Stream logs using Kinesis Data Firehose

If your log data is already being monitored by Amazon CloudWatch Logs, you can use our Kinesis Data Firehose integration to forward and enrich your log data in New Relic. Kinesis Data Firehose is a service that can stream data in real time to a variety of destinations, including our platform.

Forwarding your CloudWatch Logs or other logs compatible with a Kinesis stream to New Relic will give you enhanced ログ管理 capabilities to collect, process, explore, query, and alert on your log data.

Create the delivery stream for New Relic

To forward your logs from Kinesis Data Firehose to New Relic:

1. Make sure you have a ライセンスキー.

2. Follow the procedures to create an Amazon Kinesis Data Firehose delivery stream, and select New Relic as the destination: In the AWS Management Console, go to Amazon Kinesis.

3. Select Kinesis Data Firehose and click Create delivery stream.

4. Enter a name for the stream and select your data source.

5. When selecting the destination, click Third-party partner and then New Relic.

6. Under HTTP endpoint URL, select New Relic logs - US. This is the US endpoint (https://aws-api.newrelic.com/firehose/v1). To use the EU endpoint, complete the remaining steps in this procedure, then go to EU account configuration.

7. Paste your ライセンスキー in the API key field.

8. Ensure that Content encoding is set to GZIP.

9. Set a buffer size of 1 MiB under Buffer hints/Buffer size. (Note that the Buffer hints section is folded by default. For more information, see the buffer settings section.

10. Configure and review the remaining metadata settings.

Any optional key/value pairs you add in the AWS Management Console will result in attribute/value pairs you can use in New Relic. Kinesis Data Firehose includes these key-value pairs in each HTTP call. These Kinesis Firehose parameters can help you identify and organize your destinations.

Configure buffer size and interval

When selecting our Kinesis Firehose integration for logs in AWS, the wizard hides a small section called Buffer hints at the bottom of the screen. This section contains two very important configuration options: Buffer size and Buffer interval.

The default values are 5 MiB for Buffer size, and 60 seconds for Buffer interval. This means that Firehose accumulates logs until either they contain 5 MiB of data, or until 60 seconds have passed since the last time they were flushed to New Relic.

Why do we recommend this? New Relic accepts at most payloads of 1 MB (1.000.000 bytes) for an individual HTTP POST request to its Logs API. If the size of the accumulated logs exceeds 1 MB during any 60-second accumulation period, we will reject those logs with a 413 HTTP error.

The delivery stream allows configuring, at a minimum, a buffer size of 1 MiB (1.048.576 bytes). Even though this value is slightly greater than the 1MB allowed by New Relic's Logs API, you can use GZIP compression so that the resulting payload becomes smaller than the 1MB limit.

Configure your stream to send logs to EU accounts

If you have a New Relic account in our EU data center:

1. Follow standard procedures to create your delivery stream.

2. Once your stream has been created, navigate to the Amazon Kinesis Console, and click Delivery streams.

3. Click the name of your newly created delivery stream, and select the Configuration tab.

4. Next to the Destination settings section, click Edit.

5. Update the value for HTTP endpoint URL to our Firehose endpoint for EU accounts:

   https://aws-api.eu.newrelic.com/firehose/v1

   コピー

6. Click Save changes to ensure your stream is updated.

View log data

If everything is configured correctly and your data is being collected, you should see log data in both of these places:

• Our logs UI
• Our tools for running NRQL queries. For example, you can execute a query like this:

SELECT * FROM Log

If no data appears after you enable our log management capabilities, follow our standard log troubleshooting procedures.

What's next?

Explore logging data across your platform with our logs UI.

• Get deeper visibility into both your application and your platform performance data by forwarding your logs with our logs in context capabilities.
• Set up alerts.
• Query your data and create dashboards.

Disable log forwarding

To disable log forwarding capabilities, follow standard procedures in Kinesis Data Firehose documentation. You do not need to do anything else in New Relic.