No data appears: AWS CloudWatch metric streams

Problem

You've followed the steps to link your AWS account, configured the AWS CloudWatch metrics stream and AWS Kinesis Data Firehose, and you still don't see the expected metrics in New Relic.

Solutions

No metrics or errors appear on New Relic

If you are not seeing data in New Relic once the AWS CloudWatch Metric Stream has been connected to AWS Kinesis Data Firehose, then follow the steps below to troubleshoot your configuration:

1. Check that the Metric Stream is in a state of Running via the AWS console or API. Please refer to AWS Troubleshooting docs for additional details.

2. Check the Metric Stream metrics under AWS/CloudWatch/MetricStreams namespace. You will see a count of metric updates and errors per Metric Streams. This indicates that the Metric Stream is successfully emitting data.

3. If you see errors, confirm the IAM role specified in the Metric Streams configuration grants the CloudWatch service principal permissions to write and targets the intended Firehose resource.

4. Check the Monitoring tab of the Kinesis Data Firehose in the Kinesis console to see if the Firehose is successfully receiving data.

5. You can enable CloudWatch error logging on your Kinesis Data Firehose to get more detailed information for debugging issues. Refer to Amazon Kinesis Data Firehose official documentation for more details.

6. Confirm that you have configured your Kinesis Data Firehose with the correct destination details:

   • Ensure the license key contains your 40 hexadecimal character license key.
   • Ensure the right data center US or EU has been selected for your New Relic account. Tip: If the license_key starts with “eu” then you need to select the EU data center.

7. Check that your Kinesis Data Firehose has permissions to write to the configured destination. For example, the S3 bucket policy allows to write.

8. Check official AWS Firehose docs on troubleshooting usage and implementation:

   • https://docs.aws.amazon.com/firehose/latest/dev/http_troubleshooting.html
   • https://docs.aws.amazon.com/firehose/latest/dev/httpdeliveryrequestresponse.html
   • https://docs.aws.amazon.com/firehose/latest/dev/retry.html

9. If you are still experiencing issues, please contact AWS support for further assistance by writing up a Support Case. Include the AWS Firehose stream information and the target endpoint, issue time range, and any error messages you have received.

Missing metrics for certain AWS namespaces

New Relic doesn't apply any filter on the metrics received from the AWS CloudWatch metric stream.

If you're expecting certain metrics to be ingested and they aren't, verify the following:

• Make sure there’s no Inclusion or Exclusion filter in your CloudWatch Metric Stream.
• Make sure metrics are available in AWS CloudWatch and can be queried in the AWS CloudWatch interface. For some specific AWS services, such as ECS/ EKS container insights, enhanced monitoring needs to be explicitly enabled in AWS-side before getting access to the metrics.

Metric values discrepancies between AWS CloudWatch and New Relic

Metrics are processed, mapped, and stored as received from AWS CloudWatch metric stream. Some discrepancies might be observed when comparing AWS CloudWatch and New Relic dashboards. On limited scenarios, AWS CloudWatch applies specific functions and logic before rendering the metrics.

These guidelines should help understand the root cause of the discrepancy:

• Check that the same function is used on the metrics (for example average, min, max).
• On the New Relic side, make sure you filter the same timestamp or timeframe (considering the time zone) to show the exact same time as in AWS CloudWatch.
• When using time series, the New Relic user interface might perform some rounding based on intervals.

You can get a list of the raw metric received by time using a query like this one (note that no function is applied to the selected metric):

FROM Metric
SELECT aws.outposts.InstanceTypeCapacityUtilization
WHERE collector.name = 'cloudwatch-metric-streams'

Remember that AWS fixes the maximum resolution (1 minute, 5 minutes, etc.) for every metric reported in AWS CloudWatch.

AWS Metric Streams Operation

You can see the state of the Metric Stream(s) in the Streams tab in the CloudWatch console. In particular, a Metric Stream can be in one of two states: running or stopped.

• Running: The stream is running correctly. Even if it's running, there may not be any metric data being streamed due to the configured filters.
• Stopped: The stream has been explicitly set to the halted state (not because of an error). This state is useful to temporarily stop the streaming of data without deleting the configuration.

Errors in the Status Dashboard

New Relic relies on the AWS Config service to collect additional metadata from resources in order to enrich metrics received via CloudWatch Metric Stream.

Make sure AWS Config is enabled in your AWS Account, and ensure the linked Role has the following permission or inline policy created:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": "config:BatchGetResourceConfig",
            "Resource": "*"
        }
    ]
}