Understanding data processing in gateway

Data flow overview

Data sources

Data enters the New Relic gateway from various sources:

APM agents
Infrastructure agent
Metrics API

Data format

This incoming data is structured as complex, multi-nested JSON with numerous attributes.

Processing stages

Gateway processing: Initial data handling occurs here in your own infrastructure.
Enrichment: After the gateway, additional attributes are added AND some attribute names are changed.
Cloud rules processing: Processes data with the renamed/enriched attributes in New Relic's environment.
Final storage: Data is ultimately stored in the New Relic database (NRDB).

Considerations for writing NRQL queries for gateway rules

Renamed attributes

If you're using attributes that get renamed during the enrichment process (after gateway, before cloud rules), don't worry. New Relic automatically handles this mapping, so your NRQL queries will work correctly at both the gateway and cloud rule levels.

Added attributes

Some attributes are only added during the enrichment process (after gateway, before cloud rules). These attributes:

Can be used in NRQL queries for cloud drop rules
Cannot be used in NRQL queries for gateway drop rules (since they don't exist at that point)

Best practices

When authoring gateway rules, be aware that some attributes may not be available at the gateway level. If you need to use attributes that are only added during enrichment, consider using a cloud rule instead of a gateway rule.

Data types and attribute handling

The following table lists:

Data types for which you can create drop rules using NRQL
Data types for which drop rules aren't supported
Attributes unavailable at the gateway level
Sample queries for each supported data type

Data type	Drop rules supported?	Attributes unavailable at gateway	Sample NRQL Query
Transaction	Yes	appId, appName, containerId, entity.guid, entityGuid, host, realAgentId, transactionSubType, transactionType	`DELETE priority FROM Transaction WHERE guid = 'c2906c2e8b9f11ff'`
MyCustomEvent	Yes	appId, appName, containerId, entityGuid, host, realAgentId	`DELETE myString FROM my_event_type WHERE myFloat = 0.603`
ErrorTrace	Yes	aggregateFacet, appId, appName, applicationIds, count, entity.guid, entityGuid, error.class, message, path, exceptionClass, fingerprint, id, message, realAgentId, storageId, timestamp, transactionName, transactionUiName	`DELETE FROM ErrorTrace WHERE traceId = 'b366efe772fa6d1c8e0852558026c40e'`
TransactionError	Yes	aggregateFacet, appId, appName, containerId, entity.guid, entityGuid, host, realAgentId, transactionUiName	`DELETE FROM TransactionError WHERE error.message = 'my expected error message'`
Log	Yes	entity.guids, messageId, newrelic.logPattern, newrelic.logs.batchIndex, newrelic.source	`DELETE message FROM Log WHERE span.id = '8b583de97341d094'`
Metric	No	appId, appName, entity.guid, entityGuid, language, metricName, metricTimesliceName, newrelic.timeslice.value, scope, timestamp	Not applicable
Span	Yes	appId, appName, containerId, duration.ms, entity.guid, entity.name, entityGuid, host, id, process.id, realAgentId, trace.id	`DELETE FROM Span WHERE name = 'WebTransaction/Go/GET /log'`
SqlTrace	No	applicationIds, callCount, databaseMetricName, entity.guid, id, maxCallTime, minCallTime, path, realAgentId, sql, sqlId, storageId, timestamp, totalCallTime, uri	Not applicable
TransactionTrace	No	storageId, uri, path, agentRunId, applicationIds, duration, entity.guid, guid, id, protocolVersion, realAgentId, timestamp	Not applicable
Metric/gauge	Yes	newrelic.source (metricAPI), metricName: `{type, count, latest, max, min, sum}`	`DELETE FROM Metric WHERE metricName = 'redis_aof_rewrite_in_progress' AND value < 100`
Metric/summary	Yes	newrelic.source (metricAPI), metricName: `{type, count, max, min, sum}`	`DELETE FROM Metric WHERE scrapedTargetKind = 'user_provided'`
Metric/count	Yes	newrelic.source (metricAPI), metricName: `{type, count}`	`DELETE FROM Metric WHERE instrumentation.name = 'nri-prometheus'`
SystemSample	Yes	None	`DELETE agentName FROM SystemSample WHERE entityKey = 'vagrant'`
StorageSample	Yes	entityAndMountPoint	`DELETE FROM StorageSample WHERE inodesUsed = 161604`
NetworkSample	Yes	entityAndInterface	`DELETE instanceType,inodesTotal NetworkSample WHERE entityKey = 'vagrant'`
ProcessSample	Yes	entityAndPid	`DELETE FROM ProcessSample WHERE entityKey = 'vagrant'`
ContainerSample	Yes	entityGuid, entityType, entityId	`DELETE FROM ContainerSample WHERE agentName='ContainerSampleAgent'`