Understanding your ingest pipeline

In this section, you'll explore the ingest pipeline architecture, which visually represents how your telemetry data flows through the system. You'll see how using gateway and cloud rules can impact data volume and optimize your data management strategy.

Components of your ingest pipeline

1. MELT sources:

   • Metrics, Events, Logs, and Traces (MELT): These are the primary types of observability data you collect from various sources. They form the starting point of your ingest pipeline.

2. Data flow lines:

   • Lines emanate from each MELT source, representing the data being sent to the NRDB. Each line is annotated with a data volume, measured in millions (M), indicating the amount of data volume in transit.

3. Gateway:

   • Positioned between MELT sources and cloud rules, the gateway acts as a filter, applying your user-defined rules to drop low-value data before it exits your network. This reduces the data volume sent to the NRDB, optimizing storage and egress costs.

4. Cloud rules:

   • Cloud rules are applied within the New Relic Cloud, further filtering data before it reaches the NRDB. The diagram indicates the number of operational cloud rules.

5. NRDB:

   • The New Relic Database (NRDB) is the final destination for processed data. It stores the refined telemetry data, ready for your analysis and visualization.

Visual representation

• Without gateway and cloud rules:

  • The diagram shows direct lines from MELT sources to NRDB, with higher data volumes due to the absence of filtering.

• With gateway and cloud rules:

  • The diagram illustrates reduced data volumes as lines pass through the gateway and cloud rules. Only telemetry from APM agents configured to route through the gateway is processed.