Set up and deploy Pipeline Control gateway

Set up and deploy the Pipeline Control gateway to manage the flow of data from your monitored entities to New Relic.

Considerations for using gateway

As you integrate and use the gateway component, here are some important considerations for you to keep in mind:

Attribute and data management

Attribute support: Some attributes may not be supported by the gateway. For detailed information, refer to Data dropping at gateway.
Data/attribute handling: The gateway forwards your requests for dropping data or attributes for metric time slice data, sql_trace_data, and trace_sample_data directly to New Relic without modification.

Parsing and data integration

Log parsing: The gateway does not include Grok parsing for logs. Grok parsing is a method used to extract structured data from unstructured log messages using predefined patterns and regular expressions. You may need alternative methods to structure log data for querying and visualization.
Prometheus integration: The gateway does not support Prometheus remote write for metrics, which may affect your integration with Prometheus-based systems.

Licensing and compliance

License key management: The gateway does not accommodate multiple license keys from native OpenTelemetry Protocol (OTLP) sources, such as agent collectors or APM SDKs. Consider this when managing your licenses.
Compliance standards: The gateway isn't FedRAMP-compliant, which may affect your use if you need to meet federal security standards.

Query functionality

Query functions: The gateway does not support certain functions in the WHERE clause for metric queries. The functions include:

%[type]
%[count]
%[sum]
%[min]
%[max]
%[latest]

Example query: FROM Metric SELECT * WHERE (%[type]='summary' AND %[count]=0) OR (%[type]='gauge' AND %[count]=1 AND %[latest]=0)

Set up Pipeline Control gateway

Screenshot of the button to start Pipeline Control gateway installation

To set up Pipeline Control gateway:

In your New Relic account, in Integrations & Agents, click Set up Pipeline Control, or search for gateway and select Pipeline Control gateway.
Ensure the account selected is the one you want to use for Pipeline Control. If not, click Select an account, select the account of your choice, and then click Continue.
To enter your credentials, provide an existing license key or create a new key, and then click Continue.
Select an existing Kubernetes cluster using the drop-down menu, or create a new one.
If you create a new cluster, enter the cluster name and namespace. The default namespace is newrelic.
Download the configuration file, values-newrelic-gateway.yaml, to a directory.
Run the installation command shown on screen:
bash
```
$helm repo add newrelic https://helm-charts.newrelic.com && helm repo update && \
>helm upgrade --install newrelic-gateway --version <VERSION_NUMBER> -n newrelic \
>newrelic/agent-control --create-namespace --values /path/to/values-newrelic-gateway.yaml --devel
```
Dica
Replace <VERSION_NUMBER> with the actual version number of the gateway amd /path/to/values-newrelic-gateway.yaml with the directory where the configuration file is located.
Click Continue.
Click Test connection and wait for a successful connection.

When you create your first gateway cluster, it sets up a unified configuration across all gateway clusters within your organization.

To view details of your gateway setup, go to New Relic Control and click Gateway.

Use case: Optimize data flow across multiple regions

Imagine you're a customer managing a sophisticated observability setup with New Relic. Your operations span multiple geographic regions, each with its own unique data management challenges. Specifically, you operate Kubernetes clusters across two primary regions: Region A and Region B.

Objective: Optimize data handling and reduce unnecessary data transfer to maximize efficiency and minimize costs.

Current setup: You already have a gateway cluster (GC1) deployed. Your data-producing Kubernetes clusters in Region A are configured to route all telemetry data through GC1. This setup allows you to filter out irrelevant data, ensuring that only critical signals are sent to the New Relic backend.

Challenge with Region B: Your Kubernetes clusters in Region B also need data processing. You have considered utilizing GC1 for Region B's data, but this introduces some inefficiencies:

Increased latency: Data sent from Region B to Region A adds delay, which can impact real-time monitoring capabilities.
Higher costs: Inter-region data transfer may incur additional egress costs, increasing operational expenses.
Solution: Deploy multiple gateway clusters:
Deploy a new gateway cluster (GC2) in Region B.
Configure your Kubernetes clusters in Region B to filter their data through GC2. This reduces the distance data travels, minimizes latency, and eliminates inter-region transfer fees.
Outcome:
Improved performance: Deploying GC2 locally in Region B can help to reduce data transmission delays, which may lead to improved reaction time for critical events.
Scalable architecture: Adding GC2 provides capacity.

This strategy of deploying multiple, region-specific gateway clusters helps maintain streamlined operations and ensures your data monitoring remains both effective and economical.

Add more clusters

To add a new cluster to your gateway setup,

Open New Relic Control, and then select Gateway.
In the clusters table, click Add a cluster.
Follow the instructions in the gateway setup.
Come back to the gateway summary page. You should see your newly added cluster.

After you have created your gateway clusters, add and deploy pipeline rules to the clusters. Refer to Create pipeline rules.