Create pipeline rules

Pipeline rules encompass two types: gateway rules and cloud rules. Both are authored using the same method. Gateway rules apply organization-wide, while cloud rules are account-specific. Gateway rules can have specific differences or incompatibilities, particularly concerning agent and data type considerations.

You have two ways to create pipeline rules via the New Relic UI: the guided method or by writing your own NRQL query. The guided method provides a step-by-step wizard to help you configure rules, while the NRQL query option allows for custom query creation to filter data precisely. You can manage cloud rules via an API, too.

When creating a rule using the guided method, you can specify the type of data you want to filter, including metrics, events, logs, or traces. Additionally, you can decide where to deploy the rule as a cloud rule running in New Relic’s infrastructure, or as a gateway rule running in your environment.

Via guided method

Follow these steps to create a pipeline gateway rule via the guided method:

1. In New Relic, go to New Relic Control and open the Pipeline Rules tab.
2. Click Create a rule. In the wizard that appears, the guided method is selected by default.
3. Select the type of data you want to filter: Metrics, Events, Logs, or Traces, and then click Next.
4. Enter the rule name.
5. Select whether you want to drop data or attributes. If you want to drop entire records, select Drop data. If you want to drop specific attributes or properties on data that’s otherwise preserved, select Drop attributes.
6. Select Cloud rule or Gateway rule depending on whether you want to deploy the rule to New Relic’s cloud, or to the gateway cluster(s) hosted in your own network.
7. Optionally, add a friendly description for your rule.
8. For cloud rules, ensure that the current account is selected. For gateway, the rule will apply to all data sent through your gateway(s) in the entire organization.
9. Depending on the telemetry data type you select earlier, perform the following action: For metrics:
   • Select the metric for which you want to drop data.
   • Filter data using the NRQL WHERE clause to scope the queried data. The WHERE clause allows you to add conditional filtering across various data types.
   • If you are dropping attributes, use the drop-down menu to specify the metric attributes. For events:
   • Select the event for which you want to drop data.
   • Filter data using the NRQL WHERE clause to scope the queried data.
   • If you are dropping attributes, use the drop-down menu to specify the event attributes. For logs:
   • Select the partition for which you want to drop data.
   • Filter data using the NRQL WHERE clause to scope the queried data. Example for log filtering: level = 'INFO' This allows you to narrow down your query to specific conditions, such as filtering logs by their log level, error types, or other relevant attributes.
   • If you are dropping attributes, use the drop-down menu to specify the attributes. For traces:
   • Filter data using the NRQL WHERE clause to scope the queried data.
   • If you are dropping attributes, use the drop-down menu to specify the attributes.
10. Save the rule. The rule is deployed.

By writing NRQL query

Follow these steps to create a Pipeline rule by writing your own NRQL query:

1. In New Relic, go to New Relic Control and open the Pipeline Rules tab.
2. Click Create a rule. The Create a new rule wizard appears. The guided method is selected by default.
3. Select Write a NRQL query.
4. Enter the rule name.
5. To write your own NRQL query for filtering data, click Write your query. The query builder opens up at the bottom.
6. Write your query using the SELECT keyword, and test it by clicking Run. This will show you an example of the data that will be impacted by this rule in the future when it is ingested. However, the data at-rest will not be affected by the rule that you author.
7. When satisfied, click Save rule. The SELECT keyword will be replaced by the DELETE keyword upon saving, so that the rule will drop the data that you referenced using SELECT.
8. Select Cloud rule or Gateway rule depending on whether you want to deploy the rule to New Relic’s cloud, or to the gateway cluster(s) hosted in your own network.
9. For cloud rule, you’ll also have to select the account to which you want the rule to apply.
10. Click Save. The rule is now deployed.

Caution

New Relic cannot guarantee that this functionality will completely resolve data disclosure concerns you may have. New Relic does not review or monitor how effective the rules you develop are. Always test and retest your queries and, after the drop rule is created, make sure it works as intended.

Creating rules about sensitive data can leak information about what kinds of data you maintain, including the format of your data or systems (for example, through referencing email addresses or specific credit card numbers). Rules you create, including all information in those rules, can be viewed and edited by any user with the relevant role-based access control permissions.

Next step

Implement a load balancer for your gateway cluster.