One way to manage your data ingestion is by using Pipeline Control cloud rules. To create Pipeline cloud rules, you must be on New Relic Compute usage-based pricing.

There are two categories of rules you can create:

Drop data rule
- Drop entire data types or a data subset (with optional filter) using the Drop data action, with NRQL in the form of:
  DELETE FROM DATA_TYPE_1, DATA_TYPE_2 (WHERE OPTIONAL_FILTER)
Drop attribute rule
- Drop attributes from data types (with optional filter) using the Drop attributes action, with NRQL in the form of:
  DELETE dropAttr1, dropAttr2 FROM DATA_TYPE (WHERE OPTIONAL_FILTER)
- For this type of rule, you must pass in a non-empty list of raw attributes names in the SELECT clause.

Tip

Pipeline Control cloud rules only apply to data that arrives from the moment you create the rule, they don't delete data that's already been ingested.

To learn more about what data counts as billable or not, see Data ingest.

Cloud rules data scope

Use cloud rules to target the following data types:

APM-reported events
Browser-reported events
Mobile-reported events
Synthetics-reported events
Custom events (like those generated by the APM agent APIs or the Event API)
Log data (you can also use the UI to drop data)
Distributed tracing spans
Default infrastructure monitoring events and infrastructure integrations events. Some caveats:
- When you drop this data, the raw data is deleted, but the aggregated SystemSample, ProcessSample, NetworkSample and StorageSample events are still available (for more on this, see Data retention). Though still available, this data doesn't count towards ingest and is not billable.
- Raw infrastructure data is used for alerting, so if you drop that data, you can't alert on it. Because the aggregated data is still available, you may still see that data in charts with time ranges above 59 minutes.
Dimensional metrics. Some caveats:
- For metrics generated by the events-to-metrics service: Cloud rules won't work but these metrics can be stopped or attributes pruned by disabling or re-configuring the events-to-metric rule.
- Metric timeslice data can't be dropped with cloud rules. For more information about APM metric timeslice data see this doc.

NRQL restrictions

The limit on NRQL query length is 4096 characters. If it exceeds the length, the INVALID_NRQL_TOO_LONG error occurs. If you need to drop data based on a longer query that cannot be split, contact your New Relic support.
JOIN and subqueries are not supported.
You can provide a WHERE clause to select data with specific attributes.
Features such as LIMIT, TIMESERIES, COMPARE WITH, FACET, and other clauses cannot be used.
SINCE and UNTIL are not supported. If you have time-specific rules (say, drop everything until a time in the future), use WHERE timestamp < (epoch milliseconds in the future).
You can't use SINCE to drop historical data. Cloud rules only apply to data reported after the rule was created. If you need to delete data that has already been reported, refer to Delete existing data or contact New Relic support.

Audit rule history

To see who created and deleted cloud rules, query your account audit logs. The list endpoint also includes the user ID of the person who created the rule.

Cautions when dropping data

Cloud rules apply to each data point independently. For example, let's look at the following three data drop rules:

Important

When creating rules, you are responsible for ensuring that the rules accurately identify and discard the data that meets the conditions that you have established. You are also responsible for monitoring the rule, as well as the data you disclose to New Relic.

1. DELETE FROM MyEvent WHERE myAttr not in ('staging')
2. DELETE FROM MyEvent WHERE myAttr not in ('production')
3. DELETE FROM MyEvent WHERE myAttr in ('development')

These three rules are applied independently to each data point; in summary, all MyEvent events containing myAttr with any value will be dropped:

myAttr: 'staging' -> matches rule 2
myAttr: 'production' -> matches rule 1
myAttr: 'development' -> matches rules 1, 2, and 3
myAttr: 'uuid-random-string' -> matches rules 1 and 2

New Relic cannot guarantee that this functionality will completely resolve data disclosure concerns you may have. New Relic does not review or monitor how effective the rules you develop are. Always test and retest your queries and, after the drop rule is created, make sure it works as intended.

Creating rules about sensitive data can leak information about what kinds of data you maintain, including the format of your data or systems (for example, through referencing email addresses or specific credit card numbers). Rules you create, including all information in those rules, can be viewed and edited by any user with the relevant role-based access control permissions.

Only new data will be dropped. Existing data cannot be edited or deleted.

Managing cloud rules

To create and edit rules, you can either use the Pipeline Control UI or the NerdGraph API explorer (one.newrelic.com > Apps > NerdGraph API explorer).

Caution

Use caution when deciding to drop data. The data you drop can't be recovered. For more details on potential issues, see Caution notes.

Use Case Examples

Let's say you notice you have some event types being sent to New Relic that are not important to you. Also, stopping the source from sending those event types quickly is unrealistic, requiring changes to agents and/or API instrumentation. Using a cloud rule is an easier way to accomplish the same goal.

Here is an example NerdGraph call that drops two event types: Event1 and Event2.

mutation {
  entityManagementCreatePipelineCloudRule(
    pipelineCloudRuleEntity: {
      description: "Event1 and Event2 are unimportant, see ticket DM-1234"
      name: "Drop all data for Event1 and Event2"
      nrql: "DELETE FROM Event1, Event2"
      scope: { id: "your_nr_account_id", type: ACCOUNT }
    }
  ) {
    entity {
      id
      name
      nrql
    }
  }
}

Let’s say you have a high volume custom event type that arrives from multiple sources. If you don't find all of that data important, you can use a cloud rule. Here is an example of a cloud rule that filters out events based on specific criteria.

mutation {
  entityManagementCreatePipelineCloudRule(
    pipelineCloudRuleEntity: {
      description: "Drops all data for MyCustomEvent that comes from the LoadGeneratingApp in the dev environment, because there is too much and we don’t look at it"
      name: "Drop MyCustomEvent from LoadGeneratingApp in dev"
      nrql: "DELETE FROM MyCustomEvent WHERE appName='LoadGeneratingApp' AND environment='development'"
      scope: { id: "your_nr_account_id", type: ACCOUNT }
    }
  ) {
    entity {
      id
      name
      nrql
    }
  }
}

Let's say you noticed an event has attributes that contain Personally Identifiable Information (PII). You are working to update your services to stop sending the data, but until then, you need to cease storing further PII in New Relic. Although you could drop all of the data as it comes in the door with Drop data, the rest of the data still provides value. Therefore, you can register a cloud rule to remove only the offending PII from your data:

mutation {
  entityManagementCreatePipelineCloudRule(
    pipelineCloudRuleEntity: {
      description: "Removes the user name and email fields from MyCustomEvent"
      name: "Drop username and email from MyCustomEvent"
      nrql: "DELETE userName, userEmail FROM MyCustomEvent"
      scope: { id: "your_nr_account_id", type: ACCOUNT }
    }
  ) {
    entity {
      id
      name
      nrql
    }
  }
}

Verify your rule works

After you create a cloud rule, you might wish to verify that it is working as expected. The rule should take effect quickly after a successful registration, so try running a TIMESERIES version of the query you registered to see that the data drops off.

Note: Timeseries data is rendered with event time (not processing time) as the x-axis. Since New Relic accepts data with a timestamp up to twenty-four hours in the future, you might see some data that was sent to New Relic before the rule was created but with an event timestamp past rule creation.

Cloud rule type	NRQL
`Drop data`	Cloud rule NRQL: `DELETE FROM MyEvent WHERE foo = bar` Validation NRQL: `SELECT count(*) FROM MyEvent WHERE foo = bar TIMESERIES` This should drop to 0. To verify that it did not affect any thing else, invert the `WHERE` clause.
`Drop attributes`	Cloud rule NRQL: `DELETE dropAttr1, dropAttr2 FROM MyEvent WHERE foo = bar` Validation NRQL: `SELECT count(dropAttr1), count(dropAttr2) FROM MyEvent WHERE foo = bar TIMESERIES` Both lines should drop to 0. To verify that it did not affect events that contained these attributes and still should, invert the `WHERE` clause.

Cloud rule type

NRQL

Drop data

Cloud rule NRQL:

DELETE FROM MyEvent WHERE foo = bar

Validation NRQL:

SELECT count(*) FROM MyEvent WHERE foo = bar TIMESERIES

This should drop to 0. To verify that it did not affect any thing else, invert the WHERE clause.

Drop attributes

Cloud rule NRQL:

DELETE dropAttr1, dropAttr2 FROM MyEvent WHERE foo = bar

Validation NRQL:

SELECT count(dropAttr1), count(dropAttr2) FROM MyEvent WHERE foo = bar TIMESERIES

Both lines should drop to 0. To verify that it did not affect events that contained these attributes and still should, invert the WHERE clause.

NerdGraph Examples

Create cloud rules

Drop data:

mutation {
  entityManagementCreatePipelineCloudRule(
    pipelineCloudRuleEntity: {
      description: "Since we only care about MyEvent in staging and production, let's drop all MyEvent data in the test environment"
      name: "Drop MyEvent in test environment"
      nrql: "DELETE FROM MyEvent where environment = 'test'"
      scope: { id: "your_nr_account_id", type: ACCOUNT }
    }
  ) {
    entity {
      id
      name
      nrql
    }
  }
}

Drop attributes:

mutation {
  entityManagementCreatePipelineCloudRule(
    pipelineCloudRuleEntity: {
      description: "Since we only care about MyEvent in staging and production, let's drop all MyEvent data in the test environment"
      name: "Drop MyEvent in test environment"
      nrql: "DELETE jvmId, targetAttr FROM MyEvent where environment = 'test'"
      scope: { id: "your_nr_account_id", type: ACCOUNT }
    }
  ) {
    entity {
      id
      name
      nrql
    }
  }
}

Delete a cloud rule

mutation {
  entityManagementDelete(
    id: "MTAyNTY1MHxOR0VQfFBJUEVMSU5FX0NMT1VEX1JVTEV8MDE5NWI0NDYtNjk5My03NGE5LWEyYjktMzBjMzQ1ODM0NTUz"
  ) {
    id
  }
}

View cloud rules

Get a single cloud rule:

{
  actor {
    entityManagement {
      entity(
        id: "MTAyNTY1MHxOR0VQfFBJUEVMSU5FX0NMT1VEX1JVTEV8MDE5NWI0M2UtYmFhNy03NDk3LWI0N2ItNjUyMmEzZDFmZTFi"
      ) {
        id
        ... on EntityManagementPipelineCloudRuleEntity {
          id
          name
          description
          nrql
          metadata {
            createdBy {
              id
            }
            createdAt
          }
        }
      }
    }
  }
}

List all cloud rules:

{
  actor {
    entityManagement {
      entitySearch(query: "type = 'PIPELINE_CLOUD_RULE'") {
        entities {
          id
          type
          ... on EntityManagementPipelineCloudRuleEntity {
            id
            name
            nrql
          }
          metadata {
            createdBy {
              id
            }
          }
        }
      }
    }
  }
}

Non-droppable events and attributes

You cannot drop the following events and attributes using cloud rules:

Event type	Description
`NrAuditEvent`	Important auditing information about the API usage that can’t be dropped.
`NrIntegrationError`	Important information about issues between you and New Relic.

Attribute	Description
Any `nr.` prefixed attribute	Used internally by New Relic
`timestamp`	Integral to all data types
`appId`	Integral to all data types
`metricName`	Integral to the metric data type
`instrumentation.provider`	Used for billing and usage data

Learn more

Recommendations for learning more:

NerdGraph basics and terminology
NRQL basics
Browse the Support Forum for community discussions about cloud rules.
For a deep dive into managing data ingest for a complex organization, see Data ingest governance.

Drop data using Pipeline Control cloud rules

Tip

Cloud rules data scope

NRQL restrictions

Audit rule history

Cautions when dropping data

Important

Managing cloud rules

Caution

Use Case Examples

Drop two event types

Drop events meeting certain criteria

Drop sensitive attributes while maintaining the rest of the data

Verify your rule works

NerdGraph Examples

Create cloud rules

Delete a cloud rule

View cloud rules

Non-droppable events and attributes

Non-droppable events

Non-droppable attributes

Learn more

Drop data using Pipeline Control cloud rules

Tip

Cloud rules data scope .css-21sua1{background:none;border:none;width:0;padding:0;}

NRQL restrictions

Audit rule history

Cautions when dropping data

Important

Managing cloud rules

Caution

Use Case Examples

Drop events meeting certain criteria

Drop sensitive attributes while maintaining the rest of the data

Verify your rule works

NerdGraph Examples

Create cloud rules

Delete a cloud rule

View cloud rules

Non-droppable events and attributes

Non-droppable events

Non-droppable attributes

Learn more

Cloud rules data scope