---
title: Security Bulletin NR23-01 — Security Advisory
source: https://docs.newrelic.com/docs/security/new-relic-security/security-bulletins/security-bulletin-nr23-01-security-advisory
---

## Investigation conclusion: January 31, 2024 [#conclusion]

This is our final update to this security bulletin describing our November 2023 security incident involving unauthorized access to our staging environment. New Relic, leading cyber experts, and forensic firms conducted an extensive investigation into the incident, and together have provided updates to this security bulletin as information became available. The investigation has concluded and we are sharing additional information about our findings.

## Background [#background-details]

In November 2023, New Relic became aware of unauthorized access to our staging environment—an internal environment that provides visibility into, and information relating to, our customers’ use and operation of our services for troubleshooting purposes (“Staging Environment”). The Staging Environment maintains New Relic’s own observability data, including logs, events, traces and other diagnostic files, ensuring that we have visibility in the event of a failure in our customer facing Production environment. Notably, telemetry and application data sent to New Relic by our customers in their use of the New Relic platform does not reside in our Staging Environment.

Upon learning of the unauthorized access to the Staging Environment, we took immediate action to assess the integrity of internal applications, systems, and infrastructure. We activated our incident response plan and engaged several third-party cybersecurity experts to conduct a thorough investigation into the impact to our customers and our business.

During the course of the investigation, we learned the unauthorized actor used stolen credentials in connection with a single New Relic employee account to gain access to the Staging Environment. The unauthorized access occurred shortly before the completion of a planned migration of the remainder of our employees to our enhanced [User management model](https://docs.newrelic.com/docs/accounts/accounts-billing/new-relic-one-user-management/introduction-managing-users/) for added security.

New Relic immediately revoked access to the compromised employee account. We also analyzed the potential impact to customers by searching the Staging Environment for passwords, API keys, user identifiers, including usernames and other customer data. Our investigation also confirmed there was no lateral movement from the Staging Environment to any customer accounts in separate environments or to New Relic’s Production environment.

## Additional steps to harden environment [#additional-hardening]

We took additional steps to remediate the impact of the incident, including by redacting secrets out of our logging rules, and took steps to further harden our systems, such as implementing additional layers of technical controls, enhancing our network access controls, and accelerating the migration of our remaining employee users to our enhanced [User management model](https://docs.newrelic.com/docs/accounts/accounts-billing/new-relic-one-user-management/introduction-managing-users/). These additional steps were taken and completed.

## Incident findings [#incident-findings]

Our investigation into the Staging Environment incident is complete and has revealed the following:

-   The unauthorized actor utilized a single New Relic employee account to gain access to New Relic’s Staging Environment.
-   All activity by the unauthorized actor within New Relic’s Staging Environment has been comprehensively identified and reviewed by New Relic and our industry-leading forensic firms.
-   Between October 24 and November 15, 2023, the unauthorized actor executed specific search queries and exfiltrated these query results from the Staging Environment.
-   The last observed unauthorized activity in the Staging Environment was on November 16, 2023. There is no indication of persistent access by the unauthorized actor in New Relic’s Staging Environment.
-   A very small percentage of our customers were impacted by the search queries executed by the unauthorized actor.
-   There is no indication of lateral movement from our Staging Environment to any customers’ New Relic accounts in the separate production environment or to New Relic’s production infrastructure.

## Tactics, techniques, and procedures (TTPs) [#ttps]

In support of our industry and community as a whole, we are sharing the tactics, techniques, and procedures (TTPs) utilized by this unauthorized actor so that our community can leverage this information to identify potential risk to their environments. These TTPs were discovered and confirmed through the investigation performed by New Relic in partnership with forensic and cybersecurity experts.

The unauthorized actor used the following tactics, techniques, and procedures (TTPs):

-   Credential stuffing;
-   Use of Protonmail (proton.me) for communication;
-   Use of VPN services including NordVPN for access to public services; and
-   Programmatic Data Extraction using APIs.

## Eradication and remediation efforts [#eradication-efforts]

We understand that the best defense starts right here at New Relic. New Relic took a number of actions to eradicate the unauthorized actor’s access during the incident, including:

-   Revoking access to the compromised employee account immediately;
-   Blocking indicators of compromise associated with the attack;
-   Further hardening access controls and credential theft defenses, leveraging an industry-leading security toolset; increasing our capacity to monitor security across our entire enterprise; and
-   Providing additional cyber education and awareness to our employees.

## Customer recommendations [#customer-recommendations]

We have completed our proactive outreach to customers whose accounts were impacted from this incident in order to help them understand the impact and to communicate suggested remediation steps. If you have not received specific instructions regarding your systems, there is no action you need to take. Customers should review our [Security bulletins](https://docs.newrelic.com/docs/security/security-privacy/information-security/security-bulletins/) and [Security guides](https://docs.newrelic.com/docs/security/security-privacy/data-privacy/data-privacy-new-relic/) for best practices. There are no additional measures you need to take beyond what has already been communicated with you.

We regret any inconvenience this incident caused for our customers. Our CEO, CTO, and CISO are aligned on the future state of security at New Relic. They share the same commitment to making broad improvements to our security posture, and specifically to preventing the same type of incident from occurring in the future. We have talked to many customers, including those  not impacted directly by this incident, and have shared both our commitment to do better and the significant enhancements we have made to our security posture. We will continue making long term investments to earn back the trust of our customers. We deeply appreciate the understanding and support that customers have shown.To all of our customers—we look forward to our continued work together.

## Previous updates [#previous-updates]

**Incident report update: December 1, 2023**

Following considerable progress in our investigation, we are now in a more informed position to share with our customers additional details about the ongoing investigation and what we have learned.

### What happened—initial attack on New Relic staging environment [#what-happened]

Two weeks ago, New Relic became aware of unauthorized access to our staging environment, an internal environment that provides visibility into how our customers are using New Relic and certain logs. Telemetry and application data sent to New Relic by our customers in their use of the New Relic platform does not reside in our staging environment.

We immediately launched an investigation and discovered that an unauthorized actor used stolen credentials and social engineering in connection with a New Relic employee account. The unauthorized actor used the stolen credentials to gain access to our staging environment, where they were able to view certain data pertaining to our customers’ use of New Relic. **Customers confirmed to have been impacted by this incident have been notified with recommended next steps.** There is no indication of lateral movement from our staging environment to any customers’ New Relic accounts in the separate production environment or to New Relic’s production infrastructure.

As a result of the steps we have taken, we can also confirm that the unauthorized access to our staging environment is contained, and we see no further signs of unauthorized access or activity in our staging environment. We will continue to proactively reach out to our customers if we identify further information that may be relevant to them over the course of our ongoing investigation.

### Steps we have taken [#steps-taken]

Since learning of this incident, we took immediate action to assess the integrity of internal applications, systems, and infrastructure. We activated our incident response plan and engaged several third-party cybersecurity experts to conduct a thorough investigation into the impact to our customers and our business. Our continuity of operations and ability to serve our customers were not disrupted by this incident and continue as normal.

Our security team revoked access to the compromised employee account immediately. We have taken steps to implement additional layers of technical controls, enhance network access controls, and eliminate the attack method used to access New Relic’s staging environment. Leading cyber experts and forensics firms continue to be engaged to aid in our ongoing investigation.

We have taken this opportunity to further harden access controls and credential theft defenses, leveraging an industry-leading security toolset. We have also increased capacity to monitor security across our entire enterprise, all in order to ensure comprehensive visibility into our security posture.

### Additional information concerning potential access to customer accounts discovered during our investigation [#additional-information]

Over the course of our investigation, we observed similar indicators of compromise (IOCs) accessing a small number of customers’ New Relic accounts. Out of an abundance of caution, we proactively responded by rotating passwords and removing user API keys for the suspected compromised user accounts. Based on our investigation to date, **there is no evidence to suggest the identified log-in credentials were acquired as a result of the attack on New Relic’s staging environment.** It appears the credentials were harvested in recent large-scale social engineering and credential compromise attacks, which may have put these New Relic user accounts at risk. In cases where we identify this suspected access, we are proactively reaching out to these customers.

### Steps you can take to prevent credential-based compromises [#steps-you-can-take]

Although our investigation is ongoing, we aim to take a security forward posture by sharing our learnings to support the broader New Relic community. Our goal is to assist our customers in enhancing their own security posture and provide assistance where applicable to any investigations that may be necessary.

New Relic offers automatic controls over how users are added to New Relic, how they're managed, and how they log in. New Relic also makes available SAML, SSO, and SCIM provisioning, which is available [here](https://docs.newrelic.com/docs/accounts/accounts-billing/new-relic-one-user-management/introduction-saml-scim/). Additionally, customers configured with SAML, SSO, and SCIM, are strongly encouraged to enable MFA. If you are not taking advantage of these features, avoid reusing passwords and ensure that you regularly rotate your passwords.

We also recommend that you remain vigilant and monitor your account for suspicious activity. For example, as an additional security measure, you should regularly audit the changes made in your New Relic environment - particularly when you suspect unusual activity. New Relic makes such functionality readily available for every customer. Customers should also use automatically generated meta-events, such as [NrAuditEvent](https://docs.newrelic.com/docs/accounts/accounts/account-maintenance/query-account-audit-logs-nrauditevent/) and NrdbQuery to understand what actions your users are taking and which telemetry they are querying. Additionally, we encourage you to review our [Security bulletins](https://docs.newrelic.com/docs/security/security-privacy/information-security/security-bulletins/) and [Security guides](https://docs.newrelic.com/docs/security/security-privacy/data-privacy/data-privacy-new-relic/#product-security) for best practices.

We understand that you may have additional questions as a result of this notice. Please open a support case under security or contact our Customer Support team at [supportforum@newrelic.com](mailto:supportforum@newrelic.com) if you have further questions. They will be able to assist you or connect you with the appropriate team.

As our investigation continues, we will continue to share information with appropriate parties and the larger New Relic customer community.

Trust and transparency are paramount in our business, and we know the security of our systems is an important part of earning and keeping your trust. We apologize to our customers that this happened, and we deeply appreciate the partnership that we have forged with you. We will continue to make security investments in our infrastructure and product offering to maintain a strong security posture for our New Relic community.

**Original bulletin text: November 22, 2023**

**Release date:** November 22, 2023

**Vulnerability identifier:** NR23-01

We value our New Relic community and want to make our customers aware of a recent cybersecurity incident that we are working diligently to investigate with the support of third-party cybersecurity experts.

**Customers will be directly contacted if there are any specific actions required of you.** To be clear, if you do not hear from us, there is no action you need to take at this time.

As always, we recommend that you remain vigilant and monitor your account for suspicious activity. Additionally, we encourage you to review [Security Guides](https://docs.newrelic.com/docs/security/security-privacy/data-privacy/data-privacy-new-relic/#product-security) for best practices as well as our [Security Bulletins](https://docs.newrelic.com/docs/security/security-privacy/information-security/security-bulletins/) for updates.

We will continue to provide relevant updates as we have more information to share.