---
title: Prometheus High Availability (HA)
source: https://docs.newrelic.com/docs/infrastructure/prometheus-integrations/install-configure/prometheus-high-availability-ha
---

If you are using our Prometheus remote write integration in a high-availability (HA) configuration, you need to make sure your Prometheus servers aren't sending multiple copies of the same metrics to New Relic. This document describes how you can configure your remote write integration so that New Relic does not keep duplicated metrics.

> #### 💡 TIP
>
> This page applies only if you run your own Prometheus servers in a high-availability configuration and forward metrics with [remote write](https://docs.newrelic.com/docs/integrations/prometheus-integrations/install-configure/set-your-prometheus-remote-write-integration). If you don't already run Prometheus, the [Prometheus agent for Kubernetes](https://docs.newrelic.com/docs/infrastructure/prometheus-integrations/install-configure-prometheus-agent/install-prometheus-agent/) is a fully New Relic-managed alternative that doesn't require HA deduplication; see [Send Prometheus metric data to New Relic](https://docs.newrelic.com/docs/infrastructure/prometheus-integrations/get-started/send-prometheus-metric-data-new-relic/) to compare your options.

## Quick reference [#prometheus-labels]

New Relic deduplicates HA replicas using three values. Set all three on every replica in a high-availability cluster:

| Values               | Where you set it                   | Value across replicas in the same HA cluster | Example                  |
| -------------------- | ---------------------------------- | -------------------------------------------- | ------------------------ |
| `prometheus`         | Prometheus external label          | **Same** for all replicas                    | `monitoring-cluster`     |
| `prometheus_replica` | Prometheus external label          | **Unique** per replica                       | `replica-1`, `replica-2` |
| `prometheus_server`  | `remote_write` URL query parameter | **Same** for all replicas                    | `prod-monitoring`        |

**How it works:** New Relic organizes replicas into HA clusters based on your account, the `prometheus` label, and the `prometheus_server` parameter. In each cluster, New Relic designates one replica as the leader and stores only the data from that leader. The leader is identified by the `prometheus_replica` label.

## Set up deduplication [#set-up]

### Prometheus Operator [#kubernetes-operator]

[Prometheus Operator](https://github.com/coreos/prometheus-operator) version 0.19.0 or higher adds the `prometheus` and `prometheus_replica` external labels for you, whether you use the operator directly or via the [helm chart](https://hub.helm.sh/charts/stable/prometheus-operator):

-   `prometheus` is set to `<prometheus deployment namespace>/<prometheus deployment name>`. For example, a deployment named `prometheus-cluster1` in the `monitoring` namespace produces `monitoring/prometheus-cluster1`.
-   `prometheus_replica` is set to each replica's pod name, in the format `replica-<replica number>` (for example, `replica-1`).

Set `prometheus_server` yourself in the `remoteWrite` configuration on the `Prometheus` resource, using the same value for every replica. For example, this resource runs a two-replica HA pair with a single `remoteWrite` block, so `prometheus_server` stays consistent across both replicas:

```yaml
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: monitoring-cluster
  namespace: monitoring
spec:
  replicas: 2
  remoteWrite:
    - url: https://metric-api.newrelic.com/prometheus/v1/write?prometheus_server=prod-monitoring
      authorization:
        credentials:
          name: nr-license-key
          key: licenseKey
```

The operator then sets `prometheus` to `monitoring/monitoring-cluster` and `prometheus_replica` to each replica's pod name automatically — you only supply `prometheus_server`. If you use the `kube-prometheus-stack` helm chart, set the same `remoteWrite` block under `prometheus.prometheusSpec`.

### Standalone Prometheus [#standalone-prometheus]

Add the external labels to each replica's [configuration file](https://prometheus.io/docs/prometheus/latest/configuration/configuration/), and use an identical `remote_write` URL — including the `prometheus_server` parameter — across all replicas.

**Replica 1** (`prometheus.yml`)

```yml
global:
  external_labels:
    prometheus: monitoring-cluster
    prometheus_replica: replica-1

remote_write:
  - url: https://metric-api.newrelic.com/prometheus/v1/write?prometheus_server=prod-monitoring
    authorization:
      credentials: YOUR_LICENSE_KEY
```

**Replica 2** (`prometheus.yml`)

```yml
global:
  external_labels:
    prometheus: monitoring-cluster
    prometheus_replica: replica-2

remote_write:
  - url: https://metric-api.newrelic.com/prometheus/v1/write?prometheus_server=prod-monitoring
    authorization:
      credentials: YOUR_LICENSE_KEY
```

Only `prometheus_replica` differs between the two replicas; `prometheus` and `prometheus_server` are identical.

## Limitations [#limitations]

Keep `prometheus_server` identical across replicas. Because New Relic uses `prometheus_server` to group replicas, giving replicas _different_ values — for example, per host or per pod — places each in its own HA group. Each replica becomes the sole member of its group, is always elected leader, and retains all its data. The result is duplicate data in New Relic with no error or warning, since each replica appears as a healthy, standalone leader. To distinguish individual replicas, use the `prometheus_replica` label instead.

An account can have up to 1,500 unique Prometheus HA clusters, counted per unique combination of account, `prometheus` label, and `prometheus_server` parameter. Exceeding this limit causes data from additional HA clusters to be dropped, and New Relic generates `PrometheusHAClusterLimit` [`NrIntegrationError`](https://docs.newrelic.com/docs/data-apis/ingest-apis/metric-api/troubleshoot-nrintegrationerror-events/) events. Keeping `prometheus_server` consistent across replicas also prevents a single HA group from consuming multiple slots against this limit.

## Troubleshooting [#troubleshooting]

Prometheus Operator does not set `prometheus_server` — that value comes entirely from your `remoteWrite` configuration on the `Prometheus` resource. When all replicas share a single `remoteWrite` block (the normal case), this is consistent by default. If you inject per-replica `remoteWrite` overrides, ensure `prometheus_server` is the same across all of them.

If you still see duplicate copies of replica data, ensure you do not have [`replicaExternalLabelName` or `prometheusExternalLabelName`](https://github.com/coreos/prometheus-operator/blob/master/Documentation/api.md#prometheusspec) in your Prometheus spec or chart configuration because these overrides change the label name.

For the most consistent deduplication, keep your Prometheus `scrape_interval` at 60 seconds or less. New Relic identifies the active replica in each HA cluster from its most recently received data, and Prometheus remote write forwards new samples about once per scrape. If a cluster sends data less frequently than that, New Relic may not consistently recognize the same leader and can retain duplicated or interleaved data from more than one replica, with no error surfaced.