Monitore o Elasticsearch auto-hospedado com o OpenTelemetry

Monitore seu cluster Elasticsearch auto-hospedado instalando o OpenTelemetry Collector diretamente em servidores ou máquinas virtuais. O New Relic oferece opções de implantação flexíveis para atender à configuração da sua infraestrutura e aos requisitos de monitoramento.

Você pode escolher entre três opções de coletor:

NRDOT: Distribuição New Relic do OpenTelemetry
OTel Collector Contrib: Coletor OpenTelemetry padrão com componentes contribuídos pela comunidade
Receptor do Prometheus: para ambientes que já executam um exportador do Prometheus para Elasticsearch

Opções de instalação

Escolha a distribuição do coletor que atenda às suas necessidades:

Antes de você começar

Antes de configurar o monitoramento do Elasticsearch com o NRDOT, certifique-se de ter:

Coletor NRDOT instalado:

Você deve ter o coletor NRDOT instalado e em execução como um serviço systemd no seu host
Se não estiver instalado, siga o guia oficial de instalação do NRDOT para instalar o coletor usando o gerenciador de pacote (DEB/RPM) que configura o serviço systemd automaticamente
Privilégios de acesso necessários:
Seu New Relic
Privilégios de root ou sudo na máquina host
Privilégios de administrador de cluster do Elasticsearch com privilégio de cluster monitor ou manage (consulte a documentação de privilégios de segurança do Elasticsearch para obter detalhes)
Requisitos do sistema:
Elasticsearch versão 7.16 ou superior - Esta integração requer um cluster Elasticsearch moderno
Conectividade de rede - HTTPS de saída (porta 443) para o endpoint de ingestão OTLPda New Relic
Valores de configuração prontos:
Endpoint do Elasticsearch - Sua URL do cluster Elasticsearch (por exemplo, http://localhost:9200)
Nome do cluster - Um identificador exclusivo para seu cluster no New Relic

Configurar o monitoramento do Elasticsearch

Assim que o coletor NRDOT estiver instalado, substitua o arquivo de configuração padrão do coletor pela configuração de monitoramento do Elasticsearch. Isso habilitará a coleta de métricas do Elasticsearch. Métricas de host e logs são opcionais e podem ser adicionados separadamente.

O arquivo de configuração está localizado em: /etc/nrdot-collector/config.yaml

Dica

Faça backup da sua configuração padrão: Antes de modificar o arquivo de configuração, crie uma cópia de backup para preservar as configurações padrão:

bash

$sudo cp /etc/nrdot-collector/config.yaml /etc/nrdot-collector/config.yaml.backup

Para configurar o coletor:

Abra o arquivo de configuração com um editor de texto usando privilégios de root ou sudo:
bash
```
$sudo nano /etc/nrdot-collector/config.yaml
```

Exclua todo o conteúdo existente e substitua-o pela seguinte configuração para monitoramento do Elasticsearch:

Importante

Substitua o valor endpoint pelo endpoint do seu cluster Elasticsearch e atualize elasticsearch.cluster.name no bloco do processador com um nome exclusivo para identificar seu cluster no New Relic.

receivers:
  elasticsearch:
    endpoint: "http://localhost:9200"
    collection_interval: 15s
    metrics:
      elasticsearch.os.cpu.usage:
        enabled: true
      elasticsearch.cluster.data_nodes:
        enabled: true
      elasticsearch.cluster.health:
        enabled: true
      elasticsearch.cluster.in_flight_fetch:
        enabled: true
      elasticsearch.cluster.nodes:
        enabled: true
      elasticsearch.cluster.pending_tasks:
        enabled: true
      elasticsearch.cluster.shards:
        enabled: true
      elasticsearch.cluster.state_update.time:
        enabled: true
      elasticsearch.index.documents:
        enabled: true
      elasticsearch.index.operations.merge.current:
        enabled: true
      elasticsearch.index.operations.time:
        enabled: true
      elasticsearch.node.cache.count:
        enabled: true
      elasticsearch.node.cache.evictions:
        enabled: true
      elasticsearch.node.cache.memory.usage:
        enabled: true
      elasticsearch.node.shards.size:
        enabled: true
      elasticsearch.node.cluster.io:
        enabled: true
      elasticsearch.node.documents:
        enabled: true
      elasticsearch.node.disk.io.read:
        enabled: true
      elasticsearch.node.disk.io.write:
        enabled: true
      elasticsearch.node.fs.disk.available:
        enabled: true
      elasticsearch.node.fs.disk.total:
        enabled: true
      elasticsearch.node.http.connections:
        enabled: true
      elasticsearch.node.ingest.documents.current:
        enabled: true
      elasticsearch.node.ingest.operations.failed:
        enabled: true
      elasticsearch.node.open_files:
        enabled: true
      elasticsearch.node.operations.completed:
        enabled: true
      elasticsearch.node.operations.current:
        enabled: true
      elasticsearch.node.operations.get.completed:
        enabled: true
      elasticsearch.node.operations.time:
        enabled: true
      elasticsearch.node.shards.reserved.size:
        enabled: true
      elasticsearch.index.shards.size:
        enabled: true
      elasticsearch.os.cpu.load_avg.1m:
        enabled: true
      elasticsearch.os.cpu.load_avg.5m:
        enabled: true
      elasticsearch.os.cpu.load_avg.15m:
        enabled: true
      elasticsearch.os.memory:
        enabled: true
      jvm.gc.collections.count:
        enabled: true
      jvm.gc.collections.elapsed:
        enabled: true
      jvm.memory.heap.max:
        enabled: true
      jvm.memory.heap.used:
        enabled: true
      jvm.memory.heap.utilization:
        enabled: true
      jvm.threads.count:
        enabled: true
      elasticsearch.index.segments.count:
        enabled: true
      elasticsearch.index.operations.completed:
        enabled: true
      elasticsearch.node.script.cache_evictions:
        enabled: false
      elasticsearch.node.cluster.connections:
        enabled: false
      elasticsearch.node.pipeline.ingest.documents.preprocessed:
        enabled: false
      elasticsearch.node.thread_pool.tasks.queued:
        enabled: false
      elasticsearch.cluster.published_states.full:
        enabled: false
      jvm.memory.pool.max:
        enabled: false
      elasticsearch.node.script.compilation_limit_triggered:
        enabled: false
      elasticsearch.node.shards.data_set.size:
        enabled: false
      elasticsearch.node.pipeline.ingest.documents.current:
        enabled: false
      elasticsearch.cluster.state_update.count:
        enabled: false
      elasticsearch.node.fs.disk.free:
        enabled: false
      jvm.memory.nonheap.used:
        enabled: false
      jvm.memory.pool.used:
        enabled: false
      elasticsearch.node.translog.size:
        enabled: false
      elasticsearch.node.thread_pool.threads:
        enabled: false
      elasticsearch.cluster.state_queue:
        enabled: false
      elasticsearch.node.translog.operations:
        enabled: false
      elasticsearch.memory.indexing_pressure:
        enabled: false
      elasticsearch.node.ingest.documents:
        enabled: false
      jvm.classes.loaded:
        enabled: false
      jvm.memory.heap.committed:
        enabled: false
      elasticsearch.breaker.memory.limit:
        enabled: false
      elasticsearch.indexing_pressure.memory.total.replica_rejections:
        enabled: false
      elasticsearch.breaker.memory.estimated:
        enabled: false
      elasticsearch.cluster.published_states.differences:
        enabled: false
      jvm.memory.nonheap.committed:
        enabled: false
      elasticsearch.node.translog.uncommitted.size:
        enabled: false
      elasticsearch.node.script.compilations:
        enabled: false
      elasticsearch.node.pipeline.ingest.operations.failed:
        enabled: false
      elasticsearch.indexing_pressure.memory.limit:
        enabled: false
      elasticsearch.breaker.tripped:
        enabled: false
      elasticsearch.indexing_pressure.memory.total.primary_rejections:
        enabled: false
      elasticsearch.node.thread_pool.tasks.finished:
        enabled: false
processors:
  memory_limiter:
    check_interval: 60s
    limit_mib: ${env:NEW_RELIC_MEMORY_LIMIT_MIB:-100}
  cumulativetodelta: {}
  attributes/cluster_state_aggregate:
    include:
      match_type: strict
      metric_names:
        - elasticsearch.cluster.state_update.time
    actions:
      - key: type
        action: delete
      - key: state
        action: delete
  filter/critical_operations:
    metrics:
      datapoint:
        # Filters to keep only: query, index, get, merge
        # Affects only 4 metrics: *.operations.completed and *.operations.time
        # All other metrics pass through unchanged
        - 'attributes["operation"] == "query" or attributes["operation"] == "index" or attributes["operation"] == "get" or attributes["operation"] == "merge" or attributes["operation"] == nil'
  resource/cluster_name_override:
    attributes:
      - key: elasticsearch.cluster.name
        value: "<elasticsearch-cluster-name>"
        action: upsert
  resourcedetection:
    detectors: [ system ]
    system:
      resource_attributes:
        host.name:
          enabled: true
        host.id:
          enabled: true
        os.type:
          enabled: true
  batch:
    timeout: 30s
    send_batch_size: 2048
    send_batch_max_size: 4096
  attributes/cardinality_reduction:
    actions:
      - key: process.pid
        action: delete
      - key: process.parent_pid
        action: delete
  transform/metadata_nullify:
    metric_statements:
      - context: metric
        statements:
          - set(description, "")
          - set(unit, "")
exporters:
  otlphttp:
    endpoint: ${env:NEWRELIC_OTLP_ENDPOINT}
    headers:
      api-key: ${env:NEWRELIC_LICENSE_KEY}
    compression: gzip
    timeout: 30s
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s
service:
  pipelines:
    metrics/elasticsearch:
      receivers: [elasticsearch]
      processors: [memory_limiter, resourcedetection, resource/cluster_name_override, attributes/cardinality_reduction, filter/critical_operations, attributes/cluster_state_aggregate, cumulativetodelta, transform/metadata_nullify, batch]
      exporters: [otlphttp]

(Opcional) Para Elasticsearch seguro com autenticação e SSL, modifique a configuração do receiver:

receivers:
  elasticsearch:
    endpoint: "https://localhost:9200"
    username: "your_elasticsearch_username"
    password: "your_elasticsearch_password"
    tls:
      ca_file: "/etc/elasticsearch/certs/http_ca.crt"
      insecure_skip_verify: false
    collection_interval: 15s

(Opcional) Para coletar métricas do host, adicione o receptor hostmetrics:

receivers:
  hostmetrics:
    collection_interval: 60s
    scrapers:
      cpu:
        metrics:
          system.cpu.utilization: {enabled: true}
          system.cpu.time: {enabled: true}
      load:
        metrics:
          system.cpu.load_average.1m: {enabled: true}
          system.cpu.load_average.5m: {enabled: true}
          system.cpu.load_average.15m: {enabled: true}
      memory:
        metrics:
          system.memory.usage: {enabled: true}
          system.memory.utilization: {enabled: true}
      disk:
        metrics:
          system.disk.io: {enabled: true}
          system.disk.operations: {enabled: true}
      filesystem:
        metrics:
          system.filesystem.usage: {enabled: true}
          system.filesystem.utilization: {enabled: true}
      network:
        metrics:
          system.network.io: {enabled: true}
          system.network.packets: {enabled: true}
      process:
        metrics:
          process.cpu.utilization:
            enabled: true

E adicione aos pipelines de serviço:

service:
  pipelines:
    metrics/host:
      receivers: [hostmetrics]
      processors: [memory_limiter, resourcedetection, batch]
      exporters: [otlphttp]

(Opcional) Para coletar logs do Elasticsearch, adicione o receiver filelog. Certifique-se de que o usuário que executa o serviço do coletor (nrdot-collector) tenha acesso de leitura aos seus arquivos de log do Elasticsearch:

Se estiver executando o Elasticsearch no Linux (Host):

receivers:
  filelog:
    include:
      - /var/log/elasticsearch/elasticsearch.log
      - /var/log/elasticsearch/*.log

Se estiver executando o Elasticsearch no Docker:

receivers:
  filelog:
    include:
      - /var/lib/docker/containers/*/*.log
    operators:
      - type: move
        from: attributes.log
        to: body

E adicione aos pipelines de serviço:

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [resource/cluster_name_override]
      exporters: [otlphttp]

(Opcional) Para adicionar tags de metadados personalizadas às suas métricas, use o processador resource/static_override:

processors:
  resource/static_override:
    attributes:
      - key: env
        value: "production"
        action: upsert
service:
  pipelines:
    metrics/elasticsearch:
      receivers: [elasticsearch]
      processors: [memory_limiter, resourcedetection, resource/cluster_name_override, resource/static_override, attributes/cardinality_reduction, cumulativetodelta, transform/metadata_nullify, batch]
      exporters: [otlphttp]

Salve o arquivo de configuração.
Defina as variáveis de ambiente:
Crie um diretório de substituição do systemd:
bash
```
$sudo mkdir -p /etc/systemd/system/nrdot-collector.service.d
```
Crie o arquivo de configuração do ambiente:
bash
```
$cat <<EOF | sudo tee /etc/systemd/system/nrdot-collector.service.d/environment.conf
$[Service]
$Environment="NEWRELIC_OTLP_ENDPOINT=https://otlp.nr-data.net:4318"
$Environment="NEWRELIC_LICENSE_KEY=YOUR_LICENSE_KEY_HERE"
$Environment="NEW_RELIC_MEMORY_LIMIT_MIB=100"
$EOF
```
Atualize a configuração com seus valores:
- Substitua https://otlp.nr-data.net:4318 pelo endpoint da sua região
- Substitua YOUR_LICENSE_KEY_HERE pela sua chave de licença da New Relic
- Substitua 100 pelo limite de memória desejado em MiB para o coletor (padrão: 100 MiB). Ajuste com base nas necessidades do seu ambiente

Reinicie o coletor NRDOT para aplicar as alterações:

bash

$sudo systemctl daemon-reload
$sudo systemctl restart nrdot-collector.service

Verificar coleta de dados

Verifique se o coletor NRDOT está em execução e coletando dados sem erros:

Verifique o status do serviço do coletor:
bash
```
$sudo systemctl status nrdot-collector.service
```
Monitore os logs do coletor em busca de erros:
bash
```
$sudo journalctl -u nrdot-collector.service -f
```
Procure por conexões bem-sucedidas com o Elasticsearch e o New Relic. Se você encontrar erros, consulte o guia de solução de problemas.

Antes de você começar

Antes de configurar o OTel Collector Contrib, certifique-se de ter:

Privilégios de acesso necessários:

Seu New Relic
Privilégios de root ou sudo na máquina host
Privilégios de administrador de cluster do Elasticsearch com privilégio de cluster monitor ou manage (consulte a documentação de privilégios de segurança do Elasticsearch para obter detalhes)
Requisitos do sistema:
Elasticsearch versão 7.16 ou superior - Esta integração requer um cluster Elasticsearch moderno
Conectividade de rede - HTTPS de saída (porta 443) para o endpoint de ingestão OTLPda New Relic
Valores de configuração prontos:
Endpoint do Elasticsearch - Sua URL do cluster Elasticsearch (por exemplo, http://localhost:9200)
Nome do cluster - Um identificador exclusivo para seu cluster no New Relic
Importante
Você deve ter o OpenTelemetry Collector Contrib instalado em seu host antes de prosseguir. Recomendamos a instalação via pacotes oficiais (.deb ou .rpm) para garantir que a unidade de serviço systemd seja criada corretamente.

Configurar o monitoramento do Elasticsearch

Assim que o OTel Collector Contrib estiver instalado, substitua o arquivo de configuração padrão do coletor pela configuração de monitoramento do Elasticsearch. Isso habilitará a coleta de métricas do Elasticsearch. Métricas de host e logs são opcionais e podem ser adicionados separadamente.

O arquivo de configuração está localizado em: /etc/otelcol-contrib/config.yaml

Dica

Faça backup da sua configuração padrão: Antes de modificar o arquivo de configuração, crie uma cópia de backup para preservar as configurações padrão:

bash

$sudo cp /etc/otelcol-contrib/config.yaml /etc/otelcol-contrib/config.yaml.backup

Para configurar o coletor:

Abra o arquivo de configuração com um editor de texto usando privilégios de root ou sudo:
bash
```
$sudo nano /etc/otelcol-contrib/config.yaml
```

Exclua todo o conteúdo existente e substitua-o pela seguinte configuração para monitoramento do Elasticsearch:

Importante

receivers:
  elasticsearch:
    endpoint: "http://localhost:9200"
    collection_interval: 15s
    metrics:
      elasticsearch.os.cpu.usage:
        enabled: true
      elasticsearch.cluster.data_nodes:
        enabled: true
      elasticsearch.cluster.health:
        enabled: true
      elasticsearch.cluster.in_flight_fetch:
        enabled: true
      elasticsearch.cluster.nodes:
        enabled: true
      elasticsearch.cluster.pending_tasks:
        enabled: true
      elasticsearch.cluster.shards:
        enabled: true
      elasticsearch.cluster.state_update.time:
        enabled: true
      elasticsearch.index.documents:
        enabled: true
      elasticsearch.index.operations.merge.current:
        enabled: true
      elasticsearch.index.operations.time:
        enabled: true
      elasticsearch.node.cache.count:
        enabled: true
      elasticsearch.node.cache.evictions:
        enabled: true
      elasticsearch.node.cache.memory.usage:
        enabled: true
      elasticsearch.node.shards.size:
        enabled: true
      elasticsearch.node.cluster.io:
        enabled: true
      elasticsearch.node.documents:
        enabled: true
      elasticsearch.node.disk.io.read:
        enabled: true
      elasticsearch.node.disk.io.write:
        enabled: true
      elasticsearch.node.fs.disk.available:
        enabled: true
      elasticsearch.node.fs.disk.total:
        enabled: true
      elasticsearch.node.http.connections:
        enabled: true
      elasticsearch.node.ingest.documents.current:
        enabled: true
      elasticsearch.node.ingest.operations.failed:
        enabled: true
      elasticsearch.node.open_files:
        enabled: true
      elasticsearch.node.operations.completed:
        enabled: true
      elasticsearch.node.operations.current:
        enabled: true
      elasticsearch.node.operations.get.completed:
        enabled: true
      elasticsearch.node.operations.time:
        enabled: true
      elasticsearch.node.shards.reserved.size:
        enabled: true
      elasticsearch.index.shards.size:
        enabled: true
      elasticsearch.os.cpu.load_avg.1m:
        enabled: true
      elasticsearch.os.cpu.load_avg.5m:
        enabled: true
      elasticsearch.os.cpu.load_avg.15m:
        enabled: true
      elasticsearch.os.memory:
        enabled: true
      jvm.gc.collections.count:
        enabled: true
      jvm.gc.collections.elapsed:
        enabled: true
      jvm.memory.heap.max:
        enabled: true
      jvm.memory.heap.used:
        enabled: true
      jvm.memory.heap.utilization:
        enabled: true
      jvm.threads.count:
        enabled: true
      elasticsearch.index.segments.count:
        enabled: true
      elasticsearch.index.operations.completed:
        enabled: true
      elasticsearch.node.script.cache_evictions:
        enabled: false
      elasticsearch.node.cluster.connections:
        enabled: false
      elasticsearch.node.pipeline.ingest.documents.preprocessed:
        enabled: false
      elasticsearch.node.thread_pool.tasks.queued:
        enabled: false
      elasticsearch.cluster.published_states.full:
        enabled: false
      jvm.memory.pool.max:
        enabled: false
      elasticsearch.node.script.compilation_limit_triggered:
        enabled: false
      elasticsearch.node.shards.data_set.size:
        enabled: false
      elasticsearch.node.pipeline.ingest.documents.current:
        enabled: false
      elasticsearch.cluster.state_update.count:
        enabled: false
      elasticsearch.node.fs.disk.free:
        enabled: false
      jvm.memory.nonheap.used:
        enabled: false
      jvm.memory.pool.used:
        enabled: false
      elasticsearch.node.translog.size:
        enabled: false
      elasticsearch.node.thread_pool.threads:
        enabled: false
      elasticsearch.cluster.state_queue:
        enabled: false
      elasticsearch.node.translog.operations:
        enabled: false
      elasticsearch.memory.indexing_pressure:
        enabled: false
      elasticsearch.node.ingest.documents:
        enabled: false
      jvm.classes.loaded:
        enabled: false
      jvm.memory.heap.committed:
        enabled: false
      elasticsearch.breaker.memory.limit:
        enabled: false
      elasticsearch.indexing_pressure.memory.total.replica_rejections:
        enabled: false
      elasticsearch.breaker.memory.estimated:
        enabled: false
      elasticsearch.cluster.published_states.differences:
        enabled: false
      jvm.memory.nonheap.committed:
        enabled: false
      elasticsearch.node.translog.uncommitted.size:
        enabled: false
      elasticsearch.node.script.compilations:
        enabled: false
      elasticsearch.node.pipeline.ingest.operations.failed:
        enabled: false
      elasticsearch.indexing_pressure.memory.limit:
        enabled: false
      elasticsearch.breaker.tripped:
        enabled: false
      elasticsearch.indexing_pressure.memory.total.primary_rejections:
        enabled: false
      elasticsearch.node.thread_pool.tasks.finished:
        enabled: false
processors:
  memory_limiter:
    check_interval: 60s
    limit_mib: ${env:NEW_RELIC_MEMORY_LIMIT_MIB:-100}
  cumulativetodelta: {}
  resource/cluster_name_override:
    attributes:
      - key: elasticsearch.cluster.name
        value: "<elasticsearch-cluster-name>"
        action: upsert
  resourcedetection:
    detectors: [ system ]
    system:
      resource_attributes:
        host.name:
          enabled: true
        host.id:
          enabled: true
        os.type:
          enabled: true
  batch:
    timeout: 10s
    send_batch_size: 1024
  attributes/cardinality_reduction:
    actions:
      - key: process.pid
        action: delete
      - key: process.parent_pid
        action: delete
  transform/metadata_nullify:
    metric_statements:
      - context: metric
        statements:
          - set(description, "")
          - set(unit, "")
exporters:
  otlphttp:
    endpoint: ${env:NEWRELIC_OTLP_ENDPOINT}
    headers:
      api-key: ${env:NEWRELIC_LICENSE_KEY}
service:
  pipelines:
    metrics/elasticsearch:
      receivers: [elasticsearch]
      processors: [memory_limiter, resourcedetection, resource/cluster_name_override, attributes/cardinality_reduction, cumulativetodelta, transform/metadata_nullify, batch]
      exporters: [otlphttp]

(Opcional) Para Elasticsearch seguro com autenticação e SSL, modifique a configuração do receiver:

receivers:
  elasticsearch:
    endpoint: "https://localhost:9200"
    username: "your_elasticsearch_username"
    password: "your_elasticsearch_password"
    tls:
      ca_file: "/etc/elasticsearch/certs/http_ca.crt"
      insecure_skip_verify: false
    collection_interval: 15s

(Opcional) Para coletar métricas do host, adicione o receptor hostmetrics:

receivers:
  hostmetrics:
    collection_interval: 60s
    scrapers:
      cpu:
        metrics:
          system.cpu.utilization: {enabled: true}
          system.cpu.time: {enabled: true}
      load:
        metrics:
          system.cpu.load_average.1m: {enabled: true}
          system.cpu.load_average.5m: {enabled: true}
          system.cpu.load_average.15m: {enabled: true}
      memory:
        metrics:
          system.memory.usage: {enabled: true}
          system.memory.utilization: {enabled: true}
      disk:
        metrics:
          system.disk.io: {enabled: true}
          system.disk.operations: {enabled: true}
      filesystem:
        metrics:
          system.filesystem.usage: {enabled: true}
          system.filesystem.utilization: {enabled: true}
      network:
        metrics:
          system.network.io: {enabled: true}
          system.network.packets: {enabled: true}
      process:
        metrics:
          process.cpu.utilization:
            enabled: true

E adicione aos pipelines de serviço:

service:
  pipelines:
    metrics/host:
      receivers: [hostmetrics]
      processors: [memory_limiter, resourcedetection, batch]
      exporters: [otlphttp]

(Opcional) Para coletar logs do Elasticsearch, adicione o receiver filelog. Certifique-se de que o usuário que executa o serviço do coletor (otelcol-contrib) tenha acesso de leitura aos seus arquivos de log do Elasticsearch:

Se estiver executando o Elasticsearch no Linux (Host):

receivers:
  filelog:
    include:
      - /var/log/elasticsearch/elasticsearch.log
      - /var/log/elasticsearch/*.log

Se estiver executando o Elasticsearch no Docker:

receivers:
  filelog:
    include:
      - /var/lib/docker/containers/*/*.log
    operators:
      - type: move
        from: attributes.log
        to: body

E adicione aos pipelines de serviço:

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [resource/cluster_name_override]
      exporters: [otlphttp]

(Opcional) Para adicionar tags de metadados personalizadas às suas métricas, use o processador resource/static_override:

processors:
  resource/static_override:
    attributes:
      - key: env
        value: "production"
        action: upsert
service:
  pipelines:
    metrics/elasticsearch:
      receivers: [elasticsearch]
      processors: [memory_limiter, resourcedetection, resource/cluster_name_override, resource/static_override, attributes/cardinality_reduction, cumulativetodelta, transform/metadata_nullify, batch]
      exporters: [otlphttp]

Salve o arquivo de configuração.
Defina as variáveis de ambiente:
Crie um diretório de substituição do systemd:
bash
```
$sudo mkdir -p /etc/systemd/system/otelcol-contrib.service.d
```
Crie o arquivo de configuração do ambiente:
bash
```
$cat <<EOF | sudo tee /etc/systemd/system/otelcol-contrib.service.d/environment.conf
$[Service]
$Environment="NEWRELIC_OTLP_ENDPOINT=https://otlp.nr-data.net:4318"
$Environment="NEWRELIC_LICENSE_KEY=YOUR_LICENSE_KEY_HERE"
$Environment="NEW_RELIC_MEMORY_LIMIT_MIB=100"
$EOF
```
Atualize a configuração com seus valores:
- Substitua https://otlp.nr-data.net:4318 pelo endpoint da sua região
- Substitua YOUR_LICENSE_KEY_HERE pela sua chave de licença da New Relic
- Substitua 100 pelo limite de memória desejado em MiB para o coletor (padrão: 100 MiB). Ajuste com base nas necessidades do seu ambiente

Reinicie o OTel Collector Contrib para aplicar as alterações:

bash

$sudo systemctl daemon-reload
$sudo systemctl restart otelcol-contrib.service

Verificar coleta de dados

Verifique se o OTel Collector Contrib está em execução e coletando dados sem erros:

Verifique o status do serviço do coletor:
bash
```
$sudo systemctl status otelcol-contrib.service
```
Monitore os logs do coletor em busca de erros:
bash
```
$sudo journalctl -u otelcol-contrib.service -f
```
Procure por conexões bem-sucedidas com o Elasticsearch e o New Relic. Se você encontrar erros, consulte o guia de solução de problemas.

Use esta abordagem se você já tiver um Prometheus Elasticsearch exporter em execução no seu ambiente, ou se estiver migrando de uma stack de monitoramento baseada no Prometheus.

Dica

Recomendado: se você ainda não tiver um exportador do Prometheus em execução, use as guias NRDOT Collector ou OTel Collector Contrib em vez disso. Eles se conectam diretamente à API do Elasticsearch sem precisar de um componente exportador adicional.

Instale e configure o exportador do Elasticsearch

O exportador do Elasticsearch para Prometheus expõe métricas do Elasticsearch no formato Prometheus na porta 9114.

Baixe a versão mais recente:

bash

$wget https://github.com/prometheus-community/elasticsearch_exporter/releases/download/v1.8.0/elasticsearch_exporter-1.8.0.linux-amd64.tar.gz
$tar xzf elasticsearch_exporter-*.tar.gz
$cd elasticsearch_exporter-*

Inicie o exporter (aponte-o para sua instância Elasticsearch):
bash
```
$./elasticsearch_exporter --es.uri=http://localhost:9200
```
Dica
Para Elasticsearch seguro, use: --es.uri=https://username:password@localhost:9200 --es.ssl-skip-verify
Verifique se o exportador está em execução:
bash
```
$curl http://localhost:9114/metrics | grep elasticsearch_cluster_health
```
Você deve ver métricas no formato Prometheus como elasticsearch_cluster_health_status, elasticsearch_cluster_health_number_of_nodes, etc.

Configurar o coletor

Esta configuração funciona tanto com o NRDOT quanto com o OTel Collector Contrib. Coloque-o no local de configuração apropriado para o seu coletor:

NRDOT: /etc/nrdot-collector/config.yaml

OTel Collector Contrib: /etc/otelcol-contrib/config.yaml

Se você ainda não tem um coletor instalado, siga primeiro as etapas de instalação nas abas NRDOT Collector ou OTel Collector Contrib acima e depois retorne aqui para aplicar esta configuração.

Esta configuração coleta métricas do exportador do Elasticsearch e as traduz para nomes de métricas compatíveis com o OpenTelemetry que alimentam o dashboard do Elasticsearch da New Relic.

receivers:
  prometheus/elasticsearch:
    config:
      scrape_configs:
        - job_name: 'elasticsearch'
          scrape_interval: 15s
          static_configs:
            - targets: ['localhost:9114']  # <-- elasticsearch_exporter address

processors:
  memory_limiter:
    check_interval: 60s
    limit_mib: ${env:NEW_RELIC_MEMORY_LIMIT_MIB:-100}

  metricstransform/prom_to_otel:
    transforms:
      - include: ^elasticsearch_cluster_health_number_of_data_nodes$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.data_nodes
      - include: ^elasticsearch_cluster_health_status$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.health
        operations:
          - action: update_label
            label: color
            new_label: status
      - include: ^elasticsearch_cluster_health_number_of_in_flight_fetch$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.in_flight_fetch
      - include: ^elasticsearch_cluster_health_number_of_nodes$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.nodes
      - include: ^elasticsearch_cluster_health_number_of_pending_tasks$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.pending_tasks
      - include: ^elasticsearch_cluster_health_active_shards$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.shards
        operations:
          - action: add_label
            new_label: state
            new_value: active
      - include: ^elasticsearch_cluster_health_active_primary_shards$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.shards
        operations:
          - action: add_label
            new_label: state
            new_value: active_primary
      - include: ^elasticsearch_cluster_health_initializing_shards$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.shards
        operations:
          - action: add_label
            new_label: state
            new_value: initializing
      - include: ^elasticsearch_cluster_health_relocating_shards$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.shards
        operations:
          - action: add_label
            new_label: state
            new_value: relocating
      - include: ^elasticsearch_cluster_health_unassigned_shards$
        match_type: regexp
        action: update
        new_name: elasticsearch.cluster.shards
        operations:
          - action: add_label
            new_label: state
            new_value: unassigned
      - include: ^elasticsearch_indices_docs$
        match_type: regexp
        action: insert
        new_name: elasticsearch.node.documents
        operations:
          - action: add_label
            new_label: state
            new_value: active
      - include: ^elasticsearch_indices_docs$
        match_type: regexp
        action: insert
        new_name: elasticsearch.index.documents
        operations:
          - action: add_label
            new_label: state
            new_value: active
          - action: add_label
            new_label: aggregation
            new_value: primary_shards
          - action: add_label
            new_label: elasticsearch.index.name
            new_value: _all
      - include: ^elasticsearch_indices_docs_deleted$
        match_type: regexp
        action: insert
        new_name: elasticsearch.node.documents
        operations:
          - action: add_label
            new_label: state
            new_value: deleted
      - include: ^elasticsearch_indices_fielddata_evictions$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.cache.evictions
        operations:
          - action: add_label
            new_label: cache_name
            new_value: fielddata
      - include: ^elasticsearch_indices_query_cache_evictions$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.cache.evictions
        operations:
          - action: add_label
            new_label: cache_name
            new_value: query
      - include: ^elasticsearch_indices_fielddata_memory_size_bytes$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.cache.memory.usage
        operations:
          - action: add_label
            new_label: cache_name
            new_value: fielddata
      - include: ^elasticsearch_indices_query_cache_memory_size_bytes$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.cache.memory.usage
        operations:
          - action: add_label
            new_label: cache_name
            new_value: query
      - include: ^elasticsearch_transport_rx_size_bytes_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.cluster.io
        operations:
          - action: add_label
            new_label: direction
            new_value: received
      - include: ^elasticsearch_transport_tx_size_bytes_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.cluster.io
        operations:
          - action: add_label
            new_label: direction
            new_value: sent
      - include: ^elasticsearch_filesystem_data_available_bytes$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.fs.disk.available
      - include: ^elasticsearch_filesystem_data_size_bytes$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.fs.disk.total
      - include: ^elasticsearch_process_open_files_count$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.open_files
      - include: ^elasticsearch_indices_indexing_index_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.operations.completed
        operations:
          - action: add_label
            new_label: operation
            new_value: index
      - include: ^elasticsearch_indices_search_query_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.operations.completed
        operations:
          - action: add_label
            new_label: operation
            new_value: query
      - include: ^elasticsearch_indices_get_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.operations.completed
        operations:
          - action: add_label
            new_label: operation
            new_value: get
      - include: ^elasticsearch_indices_merges_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.operations.completed
        operations:
          - action: add_label
            new_label: operation
            new_value: merge
      - include: ^elasticsearch_indices_indexing_index_time_seconds_total$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.operations.time
        operations:
          - action: add_label
            new_label: operation
            new_value: index
          - action: experimental_scale_value
            experimental_scale: 1000
      - include: ^elasticsearch_indices_search_query_time_seconds$
        match_type: regexp
        action: update
        new_name: elasticsearch.node.operations.time
        operations:
          - action: add_label
            new_label: operation
            new_value: query
          - action: experimental_scale_value
            experimental_scale: 1000
      - include: ^elasticsearch_thread_pool_active_count$
        match_type: regexp
        action: insert
        new_name: elasticsearch.node.operations.current
        operations:
          - action: update_label
            label: type
            new_label: operation
      - include: ^elasticsearch_os_cpu_percent$
        match_type: regexp
        action: update
        new_name: elasticsearch.os.cpu.usage
      - include: ^elasticsearch_os_load1$
        match_type: regexp
        action: update
        new_name: elasticsearch.os.cpu.load_avg.1m
      - include: ^elasticsearch_os_load5$
        match_type: regexp
        action: update
        new_name: elasticsearch.os.cpu.load_avg.5m
      - include: ^elasticsearch_os_load15$
        match_type: regexp
        action: update
        new_name: elasticsearch.os.cpu.load_avg.15m
      - include: ^elasticsearch_jvm_gc_collection_seconds_count$
        match_type: regexp
        action: update
        new_name: jvm.gc.collections.count
      - include: ^elasticsearch_jvm_gc_collection_seconds_sum$
        match_type: regexp
        action: update
        new_name: jvm.gc.collections.elapsed
        operations:
          - action: experimental_scale_value
            experimental_scale: 1000
      - include: ^elasticsearch_jvm_memory_max_bytes$
        match_type: regexp
        action: update
        new_name: jvm.memory.heap.max
      - include: ^elasticsearch_jvm_memory_used_bytes$
        match_type: regexp
        action: update
        new_name: jvm.memory.heap.used
      - include: ^elasticsearch_thread_pool_threads_count$
        match_type: regexp
        action: update
        new_name: jvm.threads.count
        operations:
          - action: aggregate_labels
            label_set: [name, cluster]
            aggregation_type: sum
      - include: ^elasticsearch_indices_segments_count$
        match_type: regexp
        action: update
        new_name: elasticsearch.index.segments.count

  filter/allowed_metrics:
    metrics:
      include:
        match_type: strict
        metric_names:
          - elasticsearch.cluster.data_nodes
          - elasticsearch.cluster.health
          - elasticsearch.cluster.in_flight_fetch
          - elasticsearch.cluster.nodes
          - elasticsearch.cluster.pending_tasks
          - elasticsearch.cluster.shards
          - elasticsearch.index.documents
          - elasticsearch.index.segments.count
          - elasticsearch.node.cache.evictions
          - elasticsearch.node.cache.memory.usage
          - elasticsearch.node.cluster.io
          - elasticsearch.node.documents
          - elasticsearch.node.fs.disk.available
          - elasticsearch.node.fs.disk.total
          - elasticsearch.node.open_files
          - elasticsearch.node.operations.completed
          - elasticsearch.node.operations.current
          - elasticsearch.node.operations.time
          - elasticsearch.os.cpu.usage
          - elasticsearch.os.cpu.load_avg.1m
          - elasticsearch.os.cpu.load_avg.5m
          - elasticsearch.os.cpu.load_avg.15m
          - jvm.gc.collections.count
          - jvm.gc.collections.elapsed
          - jvm.memory.heap.max
          - jvm.memory.heap.used
          - jvm.threads.count

  transform/rename_node_name:
    metric_statements:
      - context: datapoint
        statements:
          - set(attributes["elasticsearch.node.name"], attributes["name"]) where attributes["name"] != nil
          - delete_key(attributes, "name") where attributes["elasticsearch.node.name"] != nil

  groupbyattrs/node:
    keys:
      - elasticsearch.node.name
      - elasticsearch.cluster.name

  cumulativetodelta: {}

  resource/cluster_name_override:
    attributes:
      - key: elasticsearch.cluster.name
        value: "<elasticsearch-cluster-name>"
        action: upsert

  transform/scope_override:
    metric_statements:
      - context: scope
        statements:
          - set(name, "github.com/open-telemetry/opentelemetry-collector-contrib/receiver/elasticsearchreceiver")
          - set(version, "")

  resourcedetection:
    detectors: [system]
    system:
      resource_attributes:
        host.name:
          enabled: true
        host.id:
          enabled: true
        os.type:
          enabled: true

  batch:
    timeout: 30s
    send_batch_size: 2048
    send_batch_max_size: 4096

  attributes/cardinality_reduction:
    actions:
      - key: process.pid
        action: delete
      - key: process.parent_pid
        action: delete

  transform/metadata_nullify:
    metric_statements:
      - context: metric
        statements:
          - set(description, "")
          - set(unit, "")

exporters:
  otlphttp:
    endpoint: ${env:NEWRELIC_OTLP_ENDPOINT}
    headers:
      api-key: ${env:NEWRELIC_LICENSE_KEY}
    compression: gzip
    timeout: 30s
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
      max_elapsed_time: 300s

service:
  pipelines:
    metrics/elasticsearch:
      receivers: [prometheus/elasticsearch]
      processors:
        - memory_limiter
        - metricstransform/prom_to_otel
        - filter/allowed_metrics
        - transform/rename_node_name
        - groupbyattrs/node
        - transform/scope_override
        - resourcedetection
        - resource/cluster_name_override
        - attributes/cardinality_reduction
        - cumulativetodelta
        - transform/metadata_nullify
        - batch
      exporters: [otlphttp]

Substitua os seguintes valores na configuração:

<elasticsearch-cluster-name>: o nome do seu cluster do Elasticsearch para identificação no New Relic.
localhost:9114: O endereço do seu elasticsearch_exporter se estiver sendo executado em um host ou porta diferente.

Configurar variáveis de ambiente

Crie um override do systemd para injetar as variáveis de ambiente necessárias. Substitua <collector-service> pelo nome do seu serviço de coletor (nrdot-collector ou otelcol-contrib):

bash

$sudo mkdir -p /etc/systemd/system/<collector-service>.service.d

Crie o arquivo /etc/systemd/system/<collector-service>.service.d/environment.conf:

[Service]
Environment="NEWRELIC_OTLP_ENDPOINT=https://otlp.nr-data.net:4318"
Environment="NEWRELIC_LICENSE_KEY=YOUR_NEWRELIC_LICENSE_KEY"
Environment="NEW_RELIC_MEMORY_LIMIT_MIB=100"

Substitua YOUR_NEWRELIC_LICENSE_KEY pelo seu .

Dica

Para contas da UE, use NEWRELIC_OTLP_ENDPOINT=https://otlp.eu01.nr-data.net:4318

Inicie o coletor

Substitua <collector-service> pelo nome do seu serviço de coletor (nrdot-collector ou otelcol-contrib):

bash

$sudo systemctl daemon-reload
$sudo systemctl enable <collector-service>
$sudo systemctl restart <collector-service>

Verifique o status do coletor:

bash

$sudo systemctl status <collector-service>
$sudo journalctl -u <collector-service> -f

Verifique os dados no New Relic

Após alguns minutos, verifique se os dados estão fluindo para o New Relic:

FROM Metric SELECT count(*)
WHERE metricName LIKE 'elasticsearch.%'
AND elasticsearch.cluster.name = '<elasticsearch-cluster-name>'
SINCE 10 minutes ago

Dica

Correlacione APM com Elasticsearch: Para conectar seu aplicativo APM e o cluster Elasticsearch, inclua o atributo de recurso es.cluster.name="your-cluster-name" em suas métricas APM. Isso permite a visibilidade entre serviços e uma solução de problemas mais rápida no New Relic.

Visualize seus dados do Elasticsearch

Assim que o coletor estiver em execução e enviando dados, você pode explorar suas métricas do Elasticsearch, criar consultas personalizadas e configurar dashboards de monitoramento no New Relic.

Para informações detalhadas sobre como acessar seus dados, escrever consultas NRQL e configurar alertas, consulte Encontrar e consultar dados do Elasticsearch.

Resolução de problemas

Se você encontrar problemas durante a instalação ou não vir dados na New Relic, consulte nosso guia de solução de problemas abrangente para obter soluções passo a passo para problemas comuns.

Esta tradução de máquina é fornecida para sua comodidade.

Monitore o Elasticsearch auto-hospedado com o OpenTelemetry

Opções de instalação

Antes de você começar

Configurar o monitoramento do Elasticsearch

Dica

Importante

Verificar coleta de dados

Antes de você começar

Importante

Configurar o monitoramento do Elasticsearch

Dica

Importante

Verificar coleta de dados

Dica

Instale e configure o exportador do Elasticsearch

Dica

Configurar o coletor

Configuração completa do receptor Prometheus

Configurar variáveis de ambiente

Dica

Inicie o coletor

Verifique os dados no New Relic

Dica

Visualize seus dados do Elasticsearch

Resolução de problemas

Esta tradução de máquina é fornecida para sua comodidade.

Monitore o Elasticsearch auto-hospedado com o OpenTelemetry

Opções de instalação .css-21sua1{background:none;border:none;width:0;padding:0;}

Antes de você começar

Configurar o monitoramento do Elasticsearch

Dica

Importante

Verificar coleta de dados

Antes de você começar

Importante

Configurar o monitoramento do Elasticsearch

Dica

Importante

Verificar coleta de dados

Dica

Instale e configure o exportador do Elasticsearch

Dica

Configurar o coletor

Configurar variáveis de ambiente

Dica

Inicie o coletor

Verifique os dados no New Relic

Dica

Visualize seus dados do Elasticsearch

Resolução de problemas

Opções de instalação