---
title: OpenTelemetry Kafka monitoring overview
source: https://docs.newrelic.com/docs/opentelemetry/integrations/kafka/overview
---

Monitor Apache Kafka clusters with OpenTelemetry Collector for real-time visibility and reliable data streaming. This vendor-neutral solution prevents costly downtime across self-hosted and Kubernetes environments.

## Collector options [#collector-options]

New Relic supports two OpenTelemetry Collector distributions for Kafka monitoring, both offering identical functionality with the same configuration files and monitoring capabilities.

-   **NRDOT Collector** (recommended): New Relic's distribution of OpenTelemetry Collector with New Relic support for assistance. For more information, see the [NRDOT Collector GitHub repository](https://github.com/newrelic/nrdot-collector-releases/tree/main/distributions/nrdot-collector).
-   **OpenTelemetry Collector**: The upstream community distribution. For more information, see the [OpenTelemetry Collector Contrib GitHub repository](https://github.com/open-telemetry/opentelemetry-collector-releases/tree/main/distributions/otelcol-contrib).

Choose the collector that best fits your support and operational requirements, then proceed to [set up monitoring](#get-started) for your environment.

![Dashboard showing Kafka cluster health, network throughput, active brokers, topics, partitions, and consumer groups](https://docs.newrelic.com/images/otel-kafka-dashboard.webp "OTel Kafka Dashboard")

Monitor your Kafka clusters with comprehensive dashboards showing cluster health, broker status, topic metrics, and consumer group performance.

## Why Kafka monitoring?

-   **Prevent outages**: Get alerts for broker failures, under-replicated partitions, and offline topics before they cause downtime
-   **Optimize performance**: Identify consumer lag, slow producers, and network bottlenecks that affect data processing speed
-   **Plan capacity**: Track resource usage, message rates, and connection counts to scale proactively
-   **Ensure data integrity**: Monitor replication health and partition balance to prevent data loss

### Common use case

Kafka monitoring helps you catch issues before they impact your business. Get alerted when consumer lag spikes threaten real-time dashboards, broker failures risk data loss, or network bottlenecks slow critical data pipelines. Essential for financial transactions, IoT data processing, microservices communication, e-commerce platforms, and real-time analytics.

## Get started [#get-started]

Choose your Kafka environment to begin monitoring. Each setup guide includes prerequisites, configuration steps, and troubleshooting tips.

-   [Self-hosted Kafka](https://docs.newrelic.com/docs/opentelemetry/integrations/kafka/self-hosted)

-   [Kubernetes (self-managed)](https://docs.newrelic.com/docs/opentelemetry/integrations/kafka/kubernetes-self-managed)

-   [Kubernetes (Strimzi)](https://docs.newrelic.com/docs/opentelemetry/integrations/kafka/kubernetes-strimzi)

## How it works [#how-it-works]

The collector continuously gathers performance data using specialized components:

**Data collection:**

-   **[Kafka metrics receiver](https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/kafkametricsreceiver)**: Connects to Kafka's bootstrap port for cluster health, consumer lag, topic metrics, and partition status
-   **JMX metrics collection**: Collects broker performance, JVM data, and operational insights via:
    -   **Self-hosted Kafka**: OTel Java agent or Prometheus JMX Exporter on broker JVM
    -   **Kubernetes (self-managed)**: OTel Java agent or Prometheus JMX Exporter via init container
    -   **Kubernetes (Strimzi)**: Prometheus JMX Exporter via Strimzi's `KafkaMetricsConfig`

|                   | [OTel Java agent](https://opentelemetry.io/docs/zero-code/java/agent/) | [Prometheus JMX Exporter](https://github.com/prometheus/jmx_exporter) |
| ----------------- | ---------------------------------------------------------------------- | --------------------------------------------------------------------- |
| **Architecture**  | Push-based                                                             | Pull-based                                                            |
| **Protocol**      | OTLP (gRPC or HTTP)                                                    | HTTP scrape (default port 9404)                                       |
| **Configuration** | JMX config file (`.yaml`)                                              | YAML config with metric patterns                                      |
| **Availability**  | Self-hosted, Kubernetes self-managed                                   | Self-hosted, Kubernetes self-managed, Kubernetes Strimzi              |

**Key metrics:** Consumer lag, broker health, request rates, network throughput, partition replication status, resource utilization, and JVM performance data.

For complete metric names, descriptions, and alerting recommendations, see [Kafka metrics reference](https://docs.newrelic.com/docs/opentelemetry/integrations/kafka/metrics-reference).

### Optional: Add application-level monitoring

Monitor producer and consumer applications for complete visibility from producers → brokers → consumers.

**Adds:** Request latencies, throughput metrics, error rates, and distributed traces.

**Setup:** Use the [OpenTelemetry Java agent](https://opentelemetry.io/docs/zero-code/java/agent/getting-started/) for [zero-code Kafka instrumentation](https://opentelemetry.io/docs/zero-code/java/spring-boot-starter/out-of-the-box-instrumentation/).

## Next steps [#next-steps]

**Set up monitoring:**

-   [Self-hosted Kafka](https://docs.newrelic.com/docs/opentelemetry/integrations/kafka/self-hosted)
-   [Kubernetes (self-managed)](https://docs.newrelic.com/docs/opentelemetry/integrations/kafka/kubernetes-self-managed)
-   [Kubernetes with Strimzi](https://docs.newrelic.com/docs/opentelemetry/integrations/kafka/kubernetes-strimzi)

**After setup:**

-   [Find and query your data](https://docs.newrelic.com/docs/opentelemetry/integrations/kafka/find-and-query-data)
-   [Explore Kafka metrics](https://docs.newrelic.com/docs/opentelemetry/integrations/kafka/metrics-reference)