Google Cloud Composer monitoring integration

We offer a cloud integration for reporting your GCP Composer data to our platform. Here we explain how to activate the integration and what data it collects.

Activate the integration

To enable the integration follow standard procedures to connect your GCP service.

Configuration and polling

You can change the polling frequency and filter data using configuration options.

Default polling information for the GCP Composer integration:

New Relic polling interval: 5 minutes

Find and use data

To find your integration data, go to one.newrelic.com > All capabilities > Infrastructure > GCP and select the integration.

Data is attached to the following event types:

Entity	Event Type	Provider
Environment	`GcpComposerEnvironmentSample`	`GcpComposerEnvironment`
Workflow	`GcpComposerWorkflowSample`	`GcpComposerWorkflow`

For more on how to use your data, see Understand and use integration data.

Metric data

This integration collects GCP Composer data for Environment and Workflow.

Composer Environment data

Metric	Unit	Description
`environment.api.Request`	Count	Number of Composer API requests seen so far.
`environment.api.RequestLatencies`	Milliseconds	Distribution of Composer API call latencies.
`environment.dag_processing.ParseError`	Count	Number of errors raised during parsing DAG files.
`environment.dag_processing.Processes`	Count	Number of currently running DAG parsing processes.
`environment.dag_processing.ProcessorTimeout`	Count	Number of file processors terminated due to processing timeout.
`environment.dag_processing.TotalParseTime`	Seconds	Number of seconds taken to scan and import all DAG files once.
`environment.DagbagSize`	Count	The current dag bag size.
`environment.database.cpu.ReservedCores`	Count	Number of cores reserved for the database instance.
`environment.database.cpu.UsageTime`	Seconds	CPU usage time of the database instance.
`environment.database.cpu.Utilization`	Count	CPU utilization ratio (from 0.0 to 1.0) of the database instance.
`environment.database.disk.BytesUsed`	Bytes	Used disk space in bytes on the database instance.
`environment.database.disk.Quota`	Bytes	Maximum data disk size in bytes of the database instance.
`environment.database.disk.Utilization`	Count	Disk quota usage ratio (from 0.0 to 1.0) of the database instance.
`environment.database.memory.BytesUsed`	Bytes	Memory usage of the database instance in bytes.
`environment.database.memory.Quota`	Bytes	Maximum RAM size in bytes of the database instance.
`environment.database.memory.Utilization`	Count	Memory utilization ratio (from 0.0 to 1.0) of the database instance.
`environment.executor.OpenSlots`	Count	Number of open slots on executor.
`environment.executor.RunningTasks`	Count	Number of running tasks on executor.
`environment.FinishedTaskInstance`	Count	Overall task instances.
`environment.NumCeleryWorkers`	Count	Number of Celery workers.
`environment.SchedulerHeartbeat`	Count	Scheduler heartbeats.
`environment.TaskQueueLength`	Count	Number of tasks in queue.
`environment.worker.PodEviction`	Count	The number of Airflow worker pods evictions.
`environment.ZombieTaskKilled`	Count	Number of zombie tasks killed.

Composer Workflow data

Metric	Unit	Description
`workflow.Run`	Count	Number of workflow runs completed so far.
`workflow.RunDuration`	Seconds	Duration of workflow run completion.
`workflow.task.Run`	Count	Number of workflow tasks completed so far.
`workflow.task.RunDuration`	Seconds	Duration of task completion.