Forward logs from Google Cloud Platform

Start by creating an ingest URL for your GCP Pub/Sub topic:

1. Go to the Integrations & Agents UI, click Logging, and click Google Cloud Platform.

2. Select the New Relic account where you want to forward logs, and click Continue.

3. Optional: Configure metadata (attribute-value pairs) to be included in every log event sent to the ingest URL you will generate in the next step.

4. Click Generate URL.

5. Copy your newly generated ingest URL and keep it in a safe place.

   You will use your new ingest URL to configure a Pub/Sub topic that sends logs to New Relic.

Next you will create the GCP Pub/Sub topic that your ingest URL will use.

1. Navigate to the GCP Pub/Sub Console.
2. Click Create Topic.
3. Enter a meaningful Topic ID, then configure other options as needed.
4. Click Create Topic.

Once you have created your Pub/Sub topic, create a subscription for it.

1. Return to the GCP Pub/Sub Console.

2. Click the Pub/Sub topic you previously created.

3. Scroll down and select the Subscriptions tab, then click Create Subscription and select Create a simple subscription.

   

4. Enter a Subscription ID. Then, under Delivery Type, select Push.

5. In the Endpoint URL field, paste the ingest URL you previously generated.

6. Configure remaining settings as needed, and click Create.

To finish setup, create a routing sink for your GCP Pub/Sub topic that will forward your logs to New Relic.

1. Navigate to the GCP Logs Router Console.
2. Click Create Sink.
3. Enter a Sink name and Sink description, then click Next.
4. Under Select sink service, select Cloud Pub/Sub topic, and select the Pub/Sub topic you previously created.
5. Configure remaining filters as needed, then click Create sink to complete setup.

Run the following command and follow the prompts to log into GCP and select your cloud project:

$
gcloud init

Using the wizard, you will select a cloud project to use, and you will be able to optionally select a default compute region and zone for the resources you create using gcloud or gsutil. We will not presume a default project, location or region for the following commands.

1. Clone the DataflowTemplates GitHub repository using the following command:

   bash

   Copia

   $
   git clone https://github.com/newrelic-forks/DataflowTemplates.git

2. Enter the directory you just created:

   bash

   Copia

   $
   cd DataflowTemplates

   Then continue with the next section to execute additional commands.

To set the required configuration to compile and run the Dataflow forwarder, run the following commands in your DataflowTemplates directory. The only required values are:

• PROJECT_ID

• BUCKET_NAME

• NR_LICENSE_KEY

• INPUT_SUBSCRIPTION_NAME

  You can leave the other default values as is.

  # The Google Cloud Platform project id where your logs are and where the Dataflow log forwarder will run
  PROJECT_ID=<your_project_id>
  # Temporary bucket that will store intermediary files as a result of compiling the Dataflow template. Its name must be unique.
  BUCKET_NAME=<a_nonexisting_gcs_bucket_name>
  # New Relic license key
  NR_LICENSE_KEY=<your_newrelic_license_key>
  # Input PubSub subscription to read logs from
  INPUT_SUBSCRIPTION_NAME=<your_pubsub_input_subscription_name>
  
  # Region where the created resources will be placed
  REGION=us-west1
  # Service account used to execute the Dataflow template
  SERVICE_ACCOUNT_NAME=nr-dataflow-forwarder-sa
  # File name where the service account credentials will be stored
  SERVICE_ACCOUNT_KEY_FILENAME=service-account-key.json
  # The name your Dataflow log forwarder job will have
  JOB_NAME=nr-log-forwarder

  Copia

Create a bucket in GCP to hold the generated Dataflow template by running the following command:

$
gsutil mb -p ${PROJECT_ID} -l ${REGION} gs://${BUCKET_NAME}

Execute the following commands:

1. Create the service account:

   bash

   Copia

   $
   gcloud iam service-accounts create ${SERVICE_ACCOUNT_NAME}

2. Grant permissions to the service account:

   bash

   Copia

   $
   gcloud projects add-iam-policy-binding ${PROJECT_ID} --member="serviceAccount:${SERVICE_ACCOUNT_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" --role="roles/owner"

3. Generate the service account key file:

   bash

   Copia

   $
   gcloud iam service-accounts keys create ${SERVICE_ACCOUNT_KEY_FILENAME} --iam-account=${SERVICE_ACCOUNT_NAME}@${PROJECT_ID}.iam.gserviceaccount.com

4. Reference your service account key file using the GOOGLE_APPLICATION_CREDENTIALS environment variable, as it will be used by subsequent commands:

   bash

   Copia

   $
   export GOOGLE_APPLICATION_CREDENTIALS=${SERVICE_ACCOUNT_KEY_FILENAME}

Execute the following command:

$
mvn compile exec:java \
>
    -Dexec.mainClass=com.google.cloud.teleport.templates.PubsubToNewRelic \
>
    -Dexec.cleanupDaemonThreads=false \
>
    -Dexec.args=" \
>
        --project=${PROJECT_ID} \
>
        --region=${REGION} \
>
        --enableStreamingEngine \
>
        --stagingLocation=gs://${BUCKET_NAME}/staging/ \
>
        --gcpTempLocation=gs://${BUCKET_NAME}/temp/ \
>
        --templateLocation=gs://${BUCKET_NAME}/template/PubsubToNewRelic \
>
        --runner=DataflowRunner \
>
    "

Execute the following command to start shipping logs using a Dataflow job that reads from your Pub/Sub topic:

$
gcloud dataflow jobs run ${JOB_NAME} \
>
    --gcs-location=gs://${BUCKET_NAME}/template/PubsubToNewRelic \
>
    --region=${REGION} \
>
    --parameters "inputSubscription=projects/${PROJECT_ID}/subscriptions/${INPUT_SUBSCRIPTION_NAME},licenseKey=${NR_LICENSE_KEY}"

This command only requires these two values:

• The input PubSub subscription used to read log messages
• The New Relic clave de licencia used to send your logs

$
gcloud dataflow jobs run ${JOB_NAME} \
>
    --gcs-location=gs://${BUCKET_NAME}/template/PubsubToNewRelic \
>
    --region=${REGION} \
>
    --parameters "inputSubscription=projects/${PROJECT_ID}/subscriptions/${INPUT_SUBSCRIPTION_NAME},licenseKey=${NR_LICENSE_KEY},logsApiUrl=https://log-api.eu.newrelic.com/log/v1"

Here is a reference of available options you can use to further tune up the execution of your Dataflow log forwarder job.

projects/<project-id>/subscriptions/<subscription-name>

https://log-api.newrelic.com/log/v1

https://log-api.eu.newrelic.com/log/v1

projects/{gcp_project}/locations/{key_region}/keyRings/{key_ring}/cryptoKeys/{kms_key_name}