Synthetics job manager configuration

The variables are provided at startup using the -e, --env argument.

The following table shows all the environment variables that synthetics job manager supports. PRIVATE_LOCATION_KEY is required, and all other variables are optional.

The variables are provided at startup using the -e, --env argument.

The following table displays all the environment variables that synthetics job manager supports. PRIVATE_LOCATION_KEY is required, and all other variables are optional. To run the synthetics job manager in a Podman environment, the minimum version should be release-418 or higher.

The variables are provided at startup using the --set argument.

The following list shows all the environment variables that synthetics job manager supports. synthetics.privateLocationKey is required, and all other variables are optional.

A number of additional advanced settings are available and fully documented in our Helm chart README

The variables are provided at startup using the --set argument.

The following list shows all the environment variables that synthetics job manager supports. synthetics.privateLocationKey is required, and all other variables are optional.

A number of additional advanced settings are available and fully documented in our Helm chart README

The user may create a JSON-formatted file and mount the volume where the file is located to a specified target path in the SJM container.

The file must have read permissions and contain a JSON-formatted map. Example user-defined variables file:

{
  "KEY": "VALUE",
  "user_name": "MINION",
  "my_password": "PASSW0RD123",
  "my_URL": "https://newrelic.com/",
  "ETC": "ETC"
}

Place the file in the source directory on the host. The SJM is expecting the file name to be user_defined_variables.json

Docker example:

The expected target directory is: /var/lib/newrelic/synthetics/variables/

$
docker run ... -v /variables:/var/lib/newrelic/synthetics/variables:rw ...

Podman example:

In case of SELinux, mount the volume additionally with :z or :Z. For more information, refer Podman documentation. The expected target directory is: /var/lib/newrelic/synthetics/variables/

$
podman run ... -v /variables:/var/lib/newrelic/synthetics/variables:rw,z ...

Kubernetes example:

The user has two options when providing a file to the SJM pod in Kubernetes. They may:

• Pass in a local file.
• Provide a PersistentVolume that includes the user_defined_variables.json.

Pass in a local file

This option creates a ConfigMap Kubernetes resource and mounts that to the SJM pod.

$
helm install newrelic/synthetics-job-manager ... --set-file "synthetics.userDefinedVariables.userDefinedFile=[local-path]/user_defined_variables.json" ...

Mount a PersistentVolume

This option requires the user to provide a PersistentVolume that includes the user_defined_variables.json file or a PersistentVolumeClaim to the same. For more details on helm chart installation using a PersistentVolume, follow the instructions at permanent data storage.

Once the user has prepared a PersistentVolume as described below, launch the SJM, setting the path where the user_defined_variables.json file is located, and set any other synthetics.persistence variables as necessary.

$
helm install newrelic/synthetics-job-manger ... --set synthetics.userDefinedVariables.userDefinedPath="variables"

The variables may be passed to their respective container system via environment variable.

Docker example:

Use the -e flag to set up an environment variable named USER_DEFINED_VARIABLES and give it the value of a JSON formatted map string.

$
docker run ... -e USER_DEFINED_VARIABLES='{"key":"value","name":"sjm"}' ...

Podman example:

Use the -e flag to set up an environment variable named USER_DEFINED_VARIABLES and give it the value of a JSON formatted map string.

$
podman run ... -e USER_DEFINED_VARIABLES='{"key":"value","name":"sjm"}' ...

Kubernetes example:

Use the --set-literal flag to pass in the JSON formatted string.

$
helm install newrelic/synthetics-job-manager ... --set-literal synthetics.userDefinedVariables.userDefinedJson='{"key":"value","name":"sjm"}' ...

For Docker, launch SJM mounting the directory at /var/lib/newrelic/synthetics/modules. For example:

$
docker run ... -v /example-custom-modules-dir:/var/lib/newrelic/synthetics/modules:rw ...

For podman, launch SJM mounting the directory at /var/lib/newrelic/synthetics/modules. In case of SELinux, mount the volume additionally with with :z or :Z. For more information, refer Podman documentation. For example:

$
podman run ... -v /example-custom-modules-dir:/var/lib/newrelic/synthetics/modules:rw,z ...

For Kubernetes, the directory at /var/lib/newrelic/synthetics/modules needs to exist on a PV prior to launching the SJM with custom modules enabled.

One method is to create a pod that mounts the PV just for the purpose of copying your custom modules directory to the PV. The following example uses Amazon EFS with Amazon EKS:

Create the namespace, persistent volume, and persistent volume claim

1. Make sure you've already set up your EFS filesystem and installed the EFS CSI driver on your cluster. You will also need your EFS filesystem ID for the PV's spec.csi.volumeHandle.

   bash

   Copy

   $
   kubectl apply -f - <<EOF
   $
   apiVersion: v1
   $
   kind: Namespace
   $
   metadata:
   $
     name: newrelic
   $
   
   $
   ---
   $
   kind: StorageClass
   $
   apiVersion: storage.k8s.io/v1
   $
   metadata:
   $
     name: efs-sc
   $
   provisioner: efs.csi.aws.com
   $
   
   $
   ---
   $
   apiVersion: v1
   $
   kind: PersistentVolume
   $
   metadata:
   $
     name: custom-modules-pvc
   $
   spec:
   $
     capacity:
   $
       storage: 5Gi
   $
     volumeMode: Filesystem
   $
     accessModes:
   $
       - ReadWriteMany
   $
     persistentVolumeReclaimPolicy: Retain
   $
     storageClassName: efs-sc
   $
     csi:
   $
       driver: efs.csi.aws.com
   $
       volumeHandle: <your-efs-filesystem-id>
   $
   
   $
   ---
   $
   apiVersion: v1
   $
   kind: PersistentVolumeClaim
   $
   metadata:
   $
     name: custom-modules-pvc
   $
     namespace: newrelic
   $
   spec:
   $
     accessModes:
   $
       - ReadWriteMany
   $
     storageClassName: efs-sc
   $
     resources:
   $
       requests:
   $
         storage: 5Gi
   $
   EOF

2. Switch to the newrelic namespace in your ~/.kube/config.

   bash

   Copy

   $
   kubectl config get-contexts
   $
   kubectl config set-context YOUR_CONTEXT --namespace=newrelic
   $
   kubectl config view --minify | grep namespace:

3. At this point, the PVC should be bound to the PV with RWX access mode.

   bash

   Copy

   $
   kubectl get pv,pvc
   NAME                                  CAPACITY   ACCESS MODES   RECLAIM POLICY   STATUS   CLAIM                         STORAGECLASS   VOLUMEATTRIBUTESCLASS   REASON   AGE
   persistentvolume/custom-modules-pvc   5Gi        RWX            Retain           Bound    newrelic/custom-modules-pvc   efs-sc         <unset>                          4m46s
   NAME                                       STATUS   VOLUME               CAPACITY   ACCESS MODES   STORAGECLASS   VOLUMEATTRIBUTESCLASS   AGE
   persistentvolumeclaim/custom-modules-pvc   Bound    custom-modules-pvc   5Gi        RWX            efs-sc         <unset>                 4m10s

   Create mount-custom-mods-pod to copy your custom-modules directory

   bash

   Copy

   $
   kubectl apply -f - <<EOF
   $
   apiVersion: v1
   $
   kind: Pod
   $
   metadata:
   $
     name: mount-custom-mods-pod
   $
   spec:
   $
     containers:
   $
     - name: mount-custom-mods-pod
   $
       image: nginx
   $
       resources:
   $
         requests:
   $
           memory: "64Mi"
   $
           cpu: "250m"
   $
         limits:
   $
           memory: "128Mi"
   $
           cpu: "500m"
   $
       volumeMounts:
   $
         - mountPath: "/var/lib/newrelic/synthetics/modules"
   $
           name: custom-modules-storage
   $
     volumes:
   $
     - name: custom-modules-storage
   $
       persistentVolumeClaim:
   $
         claimName: custom-modules-pvc
   $
   EOF

   At this point, the mount-custom-mods-pod should be created and configured to use the volume.

   bash

   Copy

   $
   kubectl describe po mount-custom-mods-pod | grep -A4 Volumes:
   Volumes:
     custom-modules-storage:
       Type:       PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
       ClaimName:  custom-modules-pvc
       ReadOnly:   false

   Check events for any warnings related to the PV, PVC, or mount-custom-mods-pod.

   bash

   Copy

   $
   kubectl get events --field-selector type=Warning --sort-by='.lastTimestamp'

   Copy your custom-modules directory to the PV

   It's not necessary to copy node_modules as it will be generated by the SJM on npm install.

   bash

   Copy

   $
   cd custom-modules
   $
   rm -rf node_modules && cd ..

4. Check that the mount-custom-mods-pod is running.

   bash

   Copy

   $
   kubectl get po
   NAME                    READY   STATUS    RESTARTS   AGE
   mount-custom-mods-pod   1/1     Running   0          5m43s

5. Copy to the PV.

   bash

   Copy

   $
   kubectl cp custom-modules newrelic/mount-custom-mods-pod:/var/lib/newrelic/synthetics/modules

6. Check that /var/lib/newrelic/synthetics/modules/custom-modules/package.json exists on the PV.

   bash

   Copy

   $
   kubectl exec -it mount-custom-mods-pod -- bash
   root@mount-custom-mods-pod:/# cd /var/lib/newrelic/synthetics/modules/
   root@mount-custom-mods-pod:/var/lib/newrelic/synthetics/modules# ls -l
   total 4
   drwxr-xr-x 2 root root 6144 Jun 29 03:49 custom-modules
   root@mount-custom-mods-pod:/var/lib/newrelic/synthetics/modules# ls -l custom-modules/
   total 4
   -rw-r--r-- 1 501 staff 299 Jun 29 03:49 package.json

   Launch the SJM with custom modules feature enabled

   Set values for persistence.existingClaimName and customNodeModules.customNodeModulesPath either in the command line or in a YAML file during installation. The customNodeModules.customNodeModulesPath value should specify the subpath on the Persistent Volume where your custom modules files exist. For example:

   bash

   Copy

   $
   helm upgrade --install synthetics-job-manager newrelic/synthetics-job-manager -n newrelic --set global.persistence.existingClaimName=custom-modules-pvc --set global.customNodeModules.customNodeModulesPath=custom-modules --set synthetics.privateLocationKey=YOUR_PRIVATE_LOCATION_KEY
   Release "synthetics-job-manager" does not exist. Installing it now.
   NAME: synthetics-job-manager
   LAST DEPLOYED: Fri Jun 28 16:53:28 2024
   NAMESPACE: newrelic
   STATUS: deployed
   REVISION: 1
   TEST SUITE: None

   The custom-modules directory should now contain the installed packages in node_modules.

   bash

   Copy

   $
   kubectl exec -it mount-custom-mods-pod -- bash
   root@mount-custom-mods-pod:/# cd /var/lib/newrelic/synthetics/modules/
   root@mount-custom-mods-pod:/var/lib/newrelic/synthetics/modules# ls -l custom-modules/
   total 16
   -rw-r--r--  1 root root   836 Jun 29 03:51 README
   drwxr-xr-x 18 root root  6144 Jun 29 03:51 node_modules
   -rw-r--r--  1  501 staff  299 Jun 29 03:49 package.json
   -rw-r--r--  1 root root   190 Jun 29 03:51 package.json.shasum

   If custom node modules are not detected, adjust permissions on the custom-modules directory and package.json file.

   bash

   Copy

   $
   kubectl exec -it mount-custom-mods-pod -- bash
   root@mount-custom-mods-pod:/# cd /var/lib/newrelic/synthetics/modules/
   root@mount-custom-mods-pod:/var/lib/newrelic/synthetics/modules# chmod -R 777 custom-modules
   root@mount-custom-mods-pod:/var/lib/newrelic/synthetics/modules# chown -R 2000:2000 custom-modules

To set permanent data storage on Docker:

1. Create a directory on the host where you are launching the Job Manager. This is your source directory.
2. Launch the Job Manager, mounting the source directory to the target directory /var/lib/newrelic/synthetics.

Example:

$
docker run ... -v /sjm-volume:/var/lib/newrelic/synthetics:rw ...

To set permanent data storage on Podman:

1. Create a directory on the host where you are launching the Job Manager. This is your source directory.
2. Launch the Job Manager, mounting the source directory to the target directory /var/lib/newrelic/synthetics.

Example:

$
podman run ... -v /sjm-volume:/var/lib/newrelic/synthetics:rw,z ...

Kubernetes

To set permanent data storage on Kubernetes, the user has two options:

1. Provide an existing PersistentVolumeClaim (PVC) for an existing PersistentVolume (PV), setting the synthetics.persistence.existingClaimName configuration value. Example:

   bash

   Copy

   $
   helm install ... --set synthetics.persistence.existingClaimName=sjm-claim ...

2. Provide an existing PersistentVolume (PV) name, setting the synthetics.persistence.existingVolumeName configuration value. Helm will generate a PVC for the user. The user may optionally set the following values as well:

• synthetics.persistence.storageClass: The storage class of the existing PV. If not provided, Kubernetes will use the default storage class.
• synthetics.persistence.size: The size for the claim. If not set, the default is currently 2Gi.

$
helm install ... --set synthetics.persistence.existingVolumeName=sjm-volume --set synthetics.persistence.storageClass=standard ...

To ensure your private location runs efficiently, you must provision enough CPU resources on your host to handle your monitoring workload. Many factors impact sizing, but you can quickly estimate your needs. You'll need 1 CPU core for each heavyweight monitor (i.e., simple browser, scripted browser, or scripted API monitor). Below are two formulas to help you calculate the number of cores you need, whether you're diagnosing a current setup or planning for a future one.

Formula 1: Diagnosing an Existing Location

If your current private location is struggling to keep up and you suspect jobs are queuing, use this formula to find out how many cores you actually need. It's based on the observable performance of your system.

• $C_{est}$ = Estimated CPU Cores.
• $R_{proc}$ = The rate of heavyweight jobs being processed per minute.
• $R_{growth}$ = The rate your jobManagerHeavyweightJobs queue is growing per minute.
• $D_{avg,m}$ = The average duration of heavyweight jobs in minutes.

This formula calculates your true job arrival rate by adding the jobs your system is processing to the jobs that are piling up in the queue. Multiplying this total load by the average job duration tells you exactly how many cores you need to clear all the work without queuing.

Formula 2: Forecasting a New or Future Location

If you're setting up a new private location or planning to add more monitors, use this formula to forecast your needs ahead of time.

• $C_{est}$ = Estimated CPU Cores.
• $N_{mon}$ = The total number of heavyweight monitors you plan to run.
• $D_{avg,m}$ = The average duration of a heavyweight job in minutes.
• $P_{avg,m}$ = The average period for heavyweight monitors in minutes (e.g., a monitor that runs every 5 minutes has $P_{avg,m} = 5$).

This calculates your expected workload from first principles: how many monitors you have, how often they run, and how long they take.

Important sizing factors

When using these formulas, remember to account for these factors:

• Job duration ($D_{avg,m}$): Your average should include jobs that time out (often 3 minutes), as these hold a core for their entire duration.
• Job failures and retries: When a monitor fails, it's automatically retried. These retries are additional jobs that add to the total load. A monitor that consistently fails and retries effectively multiplies its period, significantly impacting throughput.
• Scaling out: In addition to adding more cores to a host (scaling up), you can deploy additional synthetics job managers with the same private location key to load balance jobs across multiple environments (scaling out).

It's important to note that a single Synthetics Job Manager (SJM) has a throughput limit of approximately 15 heavyweight jobs per minute. This is due to an internal threading strategy that favors the efficient competition of jobs across multiple SJMs over the raw number of jobs processed per SJM. If your calculations indicate a need for higher throughput, you must scale out by deploying additional SJMs. You can check if your job queue is growing to determine if more SJMs are needed.

Adding more SJMs with the same private location key provides several advantages:

• Load balancing: Jobs for the private location are distributed across all available SJMs.
• Failover protection: If one SJM instance goes down, others can continue processing jobs.
• Higher total throughput: The total throughput for your private location becomes the sum of the throughput from each SJM (e.g., two SJMs provide up to 30 jobs/minute).

NRQL queries for diagnosis

You can run these queries in the query builder to get the inputs for the diagnostic formula. Make sure to set the time range to a long enough period to get a stable average.

1. Find the rate of jobs processed per minute ($R_{proc}$): This query counts the number of non-ping (heavyweight) jobs completed over the last day and shows the average rate per minute.

FROM SyntheticCheck
SELECT rate(uniqueCount(id), 1 minute) AS 'job rate per minute'
WHERE location = 'YOUR_PRIVATE_LOCATION' AND type != 'SIMPLE'
SINCE 1 day ago

2. Find the rate of queue growth per minute ($R_{growth}$): This query calculates the average per-minute growth of the jobManagerHeavyweightJobs queue on a time series chart. A line above zero indicates the queue is growing, while a line below zero means it's shrinking.

FROM SyntheticsPrivateLocationStatus
SELECT derivative(jobManagerHeavyweightJobs, 1 minute) AS 'queue growth rate per minute'
WHERE name = 'YOUR_PRIVATE_LOCATION'
TIMESERIES SINCE 1 day ago

3. Find total number of heavyweight monitors ($N_{mon}$): This query finds the unique count of heavyweight monitors.

FROM SyntheticCheck
SELECT uniqueCount(monitorId) AS 'monitor count'
WHERE location = 'YOUR_PRIVATE_LOCATION' AND type != 'SIMPLE'
SINCE 1 day ago

4. Find average job duration in minutes ($D_{avg,m}$): This query finds the average execution duration of completed non-ping jobs and converts the result from milliseconds to minutes. executionDuration represents the time the job took to execute on the host.

FROM SyntheticCheck
SELECT average(executionDuration)/60e3 AS 'avg job duration (m)'
WHERE location = 'YOUR_PRIVATE_LOCATION' AND type != 'SIMPLE'
SINCE 1 day ago

5. Find average heavyweight monitor period ($P_{avg,m}$): If the private location's jobManagerHeavyweightJobs queue is growing, it isn't accurate to calculate the average monitor period from existing results. This will need to be estimated from the list of monitors on the Synthetic Monitors page. Make sure to select the correct New Relic account and you may need to filter by privateLocation.

Note about ping monitors and the pingJobs queue

Ping monitors are different. They are lightweight jobs that do not consume a full CPU core each. Instead, they use a separate queue (pingJobs) and run on a pool of worker threads.

While they are less resource-intensive, a high volume of ping jobs, especially failing ones, can still cause performance issues. Keep these points in mind:

• Resource model: Ping jobs utilize worker threads, not dedicated CPU cores. The core-per-job calculation does not apply to them.
• Timeout and retry: A failing ping job can occupy a worker thread for up to 60 seconds. It first attempts an HTTP HEAD request (30-second timeout). If that fails, it immediately retries with an HTTP GET request (another 30-second timeout).
• Scaling: Although the sizing formula is different, the same principles apply. To handle a large volume of ping jobs and keep the pingJobs queue from growing, you may need to scale up and/or scale out. Scaling up means increasing cpu and memory resources per host or namespace. Scaling out means adding more instances of the ping runtime. This can be done by deploying more job managers on more hosts, in more namespaces, or even within the same namespace. Alternatively, the ping-runtime in Kubernetes allows you to set a larger number of replicas per deployment.

Each runtime used by the Kubernetes and OpenShift synthetic job manager can be sized independently by setting values in the helm chart. The node-api-runtime and node-browser-runtime are sized independently using a combination of the parallelism and completions settings.

• The parallelism setting controls how many pods of a particular runtime run concurrently.
• The completions setting controls how many pods must complete before the CronJob starts another Kubernetes Job for that runtime.

Best practices for sizing your deployment

It's often not possible to precisely calculate the needed parallelism and completions values because the average duration as seen in New Relic might not be accurate, especially if the existing private location is not working well. Follow this practical approach to dial in parallelism and completions. The equations below can be used to get ballpark values to start from.

1. Estimate completions and parallelism

Do your best to estimate the average execution duration and number of jobs per 5 minutes. This provides you with a ballpark starting point for the next step, which will involve trial and error to tune the parallelism and completions values in a working cluster. Make sure to scale them proportionally, for example, going from the defaults of 1 and 6 to 10 and 60.

Estimated Completions: This determines how long your 5-minute job load will take to complete.

-- Get average execution duration in minutes
FROM SyntheticCheck
SELECT average(executionDuration / 60e3) AS 'Avg Duration (min)'
WHERE type != 'SIMPLE' AND location = 'YOUR_PRIVATE_LOCATION'
SINCE 1 hour ago

Where $D_{avg,m}$ is your average job execution duration in minutes.

Estimated Parallelism: This determines how many workers (pods) you need running concurrently to handle your 5-minute job load.

-- Get jobs per 5 minutes
FROM SyntheticCheck
SELECT rate(uniqueCount(id), 5 minutes) AS 'Number of monitor jobs per 5 minutes'
WHERE type != 'SIMPLE' AND location = 'YOUR_PRIVATE_LOCATION'
SINCE 1 hour ago

Where $N_m$ is your number of jobs per 5 minutes. This $P_{est}$ value is your estimated parallelism.

2. Perform a Helm deploy

Perform a Helm deploy with estimated parallelism and completions values, and your best guess for ping-runtime.replicaCount given the number of cpu cores per node and the number of ping monitors that need to run per minute.

3. Monitor queue growth

With the synthetic monitors configured to send jobs to the private location, check for queue growth on a timeseries line chart for pingJobs and jobManagerHeavyweightJobs.

• If the pingJobs queue has a positive slope, increase ping-runtime.replicaCount and redeploy.
• If the jobManagerHeavyweightJobs queue has a positive slope, increase parallelism and completions proportionally until the queue is no longer growing (negative slope).

A negative slope indicates that the job manager has enough parallelism to handle the job demand. It will eventually reach zero with a negative slope.

FROM SyntheticsPrivateLocationStatus
SELECT average(jobManagerHeavyweightJobs) AS 'Heavyweight Queue Growth', average(pingJobs) AS 'Ping Queue Growth'
WHERE name = 'YOUR_PRIVATE_LOCATION'
SINCE 1 day ago TIMESERIES

4. Tune based on pod running state

With the queue decreasing or at zero, check for node-api-runtime and node-browser-runtime pods that are in a "running" state for 10+ minutes. This indicates that parallelism is set too high and there are more pods than needed.

To avoid wasting resources unnecessarily, decrease parallelism and completions to reduce the age of each "running" runtime pod. If targeting a Kubernets job age of 5 minutes, runtime pods should be in a running state for less than 5 minutes, meaning the pod was created, it quickly received a job to run and completed.

5. Scale out if necessary

If the queue is not decreasing, yet there are many pods in a "running" state for 10+ minutes, it's likely that the job manager is hitting its performance bottleneck. The next thing to do is decrease parallelism and scale out with one or more additional deployments.

For example, with parallelism: 100, completions: 600 the queue is still growing yet there are many pods in a "running" state for 10+ minutes, and the Kubernetes Job age is 20 minutes ... set parallelism: 50, completions: 200 and scale horizontally (out) by adding 2 additional deployments. This yields a total of 150 parallel pods and should reduce the K8s job age to less than 20 minutes while also reducing the number of long-lived "running" pods. Aim for a K8s job age of 5-10 minutes.

For more information on adding deployments, see Scaling out with multiple SJM deployments.

-- monitors per minute per SJM
FROM SyntheticCheck SELECT
  round(rate(uniqueCount(id), 1 minute)/uniqueCount(minionId),0.1) AS 'heavy jobs per minute per SJM',
  uniqueCount(minionId) AS 'number of SJMs (namespaces)',
  round(rate(uniqueCount(id), 1 minute),0.1) AS 'heavy jobs per minute total'
WHERE minionContainerSystem = 'KUBERNETES' AND minionDeploymentMode = 'private' AND location = 'YOUR_PRIVATE_LOCATION' AND type != 'SIMPLE' FACET location SINCE 1 hour ago TIMESERIES

Scaling out with multiple SJM deployments

To scale beyond the 15 jobs/minute throughput of a single SJM, you must install multiple, separate SJM Helm releases.

The correct strategy is to deploy multiple SJM instances, each as its own Helm release. Each SJM will compete for jobs from the same private location, providing load balancing, failover protection, and an increased total job throughput.

Horizontal scaling strategy

If you need to scale out, you can simplify maintenance by treating each SJM deployment as a fixed-capacity unit.

1. Set Parallelism: For each SJM, set parallelism to the same maximum that a single SJM can handle without creating too many long-lived "running" runtime pods. This maximizes the potential throughput of each SJM without wasting resources.
2. Set Completions: For each SJM, set completions to the same fixed value as well. Adjust as needed to target a 5 minute Kubernetes job age per runtime, i.e., node-browser-runtime and node-api-runtime.
3. Install Releases: Install as many separate Helm releases as you need to handle your total job demand, i.e., get the queue to zero or line chart to a negative slope.
4. Monitor and Add: Monitor the private location job queue. If it starts to grow (positive slope), simply install another Helm release (e.g., sjm-delta) using the same fixed configuration.

By fixing parallelism and completions to static values, increasing or decreasing capacity becomes a simpler process of adding or removing Helm releases. This helps to avoid wasting cluster resources on a parallelism value that is higher than the SJM can effectively utilize.

Installation example

When installing multiple SJM releases, you must provide a unique name for each release. All instances must be configured with the same private location key.

Setting the fullnameOverride is highly recommended to create shorter, more manageable resource names. For example, to install two SJMs named sjm-alpha and sjm-beta into the newrelic namespace (both using the same values.yaml with your fixed parallelism and completions):

$
# Install the first SJM deployment
$
helm upgrade --install sjm-alpha newrelic/synthetics-job-manager \
>
  -n newrelic \
>
  -f values.yaml \
>
  --set fullnameOverride=sjm-alpha \
>
  --set ping-runtime.fullnameOverride=sjm-alpha-ping \
>
  --set node-api-runtime.fullnameOverride=sjm-alpha-api \
>
  --set node-browser-runtime.fullnameOverride=sjm-alpha-browser

$
# Install the second SJM deployment to add capacity
$
helm upgrade --install sjm-beta newrelic/synthetics-job-manager \
>
  -n newrelic \
>
  -f values.yaml \
>
  --set fullnameOverride=sjm-beta \
>
  --set ping-runtime.fullnameOverride=sjm-beta-ping \
>
  --set node-api-runtime.fullnameOverride=sjm-beta-api \
>
  --set node-browser-runtime.fullnameOverride=sjm-beta-browser

You can continue this pattern (sjm-charlie, sjm-delta, etc.) for as many SJMs as needed to keep the job queue from growing.