You can configure the amount of memory Pixie uses. During the installation, use Helm to set the memory requests and limits, or to reduce the amount of memory Pixie uses for short-term data storage.
How does Pixie use memory?
The primary focus of the open source Pixie project is to build a real-time debugging platform. Pixie isn't intended to be a long-term durable storage solution and is best used in conjunction with New Relic. The New Relic integration queries Pixie every few minutes and persists a subset of Pixie's telemetry data in New Relic.
When you install the New Relic Pixie integration, a vizier-pem agent is deployed to each node in your cluster via a DaemonSet. The vizier-pem agents use memory for two main purposes:
- Collecting telemetry data: tracing application traffic or CPU profiles, amongst others. Those values must be stored in memory somewhere, as they're processed.
- Short-term storage of telemetry data: to power troubleshooting via the Live debugging with Pixie tab and as a temporary storage location for a subset of the telemetry data before it's stored in New Relic.
By default, vizier-pem pods have a 2Gi memory limit, and a 2Gi memory request. They set aside 60% of their allocated memory for short-term data storage, leaving the other 40% for the data collection.
Why does Pixie's memory use increase after installation?
After installing Pixie, memory usage of the vizier-pem pods increases as they begin storing telemetry data. Once you reach vizier-pem's memory limit, old telemetry data is expired to make room for new data, and so memory utilization shouldn't increase any further.
Configuring Pixie's memory usage
For most clusters, we recommend using the default 2Gi memory configuration. However, for certain low traffic clusters, Pixie can support a minimum memory limit of 1Gi. To accommodate application pods, we recommend that no more than 25% of the nodes' total memory be allocated to Pixie. For example, if your nodes have a total memory of 4Gi, you'll want to configure Pixie to use a 1Gi memory limit.
Deploy Pixie with a particular memory limit
If you want to specify a different memory limit than the default 2Gi for Pixie's vizier-pem agents, you can add the following configuration parameter to your Helm chart during installation. For example, for a 1Gi memory limit, you'd use:
$--set pixie-chart.pemMemoryLimit=1GiDeploy Pixie with a particular memory request
By default, the vizier-pem's memory request is the same as the limit. If you want to specify a different memory request, you can add the following configuration parameter to your Helm chart during installation. For example, for a 1Gi memory request you'd use:
$--set pixie-chart.pemMemoryRequest=1GiImportant
The vizier-pem memory request must be less than or equal to the vizier-pem memory limit.
Deploy Pixie with a particular short-term data storage
By default, vizier-pem pods set aside 60% of their allocated memory for short-term data storage, leaving the other 40% for the collection. For the default 2Gi memory limit, this means that a vizier-pem pod keeps 1.2Gi memory for data storage.
If you want to specify a different amount for memory for short-term storage, you can add the following configuration parameter to your Helm chart during installation. For example, for 750MiB of short-term data storage you'd use:
$--set pixie-chart.dataCollectorParams.customPEMFlags.PL_TABLE_STORE_DATA_LIMIT_MB=750Troubleshooting
For troubleshooting, see these docs: