Modern systems create a large volume of log data. You might be dealing with hundreds of gigabytes to dozens of terabytes today, and the amount will continue to increase as your system scales. When you need to search through your logs, you'll encounter hours of toil trying to uncover valuable and relevant logs. Sending all your logs to a log management tool can help reduce this toil, but you'll quickly encounter organizational hurdles and rising costs as you ingest more logs. New Relic solves this problem by providing tools to ingest only valuable logs to reduce cost, a unified UI to correlate your logs to your services, and various ways to organize your logs before your drown in them.
Whether you're setting up a log management platform for the first time or you're migrating to New Relic, this tutorial will walk you through how to use New Relic to manage a large amount of log data. You'll start by forwarding your logs to New Relic, which means sending your log data to New Relic automatically. You'll then identify what logs to ingest and which to drop. Finally you'll organize your logs through partitions and parsing.
Choosing a log management platform
Once you've identified you have a problem with managing logs, it's time to choose a log management platform. There are many platforms out there. Some focus on quick automation but sacrifice ease-of-use. Others focus on complex features, but obscure their pricing.
New Relic's philosphy when it comes to focuses on three things: we want our logs solution to be flexible, transparent, and usage-based. Let's quickly talk about what these mean:
- Flexible: Everyone needs different things from their logs. Some may need to ingest a large amount for record keeping while some may need to ingest a small amount. Some may need to heavily parse their logs while other may barely parse their logs at all. Our log management platform gives you tools to manage what you send us.
- Transparent: There are no surprises in billing. New Relic charges you only for the data you ingest at a fixed price per gigabyte.
- Usage-based: Only pay for logs you ingest. Not all logs are valuable, so there's no use in ingesting and paying for logs you will never use. In this tutorial we'll explore how to selectively ingest logs in an affordable and effective manner.
Let's begin: forward your logs
To forward your log data to New Relic, choose one or more of these options:
Log forwarding option | When to use | Install |
|---|---|---|
APM agent | By default, our APM agents do three things:
| |
Infrastructure agent | With our infrastructure agent, you can capture any logs present on your host, including your app logs. Compared to using an APM agent to report logs, this can take a little more setting up but gives you much more powerful options (for example, ability to collect custom attributes, which you can't do with agents). | |
Third-party log services | We have a wide range of integrations for other log services, including Amazon, Microsoft, Fluentd, Fluent Bit, Kubernetes, Logstash, and more. | |
Log API or TCP endpoint | When you want more precision control about what and how logs are sent to New Relic. | |
OpenTelemetry SDK | Forward logs from your apps to an OpenTelemetry collector, which can forward them to New Relic via OTLP. |
For more on log forwarding options and specific use cases, see Forward logs.