In order to understand how to search and analyze the logs that you're interested in, you’ll first need to understand the data that’s contained within the log files. This includes what information is being sent, how it looks, and how it's being tagged and labeled. The labeling aspect is especially important since that's how you’ll facet, group, slice, and segment the log data.

This page outlines how to query and analyze Loki data in Grafana.

Before you begin

Log in to Grafana and select a Loki data source. (SOCKS proxy access is required)

Step-by-step guide

Step 1: View how data is structured and organized

Click “Log labels” to see how the data is structured and organized.

Tip: Most people find the app label to be the most useful.

The image below shows all log data for the app or label that’s been selected. The bar graph shows the rate of log messages for a particular label.

Log data for the app (or label)

Log data for the app or label

In the image below, the lines at the top of the image are individual log messages. Click on a specific message to see additional details, including extracted fields and other labels that are tagged onto that specific message.

Individual log messages

Individual log messages

Step 2: Try some basic data manipulation

  • Click the time picker at the top to select a range of dates/times or a specific time window.

  • Click the split icon to get two windows side-by-side, which can be helpful for manual correlation.

  • Click the “Live” button to see ingestion and processing of log data in real-time.

  • Click the various options below the graph to see how it changes the detail view.

Step 3: Try simple queries

The power of Loki is in labeling the log messages, which enables you to quickly sort, query, and slice & dice the log data. Selecting an app label will show all logs for a given app for the chosen environment.

For example, if we want to see only the web-server logs for the reverse proxy aka “revproxy”:

  • The web server logs are generated by scraping the Nginx access logs, ie /var/log/nginx/access.log

  • This can be found by looking for a web-server log entry, clicking it, and seeing the labels.

  • So, we can update our query to only show logs with both labels (app, and specific log file): {app="revproxy",filename="/var/log/nginx/access.log"}

Tip: After entering the first label, you can start typing and the filters will auto-complete with (only) valid options

Tip: You can hit “shift-enter” as a keyboard short-cut to execute your updated query

As another example, if we want to look for something a bit more specific, errors are a common thing to look for. By appending a simple operator to our query (|= “thingy”), we can search for errors in the revproxy's access logs: {app="revproxy",filename="/var/log/nginx/access.log"} |= "Error"

To recap, the image below is showing us:

All of the web-server logs (Nginx access.log files) from all revproxy instances, that contain Error over the past 12 hours

Web-server logs from all revproxy instances containing Error (past 12 hours)

Web-server logs from all revproxy instances containing “Error” (past 12 hours)