LogQL can be used to parse data out of certain log formats such as JSON or traditional Apache log formats. This results in searchable labels which reflect the log data. This conversion happens in the client browser.

For example, if you're trying to work with vets-api logs, which come from a docker container, you can use a query like this: {app="vets-api-server"} and you'll see logs such as:

20:15:55 web.1       | {"host":"fe34c4f5d79d","application":"vets-api-server","environment":"production","timestamp":"2021-09-16T20:15:55.533505Z","level":"info","level_index":2,"pid":36,"thread":"puma server threadpool 013","duration_ms":1.9736800004466204,"duration":"1.974ms","named_tags":{"request_id":"4f839e23-869c-48e2-ba4e-7fb7e0069796","remote_ip":"52.61.71.205","user_agent":"node-fetch/1.0 (+https://github.com/bitinn/node-fetch)","ref":"226118959f22385843b359d0ac9f6adc9e863cd0","csrf_token":"alN6UVi+AntpUZ12wM3GND7ytxwNKvrfeZSrB110SjH4irQCeBsp7G5rcFG9mjxpt/6hLbaKDQhJbKyoWYlcCg=="},"name":"V0::Profile::TransactionsController","message":"Completed #statuses","payload":{"controller":"V0::Profile::TransactionsController","action":"statuses","format":"JSON","method":"GET","path":"/v0/profile/status","status":401,"view_runtime":0.47,"db_runtime":0.0,"allocations":856,"status_message":"Unauthorized"}}
JSON

As you can see, the latter part of this log is JSON, however, if you were to change your query to {app="vets-api-server"} | json you'd get JSON parsing errors because the JSON data is prefixed with a timestamp, container name, and a pipe symbol. To get the JSON data out, you must use a regexp expression to extract the JSON blob to an object and you can then parse it with the built-in JSON parser.

This page outlines how to eliminate parsing errors.

Step-by-step guide

Step 1: Regexp out the timestamp

To start, we’ll regexp out the timestamp. LogQL regexp are in Go regex format.

{app="vets-api-server", filename=~".+json.log"} | regexp "(?P<time>\\d\\d:\\d\\d:\\d\\d) (?P<process_name>\\w+.\\d) \\s+ (\\|) (?P<json>.+)"
JSON

When you run this query, you now have a label in the log called time that contains the timestamp.

New time label

New time label

Step 2: Take care of container name and pipe symbol, and stash the JSON part of the log into a named matcher

{app="vets-api-server", filename=~".+json.log"} | regexp "(?P<time>\\d\\d:\\d\\d:\\d\\d) (?P<process_name>\\w+.\\d) \\s+ (\\|) (?P<json>.+)" | line_format "{{.json}}" | json
CODE

Note the new labels in the screenshot below.

New labels

New labels

Step 3: Use the line_format operator to pass the rest to the JSON parser.

{app="vets-api-server", filename=~".+json.log"} | regexp "(?P<time>\\d\\d:\\d\\d:\\d\\d) (?P<process_name>\\w+.\\d) \\s+ (\\|) (?P<json>.+)" | line_format "{{.json}}" | json
JSON

All the fields in the JSON blob are now added to the list of labels.

List of labels

List of labels

You may now use those labels to continue to refine your log search. Please note that clicking the magnifying glass in the UI to add those things to your query may not work correctly, you should refine your query in the query input box.

Also, keep in mind that you can choose multiple labels as your initial batch of log files, like {app=~"vets-api-.+} to choose both vets-api-server and vets-api-worker logs, or even two different applications if you’re trying to correlate logs between services.

External resources:

Loki Home Page

Loki Release Log

Loki Source Code

Grafana Blog - “How labels in Loki can make log queries faster and easier”