Skip to content

Grafana Loki Workshop - Quick Reference Guide

Version: 1.0 Date: 2026-04-15 Source: https://github.com/grafana/loki-workshop Documentation: https://grafana.github.io/loki-workshop/

Table of Contents

  1. Workshop Overview
  2. Prerequisites and Environment
  3. Lab 1: Log Exploration (Logs Drilldown)
  4. Lab 2: LogQL Queries and Correlation
  5. Lab 3: Metric Queries
  6. Lab 4: Dashboard Building
  7. LogQL Reference
  8. Common Commands and Patterns
  9. Docker Compose Setup (Local Run)

Workshop Overview

Learning Objectives

The workshop covers three main areas:

  1. Navigating and exploring logs in Grafana
  2. Using the Logs Drilldown app
  3. Filtering, searching, and drilling down

  4. Instant metrics from logs

  5. Writing and running Loki queries

  6. LogQL syntax
  7. Label and line filters
  8. Query-time JSON parsing

  9. Correlation between metrics, logs, and traces

  10. Visualizing Loki query results on dashboards

  11. Metric extraction from logs
  12. Unwrapped range aggregations
  13. Building dashboard panels
  14. Line format transformations

Lab Structure

  • Lab 1: Exploring Logs (Logs Drilldown App)
  • Lab 2: Query and Correlate Logs (LogQL + Prometheus/Tempo)
  • Lab 3: Metric Queries (count_over_time, unwrap, aggregations)
  • Lab 4: Dashboard Building (95th percentile, Geomap, Log panels)

Prerequisites and Environment

Grafana Requirements

  • Version: Grafana 11 or higher
  • Access: Administrator permissions
  • Required plugin: Logs Drilldown (must be installed)

Plugin installation (self-hosted instance): 

Administration → Plugins → Logs Drilldown → Install

Required Permissions

  • Add data sources
  • Create dashboards
  • Use Explore

Data Sources

The workshop requires three data sources (connection details provided by presenter):

  1. Loki - Log aggregation and querying
  2. Prometheus - Metrics correlation
  3. Tempo - Trace correlation

Grafana Cloud Alternative

  • Free tier: https://grafana.com/auth/sign-up/create-user
  • Advantage: Managed Grafana instance with Logs Drilldown plugin pre-installed

Lab 1: Log Exploration (Logs Drilldown)

1.1. Viewing Service Logs

Opening Logs Drilldown App: 

Left menu → Drilldown → Logs

Service list: - All services listed by service_name label - Scroll and click "Show logs" button

Log View components: 1. Log volume chart - Log volume over time 2. Log lines list - Service log entries 3. Time range picker - Top right corner

1.2. Log Line Details

Expanding a log line: - Click on a log line → details appear

Field types:

Icon Type Description
I Indexed labels Labels in Loki's index
P Parsed labels Query-time parsed fields
S Structured metadata Un-indexed key-value pairs

Field actions: - Magnifying glass (+) - Add field to filter (include) - Magnifying glass (-) - Exclude field from filter

1.3. Searching and Filtering

String search: 

Search bar: favicon
- Only shows lines containing the "favicon" string

Adding field filter: 1. Expand a log line 2. Next to field: click magnifying glass (+) icon 3. Example: status_code = 404

Modifying filters: - Click on filter chips at the top - Select new value from dropdown

Example combination: 

Text: favicon
status_code: 404
→ Only lines containing "favicon" AND with 404 status_code

1.4. Metrics from Logs (Labels Tab)

Opening Labels tab: - From tab row above log volume chart: Labels

What it shows: - Breakdown by Loki labels - Chart for each label value

Drilldown: 1. Select a panel (e.g., http_method) 2. Select button → Breakdown by this label 3. Include button next to a value → Filter to only this value

Example workflow: 

Labels tab → http_method panel → Select → GET value → Include
→ Only GET requests visible

1.5. Returning to Logs Tab

  • Click Logs tab → Log list visible again
  • Applied filters persist

Lab 2: LogQL Queries and Correlation

2.1. Switching to Explore Mode

Logs Drilldown → Explore: 

Log panel context menu (⋮) → Open in Explore

What we see: - The LogQL query generated by Logs Drilldown - Log volume chart - Query editor

2.2. LogQL Syntax Basics

Basic LogQL query structure: 

{my_label="value"} |= `foo`

Components:

Part Description
{label="value"} Stream selector - which log streams to select
\|= Line filter - contains string
\|~ Regex filter - regex pattern match
\| json Parser - JSON parsing
\| logfmt Parser - logfmt parsing

Example query: 

{service_name=`web_app_3`} | json |~ `(?i)favicon\.ico`

Explanation: 1. {service_name=web_app_3} - Only web_app_3 service logs 2. | json - Parse JSON fields 3. |~(?i)favicon.ico`` - Regex filter: favicon.ico (case-insensitive)

2.3. Table View

Activating Table toggle: 

Top right of Logs panel: Table

Fields panel: - Left side panel: Detected labels and fields - Checkboxes: Add/remove columns

Example columns: - status_code - http_method - Line (original log line)

Export functions: 

Download button → Text / JSON / CSV

2.4. Prometheus Metrics Correlation

Writing Prometheus query: 

Explore → PromCorrelation data source

Using Metrics Browser: 

Metrics browser button → Select a metric: web → web_http_requests

Adding label filter: 

web_http_requests{service="web_app_3"}

Correlation setup:

  1. Split view: Split button (next to time picker)
  2. Second panel: Switch to LokiCorrelation data source
  3. Time sync: Chain button (next to time picker) - orange when active
  4. Zoom on Prometheus chart: Click + drag to select time period → Loki panel automatically follows time range

Error searching: 

{service="web_app_3", error_level="ERROR"}
→ Detect out of memory error

2.5. Tempo Traces Correlation

Finding Trace ID: 1. Zoom out on time range (zoom out button) 2. Search for log line with tempo_trace_id field

Opening trace: 1. Expand log line 2. Next to Trace ID field: Tempo or View traces button (blue) 3. Grafana Trace View opens

What we see: - Full request/response flow as spans - Bottleneck identification - Latency breakdown

Lab 3: Metric Queries

3.1. Log Range Aggregation (count_over_time)

Goal: Calculate metric based on number of log lines

Example: Googlebot request rate by status code 

sum by(status) (
count_over_time(
{filename="/var/log/nginx/json_access.log"} 
|= `Googlebot` 
| json 
[5m]
)
)

Explanation: 1. {filename="..."} - Stream selector 2. |= 'Googlebot' - Line filter 3. | json - JSON parser 4. count_over_time(...[5m]) - Count log lines in 5-minute windows 5. sum by(status) - Aggregate by status label

Query-time parsing: - Loki doesn't pre-index JSON content - | json parser can extract any field during query - These fields can be used for aggregation

3.2. Unwrapped Range Aggregation

Goal: Calculate metric based on value in log line (e.g., bytes_sent, request_time)

Example: Average bytes sent to Googlebot 

avg_over_time(
{filename="/var/log/nginx/json_access.log"} 
|= "Googlebot" 
| json 
| unwrap bytes_sent 
[5m]
) by (host)

Explanation: 1. | json - Parse JSON 2. | unwrap bytes_sent - Use the bytes_sent field value 3. avg_over_time(...[5m]) - Average calculation in 5-minute windows 4. by (host) - Breakdown by host

Example: Maximum bytes sent 

max_over_time(
{filename="/var/log/nginx/json_access.log"} 
|= "Googlebot" 
| json 
| unwrap bytes_sent 
[5m]
) by (host)

Unwrap functions:

Function Description
avg_over_time() Average value in time window
max_over_time() Maximum value
min_over_time() Minimum value
sum_over_time() Sum
quantile_over_time(0.95, ...) 95th percentile

3.3. Combining Multiple Queries

Add query button: 

+ Add query

Example: All requests vs Googlebot requests 

# Query A: All requests
sum by (request_method) (
count_over_time(
{filename="/var/log/nginx/json_access.log"} 
| json 
[5m]
)
)

# Query B: Googlebot requests by status
sum by(status) (
count_over_time(
{filename="/var/log/nginx/json_access.log"} 
|= `Googlebot` 
| json 
[5m]
)
)

→ Both queries appear on the same chart for comparison

Lab 4: Dashboard Building

4.1. Creating Dashboard

New dashboard: 

+ button (top right) → Dashboard

Save dashboard: 

Save dashboard button → Name: "My Loki Dashboard"

Set time range: 

Time picker (top right) → Last 3 hours

4.2. Panel 1: 95th Percentile Request Time

Add visualization: 

Add Visualization button → LokiNGINX datasource

Query 1: 95th percentile 

quantile_over_time(0.95,
{filename="/var/log/nginx/json_access.log"} 
| json 
| upstream_cache_status="MISS" 
| unwrap request_time 
| __error__="" 
[5m]
) by (host)

Query 2: Max request time 

max_over_time(
{filename="/var/log/nginx/json_access.log"} 
| json 
| upstream_cache_status="MISS" 
| unwrap request_time 
| __error__="" 
[1m]
) by (host)

Legend settings: 

Query 1 Options → Legend: {{host}} - 95%
Query 2 Options → Legend: {{host}} - max

Panel title: 

Panel options → Title: "95th percentile of Request Time"

What it shows: - 95% of requests faster than this value - Maximum request time every 1 minute - Only cache MISS requests (cache hits filtered out)

4.3. Panel 2: Googlebot Request Percentage (Stat Panel)

Add visualization: 

Add → Visualization → LokiNGINX datasource

Query: Percentage calculation 

sum(rate(({filename="/var/log/nginx/json_access.log"} |= "Googlebot")[10m])) 
/ 
(sum(rate(({filename="/var/log/nginx/json_access.log"} |= "Mozilla")[10m])) / 100)

Explanation: - rate(...[10m]) - Request rate per second in 10-minute windows - Googlebot requests / (Mozilla requests / 100) = percentage

Visualization settings: 

Panel type: Stat (top dropdown)
Value options → Calculation: Last
Standard Options → Unit: Misc → Percent (0-100)
Panel Title: "Current % of request by Google"

What it shows: - Large, bold number (current value) - Googlebot traffic as percentage of all (Mozilla) traffic

4.4. Panel 3: Geomap (Country Code)

Add visualization: 

Add → New visualization → LokiNGINX datasource

Query: Count by country 

sum by (geoip_country_code) (
count_over_time(
{filename="/var/log/nginx/json_access.log"} 
| json 
| __error__="" 
[1m]
)
)

Legend: 

Options → Legend: {{geoip_country_code}}

Transformation: 

Transformations tab → Add transformation → Reduce
Mode: Series to rows
Calculations: Total

Visualization: 

Panel type: Geomap

Map layers: 

Layer 1 (bottom): ArcGIS MapServer
Layer 2 (top): Markers
- Location Mode: Lookup
- Lookup field: Field
- Gazetteer: Countries
- Styles Size: Total (Min: 10, Max: 40)
- Color: Fixed Color → Red
- Fill opacity: 0.8

Panel title: 

"Total requests per country"

4.5. Panel 4: Reformatted Logs (line_format)

Add visualization: 

Add → Visualization → LokiNGINX datasource
Panel type: Logs

Query: Line format transformation 

{filename="/var/log/nginx/json_access.log"} 
| json 
| line_format " request for {{.request_uri}} with HTTP status: {{.status}} "

Explanation: 1. | json - Parse JSON fields 2. | line_format "..." - Rewrite log line 3. {{.field_name}} - Reference JSON field

What it shows: - Custom formatted log lines - Decorated with emojis - Only essential information (request_uri, status)

Panel title: 

"Logs"

4.6. Dashboard Import (Demo)

Import pre-built dashboard: 

+ menu (top right) → Import dashboard
Dashboard ID: 24306
Click Load
Set Loki datasource: LokiNGINX

Set dashboard variables: 

Label name: filename
Label value: /var/log/nginx/json_access.log

Example panels: - % of 5xx requests - Top requested pages - Top user agents - Top IP addresses - Logs panel

View query: 

Panel context menu (⋮) → Explore

LogQL Reference

Stream Selectors

Label matching: 

{label="exact_value"} # Exact match
{label=~"regex_pattern"} # Regex match
{label!="not_this_value"} # Not equal
{label!~"not_this_regex"} # Regex not match

Multiple labels (AND): 

{service="web", env="prod", region="us-east-1"}

Line Filters

|= "string" # Contains string
|~ "regex" # Regex match
!= "string" # Does NOT contain
!~ "regex" # Regex NOT match

Examples: 

{service="web"} |= "error" # Contains "error"
{service="web"} |~ "(?i)error|warn" # Regex: error OR warn (case-insensitive)
{service="web"} != "debug" # Does NOT contain "debug"

Parsers

| json # Parse JSON
| logfmt # Parse logfmt (key=value)
| pattern `<pattern>` # Custom pattern
| regexp `(?P<name>regex)` # Named capture groups

JSON parser example: 

{service="web"} 
| json 
| status_code="404" # Use parsed field as filter

Label Filters (Parsed)

| json 
| label_name="value" # Parsed label filter
| label_name=~"regex" # Parsed label regex

Line Format

Rewrite log line: 

| line_format "{{.field1}} - {{.field2}}: {{.message}}"

Example: 

{service="nginx"} 
| json 
| line_format "{{.method}} {{.path}} → {{.status}}"

Label Format

Create/modify labels: 

| label_format new_label="{{.old_label}}_suffix"

Example: 

{service="web"} 
| json 
| label_format full_path="{{.host}}{{.path}}"

Metrics Functions

Count-based (log lines)

count_over_time({service="web"}[5m]) # Count log lines
rate({service="web"}[5m]) # Logs per second
bytes_over_time({service="web"}[5m]) # Total bytes

Unwrap-based (values in logs)

| unwrap field_name # Extract numeric field

avg_over_time(... | unwrap latency [5m]) # Average value
sum_over_time(... | unwrap bytes [5m]) # Sum of values
max_over_time(... | unwrap duration [5m]) # Maximum value
min_over_time(... | unwrap size [5m]) # Minimum value
quantile_over_time(0.95, ... | unwrap latency [5m]) # 95th percentile

Aggregation Operators

sum by (label) (...) # Sum grouped by label
avg by (label) (...) # Average grouped by label
max by (label) (...) # Maximum grouped by label
min by (label) (...) # Minimum grouped by label
count by (label) (...) # Count grouped by label

Without labels (all streams combined): 

sum(...) # Total sum
avg(...) # Total average

Binary Operations

Math on metrics: 

sum(rate({service="web"}[10m])) 
/ 
sum(rate({service="web"}[10m]))

Example: Percentage 

sum(rate({service="web", level="error"}[5m])) 
/ 
sum(rate({service="web"}[5m])) 
* 100

Common Commands and Patterns

1. Error Rate Calculation

Error logs per second: 

rate({service="web", level="error"}[5m])

Error percentage: 

sum(rate({service="web", level="error"}[5m])) 
/ 
sum(rate({service="web"}[5m])) 
* 100

2. HTTP Status Code Distribution

Count by status code: 

sum by (status_code) (
count_over_time(
{service="nginx"} 
| json 
[5m]
)
)

Rate of 5xx errors: 

sum(
rate(
{service="nginx"} 
| json 
| status_code=~"5.." 
[5m]
)
)

3. Top N Queries

Top 10 requested paths: 

topk(10, 
sum by (path) (
count_over_time(
{service="nginx"} 
| json 
[1h]
)
)
)

Bottom 5 slowest endpoints: 

bottomk(5, 
avg by (endpoint) (
avg_over_time(
{service="app"} 
| json 
| unwrap duration 
[15m]
)
)
)

4. Response Time Percentiles

P50, P95, P99: 

# P50 (median)
quantile_over_time(0.50, {service="web"} | json | unwrap latency [5m]) by (endpoint)

# P95
quantile_over_time(0.95, {service="web"} | json | unwrap latency [5m]) by (endpoint)

# P99
quantile_over_time(0.99, {service="web"} | json | unwrap latency [5m]) by (endpoint)

5. Correlation Patterns

Prometheus → Loki: 

# Prometheus query
http_requests_total{service="web_app"}


# Loki query (same label)
{service="web_app"}

Loki → Tempo: 

{service="web"} 
| json 
| trace_id=~".+" # Logs with trace ID
→ Click trace_id field → View traces button

6. Alert Queries (for Alerting Rules)

High error rate: 

sum(
rate(
{service="web", level="error"}[5m]
)
) 
> 10

P95 latency too high: 

quantile_over_time(0.95, 
{service="web"} 
| json 
| unwrap latency 
[5m]
) 
> 5000

7. Filtering Out Noise

Exclude health checks: 

{service="web"} 
| json 
| path!="/health" 
| path!="/ready"

Exclude specific IPs: 

{service="nginx"} 
| json 
| remote_addr!~"192\.168\..+"

8. Multi-line Log Parsing

Pattern extraction: 

{service="java"} 
| pattern `<timestamp> <level> <class> - <message>`
| level="ERROR"

Regex with named groups: 

{service="app"} 
| regexp `(?P<timestamp>\d{4}-\d{2}-\d{2}) (?P<level>\w+) (?P<msg>.+)`
| level="FATAL"

Docker Compose Setup (Local Run)

compose.yaml

Clone repository: 

git clone https://github.com/grafana/loki-workshop.git
cd loki-workshop

Environment variables (.env file): 

# Create .env file from .env.sample
cp .env.sample .env

Required variables: 

LOKI_URL=https://your-loki-instance.com
LOKI_USERNAME=your-username
LOKI_PASSWORD=your-password

TEMPO_URL=https://your-tempo-instance.com
TEMPO_USERNAME=your-username
TEMPO_PASSWORD=your-password

Docker Compose Service

compose.yaml content: 

version: '3'

services:
grafana:
image: docker.io/grafana/grafana:latest
ports:
- "3007:3000"
environment:
GF_PLUGINS_PREINSTALL: grafana-lokiexplore-app
GF_AUTH_ANONYMOUS_ENABLED: true
GF_AUTH_ANONYMOUS_ORG_ROLE: Admin
GF_USERS_DEFAULT_THEME: light
GF_LOG_FILTERS: migrator:warn
LOKI_URL: ${LOKI_URL}
LOKI_USERNAME: ${LOKI_USERNAME}
LOKI_PASSWORD: ${LOKI_PASSWORD}
TEMPO_URL: ${TEMPO_URL}
TEMPO_USERNAME: ${TEMPO_USERNAME}
TEMPO_PASSWORD: ${TEMPO_PASSWORD}
volumes:
- ./grafana/provisioning:/etc/grafana/provisioning:z

Start: 

# Docker Compose
docker-compose up -d

# Podman Compose
podman-compose up -d

Access Grafana: 

http://localhost:3007

Stop: 

docker-compose down
# or
podman-compose down

Taskfile Usage (Task Runner)

Taskfile.yaml tasks:

# Development docs server
task dev

# Generate screenshots (for workshop docs)
task screenshots

Start development docs: 

cd docs
npm start
→ Local preview of workshop docs: http://localhost:3000

Official Documentation

  • Loki Documentation: https://grafana.com/docs/loki/latest/
  • LogQL Reference: https://grafana.com/docs/loki/latest/query/
  • Loki Workshop GitHub: https://github.com/grafana/loki-workshop
  • Workshop Docs: https://grafana.github.io/loki-workshop/

Grafana Plugins

  • Logs Drilldown: https://grafana.com/grafana/plugins/grafana-lokiexplore-app/
  • Grafana Plugin Catalog: https://grafana.com/grafana/plugins/

Example Dashboards

  • NGINX Analytics (ID: 24306): Official workshop demo dashboard
  • Grafana Dashboards: https://grafana.com/grafana/dashboards/?search=loki

Learning Resources

  • Grafana Labs YouTube: Loki tutorials, webinars
  • Grafana Community: https://community.grafana.com/
  • Loki Blog Posts: https://grafana.com/blog/tags/loki/

Common Issues and Solutions

1. Logs Drilldown not loading logs

Problem: Service list empty or logs not showing

Solution: - Check the correct data source is selected (top right corner) - Ensure LokiCorrelation data source is set - Verify time range (long enough time window?)

2. Parsed fields not appearing

Problem: No fields after JSON parser

Solution: 

# Wrong: no parser
{service="web"} |= "error"

# Correct: JSON parser added
{service="web"} |= "error" | json

3. Metric query returns "no data"

Problem: count_over_time() or unwrap query returns no data

Solution: - Verify label selector finds streams - Increase time range - Check unwrap field type (must be numeric)

# Wrong: unwrap string field
| unwrap status_code # If status_code is string "404"

# Correct: unwrap numeric field
| unwrap bytes_sent # Numeric field

4. Dashboard panel shows "N/A"

Problem: Stat or Gauge panel shows "N/A" instead of number

Solution: - Value options → Calculation: "Last" (or "Last (not null)") - Query time range must match dashboard time range

5. Geomap not displaying countries

Problem: Markers not appearing on map

Solution: - Transformation: Reduce transformation in "Series to rows" mode - Lookup field: Set to "Field" - Gazetteer: Select "Countries" - Label name must match country code format (ISO 2-letter: "US", "GB", etc.)

Next Steps

Reinforcing Basics

  1. Practice Logs Drilldown app (search, filter, labels)
  2. Write basic LogQL queries (stream selector, line filter, parser)
  3. Metric extraction from logs (count_over_time, unwrap)
  4. Build dashboard panels

Advanced Topics

  • Loki Recording Rules: Pre-calculated metrics
  • Alerting Rules: Alert notifications based on LogQL queries
  • Loki Ruler: Alert rule management
  • Multi-tenancy: Organization isolation in Loki
  • Label Cardinality: Best practices for label usage

Production Deployment

  • Loki Architecture: Distributor, Ingester, Querier components
  • Storage Backend: S3, GCS, Azure Blob
  • Retention Policies: Data lifecycle management
  • Performance Tuning: Query optimization, caching

Integration

  • Promtail: Log shipping to Loki
  • Grafana Agent: Unified observability agent
  • OpenTelemetry Collector: OTEL logs to Loki
  • FluentBit/Fluentd: Alternative log collectors

Summary

This quick reference covers the complete Grafana Loki Workshop material with practical examples and commands.

During the workshop, you learned: - Use Logs Drilldown app for fast log exploration - Write LogQL queries (stream selectors, filters, parsers) - Calculate metrics from logs (count, unwrap, aggregations) - Build dashboards with Loki queries - Use correlation between Prometheus and Tempo

Next step: - Practice queries on your own Loki instance - Build production-ready dashboards - Explore advanced LogQL features (recording rules, alerts)

Happy learning!