19 August 2026

Operating Loki in a Real Observability Platform

by Alex Carvalho

I have spent time studying and operating Grafana Loki in high-throughput environments, and this is the sixth and final post in my series on Loki’s architecture and design. In the first post, we covered labels and chunks. In the second post, we traced a log line’s path. In the third post, we explored the hash ring. In the fourth post, we dissected storage internals. In the fifth post, we looked at query execution and sharding. Now, we pull it all together: what changes when Loki moves from an architecture diagram to a production observability platform?

The core rule is:

Operating Loki successfully at scale requires choosing a deployment mode that matches your load, establishing strict per-tenant ingestion and query boundaries, actively monitoring core performance metrics, and correlating logs with metrics and traces using metadata rather than high-cardinality indexing.

Loki is a highly scalable set of tradeoffs. When configured correctly, it provides extremely cost-effective log storage. When configured poorly—usually by abusing labels or neglecting query limits—it can become unstable and expensive.

Choosing the right deployment mode

Loki can be deployed in three modes depending on scale, operational resources, and infrastructure complexity.

  1. Single Binary Mode (All-in-one)
  ┌────────────────────────────────────────────────────────┐
  │   [Distributor, Ingester, Querier, Compactor, etc.]   │
  └───────────────────────────┬────────────────────────────┘
                              ▼
                        [Local Storage]

  2. Simple Scalable Deployment (SSD)
  ┌────────────────────────────────────────────────────────┐
  │  Write Target   ◄── Ingestion (Distributors/Ingesters) │
  ├────────────────────────────────────────────────────────┤
  │  Read Target    ◄── Querying (Queriers/Frontends)      │
  ├────────────────────────────────────────────────────────┤
  │  Backend Target ◄── Housekeeping (Compactor/Ruler)     │
  └───────────────────────────┬────────────────────────────┘
                              ▼
                       [Object Storage]

  3. Microservices Mode
  ┌───────────────┐ ┌───────────────┐ ┌───────────────┐
  │ Distributors  │ │   Ingesters   │ │   Queriers    │ ...
  └───────────────┘ └───────────────┘ └───────────────┘
                              ▼
                       [Object Storage]

1. Single Binary Mode

All Loki components run inside a single process, utilizing local disk storage or object storage. This is ideal for local development, proof-of-concepts, or environments with low log volumes (e.g., < 100 GB/day). It is simple to operate but lacks high availability.

2. Simple Scalable Deployment (SSD)

This is the recommended mode for most production environments. Loki components are grouped into three target workloads, allowing them to scale independently:

Write Target: Runs stateless distributors and stateful ingesters to process incoming log writes.
Read Target: Runs queriers and query-frontends to process log queries.
Backend Target: Runs the compactor, index-gateway, and ruler for maintenance and alerts.

By separating reads and writes into distinct targets, you prevent massive queries from impacting ingestion latency.

3. Microservices Mode

Loki’s components run as completely independent processes (usually as Kubernetes deployments or daemonsets). While this offers the maximum granular control and scaling efficiency (e.g., scaling up just the queriers during peak alert times), it adds significant network overhead and operational complexity. It is reserved for large enterprise platforms ingesting tens of terabytes of logs per day.

To test these deployment patterns hands-on, I built a Kubernetes proof of concept: github.com/alex-carvalho/sandbox/devops/observability/loki/k8s. It walks through deploying Loki on Kubernetes with the configuration and storage setup needed to run the cluster in practice.

Component scaling strategies

If you choose the Simple Scalable or Microservices deployment, scaling becomes a matter of tracking different bottlenecks across the read and write paths:

Distributors: These are CPU-bound and network-bound. Scale them horizontally based on incoming packet rates or total write bytes. They require no persistent disk storage.
Ingesters: These are highly memory-intensive because they buffer and compress active logs in memory before flushing them to object storage. Scale them out when memory usage or active stream counts grow too high. Ingesters must have fast local NVMe disks for their Write-Ahead Log (WAL) to survive sudden node restarts.
Queriers: CPU-bound. Scale queriers up or down dynamically using horizontal pod autoscalers (HPA) based on queue lengths in the Query Scheduler.
Compactor: Runs as a singleton to avoid index compaction conflicts. It requires fast local disk space to download, merge, and re-upload daily TSDB index files.

Multi-tenancy and operational boundaries

Loki is a multi-tenant system by design. In a shared observability platform, multi-tenancy ensures that distinct teams or environments (e.g., dev vs. prod) are isolated and cannot degrade each other’s performance.

Isolation via X-Scope-OrgID

Loki segregates data based on the X-Scope-OrgID HTTP header passed by log agents (like Grafana Alloy or OpenTelemetry Collector) during writes, and by Grafana during queries. The distributor routes logs to ingesters using this tenant identifier, and chunks are written to tenant-specific folders in object storage.

Ingestion and Query Limits

To prevent a single misconfigured application or user from overwhelming the cluster, you must set per-tenant limits in Loki’s configuration:

limits_config:
  # Ingestion limits
  ingestion_rate_mb: 10
  ingestion_burst_size_mb: 20
  max_streams_per_user: 10000
  max_line_size: 256kb

  # Query limits
  max_query_length: 744h # Limit queries to 31 days max
  max_query_parallelism: 32
  max_entries_limit_per_query: 5000

Shuffle Sharding

To isolate tenant failures in large-scale deployments, Loki supports Shuffle Sharding. Instead of distributing a tenant’s streams across all ingesters in the cluster, Loki assigns each tenant to a virtual subset of ingesters. If Tenant A floods the system with an massive load, they will only overwhelm their assigned subset of ingesters, leaving the remaining ingesters completely healthy for other tenants.

Rule evaluation: The Ruler

The Ruler is a component that continuously evaluates Alerting and Recording rules written in LogQL. It acts like Prometheus alerting but runs queries against your log lines.

To keep the Ruler lightweight:

Avoid running rule evaluations directly on the Ruler’s local CPU.
Configure the Ruler to evaluate rules remotely via the Query Frontend. This leverages the Query Frontend’s caching and query-splitting capabilities, keeping the Ruler stable and responsive.

Meta-monitoring: Loki metrics that matter

Operating Loki requires monitoring Loki itself. The following core metrics must be tracked in your Prometheus/Mimir monitoring dashboards:

Ingestion Errors: loki_distributor_bytes_received_total compared against loki_discarded_samples_total. If discarded samples spike, check for tenants hitting rate limits, out-of-order logs, or lines exceeding the maximum line size.
WAL Replay Health: loki_ingester_wal_replay_duration_seconds. A long WAL replay time during an ingester restart means your WAL is too large, causing the pod to crash or fail readiness checks on startup.
Flush Queues: loki_ingester_flush_duration_seconds and loki_ingester_flush_queue_length. If flush queues are growing, it indicates your object storage (e.g., S3 or GCS) is throttle-limiting writes, or there is a network bottleneck.
Query Latency: Monitor p50, p90, and p99 of loki_request_duration_seconds_bucket{route="api_v1_query_range"}. Slow queries indicate a need for more queriers or better caching configuration.
Compactor Health: loki_compactor_apply_retention_duration_seconds. If retention cycles fail or take longer than 24 hours, the compactor may need more local disk IO or memory.

Building the unified observability stack

Loki should not exist in isolation. Its power is unlocked when integrated with other observability signals (metrics and traces) to dramatically reduce Mean Time to Resolution (MTTR).

The modern, open-source telemetry flow typically looks like this:

[ Application Nodes ]
         │ (Logs, Traces, Metrics via OTLP)
         ▼
 ┌───────────────┐
 │ Grafana Alloy │ (Collector & Routing)
 └─┬───┬───┬─────┘
   │   │   │
   │   │   └─► [ Mimir / Prometheus ] ──► (Metrics) ──┐
   │   └─────► [ Grafana Loki ]       ──► (Logs)     ──┼─► [ Grafana Visualisation ]
   └─────────► [ Grafana Tempo ]      ──► (Traces)   ──┘

Correlation in Practice: The Correlation ID

When an application encounters an error, a developer should be able to jump from a metric spike to the exact log lines, and from the log lines directly to the distributed trace.

Metrics to Logs: Prometheus metrics include low-cardinality labels (like service_name="checkout", env="prod"). When viewing a metric dashboard in Grafana, you can create a link that passes those exact labels to Loki, opening a LogQL query pre-filtered to the correct stream.
Logs to Traces: Inside the log line, use Structured Metadata to attach the trace_id. Grafana detects the trace_id and adds a clickable link that opens the corresponding trace in Tempo.
Traces to Logs: When looking at a distributed trace span, you can use the span’s start/end timestamps and its tags (e.g., service.name) to query Loki for logs generated by that specific service at that exact millisecond.

By using low-cardinality labels to locate the target stream and structured metadata (coupled with Bloom filters) to find the specific trace_id, you achieve lightning-fast correlation without overloading Loki’s index.

Conclusion: The Loki Tradeoffs

Building a mental model of Grafana Loki is about recognizing that every system has to pay for search capability somewhere:

Elasticsearch pays at ingestion time by writing massive inverted indices to expensive block storage.
Loki pays at query time by scanning raw compressed data in parallel across stateless querier nodes.

Loki’s architecture makes complete sense once these tradeoffs are visible. By keeping labels low-cardinality, setting tenant boundaries, caching aggressively, and using structured metadata for high-cardinality correlation, you can operate an observability platform that is cost-effective, incredibly fast, and scalable to petabytes of log data.