18 August 2026

Reading Billions of Logs: How Loki Queries Without Indexing Everything

by Alex Carvalho

I have been studying Grafana Loki’s architecture, and this is the fifth post in the series. In the first post, we covered labels, streams, and chunks. In the second post, we traced a log line’s write path. In the third post, we explored the hash ring. In the fourth post, we examined storage internals. Now, we face the read path: how does Loki query billions of logs quickly without a full-text index?

The core rule is:

Loki answers queries by using a metadata-only index to identify matching log streams and time ranges, then dynamically downloads, decompresses, and filters those target chunks in parallel, optimizing performance via massive query sharding and multi-tier caching.

By shifting the computational burden from ingestion indexing to query-time scanning, Loki optimizes for the reality of log data: we write logs constantly, but we query only a tiny fraction of them.

The query lifecycle: from client to object storage

Loki’s query architecture is divided into stateless, micro-scaled components that handle incoming queries, split them into smaller chunks, distribute them to workers, and merge the results.

Below is the sequence of a query execution across the read path:

[Client: Grafana/LogCLI]
         │
         ▼ (1. LogQL Query: {app="api"} |= "error")
┌──────────────────────────────────────────────┐
│                Query Frontend                │
│  - Checks Result Cache                       │
│  - Splits query by time (e.g. 1-hour slots)  │
│  - Shards query by log streams               │
└──────────────────────┬───────────────────────┘
                       │
                       ▼ (2. Enqueue fragments)
┌──────────────────────────────────────────────┐
│               Query Scheduler                │
│  - Manages per-tenant fair queues            │
└──────────────────────┬───────────────────────┘
                       │
                       ▼ (3. Pull fragment work)
┌──────────────────────────────────────────────┐
│                   Querier                    │
│  - Executes the query fragment concurrently  │
└──────┬───────────────────────┬───────────────┘
       │                       │
       │ (4a. Fetch Recent)    │ (4b. Lookup Indexes)
       ▼                       ▼
┌──────────────┐       ┌───────────────┐
│  Ingesters   │       │ Index Gateway │
│  (In-Memory) │       │ (TSDB shipper)│
└──────────────┘       └───────┬───────┘
                               │
                               ▼ (5. Find chunk locations)
                       ┌───────────────┐
                       │Object Storage │
                       │(S3/GCS Chunks)│
                       └───────┬───────┘
                               │
                               ▼ (6. Fetch compressed bytes)
                       ┌───────────────┐
                       │ Querier (Cont)│
                       │ - Decompress  │
                       │ - Filter/Parse│
                       │ - Deduplicate │
                       └───────┬───────┘
                               │
                               ▼ (7. Return merged response)
                       [Client: Grafana/LogCLI]

The Read-Path Components

Query Frontend: The entry point. It parses LogQL queries, checks the results cache, splits the query into smaller sub-queries over shorter time intervals, and shards them.
Query Scheduler: A queuing service that buffers sub-query fragments. It prevents queriers from getting overwhelmed and ensures tenant fairness in multi-tenant deployments by using FIFO queues per tenant.
Querier: The workhorse. It pulls sub-queries from the scheduler and executes the heavy lifting. It queries the ingesters for recent, un-flushed data, fetches historical index information from the Index Gateway, downloads the compressed chunks from object storage, and runs the actual text filtering/parsing.
Index Gateway: Serves lookups on the TSDB index. It translates label matchers (like app="api") into a list of stream IDs and matching chunk references.
Ingesters: Serve the latest logs held in memory or local WAL before they are flushed to object storage.

Query planning: splitting, sharding, and fairness

When a user runs a LogQL query across a large time window (e.g., a 30-day search), executing it sequentially on a single node would cause timeout issues. Loki bypasses this through aggressive parallelization.

Query Splitting

The Query Frontend inspects the query’s time range and splits it into smaller, fixed-size intervals (e.g., splitting a 30-day query into thirty 1-day queries, or a 24-hour query into 2-hour increments). This allows the scheduler to distribute chunks of work to multiple queriers simultaneously.

Query Sharding

In addition to time splitting, Loki shards queries by stream. If the TSDB index indicates a label matcher matches 1,000 distinct log streams, the Query Frontend shards the query so that different queriers process subsets of those streams in parallel. For metric queries (like count_over_time), this sharding allows Loki to process petabytes of raw log data per second by scaling horizontally.

Tenant Fairness

Loki ensures that a single user running a massive query does not starve other users. The Query Scheduler maintains separate queues for each tenant. If Tenant A submits 1,000 sub-queries and Tenant B submits 1, the scheduler alternates between the tenant queues, ensuring Tenant B’s query executes without waiting for Tenant A to finish.

The caching hierarchy

To reduce reads from object storage and accelerate query execution, Loki implements three layers of caching:

Result Cache: Caches the final parsed results of LogQL queries. Because logs in the past are immutable, Loki caches sub-interval query results. If a user runs a query from now-6h to now, Loki pulls the cached results for now-6h to now-5m and only executes the query for the last 5 minutes.
Index Cache: Caches TSDB postings lists and label indices. This prevents the Index Gateway from repeatedly reading index blocks from disk or object storage.
Chunk Cache: Caches the raw compressed chunk files fetched from object storage. This is highly effective when multiple users are looking at the same time range in Grafana dashboards.

Inside the Querier: Decompression and Dynamic Filtering

Once a Querier receives a query fragment, it downloads the target chunks from object storage (or local cache). Since chunks are immutable blocks of compressed log lines, the Querier must process them efficiently.

+-------------------------------------------------------------+
|                     QUERIER PROCESSING PIPELINE             |
+-------------------------------------------------------------+
| 1. Read Chunk Header -> Identify sub-blocks overlapping time|
| 2. Decompress targeted blocks (Snappy/Gzip)                 |
| 3. Seek to offset & scan lines sequentially                 |
| 4. Apply LogQL filters (|=, |~, !=)                         |
| 5. Run Parsers (| json, | logfmt) -> Extract fields         |
| 6. Compute aggregations (if metric query)                  |
+-------------------------------------------------------------+

Selective Block Decompression: A chunk file is divided into smaller compressed blocks (usually 8KB–256KB raw). The chunk header contains a mini-index mapping time ranges to block offsets. The Querier reads the header and only decompresses the specific blocks that contain log lines matching the query’s time range, saving CPU cycles.
Sequential Line Filtering: After decompression, the Querier scans the lines. It performs substring matches (using fast algorithms like Boyer-Moore) or regex matches.
LogQL Parsing & Aggregation: If the LogQL query includes parsers (e.g., | json or | logfmt), the Querier extracts keys and values on the fly. If it is a metric query (e.g., sum by (status) (rate({app="api"} | json [5m]))), the Querier aggregates the numbers locally before discarding the raw log lines, sending only the calculated metrics back to the Query Frontend.

Deduplication: Handling Replicas

Because Loki replicates log streams (typically with a Replication Factor of 3) to ensure durability, the Querier often retrieves overlapping log chunks from different ingesters or historical storage.

If left unhandled, the user would see duplicate log lines in Grafana. The Querier solves this by applying a deduplication step before merging results:

It tracks the unique stream labels, timestamps, and line content hashes.
If two log entries have the exact same timestamp, labels, and content, the Querier discards the duplicate.
This ensures that write-path replication does not pollute read-path accuracy.

Advanced Query Acceleration in Loki 3.0

While scanning compressed chunks is fast, searching for a single record (like a specific trace_id) across petabytes of logs can still be expensive if Loki has to download and scan every chunk for a given service. Loki 3.0 addresses this with two major features:

1. Structured Metadata

Historically, indexing high-cardinality fields like trace_id or user_id as labels was an anti-pattern because it exploded the number of streams in the TSDB index (high cardinality).

In Loki 3.0, Structured Metadata allows log forwarders to attach key-value metadata to individual log lines without creating separate streams. This metadata is saved inside the chunk file’s blocks alongside the logs, keeping the TSDB index small while allowing the Querier to filter on high-cardinality fields during execution.

2. Bloom Filters

To avoid downloading chunks that do not contain a specific high-cardinality metadata value (like {app="api"} | trace_id="12345-abc"), Loki 3.0 introduces experimental Bloom Filters.

Loki builds Bloom filters for the structured metadata keys and values inside each chunk. The Index Gateway and Querier can query the Bloom filter first. Because a Bloom filter can state with 100% certainty if a key-value pair is not present in a chunk, Loki can skip downloading that chunk entirely. This turns needle-in-a-haystack searches into highly targeted, low-bandwidth lookups.

Search: {app="api"} | trace_id="123-xyz"
 
                +-----------------------+
                |     Index Gateway     |
                | (TSDB stream search)  |
                +-----------------------+
                            |
                   [ Stream Chunks ]
                            v
                +-----------------------+
                |  Bloom Filter Check   |
                +-----------------------+
                 /                     \
       (Match = False)             (Match = True)
             /                             \
            v                               v
   [ Skip Downloading Chunk ]     [ Download & Scan Chunk ]
   (Saves Network & Disk IO)       (Returns Matched Line)

Summary

Loki’s query engine is a masterclass in scale-out design. By combining time splitting and stream sharding, it distributes massive scan operations across dozens of Queriers. Multi-level caching shields object storage from redundant queries, and block-level decompression limits CPU overhead.

With Loki 3.0’s introduction of Structured Metadata and Bloom Filters, Loki successfully addresses the “needle-in-a-haystack” problem, keeping ingestion cheap while making targeted lookups orders of magnitude faster.

In the next post, we will tie the series together by examining Operating Loki in a Real Observability Platform: comparing deployment modes, configuring multi-tenancy limits, and linking logs with traces and metrics.