28 July 2026
The Life of a Log in Grafana Loki
I have been studying Grafana Loki’s write path, and this is the second post in the series. In the first post, the main idea was that Loki indexes labels, groups log lines into streams, and stores the log content in compressed chunks.
The next question is what happens when a log line actually reaches Loki.
The core rule is:
A log is validated and routed by a distributor, replicated to ingesters, protected by the WAL, buffered into chunks, and eventually flushed to object storage with index data.
That flow is where Loki starts to look less like a single log database and more like a distributed storage system with separate responsibilities on the write path.
The simplified write path
A useful first version of the flow is:
Application / Alloy / OpenTelemetry Collector / Promtail
-> Distributor
-> validation, label normalization, limits
-> ring lookup
-> replication to ingesters
-> ingester WAL
-> in-memory stream
-> head chunk
-> compressed chunk
-> flush to object storage
-> TSDB index files
This is not every implementation detail, but it gives me the mental model I need before going deeper into the ring, storage internals, and queries.
The important thing is that the distributor does not permanently store the log. It is the stateless front door. The ingesters are the components that accept ownership of the stream data, keep recent data in memory, persist incoming writes to their local WAL, and later flush chunks to long-term storage.
Logs arrive as streams
Clients push logs to Loki as streams. A stream is still the same concept from the first post: one tenant plus one exact label set.
For example:
tenant: acme
labels: {app="checkout", namespace="production", container="api"}
entries:
2026-07-22T10:00:01Z payment request started
2026-07-22T10:00:02Z payment provider returned timeout
The label set matters immediately because Loki uses it for deterministic routing. Two log lines with the same labels belong to the same stream, and the stream is the unit that gets hashed and assigned to ingesters.
This is why label design is not only a query concern. Bad labels also hurt ingestion. A high-cardinality label can create too many streams, which means too many routing decisions, too many small in-memory buffers, and too many small chunks later.
The distributor is the stateless front door
The distributor is the first Loki component on the write path. It receives push requests from clients and does the early work that should happen before ingesters accept data.
Its responsibilities include:
- Checking that labels are valid.
- Normalizing labels into a deterministic order.
- Rejecting log lines that violate limits, such as lines that are too long or timestamps outside the accepted range.
- Applying tenant ingestion limits and rate limits.
- Hashing each stream and using the ring to choose ingesters.
- Forwarding each stream to the selected ingesters.
The distributor being stateless is a major design choice. It means distributors can be scaled horizontally behind a load balancer. If ingestion traffic grows, Loki can add more distributors to absorb validation, rate limiting, and fan-out work without making them owners of durable log data.
The durable responsibility starts when ingesters accept the write.
The ring decides where the stream goes
Once a stream passes validation, the distributor needs to decide which ingesters should receive it.
It does this with the ingester ring. Loki hashes the tenant and stream labels, then uses that hash to find the ingester tokens responsible for the stream. With replication enabled, it chooses multiple ingesters rather than only one.
A simplified version looks like this:
hash(tenant + labels)
-> ring position
-> ingester A
-> ingester B
-> ingester C
The key idea is that every distributor can make the same routing decision from the same ring state. The distributor does not need to ask a central leader where the stream belongs for every write.
I will go deeper into the ring in the next post, but the mental model is already useful: the ring gives Loki distributed ownership of streams.
Replication and quorum
In a distributed Loki deployment, a stream is usually written to more than one ingester. The common example is a replication factor of 3.
With a replication factor of 3, the distributor sends the same stream data to three ingesters. It does not need all three to respond before acknowledging the client. It waits for a quorum:
floor(replication_factor / 2) + 1
For 3, that means 2.
This matters because Loki can keep accepting writes during some failures, restarts, or rollouts. If one ingester is slow or unavailable but two replicas accept the write, the distributor can still return success.
The tradeoff is also important. If a write is acknowledged by two out of three ingesters, losing both successful replicas before the data is safely recovered or flushed can still lose data. Replication improves availability and reduces the chance of data loss, but it is not the whole durability story.
That is where the WAL enters.
WAL and replication solve different problems
The ingester WAL protects acknowledged data on the local disk of an ingester. If an ingester process crashes after accepting a write but before flushing chunks to object storage, it can replay the WAL on restart and rebuild its in-memory state.
Replication and WAL are easy to confuse because both are about reliability, but they protect different failure modes:
- Replication protects against losing availability or data when an ingester is unavailable.
- WAL protects an individual ingester’s acknowledged, not-yet-flushed data across a process crash.
A simple way to say it:
replication: another ingester has a copy
WAL: this ingester can recover its own copy
Loki’s documentation describes the WAL as local persistence for acknowledged data. That local part is important. A WAL on a lost or corrupted disk cannot help, which is why replication still matters.
Inside the ingester
After an ingester receives a stream, it appends incoming data to its WAL and updates its in-memory structures.
For each stream, the ingester builds chunks. The current writable chunk is often called the head chunk. New log entries for that stream are appended to it until Loki decides the chunk should be closed.
A chunk can be closed when:
- It reaches a configured size or capacity.
- It has not been updated for long enough.
- A flush is triggered.
Once a chunk is closed, Loki compresses it and creates a new writable chunk for future entries in the same stream.
This buffering is central to Loki’s cost model. Instead of writing every log line individually to long-term storage, ingesters accumulate entries into chunk objects that can be compressed and flushed more efficiently.
Flushing chunks and index data
Eventually, compressed chunks are flushed to object storage. At the same time, Loki needs index data that lets future queries find those chunks by tenant, labels, and time range.
In modern single-store deployments, Loki stores both chunks and index files in object storage. The index is still not a full-text index of log contents. It is closer to a table of contents:
labels + time range -> chunk references
That connects this post back to the first one. The write path creates the storage layout that the read path later depends on. If labels are stable and chunks are healthy, queries can use the index to find a smaller set of chunks. If labels explode into too many streams, the write path creates operational pain that also shows up during reads.
What can go wrong
The write path has several useful failure modes to remember.
Too many labels or too many label values.
This creates too many streams. Loki then has to manage more stream state in ingesters, more chunk objects, and more index entries.
Too many tiny streams.
If each stream receives only a few log lines, chunks may be small and inefficient. The system spends more work managing metadata and flushing objects instead of storing dense compressed chunks.
Bad timestamps.
Distributors and ingesters validate timestamps. Depending on configuration, old, too-new, or out-of-order entries can be rejected or handled differently than expected.
Ingester crashes before flushing.
The WAL is designed for this case. On restart, the ingester replays the WAL before becoming ready for new writes.
Lost local disk.
The WAL cannot recover data if the disk it depends on is gone or corrupted. This is why stateful ingesters need persistent storage and why replication is still part of the design.
Object storage problems.
If chunks or index files cannot be flushed, ingesters can accumulate more local and in-memory pressure. Loki’s write path depends on object storage eventually becoming available as the long-term source of truth.
The mental model
The best simplified model I have now is:
distributor:
validate, normalize, limit, route
ring:
decide which ingesters own the stream
ingester:
WAL, memory, chunks, flush
object storage:
compressed chunks and index files
The distributor is about accepting and routing. The ring is about ownership. The ingester is about buffering and durability. Object storage is where the long-term data lives.
Final thought
The life of a log in Loki is a chain of tradeoffs. The distributor keeps the front door stateless. The ring lets many distributors make consistent routing decisions. Replication keeps writes available through common failures. The WAL protects acknowledged data while it is still in ingester memory. Chunks make long-term storage efficient.
That path explains why Loki cares so much about streams. A log line is not just stored somewhere. It joins a stream, gets routed through the ring, becomes part of an ingester’s in-memory state, and eventually lands in object storage as compressed chunk data plus index references.
The next part I want to study is the ring itself: how consistent hashing lets Loki distribute stream ownership without a central leader.