The first idea that changes how I think about Loki is also the easiest one to misunderstand: Loki does not build a full-text index of every log line.

That sounds limiting at first. Logs are text, so it feels natural to index the text. But Loki is built around a different tradeoff. It indexes labels, stores log lines in compressed chunks, and scans those chunks at query time after the labels have narrowed the search space.

The core rule is:

Use labels to find the right streams; scan chunks to find the matching lines.

This is the same reason Loki often feels closer to Prometheus than to a traditional search engine. The index is not a giant map from every word to every log line. It is a smaller structure that helps Loki answer a different question first: which streams could contain the logs I care about?

The logging problem at scale

Logs grow quickly because they are produced by almost everything:

• Every service instance
• Every request path
• Every background job
• Every dependency call
• Every retry, timeout, warning, and error

At small scale, indexing all of that text can feel convenient. You send logs to a search system, type a word, and expect matching lines to appear.

At larger scale, the convenience becomes expensive. A full-text log index has to process, tokenize, index, and retain a searchable representation of the log contents. That means more CPU during ingestion, more storage for the index, more background compaction, and more operational work to keep the cluster healthy.

The difficult part is that a lot of log data is rarely queried. Many logs are useful because they exist when something goes wrong, not because every word in them needs to be indexed forever.

Loki starts from that observation. Instead of treating logs as documents that must be fully indexed, it treats them as streams that can be selected by metadata.

Why Elasticsearch is not always the right fit

Elasticsearch is powerful when the main requirement is flexible search over document contents. If I need rich text search, relevance scoring, complex inverted indexes, and exploratory queries across arbitrary fields, that model makes sense.

But operational logs often have a different access pattern.

Usually I start with context:

• Which application?
• Which namespace?
• Which environment?
• Which cluster?
• Which container?
• What time range?

Only after that do I search for a specific word, error message, trace ID, or JSON field.

In other words, I rarely need to ask, “where did this word appear across every log ever stored?” More often I ask, “inside this service, during this time window, which lines mention this failure?”

Loki is optimized for that second question. It avoids the cost of indexing every log line by relying on labels to reduce the amount of data that must be scanned.

Labels first

The most important Loki concept is the label set. A label is a key-value pair that describes where a log came from or what stable context it belongs to.

Examples:

app="checkout"
namespace="production"
cluster="us-east-1"
container="api"

A Loki query begins with a label selector:

{app="checkout", namespace="production"}

That selector does not search the log text. It selects streams whose labels match the query. After Loki has found candidate streams and chunks for the requested time range, the rest of the LogQL pipeline can filter the actual log lines:

{app="checkout", namespace="production"} |= "payment failed"

The label selector is the narrowing step. The text filter is the scanning step.

That distinction is the center of Loki’s design.

Streams

A stream is the sequence of log entries that share the exact same label set.

These two entries belong to the same stream if their labels are identical:

{app="checkout", namespace="production", container="api"}

If one label value changes, Loki sees a different stream:

{app="checkout", namespace="production", container="worker"}

This is useful because logs from the same source usually arrive together and are queried together. Loki can group them into chunks and index the metadata for those chunks instead of indexing every individual line.

The stream model also explains why labels must be chosen carefully. Labels define how many streams Loki has to manage.

Chunks

Loki stores the actual log lines in chunks. A chunk contains log entries for a stream over a time range. It is compressed and eventually flushed to storage.

A simplified way to see it is:

stream labels
    -> chunk from 10:00 to 10:15
    -> chunk from 10:15 to 10:30
    -> chunk from 10:30 to 10:45

The index can tell Loki which chunks might matter for a query. The chunk contains the real log content.

This is different from a full-text search system. A full-text index tries to answer “which documents contain this term?” directly from the index. Loki’s index answers “which chunks belong to streams matching these labels and this time range?” Then Loki reads and filters those chunks.

The benefit is that the index can stay much smaller. The cost is that queries may need to scan chunk data, especially if the label selector is broad.

Object storage

Loki is designed to use object storage as the long-term place for chunks and, in modern single-store setups, index data too. The Grafana documentation describes Loki storage around a small index plus compressed chunks in object stores such as S3.

This matters because object storage changes the cost model. Instead of keeping a large search index on expensive local disks across many indexing nodes, Loki can store compressed log data in a cheaper backend and scale the read path when queries need to inspect it.

That is one reason the design is attractive for large volumes of logs. Loki does not make querying free, but it tries to make storing logs less expensive and simpler to operate.

Cardinality is the trap

The label model only works if labels stay low-cardinality.

Cardinality means the number of distinct values a label can have. A label like environment is usually low-cardinality:

environment="dev"
environment="staging"
environment="production"

A label like trace_id is high-cardinality:

trace_id="0242ac120002"
trace_id="ae91bc78f1d4"
trace_id="f0c41e9b77aa"

Every unique label set creates a separate stream. If a value changes for every request, then Loki has to manage a huge number of short-lived streams. That creates a bigger index, more tiny chunks, more flushing, and worse query behavior.

Good labels usually describe stable origin or routing context:

• app
• namespace
• cluster
• environment
• container

Bad labels usually describe individual events:

• request_id
• trace_id
• user_id
• order_id
• timestamp
• Raw URL paths with unbounded IDs

This is the part of Loki that feels simple but has deep operational consequences. Loki is not saying metadata does not matter. It is saying that indexed labels should be metadata with controlled cardinality.

Structured metadata

The uncomfortable question is: what should I do with fields that are useful but too high-cardinality for labels?

For example, trace IDs are extremely useful when connecting logs to traces. But using trace_id as a label would create many streams, often one per request.

Structured metadata is Loki’s answer for some of this middle ground. It lets metadata travel with the log entry without becoming part of the indexed label set. The data can still be used in queries, but it does not expand the index the same way a label would.

A useful mental model is:

• Labels decide where Loki should look first.
• Structured metadata gives extra context after Loki has found candidate log entries.
• Log line filters inspect the text itself.

I will explore more about it in a future post.

Why this design works

Loki’s architecture makes more sense when I separate storage cost from query cost.

Full-text indexing pays more during ingestion and storage so arbitrary text searches can be faster later. Loki pays less during ingestion and storage by keeping a smaller index, then accepts that some queries will scan compressed chunks.

That is a good tradeoff when:

• Logs are very large in volume.
• Most queries start with service, namespace, environment, or cluster.
• The team can control label cardinality.
• Object storage is the desired long-term backend.
• The system is used for observability workflows, not general document search.

It is a worse tradeoff when:

• Users expect arbitrary full-text search across everything.
• Label selection is too broad.
• High-cardinality values are accidentally promoted to labels.
• The system produces many short-lived streams.
• Queries frequently need needle-in-a-haystack searches over huge time ranges.

Loki is not magic. It is a set of constraints. When the constraints match the workload, the design is elegant. When they are ignored, the same design can become painful.

The mental model

The best simplified model I have found is this:

labels + time range
    -> find streams
    -> find chunk references
    -> read compressed chunks
    -> filter log lines
    -> return matches

The index is there to avoid reading everything. It is not there to answer every possible text question by itself.

This is why Loki’s label design matters so much. Labels are the map. Chunks are the archive. LogQL is the tool that narrows, scans, parses, and aggregates.

Final thought

Loki does not index log contents because it is choosing a different center of gravity. It assumes that logs are usually searched with context first and text second.

That choice gives Loki its shape: labels, streams, chunks, object storage, and query-time scanning. Once that tradeoff is clear, the rest of the architecture starts to feel less surprising.

The next question is what happens when a log actually arrives. That path goes through distributors, the ring, replication, ingesters, the WAL, chunks, and finally object storage.

References

• Grafana Loki documentation: Get started
• Grafana Loki documentation: Storage
• Grafana Loki documentation: Cardinality
• Grafana Loki documentation: Structured metadata

The core rule is:

Each node periodically tells a few randomly chosen peers what it knows, and those peers repeat the process.

No single message reaches the entire cluster. Instead, information spreads over several rounds, much like a rumour. Given enough exchanges, the nodes converge on a similar view of the world.

Gossip is commonly used for information that can be eventually consistent: cluster membership, health observations, service metadata, configuration hints, or other state where a brief delay between nodes is acceptable. It is not a consensus protocol. It does not make every node agree at the same instant, nor does it decide the order of critical operations.

Why a cluster needs gossip

Imagine a cluster with hundreds of nodes. A central registry can tell every member about a join, a departure, or a failure, but that registry becomes an important dependency. It must be reachable, it must handle all update traffic, and it must be highly available if the rest of the system depends on it.

Gossip distributes both the work and the knowledge. Each node only communicates with a few peers during a round, rather than broadcasting directly to every other member. Nodes also relay information they learned from somebody else, so a new update can move through the cluster even if its original sender does not contact everyone itself.

This makes gossip useful when the cluster should continue operating through the loss of an individual node or a central control-plane component. A node still needs an initial contact point—a seed peer—to enter the group, but that seed is an entry point rather than the permanent owner of membership.

How information spreads

A simplified gossip round looks like this:

1. A node observes or creates an update, such as “node C joined” or “the version of this metadata is now 8.”
2. It selects one or more peers, usually at random.
3. It sends the update, or a compact summary of the updates it knows.
4. Recipients merge any information that is newer than their local state.
5. Those recipients choose peers in later rounds and relay the update further.

The random selection is important. Repeated random exchanges spread information widely without requiring every node to maintain a full broadcast schedule. Many implementations use a push model, where a node sends updates; a pull model, where it asks peers for missing state; or a push-pull combination.

The membership view is local

A gossip-based membership system does not usually have one authoritative table that every process reads from. Each process keeps its own local view and updates it as messages arrive.

For a short time, node A may know about a join or a failure that node B has not heard about yet. That is not necessarily an error; it is the expected consequence of eventual consistency. The protocol is designed so that the difference disappears as updates are relayed.

To make merging deterministic, an update needs a version. A simple representation is:

type Version struct {
    Epoch     int64
    Heartbeat uint64
}

The exact format varies, but the rule is the same: a receiver accepts a record only if it is newer than what it already knows. A heartbeat can advance while a process is alive. An epoch, incarnation number, or generation lets a restarted process begin a fresh sequence without being mistaken for an old message.

For example, an implementation might order versions like this:

incoming epoch > local epoch
or
incoming epoch == local epoch and incoming heartbeat > local heartbeat

This protects the cluster from accepting stale state after messages are delayed, duplicated, or received out of order. A new epoch wins even when its heartbeat starts again at 1.

Failure detection is a suspicion, not a fact

Gossip is often paired with failure detection, but they are separate concerns. Gossip spreads the observation; failure detection decides when there is enough evidence to treat a peer as unavailable.

One simple approach is to track when a peer last sent newer state:

no newer heartbeat for longer than the timeout → suspect the peer

The word suspect matters. A node may be slow, temporarily unreachable, partitioned from the observer, or affected by packet loss. None of those conditions proves that its process has stopped. More complete membership protocols therefore use suspicion periods, indirect checks, and refutation mechanisms before spreading a final failure state.

Timeouts are a real tradeoff:

• A shorter timeout reacts quickly but increases false positives.
• A longer timeout is more conservative but leaves failed members visible for longer.
• A network partition can make two healthy groups each conclude that the other group is unavailable.

The important lesson is that failure detection describes an observer’s view of the network, not an absolute fact about another machine.

What gossip gives us

Gossip is a good fit when a system values availability, decentralisation, and scalable dissemination.

• No central dissemination bottleneck: each member helps distribute updates.
• Resilience to individual failures: the loss of one node does not prevent other nodes from sharing information.
• Scalable communication pattern: a node talks to only a small subset of the cluster in each round.
• Simple membership growth: a new member can join through a seed peer and learn about the rest of the cluster over time.
• Natural fit for soft state: information such as membership and health can tolerate brief disagreement while it converges.

These properties are why gossip appears in systems such as Apache Cassandra, which uses it to share cluster state, and in HashiCorp’s Memberlist, a library for cluster membership and failure detection.

The costs and limitations

Gossip avoids a central bottleneck by accepting different costs. It is not the right primitive for every distributed-systems problem.

• Eventual, not immediate, consistency: different nodes can briefly hold different views of the cluster.
• Network overhead: nodes keep sending background messages even when little state changes.
• Convergence time: an update needs several rounds to reach most or all members.
• False failure reports: timeouts and partitions can make healthy nodes look unavailable.
• Payload management: naively sending the full membership table becomes expensive as the cluster grows.
• No agreement on critical decisions: gossip cannot safely replace consensus when the system needs one ordered, durable answer—such as electing a leader, committing a transaction, or issuing a unique lock.

Good implementations control these costs with bounded fan-out, compact state, retransmission limits, failure-suspicion rules, and tuning for the network they operate in. The details differ by system, but the central balance is consistent: faster dissemination costs more traffic; more conservative detection reduces false positives but delays reaction.

A small Go proof of concept

To make the protocol concrete, I built a minimal Go implementation. It uses UDP, gossips a full table of (epoch, heartbeat) values, selects random active peers, and marks a peer as failed when its heartbeat stops advancing past a timeout.

It is intentionally a learning tool rather than a production design. In particular, sending the complete table makes the merge rule easy to see, but makes each message grow with membership size.

I then rebuilt the same membership experiment with HashiCorp Memberlist. The application provides node identity, bind address, seed peers, and join/leave callbacks, while Memberlist handles the specialised membership and failure-detection machinery. A clean shutdown calls Leave, so the rest of the cluster can learn about the departure without waiting for a timeout.

Final thought

Gossip does not create a perfectly synchronized cluster. It gives a distributed system a decentralised way to spread information until members converge, without making one node responsible for every update.

The central tradeoff is also its strength: gossip gives up immediate global agreement in exchange for resilience and scalable dissemination. The two PoCs helped me see the mechanics, but the protocol is useful because of that broader design choice.

References

• Minimal heartbeat gossip PoC
• HashiCorp Memberlist gossip PoC
• HashiCorp Memberlist
• Apache Cassandra: Gossip

The core rule is:

Before changing the main data structure, write the change to a durable log.

That rule is the foundation for crash recovery in many databases and storage engines. If a process dies after the log record is safely persisted but before the main data files are updated, the system can recover by replaying the log instead of guessing what happened.

Writing every changed data page directly to disk on every commit would be expensive. A WAL makes the commit path cheaper because the database can append records sequentially, flush that log, and apply the heavier data-file work later.

PostgreSQL describes this same central idea in its WAL documentation: data-file changes are written only after the corresponding WAL records have been flushed to permanent storage.

What WAL gives us

• Durability: once the log record is safely flushed, the change can be recovered.
• Crash recovery: after a restart, the database can replay log records that were not yet reflected in the main storage.
• Sequential write path: appending to a log is usually cheaper than immediately rewriting many random data pages.
• Replication and backups: many systems stream or archive WAL records to replicas or backup storage.
• Point-in-time recovery: if old WAL segments are archived, the database can restore a previous backup and replay logs until a chosen moment.

WAL does not make storage simple. It moves complexity into a controlled place: the log format, flush policy, recovery rules, checkpoints, and cleanup.

The write path

A simplified WAL write path looks like this:

1. A transaction changes data.
2. The database creates one or more WAL records describing the change.
3. The WAL records are appended to the current log file.
4. On commit, the required WAL bytes are flushed to durable storage.
5. The change can be applied to memory or data pages.
6. Dirty pages can be written to the main data files later.

The important ordering is: the log must reach durable storage before the database depends on the corresponding data change being durable.

Some systems flush on every commit. Others group many commits into one flush to improve throughput. Some expose weaker durability settings where a committed transaction may be lost after a power failure. That is a performance and durability tradeoff.

WAL record anatomy (simple representation)

A WAL is a sequence of records. The exact format depends on the database, but a record usually contains enough information to validate, order, and replay a change.

Common fields are: an LSN to place the record in the log timeline, a record type to say what happened, optional transaction metadata, a payload length, the payload itself, and often a checksum to detect corrupted or partial writes.

The diagram is simplified, but the principle is the same: recovery must be able to scan a byte stream and know exactly where each record starts, where it ends, whether it is valid, and how it should be replayed.

LSN: the timeline of the log

The LSN gives the log a stable order. It can be a record number, byte offset, or another monotonically increasing value.

LSNs help the database answer practical questions: which changes happened first, what has already been flushed, where recovery should start, which segments can be deleted, and how far a replica is behind. In many systems, data pages also store the latest LSN that affected them, so recovery can skip changes already present on disk.

Checksums and partial writes

Crashes do not always leave files in a clean state. The last WAL record might be complete, missing bytes, or partially written. Storage can also return corrupted data.

That is why production WAL formats usually include checksums or CRCs. During recovery, the database validates record length and checksum, then stops at the first invalid tail record instead of replaying garbage bytes.

Commit records and transaction boundaries

For a single operation, WAL recovery is easy: replay the operation.

Transactions make the problem more interesting. A transaction may generate many WAL records before it commits. After a crash, recovery must know whether those records belong to a committed transaction or an incomplete one.

Many systems solve this by writing explicit transaction records:

BEGIN tx42
UPDATE account:1
UPDATE account:2
COMMIT tx42

If recovery sees the updates but not the commit, it should not expose the transaction as committed. Depending on the engine design, recovery may redo committed work, undo incomplete work, or use a combination of redo and undo.

Checkpoints

If the database only appended to WAL forever, recovery would eventually become slow. Replaying months of logs after every restart is not practical.

A checkpoint records that the main data files are durable up to a known point in the log. After a checkpoint, recovery can start near that point instead of replaying from the beginning of time.

The simplified idea is:

1. Flush dirty data pages to disk.
2. Record a checkpoint LSN.
3. During recovery, start from the latest valid checkpoint.
4. Keep only WAL segments still needed for recovery, replicas, backups, or point-in-time restore.

The details differ by engine, but the motivation is similar: bound recovery time and control log growth.

Rotation, truncation, and archiving

WAL files are usually split into segments instead of one giant file. Segmenting makes log management easier.

A database may:

• Rotate to a new WAL segment after the current segment reaches a size limit.
• Recycle old segment files by reusing their space.
• Truncate log records that are no longer needed.
• Archive old segments for backups or point-in-time recovery.
• Retain segments needed by replicas that are still catching up.

This is one of the operationally important parts of WAL. If old logs cannot be removed because a replica is down or archiving is broken, disk usage can grow until the database is in trouble.

Recovery

On startup after an unclean shutdown, the database runs recovery.

A simplified recovery process looks like:

1. Find the latest valid checkpoint.
2. Open WAL from the checkpoint position.
3. Read records in LSN order.
4. Validate each record checksum and length.
5. Redo committed changes that are not reflected in data files.
6. Ignore, roll back, or clean up incomplete transactions, depending on the engine.
7. Stop at the first invalid or incomplete tail record.
8. Make the database available again.

The key idea is that WAL turns a crash into a deterministic replay problem.

WAL and performance

WAL improves the write path because sequential appends are efficient. It also allows the database to delay random data-page writes.

But WAL is not free:

• Every durable commit eventually needs a flush.
• Checksums, encoding, and compression cost CPU.
• Checkpoints can create I/O spikes.
• Long-running transactions or replicas can prevent log cleanup.
• Large WAL files can slow recovery.
• Misconfigured durability settings can trade correctness for speed.

Good WAL design is always a balance between latency, throughput, recovery time, and storage usage.

WAL in real systems

This idea appears in many storage systems, sometimes with slightly different names:

• PostgreSQL: uses WAL for crash recovery, replication, backups, and point-in-time recovery.
• SQLite: has a WAL mode where changes are appended to a separate WAL file before being checkpointed back into the database file.
• MySQL/InnoDB: uses a redo log for crash recovery. The redo log stores changes that can be replayed during initialization after an unexpected shutdown.
• RocksDB: uses WAL files to reconstruct memtables after a failure.
• Cassandra: uses a commit log before data is written into memtables and later flushed to SSTables.
• etcd: stores Raft entries in a WAL so the node can recover consensus state after restart.
• Prometheus: uses a WAL to protect the in-memory head block of its local time-series database. On restart, Prometheus replays the WAL to recover samples that were not yet compacted into blocks.
• Grafana Loki: uses a WAL in ingesters to persist acknowledged log data on local disk before it is flushed to long-term storage.

The names are not always identical, but the pattern is similar: append first, recover later.

A small Go proof of concept

To make the idea concrete, I built a small Go proof of concept: github.com/alex-carvalho/sandbox/devops/storage/wal.

It is intentionally small: a key-value map, a binary wal.log, and a recovery step that rebuilds memory from the log.

The demo writes a few operations:

write(wal.OpSet, "username1", "john")
write(wal.OpSet, "username2", "alex")
write(wal.OpDelete, "username1", "")

Each record contains:

• LSN: a monotonically increasing Log Sequence Number.
• OpType: the operation type, currently SET or DELETE.
• KeyLen: the number of bytes in the key.
• ValLen: the number of bytes in the value.
• Key: the key bytes.
• Value: the value bytes.

Final thought

The main lesson is that durability does not require immediately rewriting the whole database. A sequential log can act as the bridge between fast in-memory changes and safe on-disk recovery.

That is the elegance of WAL: the database can move quickly while still leaving itself a path back after a crash.

References

• My Go WAL PoC
• PostgreSQL: Write-Ahead Logging
• SQLite: Write-Ahead Logging
• MySQL InnoDB: Redo Log
• RocksDB: Write Ahead Log File Format
• Cassandra: Storage Engine
• etcd: WAL package
• Prometheus: Storage
• Grafana Loki: Write Ahead Log