keyboard-shortcut
d

CAP theorem

5min read

an image

The CAP theorem, proposed by Eric Brewer, describes the trade-offs in distributed systems between three key properties:

  • Consistency (C): Every read receives the most recent write data or an error.
  • Availability (A): Every request receives a (non-error) response, even if some nodes are down.
  • Partition Tolerance (P): The system continues to operate despite network partitions or communication failures.

According to the theorem, a distributed system can only guarantee two of these properties at the same time:

Combination Description Examples
CA Consistent and Available (but not Partition Tolerant) Traditional relational databases
CP Consistent and Partition Tolerant (but may sacrifice Availability) MongoDB (in some modes)
AP Available and Partition Tolerant (but may sacrifice Consistency) Cassandra

No distributed system is safe from network failures, thus network partitioning generally has to be tolerated. In the presence of a partition, you're forced to choose between consistency or availability. When choosing consistency over availability, the system will return an error or a time out if particular information cannot be guaranteed to be up to date due to network partitioning. When choosing availability over consistency, the system will always process the query and try to return the most recent available version of the information, even if it cannot guarantee it is up to date due to network partitioning.

In the absence of a network partition, both availability and consistency can be satisfied.

Most modern architectures choose between CP and AP, depending on application needs. Database systems designed with traditional ACID guarantees in mind such as RDBMS choose consistency

Cassandra

Cassandra trades strict consistency for high availability and partition tolerance, which introduces several possible consistency issues. Here are the key ones:

⚖️ 1. Eventual Consistency

Cassandra uses eventual consistency, meaning that after a write, different replicas may temporarily hold different versions of the same data.

If you read from a replica that hasn’t yet received the latest update, you’ll get stale data.

The system will eventually converge, but reads immediately after writes can be inconsistent.

🔁 2. Read/Write Conflicts

When multiple nodes accept writes for the same key during a network partition, Cassandra must later reconcile conflicting versions.

This is usually handled by timestamps - the latest write wins - but that can lead to lost updates if clocks are skewed or concurrent writes happen.

🧱 3. Inconsistent Reads

Because reads can be served by any replica, you might get different results from consecutive reads of the same key until replicas synchronize.

You can control this using consistency levels (the number of nodes that need to agree e.g., QUORUM, ONE, ALL), but higher consistency levels increase latency and reduce availability.

🕰️ 4. Write Visibility Delays

Even if a write returns success, it might not yet be visible to all nodes.

A client reading immediately after writing may not see its own update — a phenomenon known as read-after-write inconsistency. This happens when you write to one node, but query another node that the data hasn't propogated to yet.

⚔️ 5. Tombstone and Repair Issues

Cassandra uses tombstones to mark deletions, which are reconciled later during compaction or repair.

Until that process completes, deleted data may reappear if an old replica still holds it.