Cockroach Labs Estate economics What idle nodes, provisioned bytes and peak-sized bills cost, worked out from your estate.
  1. Nodes 3 baseline
  2. Avg CPU utilization 15.0% idle hardware
  3. Storage provisioned 60 GB 10 GB of data
  4. Est. monthly saving 0%

Step 5 of 8 - Storage

A replacement KV node has nothing to copy

The same node killed in both models.

Kill a KV node n1 n2 n3 n4 n5
KV nodesn1servingn2downn3servingn4servingn5servingLocal disks: the store dies with the nodeKV node topologyKV node 2 has failed. Its store died with it, so the replacement must copy 10 GB from the surviving replicas before it can serve.

Stateful node on EBS

Replacement must be re-populated before it can serve

0% after 0 s, 5 min still to goStateful node on EBS progressReplacement must be re-populated before it can serve. 0% after 0 s, 5 min still to go.

Re-populating the replacement: 0 GB of 10 GB copied in 0 s.

Data to re-populate20 ranges x 512 MB10 GB
Copy raterebalance snapshots are rate-limited per store32 MiB/s
Time to serving10 GB at 32 MiB/s, and the docs give a band of minutes to hours5 min
Serving?not until the copy completesno
KV nodesn1servingn2downn3servingn4servingn5servingShared storage layer: the store outlives the nodeKV node topologyKV node 2 has failed. Its store is in the storage layer, so the replacement locks the same store and resumes without copying data.

Stateless KV node

A new node locks the same store and resumes

0% after 0 s, under a second still to goStateless KV node progressA new node locks the same store and resumes. 0% after 0 s, under a second still to go.

Store re-locked. Serving.

Data movednone: the store is in the storage layer already0 GB
Metadatastore lock handoff1 KB
Time to serving1 KB at 32 MiB/sunder a second
Serving?as soon as the lock is heldyes

How the handoff stays safe

A replacement locks the same store from Plenum and resumes. Fencing the old owner's write authorization is what makes the handoff safe.

How this is worked out
  • Both bars run in real time, like the scaling page. Bringing a node back is the same rate-limited copy: 10 GB onto the replacement at 32 MiB/s, CockroachDB's default kv.snapshot_rebalance.max_rate, so it can't serve for 5 min. That's the optimistic end, and it lands inside the "minutes to hours" the design docs give.
  • The other lane writes 1 KB to hand the store's lock to the replacement, which at the same rate is under a second. Nothing is waiting on a copy.