Step 5 of 8 - Storage
A replacement KV node has nothing to copy
The same node killed in both models.
Stateful node on EBS
Replacement must be re-populated before it can serve
| Data to re-populate | 2000 ranges x 512 MB | 1.0 TB |
|---|---|---|
| Copy rate | rebalance snapshots are rate-limited per store | 32 MiB/s |
| Time to serving | 1.0 TB at 32 MiB/s, and the docs give a band of minutes to hours | 8 h 53 m |
| Serving? | not until the copy completes | no |
Stateless KV node
A new node locks the same store and resumes
| Data moved | none: the store is in the storage layer already | 0 GB |
|---|---|---|
| Metadata | store lock handoff | 125 KB |
| Time to serving | 125 KB at 32 MiB/s | under a second |
| Serving? | as soon as the lock is held | yes |
How the handoff stays safe
A replacement locks the same store from Plenum and resumes. Fencing the old owner's write authorization is what makes the handoff safe.
How this is worked out
-
Both bars run in real time, like the scaling page. Bringing a node back is
the same rate-limited copy: 1.0 TB onto the
replacement at 32 MiB/s, CockroachDB's default
kv.snapshot_rebalance.max_rate, so it can't serve for 8 h 53 m. That's the optimistic end, and it lands inside the "minutes to hours" the design docs give. - The other lane writes 125 KB to hand the store's lock to the replacement, which at the same rate is under a second. Nothing is waiting on a copy.