Step 4 of 8 - Storage
Scaling without copying the data
One action, both lanes. The only difference is where the bytes already are.
Stateful nodes on EBS
7.5 TB to move
Rebalancing ranges: 0 GB of 7.5 TB copied in 0 s.
| Data to move | 15030 ranges x 512 MB | 7.5 TB |
|---|---|---|
| Copy rate | rebalance snapshots are rate-limited per store | 32 MiB/s |
| Time to complete | 7.5 TB at 32 MiB/s, and the docs give a band of minutes to hours | 2 d 19 h |
| Consequence | can't shrink quickly, so provision for peak | peak-provisioned |
Stateless KV nodes, disaggregated storage
939 KB of metadata
Creating hard links: the ranges are already in the storage layer.
| Hard links created | 15030 ranges shared, not copied | 15030 links |
|---|---|---|
| Metadata written | 15030 x 64 bytes | 939 KB |
| Time to complete | 939 KB at 32 MiB/s | under a second |
| Bulk data moved | none: the range's SSTables are shared, not copied | 0 GB |
| Still sent | the range's store-local state, which no hard link shares | small |
How this is worked out
- Both bars run in real time. A second on screen is a second of the actual operation: the hard links are written in under a second and the copy needs 2 d 19 h, so the left-hand bar is still going when you leave the page. There's nothing to wait for.
-
Moving a range is rate-limited per store, so the duration is arithmetic:
7.5 TB at 32 MiB/s, which is
CockroachDB's default
kv.snapshot_rebalance.max_rate. It's the optimistic end - a real rebalance shares the link with your queries - and it lands inside the "minutes to hours" the design docs give.