How Shadow Mode Holds Up Under Load With Sampling and a Full Queue

Shadow mode makes one promise above the others: the candidate can never slow down or break what clients get. On a quiet service that’s easy. This case puts it under steady load, with a candidate that can’t keep up, to see what gives and what the report says about it.

The pair is the same as in the Datasette case study: Datasette 0.65.5 as the primary and 1.0a41 as the candidate, serving the same parks database. The traffic is four table reads with _shape=array, which answer the same on both versions, so the report is all about counts. Eight keep-alive clients send 6,000 requests per run. The primary, the candidate, shadow and the clients all share one two-core Linux machine, which makes the candidate the bottleneck on purpose: every copy it answers takes CPU from the primary. All of it ran on the downloaded aftermarket 0.10.1, and the script and reports are under test/results/case-studies/shadow/load in the code repository.

The baseline

Straight to the primary, no shadow:

requests: 6000 statuses: {200: 6000} threads: 8
elapsed: 27.5s throughput: 218 req/s
latency ms: p50 27.5 p90 68.6 p99 132.4 max 400.1

The defaults

aftermarket shadow --primary http://127.0.0.1:9101 --candidate http://127.0.0.1:9102 \
  --listen 127.0.0.1:8190 --report report.md --json report.json

A queue of 64 copies and four workers. The clients got 6,000 answers, all 200, at 198 requests a second with a median of 32.8 ms and a p99 of 129.3 ms. The report:

4430 case(s): 4430 same, 0 different, 0 skipped, 0 error(s)

- 6000 request(s) served from A; 4430 copied to B
- Not copied: 0 outside when:, 0 left out by sampling (1), 0 writes without --include-writes, 1570 dropped because the queue (64) was full, 0 that couldn't be compared safely

The candidate fell behind, the queue filled, and 1,570 copies were dropped and counted. No client waited for them. That’s the design: a dropped copy costs a little coverage, a blocked request would cost a user.

Sampling

On a busy service you don’t need every request. --sample 0.1 copies one in ten:

600 case(s): 600 same, 0 different, 0 skipped, 0 error(s)

- 6000 request(s) served from A; 600 copied to B
- Not copied: 0 outside when:, 5400 left out by sampling (0.1), 0 writes without --include-writes, 0 dropped because the queue (64) was full, 0 that couldn't be compared safely

Exactly 600, because the share is counted, not drawn at random, and nothing dropped. The candidate kept up easily with a tenth of the traffic. That’s the setting to start with on a real service: the comparisons you get are all complete, and the candidate stays idle most of the time.

A bigger queue

The other way out is more room and more workers, --queue 1024 --workers 8:

5484 case(s): 5484 same, 0 different, 0 skipped, 0 error(s)

- 6000 request(s) served from A; 5484 copied to B
- Not copied: 0 outside when:, 0 left out by sampling (1), 0 writes without --include-writes, 516 dropped because the queue (1024) was full, 0 that couldn't be compared safely

Fewer drops, 516, but a queue only buys time. When the candidate is slower than the traffic for long enough, any queue fills. It also costs memory: each waiting copy holds its request and the primary’s answer, up to max_body each.

About the timings

The client timings moved around between runs more than shadow could explain. The sampled run, which sent the fewest copies, was the slowest at 164 requests a second, while the run with every copy did 198. On a shared two-core machine with other jobs running, those numbers say more about the machine than about shadow, so they’re in the results files as measured and nothing here leans on them. The counts don’t move with the machine: every request was served, every copy was either compared or counted as dropped, and the totals add up to 6,000 in each run. The shadow mode post has the proper latency measurements, where shadow added about 0.2 ms at the median against a direct call.

What to take from it

Read the drop count before the differences. A report with 0 different and 1,570 dropped is a report on 4,430 requests, not 6,000. If drops are high, sample or give the candidate more capacity; don’t raise the queue and hope. The shadow mode recipes have the settings, and the AfterTest and AfterDiff reference has every option.