Shadow Mode Recipes for Testing a New Version on Live Traffic

Short, practical setups for aftermarket shadow, each one run before it went here. The commands come from runs of the downloaded 0.10.1 against Datasette, JSON Server and the customer-api example, recorded under test/results/case-studies/shadow in the code repository. In all of them the primary is the version clients use today and the candidate is the new one.

Shadow a rewrite

Start with reads only. That’s the default, and it can’t change anything on the candidate:

aftermarket shadow --primary http://127.0.0.1:9201 --candidate http://127.0.0.1:9202 \
  --listen 127.0.0.1:8290 --report report.md --json report.json

Point the client, or a copy of production traffic, at port 8290. The report counts the writes it didn’t send (6 writes without --include-writes in the JSON Server run), so you can see how much of the traffic you haven’t tested yet. The JSON Server case study goes through what a rewrite looks like in the report.

Find the noise before you compare

Run the current version against a second copy of itself first. Whatever differs there differs on its own: timings, generated ids, timestamps, absolute URLs built from the host. Datasette 0.65.5 against itself differed in 13 of 22 cases this way. JSON Server 0.17.4 against itself differed in none. It’s quick, and it saves reading differences that mean nothing.

Ignore timestamps and timings

--ignore takes a dot path, with * for any key or list element. For Datasette:

aftermarket shadow --primary http://127.0.0.1:9101 --candidate http://127.0.0.1:9103 \
  --listen 127.0.0.1:8390 --config shadow.yaml \
  --ignore query_ms --ignore next_url --ignore 'suggested_facets.*.toggle_url' \
  --ignore 'facet_results.*.results.*.toggle_url'

With these four rules, the self-comparison that showed 13 differences read 18 case(s): 18 same. For a typical API it’s --ignore updated_at or --ignore 'data.*.updated_at' for a list. Quote paths with * so the shell leaves them alone.

HTML and other bodies that aren’t JSON are compared byte for byte, so a page with a render time in it will always differ. Leave those out with when: in the config:

version: 1
when:
  method: GET
  path: "/**.json"

Requests outside when: are still served from the primary and counted in the report.

Ignore generated ids, carefully

When the server picks ids, a create gives different ids on each side. --ignore id --ignore '*.id' covers an id at the top of an answer and in every item of a list. On the JSON Server pair with writes, that cleared the id differences from the POST and from every read of the list. What it can’t clear is the follow-up: the client reads, edits and deletes the new record by the id the primary gave it, and the candidate never had that id. Those showed up as status: 200 vs 404 three times. It also hid the change from numbers to strings, which was the most important finding of that run, so read the report once without the rule before adding it.

Sample a busy service

aftermarket shadow --primary http://old:9000 --candidate http://new:9000 --sample 0.1

One request in ten is copied, counted exactly. Under steady load on a two-core machine, the defaults dropped 1,570 of 6,000 copies because the candidate couldn’t keep up; with --sample 0.1, 600 were copied and none were dropped. Clients never wait for copies either way. The load case study has the numbers. If you’d rather copy everything, --queue and --workers give the candidate more room, but a queue only absorbs bursts.

Copy writes to a separate database

aftermarket shadow --primary http://127.0.0.1:9201 --candidate http://127.0.0.1:9202 \
  --listen 127.0.0.1:8290 --include-writes --workers 1

Give the candidate its own database, or its own data file, loaded from the same starting point as the primary’s. A candidate sharing the primary’s database would make every write twice. --workers 1 sends copies one at a time, so writes reach the candidate in the order the primary got them. Expect records created by the server to drift apart, as above, and judge writes by status and shape.

Keep credentials out of the report

shadow reads record’s config, and its redact: lists apply to the report and the log. The customer-api example’s record.yaml has:

redact:
  headers: [X-Request-Token]
  query: [api_key]
  json: [email, customer.email, data.*.email]

With v1 as the primary, v2 as the candidate, and a client sending Authorization: Bearer ... and ?api_key=...:

aftermarket shadow --primary http://127.0.0.1:9301 --candidate http://127.0.0.1:9302 \
  --listen 127.0.0.1:8390 --config examples/customer-api/record.yaml

`GET /api/v1/customer?id=7&api_key=REDACTED`

- status: 200 vs 404
- customer_id: only in A (7)
- email: only in A (REDACTED)
- error: only in B (an object)
- name: only in A ("Ada Lovelace")

Neither the token, the key nor either email address appeared anywhere in the Markdown report, the JSON report or the audit log. Credential headers such as Authorization and Cookie, and the usual credential query keys, are redacted without being listed. The candidate still gets the real values, since it has to answer the same request.

Read the report

The top of the report says what was compared:

- Headers compared: Content-Type, Location
- Ignored: query_ms, next_url, suggested_facets.*.toggle_url, facet_results.*.results.*.toggle_url

18 case(s): 3 same, 15 different, 0 skipped, 0 error(s)

Then the traffic block, which is worth reading before any difference:

- 22 request(s) served from A; 18 copied to B
- Not copied: 4 outside when:, 0 left out by sampling (1), 0 writes without --include-writes, 0 dropped because the queue (64) was full, 0 that couldn't be compared safely

It tells you how much of the traffic the comparison covers. Then each differing case, A being the primary and B the candidate. only in A means the candidate dropped a field, only in B means it added one, and a list vs an object means the shape changed. --header Location is worth adding whenever redirects might be involved, since a 302 with an empty body otherwise says little. The file is rewritten as comparisons come in, at most once a second, so you can keep it open while traffic runs; Ctrl-C writes the final version. The JSON report has the same content for scripts.

The AfterTest and AfterDiff reference has every option, and the shadow mode post explains how copies stay out of the clients’ way.