AfterDiff Gets a Live Shadow Mode to Test New Versions on Real Traffic

Aftermarket 0.10.0 adds aftermarket shadow. It’s AfterDiff, but on live traffic instead of a recording.

Replay is good at one thing. You record what a client sends, play it to the old and the new version, and read the differences. But a recording only holds what happened while it ran. The odd request that comes once a week isn’t in the file, and neither is the client nobody remembered.

Shadow puts the new version next to the old one while real users keep going. Every request is served by the current service, the primary. The client gets its answer, byte for byte. Then a copy of the same request goes to the new version, the candidate, and the two answers are compared the same way replay compares them.

One command

aftermarket shadow --primary http://127.0.0.1:9001 --candidate http://127.0.0.1:9002 \
  --listen 127.0.0.1:8090 --report shadow-report.md

Point clients at shadow. The report is rewritten as differences come in, in replay’s format, so anything that already reads replay reports reads this one. A difference looks like this:

### 1 GET /api/v1/customer

`GET /api/v1/customer?id=7`

- status: 200 vs 404
- customer_id: only in A (7)
- error: only in B (an object)

On top of replay’s report there are counts: requests served, copies sent, copies dropped, and anything that couldn’t be compared, with the reason. --sample 0.1 copies one request in ten. --ignore leaves out fields that differ every time, such as timestamps. The record config works here too: when: picks the traffic and redact: keeps values out of the report.

The candidate stays out of the way

This was the part that mattered most. A new version is the one most likely to be slow or broken, and none of that can be allowed to reach a user.

So the copy is made only after the primary’s answer has gone out. It goes into a queue of fixed size, and handing it over never waits. If the queue is full, the copy is dropped and counted. A few workers send the copies, each with its own HTTP client and a five second timeout. A candidate that hangs only ties up a worker.

Writes are not copied unless you ask with --include-writes, the same rule as replay. If you do, give the candidate its own database, or every write happens twice.

How it was tested

The tests aim at four promises.

Answers don’t change. One test sends 300 random requests straight to a server and then through shadow, and compares every byte both ways: binary bodies, gzip, chunked bodies, odd paths, HEAD, 204 and 304. The candidate in that test takes turns answering differently, failing, answering late, dropping the connection and never answering.

A bad candidate adds no wait. Another test times 150 requests in a row against six candidates: one that answers the same, one that’s down, one that takes a second per answer, one that never answers, one that fails with 500 and one that drops the connection. Every answer has to be right, and no request may take more than 400 ms, while the slow candidate takes a full second.

Differences match replay. The same traffic goes through shadow, and through record followed by replay. The two reports have to list the same cases with the same differences, line for line. The customer-api example does it too. With the old client’s suite running live against v1, a candidate of v2 behind AfterAPI gives 12 of 12 the same, and v2 on its own gives 12 of 12 different, listed exactly as replay lists them. The client passes all twelve checks both times, since it only ever sees v1.

Credentials stay out. Requests carry an Authorization header, cookies and keys like api_key and token, and both services answer with cookies, session headers and secret JSON fields that differ. The candidate still gets the real credentials. The report says header Set-Cookie: differs (REDACTED) and password: differs (REDACTED), and none of the values appear in the report or the log.

The recorded results have 554 tests, all passing. They also time the primary path over 20,000 requests per candidate. Against a direct call, shadow added 214 µs at the median with a candidate that answers the same, 217 µs with the candidate down, 208 µs with one that fails every time, and 149 µs with one that takes a second per answer, where 23,854 of 23,966 copies were dropped because the queue stayed full. Record alone adds 197 µs on the same machine, so most of that is the proxy itself. The p99 figures ran from 3.7 to 7 ms on a shared two-core machine that was busy with other work, and they’re in the results file as measured.

What it doesn’t do

The candidate sees each request a moment after the primary did. With several workers, copies can arrive out of order, so use --workers 1 when the order of writes matters. Values in a URL path, and JSON fields the config doesn’t list, can show in a difference.

The AfterTest and AfterDiff reference has every option, the quick start has a shadow step in part three, and the binaries are on the download page.

Three case studies have since run it against real open source services: a Datasette upgrade, the JSON Server rewrite with writes and the Datasette pair under load. The shadow mode recipes collect the setups that came out of them, and 0.10.1 fixed five bugs a review found.