How AfterTest and AfterDiff Were Tested With Secrets in Every Request

A recorder sits on real traffic and writes it to disk. So it makes two promises, and the tests go after both. It must not change a single byte on the way through. And no secret may end up in the file.

Not a byte changed

One test sends 300 random requests straight to a test server and then through record, and compares what each side saw. The traffic is deliberately awkward: binary bodies, gzip, chunked bodies, bodies of unknown length, %2F and ü in paths, ; in query strings, HEAD, 204 and 304. Every request also carries an Authorization header. The upstream must receive the same method, path, headers and body both times, and the client must get the same answer.

Then every one of those 300 exchanges has to be accounted for, as recorded or skipped. The recording must parse as a fixture, and every case in it must pass when run back through a proxy that changes nothing. Further tests run a recording through patch test and adapt test with rules that match nothing, so the cases record writes are the same kind the other tools read. Another checks that answers stream rather than wait for the whole body.

No secret in the file

The redaction test sends fifteen different secrets in one exchange, each a unique string. They ride in Authorization, Proxy-Authorization, Cookie and two Set-Cookie headers, X-Api-Key, custom headers, query values, nested JSON fields and every email in a JSON list. The test checks that the service and the client both got the real values. Then it searches the fixture file and the audit log for each secret. None may be there, and everything around them, such as the other query keys and the other JSON fields, must still be there.

What the review found

Before release, an independent review went looking for ways to break record and replay. It found one serious problem: a WebSocket or any other connection upgrade failed with a 502 through record, which broke the “changes nothing” promise. Upgrades now pass through and are logged as not recorded.

The rest were smaller, and several were about secrets:

Query redaction only split on &, so ?a=1;token=x kept its token.
JSON sent without a JSON Content-Type wasn’t checked for listed fields.
A password or query string in the upstream URL showed up in the file’s header comment.

Since then, record checks every body that parses as JSON. It splits queries on ; as well as &, and redacts common credential query keys such as api_key, access_token and token by default. Other findings:

A request with no path could make the whole file unreadable.
Replay compared very long numbers too loosely.
A replay that skipped every case still passed.
A busy port left an empty fixture behind.

Each finding was fixed and has its own test. The recorded run has 272 tests, all passing with Go’s race detector on, and CI runs them on Linux and macOS.

The loop on the example

The end-to-end test records the customer-api client’s twelve checks against v1, then replays them. Against v2 through adapt, all twelve answers must be the same. Against v2 directly, all twelve must differ. Sending reads only, the eight reads must differ and the four writes must be skipped.

What it costs

Over loopback on a 2-CPU Linux machine, with 20,000 requests, a customer lookup straight to the service took 25 µs at the median. Through record, with every exchange written to a fixture file on disk, it took 106 µs. So record adds about 81 µs at the median. As before, these are small answers on a small machine, so read them as an order of magnitude. The raw numbers are in test/results in the code repository.