Aftermarket Fixes Proxy Answers Cut Off Under Load

Aftermarket 0.7.1 is a fix release, and the bug it fixes was hiding in plain sight. For a couple of days CI failed now and then in the random pass-through tests, with an “unexpected EOF” while reading an answer. The same tests passed every time on the build machine. It got written off as a flaky test.

It wasn’t. Every failure was on the call through the proxy, never on the direct one. Under enough CPU load the failure showed up locally too, about once in a hundred runs, and from there it could be traced.

What went wrong

Take a client that sends a request with a body, and an upstream that answers quickly. The proxy streams the body upstream from one goroutine and copies the answer back from another. Go’s HTTP server, as soon as the answer starts going out, finishes reading the client’s body and closes it. That protects clients that send everything before they read anything. But Go’s transport was still doing one last read of that body, just to see it end. That read failed, the transport took the request as failed, and it dropped the upstream connection in the middle of the answer.

So the client got a short answer. Rarely, mostly on busy machines, and only for requests with a body. patch, adapt and record could all hit it. AfterGuard could too, when the app answered without reading the whole body.

The fix

The proxies now tell Go’s server to leave the request body alone until the proxy is done with it. The server still finishes the body once the request is over. An independent review of the fix found one place where that would have broken something, a health check sent with a body, and it’s covered now.

New tests make the race happen every time instead of once in a hundred. A stand-in upstream answers after reading all of the body, half of it or none of it, then checks the body is still readable while the answer goes out. Without the fix they fail on every run. With it, the random pass-through tests ran 390 times under heavy load without a failure.

The review turned up one more small thing. After a client’s body stalled and got a 408, patch and adapt kept waiting on that client. They now let go of the connection straight away.

The suite now has 452 tests. Binaries are on the download page.