case studies

Testing a Datasette Upgrade on Live Read Traffic With Shadow Mode

Datasette is an open source tool that publishes a SQLite database as a website and a JSON API. Its 1.0 line has been in alpha for a while, and the changelog says plainly that the JSON API is getting a new design before 1.0 ships. If you have scripts or dashboards reading a Datasette instance, you’d want to know what they will see before you upgrade it.

This case puts aftermarket shadow in front of the current stable release, 0.65.5, with the latest alpha, 1.0a41, as the candidate. Both serve the same small database of twelve national parks. Every command and every line of output below comes from a real run of the downloaded aftermarket 0.10.1 on Linux. The script, the traffic and the full reports are in the code repository under test/results/case-studies/shadow/datasette, so the run can be repeated.

The setup

Two Datasette installs side by side, each in its own Python virtual environment, on ports 9101 and 9102. Shadow listens on 8190 and the client talks only to it:

aftermarket shadow --primary http://127.0.0.1:9101 --candidate http://127.0.0.1:9102 \
  --listen 127.0.0.1:8190 --report report.md --json report.json

The traffic is 22 requests a reader of the API would make: table pages in JSON with filters, sorting, facets and paging, single rows, a missing row, two arbitrary SQL queries, _shape=array and _shape=objects, the CSV export, two HTML pages and the instance’s version and metadata endpoints.

Find the noise first

Before comparing two versions, compare a version with itself. A second copy of 0.65.5 went on port 9103 as the candidate, with the same traffic. 13 of 22 cases differed, with nothing changed at all:

### 1 GET /parks/parks.json

`GET /parks/parks.json`

- query_ms: 12.344115000814782 vs 16.873082999154576
- suggested_facets.0.toggle_url: "http://127.0.0.1:9101/parks/parks.json?_facet=state" vs "http://127.0.0.1:9103/parks/parks.json?_facet=state"
- suggested_facets.1.toggle_url: "http://127.0.0.1:9101/parks/parks.json?_facet=established" vs "http://127.0.0.1:9103/parks/parks.json?_facet=established"

Two kinds of noise. query_ms is how long the query took, so it’s different on every answer. The other is absolute URLs: Datasette builds next_url and the facet links from the host it was called on, and each side was called on its own port. The two HTML pages differed too, and HTML is compared byte for byte, so there’s no field to leave out there.

That gives four ignore rules and a config that copies only the JSON API:

version: 1
when:
  method: GET
  path: "/**.json"

--ignore query_ms --ignore next_url --ignore 'suggested_facets.*.toggle_url' \
--ignore 'facet_results.*.results.*.toggle_url'

With those, the same self-comparison read 18 case(s): 18 same. Now any difference left is a real one.

What the candidate answers differently

With 1.0a41 as the candidate and nothing ignored, 17 of 22 differed. With the config, the ignore rules and --header Location added so redirects show where they go, the report read:

18 case(s): 3 same, 15 different, 0 skipped, 0 error(s)

Four requests were outside when: (the CSV exports and the HTML pages) and were served without a copy. The three that matched were the two _shape=array table reads and the plugins list. The CSV exports matched too in the run without the config.

The biggest change is the default JSON shape. A table read in 0.65.5 returns rows as lists next to a columns list, plus a lot of context. 1.0a41 returns rows as objects with ok and truncated, and the context moved behind ?_extra=. Single rows changed the same way:

### 10 GET /parks/parks/3.json

`GET /parks/parks/3.json`

- columns: only in A (a list)
- ok: only in B (true)
- rows.0: a list vs an object
- truncated: only in B (false)
- units: only in A (an object)

_shape=objects already gave objects in 0.65.5, so the rows match there, but the 15 context keys it carried aren’t in the 1.0 answer. A reader that only takes rows is fine. One that reads filtered_table_rows_count or columns isn’t.

Arbitrary SQL moved. Both SQL queries got a redirect instead of data:

### 14 GET /parks.json

`GET /parks.json?sql=select+name+from+parks+where+area_km2+>+5000&_shape=array`

- status: 200 vs 302
- header Content-Type: "application/json; charset=utf-8" vs "text/plain"
- header Location: "" vs "/parks/-/query.json?sql=select+name+from+parks+where+area_km2+>+5000&_shape=array"
- body: 69 bytes vs 0 bytes, first difference at byte 0

That matches the 1.0a14 changelog entry, which moved SQL queries to /database/-/query and redirects the old address. Following the redirect by hand gave the same three parks as 0.65.5 for this _shape=array query. So a client that follows redirects keeps working for array-shaped queries, and one that doesn’t gets an empty 302.

The smaller ones are the kind nobody remembers to check. A missing row still answers 404, with a new message:

- error: "Record not found" vs "Row not found"

and /-/metadata.json answers 404 in 1.0a41, since metadata moved into Datasette’s internal database in the same alpha. The version endpoint differs as you’d expect, and the database page gained metadata, table_columns and query counts.

What a team would take from it

None of this is hidden; the Datasette changelog and upgrade guide describe the new JSON design and the query move. What shadow adds is the list for your own traffic. In this run the change came down to a short to-do list: switch readers of table JSON to _shape=array or to the new row objects, make sure the HTTP client follows redirects or call /-/query.json directly, and drop any use of /-/metadata.json. The noise run took a minute and saved reading thirteen differences that meant nothing.

The shadow mode post explains how the copies stay out of the client’s way, and the AfterTest and AfterDiff reference has every option. The JSON Server case study covers writes, and the load case study runs this same pair under load.