When one of Ghostget's generated tests fails, the report includes the seed that started the random generator and the shrink path to the smallest failing input. Passing those two values back reruns that exact case, on any machine, as often as needed. For two of its data formats, the tests also compare Ghostget's output with a separate Rust program written to the same rules.
This matters because of what Ghostget does. It sits between an AI agent and a person's own accounts, and after that person confirms a preview, it sends messages and posts there for them. A bug that shows up once in CI and then passes on the rerun is one Ghostget cannot afford to shrug off as flaky.
A failure you cannot repeat is a failure you keep
A lot of software now gets written quickly by a model, looks finished, and breaks on the second use. Call it vibe-coded slop. The tests that come with it usually check the three cases the author thought of, and the bug lives in a fourth: a name with an unusual character, an empty reply, a network drop halfway through a save.
Under that sits a fragile foundation: when a product is built on pieces nobody checked, each new feature inherits their bugs, and the failures that do show up are hard to pin down. "It only breaks if you save, lose connection, then reopen" is the kind of report that is slow to reproduce by hand. If the test that caught it cannot repeat it, the fix is a guess.
When every failure arrives with its own recipe
With property tests, you write the rule the code must follow, and the computer invents hundreds of inputs to try against it. When one breaks the rule, the computer shrinks it to the smallest input that still breaks it and writes down exactly how it produced that input.
That note turns a failure into a recipe: anyone can run it again, on any machine, and watch the same thing go wrong. Once the bug is fixed, the recipe can be checked into the repository, where that test replays it before trying new inputs, so the same bug cannot return quietly. Far fewer failures end up as "it failed once and we never saw it again".
Ghostget adds one more step for its data formats. For two of them, a second program written separately in another language follows the same written rules, and the tests compare the two outputs on generated inputs. When they disagree, one of them has misread the standard.
Rules first, then generated inputs
Ghostget's generated tests use fast-check, a TypeScript library for property tests. A property is a rule stated over all inputs of some kind. Here is the shape of one, written for this post:
// Rule: reading a service's JSON result and writing it back changes nothing.
assertProperty(
fc.property(fc.jsonValue(), (output) => {
const parsed = parseResult({ status: "succeeded", output });
expect(parsed.output).toEqual(output);
}),
);
fc.jsonValue() produces arbitrary JSON: nested objects, arrays, odd strings, large and tiny numbers. By default each property tries 200 generated cases, and a property that runs longer than 10 seconds counts as a failure. A nightly job multiplies every property's run count, and its time limit, by 20 unless someone asks for a different factor. That job blocks no merge or release; a failure there is triaged.
Every property in the repository goes through two shared helpers, assertProperty and assertAsyncProperty. A policy test rejects any direct call to fast-check's own runner, so the replay rules below apply to all of them.
Each failure prints its seed and path
When a property fails, fast-check reports two things: the seed that started its random generator, and the shrink path that leads from the first failing input to the smallest one. Together they are a replay coordinate. A contributor reruns exactly that case, and only that property, like this:
GHOSTGET_PROPERTY_SEED=<seed from the report> \
GHOSTGET_PROPERTY_PATH=<path from the report> \
bun test <test file> --test-name-pattern '^<property name>$'
The helpers are strict about this input. A path without its seed is refused, because a path means nothing without the seed that produced it. The seed must be a whole number that fits in 32 bits, and a path must be short, well formed, and limited in how much replay work it asks for. A mistyped coordinate fails loudly instead of replaying some other case.
One failure that became a permanent test
A required CI run failed in the property that checks how Ghostget reads back the result of a read action. The fix itself landed separately, and on 2026-09-23 the repository added a named test that carries the failure's coordinate. Shrunk, the failing output was this:
{"__proto__":0}
In JavaScript, __proto__ is a special name. JSON.parse keeps it as an ordinary key, but code that copies keys one at a time by assignment can end up changing the copied object's prototype instead, and the key disappears. The reader was fixed, and the fix is now held by three guards:
- A named example test round-trips
{"__proto__":0}and a nested variant, and checks that the key is still there. - On 2026-09-24 the seed and path went into a small checked-in seed corpus, so the property replays that coordinate before its ordinary random run, every time.
- A second test asks fast-check to generate the value at that coordinate and checks that it is still
{"__proto__":0}. If a fast-check upgrade moves what a coordinate produces, that test fails and says the corpus entry needs to be found again.
On 2026-09-24 the corpus held this one entry. Contributors add to it when a failure is found, or when a seed is worth replaying for another reason.
Sequences of actions, with crashes in between
Some rules concern the order of actions. For those, Ghostget uses fast-check's command sequences, where the generator invents a list of actions and a simple model says what should be true after each one. The crash tests are the clearest case. The strictest of their rules is about sends: after any crash, a confirmed request reaches the service at most once, and a request that did cross is never forgotten and sent again. A simpler rule, about private files, reads like this in a sketch written for this post:
// Rule: after any sequence of writes, each possibly interrupted by a crash,
// the file holds the old value or the new value, never a torn mix.
fc.commands([writeThatMayCrash, readBack]);
The harness kills the process at a generated point, at one of the steps that make a change durable (creating, writing, renaming, or removing a file or directory, or flushing it to disk). It injects four kinds of fault there: a crash just before the step, one just after, a torn data write, and a simulated power loss that rolls back every change no flush had made durable. The crash position is part of the generated input, so the same seed and path select the same failing sequence and crash point.
Checking the output against a second program
Two of Ghostget's outputs follow written rules, so the tests compare them with a separate program: a small Rust tool in the repository, used only for testing and never shipped.
Canonical JSON. Ghostget writes some JSON in one fixed form so that the same value always hashes to the same digest. The rules for that form come from RFC 8785. The Rust tool implements RFC 8785 separately, with its own strict parser, and the tests feed both sides the same generated values and require identical text. Key order is where a second reading pays off: RFC 8785 sorts object keys by UTF-16 code units, so {"B":1,"a":2} is already in order, because capital letters come before lowercase ones. Sorting by a language's usual alphabetical order would put a first and produce a different digest.
Public URLs. Before Ghostget's web gateway requests a URL for an agent, it decides whether the URL is one it will accept. The Rust tool parses each candidate with the Rust url crate, applies the gateway's written policy. The tests require both sides to make the same decision. For every URL either side accepts, the tests also require the same parsed parts, apart from the named parser differences below.
The repository also tests the comparison tests. It plants known mistakes, such as sorting keys by locale order or by UTF-8 bytes, or dropping the plus sign in a number's exponent, and checks that the comparison catches each one. Where the two URL parsers are known to differ, for example in how they encode ^ in a path, a test names the difference and checks that Ghostget refuses those inputs.
What these tests leave unproved
Ghostget keeps a public register of its claims, and on 2026-09-24 it listed 16 property claims (15 with checks that run, 1 planned), 15 command-sequence model claims (2 with checks that run, 13 planned), and 4 comparison claims (3 with checks that run, 1 planned). One of the three that run compares fixed test vectors from a separate Python generator rather than a second live program. The planned comparison checks the address filter that runs after a web host's name is resolved; no independent program covers it yet.
Generated inputs are samples. The default of 200 cases per property, multiplied by 20 at night, is not a proof over every input. The crash tests run one or two generated schedules per property in each CI run. The RFC 8785 program was written from the standard without reference to the TypeScript, but by the same author, so a misreading that both share would pass. One known gap is recorded: for a lone surrogate, Ghostget's canonical JSON writes an escape where RFC 8785 refuses the input, and a test pins that difference. The URL oracle also restates Ghostget's written policy, so a rule missing from both the policy text and the code goes unnoticed. None of these tests exercises a live service, browser, or operating system.
Latest release: v0.18.38. The register and the tests described here live in the Ghostget repository on GitHub, under the MIT license.