writing code with ai has made me spend more time thinking about how to test it. this week, while building an internal feature flag system at Pandai AI, i've been building fuzz harnesses to try the combinations i wouldn't think to test by hand.
our apps read flags through the API defined by the OpenFeature specification, with our own storage and provider behind it.
write the rule, generate the cases
property based testing generates test cases and checks that your code follows the rules you set. by running generated inputs through your code, the harness can check thousands of cases you would otherwise have to write by hand.
imagine getting a new feature, only to lose it when you refresh and get it back on the next refresh. that gives us a useful rule for a gradual rollout: the same user gets the same version while the settings stay unchanged. the test can generate thousands of user IDs and evaluate each repeatedly, looking for anyone who switches versions.
for the flag system, i use Hypothesis in Python and Hegel in TypeScript to generate inputs and refresh sequences involving changing values, malformed data, and outages. i take the same approach into the browser with Bombadil, checking things like whether chat messages stay in order when you send a message or stop a response.
the bug that only happens when...
a duplicate-charge bug might need this exact sequence: payment succeeds, reply gets lost, client retries during a leader crash. testing retries and crashes separately won't reproduce it.
reading about Antithesis has made me more interested in testing these combinations, and the work on TigerBeetle and FoundationDB shows what that looks like when money and data depend on getting it right.
TigerBeetle is a database for financial transactions. its simulator deliberately crashes processes and breaks the network while the database runs, checking that the data stays correct through the failures. it then restores enough of the system to check that transactions can resume. keeping the numbers right through that mess is the engineering that draws me to it.
FoundationDB is a key piece of infrastructure at Apple and Snowflake. Snowflake uses it to store the metadata its data warehouse depends on. its team simulates different workloads and failures and can replay a failing run, turning an unlikely sequence of failures into something the developers can reproduce and debug.
give the ai something to check
being able to reproduce a failure matters when debugging already takes a sizeable part of the week. in a 2024 survey of 484 Microsoft developers, about 9% of the workweek went to debugging, compared with 11% to coding new features.
keeping a test for that failure can make the work useful beyond one fix. a February 2026 study of 642 bug-reproducing tests across 15 Python projects found no statistically significant difference from other tests in size, assertion count, or complexity. reproducing a bug does not necessarily need a huge test. that's what i want from a failing run: a case i can keep, so the same mistake gets caught next time.
on the Pandai AI feature flags, i'm spending less time going back and forth with my ai because a failing case gives it something concrete to fix and a way to check the change by running the harness again.
i've increased our throughput this way too. i give my ai measurable targets: navigation feedback within 70ms, a 70ms response-time goal in the test setup, and database connection pool stress tests with concurrent requests. those tests also exercise what happens when every connection is busy, a query times out, or the database goes away, so the optimization work includes how the system behaves under pressure.
while i make those changes, Hegel generates sequences of users, roles, and permission changes through the real API and a local test database. the checks verify that revoking access takes effect on the next request and that concurrent users cannot change each other's data. alongside the timing and load measurements, that helps me see whether i've made the app faster without breaking its rules.
keeping Realm responsive
i'm doing this in Realm too, my next app: a desktop workspace for RLM coding agents. i fuzz composer controls to catch memory growth after repeated interactions, and stream around 100k tokens of Markdown at high speed through the renderer with frame-time checks.
pushing that much text through the app is how i learned about the browser's main thread. inside Electron, the renderer's JavaScript, input handlers, and layout work share that thread. if parsing and rendering the stream hold it for too long, typing and window updates have to wait. the text can arrive quickly while the app feels slow. i'll go deeper into that in the next article on high-performance UI: keeping typing, scrolling, and resizing smooth while the app is busy rendering a wall of text.
that changed what i look for in the tests: can i still type and resize the window while the stream is coming in? batching updates and breaking long tasks into smaller pieces can leave room for those interactions. the useful measurement is how responsive the app stays while it's working, alongside how quickly it finishes rendering.
Bombadil explores the UI inside Electron, resizing the native window as it goes, with checks for horizontal overflow, hidden composer controls, and drafts lost during resizing. Hegel adds generated renderer and startup cases, giving me concrete workloads to measure and optimize, including the ones that get annoying only after you've used the app for a while.
next time you catch me on my laptop, just know i'm buzzing, fuzzing my systems.