Devlog #17 — The Instrument

Date: July 4, 2026

Focus: After the optimization campaign came a harder question — “think Factorio: thousands of entities at 1000 fps” — and the honest answer to it isn’t more optimization. It’s an instrument. This is the devlog where EchoWarrior grows a benchmark mode, learns to measure itself without lying, writes reports a human being can actually read — and then, in its very first run, catches a bug in itself and hands back a curve that humbles the whole campaign.

Last devlog ended with a scoreboard: fewer pixels wasted, fewer draw calls, less arithmetic per frame. Satisfying — and quietly unfalsifiable. Every claim came from reading code and reasoning carefully, which is a fine way to find wins and a terrible way to keep them. If the next feature quietly costs two milliseconds, nothing in the project would have noticed.

So when the ambition escalated to Factorio numbers — thousands of live entities, frame times deep under the refresh rate — the first deliverable couldn’t be another optimization. It had to be the thing that makes optimization claims checkable: a benchmark that produces the same scene every time, measures it without disturbing it, and writes down what it saw.

1. A benchmark is just a game mode

EchoWarrior already had the right architecture for this, almost by accident. The start menu had recently grown an Arena — a sandbox mode for inspecting shaders and tuning values. A benchmark is the same idea with a different obsession: a new Stress Test entry that opens an empty run — no intro dialogue, no wave director — and fills it with a horde.

Everything about it lives in a TOML file, like everything else in this game:

enemy_count = 10000
stages = []                  # empty = the 1-2-5 ladder below
stage_warmup_seconds = 1.5
stage_sample_seconds = 4.0
immortal_horde = true
invulnerable = true

The detail I care most about is the one nobody sees: enemies spawn on a golden-angle spiral, indexed deterministically. Run the benchmark today and after next month’s refactor, and enemy #4 217 stands on the same blade of grass both times. A benchmark that shuffles its scene between runs isn’t measuring the code — it’s measuring the shuffle.

2. Gradual steps, because curves beat points

The first draft did the obvious thing: 1, 10, 100, 1000, 10000. Five points, four decades of nothing between them. Whatever interesting behavior lives between 1 000 and 10 000 enemies — and that’s exactly where it lives — would be invisible.

The fix is the classic 1-2-5 ladder: 1, 2, 5, 10, 20, 50, 100, 200, 500, 1 000, 2 000, 5 000, 10 000. Thirteen stages, evenly spaced on a logarithmic axis, which is the axis entity counts want to be read on. Each stage tops up the horde along the same spiral (the spawn cursor persists across stages), warms up briefly, samples, and moves on. The whole ladder takes about seventy seconds.

3. The paranoid parts

Measuring a game from inside the game is an exercise in not stepping on your own experiment. The list of ways this benchmark refuses to lie got long enough to be its own section:

• Sampling never allocates. Frame times go into a buffer reserved up front; if it ever filled, extra frames would simply go unrecorded rather than trigger a reallocation mid-window.
• File writes happen only between stages. The CSV is refreshed at every stage boundary — so quitting early still leaves partial results — and the next stage’s warmup absorbs the write hitch. IO never touches a sampling window.
• The horde is immortal. The player keeps auto-attacking (that load is real and should be measured), but dead enemies snap back to full health instead of despawning. Otherwise stage populations decay while you measure them — at the single-enemy stage, the benchmark’s subject would be dead within two seconds.
• No XP. A level-up card pausing the simulation mid-stage would stop the clock in the most confusing way possible.
• Interruptions aren’t frames. Alt-tab away and back, and the resume frame arrives claiming it took two thousand milliseconds. It goes in the bin: anything over half a second is an interruption, not gameplay, and one of them would otherwise own the p99 column.
• The player is invulnerable — dying at stage eleven of thirteen is data loss with extra steps.

4. The vsync trap

Here’s the failure mode that would have made the whole instrument worthless on arrival: by default, the display driver syncs frame presentation to the monitor. Benchmark a scene that could run at 800 fps and vsync will report a perfectly smooth 165 — or 60 — for every light stage on the ladder. The low half of the curve measures the monitor.

Two changes. First, the window now takes an explicit swap-interval hint, and it defaults to off — no vsync, no frame caps, no pacing tricks anywhere in the loop. Second, because a driver control panel can override that hint, the report doesn’t trust it: if the lightest stages’ median frame time sits suspiciously close to a common refresh interval — 60, 72, 75, 90, 120, 144, 165, 240 Hz — a warning banner says so, plainly, with instructions. Trust, but verify against the physics of your own monitor.

5. Reports for two audiences

Each run writes two files into the platform data folder. The CSV is for machines and spreadsheets — stable header, one row per stage, a machine-readable summary comment. Any tool that has ever graphed anything can graph it.

The HTML report is for people, and it opens the way a person needs it to: with a sentence — “the game holds a smooth 60 fps up to about N enemies; at the heaviest stage a typical frame took X ms” — generated from the data, in words. Under that, three big numbers — enemies at a smooth 60 fps, fps at the heaviest stage, the 1-in-20 slowest frame — then the scaling chart. The chart carries dashed frame-budget guides at the 60 and 120 fps lines, which quietly do the unit conversion nobody should have to do in their head: the moment the p95 curve crosses a dashed line, that’s the horde size where that refresh rate stops being smooth. The full raw table sits below for tuning work, untouched.

The page is one self-contained file — SVG chart included, generated straight from Rust, no external assets, no scripts, no toolchain. (The obvious move was “write a Python script that makes a graph.” This project already fought Python once, lost the fight to a corporate allowlist, and chose Lua for scripting because of it. The graph gets rendered by the same language that renders the game.) The two series colors went through a colorblind-safety validator in both light and dark mode, and the rendered page was screenshot-checked in a headless browser before shipping — a chart you never looked at is a chart you shipped broken.

6. First light — and the instrument bites its maker

I ran a first shakedown pass the moment it built. Treat everything in this section as exactly that — one partial run on one machine, aborted after the 50-enemy stage, to be redone properly — but even a shakedown run turned out to be worth a devlog section, because the instrument did two things in its first seventy seconds of existence.

It caught a bug in itself. Every stage reported draws = 0. The draw-call capture armed macroquad’s telemetry every frame — and it turns out the frame reset clears the previous record whenever a new capture request is pending, so the benchmark was always reading a list that had been emptied a microsecond earlier. Classic instrumentation lesson: the first thing a new measuring device measures is its own wiring. Fixed the same evening — capture is now a one-shot: arm when the sampling window closes, let the next frame record completely, read the frame after.

And it drew a curve nobody asked for. Here are the real rows from that partial run, exactly as the CSV recorded them:

entities  avg_ms   p95_ms   fps
       1  18.7     21.9     53
       2  25.8     30.4     39
       5  38.5     45.2     26
      10  49.7     57.1     20
      20  63.2     70.4     16
      50  100.4    126.5    10

Read that twice. A single enemy on an empty field costs nearly nineteen milliseconds — and every additional enemy adds roughly a millisecond, linearly, which is orders of magnitude more than any sane per-entity budget. The whole optimization campaign made the pipeline demonstrably better by construction — and the first end-to-end measurement says something enormous and unaccounted-for is eating the frame anyway, before entity count even enters the picture, and something else is charging a per-enemy toll that no amount of draw-call batching explains.

I have suspicions. I am deliberately not printing them, because the entire moral of this post is that suspicions are what the instrument exists to replace. The next run happens with the fixed draws counter and the F1 frame profile open, and the diagnosis — the real one, with receipts — is the next devlog.

7. What the instrument is for

None of this makes the game one frame faster. That’s the point. The last devlog’s optimizations were arrows; this is the target that tells you whether they landed. From here on, the workflow for every performance change is fixed: run the ladder, keep the report, make the change, run the ladder again, diff two files. And as section 6 shows, the very first thing it did was replace a comfortable story with an uncomfortable measurement.

That’s not the instrument failing. That’s the instrument working.

Measure honestly, or don’t bother measuring.