Scripting & deterministic simulation testing

The simulator can be driven from a script: a text file of commands, each stamped with the tick it runs on. Pass it with --script/-i and the run needs no client at all.

python cli/robosim.py --headless -p 0 -i test/robosim/integration/dst/scripts/connect.txt

That one flag is what most of robosim's test suite is built on. This page covers the format, how a script becomes a test, what is and is not reproducible today, and where the idea comes from.

Why: a socket makes a bad test harness

Robosim's tests used to be bash: start the server on a port, pipe commands through cli/robosim-ncat.py, grep the replies, sleep between steps. It worked, and it had three problems that no amount of care fixes.

1. Time leaked into the assertions. A test that sends a command and samples the state "shortly after" is measuring the machine, not the simulator. One of them failed 3/3 purely because a neighbouring test waited ~0.3s before it and shifted its sampling window.
2. The assertions were weak by construction. grep -q collision_error over 300 replies passes if one of them arrives. "The collision is reported on every tick it happens" is not expressible, so a bug where it is reported only sometimes is invisible.
3. Cost per test. A port, a server process, a readiness poll, a cleanup trap, and jq — dozens of lines of ceremony before the first assertion, in every suite.

A script removes all three: the tick index replaces sleep, the whole reply list is in hand at once, and there is no process to manage.

The format

One command per line:

<tick> <command> <json arguments>

0 connect {}
1 move {"control_input": [1.0, 0.0, 0.0, 0.0, 0.0, 0.0]}
2 robot_get_state {"robot_ix": 0}
3 sim_exit {}

Rules, all enforced by _build_script_lines (src/robosim/robosim.py):

• the tick is a non-negative integer — no floats, no negatives (they are a ValueError, not a silent round);
• blank lines and # comments are skipped, which is what lets a fixture be commented prose around its commands;
• several lines may share a tick; they run in file order within it;
• the arguments are a JSON object, merged with {"cmd": <command>} to form the message. So a script line is the same payload a socket client sends, minus the cmd key, which the line already carries.

That last point is the design decision worth naming. The vocabulary and the payloads are exactly robosim-ncat's raw-JSON mode — the same commands, the same fields, validated by the same microspec schemas (src/robosim/protocol.json for the core commands, each plugin's own declared endpoints for its own). What a script deliberately does not borrow is ncat's human shorthand (move 1 0 0 0 0 0, robot_get_state 0), which exists to save typing at an interactive prompt. A shorthand is a second parser, and a second parser drifts from the spec: a field added to protocol.json would work over the socket and silently not work in a script. One JSON payload, one schema, one place to be wrong.

Scripts are ordinary files, so the simulator's own --script flag and the test suite read the same file. A failing test is replayable against a live server by copy-pasting its command line.

The map editor's -i: same flag, no ticks

cli/mapeditor.py -i <file> reads a different script: plain CLI lines, exactly what a client sends over the editor's socket, one per line, no tick column. They run first, one line per frame.

help {}
entity_spawn {"components": ["HasTag", "HasPose"], "data": {"tag": "crate", "pose": [[1,0,0,7],[0,1,0,3],[0,0,1,7],[0,0,0,1]]}}
me_save_state {}
exit {}

There is no tick because the editor has no clock and no physics: between two commands nothing happens but world.update(), so a tick number would be a number that changes nothing. What the line boundary does buy is that world.update() — microecs applies structural changes (add_entity, add_component, remove_component) lazily, so a line and the line under it are different frames. Put a spawn and a save in the same frame and the save raises AttributeError: Entity 0 not in world. Call world.update() if it was just added.

exit {} sets state.must_exit and breaks the loop, which is what makes a scripted run terminate with no window to close. A script without it is the other use: -i scene.txt builds a scene and hands the editor back to the operator. # comments and blank lines are skipped before the manager sees them (it would answer them with an error otherwise).

The editor has its own test suite in test/mapeditor/integration/, the same "the file is the test" pattern as the DST suite — one test per scripts/*.txt — with three differences: replies are strings compared verbatim (#/.../ for a regex), there is no tick column, and every fixture must end in exit {}, since there is no runtime bound to catch a forgotten one.

A script becomes a test: the fixture file IS the test

main() returns the reply of every executed line, in order, so a test is "run this file, check these replies". The fixtures live in test/robosim/integration/dst/scripts/ and the runner is test/robosim/integration/dst/test_i_scripts.py — one test per .txt, generated at collection time. Adding a test means adding a text file, never a Python file.

A fixture states its claims three ways, all inside the file:

#expected — one reply

The comment immediately under a command is that command's expected reply, compared against the reply's canonical JSON (sort_keys=True). A blank line ends it, so a command with no expectation can still be followed by a paragraph of prose.

0 connect {}
#{"eid": 14, "robot_ix": 0, "status": "connected"}

1 connect {}
#{"error": "already connected"}

Wrap it in slashes for a regex (re.fullmatch) — the escape hatch for a reply that is large or carries a timestamp:

1 robot_get_state_with_camera {"robot_ix": 0}
#/\{"fpv_compressed": "<bytes>", "fpv_frame_id": \d+, .*\}/

# assert: — something about several replies

An #expected line cannot say "it moved", or "the second displacement is smaller than the first". A # assert: header is a plain Python expression over the whole run, evaluated once:

# assert: states[1][0] < states[0][0]
# assert: 0 < states[2][0] - states[3][0] < states[1][0] - states[2][0]
# assert: len(collisions) > 0

The names are deliberately few (see script_utils.assert_expressions): replies (every reply), states (the xyz of each robot_get_state reply), moves (the data of each applied move), and collisions (every reported collision). They index into filtered lists, not into replies, so inserting a move line does not renumber what an expression talks about.

This is an escape hatch, not the main road. Every one of these exists because the exact numbers are not reproducible yet — see below — and each should become an #expected golden when they are.

# log: — something the sim told the operator

For the rare claim that has no reply of its own, like "an unknown plugin in a loaded state is skipped with a warning naming it". Matched against the run's stderr. Use sparingly: a reply is a contract, a log line is not.

Headers: # map: and # plugins:

A fixture configures its own world and plugin set, so it depends on nothing global:

# map: test/robosim/integration/dst/maps/minimal.json
# plugins: manual_move race

Running them

A test belongs in the first layer that can reach it:

• test/robosim/integration/ — DST script fixtures (pytest). A fixture is a .txt of tick-stamped commands with expected replies as # comments, run in-process by the simulator. Adding a test means adding a .txt.
• test/e2e/ — bash, over a real socket. Only for what the layer above cannot reach: the transport contract, the performance suites (they need a real clock), and the fuzz replay corpus.

python -m pytest test -q                            # robolib unit tests + the DST suite (~12s; -n 16 parallelizes)
python -m pytest test/robosim/integration -q -k mission     # one fixture
[CI=true] bash test/e2e/run_all.sh                  # socket tests (CI skips the slow perf + fuzz suites)
bash test/lint/run_all.sh                           # ruff + pylint, robosim and vendored pkg/
python cli/robosim.py --headless -p 0 -i test/robosim/integration/dst/scripts/mission.txt   # replay it live

Two static guards run alongside, with no simulator at all: the generated spec_coverage.txt must match what generate_spec_coverage.py produces from protocol.json, and every endpoint the server declares — plugin endpoints included — must be called by some fixture, or be listed as an explicit exemption with a reason. Add an endpoint and the suite tells you it is uncovered.

What is reproducible today, and what is not

The goal is TigerBeetle's: same seed and same initial state ⇒ the same run, byte for byte. Robosim is partway there, and the honest status matters because it decides whether a fixture can pin a number.

Reproducible. The tick a command runs on, the order of commands within a tick, and therefore which commands have been applied before any given read. That alone removes every sleep from the suite.

Not yet reproducible: how much physics happens per tick. Clock.tick() reads the wall clock:

frame_time = now - self.prev_time    # the only nondeterministic input
self.accumulator += frame_time

drain() then yields floor(accumulator / dt) physics subticks, capped at MAX_SUBTICKS_PER_RENDER_TICK (3). A slow frame silently buys an extra subtick, so after N ticks the robot has had somewhere between N and 3N physics steps and its pose is a function of machine load. That is why the physics fixtures assert inequalities instead of goldens. The fix is an injectable clock (tracked as task 193): a simulated clock advances by exactly dt per tick, so a tick that takes three real seconds still counts as one dt and runs one subtick. The sim runs slower or faster than real time and does not care.

A sharp edge: reply order within a tick. Replies come back in three groups, not in file order.

So 1 move {..} followed by 1 sim_get_info {} returns sim_get_info first. It is fully deterministic, just not submission order, and a socket client never observes it (each has its own channel and its own request/response lockstep). Only a script, which collects every reply into one list, can see it. script_utils.load_cmds rejects a fixture that writes a plugin command before a core one in the same tick; within the plugin groups, keep a tick's commands either all valid or all invalid.

What still needs a socket

test/e2e/ is down to two correctness suites, and that is on purpose.

connect-and-move is the socket itself. A script client has no socket, so it cannot exercise: the connection manager's accept gating (a fresh connection must connect first), the FPV fast path answered entirely on the client thread, the main2tcp handoff back to a parked put_then_get, or EOF releasing a robot. A DST fixture cheats by construction — in process, and on a fake clock once task 193 lands — so something has to drive the real thing.

editor-to-server is the other thing a single process cannot fake: two apps and a file. The editor saves resources/state.json, the server boots on it (--map_path), moves a robot, saves it back, and the editor boots on that and re-saves it byte-identically. Both apps' world_to_dict/world_from_dict are pinned against each other by a real json.dump between two OS processes.

Tests move from test/e2e/ to test/robosim/integration/, never back. If an assertion can be made without a socket, it does not belong in bash. Porting the suite this way took the fast gate from 85s to 3.4s and the CI pipeline from ~2:20 to ~1:00. It also turned up bugs the bash suite had been running past for months — a collision that is silently never reported when a render tick drains more than one subtick, resource leaks that made a second main() call in one process unsafe, and a schema-valid collision_cell_size that leaves the simulator technically responsive and practically unusable. None of them were reachable by a suite that asserts "a reply came back" with a sleep in front of it.

Prior art

Deterministic simulation testing is not ours; we are copying people who did it properly.

• TigerBeetle — the project's north star. Its VOPR runs a whole cluster in one process against a simulated network, clock and storage, then fuzzes over seeds; a failing seed is the bug report. The separation robosim copies is the one between a deterministic state machine and the message bus around it — which is exactly what made scripts possible here (see Architecture: every handler returns replies, only protocol.py touches a channel).
• FoundationDB — the canonical example of building the simulator before trusting the database. Will Wilson's talk "Testing Distributed Systems w/ Deterministic Simulation" (Strange Loop 2014) is the standard introduction; the same people later built Antithesis around the idea.
• Deterministic lockstep in games — fixed timestep plus identical inputs on every machine gives identical state, which is how RTS titles synchronise thousands of units over a modem's worth of bandwidth. Same principle as (map, script) ⇒ same run; also why task 193 matters, since a variable-substep loop is exactly what breaks lockstep.
• Record/replay in physics engines (e.g. box2d/box3d-style input recording) — capture the inputs, replay them to reproduce a bug instead of describing it.
• Expect/golden testing — the "fixture file is the test, and it holds the expected output" shape, as in Jane Street's expect_test, Rust's insta, or cram for CLIs. Our #expected lines are that idea applied to protocol replies.
• sans-I/O protocol libraries (h11, hyper-h2) — the core parses and returns events; the caller owns the socket. PluginsManager returning (Message, Response) pairs instead of writing to a channel is the same move, and it is why a plugin endpoint can be driven from a script with no transport at all.

Further reading

• Architecture — the tick, and why every handler returns replies instead of sending them
• Plugins — how a plugin endpoint is routed and answered
• Protocol — the wire format and the command reference