Safety cannot be a policy the model talks its way past

A wrong answer in an information system costs a retry. A wrong actuator command costs a part, a person, or a programme. That gap is wide enough to change where the guarantees live, not merely how thoroughly the thing is tested before it ships.

The mistake we see most is treating safety as a very well-behaved component inside the control system — a module with good test coverage, reviewed carefully, sitting in the same process and the same decision graph as everything else. It has to sit outside it.

Why inside is the wrong place

A safety check that lives inside the control system shares its failure modes. It runs on the same scheduler, depends on the same state, and is reachable by the same code paths. When the controller enters a state its designers did not anticipate, the check is inside that state with it.

Worse, a check that is part of the decision graph is something the decision process can optimise against. This is not a hypothetical about deceptive models; it is ordinary engineering pressure. If a limit is a parameter the controller can read, some future version will read it, and the behaviour will be shaped by where the limit is rather than by what is actually safe.

Independence means the mechanism does not consult the thing it constrains, cannot be reconfigured by it, and continues to function when it has failed entirely — including when it has failed in a way nobody enumerated.

Deterministic, and independent of anything learned

A learned policy may propose bounded actions. It may not bypass command validation, emergency stop, the watchdog, or actuator power isolation. Those mechanisms live outside the learned system, do not consult it, and are not tuned by it.

Command validation rejects commands outside the envelope before they reach hardware, on limits that are declared rather than inferred. The emergency stop is reachable without the controller cooperating. The watchdog assumes that silence means failure, because a controller that has stopped reporting is not a controller that is thinking. Power isolation exists because the correct response to a sufficiently confused machine is not a better command, it is no torque.

Confidence is not an input. A policy that is extremely sure it should exceed a limit is describing a policy failure, not an exception to the limit. This is the load-bearing sentence: the moment confidence can widen the envelope, the envelope was only ever advisory, and every subsequent safety argument is about the quality of a probability estimate rather than about a bound.

This is the same rule the software side applies to agents — propose, do not decide — enforced in the one place where it is least negotiable.

The same contracts in simulation and in metal

Simulation and hardware execute against identical runtime contracts, with the differences confined behind the hardware abstraction. Where that is not true you validate one artifact and ship a different one, and the divergence gets discovered by the machine rather than by the test.

The failure is quiet and expensive. Simulation runs establish confidence, that confidence is transferred to the physical build by assumption, and the first honest measurement happens when something moves. Every hour of simulation spent against contracts the hardware does not share is an hour of evidence about a system that was never going to be built.

Generated artifacts — simulator descriptions, diagrams, dashboards, exported geometry — are projections of the product model, never sources of truth. They may explain the system. They may not define a part, a limit, or a release state. A rendering is a convenient thing to review and a catastrophic thing to build from, because it looks authoritative and carries none of the constraints that made it.

The loop this sits inside

Define, simulate, fabricate, operate, capture evidence, diagnose, calibrate, improve, and around again. The loop matters because each pass produces evidence from both the simulated and the physical article, and the difference between them is the most informative signal the programme generates.

Treating that difference as noise to be smoothed over is how a simulation stack slowly stops predicting anything. Treating it as the finding is how the model gets better, and it is why evidence capture belongs in the loop rather than in a report written afterwards.

The obvious objection

Does hard-bounding the learned system cap what it can achieve? It caps one specific class of action: the class that exceeds a declared physical limit. Inside those limits the policy has the entire envelope, and the envelope is usually far larger than early behaviour explores.

The trade is worth making because the alternative is not more capability, it is a capability claim that cannot be certified. A machine whose safety argument rests on the good behaviour of a learned policy has no safety argument that survives the policy being retrained — and it will be retrained.