Athrill found that its RISC-V firmware self-tests could print FAIL and still finish normally. Nothing reported that failure to CTest, the test runner built into CMake, so automated builds could stay green. One floating-point test was worse: it selected its expected answer from the simulator's observed behavior, allowing a missing processor exception to certify itself as correct.
The project added a shared reporter that counts failures and ends with one unambiguous pass or fail line. A Python runner starts the simulator, rejects any earlier FAIL, and only passes when it sees the expected final line. CMake now registers integer, floating-point, device, and optional operating-system boot tests. The expected values are derived in the test instead of copied from the implementation being checked.
The first run exposed a real floating-point conversion failure, which is better evidence than a large passing test count. Firmware teams using simulators should deliberately corrupt one expected value and confirm that continuous integration (CI) turns red; if that experiment stays green, the self-test is still only console output.