Skip to content
hw.dev
hw.dev/signal/margin-bench-plant-acceptance-tests-2026
Signalmargin-bench

Margin Bench Grades AI Engineers With Plant Acceptance Tests

margin-bench evaluates AI agents by letting them commission hidden simulated plants, then grading the resulting controller or signal chain with deterministic acceptance tests.

Thesis connection

Deterministic plant acceptance tests replace model-graded demos with replayable evidence about whether an AI agent can complete engineering work.

#testing#verification#ai-hardware#software
Read Original

Margin Bench asks a better question than whether an AI agent can explain control theory: can it commission a plant that passes acceptance? The open benchmark gives a model hidden simulated equipment, engineering tools, a plant-time budget, and written requirements. A fresh copy of the plant then grades the controller tuning or DSP chain with deterministic tests, never another model.

Its first environments expose the shape of a credible hardware-agent benchmark. One agent must tune a delayed, noisy tank-level loop without tripping spill or dry-pump alarms. Another must build a weighing-scale signal chain under a 12 multiply-adds-per-sample budget while rejecting mains hum and converter noise. Seeded parameters stay hidden, every tool call is replayable, and the test suite proves the reference engineer passes 20 seeds while known-bad strategies fail all 20. No production model results are published yet, which is a useful boundary rather than a weakness disguised as a leaderboard.

Teams building AI for test, controls, or electronics should copy this evaluation pattern before buying a model benchmark. Budget a week to encode one real commissioning task as a hidden plant plus acceptance test. Any agent that cannot beat the known-bad baseline there is not ready for the lab.