The benchmarks used to evaluate LLMs on RTL generation are broken, and GateTruth now has numbers to prove it. The paper applies mutation testing to RTL benchmark testbenches -- a standard hardware verification technique that injects semantic mutants into a reference design and measures what fraction the testbench catches. Of 46 auditable designs in RTLLM v2.0, 72% fall below the 95% mutation kill floor that GateTruth's own 68-task suite is held to. Three designs score 0% outright, meaning their testbenches pass every broken implementation GateTruth can construct.
The mechanism is straightforward and the implications are uncomfortable. A testbench that never fails is not evidence of a correct design; it may never stimulate the logic that is actually broken. When LLM RTL generation papers report pass rates on RTLLM v2.0, they are measuring the LLM's ability to produce code that satisfies a testbench that cannot tell good RTL from bad. Every leaderboard built on RTLLM v2.0 pass rates is measuring something other than RTL correctness. GateTruth also tried to audit NVIDIA's CVDP benchmark and found it structurally impossible: NVIDIA withholds the reference solutions that mutation testing requires, making external quality audits impossible by design.
This matters most for the teams using benchmark pass rates to make tooling decisions. If the benchmark cannot distinguish correct from broken, improving your score is not the same as improving your RTL. GateTruth ships its mutation engine and its own 68-task dual-track suite as open source, with 46 of 60 Track A testbenches certified above the 95% floor. The practical move for anyone building or evaluating LLM-assisted RTL tools is to run GateTruth's engine against whatever benchmark they are using before treating any pass rate as load-bearing evidence.