GateTruth and a concurrent GPU kernel audit, from independent teams, found the same structural failure across two domains: the benchmarks used to measure AI hardware code quality cannot catch the bugs they were built to detect. Every RTLLM pass rate cited in product documentation or a research paper is a number the field has not verified means what it claims.
The Same Failure, Two Domains
GateTruth (arXiv:2608.12635) applied mutation testing to 46 auditable designs from RTLLM v2.0, the benchmark that has been the shared currency for comparing LLM tools on RTL generation since HKUST published RTLLM v1.0 at ASP-DAC 2024 (arXiv:2308.05345). Mutation testing is not exotic: it has been the standard method for evaluating testbench quality in software since DeMillo, Lipton, and Sayward introduced it in 1978. The technique injects semantic variants into a reference design and measures what fraction the testbench catches. GateTruth applied it to RTLLM and found 33 of 46 designs fall below the 95% mutation kill threshold, the bar below which a testbench cannot be trusted to distinguish correct from subtly wrong.
The GateTruth paper notes that across every RTLLM paper, repository, and documentation set the authors accessed, none had applied mutation testing to its own testbenches. The benchmark-building community never verified the benchmarks.
Three days later, Shah and Shrestha (arXiv:2608.12700) published an adversarial audit of LLM-generated GPU kernels. The standard correctness test for these kernels is a handful of random inputs at one fixed shape, compared to a reference at a tolerance. A public system had already accepted 2,638 kernels under that bar. The audit found 39.5% broken beyond any tolerance argument, and 62.1% carrying at least one violation. The test the system was running had no way to catch what it did not test: different shapes, different sparsity patterns, edge cases outside the one fixed input. Convenience-optimized acceptance tests accepted 1,040 broken kernels without flagging a single one.
The structural finding is the same in both papers: tests designed for ease of running were not designed to cover the failure modes that matter.
How the Measurement Gap Built
RTLLM v1.0 launched at ASP-DAC 2024 because the field needed a way to compare LLMs on RTL tasks. Synopsys, Cadence, and a growing set of AI hardware codegen startups began reporting RTLLM scores in product documentation. Pass rates became the shared shorthand for AI RTL quality claims. RTLLM v2.0 (OpenLLM-RTL, arXiv:2503.15112, March 2025) expanded the suite, and adoption accelerated. By mid-2026, RTLLM scores were cited in at least a dozen papers and in commercial product pages as evidence that one tool generates more correct RTL than another.
The auditing lag is predictable. Mutation testing is computationally expensive and requires domain expertise to configure correctly for RTL. The benchmark-building phase preceded the benchmark-auditing phase by roughly two years, the same pattern software has seen every time a benchmark proliferates faster than the rigor to validate it. No one asked the obvious question: can the testbenches actually detect the failure modes they are supposed to catch?
GateTruth answers that question and ships the tooling to close it. The open-source repo includes a mutation-tested 60+8-task reference suite that removes the setup barrier. The remaining constraint is adoption: benchmark maintainers accepting that their current testbenches need replacement, and the tool vendor community retiring pass rate claims until that happens.
Who Benefits, Who Is Exposed
Teams that have already been skeptical of RTLLM pass rates now have numbers. Hardware engineers who know that a testbench written by the same LLM that wrote the design is not an independent verifier had an intuition; GateTruth made it concrete and reproducible.
SparseDitto (arXiv:2608.05033) sidesteps the evaluation problem by generating GPU kernels per matrix and measuring throughput directly against hardware, not against a format-optimized library baseline. That is a harder correctness gate than benchmark compliance and closes on the metric that matters: does this kernel outperform the library on this matrix, on actual hardware, across the shapes that the application uses?
The exposed parties are specific. EDA vendors and AI codegen startups that have been marketing their tools using RTLLM pass rates face a credibility problem, not just a marketing one. Every benchmark score published against testbenches that fall below the 95% mutation kill threshold is potentially inflated. The tools may still be good; the score is not the right evidence. If the benchmark maintainers at HKUST release mutation-tested replacements and the pass rates shift, every product comparison built on the prior benchmark needs to be rerun.
Hardware teams that purchased AI RTL tools based on RTLLM scores need to re-evaluate those tools against their own designs and their own testbenches, with mutation testing applied to the testbenches themselves.
What Builders Should Do
Before trusting AI-generated RTL in a production flow, run mutation testing on your testbench. GateTruth ships the open-source tooling (github.com/meetbhadra701-cloud/GateTruth). Budget one engineer-day for setup and a single design run. A mutation kill rate below 95% on your validation designs means you are running a verification gate that cannot reliably catch the failure modes it is supposed to catch. The cost of finding that out now is one day. The cost of finding that out after a respin is not.
For GPU kernels: the standard single-shape acceptance test is the problem the Shah and Shrestha audit named. Test at multiple matrix shapes and sparsity patterns representative of your actual workload. The default cuSPARSE benchmark is a performance comparison, not a correctness gate.
What Would Kill This Claim
Two limits bound the analysis. First, mutation testing catches functional correctness failures at the RTL level. It does not catch timing violations, power overruns, or physical design failures that only appear after synthesis. The 72% of RTLLM testbenches that fall below threshold cannot reliably verify functional correctness, but LLM RTL quality is a multi-layer problem and this paper addresses only one layer of it. A design that clears a mutation-tested testbench can still fail synthesis, and a synthesis-passing design can still fail physical sign-off.
Second, the benchmark maintainers may move fast. If HKUST releases a mutation-tested RTLLM v3.0 in the next six months and the tool-vendor pass rates hold under the new testbenches, the "broken ruler" claim weakens to "the ruler was wrong and now it is fixed." That is a good outcome for the field, and the claim to watch is whether the vendor pass rates survive the methodology upgrade. If they do not, the RTLLM leaderboard from 2024 through mid-2026 is a ranking of tools on a test they were never supposed to be ranked by. Inside 12 months, the field will know which of those outcomes it got.