Skip to content
hw.dev
hw.dev/signal/llm-gpu-kernel-verifier-adversarial-2026
SignalarXiv

39.5% of LLM-Generated GPU Kernels Are Broken. The Test Suite Did Not Know.

A 12-gate adversarial verifier found 39.5% of machine-generated GPU kernels broken and 62.1% carrying at least one violation -- kernels the standard test suite had already accepted as correct.

#verification#testing#tools#ai-hardware
Read Original

The standard correctness test for LLM-generated GPU kernels is: run it on a few random inputs at one fixed shape, compare output to a reference, accept if close enough. Shah and Shrestha audited 2,638 kernels a public system had already accepted under that bar and found 39.5% broken beyond any tolerance argument, and 62.1% carrying at least one violation. The field's standard test accepted 1,487 kernels the adversarial verifier rejects. That is not a small gap in the test suite. That is the test suite measuring the wrong thing.

The verifier they built has 12 gates, each a property a correct kernel must satisfy. Several are tolerance-free: a kernel that returns an ordinary float where the reference returns NaN fails, period, and no choice of epsilon explains it away. Other gates catch shape sensitivity (correct at the training shape, wrong when dimensions change), run-to-run nondeterminism, and precision accumulation errors (fp16 accumulation where the reference uses fp32). These are the failure modes that pass a one-shape random test and then break in production, at scale, on inputs that were not in the benchmark.

The paper also ships what it verifies with the verifier: the first native Blackwell tcgen05 training backward for the gated-linear-recurrence family, including the reverse-state stage the field still runs on a fallback. Correctness established against a double-precision oracle, five family members trained through it. That is the pattern the field needs -- build the verification harness, then use it to certify your own output before publishing correctness claims.

Teams using LLM-assisted kernel generation for AI accelerator workloads should run this verifier, or build an equivalent, before shipping any kernel to production. The cost of a tolerance-free gate is a test that actually rejects broken kernels. The cost of not having one is 39.5% of your accepted kernels carrying silent bugs through to deployment.