Hardware teams designing inference accelerators typically pick a quantization scheme on GPU benchmarks and hand the winner to an RTL team. Quantization algorithm and silicon target are designed in sequence. HBQ (Hierarchical Block Quantization), accepted at MICRO 2026, treats that sequential workflow as the root problem. The paper co-designs a block quantization algorithm with a 28nm ASIC and delivers 2.3x area efficiency and 4.6x energy efficiency over weight-only quantization at the same accuracy, plus 1.5x-3.0x speedup and 1.6x-3.3x energy reduction over prior block quantization methods.
The core constraint HBQ resolves is a precision trade-off that prior block quantization methods left unaddressed. Small blocks preserve accuracy but add dequantization and accumulation overhead that scales with block count, reducing hardware efficiency. Large blocks reduce overhead but degrade accuracy. HBQ uses large blocks to maximize efficiency and adds hierarchical significand (SIG) scaling as a second-level correction that compensates for large-block quantization errors without meaningful area overhead. The 28nm ASIC applies HBQ to weights, activations, and KV cache simultaneously, and adds a partial-sum block quantization scheme to reduce energy in the multiply-accumulate fabric. HBQ-A reaches W4A16-level accuracy at W4A5 compute cost; HBQ-E cuts area by an additional 17% while still outperforming all prior block quantization methods on accuracy.
Inference accelerator teams benchmarking quantization schemes on GPU rigs and then handing the winner to a VLSI team are leaving 2-4x area and energy efficiency on the table. MICRO 2026 is a tier-one systems venue. Within 18 months, hardware-aware quantization co-design of the kind HBQ demonstrates will be the expected baseline for any inference accelerator tape-out, not a differentiating research contribution. Teams still running quantization and silicon design as sequential decisions have a cost structure problem relative to what the field will consider standard practice after this publication lands.