LightRot puts a concrete number on 4-bit LLM inference efficiency at a mature process node: 27.4 TOPS/W in 28nm CMOS, validated against LLaMA2-13B and LLaMA3-8B on MT-Bench. That is the baseline hardware teams designing inference accelerators now have to work from.
The problem being solved is an algorithm-hardware mismatch. Rotation transforms smooth out quantization error by rotating the weight distribution before quantizing, but they conflict with group quantization at 4-bit: the rotation introduces correlations that group scaling factors cannot capture efficiently, degrading accuracy or forcing higher-precision scaling that eats the area budget. LightRot's Grouped Local Rotation (GLR) and Outlier Direction Aligning (ODA) algorithms make the rotation and the group quantization compatible. The hierarchical FHT unit in silicon executes that rotation cheaply. The key move is that the algorithm was constrained by the hardware cost model from the start, not adapted to hardware as an afterthought.
27.4 TOPS/W at 28nm is a node-adjusted starting point. Teams moving this pattern to 7nm or 5nm can expect meaningful gains from process scaling alone. The open question is tail accuracy: MT-Bench tests conversational quality, but the failure modes that matter in production (numeric reasoning, long-context coherence, domain-specific vocabulary) are harder to characterize. If the 4-bit accuracy profile degrades on those workloads, the efficiency gains trade against output quality in ways that silicon designers cannot patch after tape-out.