The premise of NELSSA, accepted to MICRO 2026, is that short-context and long-context LLM requests have fundamentally different resource profiles and should not share the same hardware tier. GPUs are optimized for large batches of short sequences. Long-context decodes are bottlenecked by KV cache memory bandwidth, not compute. NELSSA splits the workload: GPUs handle short-context requests, and CXL-attached processing-near-memory (PNM) devices handle long-context ones. The result is 5.5x improvement in decode throughput (tokens per second) and 15x reduction in P99 latency across mixed-length workloads, on a physical prototype with RDMA and RPC orchestration over a CXL-enabled infrastructure.
The prototype is the part that earns the MICRO acceptance. Every paper proposes disaggregation; this one built it with real PNM hardware, device-level sparse attention kernels on the PNM side, GPU decode kernels on the GPU side, and a host-side runtime that manages scheduling and cross-tier memory movement. The PNM tier also handles runtime migration when a short context grows into a long one mid-session, so there is no recomputation penalty for misrouted requests.
The practical implication for AI inference infrastructure teams is that CXL memory expansion is no longer just a capacity story. It is a latency architecture. If a production LLM serving system handles mixed-length workloads (which every chat application does), the GPU-only tier is penalizing the short requests to accommodate the long ones and vice versa. The question is not whether PNM-augmented serving is the right direction; NELSSA answers that with a working prototype. The question is what the integration cost looks like against the inference serving platforms in production today. Teams building new AI inference infrastructure in 2026 should treat CXL PNM as a first-class tier in the design, not an optional add-on.