Google's 8th generation TPU is not one chip. It is two: one for training, one for serving. That split is the most honest thing the industry has said about the AI workload gap in years. Training and inference have been served by the same silicon family since the first TPU. The 8th generation abandoning that assumption is a design decision with a specific cost: you now need two roadmaps, two tape-out cycles, two supply chains, and two software stacks. Google decided that cost is less than the efficiency penalty of a generalist die.
The divergence is real and quantifiable. Training wants maximum matrix throughput, high all-to-all interconnect bandwidth, and long-running job stability. Inference in the agentic era means variable batch sizes, long context windows with large KV caches, latency-per-token SLAs, and cost-per-query pressure across millions of requests per second. Those two constraint sets pull the memory hierarchy, interconnect topology, and compute precision in opposite directions. A die that optimizes both is paying a significant area and power tax on each.
The subtitle "in the Agentic Era" is load-bearing. Agentic workloads generate more inference than batch training as a fraction of total compute (longer context, more back-and-forth, tighter latency requirements). Google is designing to that shift, not to the 2024 training-dominant assumption.
Every team still running a unified training-and-inference cluster on the same accelerator family is now looking at the 8th Gen TPU decision and running the math. The mid-market (companies that cannot fund two silicon programs) is the contested ground for whoever builds the first unified die that matches the 8th Gen TPU's per-workload efficiency. That will not be a GPU vendor. It will be a new entrant or an ODM with a co-design model.