The Arithmetic of Inference: How a Modern Inference API Actually Spends Your FLOPs
A deep-dive into the layer underneath "fast, scalable, efficient": the roofline arithmetic that decides whether a token is cheap, the KV-cache accounting that caps batch size, and the routing and scaling that turn a pile of GPUs into a serving system.