LLM inference : What actually happens in the 2 seconds after you hit enter on Claude

You can buy the best AI server on the market; eight top-end GPUs, about 1.4TB of memory, 500K$, load a frontier open-weight model onto it, and serve a mere ~10 requests per second. Understanding why requires understanding what a transformer is physically doing between your prompt and its answer.

This is a science-popularisation talk: every term is built up from scratch, with no maths and no ML background assumed.
Using that as a thread, we cover:

What happens between prompt and answer: how your text becomes numbers, how the model relates words to each other (that's "attention"), and how it predicts one word at a time, over and over.
Why memory is the real constraint, not raw compute => the model's weights and its running memory of the conversation both live in the same GPU and compete for space. That running memory is the "KV cache", and it exists as a direct consequence of how attention works

The levers that change the equation : storing the model's numbers less precisely (quantization), serving many users in a single GPU pass (continuous batching), and letting the model guess several words ahead (speculative decoding), P/D Disaggregation, distrubuting inference at scale*. Together: a 2–10× gain with no loss in answer quality

Key takeaways:

A mental model of the transformer you can actually reason with when choosing a model or sizing infrastructure
Why the metric that matters is how many users you can serve within your latency budget: not tokens per second. Consumer tools run at 50–100 tok/s, and most use cases never need more
Directly useful if you self-host with Ollama or LM Studio: the same mechanism explains why your context window eats your RAM, why a quantized model suddenly fits, and which settings are worth touching.
Why the big players run freight trucks rather than Ferraris: drawn from two Theodo projects operating low-level inference in production

The session opens with 2 minute following a single sentence through the model, from raw text to a predicted next word. 4 minutes then unpack the puzzle: where the GPU's memory actually goes, why the model's running memory grows with every token of context, and how that dictates the ceiling. The final 2 minutes cover the levers that lift it and reframe the goal from raw speed to serving a broad user base reliably.

Hardware figures will be refreshed to the current generation by the time of the talk.

Based on the article published on Medium here: https://medium.com/@guillaume.gonde_31086/scaling-generative-ai-inference-5a13c01f4e46