Inference Engineering · learning portal
0 / 0

Study companion · Philip Kiely, Inference Engineering (Baseten Books, 2026)

Learn to serve AI models fast, cheap, and reliably.

This portal turns the 259-page book into a trackable course. Part 1 is a recommended study path with the best external materials. Part 2 is the book itself, chapter by chapter, with study notes, key terms, and self-check questions. Tick a box when you've studied a section — progress is saved in this browser and survives reloads.

0 sections done
0 remaining
7 chapters + 2 appendices
6 study-path stages
Part 1 · Recommended study path

How I'd learn inference engineering

Work the stages in order — each one builds the mental model the next one assumes. Pair every stage with the matching book chapters from Part 2, and do the hands-on project in Stage 6 as early as you can tolerate; nothing teaches inference like watching your own deployment run out of memory. Items marked ★ are the highest-leverage picks if you're short on time.

Stage 1Transformer & LLM inference fundamentals

Before optimizing inference you need to know exactly what happens in a forward pass: tokens, embeddings, attention, logits, sampling, and the prefill/decode split. Pair with book chapters 0–2.

Stage 2GPUs and the performance mental model

Everything in inference optimization reduces to one question: is this operation compute-bound or memory-bound? Learn GPU anatomy and the roofline model. Pair with chapters 2.4 and 3.

Stage 3The serving stack: engines and attention kernels

Now learn the tools you'll actually operate. Read the two papers that shaped modern serving (FlashAttention, PagedAttention), then get fluent in one engine and conversant in the other two. Pair with chapter 4.

Stage 4Optimization techniques research

The applied-research layer: quantization, speculation, caching, parallelism. Read alongside chapter 5 — the book gives you the map, these papers give the territory.

Stage 5Production infrastructure

Serving one GPU well is runtime work; serving thousands is systems work. Pair with chapter 7.

Stage 6Hands-on projects (do these, actually)

Rent a GPU by the hour (RunPod, Lambda, Modal, or Baseten) — a full project costs a few dollars. Each project cements one chapter of theory.

Part 2 · The book, chapter by chapter

Course: Inference Engineering

For each section: read the pages in your PDF, review the note here, then mark it done. Finish a chapter by answering its self-check questions from memory before revealing the answers. Page numbers match the printed book (PDF page ≈ printed page + 7).

CH 0 · pp. 15–22 Inference The framing chapter: why inference is now its own discipline, and the three layers every serious inference platform needs.
Key termsruntime / infrastructure / toolingopen vs. closed modelsKV cacheprefill / decode
Self-check
Traffic overwhelms a perfectly optimized single model server. Which layer owns the fix?
Infrastructure. No runtime optimization survives unbounded traffic; you need autoscaling, routing, and capacity management — a systems problem, not a CUDA problem.
Name the two phases of LLM inference and what each produces.
Prefill processes the whole input sequence and builds the KV cache (determines TTFT); decode runs autoregressive forward passes that each emit one token (determines TPS).
CH 1 · pp. 23–37 Prerequisites What must be true about your product before inference engineering pays off: use case, model choice, and the metrics you'll optimize.
Key termsTTFTperceived vs. total TPSITLP50/P90/P99evalsdistillation
Self-check
Your P50 TTFT is great but users complain the app "sometimes hangs." What do you look at?
P90/P99 latency. Right-skewed distributions hide outliers behind good medians; one slow request in ten ruins perceived reliability.
When is it wrong to move off pay-per-token APIs?
Before product-market fit or clear volume: dedicated GPUs raise your spend floor and add engineering surface area with no offsetting scale, specialization, or orchestration need.
One model serves both a live dictation app and nightly batch transcription. What's the recommended setup?
Two deployments of the same model — one tuned for latency (online), one for throughput (offline batch) — assuming both have enough volume.
CH 2 · pp. 39–69 Models How LLMs and diffusion models actually compute, and where the bottlenecks live. The most conceptually dense chapter — take it slowly.
Key termslogits · samplingchat templateMoE · routerlatent space · VAEarithmetic intensityroofline modelFlashAttentionPagedAttention
Self-check
Why is decode memory-bound while prefill is compute-bound?
Prefill loads weights once and does massive parallel matmuls over the whole input (high intensity). Decode reloads the full weights for every single token and does comparatively tiny vector-matrix math (low intensity), so bandwidth is the ceiling.
Why doesn't an MoE model's "22B active parameters" mean batched serving only touches 22B?
Different requests in a batch activate different experts, and the router picks per token per layer — so across a busy batch nearly all experts are hot unless you achieve sparsity via large-scale expert parallelism.
FlashAttention doesn't change attention's O(N²) complexity. Why does it help so much?
It eliminates redundant reads/writes of intermediate matrices (S, P) by fusing the computation and tiling it to fit the GPU's memory hierarchy — the same math with far less memory traffic, which is what actually limits the kernel.
Why do image generators work in latent space rather than pixel space?
Attention must consider the whole image at once; a 1024×1024 image has 1M+ pixels, infeasible for attention. A ~128×128 latent representation is ~1% the size, and the VAE converts back to pixels at the end.
CH 3 · pp. 71–91 Hardware GPU anatomy, NVIDIA's generations, instances and interconnects, the competition, and inference at the edge.
Key termsSM · Tensor CoreHBM · VRAMdense vs. sparse FLOPSNVLink · NVSwitch · InfiniBandMIGSXM vs. PCIe
Self-check
You need more tokens per second per user on a chat app. H100 → which upgrade, and why?
H200 (or B200): decode is bound by memory bandwidth, and H200 raises bandwidth 3.35 → 4.8 TB/s. More FLOPS alone wouldn't move TPS.
Why are L40s usually a poor inference choice despite decent specs?
No NVLink (rules out efficient multi-GPU), and for the same memory footprint fractional H100 MIGs deliver much higher compute and bandwidth.
Why does the NVLink-vs-InfiniBand bandwidth gap dictate parallelism choices?
Tensor parallelism needs all-reduce sync every layer — fine at NVLink speeds inside a node, ruinous over InfiniBand. Across nodes you switch to pipeline or expert parallelism, which communicate far less.
CH 4 · pp. 93–115 Software The stack in rising abstraction: CUDA → PyTorch → inference engines → Dynamo, plus how to benchmark any of it honestly.
Key termskernel fusioncuBLAS · GEMMtorch.compilesafetensors vs. ONNXcontinuous batchingshadowingISL / OSL
Self-check
Why does kernel fusion matter more during decode than prefill?
Decode is memory-bandwidth-bound, and fusion's whole benefit is removing intermediate reads/writes to memory. In compute-bound prefill, saved memory traffic buys less.
Pick the engine: (a) brand-new open model, day one; (b) trillion-param MoE at high throughput; (c) squeeze maximum perf from H100s on a supported model.
(a) vLLM — broadest day-zero support. (b) SGLang — built for large-scale MoE serving. (c) TensorRT-LLM — best kernels, worth the extra engineering.
Your benchmark uses uniform 512-token prompts at constant rate. Name two ways it will mislead you.
No jitter/concurrency variation → unrealistic batching behavior; identical prompts → inflated prefix-cache hits and draft acceptance. Both overstate production performance.
CH 5 · pp. 117–151 Techniques The heart of the book: quantization, speculative decoding, caching, parallelism, and disaggregation — and when each applies.
Key termsFP8 · NVFP4 · microscalingscale factor · granularityEAGLE · n-gramacceptance rateprefix cachingG1–G4 tiersTP / EP / PPxPyD
Self-check
Rank by quantization risk: weights, attention/softmax, KV cache, activations.
Weights (safest) → activations → KV cache → attention/softmax (riskiest — errors compound across every subsequent token; nearly always left in original precision).
Why does speculative decoding stop helping at high batch sizes?
It spends spare compute to validate drafts. At high batch sizes decode's compute is already saturated by the batch, so there's nothing spare — engines dynamically disable speculation.
A RAG app puts the user's question first, then retrieved docs. What's wrong?
Prefix caching dies at the first differing token — the user question — so the (large, repeated) docs never hit cache. Put stable content first, novel tokens last.
Serving a 400B dense model across two nodes: which parallelism, and why not TP16?
TP8 inside each node + PP2 between nodes. TP needs per-layer all-reduce, which InfiniBand (≪ NVLink) can't sustain across nodes.
Your traffic is short prompts with high cache hit rates. Should you disaggregate?
No — condition three fails. Decode engines handle short/cached prefill fine locally; spend the extra GPUs on horizontal replicas instead.
CH 6 · pp. 153–176 Modalities Beyond text: vision, embeddings, speech in and out, image and video generation — mostly LLM tricks with modality-specific twists.
Key termsvision encoder · downsamplingMatryoshka embeddingsVAD · RTF · diarizationTTFB · SNACclassifier-free guidancecontext parallelism · ring attention
Self-check
Why is a 4-second video clip such a problem for a VLM?
24 fps × ~1,000 tokens/frame ≈ 100K input tokens — infeasible without aggressive resolution and frame-rate downsampling, and still a huge prefill/KV load.
Why do video models use Context Parallelism instead of Tensor Parallelism?
The models are small enough to replicate on every GPU; the bottleneck is the giant attention computation over latent space. CP splits that context (ring attention) rather than splitting weights.
Your TTS model can generate 300 TPS but audio plays at 90 TPS. What do you do with the headroom?
Serve more concurrent real-time streams per GPU (raise batch size / WebSocket slots) — faster-than-real-time generation per user has no value.
CH 7 · pp. 177–208 Production Getting optimized inference into the real world: containers, autoscaling, multi-cloud, deployment, observability, and client code.
Key termspinned dependenciescold startcontinuous batchingscale to zerocanary deploymentactive-activeTCOWebSockets vs. gRPC
Self-check
List the four components of a cold start and one fix for each.
GPU procurement (warm pools / contract terms); image load (smaller pinned images); weight load (quantized weights + in-datacenter caches); engine start (cache compiled engines, matched exactly to GPU + CUDA + deps).
Why canary instead of blue-green for large inference deployments?
Blue-green needs a full duplicate fleet (100 GPUs → another 100) before cutover. Canary shifts traffic gradually while autoscaling shrinks the old deployment, so total GPU overhead stays small.
Latency spiked but request volume is flat. Name two other metrics that could explain it.
Input/output sequence lengths (a few huge prompts saturate prefill) and queue depth / replica count (scale-up lag or a cordoned node shrinking capacity). Metrics must be read together.
APP A–B · pp. 209–255 Appendices: Glossary & Recommended Reading Reference material — use the glossary as a spaced-repetition source and the reading list as your stage-4-and-beyond queue.

Source: Inference Engineering by Philip Kiely (Baseten Books, © 2026, ISBN 979-8-9943597-2-3), from your PDF. Notes are condensed study aids for personal use alongside the book, not a substitute for it. Progress is stored only in this browser (localStorage) — clearing site data resets it.