DigitalOcean — Agentic Inference Cloud
Extracting Maximum Value
from Every TFLOP
A production inference optimization strategy —
with a working proof-of-concept measured on real hardware.
3 pillars
Kernels & precision · Distributed execution · Resiliency & telemetry
2 PoCs
Quantization KPI benchmark · Cache-aware routing
3.2x
Multi-turn TTFT win, measured, from routing policy alone
Everything on these slides is either measured by me locally or cited from a production source. Physics first, then the three pillars, then what I'd build.
The framing
Decode is memory-bound. Everything follows from bytes-per-token.
- Generating one token requires streaming every weight byte from HBM. An H200 moves ~4.8 TB/s; MI300X ~5.3 TB/s with 192 GB capacity.
- Prefill processes thousands of tokens in one pass → compute-bound.
Decode processes one → bandwidth-bound.
- Every optimization is one of: move fewer bytes (quantization, sparse attention), amortize bytes (batching), or don't recompute (KV & prefix caching).
Measured proof (Qwen2.5-1.5B, llama.cpp/Metal): FP16→Q4 is 3.2x smaller and decodes 2.3x faster — but TTFT at a 2,000-word prompt is unchanged (0.43s vs 0.45s). Quantization accelerates the bandwidth-bound phase only.
Same roofline on the fleet, different constants. This asymmetry — prefill compute-bound, decode bandwidth-bound — is the root cause behind pillar 2's disaggregation.
Pillar 1
Kernel & precision engineering
| Model class | Precision policy | Why |
|---|
| 200B+ MoE (DeepSeek V3.2, Qwen 3.5) | FP8 W8A8 + FP8 KV; NVFP4 on B300 | Large + MoE models are most robust to low precision (~99% recovery) |
| Dense long-context reasoners | FP8 weights, FP8 KV, eval-gated FP4 | Reasoning chains degrade first under quantization — gate with task evals, not perplexity alone |
| Small draft models | INT4/Q4 weight-only | Drafts are verified by the target — errors cost acceptance rate, not correctness |
Kernels: FlashAttention-3/4-class attention (SRAM tiling, online softmax), fused MoE dispatch (DeepEP-style all-to-all), FP8 GEMMs per architecture. Heterogeneous fleet ⇒ maintain per-hardware artifacts: CUDA/Hopper, CUDA/Blackwell, ROCm/CDNA3 — same model, three builds, one CI eval gate.
Quality bar: every quantized artifact passes a regression gate (perplexity + task-specific evals) before any tenant sees it. My mini-benchmark ran the same gate: all three precisions scored identically (6/8) — misses were model-capability, not quantization.
FP8 is the default, FP4 is earned per-model via evals. KV-cache quantization matters as much as weights at long context.
Pillar 2
Distributed execution: place communication where bandwidth lives
| Strategy | Comm pattern | Placement rule (NVLink/xGMI in-node, 25G VPC across) |
|---|
| Tensor Parallel | All-reduce, every layer | Inside the node only (TP ≤ 8) |
| Expert Parallel | All-to-all, per MoE layer | Inside node; wide-EP only over RDMA fabric |
| Pipeline Parallel | Point-to-point activations | The cross-node axis — tolerates 25Gbps |
| Data Parallel | None (replicas) | Cross-node scale-out; needs a smart router |
Disaggregated prefill/decode: separate pools, each tuned for its bound (compute vs bandwidth); KV handoff over RDMA. Kills the ITL spikes that long prefills inflict on streaming users. Deploy where transfer bandwidth ≫ KV size.
Agentic multi-turn: prefix caching turns each turn's re-sent conversation into an incremental prefill (measured: 26–48x TTFT on cache hit) + continuous batching with chunked prefill for smooth ITL.
One sentence per row: the interconnect decides the parallelism, not the other way around. 200B MoE = EP+TP inside an 8x node, DP across nodes, PP only if it can't fit.
Pillar 3
Resiliency & observability
Cold start: make the bytes cheap or already there
- Hierarchy: node page cache → local NVMe → cluster peers (P2P/RDMA) → object storage, with streaming loaders (load-then-serve → serve-while-loading).
- Measured at mini scale: same model, cold disk load 7.4s vs warm page cache 0.6s — 12x from cache placement alone.
- Quantization is also a cold-start optimization: FP8 halves the bytes moved. 200GB over 25Gbps ≈ 80s; over 400G RDMA ≈ 5s.
Telemetry: percentiles or it didn't happen
- Per-request streaming timestamps → TTFT (queue+prefill), TPOT (decode speed), ITL distribution (jitter/stalls), rolled up at p50/p95/p99 per tenant, model, and hardware type.
- Business KPI: tokens/sec/dollar; engineering target: goodput — max throughput subject to SLOs (e.g. p99 TTFT < 500ms, p99 ITL < 50ms).
- Benchmark methodology: Poisson arrivals, realistic prompt/output mixes, never back-to-back load.
The metrics conflict — batching helps tokens/$/s but hurts ITL. Goodput-under-SLO is the objective function that resolves the conflict; it's also the right autoscaling signal.
Proof of concept — measured
Cache-aware routing: 3.2x mean, ~7x p95 TTFT — from policy alone
Setup: 2 replicas, 8 concurrent multi-turn agent sessions, toy router with two policies (poc_routing.py).
Round-robin bounces sessions between replicas → every turn re-prefills the whole growing conversation on a replica that never saw it.
Cache-aware pins a session to the replica holding its KV prefix.
110→34ms
turns 2+ mean TTFT
260→36ms
turns 2+ p95 TTFT
Industry validation: llm-d's Endpoint Picker (CNCF) is built on exactly this; Google's GKE Inference Gateway reported 35% TTFT and 2x p95 in production from cache-aware routing alone.
Once every engine has prefix caching, the load balancer becomes part of the inference stack. Note the tail effect is bigger than the mean — and SLOs are written against the tail. My 2-replica setup is round-robin's best case (hit rate ~1/N); production penalty is worse.
Strategy
2026 techniques mapped onto the AI-Native Cloud
| Where it lands | Technique (source) | Customer impact |
|---|
| Inference Router | Prefix-cache-aware endpoint picking (llm-d EPP pattern) | 3–7x multi-turn TTFT for agentic tenants — software-only, my PoC + Google's prod numbers |
| Serverless fleet | Cluster KV pool over the 400G RoCE fabric (Mooncake / LMCache tiering) | Any replica reuses any prefix; agent sessions survive rebalancing; higher effective batch |
| Inference Optimized Image | EAGLE-3 / MTP speculative decoding, FP8 KV defaults | 2–3x TPOT on chat/code; MTP heads come free with DeepSeek-class models |
| MoE serving (DeepSeek, Qwen 397B) | Wide-EP with DeepEP/EPLB-style dispatch + P/D disaggregation over RDMA | DeepSeek's own stack sustains 73.7k in / 14.8k out tok/s per H800 node — the margin exists |
| B300 (Blackwell Ultra) fleet | NVFP4 artifacts, eval-gated per model | ~3.5x weight memory vs FP16 at ~99% recovery on large/MoE models → more models per GPU |
| Scale-to-zero serverless | Hierarchical weight caching + streaming loaders | Cold start from minutes to seconds; directly answers the 100GB+ model problem |
All additive to what exists today — Router preview, RoCE fabric, and the optimized image are the natural landing spots. Sequenced: routing first (weeks, pure software), speculative decoding + FP8 KV next (a quarter), KV pooling + disaggregation after (needs the fabric), NVFP4 rides the B300 ramp.
Close
What I'd do first, and what it's worth
Phase 1
Cache-aware routing — weeks, software-only, 3–7x multi-turn TTFT (proven twice: my PoC, Google prod)
Phase 2
Spec decode + FP8 KV defaults — one quarter, 2–3x TPOT, no quality loss (verify-based)
Phase 3
KV pooling + P/D disaggregation — 2–3 quarters, tail-latency SLOs at high utilization
Phase 4
NVFP4 + sparse attention — with B300 ramp, next 2–4x cost curve
The through-line: every phase attacks bytes-per-token or bytes-not-recomputed, measured by the same four KPIs — TTFT, TPOT, ITL, tokens/sec/dollar — and gated by the same quality bar. Nothing here is speculative: each item ships today in llm-d, vLLM, SGLang, or DeepSeek's open stack.
Appendix (→): demo architecture & rejected alternatives · five failure-mode walkthroughs · throughput math & 10x plan · benchmark detail · references.
End on the roadmap. Q&A from here — appendix slides are ready for the code-review questions.
Appendix A1 — demo repo
Tech choices & the alternatives we rejected
| Choice | What we use | Rejected alternative | Why |
|---|
| Inference engine | llama.cpp (llama-server, Metal) | vLLM · Ollama | vLLM has no Mac GPU path; Ollama hides the flags the experiments need (cache_prompt, slots). On the fleet: vLLM/SGLang/TensorRT-LLM. |
| Framework | Python stdlib only (urllib, subprocess) | FastAPI + aiohttp + locust | Zero dependencies = runs anywhere in the interview; the harness is the product, not the server. |
| Storage | Flat results.json / md files | SQLite / Postgres | KBs of results, versionable and diffable in git. A DB adds ops surface for zero queries. |
| Queue | llama-server's in-process slot queue (1 slot) | Redis / Celery | Benchmark is deliberately sequential — a queue would add noise to latency measurements. Production gateway needs real admission control (A3). |
| Deploy target | Local Apple Silicon | Docker on GPU Droplet | Must run offline on a laptop mid-interview. Same methodology transfers to DOKS + GPU Droplets unchanged. |
Every choice optimizes for reproducibility-in-the-room over production fidelity — deliberately, and I can name the production substitute for each.
Appendix A2 — failure modes 1–3 (bench.py)
Crash, restart, duplicate
1 — Worker (llama-server) crashes mid-job
Current: the stream read in stream_completion() raises; finally: (bench.py:205) stops the server — but main() builds all results in one list comprehension (:213) and writes JSON only at the very end (:215). Risk: a crash at the last quant loses the entire run. Fix: json.dump incrementally after each bench_quant() (checkpoint per unit of work).
2 — Process restarts: what state is lost?
Current: all results live in Python memory until the final write — total loss. Worse: the llama-server child is orphaned and keeps port 8933; the next run's health poll (:69) gets 200 OK from the old server and benchmarks the wrong model silently. Risk: silent wrong-model results. Fix: atexit handler to kill the child + assert the served model's identity via /props before benchmarking.
3 — Duplicate submission
Current: the harness is sequential so it never duplicates; but a duplicated prompt would hit the server's slot cache and measure a warm prefill as cold — which is exactly why every cold measurement carries a nonce (:159, :186). Server-side there is no idempotency: a retried POST generates (and would bill) twice. Risk: skewed data locally; double-spend in production. Fix: idempotency keys at the gateway; keep the nonce discipline in the harness.
The orphan-server case is the sneaky one — the health check passes for the wrong reason. Identity-check what you benchmark.
Appendix A3 — failure modes 4–5 + backpressure
Burst traffic and the job that never finishes
4 — Queue full under burst
Current: the server is launched without --parallel (bench.py:57–64) → one slot. Concurrent requests queue inside llama-server; each client waits up to the 600s socket timeout (:109). No 429, no shedding — head-of-line blocking, latency grows unboundedly, then a timeout avalanche. Risk: burst turns into a total stall with zero signal to callers. Fix: --parallel N + gateway admission control that returns 429 with Retry-After when queue depth > threshold.
5 — A job that never finishes
Current: two hazards. (a) The health poll is while True (:69) with no deadline — a server that's alive but never becomes healthy (bad model file, VRAM exhaustion) loops forever. (b) The 600s timeout is per-socket-read: a stream that trickles one token per 599s never times out; only max_tokens bounds it. Risk: a hung run that looks "in progress" indefinitely. Fix: deadline on the health poll (~120s) and a wall-clock budget per request enforced in the client.
Production translation: these five are exactly queue-depth SLOs, checkpointed state, idempotency keys, admission control, and request deadlines — the standard reliability kit, discovered bottom-up from 220 lines of Python.
The point isn't that a demo script has gaps — it's that I know precisely where they are and what each costs.
Appendix A4 — capacity
Throughput math from actual config, and the 10x path
Today (measured constants, 1 slot, ctx 8192)
- Max concurrent jobs: 1. Everything else queues (no
--parallel).
- 256-token response ≈ prefill 0.05–0.45s + decode 256/176.5 ≈ 1.45s (Q4) → ~1.5–1.9s per request → ~0.5–0.7 req/s. FP16: 256/75.8 ≈ 3.4s → ~0.3 req/s.
- Health-poll sleep 0.1s (:78) adds ≤100ms per server start — noise.
- First bottleneck: single-slot serialization — not CPU, not RAM. At batch 1 the GPU streams 1.1–3.6GB of weights per token for one stream; bandwidth amortizes across zero neighbors.
Scaling to 10x — what changes first
- First:
--parallel 8 + continuous batching. Decode steps share each weight pass; throughput ≈ 5–8x for a mild ITL cost. Free until KV memory binds: 8 slots × 8192 ctx × ~110MB ≈ 0.9GB — fits.
- Then: checkpoint results + admission control (from A2/A3) so the extra load is survivable, not just fast.
- Then: replicas behind the cache-aware router — the PoC already proves the routing layer; linear scale-out with preserved cache hit rates.
- Bottleneck order: slot count → KV memory → memory bandwidth → only then compute.
Rough math, real config values. The answer generalizes: on the fleet the same order holds — batching first, memory second, silicon last.
Appendix A5 — benchmark detail
The measured trade-off surface
Quality gate: 6/8 exact-match at all three precisions (same two misses — capability, not quantization). Cold start: 7.4s cold disk vs 0.6s warm page cache (12x). Full data: demo/results/results.json, routing.json.
Left: TTFT grows with prompt length and is quantization-independent — prefill is compute-bound. Right: prefix caching collapses TTFT 26–48x — the single biggest agentic-workload lever.
Appendix A6
Selected references
Systems canon
- Orca (OSDI '22) — continuous batching
- vLLM / PagedAttention (SOSP '23) — arXiv:2309.06180
- DistServe (OSDI '24) — P/D disaggregation, goodput — arXiv:2401.09670
- Sarathi-Serve (OSDI '24) — chunked prefill — arXiv:2403.02310
- Mooncake (FAST '25 Best Paper) — KV-centric serving — arXiv:2407.00079
- FlashAttention 1–4 — arXiv:2205.14135 · together.ai/blog/flashattention-4
2026 techniques used in this deck
- llm-d (CNCF) — cache-aware routing — github.com/llm-d
- GKE Inference Gateway — 35% TTFT / 2x p95 in prod — Google Cloud blog
- DeepSeek open-infra — FlashMLA, DeepEP, EPLB, 545% margin disclosure
- NVFP4 — NVIDIA dev blog; ~99% recovery at 70B+ (Red Hat)
- EAGLE-3 — arXiv:2503.01840 · DSA sparse attention — arXiv:2512.02556
- LMCache + Mooncake Store — vLLM cluster KV pooling (May 2026)
Full annotated bibliography: inference-trends-2026.md §6.
← → to navigate · Home/End to jump