Tag: inference
All the articles with the tag "inference".
-
Production Inference Optimization Strategy — Full Deck (All Deliverables)
The complete slide deck covering the production inference optimization strategy: architecture, benchmarks, routing, and cost analysis.
-
Architectural Specification — Production Inference Optimization Strategy
Architectural specification for a production inference optimization strategy for the agentic inference cloud.
-
Inference Optimization Mini-Benchmark: FP16 vs Q8_0 vs Q4_K_M
Hands-on benchmark of quantization levels on a 1.5B model: decode throughput, TPOT, and memory footprint compared across FP16, Q8_0, and Q4_K_M.
-
Serving the Agentic Inference Era — Optimization Strategy (Slides)
Slide deck on optimizing LLM inference for the agentic era: quantization, KV cache, batching, and bandwidth-bound decoding.