Topic · 5 stories

#inference

Reporting, analysis and practical guides connected to inference.

Latest in #inference

Newest first

Speculative Decoding: How LLMs Write Tokens 3x Faster

Speculative decoding accelerates LLM inference by having a cheap drafter propose several tokens that the target model verifies in one parallel forward pass, with a rejection-sampling step that makes the output distribution provably identical to the target model's. The landscape runs from separate draft models through Medusa (2.2-3.6x) and EAGLE-3 (up to 6.5x) to DeepSeek's co-trained MTP heads (85-90% second-token acceptance) and model-free n-gram lookup. The catch: speculation spends surplus compute to save memory bandwidth, so gains shrink as batch size rises and can go negative once the GPU becomes compute-bound.

Aug 5, 2026 · 10 min read

Deep Dives

FlashAttention: The IO-Aware Trick That Made Long Context Cheap

FlashAttention is an IO-aware, exact attention algorithm from 2022 that avoids writing the full N-by-N attention matrix to slow GPU HBM. Using tiling, an online-softmax running-statistics trick, kernel fusion, and recomputation, it cuts memory from O(N^2) to O(N) and delivered up to 7.6x speedups. FlashAttention-2 reached ~70% of A100 peak FLOPs; FlashAttention-3 (2024) exploits Hopper asynchrony and FP8 to hit ~840 TFLOPs BF16 (~75% H100 utilization). It now powers PyTorch, vLLM, and long-context serving.

Aug 1, 2026 · 9 min read

Deep Dives

RoPE: The Rotary Embeddings Behind Every Modern LLM

RoPE (Rotary Position Embeddings), introduced in the 2021 RoFormer paper, injects position into transformers by rotating query and key vectors so attention scores depend only on relative distance. It became the default across LLaMA, Mistral, Qwen and more. Because RoPE fails to extrapolate past its training length, methods like Position Interpolation, NTK-Aware scaling, and YaRN extend it to 128K-token context windows.

Jul 30, 2026 · 8 min read

AI News

Etched: The $5B Sohu Chip Betting the Transformer Never Dies

Etched, a startup building the transformer-only Sohu inference ASIC, has booked over $1 billion in contracts and reached a $5 billion valuation, with reports of new rounds valuing it up to $20 billion. Sohu hard-wires the transformer graph into silicon on TSMC N4P with 144GB HBM3E, and Etched claims an 8-chip server exceeds 500,000 Llama 70B tokens/sec. No independent benchmarks exist yet.

Jul 25, 2026 · 5 min read

Deep Dives

TurboQuant: Google's 6x KV Cache Compression Hits 3-Bit With Zero Loss

Google's TurboQuant compresses KV cache 6x at 3 bits with zero loss, speeding up attention.

May 11, 2026 · 5 min read