AboutNewsletter
Section · 34 stories

Deep Dives

Long-form analysis and technical deep dives

All Deep Dives

Newest first
Deep Dives

Multimodal Inference: Split the Encoder Only When It Pays

Encoder-prefill-decode disaggregation can reduce multimodal time to first token and increase same-SLO throughput, but only when vision encoding is a large enough share of request time. Teams should trace each serving stage, benchmark a workload matrix, optimize media preprocessing and caching, and promote only the traffic buckets that clear a reversible latency and cost gate.

Sep 10, 2026 · 8 min read
Deep Dives

Speculative Decoding: How LLMs Write Tokens 3x Faster

Speculative decoding accelerates LLM inference by having a cheap drafter propose several tokens that the target model verifies in one parallel forward pass, with a rejection-sampling step that makes the output distribution provably identical to the target model's. The landscape runs from separate draft models through Medusa (2.2-3.6x) and EAGLE-3 (up to 6.5x) to DeepSeek's co-trained MTP heads (85-90% second-token acceptance) and model-free n-gram lookup. The catch: speculation spends surplus compute to save memory bandwidth, so gains shrink as batch size rises and can go negative once the GPU becomes compute-bound.

Aug 5, 2026 · 10 min read
Deep Dives

FlashAttention: The IO-Aware Trick That Made Long Context Cheap

FlashAttention is an IO-aware, exact attention algorithm from 2022 that avoids writing the full N-by-N attention matrix to slow GPU HBM. Using tiling, an online-softmax running-statistics trick, kernel fusion, and recomputation, it cuts memory from O(N^2) to O(N) and delivered up to 7.6x speedups. FlashAttention-2 reached ~70% of A100 peak FLOPs; FlashAttention-3 (2024) exploits Hopper asynchrony and FP8 to hit ~840 TFLOPs BF16 (~75% H100 utilization). It now powers PyTorch, vLLM, and long-context serving.

Aug 1, 2026 · 9 min read
Deep Dives

RoPE: The Rotary Embeddings Behind Every Modern LLM

RoPE (Rotary Position Embeddings), introduced in the 2021 RoFormer paper, injects position into transformers by rotating query and key vectors so attention scores depend only on relative distance. It became the default across LLaMA, Mistral, Qwen and more. Because RoPE fails to extrapolate past its training length, methods like Position Interpolation, NTK-Aware scaling, and YaRN extend it to 128K-token context windows.

Jul 30, 2026 · 8 min read
Deep Dives

GRPO: The Critic-Free RL Algorithm Behind DeepSeek-R1

GRPO (Group Relative Policy Optimization) is a critic-free reinforcement learning algorithm introduced in the DeepSeekMath paper (arXiv 2402.03300). Instead of training a separate value model like PPO, it samples a group of responses per prompt and computes each response's advantage relative to the group's mean and standard deviation. It powered DeepSeek-R1's emergent reasoning and is the central baseline for reinforcement learning with verifiable rewards in 2026, spawning variants like Dr. GRPO, DAPO, and GSPO.

Jul 22, 2026 · 6 min read
Deep Dives

DPO: How Direct Preference Optimization Replaced RLHF

Direct Preference Optimization (DPO), introduced in a 2023 NeurIPS paper by Rafailov et al., aligns language models directly on preference pairs without training a separate reward model or running reinforcement learning. It replaces RLHF's fragile four-model PPO pipeline with a single supervised loss governed mainly by one parameter, beta, and works best stacked after SFT on subjective tasks — not on problems with a single correct answer.

Jul 13, 2026 · 9 min read
Deep Dives

Mixture of Experts: How Sparse Models Beat Dense LLMs

Mixture of Experts (MoE) replaces a transformer's single feed-forward network with many smaller expert networks plus a learned router that sends each token to only its top-k experts (sparse activation). This decouples total parameters (which set memory) from active parameters (which set compute). Mixtral 8x7B has 46.7B total but 12.9B active via top-2 routing; DeepSeek-V3 has 671B total but 37B active (5.5%) using 256 routed experts plus one shared expert and top-8 routing. The design traces to Shazeer et al. (2017) and Google's Switch Transformer (2021, top-1 routing, 1.6T params). Trade-offs include memory footprint, load-balancing difficulty, training instability, communication overhead, and harder fine-tuning.

Jul 10, 2026 · 6 min read
Deep Dives

LoRA and QLoRA: Fine-Tune Massive LLMs on a Single GPU

LoRA (2021) freezes a model's weights and trains tiny low-rank matrices, cutting GPT-3's trainable parameters 10,000x with no inference latency. QLoRA (2023) quantizes the frozen base to 4-bit NF4, fitting a 65B model on one 48GB GPU at ~33% less memory but ~39% more training time. Rank sets capacity; alpha (via alpha/r) sets scale. Adapt attention projections first and raise rank only when quality demands it.

Jul 3, 2026 · 8 min read
Deep Dives

LLM Quantization: GGUF vs AWQ vs GPTQ in 2026

A practical breakdown of the three dominant LLM quantization formats in 2026. GGUF is the portable, CPU-friendly default (use Q4_K_M); AWQ wins on 4-bit quality for GPU serving via activation-aware precision; GPTQ remains a solid NVIDIA-focused option. Quantization is lossy, so test on your real workload.

Jun 25, 2026 · 7 min read
Deep Dives

Model Collapse: Why AI Trained on AI Slowly Falls Apart

Model collapse is the progressive degradation of generative models trained recursively on synthetic data, documented in Nature (Shumailov et al., 2024). Errors compound and rare data vanishes, but research (Gerstgrasser et al., 2024) shows accumulating real data alongside synthetic data, tracking ratios, and verifying generations prevents it.

Jun 19, 2026 · 8 min read
Deep Dives

Test-Time Compute: Why Reasoning Models Think Before Answering

Test-time compute spends extra computation during inference, not training, to improve answers. It powers reasoning models like OpenAI o1 and DeepSeek-R1. Two strategies exist: sequential scaling (longer chains of thought, e.g. the s1 paper's budget forcing) and parallel scaling (Best-of-N, majority voting). More thinking is not always better, overthinking degrades accuracy, and hidden reasoning tokens are billable. Match compute to task difficulty.

Jun 17, 2026 · 8 min read
Deep Dives

Speculative Decoding: How a Tiny Draft Model Doubles LLM Speed

Speculative decoding speeds up LLM inference 2-6x by having a small draft model propose tokens that the target model verifies in parallel via rejection sampling, guaranteeing lossless output. EAGLE-3 and Medusa reduce or remove the separate draft model. Gains are largest at low batch sizes.

Jun 15, 2026 · 7 min read
Deep Dives

Diffusion LLMs: How Text Diffusion Is Challenging Autoregression

Diffusion language models (dLLMs) abandon left-to-right autoregressive generation, instead refining masked noise into text over a few parallel denoising steps. Inception Labs' Mercury Coder runs at 1,100+ tokens per second on H100s versus 50-200 for autoregressive models, and LLaDA 8B's bidirectional design breaks the reversal curse. They still trail the best models on hard reasoning benchmarks, but the one-token-at-a-time assumption is no longer a law of nature.

Jun 12, 2026 · 8 min read
Search TeqVolt

Find an article

Type a keyword or browse a section.