Tag
quantization
3 articles
LoRA and QLoRA: Fine-Tune Massive LLMs on a Single GPU
LoRA (2021) freezes a model's weights and trains tiny low-rank matrices, cutting GPT-3's trainable parameters 10,000x with no inference latency. QLoRA (2023) quantizes the frozen base to 4-bit NF4, fitting a 65B model on one 48GB GPU at ~33% less memory but ~39% more training time. Rank sets capacity; alpha (via alpha/r) sets scale. Adapt attention projections first and raise rank only when quality demands it.
By Aisha Patel · 8 min · Jul 3, 2026
LLM Quantization: GGUF vs AWQ vs GPTQ in 2026
A practical breakdown of the three dominant LLM quantization formats in 2026. GGUF is the portable, CPU-friendly default (use Q4_K_M); AWQ wins on 4-bit quality for GPU serving via activation-aware precision; GPTQ remains a solid NVIDIA-focused option. Quantization is lossy, so test on your real workload.
By Aisha Patel · 7 min · Jun 25, 2026
TurboQuant: Google's 6x KV Cache Compression Hits 3-Bit With Zero Loss
Google's TurboQuant compresses KV cache 6x at 3 bits with zero loss, speeding up attention.
By Aisha Patel · 5 min · May 11, 2026