Open Source
Marin 535B: The Training Run Is the Real Open-Source Release
Marin’s 535B run shows how public scaling ladders and failure logs can make frontier-model training auditable before weights exist.
Topic · 48 stories
#llm
Reporting, analysis and practical guides connected to llm.
Latest in #llm
Newest first
Tech Tips
Promptfoo: Test Your LLM Prompts Like Unit Tests
Promptfoo is an open-source CLI (24.1k GitHub stars) that treats LLM prompts as testable code using YAML test cases and assertions. OpenAI announced it was acquiring the company on March 9, 2026, committing to continue the open-source project. This guide covers installation (Node.js 22.22.0 or newer, Node 24 recommended), scaffolding a first eval with npx promptfoo@latest init, choosing assertion types by cost tier, guarding cost and latency, RAG and agent evals, CI gating strategy, the red teaming half of the tool, and three honest limits of eval suites.
Open Source
SpeakoFlow: The Open-Source Voice Assistant That Stays Local
SpeakoFlow is a free, MIT-licensed local-first voice assistant for Windows, macOS and Linux, built by solo developer Abhishek Barali as a fork of CJ Pais's Handy. It combines hotkey dictation, a 'Hey Flow' generative writing mode, an assistant panel with screen vision, live translation and AI cleanup. Speech-to-text always runs on-device via whisper.cpp and Parakeet; the assistant can run fully offline through a built-in llama.cpp engine, through Ollama or LM Studio, or through any OpenAI-compatible cloud provider with your own key. There is no account and no telemetry. The trade-offs are real: the binaries are not code-signed on Windows or macOS, requiring a manual quarantine-clearing step on Mac, and the project is at v1.0.1 with 126 commits, six GitHub stars and one maintainer.
AI News
DeepSeek V4 Flash 0731: Frontier Agent Work at $0.14
DeepSeek upgraded its deepseek-v4-flash API to the 0731 public beta on July 31, 2026 — an API-only post-training update that leaves the 284B/13B MoE architecture, 1M context window and $0.14/$0.28 pricing untouched. Artificial Analysis measures a 10-point Intelligence Index jump to 50 and a GDPval-AA v2 rise from 1189 to 1559 Elo, with Cost per Task roughly 60% below GPT-5.6 Luna. Accuracy on AA-Omniscience is unchanged at 37%, and the 0731 weights are not open — only the April 24 checkpoint is on Hugging Face under MIT.
Deep Dives
Speculative Decoding: How LLMs Write Tokens 3x Faster
Speculative decoding accelerates LLM inference by having a cheap drafter propose several tokens that the target model verifies in one parallel forward pass, with a rejection-sampling step that makes the output distribution provably identical to the target model's. The landscape runs from separate draft models through Medusa (2.2-3.6x) and EAGLE-3 (up to 6.5x) to DeepSeek's co-trained MTP heads (85-90% second-token acceptance) and model-free n-gram lookup. The catch: speculation spends surplus compute to save memory bandwidth, so gains shrink as batch size rises and can go negative once the GPU becomes compute-bound.
Deep Dives
FlashAttention: The IO-Aware Trick That Made Long Context Cheap
FlashAttention is an IO-aware, exact attention algorithm from 2022 that avoids writing the full N-by-N attention matrix to slow GPU HBM. Using tiling, an online-softmax running-statistics trick, kernel fusion, and recomputation, it cuts memory from O(N^2) to O(N) and delivered up to 7.6x speedups. FlashAttention-2 reached ~70% of A100 peak FLOPs; FlashAttention-3 (2024) exploits Hopper asynchrony and FP8 to hit ~840 TFLOPs BF16 (~75% H100 utilization). It now powers PyTorch, vLLM, and long-context serving.
Open Source
Meetily: The Open-Source AI Notetaker That Runs 100% Local
Meetily is a privacy-first, open-source AI meeting assistant with 27.4K GitHub stars and an MIT license. Built on Rust and Tauri, it runs Whisper or Parakeet transcription and Ollama summarization entirely on your own device, so meeting audio never touches the cloud. It supports macOS and Windows, with flexible summary providers and a commercial PRO tier.
Deep Dives
RoPE: The Rotary Embeddings Behind Every Modern LLM
RoPE (Rotary Position Embeddings), introduced in the 2021 RoFormer paper, injects position into transformers by rotating query and key vectors so attention scores depend only on relative distance. It became the default across LLaMA, Mistral, Qwen and more. Because RoPE fails to extrapolate past its training length, methods like Position Interpolation, NTK-Aware scaling, and YaRN extend it to 128K-token context windows.
Deep Dives
Mamba: The State Space Models Challenging the Transformer
Mamba's selective state space models scale linearly and rival Transformers, and 2026's frontier models increasingly blend the two into hybrids.
Open Source
OpenClaw: The 383K-Star AI Agent With a Security Problem
OpenClaw is a free, self-hosted, model-agnostic AI agent that runs as a persistent background daemon and acts across WhatsApp, Telegram, Slack, and Discord. It became the fastest-growing repo in GitHub history (383K+ stars) but carries serious security flaws: authentication off by default, plaintext credential storage, tens of thousands of internet-exposed instances, and fake installers spreading infostealer malware. Run it only from the official repo, behind a VPN, with auth on and scoped credentials.
Deep Dives
GRPO: The Critic-Free RL Algorithm Behind DeepSeek-R1
GRPO (Group Relative Policy Optimization) is a critic-free reinforcement learning algorithm introduced in the DeepSeekMath paper (arXiv 2402.03300). Instead of training a separate value model like PPO, it samples a group of responses per prompt and computes each response's advantage relative to the group's mean and standard deviation. It powered DeepSeek-R1's emergent reasoning and is the central baseline for reinforcement learning with verifiable rewards in 2026, spawning variants like Dr. GRPO, DAPO, and GSPO.
Tech Tips
LiteLLM: One Unified API for Every LLM Provider in 2026
LiteLLM is an open-source gateway that gives developers a single OpenAI-format interface to call 100+ LLM providers. This tutorial covers installing the SDK and Proxy Server, switching providers by changing a model string, unified exception handling, streaming, and adding cost tracking, observability, virtual keys, and budgets.
Tech Tips
Langfuse: LLM Observability That Debugs Your AI Agents
Langfuse is an open-source, MIT-licensed LLM observability platform acquired by ClickHouse in January 2026. It provides hierarchical tracing, prompt management, evaluations, and datasets. Its OpenTelemetry-based Python SDK v3 uses the @observe decorator and integrates with LangChain, the OpenAI SDK, Anthropic, and LiteLLM.
AI News
Cognition SWE-1.7: Near-Frontier Coding at $2 a Task
Cognition released SWE-1.7 on July 8, 2026, a software-engineering model built by reinforcement-learning on top of Moonshot AI's Kimi K2.7 base and served through Cerebras at ~1,000 tokens/second inside the Devin agent. It scores 42.3% on FrontierCode 1.1 and 81.5% on Terminal-Bench 2.1, trailing Opus 4.8 by a few points at roughly $1.97 per task, positioning it as a near-frontier option at a fraction of frontier cost.
Deep Dives
DPO: How Direct Preference Optimization Replaced RLHF
Direct Preference Optimization (DPO), introduced in a 2023 NeurIPS paper by Rafailov et al., aligns language models directly on preference pairs without training a separate reward model or running reinforcement learning. It replaces RLHF's fragile four-model PPO pipeline with a single supervised loss governed mainly by one parameter, beta, and works best stacked after SFT on subjective tasks — not on problems with a single correct answer.
AI News
GPT-5.6: OpenAI's Sol, Terra, and Luna Go Public
OpenAI made its three-tier GPT-5.6 family (Sol, Terra, Luna) generally available on July 9, 2026 after government safety review. Pricing runs from Luna at $1/$6 to Sol at $5/$30 per 1M tokens, with a Sol Fast option at $12.50/$75 on Cerebras. The release adds Programmatic Tool Calling in the Responses API (63.5% fewer tokens, 50.1% fewer turns) and longer prompt caching, but Sol's 64.6% on SWE-Bench Pro still trails Claude Mythos 5 (80.3%).
Tech Tips
Unsloth: Fine-Tune LLMs 2x Faster on a Single GPU
Unsloth is an open-source library that fine-tunes open LLMs (Llama, Qwen, Mistral, Gemma, gpt-oss) roughly 2x faster and with up to 70% less VRAM than a stock Hugging Face setup, without sacrificing accuracy. It achieves this with custom OpenAI Triton kernels and a manual backpropagation engine, and fuses LoRA with 4-bit quantization. It runs on any NVIDIA GPU with CUDA Capability 7.0+, including the free Colab T4. Install with 'pip install unsloth' and use FastLanguageModel.from_pretrained plus get_peft_model to attach LoRA adapters before training with trl's SFTTrainer.
Deep Dives
Mixture of Experts: How Sparse Models Beat Dense LLMs
Mixture of Experts (MoE) replaces a transformer's single feed-forward network with many smaller expert networks plus a learned router that sends each token to only its top-k experts (sparse activation). This decouples total parameters (which set memory) from active parameters (which set compute). Mixtral 8x7B has 46.7B total but 12.9B active via top-2 routing; DeepSeek-V3 has 671B total but 37B active (5.5%) using 256 routed experts plus one shared expert and top-8 routing. The design traces to Shazeer et al. (2017) and Google's Switch Transformer (2021, top-1 routing, 1.6T params). Trade-offs include memory footprint, load-balancing difficulty, training instability, communication overhead, and harder fine-tuning.
AI News
GPT-Realtime-2.1: OpenAI Adds Reasoning to Its Voice API
On July 6, 2026, OpenAI released GPT-Realtime-2.1 and GPT-Realtime-2.1-mini for the Realtime API. The headline change is reasoning in the low-cost mini tier, plus a 25% cut in p95 latency from better caching. The mini holds the prior gpt-realtime-mini price (0 audio in, 0 audio out per 1M) while the full model runs 2/4. Reasoning effort is configurable from minimal to xhigh.
Ethics & AI
AI Hallucinations in Court: 1,725 Cases and a $110K Wake-Up Call
AI hallucinations in court filings have grown from the 2023 Mata v. Avianca case (a $5,000 sanction for six fabricated ChatGPT citations) into a documented worldwide phenomenon. Damien Charlotin's database catalogs 1,725 cases as of July 5, 2026, led by the US (1,187), Canada (190), and Australia (96). Self-represented litigants account for 1,016 cases, lawyers 667. In December 2025, an Oregon federal judge imposed a record $110,000 penalty in Couvrette v. Wisnovsky for 15 fake cases and 8 fabricated quotations. At least 25 federal courts now require AI-use certifications.
Deep Dives
LoRA and QLoRA: Fine-Tune Massive LLMs on a Single GPU
LoRA (2021) freezes a model's weights and trains tiny low-rank matrices, cutting GPT-3's trainable parameters 10,000x with no inference latency. QLoRA (2023) quantizes the frozen base to 4-bit NF4, fitting a 65B model on one 48GB GPU at ~33% less memory but ~39% more training time. Rank sets capacity; alpha (via alpha/r) sets scale. Adapt attention projections first and raise rank only when quality demands it.
Tech Tips
DSPy: Program Your LLMs Instead of Prompting Them
DSPy is a Stanford NLP Python framework (v3.3, MIT-licensed, 6.4M+ monthly downloads) for programming LLMs instead of hand-writing prompts. You declare tasks as typed signatures, compose them as modules like Predict/ChainOfThought/ReAct, define a metric, then run optimizers such as GEPA or MIPROv2 to auto-tune prompts — often lifting a baseline from ~62% to ~89% on the same model. Used in production by Shopify, Databricks, Dropbox, and Replit.
Tech Tips
vLLM: Serve LLMs 24x Faster Than Hugging Face Transformers
vLLM is the default open-source LLM serving engine in 2026. PagedAttention cuts KV-cache memory waste from 60-80% to under 4%, and continuous batching keeps the GPU full, together delivering 14-24x the throughput of Hugging Face Transformers. Install with pip, launch an OpenAI-compatible server via 'vllm serve', then tune --gpu-memory-utilization, --max-num-batched-tokens, --tensor-parallel-size, and chunked prefill against real traffic.
AI News
Grok 4.3: xAI's Frontier Model Hits Amazon Bedrock
Grok 4.3 is generally available on Amazon Bedrock with a 1M-token context window, $1.25/$2.50 pricing, and a top hallucination-rate score.
AI News
GLM-5.2: Zhipu's Open-Weight Model Beats GPT-5.5 at 1/6 the Cost
Z.AI released GLM-5.2 on June 16, 2026: a 753B-parameter MoE model under an MIT license with a 1M-token context. It tops open-weight coding benchmarks, beating GPT-5.5 on SWE-bench Pro, FrontierSWE and PostTrainBench at roughly one-sixth the cost.
Deep Dives
LLM Quantization: GGUF vs AWQ vs GPTQ in 2026
A practical breakdown of the three dominant LLM quantization formats in 2026. GGUF is the portable, CPU-friendly default (use Q4_K_M); AWQ wins on 4-bit quality for GPU serving via activation-aware precision; GPTQ remains a solid NVIDIA-focused option. Quantization is lossy, so test on your real workload.
Tech Tips
Ollama: Run Local LLMs Like a Pro in 2026
A hands-on guide to Ollama, the default local-LLM runner in 2026 (v0.30.10). Covers install, pulling and running models, calling them from the OpenAI SDK at localhost:11434, structured JSON outputs, tool calling, and Modelfiles, plus how to size a model to your hardware.
Reviews
OpenCode: The Open-Source AI Coding Agent at 178K Stars
OpenCode is an open-source (MIT), terminal-native AI coding agent with 178K GitHub stars. It is model-agnostic, connecting to 75+ providers (Anthropic, OpenAI, Google, Ollama) with bring-your-own keys. LSP integration feeds compiler diagnostics back to the model; built-in build and plan agents plus a general subagent. Runs locally/air-gapped, ships frequently (v1.17.9, 826 releases), and now has a desktop beta. Trade-offs: a terminal learning curve, you pay your own API bills, and quality depends on the model you plug in.
Tech Tips
Structured Outputs: Force LLMs to Return Valid JSON
A practical guide to OpenAI Structured Outputs: the difference from JSON mode, function calling vs response_format, strict schema rules, constrained decoding, limits, and cross-provider options.
Deep Dives
KV Cache: The Memory Trick Behind Fast LLM Inference
A deep dive into the KV cache in LLM inference: why autoregressive decoding needs it, how it dominates GPU memory, the 60-80% waste of contiguous allocation, and how vLLM's PagedAttention fixed it.
Deep Dives
Model Collapse: Why AI Trained on AI Slowly Falls Apart
Model collapse is the progressive degradation of generative models trained recursively on synthetic data, documented in Nature (Shumailov et al., 2024). Errors compound and rare data vanishes, but research (Gerstgrasser et al., 2024) shows accumulating real data alongside synthetic data, tracking ratios, and verifying generations prevents it.
Deep Dives
Test-Time Compute: Why Reasoning Models Think Before Answering
Test-time compute spends extra computation during inference, not training, to improve answers. It powers reasoning models like OpenAI o1 and DeepSeek-R1. Two strategies exist: sequential scaling (longer chains of thought, e.g. the s1 paper's budget forcing) and parallel scaling (Best-of-N, majority voting). More thinking is not always better, overthinking degrades accuracy, and hidden reasoning tokens are billable. Match compute to task difficulty.
AI News
MiniMax M3: Open-Weight Frontier Coding Model With 1M Context
MiniMax M3 is an open-weight model pairing a 1M-token context and revived sparse attention with frontier coding benchmarks at 15x lower cost than Claude Opus 4.7.
Tech Tips
Context Engineering: A Practical Playbook for Reliable AI Agents
Context engineering is the discipline of curating tools, prompts, retrieval, and memory each turn so AI agents stay reliable over long-horizon tasks.
Deep Dives
Speculative Decoding: How a Tiny Draft Model Doubles LLM Speed
Speculative decoding speeds up LLM inference 2-6x by having a small draft model propose tokens that the target model verifies in parallel via rejection sampling, guaranteeing lossless output. EAGLE-3 and Medusa reduce or remove the separate draft model. Gains are largest at low batch sizes.
Deep Dives
Diffusion LLMs: How Text Diffusion Is Challenging Autoregression
Diffusion language models (dLLMs) abandon left-to-right autoregressive generation, instead refining masked noise into text over a few parallel denoising steps. Inception Labs' Mercury Coder runs at 1,100+ tokens per second on H100s versus 50-200 for autoregressive models, and LLaDA 8B's bidirectional design breaks the reversal curse. They still trail the best models on hard reasoning benchmarks, but the one-token-at-a-time assumption is no longer a law of nature.
Tech Tips
Prompt Caching: How to Cut LLM API Costs by Up to 90%
Prompt caching stores the computed KV attention tensors for a repeated prompt prefix so the model skips recomputation, cutting input cost and latency. Anthropic (explicit cache_control, ~90% read discount), OpenAI (automatic, 50% off, 1,024-token minimum), and Google Gemini (implicit plus explicit cache objects, up to 90%) all support it. The one rule that determines hit rate: put all static content at the front of the prompt and all dynamic content at the back.
AI News
Gemma 4 12B: Google's Encoder-Free Multimodal Laptop Model
Google released Gemma 4 12B on June 3, 2026, a multimodal open model with an encoder-free architecture that feeds vision and audio directly into the LLM backbone. It runs locally on 16GB of memory, approaches the 26B MoE on benchmarks, uses Multi-Token Prediction drafters for low latency, and ships under Apache 2.0 with broad tooling support.
Tech Tips
RAG Grounding: 7 Ways to Stop LLM Hallucinations in Production
A practitioner's guide to grounding retrieval-augmented generation systems. Covers fixing retrieval first, hybrid dense-plus-keyword search, cross-encoder reranking, contextual compression, refusal prompting, verified citations, Chain-of-Verification, confidence-threshold abstention, and measuring faithfulness with RAGAS.
AI News
DeepSeek V4-Pro: 75% Price Cut Becomes Permanent
On May 22, 2026, DeepSeek made its 75% promotional discount on V4-Pro permanent rather than letting it expire May 31. New permanent rates: $0.435/M input, $0.87/M output, $0.003625/M cache hit. That puts V4-Pro output roughly 34x cheaper than GPT-5.5 and 17x cheaper than Claude Opus 4.7, while landing within 3-7 points on coding and reasoning benchmarks. The underrated detail is the cache-hit price, which can cut input cost ~88% for agents with stable prefixes. Teams should re-run their build math and route the easy majority of traffic to V4-Pro.
AI News
SubQ: The 12M-Token Subquadratic LLM Splitting AI Researchers
SubQ is a new 12M-token subquadratic LLM claiming massive context and low compute, sparking debate among researchers.
Deep Dives
TurboQuant: Google's 6x KV Cache Compression Hits 3-Bit With Zero Loss
Google's TurboQuant compresses KV cache 6x at 3 bits with zero loss, speeding up attention.
AI News
DeepSeek V4 Pro: 1.6T Open-Weights Model Hits #2 on the Index
DeepSeek V4 Pro is a top 1.6T open-weights model for agents, but has a high hallucination rate.
AI News
Claude Opus 4.7: Anthropic's New Flagship Clears SWE-Bench Pro
Anthropic's Claude Opus 4.7 excels on SWE-bench Pro with enhanced vision and new features.
AI News
Qwen 3.6 Plus: Alibaba's Free Preview Beats Claude Opus on Agent Tasks
Alibaba's Qwen 3.6 Plus Preview surpasses Claude Opus on agent tasks with impressive speed and context.
Tech Tips
Caveman: The Claude Code Skill That Cuts 65% of Output Tokens
Caveman, a Claude Code skill, dramatically cuts AI output tokens by 65%, optimizing agent interactions.
Tech Tips
Edgee Codex Compressor: The Rust Gateway That Cuts Codex Costs 35.6%
Edgee Codex Compressor, a Rust gateway, cuts LLM costs by 35.6% by compressing tool output.
AI News
GPT-5.4: OpenAI's Five-Variant Strategy Reshapes the AI Market
OpenAI's GPT-5.4, with five variants and expert-level computer use, is reshaping the AI market.
