Microsoft has been quietly assembling something rare in 2026: a credible, open-source, frontier-class voice AI stack. The umbrella project is VibeVoice, hosted at microsoft/VibeVoice on GitHub under an MIT license, and it has just crossed 33,000 stars with 3,700 forks — numbers that put it ahead of nearly every other voice model release this year.
But VibeVoice isn't one model. It's a family of three, each tuned for a different problem in the speech pipeline. And one of them was so capable that Microsoft pulled the code, re-thought the threat model, and shipped only what could be defended in production.
The three models
The project ships under three model cards, each available on Hugging Face:
| Model | Size | Purpose | Status |
|---|---|---|---|
| VibeVoice-ASR | 7B | 60-minute long-form speech recognition with diarization | Active, on HF |
| VibeVoice-TTS | 1.5B | 90-minute long-form multi-speaker TTS | Code removed; weights restricted |
| VibeVoice-Realtime | 0.5B | ~300 ms first-audible-latency streaming TTS | Active, on HF and Colab |
The architectural through-line connecting all three is unusual. VibeVoice runs on continuous speech tokenizers — both Acoustic and Semantic — operating at an ultra-low frame rate of 7.5 Hz. That's roughly an order of magnitude lower than what most speech models use, and it's the trick that makes 60- and 90-minute single-pass inference computationally feasible. On top of those tokenizers, VibeVoice uses a next-token diffusion framework: an LLM (the Qwen2.5 1.5B base) handles textual context and dialogue flow, while a diffusion head produces the high-fidelity acoustics.
VibeVoice-ASR: the headline model
The ASR model is where VibeVoice's claim to "frontier" carries the most weight. VibeVoice-ASR-7B processes up to 60 minutes of continuous audio in a single pass within a 64K-token context — meaning no chunking, no lost speaker context, no semantic drift across an hour-long meeting.
Its output isn't a flat transcript. The model jointly performs ASR, speaker diarization, and timestamping, returning structured Who / When / What records. You can also pass customized hotwords — proper nouns, domain terms, internal codenames — and they're respected during decoding, which is a real problem for off-the-shelf Whisper deployments handling enterprise content.
VibeVoice-ASR went open-source on January 21, 2026, supports 50+ languages natively, and was merged into Hugging Face Transformers v5.3.0 on March 6, 2026 — so you can now load it the same way you'd load any other audio model in the ecosystem.
# After installing transformers >= 5.3.0
from transformers import AutoProcessor, VibeVoiceAsrForConditionalGeneration
model_id = "microsoft/VibeVoice-ASR-HF"
processor = AutoProcessor.from_pretrained(model_id)
model = VibeVoiceAsrForConditionalGeneration.from_pretrained(model_id, device_map="auto")
Microsoft has also published finetuning code and vLLM inference support for the ASR model — a meaningful concession to teams who need throughput, not just a demo.
The ASR technical report is available on arXiv (2601.18184), and there's a public playground at aka.ms/vibevoice-asr for quick evaluation without a local install.
VibeVoice-Realtime: streaming under 300 ms
The smallest and most deployment-friendly model in the family is VibeVoice-Realtime-0.5B, open-sourced on December 3, 2025. Its design priorities are different: instead of long-form generation, it's optimized for streaming text input with a first-audible-latency around 300 milliseconds — fast enough for live agents, voice copilots, or any system where waiting two seconds to speak would feel broken.
It still handles long-form output (~10 minutes per session), and the December 2025 update added experimental speakers in nine non-English languages — DE, FR, IT, JP, KR, NL, PL, PT, ES — plus 11 distinct English style voices. There's a Colab demo linked from the repository for anyone who wants to hear it before installing.
The story of the missing TTS model
The most interesting paragraph in the README isn't about a feature — it's about a withdrawal:
"VibeVoice is an open-source research framework intended to advance collaboration in the speech synthesis community. After release, we discovered instances where the tool was used in ways inconsistent with the stated intent. Since responsible use of AI is one of Microsoft's guiding principles, we have removed the VibeVoice-TTS code from this repository." — Microsoft, September 5, 2025
VibeVoice-TTS-1.5B was originally open-sourced on August 25, 2025 and accepted as an Oral at ICLR 2026. It can synthesize up to 90 minutes of conversational speech with up to 4 distinct speakers in a single pass — a benchmark very few proprietary systems can match. Within weeks of release, it was being used in ways Microsoft wasn't comfortable with, and the company pulled the code. The "Quick Try" column on the model card now reads simply: Disabled.
This is a legitimately interesting precedent. Most labs ship TTS models, watch deepfakes proliferate, and shrug. Microsoft pulled the rug — at the cost of a freshly published ICLR paper — and committed instead to ASR and a smaller real-time model where the abuse vectors are narrower. Whether you see this as principled or paternalistic depends on your priors, but it's an unusual move worth naming.
How the architecture actually works
The 7.5 Hz continuous tokenizer is the technical detail that makes everything else possible. Most speech codecs operate at 50–75 Hz; running an LLM over a 60-minute audio clip at 50 Hz would require roughly 180,000 tokens of context — well beyond what's tractable. VibeVoice's tokenizer compresses the same hour into a 27,000-token range, which fits comfortably inside the 64K window the ASR model is trained against.
The diffusion head, meanwhile, is what gives the TTS variants their long-horizon coherence. Diffusion lets the model refine acoustic detail across an entire passage rather than committing token-by-token to a forward sample, which is why VibeVoice-TTS could maintain speaker consistency across 90 minutes without the drift that plagues autoregressive TTS at long horizons.
What's actually on the table for builders today
If you want to use VibeVoice in May 2026, here's what's available:
- VibeVoice-ASR — full open weights, MIT license, native Transformers support. Download from
microsoft/VibeVoice-ASRon Hugging Face. - VibeVoice-Realtime-0.5B — full open weights, MIT license, Colab notebook included. Download from
microsoft/VibeVoice-Realtime-0.5B. - VibeVoice-TTS-1.5B — weights still listed but the inference code has been removed; not a practical option for production.
- Finetuning — official ASR finetuning code and vLLM integration are documented in the repo.
Notable third-party adoption is already showing up: Vibing, an open-source voice-powered input method for macOS and Windows, ships with VibeVoice-ASR underneath. Expect more — the structured-output format and 50+ language coverage make it a natural fit for voice-driven productivity tools that don't want to depend on closed APIs.
The Bottom Line
VibeVoice is the most credible open-source voice AI release of 2026 — and the fact that it ships without its strongest model is what makes it interesting. VibeVoice-ASR is genuinely state-of-the-art for long-form recognition, the 7.5 Hz tokenizer trick is doing real architectural work, and the VibeVoice-Realtime model is one of the few openly licensed options for sub-300ms streaming TTS. The TTS-1.5B withdrawal signals that Microsoft is willing to police its own releases when the abuse picture gets ugly. For a team building voice features in 2026, that combination — frontier capability, MIT license, and a vendor that demonstrably enforces its own use policy — is a more useful baseline than any closed API on the market.
