DRAFT: VibeVoice-1.5B-hf (Transformers-compatible version)

VibeVoice is a novel framework designed for generating expressive, long-form, multi-speaker conversational audio, such as podcasts, from text. It addresses significant challenges in traditional Text-to-Speech (TTS) systems, particularly in scalability, speaker consistency, and natural turn-taking.

A core innovation of VibeVoice is its use of continuous speech tokenizers (Acoustic and Semantic) operating at an ultra-low frame rate of 7.5 Hz. These tokenizers efficiently preserve audio fidelity while significantly boosting computational efficiency for processing long sequences. VibeVoice employs a next-token diffusion framework, leveraging a Large Language Model (LLM) to understand textual context and dialogue flow, and a diffusion head to generate high-fidelity acoustic details.

The model can synthesize speech up to 90 minutes long with up to 4 distinct speakers, surpassing the typical 1-2 speaker limits of many prior models.

➡️ Technical Report: VibeVoice Technical Report

➡️ Project Page: microsoft/VibeVoice

Models

Model	Context Length	Generation Length	Weight
VibeVoice-1.5B	64K	~90 min	This model
VibeVoice-7B	32K	~45 min	Coming soon
VibeVoice-0.5B-Realtime	8K	~10 min	HF link
VibeVoice-ASR	64K	-	HF link
VibeVoice-AcousticTokenizer	-	-	HF link

Usage

Setup

Until VibeVoice is part of a Transformers release, you can install it from source:

pip install git+https://github.com/huggingface/transformers.git

A noise scheduler is needed as audio generation relies on a diffusion process. The easiest approach (and as done by the model developers) is to use a noise scheduler from the diffusers library as shown in the examples below.

pip install diffusers
pip install soundfile   # for saving audio

Loading the model

from transformers import AutoProcessor, VibeVoiceForConditionalGeneration

repo_id = "bezzam/VibeVoice-1.5B-hf"
processor = AutoProcessor.from_pretrained(repo_id)
model = VibeVoiceForConditionalGeneration.from_pretrained(repo_id)

Text-to-speech (TTS)

import os
import diffusers
from transformers import AutoProcessor, VibeVoiceForConditionalGeneration, set_seed

model_id = "bezzam/VibeVoice-1.5B-hf"
text = "Hello, nice to meet you. How are you?"
set_seed(42)  # for deterministic results

# Load model
processor = AutoProcessor.from_pretrained(model_id)
model = VibeVoiceForConditionalGeneration.from_pretrained(model_id, device_map="auto")
sampling_rate = processor.feature_extractor.sampling_rate

# Prepare input
chat_template = [{"role": "0", "content": [{"type": "text", "text": text}]}]
inputs = processor.apply_chat_template(
    chat_template,
    tokenize=True,
    return_dict=True,
).to(model.device, model.dtype)

# Generate!
noise_scheduler = diffusers.DPMSolverMultistepScheduler(
    beta_schedule="squaredcos_cap_v2",
    prediction_type="v_prediction",
)
audio = model.generate(**inputs, noise_scheduler=noise_scheduler)

# Save to file
fn = f"{os.path.basename(model_id)}_tts.wav"
processor.save_audio(audio, fn)
print(f"Saved output to {fn}")

Generating a podcast from a script

Below is an example to generate a conversation between two speakers.

The example below also used the monitor_progress option to track the generation progress.

import os
import time
import diffusers
import torch
from tqdm import tqdm
from transformers import AutoProcessor, VibeVoiceForConditionalGeneration, set_seed

model_id = "bezzam/VibeVoice-1.5B-hf"
max_new_tokens = 400  # `None` to ensure full generation
set_seed(42)  # for deterministic results

chat_template = [
    {
        "role": "0",
        "content": [
            {
                "type": "text",
                "text": "Hello everyone, and welcome to the VibeVoice podcast. I'm your host, Linda, and today we're getting into one of the biggest debates in all of sports: who's the greatest basketball player of all time? I'm so excited to have Thomas here to talk about it with me.",
            },
        ],
    },
    {
        "role": "1",
        "content": [
            {
                "type": "text",
                "text": "Thanks so much for having me, Linda. You're absolutely right—this question always brings out some seriously strong feelings.",
            },
        ],
    },
    {
        "role": "0",
        "content": [
            {
                "type": "text",
                "text": "Okay, so let's get right into it. For me, it has to be Michael Jordan. Six trips to the Finals, six championships. That kind of perfection is just incredible.",
            },
        ],
    },
    {
        "role": "1",
        "content": [
            {
                "type": "text",
                "text": "Oh man, the first thing that always pops into my head is that shot against the Cleveland Cavaliers back in '89. Jordan just rises, hangs in the air forever, and just sinks it",
            },
        ],
    },
]

# Load model
processor = AutoProcessor.from_pretrained(model_id)
model = VibeVoiceForConditionalGeneration.from_pretrained(model_id, device_map="auto")
sampling_rate = processor.feature_extractor.sampling_rate

# prepare inputs
inputs = processor.apply_chat_template(chat_template, tokenize=True, return_dict=True, sampling_rate=sampling_rate).to(
    model.device, model.dtype
)

# Generate audio with a callback to track progress
noise_scheduler = diffusers.DPMSolverMultistepScheduler(
    beta_schedule="squaredcos_cap_v2",
    prediction_type="v_prediction",
)
start_time = time.time()
completed_samples = set()
with tqdm(desc="Generating") as pbar:

    def monitor_progress(p_batch):
        # p_batch format: [current_step, max_step] for each sample
        active_samples = p_batch[:, 0] < p_batch[:, 1]
        if active_samples.any():
            active_progress = p_batch[active_samples]
            max_active_idx = torch.argmax(active_progress[:, 0])
            p = active_progress[max_active_idx].detach().cpu()
        else:
            p = p_batch[0].detach().cpu()

        pbar.total = int(p[1])
        pbar.n = int(p[0])
        pbar.update()

    audio = model.generate(
        **inputs,
        max_new_tokens=max_new_tokens,
        monitor_progress=monitor_progress,
        noise_scheduler=noise_scheduler,
    )
generation_time = time.time() - start_time
print(f"Generation time: {generation_time:.2f} seconds")

# Save audio
fn = f"{os.path.basename(model_id)}_script.wav"
processor.save_audio(audio, fn)
print(f"Saved output to {fn}")

Batch generation

For batch processing, a list of conversations can be passed to processor.apply_chat_template to prepare the inputs:

inputs = processor.apply_chat_template(
    [conversation1, conversation2], 
    tokenize=True,
    return_dict=True
)

Below is a full example:

import os
import time
import diffusers
import torch
from tqdm import tqdm
from transformers import AutoProcessor, VibeVoiceForConditionalGeneration, set_seed

model_id = "bezzam/VibeVoice-1.5B-hf"
max_new_tokens = 400  # `None` to ensure full generation
set_seed(42)  # for deterministic results

chat_template = [
    [
        {
            "role": "0",
            "content": [
                {
                    "type": "text",
                    "text": "Hello everyone, and welcome to the VibeVoice podcast. I'm your host, Linda, and today we're getting into one of the biggest debates in all of sports: who's the greatest basketball player of all time? I'm so excited to have Thomas here to talk about it with me.",
                },
            ],
        },
        {
            "role": "1",
            "content": [
                {
                    "type": "text",
                    "text": "Thanks so much for having me, Linda. You're absolutely right—this question always brings out some seriously strong feelings.",
                },
            ],
        },
        {
            "role": "0",
            "content": [
                {
                    "type": "text",
                    "text": "Okay, so let's get right into it. For me, it has to be Michael Jordan. Six trips to the Finals, six championships. That kind of perfection is just incredible.",
                },
            ],
        },
        {
            "role": "1",
            "content": [
                {
                    "type": "text",
                    "text": "Oh man, the first thing that always pops into my head is that shot against the Cleveland Cavaliers back in '89. Jordan just rises, hangs in the air forever, and just sinks it",
                },
            ],
        },
    ],
    [
        {
            "role": "0",
            "content": [
                {
                    "type": "text",
                    "text": "Hello and welcome to Planet in Peril. I'm your host, Alice. We're here today to discuss a really sobering new report that looks back at the last ten years of climate change, from 2015 to 2025. It paints a picture not just of steady warming, but of a dangerous acceleration. And to help us unpack this, I'm joined by our expert panel. Welcome Carter, Frank, and Maya.",
                },
            ],
        },
        {
            "role": "1",
            "content": [
                {"type": "text", "text": "Hi Alice, it's great to be here. I'm Carter."},
            ],
        },
        {
            "role": "2",
            "content": [
                {"type": "text", "text": "Hello, uh, I'm Frank. Good to be on."},
            ],
        },
        {
            "role": "3",
            "content": [
                {"type": "text", "text": "And I'm Maya. Thanks for having me."},
            ],
        },
    ],
]

# Load model
processor = AutoProcessor.from_pretrained(model_id)
model = VibeVoiceForConditionalGeneration.from_pretrained(model_id, device_map="auto")
sampling_rate = processor.feature_extractor.sampling_rate

# prepare inputs
inputs = processor.apply_chat_template(
    chat_template,
    return_dict=True,
    tokenize=True,
    sampling_rate=sampling_rate,
).to(model.device, model.dtype)

# Generate audio with a callback to track progress
noise_scheduler = diffusers.DPMSolverMultistepScheduler(
    beta_schedule="squaredcos_cap_v2",
    prediction_type="v_prediction",
)
start_time = time.time()
completed_samples = set()
with tqdm(desc="Generating") as pbar:

    def monitor_progress(p_batch):
        # p_batch format: [current_step, max_step] for each sample
        active_samples = p_batch[:, 0] < p_batch[:, 1]
        if active_samples.any():
            active_progress = p_batch[active_samples]
            max_active_idx = torch.argmax(active_progress[:, 0])
            p = active_progress[max_active_idx].detach().cpu()
        else:
            p = p_batch[0].detach().cpu()

        pbar.total = int(p[1])
        pbar.n = int(p[0])
        pbar.update()

    audio = model.generate(
        **inputs,
        max_new_tokens=max_new_tokens,
        monitor_progress=monitor_progress,
        noise_scheduler=noise_scheduler,
    )
generation_time = time.time() - start_time
print(f"Generation time: {generation_time:.2f} seconds")

# Save audio
output_dir = f"{os.path.basename(model_id)}_batch"
processor.save_audio(audio, output_dir)
print(f"Saved output to {output_dir}")

Pipeline usage

VibeVoice can also be loaded as a pipeline:

import os
import diffusers
import soundfile as sf
from transformers import pipeline, set_seed

model_id = "bezzam/VibeVoice-1.5B-hf"
text = "Hello, nice to meet you. How are you?"
set_seed(42)  # for deterministic results

# Load pipeline with noise scheduler
noise_scheduler = diffusers.DPMSolverMultistepScheduler(
    beta_schedule="squaredcos_cap_v2",
    prediction_type="v_prediction",
)
pipe = pipeline("text-to-speech", model=model_id, noise_scheduler=noise_scheduler)

# Generate!
chat_template = [
    {
        "role": "0",
        "content": [
            {"type": "text", "text": text},
        ],
    }
]
# optional kwargs for generation
generate_kwargs = {"guidance_scale": 1.3, "num_diffusion_steps": 10}
output = pipe(chat_template, generate_kwargs=generate_kwargs)

# Save to file
fn = f"{os.path.basename(model_id)}_pipeline.wav"
sf.write(fn, output["audio"], output["sampling_rate"])
print(f"Saved output to {fn}")

Training

VibeVoice can be trained with the loss outputted by the model.

import diffusers
from transformers import AutoProcessor, VibeVoiceForConditionalGeneration

model_id = "bezzam/VibeVoice-1.5B-hf"
processor = AutoProcessor.from_pretrained(model_id)
model = VibeVoiceForConditionalGeneration.from_pretrained(model_id, device_map="auto")
model.train()

# Prepare batch of 2
chat_template = [
    [
        {
            "role": "0",
            "content": [
                {
                    "type": "text",
                    "text": "VibeVoice is this novel framework designed for generating expressive, long-form, multi-speaker, conversational audio.",
                },
                {
                    "type": "audio",
                    "url": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/realtime_model/vibevoice_tts_german.wav",
                },
            ],
        }
    ],
    # NOTE: multiple speakers not supported yet
    [
        {
            "role": "0",
            "content": [
                {
                    "type": "text",
                    "text": "Hello everyone and welcome to the VibeVoice podcast. I'm your host, Alex, and today we're getting into one of the biggest debates in all of sports: who's the greatest basketball player of all time? I'm so excited to have Sam here to talk about it with me. Thanks so much for having me, Alex. And you're absolutely right. This question always brings out some seriously strong feelings. Okay, so let's get right into it. For me, it has to be Michael Jordan. Six trips to the finals, six championships. That kind of perfection is just incredible. Oh man, the first thing that always pops into my head is that shot against the Cleveland Cavaliers back in '89. Jordan just rises, hangs in the air forever, and just sinks it.",
                },
                {
                    "type": "audio",
                    "url": "https://huggingface.co/datasets/bezzam/vibevoice_samples/resolve/main/example_output/VibeVoice-1.5B_output.wav",
                },
            ],
        }
    ],
]

# Process with apply_chat_template and output_labels=True for training
inputs = processor.apply_chat_template(
    chat_template,
    tokenize=True,
    return_dict=True,
    output_labels=True,
    sampling_rate=processor.feature_extractor.sampling_rate,
).to(model.device, model.dtype)

# Forward pass
noise_scheduler = diffusers.DPMSolverMultistepScheduler(
    beta_schedule="squaredcos_cap_v2",
    prediction_type="v_prediction",
)
outputs = model(**inputs, noise_scheduler=noise_scheduler, ddpm_batch_multiplier=2, num_diffusion_steps=2)

# Compute losses
lm_loss = outputs.loss
diffusion_loss = outputs.diffusion_loss
total_loss = lm_loss + diffusion_loss

print(f"LM loss: {lm_loss.item():.4f}")
print(f"Diffusion loss: {diffusion_loss.item():.4f}")
print(f"Total loss: {total_loss.item():.4f}")

# Backward pass
total_loss.backward()

Torch compile (2x speed-up on A100)

The model can be compiled for faster inference/training.

import time
import diffusers
import torch
from transformers import AutoProcessor, VibeVoiceForConditionalGeneration


model_id = "bezzam/VibeVoice-1.5B-hf"
num_warmup = 5
num_runs = 10

torch.set_float32_matmul_precision("high")

# Load processor + model
processor = AutoProcessor.from_pretrained(model_id)
model = VibeVoiceForConditionalGeneration.from_pretrained(
    model_id,
    torch_dtype=torch.bfloat16,
).to("cuda")

# Prepare static inputs
chat_template = [
    [
        {
            "role": "0",
            "content": [
                {"type": "text", "text": "VibeVoice is a novel framework for generating expressive audio."},
            ],
        }
    ],
] * 4  # batch size 4
inputs = processor.apply_chat_template(
    chat_template,
    tokenize=True,
    return_dict=True,
    sampling_rate=processor.feature_extractor.sampling_rate,
).to("cuda", torch.bfloat16)

noise_scheduler = diffusers.DPMSolverMultistepScheduler(
    beta_schedule="squaredcos_cap_v2",
    prediction_type="v_prediction",
)

# Benchmark without compile
print("Warming up without compile...")
with torch.no_grad():
    for _ in range(num_warmup):
        _ = model(**inputs, noise_scheduler=noise_scheduler)
torch.cuda.synchronize()

print("\n=== Benchmarking without torch.compile ===")
torch.cuda.synchronize()
start = time.time()
with torch.no_grad():
    for _ in range(num_runs):
        _ = model(**inputs, noise_scheduler=noise_scheduler)
torch.cuda.synchronize()

no_compile_time = (time.time() - start) / num_runs
print(f"Average inference time without compile: {no_compile_time:.4f}s")

# Compile the model
print("\nApplying torch.compile...")
model = torch.compile(model)

print("Warming up compiled model...")
with torch.no_grad():
    for _ in range(num_warmup):
        _ = model(**inputs, noise_scheduler=noise_scheduler)
torch.cuda.synchronize()

print("\n=== Benchmarking with torch.compile ===")
torch.cuda.synchronize()
start = time.time()
with torch.no_grad():
    for _ in range(num_runs):
        _ = model(**inputs, noise_scheduler=noise_scheduler)
torch.cuda.synchronize()

compile_time = (time.time() - start) / num_runs
print(f"Average inference time with compile: {compile_time:.4f}s")

speedup = no_compile_time / compile_time
print(f"\n✓ Speedup: {speedup:.2f}x faster")

Training Details

Transformer-based Large Language Model (LLM) integrated with specialized acoustic and semantic tokenizers and a diffusion-based decoding head.

• LLM: Qwen2.5-1.5B for this release.
• Tokenizers:
  • Acoustic Tokenizer: Based on a σ-VAE variant (proposed in LatentLM), with a mirror-symmetric encoder-decoder structure featuring 7 stages of modified Transformer blocks. Achieves 3200x downsampling from 24kHz input. Encoder/decoder components are ~340M parameters each.
  • Semantic Tokenizer: Encoder mirrors the Acoustic Tokenizer's architecture (without VAE components). Trained with an ASR proxy task.
• Diffusion Head: Lightweight module (4 layers, ~123M parameters) conditioned on LLM hidden states. Predicts acoustic VAE features using a Denoising Diffusion Probabilistic Models (DDPM) process. Uses Classifier-Free Guidance (CFG) and DPM-Solver (and variants) during inference.
• Context Length: Trained with a curriculum increasing up to 65,536 tokens.
• Training Stages:
  • Tokenizer Pre-training: Acoustic and Semantic tokenizers are pre-trained separately.
  • VibeVoice Training: Pre-trained tokenizers are frozen; only the LLM and diffusion head parameters are trained. A curriculum learning strategy is used for input sequence length (4k -> 16K -> 32K -> 64K). Text tokenizer not explicitly specified, but the LLM (Qwen2.5) typically uses its own. Audio is "tokenized" via the acoustic and semantic tokenizers.

Responsible Usage

Direct intended uses

The VibeVoice model is limited to research purpose use exploring highly realistic audio dialogue generation detailed in the tech report.

Out-of-scope uses

Use in any manner that violates applicable laws or regulations (including trade compliance laws). Use in any other way that is prohibited by MIT License. Use to generate any text transcript. Furthermore, this release is not intended or licensed for any of the following scenarios:

• Voice impersonation without explicit, recorded consent – cloning a real individual’s voice for satire, advertising, ransom, social‑engineering, or authentication bypass.
• Disinformation or impersonation – creating audio presented as genuine recordings of real people or events.
• Real‑time or low‑latency voice conversion – telephone or video‑conference “live deep‑fake” applications.
• Unsupported language – the model is trained only on English and Chinese data; outputs in other languages are unsupported and may be unintelligible or offensive.
• Generation of background ambience, Foley, or music – VibeVoice is speech‑only and will not produce coherent non‑speech audio.

Risks and limitations

While efforts have been made to optimize it through various techniques, it may still produce outputs that are unexpected, biased, or inaccurate. VibeVoice inherits any biases, errors, or omissions produced by its base model (specifically, Qwen2.5 1.5b in this release). Potential for Deepfakes and Disinformation: High-quality synthetic speech can be misused to create convincing fake audio content for impersonation, fraud, or spreading disinformation. Users must ensure transcripts are reliable, check content accuracy, and avoid using generated content in misleading ways. Users are expected to use the generated content and to deploy the models in a lawful manner, in full compliance with all applicable laws and regulations in the relevant jurisdictions. It is best practice to disclose the use of AI when sharing AI-generated content. English and Chinese only: Transcripts in language other than English or Chinese may result in unexpected audio outputs. Non-Speech Audio: The model focuses solely on speech synthesis and does not handle background noise, music, or other sound effects. Overlapping Speech: The current model does not explicitly model or generate overlapping speech segments in conversations.

Recommendations

We do not recommend using VibeVoice in commercial or real-world applications without further testing and development. This model is intended for research and development purposes only. Please use responsibly.

To mitigate the risks of misuse, we have: Embedded an audible disclaimer (e.g. “This segment was generated by AI”) automatically into every synthesized audio file. Added an imperceptible watermark to generated audio so third parties can verify VibeVoice provenance. Please see contact information at the end of this model card. Logged inference requests (hashed) for abuse pattern detection and publishing aggregated statistics quarterly. Users are responsible for sourcing their datasets legally and ethically. This may include securing appropriate rights and/or anonymizing data prior to use with VibeVoice. Users are reminded to be mindful of data privacy concerns.

Contact

This project was conducted by members of Microsoft Research. We welcome feedback and collaboration from our audience. If you have suggestions, questions, or observe unexpected/offensive behavior in our technology, please contact us at VibeVoice@microsoft.com. If the team receives reports of undesired behavior or identifies issues independently, we will update this repository with appropriate mitigations.