VibeVoice-ASR-Streaming-1.5B-DFlash2-Drafter

A DFlash 2 block drafter for VibeVoice-ASR-Streaming-1.5B (Ar4ikov/VibeVoice-ASR-Streaming-1.5B-AWQ-W4A16-ASYM), for speculative decoding in vibevoice.c. It proposes 8 tokens in one pass; vibevoice.c checks them in one pass of the model and keeps the ones the model agrees with, plus one of its own.

The check is exact: every checked row is computed with the arithmetic of the model's own one-token decode step, so the transcript with this drafter is byte-for-byte the transcript without it β€” words, timestamps, speakers. The drafter only changes how many passes it takes.

Use

Needs vibevoice.c with DFlash 2 support: branch dflash2 (PR #48), in the next release. The model and this drafter in one download: Ar4ikov/VibeVoice-ASR-Streaming-1.5B-AWQ-W4A16-ASYM-DFlash2 (the INT4 drafter in drafter/, used without --draft).

vv_cli --model ./VibeVoice-ASR-Streaming-1.5B-AWQ-W4A16-ASYM --audio talk.wav --draft ./VibeVoice-ASR-Streaming-1.5B-DFlash2-Drafter
vv_cli serve --model ./VibeVoice-ASR-Streaming-1.5B-AWQ-W4A16-ASYM --draft ./VibeVoice-ASR-Streaming-1.5B-DFlash2-Drafter --slots 4           # streaming sessions (WebSocket, SSE) too
flag
--draft <dir> this repository
--draft-block <n> rows of the drafted block the model checks per pass (the drafter drafts 8; without the flag: 4 or 8, whichever keeps more tokens per ms)
--draft-check exact|fast exact (default): the transcript without a drafter on the same --attn, bit for bit; fast: flashinfer attention and prefill projections for the check, the greedy transcript within rounding
--draft-quant int4|f16 how the runtime holds a BF16 drafter (INT4 by default)

It drafts for any VibeVoice-ASR-Streaming-1.5B checkpoint of the same model β€” microsoft/VibeVoice-ASR-Streaming-1.5B β€” since it reads only the model's hidden states, embedding and LM head; it was trained on the AWQ checkpoint's transcripts, so that is where it drafts best.

Results

vibevoice.c 7d7d43e (branch dflash2), RTX 3090, VibeVoice-ASR-Streaming-1.5B-AWQ-W4A16-ASYM, greedy decoding, decode tokens per second:

plain drafted speedup tokens per block same transcript
20 held-out clips, 8 rows 409 tok/s 869 tok/s 2.12x 3.52 20/20
2-minute file, 8 rows 416 tok/s 1040 tok/s 2.50x 3.99 yes
2-minute file, 4 rows 416 tok/s 837 tok/s 2.01x 2.96 yes
32-minute file, 8 rows 341 tok/s 729 tok/s 2.13x 4.06 yes
32-minute file, 4 rows 341 tok/s 659 tok/s 1.93x 3.00 yes

Streaming sessions (22 + 4 frames a chunk); plain = the same binary without --draft, decode tokens per second, --draft-check exact.

The drafter

  • DFlash 2: 5 Qwen3-style layers (hidden 1536, 12/2 heads, intermediate 4480), 227M parameters, block 8.
  • KV injection: for every position the model has processed, the outputs of its layers 1, 7, 13, 19 and 25 are concatenated, projected (fc) and normalized; every drafter layer turns them into keys and values. The drafter reads the model's own view of the audio and of the transcript so far.
  • Two-tap dynamic convolution around attention and MLP (per-row kernels predicted from the row), and a candidate selector that re-ranks the model head's top 16 per row with a pairwise predecessor/successor term (rank 256).
  • The embedding and the LM head are the model's, frozen. The drafter scores a draft vocabulary of 32768 ids (draft_vocab): the ids its training transcripts use, plus every stop id.
  • Accepted drafts on 43 held-out clips' traces: 3.616 tokens per block of 8 (the selector's, with every earlier draft right).

Training

Self-distillation: the drafter learns what this model writes, timestamps and speaker ids included. tools/dflash in vibevoice.c holds the pipeline:

  1. ~140 hours of audio, 2872 training clips and 43 held out, from LibriSpeech, FLEURS (8 languages), SOVA, AMI, earnings calls and VoxConverse β€” audio only; the datasets' own text is never used.
  2. The model's own greedy transcripts of every clip (vv_dflash_data gen).
  3. Traces: every position's token, its role and the five tapped layers' outputs, replayed through the runtime as streaming sessions (vv_dflash_data trace).
  4. train.py (PyTorch, flex attention): anchors at generated positions, rows 1..7 scored against the next 7 tokens with weights e^(-k/4); cross-entropy of the head plus the selector's; AdamW, cosine schedule, BF16.

Limits

  • vibevoice.c only: the checkpoint keeps the published DFlash 2 names, but the KV injection, the selector and the draft vocabulary are as vibevoice.c implements them (docs/DFLASH.md).
  • CUDA only (the CPU path and Metal decode without the drafter). Greedy only.
  • Audio the corpus covered little (Mandarin, Russian) keeps fewer drafts; the runtime then falls back to plain steps and stays near plain speed.

License

MIT, like VibeVoice.

Downloads last month
54
Safetensors
Model size
0.2B params
Tensor type
I32
Β·
BF16
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for Ar4ikov/VibeVoice-ASR-Streaming-1.5B-DFlash2-Drafter

Collection including Ar4ikov/VibeVoice-ASR-Streaming-1.5B-DFlash2-Drafter