yury-kapoyko/Ling-3.0-tiny-UBTree-dflash-GGUF

Ling-3.0-tiny-UBTree — DFlash draft (GGUF, for llama.cpp)

A GGUF conversion of inclusionAI/Ling-3.0-tiny-UBTree that runs in llama.cpp as a draft-dflash speculative-decoding draft for inclusionAI/Ling-3.0-tiny.

The upstream checkpoint is a UBTree draft: a 5-layer DFlash (block-diffusion) backbone plus a learned bigram ("UBTree") selector for tree drafting. llama.cpp does not implement UBTree tree drafting/verification, so this conversion keeps the DFlash backbone and drops the 7 ubtree_selector.* tensors (the selector is not used at runtime). The result behaves as a plain DFlash block drafter: it proposes a block of tokens per step (block size 16, anchor-inclusive), which the target model then verifies.

Usage (llama.cpp)

Requires a llama.cpp build with DFlash support (--spec-type draft-dflash, upstream PR #22105 or newer). No source modifications are needed for inference — the GGUF is a standard dflash architecture file; everything is driven by metadata.

llama-server \
    -m  Ling-3.0-tiny-Q4_K_M.gguf \
    -md Ling-3.0-tiny-UBTree-dflash-bf16.gguf \
    --spec-type draft-dflash --spec-draft-n-max 6 \
    -fa on -c 32768

Notes:

• --spec-draft-n-max is clamped to the draft's block size (16 → up to 15 drafted tokens). On MoE targets, smaller blocks (e.g. 6) usually win: each extra drafted token widens the expert union read during verification, so large blocks cost more than they gain.
• The draft has its own KV cache, sized like the target context and typed f16 by default — on large contexts this can exceed the target's KV. Use -ctkd/-ctvd (e.g. -ctkd q4_0 -ctvd q4_0) to quantize it.
• The draft shares token embeddings and the LM head with the target; they are loaded from the target model at runtime, so the draft GGUF is small (~360 MB).
• Speculative decoding only pays off when the target model fits entirely in GPU memory; with CPU/mmap-resident weights, block verification has no cost advantage.
• To reduce the draft footprint, it can be requantized (llama-quantize ... q8_0).

Deployment notes (measured)

AMD RX 7600 XT 16 GB, ROCm llama.cpp, target Ling-3.0-tiny-Q8_0, -fa on -c 131072 -ctk q8_0 -ctv q8_0 -np 8, greedy decoding, 3000-token completions. Draft: bf16, --spec-type draft-dflash --spec-draft-n-max 6 -ctkd q4_0 -ctvd q4_0.

Workload	Baseline	+ DFlash draft	Gain	Acceptance / mean len
Prose generation	78.8 tok/s	84.8 tok/s	+8%	0.28 / 2.7
Code generation (Python)	79.0 tok/s	127.2 tok/s	+61%	0.53 / 4.2

VRAM: 9.3 GB baseline, 13.2 GB with draft.

Conversion routine

Uses the stock llama.cpp converter (convert_hf_to_gguf.py, recent master) with a small micro-patch so it accepts the UBTree checkpoint in DFlash-only mode:

1. In conversion/__init__.py, add to TEXT_MODEL_MAP:

   "Ling3UBTreeModel": "qwen",
   

2. In conversion/qwen.py, register the architecture on the existing DFlashModel class and filter out the selector tensors:

   @ModelBase.register("DFlashDraftModel", "DFlash2DraftModel", "Ling3UBTreeModel")
   class DFlashModel(Qwen3Model):
       ...
       @classmethod
       def filter_tensors(cls, item):
           name, gen = item
           if name.startswith("ubtree_selector.") or name.startswith("model.ubtree_selector."):
               return None  # UBTree bigram selector: unused in DFlash-only mode
           ...
   

3. Convert, pointing at the target model's HF directory (needed for the tokenizer and shared embeddings):

   python convert_hf_to_gguf.py inclusionAI/Ling-3.0-tiny-UBTree \
       --target-model-dir inclusionAI/Ling-3.0-tiny \
       --outtype bf16 --outfile Ling-3.0-tiny-UBTree-dflash-bf16.gguf
   

The output contains 58 tensors (~362 MB in BF16): 5 transformer layers, the target-feature fusion projection (fc, target hidden layers 4/8/12/16/22), and norms.

Limitations

• DFlash-only: the UBTree bigram selector and tree verification are not used (not implemented in llama.cpp). Full UBTree serving currently requires the (not yet publicly released) SGLang integration from inclusionAI.
• The draft is tied to Ling-3.0-tiny; it cannot draft for other target models and cannot generate text on its own.
• Ling-3.0-tiny is a MoE model: verifying a block of drafted tokens activates the union of their routed experts, so a verification pass reads more of the model than a single-token decode. End-to-end speedups are therefore smaller than the upstream paper's figures, which were measured with a different serving stack on datacenter GPUs.

Credits

• Target model and UBTree draft: inclusionAI (MIT license)
• UBTree paper: arXiv:2609.39972
• DFlash support in llama.cpp: ggml-org/llama.cpp#22105