Qwen3.6-29B-REAP-Opus-Distill-MTPLX-3bit

MTPLX 3-bit conversion of barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP โ€” the REAP-pruned (205/256 experts) Qwen3.6-35B-A3B MoE merged with an Opus-reasoning-distilled LoRA, plus its native MTP layer โ€” for speculative decoding on Apple Silicon via mtplx.

Sibling formats: the bf16 safetensors source above and the GGUF collection for llama.cpp/Ollama/LM Studio.

Converted straight from the bf16 safetensors weights (not from any GGUF quant), so there is no dequantize-requantize drift in the chain.

Why this release exists

MTPLX grafts multi-token prediction (MTP) onto a standard checkpoint and exposes it through MLX on Apple Silicon: the draft head proposes 1โ€“3 tokens per step, the base model verifies them, and accepted tokens are emitted in a single forward pass. For this MoE (29B total, ~3B active per token) the 3-bit trunk cuts the on-disk footprint to ~13.8 GB and keeps the MTP sidecar in bf16, unlocking speculative speedups without a separate draft model.

This repo is the 3-bit affine point of that family. If you can afford the VRAM/unified-memory budget, the sibling 4-bit MTPLX build (when published) will be closer to lossless; for tight budgets this 3-bit file is the entry point.

Quality benchmarks (of the source safetensors model)

Measured with lm-evaluation-harness (HF backend, bfloat16, chat template disabled) on the underlying safetensors checkpoint. Same base chain as the GGUF card: unpruned Qwen3.6-35B-A3B โ†’ REAP 205/256 (no LoRA, RangerX/Qwen3.6-35B-REAP-Pruned-ratio-0.2) โ†’ this checkpoint.

Task Axis Qwen3.6-35B-A3B RangerX REAP-20% This model
wikitext perplexity (lower=better) sanity 7.85 10.06 10.06
MMLU out-of-calibration 0.8409 0.8152 0.8257
HellaSwag (0-shot, acc_norm) out-of-calibration 0.7420 0.7440 0.7340
ARC-Challenge (0-shot, acc_norm) out-of-calibration 0.5320 0.5340 0.6160

Key finding: the reasoning LoRA pushes ARC-Challenge above the unpruned 256-expert base (0.616 vs 0.532) โ€” a genuine skill transfer from the Opus chain-of-thought data. See the GGUF card and the safetensors card for full methodology.

Available MTPLX conversions

Repo Bits Size (on disk) Use case
this repo (3-bit) 3 ~13.8 GB (12.8 GiB) tight unified memory, entry point
barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP bf16 ~56 GB reference / re-quant source

No separate mmproj/vision file โ€” this MoE is text-only.

Which one to pick:

  • Best quality โ†’ bf16 source
  • Apple Silicon with โ‰ค24 GB unified memory โ†’ this 3-bit MTPLX
  • llama.cpp / Ollama / LM Studio workflows โ†’ use the GGUF repo instead (IQ/MTP-GGUF quants)

Conversion notes

  • Source: barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP (commit 9fa24b5, 16 shards + model-mtp.safetensors), forge-local via mtplx==2.9.1
  • Recipe: body_bits=3, body_group_size=64, body_mode=affine, mtp_policy=keep_bf16, contract_calibration_depth=1 / limit=8 / windows=4, concat_orders=embedding_hidden, MoE-aware (qwen3-next-mtp, mtp_sidecar=bf16-qwen-moe)
  • MTP contract: base_hidden_variant=pre_norm, hidden_variant=post_norm, concat_order=embedding_hidden, mtp_position_mode=local, mtp_quant_group_size=64, mtp_quant_mode=affine
  • Output: 3 safetensors shards + mtp.safetensors (1.3 GB, bf16 sidecar) + model.safetensors.index.json; architecture Qwen3_5MoeForCausalLM (40 layers, hidden 2048, 205/256 experts, hybrid Gated-DeltaNet / full-attention)
  • Forged at: 2026-08-23T18:26:09+03:00 on Apple M5 Pro (18 CPU / 20 GPU, 24 GB unified memory) โ€” tuned for 24 GB Macs ยท macOS 27.0 arm64, mtplx_runtime.json ships in-repo as provenance

Vision

Not applicable โ€” text-only MoE. No model-vision.safetensors, no mmproj.

MTPLX usage

# install
pip install -U mtplx

# single-turn generation (auto-selects the verified depth if you omit --depth)
mtplx run --model barozp/Qwen3.6-29B-REAP-Opus-Distill-MTPLX-3bit \
  --prompt "Explain the Monty Hall problem briefly." \
  --depth 1 --max-tokens 512

# OpenAI-compatible local server
mtplx serve --model barozp/Qwen3.6-29B-REAP-Opus-Distill-MTPLX-3bit \
  --depth 1 --port 8080
# then: curl http://localhost:8080/v1/chat/completions ...

# text-only chat (temperature 0.6, top_p 0.95, top_k 20 are the verified defaults)
mtplx run --model barozp/Qwen3.6-29B-REAP-Opus-Distill-MTPLX-3bit \
  --prompt "Write a short story about a robot learning to paint." \
  --depth 1

Depth is the number of draft tokens proposed per step. Higher depth is not always faster โ€” use the verified optimum below.

Verification results

Verified locally with mtplx forge verify (MLX backend, greedy + quality gate, 512-token budget). Depth 0 is plain autoregressive (no MTP); depth N proposes N draft tokens.

Depth tok/s vs AR Acceptance by position Quality
0 (AR) 91.5 1.00x โ€” pass
1 101.5 1.08x 58.2% pass
2 84.7 0.93x 52.6% / 15.5% pass
3 71.8 0.78x 50.3% / 17.4% / 2.4% pass
  • Recommended depth: 1 (mtp_depth_wins at D1; D2/D3 lose throughput on this MoE+3-bit combo despite passing quality)
  • Verdict: mtp_depth_wins; quality_rejected=[], failure_reasons=[]
  • Hardware: Apple M5 Pro (18 CPU / 20 GPU, 24 GB) ยท macOS 27.0 arm64 (Apple Silicon, MLX), mtplx 2.9.1, artifact sha256:6af5c44e...
  • All depths passed the quality gate; the gate is strict (rejects on drift), so any quality_passed=true row is safe to serve. Throughput is the tiebreaker.

Note: This MoE's head is unusually sensitive to quantization at 3-bit (single-position acceptance 58% vs 83โ€“98% on the Qwen3.8 dense family). Expect the best speedup at D1 only; the dense VLM siblings in this collection sustain D2/D3 at >2x.

Training details (source safetensors model)

  • Base: Qwen/Qwen3.6-35B-A3B โ€” MoE, REAP-pruned to 205/256 experts
  • Method: LoRA r=64, alpha=64, dropout 0.05, merged into base weights
  • Data: barozp/opus-reasoning-distill-train (Opus reasoning traces)
  • Vision + MTP: vision N/A; MTP layer carried over from base and kept in bf16 for MTPLX

Source chain

Qwen/Qwen3.6-35B-A3B (base) โ†’ RangerX/Qwen3.6-35B-REAP-Pruned-ratio-0.2 (REAP 20% pruning) โ†’ barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill (LoRA finetune) โ†’ barozp/Qwen3.6-29B-REAP-Opus-Reasoning-Distill-MTP (MTP head) โ†’ this repo (MTPLX 3-bit conversion)

Downloads last month
68
Safetensors
Model size
28B params
Tensor type
U32
ยท
BF16
ยท
MLX
Hardware compatibility
Log In to add your hardware

3-bit

Inference Providers NEW
This model isn't deployed by any Inference Provider. ๐Ÿ™‹ Ask for provider support

Model tree for barozp/Qwen3.6-29B-REAP-Opus-Distill-MTPLX-3bit

Collections including barozp/Qwen3.6-29B-REAP-Opus-Distill-MTPLX-3bit