Instructions to use biscet/Whittle-35B-A3B-DFlash2 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks
Google Colab
Kaggle
Local Apps Settings

How to use biscet/Whittle-35B-A3B-DFlash2 with llama.cpp:

Install (macOS, Linux)

curl -LsSf https://llama.app/install.sh | sh
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf biscet/Whittle-35B-A3B-DFlash2:F16
# Run inference directly in the terminal:
llama cli -hf biscet/Whittle-35B-A3B-DFlash2:F16

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama serve -hf biscet/Whittle-35B-A3B-DFlash2:F16
# Run inference directly in the terminal:
llama cli -hf biscet/Whittle-35B-A3B-DFlash2:F16

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-app
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf biscet/Whittle-35B-A3B-DFlash2:F16
# Run inference directly in the terminal:
./build/bin/llama cli -hf biscet/Whittle-35B-A3B-DFlash2:F16

Use Docker

docker model run hf.co/biscet/Whittle-35B-A3B-DFlash2:F16

Llama new
LM Studio
Jan

vLLM

How to use biscet/Whittle-35B-A3B-DFlash2 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "biscet/Whittle-35B-A3B-DFlash2"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "biscet/Whittle-35B-A3B-DFlash2",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/biscet/Whittle-35B-A3B-DFlash2:F16

Ollama
How to use biscet/Whittle-35B-A3B-DFlash2 with Ollama:
```
ollama run hf.co/biscet/Whittle-35B-A3B-DFlash2:F16
```
Unsloth Desktop

How to use biscet/Whittle-35B-A3B-DFlash2 with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf biscet/Whittle-35B-A3B-DFlash2:F16

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "biscet/Whittle-35B-A3B-DFlash2:F16"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Docker Model Runner
How to use biscet/Whittle-35B-A3B-DFlash2 with Docker Model Runner:
```
docker model run hf.co/biscet/Whittle-35B-A3B-DFlash2:F16
```

Lemonade

How to use biscet/Whittle-35B-A3B-DFlash2 with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull biscet/Whittle-35B-A3B-DFlash2:F16

Run and chat with the model

lemonade run user.Whittle-35B-A3B-DFlash2-F16

List all available models

lemonade list

Hermes Agent

How to use biscet/Whittle-35B-A3B-DFlash2 with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf biscet/Whittle-35B-A3B-DFlash2:F16

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default biscet/Whittle-35B-A3B-DFlash2:F16

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use biscet/Whittle-35B-A3B-DFlash2 with OpenClaw:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama serve -hf biscet/Whittle-35B-A3B-DFlash2:F16

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "biscet/Whittle-35B-A3B-DFlash2:F16" \
  --custom-provider-id llama-cpp \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Whittle 35B-A3B DFlash2

A focused draft for Whittle.

526M parameters · F16 GGUF · v1.2.0 · Apache-2.0

Download GGUF · Training data · Release notes

Stage one complete — November 13, 2026. v1.2.0 packages the best saved checkpoint at continuation step 9,074, selected by validation loss. The next training stage is planned around more complex tasks to improve drafting efficiency.

A DFlash2 drafting adapter for speculative decoding with Whittle-Qwen-3.8-35B-A3B, adapted to its Q5_K_M GGUF. Run it alongside the separate Whittle target in a compatible DFlash2 runtime. The target stays frozen while all draft parameters are fine-tuned.

At a glance

	Specification
Release	v1.2.0, completed first training stage
Draft	526,251,520 parameters · 6 layers · 16-token architecture block
Initialization	v1.0.0 adapted draft; original ancestry: Ornith 1.5 DFlash2
Target	Whittle Q5_K_M GGUF
Download	F16 GGUF, about 0.991 GiB; matching tensors in `tensors/best-model.safetensors`
License	Apache-2.0, with inherited notices in NOTICE

What's improved

The same 256-example validation monitor was used throughout this continuation. The baseline is the v1.0.0 parent evaluated on this monitor; these figures are offline draft-training metrics.

Metric	v1.0.0 parent	v1.2.0, step 9,074	Change
Validation loss ↓	4.2199	3.8691	−8.31%
Draft-token accuracy ↑	46.90%	50.39%	+3.49 percentage points
Correct-prefix proxy ↑	2.3965 tokens	2.6211 tokens	+9.37%
Gold-token recall in top 16 ↑	71.23%	73.83%	+2.59 percentage points

Stage one ended after 142 of 176 first-pass shards (80.7%), covering 36,296 usable training traces. Training was intentionally stopped before the first full epoch was complete. The best saved checkpoint is also the last committed checkpoint; selection follows the lowest monitor loss. Full numeric history: EVALUATION.json and training/validation-history.jsonl.

A new tokens/s or runtime-acceptance benchmark has not been run for v1.2.0. Better validation metrics are promising but do not establish a serving-speed gain or better final answers. The reserved test benchmark remains pending.

Training data and recipe

This training stage additionally used DFlash DeepSeek TraceMix: 50,000 RU/EN task contexts, split into 45,000 training, 2,500 validation and 2,500 test tasks. It mixes programming and project-code work, text tasks, reasoning and agent/tool-use contexts. Earlier adaptation used DFlash TraceMix Lite and prompts selected from Tulu 3 SFT Mixture.

Whittle Q5 supplied the training completions and hidden features. DeepSeek Flash acted as a training mentor, reviewing aggregate metrics and advising continuation and learning-rate changes. DeepSeek-generated agent histories are part of the broader dataset; the 50,000 contexts are not 50,000 DeepSeek answers.

All 526M draft parameters were updated with AdamW, batch 4 and a 384-token feature context, using FP32 master weights and FP16 autocast. The base LR was reduced from 5e-6 to 2.5e-6 after step 7,026; projection LR remained 3× base. Each shard was validated before commit, with automatic retry of recoverable FP16 gradient overflows. See TRAINING.md, DATA_SOURCES.md and PROVENANCE.json.

Next training stage

The next stage is planned to use more complex project tasks, multi-file changes, tool interactions and longer dialogues. The aim is better token acceptance and end-to-end drafting efficiency on real workloads. The next training stage has not started, and its gains have not been measured.

Files and use

Whittle-35B-A3B-DFlash2-v1.2.0-F16.gguf — inference draft, with author, version, license and completed-stage status embedded in GGUF metadata.
tensors/best-model.safetensors — the same 96 trained tensors with their original F16/F32 dtypes, in PyTorch module naming, plus tensor inventory. No optimizer state or target weights.
training/v1.2.0/ — portable shard trainer, AMP retry helper and recipe; training/validation-history.jsonl records the monitor trajectory.

Use the compatible Linux/NVIDIA DFlash2 runtime and a separate Whittle Q5 target. USAGE.md lists the launch parameters and download verification. Checksums are in SHA256SUMS.

Previous release measurements and limitations

The v1.0.0 local long-generation test recorded 74.08 decode tokens/s, 67.62 end-to-end tokens/s and 59.53% draft acceptance; its no-draft control recorded 53.60 decode tokens/s. Hardware: V100 32 GB for target computation and draft, RTX 3060 12 GB for target n-gram memory; custom llama.cpp runtime, at most three proposed tokens and BF16 draft KV. These are historical results and do not measure v1.2.0. Archived evaluation.

The older test did not confirm exact greedy equivalence, failed both 64K marker checks with the draft, and did not test a filled 256K context. Validation pools were reused during selection. Those open issues have not been re-evaluated for v1.2.0. Other target quantizations, stock llama.cpp and Apple Metal remain unvalidated. Anonymization of publication text does not imply unlearning of earlier training data; see PRIVACY.md.

main contains v1.2.0. The previous Nightly is preserved on the v1.1.0 branch; the existing v1.0.0 branch remains available.

_{Adaptation by Egor Prokhorov · Published by biscet · License & notices}