Instructions to use Hygaard/Inkling-2-Small-oQe-3.07bpw-MLX with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use Hygaard/Inkling-2-Small-oQe-3.07bpw-MLX with MLX:

# Make sure mlx-vlm is installed
# pip install --upgrade mlx-vlm

from mlx_vlm import load, generate
from mlx_vlm.prompt_utils import apply_chat_template
from mlx_vlm.utils import load_config

# Load the model
model, processor = load("Hygaard/Inkling-2-Small-oQe-3.07bpw-MLX")
config = load_config("Hygaard/Inkling-2-Small-oQe-3.07bpw-MLX")

# Prepare input
image = ["http://images.cocodataset.org/val2017/000000039769.jpg"]
prompt = "Describe this image."

# Apply chat template
formatted_prompt = apply_chat_template(
    processor, config, prompt, num_images=1
)

# Generate output
output = generate(model, processor, formatted_prompt, image)
print(output)

Notebooks
Google Colab
Kaggle
Local Apps Settings
LM Studio

How to use Hygaard/Inkling-2-Small-oQe-3.07bpw-MLX with Pi:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "Hygaard/Inkling-2-Small-oQe-3.07bpw-MLX"

Configure the model in Pi

# Install Pi:
npm install -g @earendil-works/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "mlx-lm": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "Hygaard/Inkling-2-Small-oQe-3.07bpw-MLX"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent

How to use Hygaard/Inkling-2-Small-oQe-3.07bpw-MLX with Hermes Agent:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "Hygaard/Inkling-2-Small-oQe-3.07bpw-MLX"

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default Hygaard/Inkling-2-Small-oQe-3.07bpw-MLX

Run Hermes

hermes

Atomic Chat

OpenClaw

How to use Hygaard/Inkling-2-Small-oQe-3.07bpw-MLX with OpenClaw:

Start the MLX server

# Install MLX LM:
uv tool install mlx-lm
# Start a local OpenAI-compatible server:
mlx_lm.server --model "Hygaard/Inkling-2-Small-oQe-3.07bpw-MLX"

Configure OpenClaw

# Install OpenClaw:
npm install -g openclaw@latest
# Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local \
  --auth-choice custom-api-key \
  --custom-base-url http://127.0.0.1:8080/v1 \
  --custom-model-id "Hygaard/Inkling-2-Small-oQe-3.07bpw-MLX" \
  --custom-provider-id mlx-lm \
  --custom-compatibility openai \
  --custom-text-input \
  --accept-risk \
  --skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Inkling-2-Small oQe-3.07bpw-MLX

Mixed 2/3-bit MLX conversion of Thinking Machines’ Inkling-2-Small-MXFP8, targeting fully resident inference on 256 GiB Apple Silicon systems.

3.0732 effective BPW · 213.142 GiB including MTP
Main weights: 209.954 GiB. Original MTP sidecar: 3.188 GiB.

Conversion

Source: thinkingmachines/Inkling-2-Small-MXFP8 at 086a047e8b7ff2ac902dd52355328ddb66b16914.
Expert precision: per-projection 2/3-bit affine allocation, group size 128.
Protected projections: 8-bit attention, token embeddings and output head; embedding/head groups are 64.
Preserved: all 48 text layers, all 384 routed experts per layer, top-8 routing, BF16 vision, original BF16 MTP sidecar, tokenizer and chat template.
Format: 82 main MLX safetensors shards plus mtp.safetensors.

Mixed precision

Component	Stored precision	Original scalar parameters	Stored GiB
Routed experts	2/3-bit affine	587,068,342,272	202.658203
Text attention and auxiliaries	8-bit affine / BF16	4,714,205,184	4.529011
Norms, convolutions and residual controls	BF16 / FP32	2,810,880	0.006197
Routers	FP32	84,953,088	0.316475
Token embeddings and output head	8-bit affine	1,852,637,184	1.833241
Vision encoder and projector	BF16	327,693,312	0.610376
MTP / draft	BF16	1,711,710,384	3.188309
Safetensors headers	Metadata	—	0.000185
Total	3.0732 effective BPW	595,762,352,304	213.141997

Affine scales/biases are BF16 and included in stored bytes. Effective BPW is 8 × 228,859,476,464 weight-file bytes ÷ 595,762,352,304 original scalar parameters, including MTP and file headers but excluding tokenizer/config files, KV and runtime memory. Main-only storage is 3.0359 BPW. Exact assignments are in MODULES.csv, TENSORS.csv and PRECISION.json.

Per-block precision map

Text block	Expert w13 bits	Expert w2 bits	Attention bits / group	Affine metadata / routers
0	3	3	8 / 128	BF16 / F32
1	3	3	8 / 128	BF16 / F32
2	3	3	8 / 128	BF16 / F32
3	3	3	8 / 128	BF16 / F32
4	3	3	8 / 128	BF16 / F32
5	3	3	8 / 128	BF16 / F32
6	2	2	8 / 128	BF16 / F32
7	3	3	8 / 128	BF16 / F32
8	3	3	8 / 128	BF16 / F32
9	3	3	8 / 128	BF16 / F32
10	3	3	8 / 128	BF16 / F32
11	3	3	8 / 128	BF16 / F32
12	3	3	8 / 128	BF16 / F32
13	3	3	8 / 128	BF16 / F32
14	3	3	8 / 128	BF16 / F32
15	3	3	8 / 128	BF16 / F32
16	3	3	8 / 128	BF16 / F32
17	3	3	8 / 128	BF16 / F32
18	2	2	8 / 128	BF16 / F32
19	2	2	8 / 128	BF16 / F32
20	3	3	8 / 128	BF16 / F32
21	3	3	8 / 128	BF16 / F32
22	3	3	8 / 128	BF16 / F32
23	3	3	8 / 128	BF16 / F32
24	2	2	8 / 128	BF16 / F32
25	2	2	8 / 128	BF16 / F32
26	2	2	8 / 128	BF16 / F32
27	3	3	8 / 128	BF16 / F32
28	3	3	8 / 128	BF16 / F32
29	3	3	8 / 128	BF16 / F32
30	2	2	8 / 128	BF16 / F32
31	2	3	8 / 128	BF16 / F32
32	2	3	8 / 128	BF16 / F32
33	2	3	8 / 128	BF16 / F32
34	3	3	8 / 128	BF16 / F32
35	3	3	8 / 128	BF16 / F32
36	3	3	8 / 128	BF16 / F32
37	3	3	8 / 128	BF16 / F32
38	3	3	8 / 128	BF16 / F32
39	3	3	8 / 128	BF16 / F32
40	3	3	8 / 128	BF16 / F32
41	3	3	8 / 128	BF16 / F32
42	2	2	8 / 128	BF16 / F32
43	2	2	8 / 128	BF16 / F32
44	2	2	8 / 128	BF16 / F32
45	2	2	8 / 128	BF16 / F32
46	2	3	8 / 128	BF16 / F32
47	3	3	8 / 128	BF16 / F32

Expert groups are 128; embedding/head groups are 64. Exact tensor shapes, module assignments and bytes are in TENSORS.csv, MODULES.csv and PRECISION.json.

Running and measured performance

Tested fully resident on a dedicated 256 GiB M5 Ultra, without layer-weight paging.

Execution profile	Total request budget	Decode tok/s
Native standard KV, MTP off	8K	40.47
Native standard KV, MTP on	8K	74.75

Measured on two fixed text tasks per row, one sequence and no prefix reuse. All four responses passed the task checks. MTP-off/on outputs differed; exact token equivalence is not claimed. Request budgets include generation reserve.

Requires oMLX commit 624cff0e23519ba46a4d5ed2802c2c236ad1a9ac with the included loader correction. See RUNTIME.md for dependencies and loading details. Broader quality/vision evaluation and fresh-install/HTTP serving validation remain pending.

Credits and license

Model: Thinking Machines Lab. Runtime and conversion tooling: oMLX / oQe. Community conversion: Hygaard.

Licensed under Apache-2.0. See LICENSE, SOURCE-NOTICE.md and CONVERSION.json for attribution and conversion provenance.

Downloads last month: 37

Safetensors

Model size

594B params

Tensor type

U32

BF16

F32

MLX

Hardware compatibility

2-bit

Model tree for Hygaard/Inkling-2-Small-oQe-3.07bpw-MLX

Base model

thinkingmachines/Inkling-2-Small-MXFP8

Quantized

(5)

this model