Instructions to use moonshotai/Kimi-K3 with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use moonshotai/Kimi-K3 with Transformers:

# Use a pipeline as a high-level helper
from transformers import pipeline

pipe = pipeline("image-text-to-text", model="moonshotai/Kimi-K3", trust_remote_code=True)
messages = [
    {
        "role": "user",
        "content": [
            {"type": "image", "url": "https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/p-blog/candy.JPG"},
            {"type": "text", "text": "What animal is on the candy?"}
        ]
    },
]
pipe(text=messages)

# Load model directly
from transformers import AutoModel
model = AutoModel.from_pretrained("moonshotai/Kimi-K3", trust_remote_code=True, device_map="auto")

Inference
HuggingChat
Notebooks
Google Colab
Kaggle
Local Apps Settings

vLLM

How to use moonshotai/Kimi-K3 with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "moonshotai/Kimi-K3"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "moonshotai/Kimi-K3",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker

docker model run hf.co/moonshotai/Kimi-K3

SGLang

How to use moonshotai/Kimi-K3 with SGLang:

Install from pip and serve model

# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "moonshotai/Kimi-K3" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "moonshotai/Kimi-K3",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Use Docker images

docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "moonshotai/Kimi-K3" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "moonshotai/Kimi-K3",
		"messages": [
			{
				"role": "user",
				"content": [
					{
						"type": "text",
						"text": "Describe this image in one sentence."
					},
					{
						"type": "image_url",
						"image_url": {
							"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
						}
					}
				]
			}
		]
	}'

Docker Model Runner
How to use moonshotai/Kimi-K3 with Docker Model Runner:
```
docker model run hf.co/moonshotai/Kimi-K3
```

[Feature Request] Transparent "Auto Max" mode — K3 for planning/verification, K2.6 for execution #31

#167

by Arkovski - opened 7 days ago

Discussion

Arkovski

7 days ago

I did GitHub ticket too <3
https://github.com/MoonshotAI/Kimi-K3/issues/31

Summary
I'd like to propose an optional, fully transparent "Auto Max" mode: use K3 for planning and verification, and a cheaper model (e.g. K2.6) for execution steps — similar in spirit to Anthropic's "opusplan" mode (Opus plans, Sonnet executes) and OpenAI's router — but with one key difference: make the model mix explicit and visible to the user.

Motivation

Frontier models like K3 are expensive and slower to serve (large MoE, decode is memory-bandwidth-bound), which users feel as latency in the app.

Closed competitors (OpenAI's "Auto" router, Anthropic's plan/execute split in Claude Code) silently route most tokens to cheaper models while charging frontier prices. Benchmarks and hard queries still hit the big model, so perceived quality stays high at a fraction of the serving cost.

Moonshot's advantage is openness and trust. A labeled routing mode would deliver the same economics without the trust erosion — "planned by K3, executed by K2.6, verified by K3" shown per phase/turn.

Proposed design

Three explicit tiers in the app: Fast (cheap model), Auto Max (routed), Max (literal full K3, current behavior — unchanged).

```
Auto Max flow for agentic tasks:
```

   K3 generates the plan and decomposes it into steps.

     K2.6 executes the steps (with tool calls).

    K3 verifies outputs at checkpoints and re-plans on failure.

Per-turn/per-phase labels so users always know which model produced what.

Opt-in only — never the silent default.

Why this fits agentic tasks better than chat

The plan → execute split happens once (or a few times), so the KV-cache re-prefill cost of switching models is amortized over long execution runs.

In free-flowing chat, per-query routing (sending the whole easy query to the cheaper model) is probably the better mechanism — the cache handoff tax would otherwise eat the savings.

A verification loop (K3 reviewing K2.6's execution) mitigates the known failure mode where the executor silently derails a step the planner would have caught.

Why it fits Moonshot's philosophy
The open-weight releases are why many of us trust and use Kimi. A transparent router is the honest version of what the closed labs already do quietly — it would be both a UX win and a statement that efficiency doesn't require secrecy.
Happy to help test or spec this further. Thanks for the amazing work on K2/K3 and for keeping the weights open!

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment