Instructions to use anikifoss/Kimi-K2-Instruct-DQ4_K with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Libraries

How to use anikifoss/Kimi-K2-Instruct-DQ4_K with llama-cpp-python:

# !pip install llama-cpp-python

from llama_cpp import Llama

llm = Llama.from_pretrained(
	repo_id="anikifoss/Kimi-K2-Instruct-DQ4_K",
	filename="Kimi-K2-Instruct-DQ4_K-00001-of-00014.gguf",
)

llm.create_chat_completion(
	messages = [
		{
			"role": "user",
			"content": "What is the capital of France?"
		}
	]
)

Notebooks
Google Colab
Kaggle
Local Apps

llama.cpp

How to use anikifoss/Kimi-K2-Instruct-DQ4_K with llama.cpp:

Install from brew

brew install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf anikifoss/Kimi-K2-Instruct-DQ4_K
# Run inference directly in the terminal:
llama-cli -hf anikifoss/Kimi-K2-Instruct-DQ4_K

Install from WinGet (Windows)

winget install llama.cpp
# Start a local OpenAI-compatible server with a web UI:
llama-server -hf anikifoss/Kimi-K2-Instruct-DQ4_K
# Run inference directly in the terminal:
llama-cli -hf anikifoss/Kimi-K2-Instruct-DQ4_K

Use pre-built binary

# Download pre-built binary from:
# https://github.com/ggerganov/llama.cpp/releases
# Start a local OpenAI-compatible server with a web UI:
./llama-server -hf anikifoss/Kimi-K2-Instruct-DQ4_K
# Run inference directly in the terminal:
./llama-cli -hf anikifoss/Kimi-K2-Instruct-DQ4_K

Build from source code

git clone https://github.com/ggerganov/llama.cpp.git
cd llama.cpp
cmake -B build
cmake --build build -j --target llama-server llama-cli
# Start a local OpenAI-compatible server with a web UI:
./build/bin/llama-server -hf anikifoss/Kimi-K2-Instruct-DQ4_K
# Run inference directly in the terminal:
./build/bin/llama-cli -hf anikifoss/Kimi-K2-Instruct-DQ4_K

Use Docker

docker model run hf.co/anikifoss/Kimi-K2-Instruct-DQ4_K

LM Studio
Jan

vLLM

How to use anikifoss/Kimi-K2-Instruct-DQ4_K with vLLM:

Install from pip and serve model

# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "anikifoss/Kimi-K2-Instruct-DQ4_K"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "anikifoss/Kimi-K2-Instruct-DQ4_K",
		"messages": [
			{
				"role": "user",
				"content": "What is the capital of France?"
			}
		]
	}'

Use Docker

docker model run hf.co/anikifoss/Kimi-K2-Instruct-DQ4_K

Ollama
How to use anikifoss/Kimi-K2-Instruct-DQ4_K with Ollama:
```
ollama run hf.co/anikifoss/Kimi-K2-Instruct-DQ4_K
```

Unsloth Studio new

How to use anikifoss/Kimi-K2-Instruct-DQ4_K with Unsloth Studio:

Install Unsloth Studio (macOS, Linux, WSL)

curl -fsSL https://unsloth.ai/install.sh | sh
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for anikifoss/Kimi-K2-Instruct-DQ4_K to start chatting

Install Unsloth Studio (Windows)

irm https://unsloth.ai/install.ps1 | iex
# Run unsloth studio
unsloth studio -H 0.0.0.0 -p 8888
# Then open http://localhost:8888 in your browser
# Search for anikifoss/Kimi-K2-Instruct-DQ4_K to start chatting

Using HuggingFace Spaces for Unsloth

# No setup required
# Open https://huggingface.co/spaces/unsloth/studio in your browser
# Search for anikifoss/Kimi-K2-Instruct-DQ4_K to start chatting

Pi new

How to use anikifoss/Kimi-K2-Instruct-DQ4_K with Pi:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf anikifoss/Kimi-K2-Instruct-DQ4_K

Configure the model in Pi

# Install Pi:
npm install -g @mariozechner/pi-coding-agent
# Add to ~/.pi/agent/models.json:
{
  "providers": {
    "llama-cpp": {
      "baseUrl": "http://localhost:8080/v1",
      "api": "openai-completions",
      "apiKey": "none",
      "models": [
        {
          "id": "anikifoss/Kimi-K2-Instruct-DQ4_K"
        }
      ]
    }
  }
}

Run Pi

# Start Pi in your project directory:
pi

Hermes Agent new

How to use anikifoss/Kimi-K2-Instruct-DQ4_K with Hermes Agent:

Start the llama.cpp server

# Install llama.cpp:
brew install llama.cpp
# Start a local OpenAI-compatible server:
llama-server -hf anikifoss/Kimi-K2-Instruct-DQ4_K

Configure Hermes

# Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash
hermes setup
# Point Hermes at the local server:
hermes config set model.provider custom
hermes config set model.base_url http://127.0.0.1:8080/v1
hermes config set model.default anikifoss/Kimi-K2-Instruct-DQ4_K

Run Hermes

hermes

Docker Model Runner
How to use anikifoss/Kimi-K2-Instruct-DQ4_K with Docker Model Runner:
```
docker model run hf.co/anikifoss/Kimi-K2-Instruct-DQ4_K
```

Lemonade

How to use anikifoss/Kimi-K2-Instruct-DQ4_K with Lemonade:

Pull the model

# Download Lemonade from https://lemonade-server.ai/
lemonade pull anikifoss/Kimi-K2-Instruct-DQ4_K

Run and chat with the model

lemonade run user.Kimi-K2-Instruct-DQ4_K-{{QUANT_TAG}}

List all available models

lemonade list

What sort of performance numbers are you seeing with llama.cpp and ik_llama?

by segmond - opened Jul 15, 2025

Discussion

segmond

Jul 15, 2025

What sort of performance numbers are you seeing with llama.cpp and ik_llama? Assuming the specs you have 32gb vram and 512/700+gb sys ram

anikifoss

Owner Jul 15, 2025

With ik_llama.cpp, for programming tasks with 32k context, I'm seeing roughly 115 tokens/second for Prompt Processing and 7-8 tokens/second for Token Generation. Not using llama.cpp much, but from my memory it's roughly 20% slower.

Keep in mind, that Kimi-K2 is still pretty new, so ik_llama may further improve the performance with custom mla code to better support this model.

sousekd

Jul 21, 2025

•

edited Jul 21, 2025

With ik_llama.cpp, 32K f16 context, I managed to offload one layer to 32 GB VRAM... which improved the performance by exactly nothing :).
Anyway, here are benchmark results for Epyc 9355 + RTX 5090:

PP	TG	N_KV	T_PP s	S_PP t/s	T_TG s	S_TG t/s
2048	512	0	11.856	172.73	29.564	17.32
2048	512	2048	11.978	170.98	29.643	17.27
2048	512	4096	12.277	166.81	30.050	17.04
2048	512	6144	12.052	169.92	29.936	17.10
2048	512	8192	13.077	156.62	31.023	16.50
2048	512	10240	12.377	165.47	30.881	16.58
2048	512	12288	13.642	150.12	31.459	16.28
2048	512	14336	12.842	159.47	31.181	16.42
2048	512	16384	14.320	143.01	31.280	16.37
2048	512	18432	13.583	150.77	31.136	16.44
2048	512	20480	13.586	150.75	31.514	16.25
2048	512	22528	13.917	147.16	31.592	16.21
2048	512	24576	14.110	145.15	31.687	16.16
2048	512	26624	14.673	139.57	31.622	16.19
2048	512	28672	16.288	125.74	32.000	16.00
2048	512	30720	14.446	141.77	32.171	15.91

Thank you @anikifoss .

anikifoss

Owner Jul 21, 2025

@sousekd these are amazing numbers, thanks for sharing! How many memory channels do you have in your Epyc 9355 system?

sousekd

Jul 21, 2025

•

edited Jul 21, 2025

@anikifoss It's 12 channels of DDR5-6400. Depending on the server's mood, the OCCT memory benchmark reports around 600 GB/s read and 420 GB/s write bandwidth. The CPU itself isn’t even that expensive. Unfortunately, the same can't be said for the RAM modules :).

My hope is that, eventually, inference engines will become more NUMA-aware, so that adding a second CPU with its own set of RAM would allow me to run Q8 quantizations of these large models. But we might get entirely new architectures with shared memory between the CPU and GPU before that happens...

anikifoss

Owner Jul 23, 2025

That's pretty high RAM bandwidth, I only get about 280GB/sec due to Threadripper's limited CCDs (only 4 CCDs on my CPU). With dual socket and higher clocks, it should be possible to hit 30 t/s with kimi-k2. That would be pretty amazing!

sousekd

Jul 23, 2025

only 4 CCDs on my CPU

Yeah, I still don't understand the impact of CCDs on memory performance. I've seen people on various forums recommending as many CCDs as possible, but at the same time, I’ve read that cross-CCD bandwidth is limited. Based on that, I’d assume fewer CCDs might result in less cross-CCD memory access...

Anyway, here is a useful chart showing CCD layouts for anyone considering EPYC, and here is a detailed explanation of its memory architecture.

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment