Mixed Precision GGUF layer quantization of GLM-Z1-32B-0414 by zai-org

Original model: https://huggingface.co/zai-org/GLM-Z1-32B-0414

The hybrid quant employs different quantization levels on a per layer basis to enable both high performance and small file size at the same time. An extended layer definition E quant Q4_E_H for the model is defined as follows (updated 10/12/2026):

   LAYER_TYPES='[
   ["A","attn","Q","attn_q","K","attn_k","V","attn_v","O","attn_o","S","ssm","F","ffn","G","ffn_g","U","ffn_u","D","ffn_d"],
   ["MAP","VOD","0","QN_K","2","Q2_K","3","Q3_K","4","Q4_K","5","Q5_K","6","Q6_K","8","Q8_0","h","F16","f","F32"],
   [0 ,"Q5_K_886"],[1 ,"Q5_K_665"],[2 ,"Q4_K_655"],[3 ,"Q4_K_655"],
   [4 ,"Q4_K_644"],[5 ,"Q4_K_644"],[6 ,"Q4_K_644"],[7 ,"Q4_K_644"],
   [8 ,"Q4_K_544"],[9 ,"Q4_K_544"],[10,"Q4_K_544"],[11,"Q4_K_544"],
   [12,"Q4_K_544"],[13,"Q4_K_544"],[14,"Q4_K_544"],[15,"Q4_K_544"],
   [16,"Q4_K_544"],[17,"Q4_K_544"],[18,"Q4_K_544"],[19,"Q4_K_544"],
   [20,"Q4_K_544"],[21,"Q4_K_544"],[22,"Q4_K_544"],[23,"Q4_K_544"],
   [24,"Q4_K_544"],[25,"Q4_K_544"],[26,"Q4_K_544"],[27,"Q4_K_544"],
   [28,"Q4_K_544"],[29,"Q4_K_544"],[30,"Q4_K_544"],[31,"Q4_K_544"],
   [32,"Q4_K_544"],[33,"Q4_K_544"],[34,"Q4_K_544"],[35,"Q4_K_544"],
   [36,"Q4_K_544"],[37,"Q4_K_544"],[38,"Q4_K_544"],[39,"Q4_K_544"],
   [40,"Q4_K_544"],[41,"Q4_K_544"],[42,"Q4_K_544"],[43,"Q4_K_544"],
   [44,"Q4_K_544"],[45,"Q4_K_544"],[46,"Q4_K_544"],[47,"Q4_K_544"],
   [48,"Q4_K_544"],[49,"Q4_K_544"],[50,"Q4_K_544"],[51,"Q4_K_544"],
   [52,"Q4_K_555"],[53,"Q4_K_445"],[54,"Q4_K_655"],[55,"Q4_K_655"],
   [56,"Q4_K_655"],[57,"Q4_K_666"],[58,"Q4_K_666"],[59,"Q5_K_668"],
   [60,"Q6_K_886"]
   ]'
   FLAGS="--token-embedding-type Q4_K --output-tensor-type Q6_K --layer-types-high"

The layer quants were optimized with smallest quant across layers at Q4_K to minimize entropy loss and provide extremely strong performance across a set of reasoning prompts using greedy sample. The final bpw results in ~0.5G smaller file size than Q4_K_M.

Comparison:

Quant size PPL Comment
Q4_K_M 19.7e9 7.7 default embedding/output
Q4_E_H 19.3e9 7.7 Hybrid E quant with Q4_K embedding Q6_K output

Usage:

This is a RL trained thinking model. The layer quants for this model were evaluated on a set of test/eval prompts using greedy sampling. The quant/model show excellent performance in providing both minimal overthinking and/or getting stuck in infinite generations.

The model can be speculated using Qwen3 0.6B if the inference engine can support dynamic vocab translation between draft and target models. Approx performance using a downstream speculator with llama.cpp on two 4070s (12G VRAM) with layers and context fully in GPU and connected via RPC:

Q QKV ND NKV gen tps Comment
Q4_E_H F16 0 32k 22 No draft
Q4_E_H F16 4 16k 42 Spec 4
Q4_E_H Q8_0 4 18k 41 Spec 4

Benchmarks:

Math benchmarks for the model are given here: https://huggingface.co/spaces/steampunque/benchlm

Download the file from below:

Link Type Size/e9 B Notes
GLM-Z1-32B-0414.Q4_E_H.gguf Q4_E_H 19.3e9 B ~0.5B smaller than Q4_K_M

A discussion thread about the hybrid layer quant approach can be found here on the llama.cpp git repository:

https://github.com/ggml-org/llama.cpp/discussions/13040

Downloads last month
28
GGUF
Model size
33B params
Architecture
glm4
Hardware compatibility
Log In to add your hardware

We're not able to determine the quantization variants.

Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for steampunque/GLM-Z1-32B-0414-MP-GGUF

Quantized
(17)
this model