| uv run python custom_mlx_lm/custom_convert.py --hf-path . --mlx-path MobileLLM-R1-950M-mixed-4bit-mlx --dynamic-quant --target-bpw 4.5 --group-size 64 --report-ppl |
| Loading model from .... |
| Loading calibration data... |
| Token indices sequence length is longer than the specified maximum sequence length for this model (110205 > 32768). Running this sequence through the model will result in indexing errors |
| Calculating perplexity of original model... |
| Original PPL: 50.262 |
| Starting advanced mixed-precision quantization... |
| huggingface/tokenizers: The current process just got forked, after parallelism has already been used. Disabling parallelism to avoid deadlocks... |
| To disable this warning, you can either: |
| - Avoid using `tokenizers` before the fork if possible |
| - Explicitly set the environment variable TOKENIZERS_PARALLELISM=(true | false) |
| Estimating sensitivities: 100%|████████████████████████████████████| 54/54 [02:03<00:00, 2.28s/it] |
| Calculating perplexity of quantized model... |
| Quantized PPL: 59.059 |
|
|
| ✅ Model saved to MobileLLM-R1-950M-mixed-4bit-mlx |
|
|
| uv run python custom_mlx_lm/quant_summary.py --model-path MobileLLM-R1-950M-mixed-4bit-mlx --show 8 |
| Method: mixed_precision_dynamic |
| Group size: 64 |
| Total linear layers: 154 |
| 4-bit layers: 153 |
| 8-bit layers: 1 |
|
|
| Examples (8-bit): |
| - layers.0.attention.o_proj |
|
|
| Examples (4-bit): |
| - layers.0.attention.k_proj |
| - layers.0.attention.q_proj |
| - layers.0.attention.v_proj |
| - layers.0.feed_forward.down_proj |
| - layers.0.feed_forward.gate_proj |
| - layers.0.feed_forward.up_proj |
| - layers.1.attention.k_proj |
| - layers.1.attention.o_proj |
|
|
| weights.npz contains quantized tensors: True |
