# Understanding how changing bnb\_4bit\_compute\_dtype affects outputs

**URL:** <https://discuss.huggingface.co/t/understanding-how-changing-bnb-4bit-compute-dtype-affects-outputs/52167>\
**Category:** 🤗Transformers\
**Created:** [August 25, 2023, 1:36pm UTC](https://discuss.huggingface.co/t/understanding-how-changing-bnb-4bit-compute-dtype-affects-outputs/52167 "2023-08-25T13:36:56Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![samlhuillier](https://avatars.discourse-cdn.com/v4/letter/s/9d8465/32.png) [@samlhuillier](https://discuss.huggingface.co/u/samlhuillier)\
**Post date:** [August 25, 2023, 1:36pm UTC](https://discuss.huggingface.co/t/understanding-how-changing-bnb-4bit-compute-dtype-affects-outputs/52167/1 "2023-08-25T13:36:56Z")

</div>

I’m struggling to understand the impact of bnb\_4bit\_compute\_dtype. Specifically, I’m thinking that if i were to use the quantization config below and switched the bnb\_4bit\_compute\_dtype to float32 - nothing should change in terms of outputs/quality of the model? My thinking is that a 4 bit value should fit into both 16 and 32 bits.

```plaintext
    load_in_4bit=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.float16
)

model_4bit = AutoModelForCausalLM.from_pretrained(model_id, quantization_config=bnb_config)

```

Does anyone have an explanation for this?

---

<div class="post-metadata">

**Author:** ![shahzebnaveed](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/shahzebnaveed/32/23733_2.png) [@shahzebnaveed](https://discuss.huggingface.co/u/shahzebnaveed)\
**Post date:** [February 10, 2024, 8:24am UTC](https://discuss.huggingface.co/t/understanding-how-changing-bnb-4bit-compute-dtype-affects-outputs/52167/2 "2024-02-10T08:24:31Z")

</div>

From QLoRA paper:

_“QLORA has one low-precision storage data type, in our case usually 4-bit, and one computation data type that is usually BFloat16. In practice, this means whenever a QLORA weight tensor is used, we dequantize the tensor to BFloat16, and then perform a matrix multiplication in 16-bit.for CausalLM models, the last lm\_head is kept in its original dtype.”_

This simply means that the tensors are stored in 4-bit quantized format as proposed by QLoRa and whenever a computation needs to be performed, these are pulled in the “computation” data type which is usually FP16 or BF16.

Note that in QLoRa (a parameter-efficient fine-tuning technique), the purpose is to freeze the original layers (which are now quantized in 4-bit and only converted to bnb\_4bit\_compute\_dtype when any calculations are to be performed) and only train the new adapter weights in FP32 or in mixed-precision if `fp16=True` is specified while fine-tuning.