# Qlora - 8 bit quantization using bitsandbytes gives error for owl-vit model **URL:** \ **Category:** Intermediate\ **Created:** [February 29, 2024, 6:27am UTC](https://discuss.huggingface.co/t/qlora-8-bit-quantization-using-bitsandbytes-gives-error-for-owl-vit-model/75274 "2024-02-29T06:27:15Z")\ **Posts on this page:** 2\ **Page:** 1
**Author:** ![solomonpm](https://avatars.discourse-cdn.com/v4/letter/s/8e7dd6/32.png) [@solomonpm](https://discuss.huggingface.co/u/solomonpm)\ **Post date:** [February 29, 2024, 6:27am UTC](https://discuss.huggingface.co/t/qlora-8-bit-quantization-using-bitsandbytes-gives-error-for-owl-vit-model/75274/1 "2024-02-29T06:27:15Z")
Hi Team, when i am running the above qlora code for owl-vit model (google/owlvit-base-patch32) with below 4 bits bnbconfig , the fine tuning is taking place without any error. bnb\_config = BitsAndBytesConfig( load\_in\_4bit=True, bnb\_4bit\_use\_double\_quant=True, bnb\_4bit\_quant\_type=“nf4”, bnb\_4bit\_compute\_dtype=torch.bfloat16 ) once i change the config with below information bnb\_config = BitsAndBytesConfig( load\_in\_8bit=True ) i receive the following error trace. ######################################################################################### RuntimeError Traceback (most recent call last) Cell In[25], line 1 ----\> 1 trainer.train() File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1537, in Trainer.train(self, resume\_from\_checkpoint, trial, ignore\_keys\_for\_eval, \*\*kwargs) 1535 hf\_hub\_utils.enable\_progress\_bars() 1536 else: → 1537 return inner\_training\_loop( 1538 args=args, 1539 resume\_from\_checkpoint=resume\_from\_checkpoint, 1540 trial=trial, 1541 ignore\_keys\_for\_eval=ignore\_keys\_for\_eval, 1542 ) File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:1854, in Trainer.\_inner\_training\_loop(self, batch\_size, args, resume\_from\_checkpoint, trial, ignore\_keys\_for\_eval) 1851 self.control = self.callback\_handler.on\_step\_begin(args, self.state, self.control) 1853 with self.accelerator.accumulate(model): → 1854 tr\_loss\_step = self.training\_step(model, inputs) 1856 if ( 1857 args.logging\_nan\_inf\_filter 1858 and not is\_torch\_tpu\_available() 1859 and (torch.isnan(tr\_loss\_step) or torch.isinf(tr\_loss\_step)) 1860 ): 1861 # if loss is nan or inf simply add the average of previous logged losses 1862 tr\_loss += tr\_loss / (1 + self.state.global\_step - self.\_globalstep\_last\_logged) File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/transformers/trainer.py:2744, in Trainer.training\_step(self, model, inputs) 2742 scaled\_loss.backward() 2743 else: → 2744 self.accelerator.backward(loss) 2746 return loss.detach() / self.args.gradient\_accumulation\_steps File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/accelerate/accelerator.py:1907, in Accelerator.backward(self, loss, \*\*kwargs) 1905 return 1906 elif self.scaler is not None: → 1907 self.scaler.scale(loss).backward(\*\*kwargs) 1908 else: 1909 loss.backward(\*\*kwargs) File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/\_tensor.py:492, in Tensor.backward(self, gradient, retain\_graph, create\_graph, inputs) 482 if has\_torch\_function\_unary(self): 483 return handle\_torch\_function( 484 Tensor.backward, 485 (self,), (…) 490 inputs=inputs, 491 ) → 492 torch.autograd.backward( 493 self, gradient, retain\_graph, create\_graph, inputs=inputs 494 ) File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/init.py:251, in backward(tensors, grad\_tensors, retain\_graph, create\_graph, grad\_variables, inputs) 246 retain\_graph = create\_graph 248 # The reason we repeat the same comment below is that 249 # some Python versions print out the first line of a multi-line function 250 # calls in the traceback and some print out the last line → 251 Variable.execution\_engine.run\_backward( # Calls into the C++ engine to run the backward pass 252 tensors, 253 grad\_tensors, 254 retain\_graph, 255 create\_graph, 256 inputs, 257 allow\_unreachable=True, 258 accumulate\_grad=True, 259 ) File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/torch/autograd/function.py:288, in BackwardCFunction.apply(self, \*args) 282 raise RuntimeError( 283 "Implementing both ‘backward’ and ‘vjp’ for a custom " 284 "Function is not allowed. You should only implement one " 285 “of them.” 286 ) 287 user\_fn = vjp\_fn if vjp\_fn is not Function.vjp else backward\_fn → 288 return user\_fn(self, \*args) File ~/miniconda3/envs/testenv/lib/python3.10/site-packages/bitsandbytes/autograd/functions.py:491, in MatMul8bitLt.backward(ctx, grad\_output) 485 print(“state.CxB”,state.CxB) 486 print("State ",state) 488 CB = ( 489 undo\_layout(state.CxB, state.tile\_indices) 490 .to(ctx.dtype\_A) → 491 .mul(state.SCB.unsqueeze(1).mul(1.0 / 127.0)) 492 ) 493 grad\_A = torch.matmul(grad\_output, CB).view(ctx.grad\_shape).to(ctx.dtype\_A) 494 else: RuntimeError: The size of tensor a (32) must match the size of tensor b (4) at non-singleton dimension 0 ####################################################################################### need your help. thanks. ---
**Author:** ![ljxw88](https://avatars.discourse-cdn.com/v4/letter/l/8c91f0/32.png) [@ljxw88](https://discuss.huggingface.co/u/ljxw88)\ **Post date:** [April 12, 2024, 8:16pm UTC](https://discuss.huggingface.co/t/qlora-8-bit-quantization-using-bitsandbytes-gives-error-for-owl-vit-model/75274/2 "2024-04-12T20:16:26Z")
I am experiencing the same issue, simply changing from 8bit to 4bit resolves the issue. But I still want to use 8bit. Any solution?