How to run this with 8-bit quantized (or lower)

by lexat - opened Apr 22

Apr 22

I have only 12GB of VRAM. Is it possible to run it?

Owner Apr 24

Yes! For 8-bit quantization: It works fine for short texts, but long contexts will likely cause an OOM (Out of Memory) error.

Apr 24

•

Can you please provide command or instruction how to run it with 12GB of VRAM?

Upload images, audio, and videos by dragging in the text input, pasting, or clicking here.

Tap or paste here to upload images

· Sign up or log in to comment