# Does autogpt-q require float16?

**URL:** <https://discuss.huggingface.co/t/does-autogpt-q-require-float16/52542>\
**Category:** 🤗Transformers\
**Created:** [August 28, 2023, 7:17pm UTC](https://discuss.huggingface.co/t/does-autogpt-q-require-float16/52542 "2023-08-28T19:17:24Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![jmhessel](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/jmhessel/32/18958_2.png) [@jmhessel](https://discuss.huggingface.co/u/jmhessel)\
**Post date:** [August 28, 2023, 7:17pm UTC](https://discuss.huggingface.co/t/does-autogpt-q-require-float16/52542/1 "2023-08-28T19:17:24Z")

</div>

Hi! I’ve been excited about the recent quantization integration of auto-gptq and appreciate the work folks did to make this happen. I was trying to play around a bit, and saw this line

> <https://github.com/huggingface/transformers/blob/50573c648ae953dcc1b94d663651f07fb02268f4/src/transformers/modeling_utils.py#L2579-L2583>

which seems to imply that auto gptq only supports float16 vs. bfloat16… I uncommented this line and things seem to be working fine — do folks have thoughts about this? TheBloke seems to sometimes run quantization with bfloat16 ([[BUG]CUDA OUT OF MEMORY · Issue #179 · PanQiWei/AutoGPTQ · GitHub](https://github.com/PanQiWei/AutoGPTQ/issues/179#issuecomment-1611257490)) so I am not sure why this line is here, but could be missing something (e.g., maybe some of the CUDA kernels are float16 only)?