# Fp16, bf16 in TrainingArgs vs BitsAndBytesConfig

**URL:** <https://discuss.huggingface.co/t/fp16-bf16-in-trainingargs-vs-bitsandbytesconfig/45154>\
**Category:** Beginners\
**Created:** [June 30, 2023, 5:02pm UTC](https://discuss.huggingface.co/t/fp16-bf16-in-trainingargs-vs-bitsandbytesconfig/45154 "2023-06-30T17:02:00Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![MorphzZ](https://avatars.discourse-cdn.com/v4/letter/m/e99b99/32.png) [@MorphzZ](https://discuss.huggingface.co/u/MorphzZ)\
**Post date:** [June 30, 2023, 5:02pm UTC](https://discuss.huggingface.co/t/fp16-bf16-in-trainingargs-vs-bitsandbytesconfig/45154/1 "2023-06-30T17:02:00Z")

</div>

Hello, I was going through this excellent article on perf tuning: [Efficient Training on a Single GPU](https://huggingface.co/docs/transformers/v4.21.1/en/perf_train_gpu_one#fp16-training)

1. The first question I have w.r.t. TrainingArgs is that are fp16, bf16, tf32 mutually exclusive? i.e., you would only set one of them to be True?
2. Second, what should be the order (best to worst). I understand bf16 is better than f16. but where does tf32 fall?
3. Third, if I am using BitsAndBytesConfig, it also has a field bnb\_compute\_dtype. is it the same thing as above (meaning if I am setting the flags in TrainingArgs I don’t need to set bnb\_compute\_dtype and vice-versa) or different? and why?

Thanks.