# PEFT LORA for Text Classification?

**URL:** <https://discuss.huggingface.co/t/peft-lora-for-text-classification/42321>\
**Category:** 🤗Transformers\
**Created:** [June 6, 2023, 9:45pm UTC](https://discuss.huggingface.co/t/peft-lora-for-text-classification/42321 "2023-06-06T21:45:11Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![maadnfritz](https://avatars.discourse-cdn.com/v4/letter/m/9dc877/32.png) [@maadnfritz](https://discuss.huggingface.co/u/maadnfritz)\
**Post date:** [June 6, 2023, 9:45pm UTC](https://discuss.huggingface.co/t/peft-lora-for-text-classification/42321/1 "2023-06-06T21:45:11Z")

</div>

I am working on a project for text classification. I started out with BERT and AutoModelForSequenceClassification and now i want to move up the food chain and try some larger models. to do that on a single A100 i was hoping to use PEFT, LORA and bitsandbytes or accelerate for example. Bit the examples i have found all use AutoModelForCausalLM.

I tried to adapt one of the tutorials to use BloomForSequenceClassification but the  
PEFT tutorial suggested " Finally, we need to apply some post-processing on the 8-bit model to enable training, let’s freeze all our layers, and cast the layer-norm in `float32` for stability. We also cast the output of the last layer in `float32` for the same reasons." and had code  
model.lm\_head = CastOutputToFloat(model.lm\_head)  
Which is fine for AutoModelForCausalLM but does not work for BloomForSequenceClassification because it does not have an lm\_head layer.

Which leads me to ask, can one use PEFT and LORA with a AutoModelForSequenceClassification?

Alternatively, can one use a AutoModelForCausalLM for text classification?

cheers

---

<div class="post-metadata">

**Author:** ![jonathanlyj](https://avatars.discourse-cdn.com/v4/letter/j/a9a28c/32.png) [@jonathanlyj](https://discuss.huggingface.co/u/jonathanlyj)\
**Post date:** [December 8, 2023, 6:12am UTC](https://discuss.huggingface.co/t/peft-lora-for-text-classification/42321/2 "2023-12-08T06:12:24Z")

</div>

I have exactly the same problem. If removing CastOutputToFloat line entirely, another error is triggered. Is there a solution for this?