# How do I fine-tune roberta-large for text classification

**URL:** <https://discuss.huggingface.co/t/how-do-i-fine-tune-roberta-large-for-text-classification/12845>\
**Category:** Beginners\
**Created:** [December 15, 2021, 11:25am UTC](https://discuss.huggingface.co/t/how-do-i-fine-tune-roberta-large-for-text-classification/12845 "2021-12-15T11:25:09Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![nickmuchi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nickmuchi/32/29420_2.png) [@nickmuchi](https://discuss.huggingface.co/u/nickmuchi)\
**Post date:** [December 15, 2021, 11:25am UTC](https://discuss.huggingface.co/t/how-do-i-fine-tune-roberta-large-for-text-classification/12845/1 "2021-12-15T11:25:09Z")

</div>

Hi there,

I have been doing the HF course and decided to apply what I have learned but I have unfortunately encountered some errors at the model.fit() stage.

I extracted BBC text data as an excel file from kaggle and converted it to a DatasetDict as below:

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/4/40b3b11781223c84755e931ed2b1a5807e729739.png)

Loaded the tokenizer and tokenized the text features

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/1/19f994c71257d7b8caa5891cbf844c296cef9561.png)

Train\_test\_val split

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/c/cca63e9095843dbf2504654221ba8c5e0a3f3dc2.png)

Converted my data to tf\_data and padded with DataCollator and instantiated the model

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/b/b809efc0cab17d9a6e83c742014d12955516748b.png)

Optimizer and compile:

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/7/7fb821cb30376475791ca67738e15cc9f726cd38.png)

GEtting the error at the below stage

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/2/2ebfc05d3bd739a4f95b344a3897a033d329c7b4.png)

Not sure what I am doing wrong here as I tried following the steps in the course, thanks in advance

---

<div class="post-metadata">

**Author:** ![nickmuchi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nickmuchi/32/29420_2.png) [@nickmuchi](https://discuss.huggingface.co/u/nickmuchi)\
**Post date:** [December 15, 2021, 3:10pm UTC](https://discuss.huggingface.co/t/how-do-i-fine-tune-roberta-large-for-text-classification/12845/2 "2021-12-15T15:10:14Z")

</div>

Managed to get it to work by changing ‘label’ to “labels” but now I have a different error during model.fit

labels.shape is inconsistent with logits.shape

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/1/121372d26e1ba6f98e13383b975b918c61bdc63f.png)

---

<div class="post-metadata">

**Author:** ![nickmuchi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nickmuchi/32/29420_2.png) [@nickmuchi](https://discuss.huggingface.co/u/nickmuchi)\
**Post date:** [December 16, 2021, 2:29am UTC](https://discuss.huggingface.co/t/how-do-i-fine-tune-roberta-large-for-text-classification/12845/3 "2021-12-16T02:29:17Z")

</div>

@lewtun are you able to assist please?

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [December 16, 2021, 9:53am UTC](https://discuss.huggingface.co/t/how-do-i-fine-tune-roberta-large-for-text-classification/12845/4 "2021-12-16T09:53:52Z")

</div>

Hi,

It depends on what loss function the model has defined. As explained on [Stackoverflow](https://stackoverflow.com/a/62286888), your labels must either be 1-dimensional or 2-dimensional, depending on the loss function being used.

i.e. they must either be of shape (batch\_size,) in which case they contain the class indices, or of shape (batch\_size, num\_labels), in which case they contain one-hot encoded targets.

---

<div class="post-metadata">

**Author:** ![merve](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/merve/32/49809_2.png) [@merve](https://discuss.huggingface.co/u/merve)\
**Post date:** [December 16, 2021, 10:05am UTC](https://discuss.huggingface.co/t/how-do-i-fine-tune-roberta-large-for-text-classification/12845/5 "2021-12-16T10:05:07Z")

</div>

@Rocketknight1

---

<div class="post-metadata">

**Author:** ![Rocketknight1](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/rocketknight1/32/21055_2.png) [@Rocketknight1](https://discuss.huggingface.co/u/Rocketknight1)\
**Post date:** [December 16, 2021, 4:43pm UTC](https://discuss.huggingface.co/t/how-do-i-fine-tune-roberta-large-for-text-classification/12845/6 "2021-12-16T16:43:53Z")

</div>

Hi @nickmuchi, the key is in the warning that pops up when you `compile()`! When you compile without specifying a `loss`, the model will compute loss internally. For this to work, though, the labels need to be in your input dict. We talk about this in the [Debugging your Training Pipeline](https://huggingface.co/course/chapter8/4?fw=tf) section of the course.

There are two solutions here. One is to change your calls to `to_tf_dataset()`. Instead of

```auto
columns=["attention_mask", "input_ids"],
label_cols=["labels"]

```

do

```auto
columns=["attention_mask", "input_ids", "labels"],

```

This will put your labels in the input dict, and the model will be able to compute a loss in the forward pass. This is simple, but might cause issues with your accuracy metric. The alternative option is to leave the labels where they are, but instead to use a proper Keras loss. In that case, you would leave the call to `to_tf_dataset()` unchanged, but change your `compile()` call to

```auto
model.compile(
optimizer=optimizer, 
metrics=['accuracy'],
loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)
)

```

That should work, and will allow you to keep using the `accuracy` metric too. Let me know if you encounter any other problems!

---

<div class="post-metadata">

**Author:** ![nickmuchi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nickmuchi/32/29420_2.png) [@nickmuchi](https://discuss.huggingface.co/u/nickmuchi)\
**Post date:** [December 16, 2021, 10:35pm UTC](https://discuss.huggingface.co/t/how-do-i-fine-tune-roberta-large-for-text-classification/12845/7 "2021-12-16T22:35:24Z")

</div>

> [@Rocketknight1](#):
>
> `tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True)`

Thanks for your response, so I tried you suggestion but still getting the error and to be clear my labels are NOT one hot encoded.

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/d7762a3e88861096fb7df6f0402c75d22b1f04f0.png)

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/b/b5e4f80e7fa31469b8d7a0baa7293d9c6d76c00b.png)

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/c/c77da6d66eb6133999b60b9cd3ed2e53bbc6a1da.png)

Weird thing is that it worked when, instead of using tf\_to\_dataset, I used tf\_from\_tensor\_slices. Wanted to use the former as I was following along that part of the course

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/2/2b45363cf4961a3ae666684c31d153c678b3509d.png)

appreciate your time and help

---

<div class="post-metadata">

**Author:** ![nickmuchi](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nickmuchi/32/29420_2.png) [@nickmuchi](https://discuss.huggingface.co/u/nickmuchi)\
**Post date:** [December 17, 2021, 10:48am UTC](https://discuss.huggingface.co/t/how-do-i-fine-tune-roberta-large-for-text-classification/12845/8 "2021-12-17T10:48:05Z")

</div>

sorry I completely misunderstood what you said, reread and implemented and it worked!!! thank you!