# Saving fine-tuned MT5ForSequenceClassification

**URL:** <https://discuss.huggingface.co/t/saving-fine-tuned-mt5forsequenceclassification/68413>\
**Category:** 🤗Transformers\
**Created:** [January 8, 2024, 6:39pm UTC](https://discuss.huggingface.co/t/saving-fine-tuned-mt5forsequenceclassification/68413 "2024-01-08T18:39:29Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![adenhaus](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/adenhaus/32/19720_2.png) [@adenhaus](https://discuss.huggingface.co/u/adenhaus)\
**Post date:** [January 8, 2024, 6:39pm UTC](https://discuss.huggingface.co/t/saving-fine-tuned-mt5forsequenceclassification/68413/1 "2024-01-08T18:39:30Z")

</div>

Hi

I fine-tuned [MT5ForSequenceClassification](https://huggingface.co/docs/transformers/model_doc/mt5#transformers.MT5ForSequenceClassification) on a dataset as a regression. However, after fine-tuning, the saved model’s config says the architecture is MT5ForConditionalGeneration. Is this supposed to be like that, or is something wrong? Why did it change?

Also, I don’t know how to do inference with my saved model. The [paper](https://arxiv.org/pdf/2211.00142.pdf) I am trying to replicate says _“We adapt mT5 into a regression-metric by applying a RMSE loss between the logits of a special classification token and the label which is either 0 or 1. During inference, we then force-decode the classification token and extract its probability.”_

But I’m not sure how to do the last sentence.

My eval loss is 0.2576, which isn’t bad, so I should be able to get reasonable results with inference but I haven’t been able to make it work.

Any help would be appreciated. I am using [this](https://github.com/huggingface/transformers/blob/main/examples/pytorch/text-classification/run_classification.py) training script with these args:

```auto
    --model_name_or_path ${MODEL_SCRATCH} \
    --train_file ${DATA_SCRATCH}/train_stata.csv \
    --validation_file ${DATA_SCRATCH}/dev_stata.csv \
    --test_file ${DATA_SCRATCH}/test_stata.csv \
    --do_regression True \
    --metric_name rmse \
    --text_column_name "linearized_input,output" \
    --label_column_name attributable \
    --do_train \
    --do_eval \
    --do_predict \
    --max_seq_length 2048 \
    --per_device_train_batch_size 8 \
    --learning_rate 1e-4 \
    --lr_scheduler_type constant \
    --ignore_mismatched_sizes True \
    --num_train_epochs 1 \
    --output_dir ${OUTPUT_DIR} \
    --overwrite_output_dir True \
    --gradient_checkpointing True \
    --gradient_accumulation_steps 1 \
    --eval_accumulation_steps 1 \
    --text_column_delimiter "[output]" \
    --save_total_limit 1 \
    --load_best_model_at_end True \

```

And here is some of the output logs:

```auto
[INFO|modeling_utils.py:3373] 2024-01-07 22:03:10,696 >> loading weights file /disk/scratch/s2029717/mt5-large-seq/model.safetensors
[INFO|modeling_utils.py:4227] 2024-01-07 22:03:17,330 >> All model checkpoint weights were used when initializing MT5ForSequenceClassification.

[WARNING|modeling_utils.py:4248] 2024-01-07 22:03:17,330 >> Some weights of MT5ForSequenceClassification were not initialized from the model checkpoint at /disk/scratch/s2029717/mt5-large-seq and are newly initialized because the shapes did not match:
- classification_head.out_proj.bias: found shape torch.Size([2]) in the checkpoint and torch.Size([1]) in the model instantiated
- classification_head.out_proj.weight: found shape torch.Size([2, 1024]) in the checkpoint and torch.Size([1, 1024]) in the model instantiated
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.

```

Why am I getting this warning when I am loading a ForSequenceClassification model?

You can see the model files [here](https://huggingface.co/adenhaus/mt5-large-stata/tree/main).

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [January 8, 2024, 7:41pm UTC](https://discuss.huggingface.co/t/saving-fine-tuned-mt5forsequenceclassification/68413/2 "2024-01-08T19:41:38Z")

</div>

Hi,

It seems like the model you saved is an `MT5ForConditionalGeneration` model as seen [here](https://huggingface.co/adenhaus/mt5-large-stata/blob/main/config.json#L4). Hence if you load it using `MT5ForSequenceClassification.from_pretrained` it will tell you that not all the weights are initialized from the checkpoint.

Regarding the paper, that would require adding a special classification token to the vocabulary of the model + tokenizer and make sure the final hidden state of that token is turned into logits. Could look something like:

```auto
# in the init of the model:
self.classifier = nn.Linear(hidden_size, num_labels)

(...)

# in the forward of the model:
# extract final hidden state of the special classification token, assuming it's the first one
# this gives us a tensor of shape (batch_size, hidden_size)
cls_features = final_hidden_states[:,0,:]

# apply classifier to turn `cls_features` into `logits` of shape (batch_size, num_labels)
logits = self.classifier(cls_features)

```

---

<div class="post-metadata">

**Author:** ![adenhaus](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/adenhaus/32/19720_2.png) [@adenhaus](https://discuss.huggingface.co/u/adenhaus)\
**Post date:** [January 8, 2024, 7:48pm UTC](https://discuss.huggingface.co/t/saving-fine-tuned-mt5forsequenceclassification/68413/3 "2024-01-08T19:48:20Z")

</div>

Hi @nielsr, thanks for the reply

> It seems like the model you saved is an `MT5ForConditionalGeneration` model

Yeah I know, but during training I loaded a pretrained `MT5ForSequenceClassification` model, so what I’m confused about is why would it save my finetuned model as `MT5ForConditionalGeneration`?

This is from the fine-tuning logs:

```auto
[INFO|configuration_utils.py:737] 2024-01-07 22:03:06,753 >> loading configuration file /disk/scratch/s2029717/mt5-large-seq/config.json
[INFO|configuration_utils.py:802] 2024-01-07 22:03:06,817 >> Model config MT5Config {
  "_name_or_path": "/disk/scratch/s2029717/mt5-large-seq",
  "architectures": [
    "MT5ForSequenceClassification"
  ],
  "classifier_dropout": 0.0,
  "d_ff": 2816,
  "d_kv": 64,
  "d_model": 1024,
  "decoder_start_token_id": 0,
  "dense_act_fn": "gelu_new",
  "dropout_rate": 0.1,
  "eos_token_id": 1,
  "feed_forward_proj": "gated-gelu",
  "finetuning_task": "text-classification",
  "id2label": {
    "0": "LABEL_0"
  },
  "initializer_factor": 1.0,
  "is_encoder_decoder": true,
  "is_gated_act": true,
  "label2id": {
    "LABEL_0": 0
  },
  "layer_norm_epsilon": 1e-06,
  "model_type": "mt5",
  "num_decoder_layers": 24,
  "num_heads": 16,
  "num_layers": 24,
  "output_past": true,
  "pad_token_id": 0,
  "relative_attention_max_distance": 128,
  "relative_attention_num_buckets": 32,
  "tie_word_embeddings": false,
  "tokenizer_class": "T5Tokenizer",
  "torch_dtype": "float32",
  "transformers_version": "4.37.0.dev0",
  "use_cache": true,
  "vocab_size": 250112
}

```

---

<div class="post-metadata">

**Author:** ![adenhaus](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/adenhaus/32/19720_2.png) [@adenhaus](https://discuss.huggingface.co/u/adenhaus)\
**Post date:** [January 9, 2024, 7:30am UTC](https://discuss.huggingface.co/t/saving-fine-tuned-mt5forsequenceclassification/68413/4 "2024-01-09T07:30:30Z")

</div>

@nielsr also, I still don’t know how to do inference with the fine-tuned model.

I tried the code [here](https://huggingface.co/docs/transformers/tasks/sequence_classification#:~:text=Tokenize%20the%20text%20and%20return%20PyTorch%20tensors%3A) but this predicts a class, when since I am doing a regression I want to extract the probability of my single class instead.

---

<div class="post-metadata">

**Author:** ![adenhaus](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/adenhaus/32/19720_2.png) [@adenhaus](https://discuss.huggingface.co/u/adenhaus)\
**Post date:** [January 24, 2024, 6:00pm UTC](https://discuss.huggingface.co/t/saving-fine-tuned-mt5forsequenceclassification/68413/5 "2024-01-24T18:00:35Z")

</div>

Bit embarrassing, the error was there was a mistake in the script I used to upload the model to the Hub. It was using AutoSeq2SeqLM instead of AutoSequenceClassification.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/hellohellohello/original/2X/d/de4155eb4aa4108ecb32a1389d7cc37ae69f88b7.png) [@system](https://discuss.huggingface.co/u/system)\
**Post date:** [January 25, 2024, 6:01am UTC](https://discuss.huggingface.co/t/saving-fine-tuned-mt5forsequenceclassification/68413/6 "2024-01-25T06:01:08Z")

</div>

Closed on January 25, 2024, 6:01am UTC

This topic was automatically closed 12 hours after the last reply. New replies are no longer allowed.