# Fine-tune BERT and Camembert for regression problem **URL:** \ **Category:** Beginners\ **Created:** [July 16, 2020, 9:10pm UTC](https://discuss.huggingface.co/t/fine-tune-bert-and-camembert-for-regression-problem/332 "2020-07-16T21:10:12Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![sundaravel](https://avatars.discourse-cdn.com/v4/letter/s/bc8723/32.png) [@sundaravel](https://discuss.huggingface.co/u/sundaravel)\ **Post date:** [July 16, 2020, 9:10pm UTC](https://discuss.huggingface.co/t/fine-tune-bert-and-camembert-for-regression-problem/332/1 "2020-07-16T21:10:13Z")
I am fine tuning the Bert model on sentence ratings given on a scale of 1 to 9, but rather measuring its accuracy of classifying into the same score/category/bin as the judges, I just want BERT’s score on a continuous scale, like 1,1.1,1.2… to 9. I also need to figure out how to do this using CamemBERT as well. What are all changes to be made in BertForSequenceClassification and CamembertForSequenceClassification module and what are all the changes to be made in preprocessing (like encode\_plus )? ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [July 17, 2020, 4:35am UTC](https://discuss.huggingface.co/t/fine-tune-bert-and-camembert-for-regression-problem/332/2 "2020-07-17T04:35:05Z")
Hi @sundaravel, you can check the source code for `BertForSequenceClassification` [here](https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L1227). It also has code for regression problem. Specifically for regression your last layer will be of shape (hidden\_size, 1) and use MSE loss instead of cross entropy ---
**Author:** ![iaranguiz](https://avatars.discourse-cdn.com/v4/letter/i/a698b9/32.png) [@iaranguiz](https://discuss.huggingface.co/u/iaranguiz)\ **Post date:** [December 27, 2020, 11:40pm UTC](https://discuss.huggingface.co/t/fine-tune-bert-and-camembert-for-regression-problem/332/3 "2020-12-27T23:40:22Z")
Hi @valhalla, i just have created my account in this forum, and i can’t see the link you wrote in your commentary. It seems broken. Do you have another one link to get the source code? Kind regards ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [December 29, 2020, 6:43am UTC](https://discuss.huggingface.co/t/fine-tune-bert-and-camembert-for-regression-problem/332/4 "2020-12-29T06:43:11Z")
Aah, yes. The dir structure is changed so the link is no more. You can now find the class [here](https://github.com/huggingface/transformers/blob/master/src/transformers/models/bert/modeling_bert.py#L1449) ---
**Author:** ![iaranguiz](https://avatars.discourse-cdn.com/v4/letter/i/a698b9/32.png) [@iaranguiz](https://discuss.huggingface.co/u/iaranguiz)\ **Post date:** [December 29, 2020, 2:40pm UTC](https://discuss.huggingface.co/t/fine-tune-bert-and-camembert-for-regression-problem/332/5 "2020-12-29T14:40:10Z")
Thank you very much! ---
**Author:** ![thecity2](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thecity2/32/947_2.png) [@thecity2](https://discuss.huggingface.co/u/thecity2)\ **Post date:** [January 22, 2021, 9:51pm UTC](https://discuss.huggingface.co/t/fine-tune-bert-and-camembert-for-regression-problem/332/6 "2021-01-22T21:51:36Z")
I’m also trying to do regression using BERT and getting an error about Longs? Not sure what I’m doing wrong here. ``` from transformers import DistilBertForSequenceClassification, AdamW, BertConfig device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu') config = BertConfig() config.num_labels = 1 model = BertForSequenceClassification(config).from_pretrained('bert-base-cased') model.to(device) model.train() optim = AdamW(model.parameters(), lr=5e-5) for epoch in range(3): for batch in dataloader: optim.zero_grad() input_ids = batch['input_ids'].to(device) print(input_ids) attention_mask = batch['attention_mask'].to(device) print(attention_mask) token_type_ids = batch['token_type_ids'].to(device) print(token_type_ids) labels = batch['labels'].to(device) print(labels) outputs = model(input_ids=input_ids, attention_mask=attention_mask, token_type_ids=token_type_ids, labels=labels) loss = outputs[0] loss.backward() optim.step() model.eval() ``` Here’s the error (I printed out some data as well): ``` tensor([[101, 1198, 4841, ..., 0, 0, 0], [101, 2960, 2254, ..., 0, 0, 0], [101, 2866, 182, ..., 0, 0, 0], ..., [101, 178, 112, ..., 0, 0, 0], [101, 9294, 1128, ..., 0, 0, 0], [101, 1268, 1185, ..., 0, 0, 0]], device='cuda:0') tensor([[1, 1, 1, ..., 0, 0, 0], [1, 1, 1, ..., 0, 0, 0], [1, 1, 1, ..., 0, 0, 0], ..., [1, 1, 1, ..., 0, 0, 0], [1, 1, 1, ..., 0, 0, 0], [1, 1, 1, ..., 0, 0, 0]], device='cuda:0') tensor([[0, 0, 0, ..., 0, 0, 0], [0, 0, 0, ..., 0, 0, 0], [0, 0, 0, ..., 0, 0, 0], ..., [0, 0, 0, ..., 0, 0, 0], [0, 0, 0, ..., 0, 0, 0], [0, 0, 0, ..., 0, 0, 0]], device='cuda:0') tensor([0.2135, 0.9005, 0.4206, 0.2755, 0.5373, 0.5537, 0.2492, 0.4841, 0.8241, 0.3545, 0.2813, 0.5674, 0.4098, 0.5857, 0.9476, 0.6094, 0.2778, 0.2974, 0.3362, 0.3490, 0.9035, 0.7904, 0.4856, 0.1117, 0.3851, 0.7932, 0.9066, 0.3630, 0.2709, 0.8578, 0.2255, 0.3292], device='cuda:0') --------------------------------------------------------------------------- RuntimeError Traceback (most recent call last) in () 24 print(labels) 25 outputs = model(input_ids=input_ids, attention_mask=attention_mask, ---> 26 token_type_ids=token_type_ids, labels=labels) 27 loss = outputs[0] 28 loss.backward() 5 frames /usr/local/lib/python3.6/dist-packages/torch/nn/functional.py in nll_loss(input, target, weight, size_average, ignore_index, reduce, reduction) 2262 .format(input.size(0), target.size(0))) 2263 if dim == 2: -> 2264 ret = torch._C._nn.nll_loss(input, target, weight, _Reduction.get_enum(reduction), ignore_index) 2265 elif dim == 4: 2266 ret = torch._C._nn.nll_loss2d(input, target, weight, _Reduction.get_enum(reduction), ignore_index) RuntimeError: Expected object of scalar type Long but got scalar type Float for argument #2 'target' in call to _thnn_nll_loss_forward ``` Any help is appreciated! ---
**Author:** ![rgwatwormhill](https://avatars.discourse-cdn.com/v4/letter/r/a9adbd/32.png) [@rgwatwormhill](https://discuss.huggingface.co/u/rgwatwormhill)\ **Post date:** [January 23, 2021, 6:15pm UTC](https://discuss.huggingface.co/t/fine-tune-bert-and-camembert-for-regression-problem/332/7 "2021-01-23T18:15:23Z")
[I’m guessing here] ‘target’ is probably ‘labels’. What type are your labels initially? You could try specifying their type explicitly, using something like > tb\_labels = batch[2].to(device, dtype = torch.float) Try this link [python - RuntimeError: expected scalar type Long but found Float - Stack Overflow](https://stackoverflow.com/questions/60440292/runtimeerror-expected-scalar-type-long-but-found-float) ---
**Author:** ![thecity2](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thecity2/32/947_2.png) [@thecity2](https://discuss.huggingface.co/u/thecity2)\ **Post date:** [January 23, 2021, 10:44pm UTC](https://discuss.huggingface.co/t/fine-tune-bert-and-camembert-for-regression-problem/332/8 "2021-01-23T22:44:26Z")
The targets are floats. I’ll try this, thanks for the suggestion! ---
**Author:** ![thecity2](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thecity2/32/947_2.png) [@thecity2](https://discuss.huggingface.co/u/thecity2)\ **Post date:** [January 25, 2021, 8:41pm UTC](https://discuss.huggingface.co/t/fine-tune-bert-and-camembert-for-regression-problem/332/9 "2021-01-25T20:41:20Z")
@rgwatwormhill Shoot, that didn’t work unfortunately. ---
**Author:** ![thecity2](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thecity2/32/947_2.png) [@thecity2](https://discuss.huggingface.co/u/thecity2)\ **Post date:** [January 25, 2021, 9:18pm UTC](https://discuss.huggingface.co/t/fine-tune-bert-and-camembert-for-regression-problem/332/10 "2021-01-25T21:18:47Z")
``` r""" labels (:obj:`torch.LongTensor` of shape :obj:`(batch_size,)`, `optional`): Labels for computing the sequence classification/regression loss. Indices should be in :obj:`[0, ..., config.num_labels - 1]`. If :obj:`config.num_labels == 1` a regression loss is computed (Mean-Square loss), If :obj:`config.num_labels > 1` a classification loss is computed (Cross-Entropy). """ ``` I have a couple of questions. The first is it seems odd that `torch.LongTensor` would make sense for a regression problem. Not sure how that is compatible. Second question is would I need to add an additional layer to BERT to be able to do regression? Or remove the logits layer and replace it? ---
**Author:** ![valhalla](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/valhalla/32/4179_2.png) [@valhalla](https://discuss.huggingface.co/u/valhalla)\ **Post date:** [January 26, 2021, 6:52am UTC](https://discuss.huggingface.co/t/fine-tune-bert-and-camembert-for-regression-problem/332/11 "2021-01-26T06:52:10Z")
looks like the model is initialized incorrectly, for regression we need use `num_labels=1`, and you can do it using two ways ```auto config = BertConfig.from("...", num_labels=1) model = BertForSequenceClassificatio.from_pretrained("...", config=config) ``` or ```auto model = BertForSequenceClassificatio.from_pretrained("...", num_labels=1) ``` creating the model from config and the again using `from_pretrained` will override the config params. So in your code the model still has `num_labels=2` > The first is it seems odd that `torch.LongTensor` would make sense for a regression problem Yes, the doscstring should be corrected. But you can still pass float tensor for a regression problem. ---
**Author:** ![thecity2](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thecity2/32/947_2.png) [@thecity2](https://discuss.huggingface.co/u/thecity2)\ **Post date:** [January 26, 2021, 4:05pm UTC](https://discuss.huggingface.co/t/fine-tune-bert-and-camembert-for-regression-problem/332/12 "2021-01-26T16:05:54Z")
@valhalla Thank you so much! Such a small detail haha. Got it working. ---
**Author:** ![thecity2](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thecity2/32/947_2.png) [@thecity2](https://discuss.huggingface.co/u/thecity2)\ **Post date:** [January 28, 2021, 4:17pm UTC](https://discuss.huggingface.co/t/fine-tune-bert-and-camembert-for-regression-problem/332/13 "2021-01-28T16:17:50Z")
Well I got it “working” in that there are no errors now, but surprisingly to me, I am seeing the validation loss increase with every epoch. I’m not sure what I’m doing wrong. ``` device = torch.device('cuda') if torch.cuda.is_available() else torch.device('cpu') print(f'Device used for training: {device}') model = DistilBertForSequenceClassification.from_pretrained('distilbert-base-uncased', num_labels=1) model.to(device) model.train() optim = AdamW(model.parameters(), lr=5e-5) for epoch in range(10): for batch in tqdm.notebook.tqdm(train_dataloader): optim.zero_grad() input_ids = batch['input_ids'].to(device) attention_mask = batch['attention_mask'].to(device) # token_type_ids = batch['token_type_ids'].to(device) labels = batch['labels'].to(device) outputs = model(input_ids=input_ids, attention_mask=attention_mask, labels=labels) loss = outputs[0] loss.backward() optim.step() validation_loss = validate(test_dataset, model) print(f"Validation loss in epoch {epoch}: {validation_loss}") ``` Here is my validate function: ``` def validate(test_dataset, model): torch.cuda.empty_cache() test_dataloader = torch.utils.data.DataLoader(test_dataset, batch_size=128) total_loss = 0 with torch.no_grad(): for batch in tqdm.notebook.tqdm(test_dataloader): input_ids = batch['input_ids'].to(device) attention_mask = batch['attention_mask'].to(device) # token_type_ids = batch['token_type_ids'].to(device) labels = batch['labels'].to(device) outputs = model(input_ids=input_ids, attention_mask=attention_mask, labels=labels) total_loss += outputs[0].item() return(total_loss) ``` Here is some output showing the validation loss increasing: ``` `Device used for training: cuda` 100% 2871/2871 [1:49:25<00:00, 2.29s/it] 100% 1014/1014 [09:34<00:00, 1.76it/s] Validation loss in epoch 0: 70428.39735794067 100% 2871/2871 [2:00:01<00:00, 2.51s/it] 100% 1014/1014 [09:47<00:00, 1.73it/s] Validation loss in epoch 1: 69269.67090129852 100% 2871/2871 [2:01:22<00:00, 2.54s/it] 100% 1014/1014 [10:04<00:00, 1.68it/s] Validation loss in epoch 2: 70188.32639312744 100% 2871/2871 [1:46:28<00:00, 2.23s/it] 100% 1014/1014 [09:35<00:00, 1.76it/s] Validation loss in epoch 3: 72369.78367424011 100% 2871/2871 [1:47:28<00:00, 2.25s/it] 100% 1014/1014 [10:30<00:00, 1.61it/s] Validation loss in epoch 4: 73700.79190158844 100% 2871/2871 [1:46:24<00:00, 2.22s/it] 100% 1014/1014 [09:24<00:00, 1.79it/s] Validation loss in epoch 5: 74986.73181152344 ``` ---
**Author:** ![rgwatwormhill](https://avatars.discourse-cdn.com/v4/letter/r/a9adbd/32.png) [@rgwatwormhill](https://discuss.huggingface.co/u/rgwatwormhill)\ **Post date:** [January 31, 2021, 12:14pm UTC](https://discuss.huggingface.co/t/fine-tune-bert-and-camembert-for-regression-problem/332/14 "2021-01-31T12:14:26Z")
Is the training loss also increasing? ---
**Author:** ![RylanSchaeffer](https://avatars.discourse-cdn.com/v4/letter/r/6f9a4e/32.png) [@RylanSchaeffer](https://discuss.huggingface.co/u/RylanSchaeffer)\ **Post date:** [October 3, 2021, 9:38pm UTC](https://discuss.huggingface.co/t/fine-tune-bert-and-camembert-for-regression-problem/332/15 "2021-10-03T21:38:49Z")
@valhalla can you post a code snippet showing how to set the number of output labels? ---
**Author:** ![RylanSchaeffer](https://avatars.discourse-cdn.com/v4/letter/r/6f9a4e/32.png) [@RylanSchaeffer](https://discuss.huggingface.co/u/RylanSchaeffer)\ **Post date:** [October 3, 2021, 10:55pm UTC](https://discuss.huggingface.co/t/fine-tune-bert-and-camembert-for-regression-problem/332/16 "2021-10-03T22:55:51Z")
@thecity2 can you show how you modified the loss function to something suitable for regression? ---
**Author:** ![Mahsaseifikar](https://avatars.discourse-cdn.com/v4/letter/m/9f8e36/32.png) [@Mahsaseifikar](https://discuss.huggingface.co/u/Mahsaseifikar)\ **Post date:** [November 24, 2021, 5:11pm UTC](https://discuss.huggingface.co/t/fine-tune-bert-and-camembert-for-regression-problem/332/17 "2021-11-24T17:11:44Z")
> [@rgwatwormhill](#): > > , dtype = torch.float I have a quick question, do we need to scale output for example range between [0-10] for regression problems? ---
**Author:** ![rgwatwormhill](https://avatars.discourse-cdn.com/v4/letter/r/a9adbd/32.png) [@rgwatwormhill](https://discuss.huggingface.co/u/rgwatwormhill)\ **Post date:** [December 4, 2021, 12:24pm UTC](https://discuss.huggingface.co/t/fine-tune-bert-and-camembert-for-regression-problem/332/18 "2021-12-04T12:24:32Z")
Hi @Mahsaseifikar , No. ( I don’t think so). I am not an expert, and it is a year since I last looked at any Bert code, but I don’t remember ever having to think about what scale to use for the output in my regression problem.