# Questions about my first code on fine-tuning BERT model for text-classification

**URL:** <https://discuss.huggingface.co/t/questions-about-my-first-code-on-fine-tuning-bert-model-for-text-classification/17197>\
**Category:** Beginners\
**Created:** [April 26, 2022, 8:59am UTC](https://discuss.huggingface.co/t/questions-about-my-first-code-on-fine-tuning-bert-model-for-text-classification/17197 "2022-04-26T08:59:57Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![aomar85](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/aomar85/32/12631_2.png) [@aomar85](https://discuss.huggingface.co/u/aomar85)\
**Post date:** [April 26, 2022, 8:59am UTC](https://discuss.huggingface.co/t/questions-about-my-first-code-on-fine-tuning-bert-model-for-text-classification/17197/1 "2022-04-26T08:59:57Z")

</div>

Dear All,

I am finetuing BERT model for the sequence to sequence binary text classification task for the Arabic language. For that purpose I am using **aubmindlab/bert-base-arabertv02** checkpoint .

The code is written to support multi-checkpoint usage using the **AutoTokenizer** class, and the **AutoModelForSequenceClassification** , as I am intended to experiment with more than one checkpoint. The dataset I am using is an imbalanced dataset, so I am using a custom Trainer with the weighted class as suggested by Hugggingface docs and tutorials.

**I need to understand a couple of things to make sure I am doing well.**

- In the training args I provided my Hugging face model repo to set up an early stopping callback, so every time I am doing an experiment, the model is being loaded from the repo instead of downloading the model from the provided checkpoint, please refer to the image. I am wondering does this make the bias for the results of the model after being trained?!

- As mentioned earlier, the task is binary classification with [0,1] classes. In the custom Trainer, I used **CrossEntropyLoss** to calculate the loss function, am I using the right loss function?

- I have a slightly large dataset with [1,093,402] records. I split it into Train and Validate data set using Stratified K fold because it is imbalanced into (988,789) for training, and (104,613) for validating the model. Besides, I have a small separate hold-out test dataset (15,206 records) from another source that I am using for comparing results. Does the small size of the test dataset affect the model performance, as I am not getting as good results as the validation ones? Besides, Should I use the **trainer.evaluate** or **trainer.predict** method to evaluate the final performance of the pre-trained model??

- After using CustomTrainer with weighted loss, Do I have to consider the dataset as if it is balanced when it comes to choosing the metrics (i.e accuracy), or do I have to stick with the macro f1 score because it is imbalanced?

- Does using the code implemntaion is enough toovercome overfetting ?

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/2/2ae36284068a21b6f5bdbf2073d20baa7f024186.png)

My Dataset Description :

```auto
DatasetDict({
    train: Dataset({
        features: ['Unnamed: 0', 'WikidataArabicDescrption', 'WikipediaArabicDescrption', 'label'],
        num_rows: 988789
    })
    valid: Dataset({
        features: ['Unnamed: 0', 'WikidataArabicDescrption', 'WikipediaArabicDescrption', 'label'],
        num_rows: 104613
    })
    test: Dataset({
        features: ['Unnamed: 0', 'WikidataArabicDescrption', 'WikipediaArabicDescrption', 'label'],
        num_rows: 15206
    })
})

```

My Training Args:

```auto
# Defining the TrainingArguments() arguments

logging_steps = len(my_dataset['train']) // hyperparameter_defaults['batch_size']

training_args = TrainingArguments(

   f"aomar85/WikiBERT",

   num_train_epochs = hyperparameter_defaults['num_train_epochs'],

   evaluation_strategy = hyperparameter_defaults['evaluation_strategy'],

   

    # evaluation_strategy (`str` or [`~trainer_utils.IntervalStrategy`], *optional*, defaults to `"no"`):

    # The evaluation strategy to adopt during training. Possible values are:

    # - `"no"`: No evaluation is done during training.

    # - `"steps"`: Evaluation is done (and logged) every `eval_steps`.

    # - `"epoch"`: Evaluation is done at the end of each epoch.

   eval_steps = hyperparameter_defaults['eval_steps'], # Evaluation and Save happens every 50 steps

   save_total_limit = 5, # Only last 5 models are saved. Older ones are deleted.

   logging_strategy = hyperparameter_defaults['logging_strategy'],

    # logging_steps (default 500): Number of update steps between two logs if

    # logging_strategy="steps".

   #logging_steps=logging_steps,

    # save_strategy (default "steps"):

    # The checkpoint save strategy to adopt during training. Possible values are:

    # "no": No save is done during training.

    # "epoch": Save is done at the end of each epoch.

    # "steps": Save is done every save_steps (default 500).

   save_strategy= hyperparameter_defaults['save_strategy'],

    # save_steps (default: 500): Number of updates steps before two checkpoint

    # saves if save_strategy="steps".

   save_steps=hyperparameter_defaults['save_steps'],

   #run_name = run_name,

   disable_tqdm = False, # added by me based on trainer glosses

   seed = hyperparameter_defaults['seed'], # added by me based on trainer glosses

   learning_rate=hyperparameter_defaults['learning_rate'],

   # learning_rate (default 5e-5): The initial learning rate for AdamW optimizer.

    # Adam algorithm with weight decay fix as introduced in the paper

    # Decoupled Weight Decay Regularization.

   #lr_scheduler_type = 'cosine', # added by me based on trainer glosses

   per_device_train_batch_size=hyperparameter_defaults['batch_size'],

   per_device_eval_batch_size=hyperparameter_defaults['batch_size'],

   weight_decay=hyperparameter_defaults['weight_decay'],

   

    # weight_decay (`float`, *optional*, defaults to 0):

    # The weight decay to apply (if not zero) to all layers except all bias and LayerNorm weights in [`AdamW`]

    # optimizer.

   fp16 =True, 

   push_to_hub=True,

    # load_best_model_at_end (default False): Whether or not to load the best model

    # found during training at the end of training.

   metric_for_best_model = hyperparameter_defaults['metric_for_best_model'], # 'f1' #eval_loss eval_loss

    # metric_for_best_model:

    # Use in conjunction with load_best_model_at_end to specify the metric to use

    # to compare two different models. Must be the name of a metric returned by

    # the evaluation with or without the prefix "eval_".

    #If you set this value, `greater_is_better` will default to `True`. Don't forget to set it to `False` if

    #your metric is better when lower.

   #greater_is_better = False , 

   load_best_model_at_end = True,

    #load_best_model_at_end (`bool`, *optional*, defaults to `False`):

    #Whether or not to load the best model found during training at the end of training.

   report_to="all"

    # report_to:

    # The list of integrations to report the results and logs to. Supported

    # platforms are "azure_ml", "comet_ml", "mlflow", "tensorboard" and "wandb".

    # Use "all" to report to all integrations installed, "none" for no integrations.

   )

```

The Custom Trainer :

```auto
class WeightedLossTrainer(Trainer):

    def compute_loss(self, model, inputs, return_outputs=False):

             

        # forward pass

        # Feed inputs to model and extract logits

        outputs = model(**inputs)

        logits = outputs.get("logits")

        # Extract labels

        labels = inputs.get("labels")

        # compute custom loss (suppose one has 2 labels with different weights)

        loss_func = nn.CrossEntropyLoss(weight=class_weights)

        # compute loss

        loss = loss_func(logits, labels)

        return (loss, outputs) if return_outputs else loss

```

The class weights:

```auto
class_weights= (1- (train_df['label'].value_counts().sort_index() / len(train_df['label']))).values
class_weights = torch.from_numpy(class_weights).float().to("cuda").

```

My Trainer instance:

```auto
model = AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=2)

trainer = WeightedLossTrainer(

    model,

    training_args,

    train_dataset=tokenized_datasets["train"],

    eval_dataset=tokenized_datasets["valid"],

    data_collator=data_collator,

    tokenizer=tokenizer,

    compute_metrics=compute_metrics,

    callbacks = [EarlyStoppingCallback(early_stopping_patience=3)]

)

```

Thanks in advance
