# Which parameter is causing the decrease in Learning rate every epoch?

**URL:** <https://discuss.huggingface.co/t/which-parameter-is-causing-the-decrease-in-learning-rate-every-epoch/13015>\
**Category:** Beginners\
**Created:** [December 21, 2021, 6:53pm UTC](https://discuss.huggingface.co/t/which-parameter-is-causing-the-decrease-in-learning-rate-every-epoch/13015 "2021-12-21T18:53:17Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![NDugar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ndugar/32/40501_2.png) [@NDugar](https://discuss.huggingface.co/u/NDugar)\
**Post date:** [December 21, 2021, 6:53pm UTC](https://discuss.huggingface.co/t/which-parameter-is-causing-the-decrease-in-learning-rate-every-epoch/13015/1 "2021-12-21T18:53:17Z")

</div>

Hey, I have been trying to train my model on mnli and the learning rate seems to keep decreasing for no reason. Can someone help me? -

```auto
train_args = TrainingArguments(
    output_dir=f'./resultsv3/output',
    logging_dir=f'./resultsv3/output/logs',
    learning_rate=3e-6,
    per_device_train_batch_size=4,
    per_device_eval_batch_size=4,
    num_train_epochs=4,
    load_best_model_at_end=True,
    metric_for_best_model="accuracy",
    fp16=True,
    fp16_full_eval=True,
    evaluation_strategy="epoch",
    save_strategy = "epoch",
    save_total_limit=5,
    logging_strategy="epoch",
    report_to="all")

def compute_metrics(eval_pred):
	predictions, labels = eval_pred
	predictions = np.argmax(predictions, axis=1)
	return metric.compute(predictions=predictions, references=labels)

trainer = Trainer( 
    model=model,
    tokenizer=tokenizer,
    args=train_args,
    data_collator=data_collator,
    train_dataset=encoded_dataset_train,  
    eval_dataset=encoded_dataset_test,
    compute_metrics=compute_metrics
)

```

which parameter is causing the decrease in Learning rate every epoch?

---

<div class="post-metadata">

**Author:** ![osanseviero](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/osanseviero/32/38738_2.png) [@osanseviero](https://discuss.huggingface.co/u/osanseviero)\
**Post date:** [December 21, 2021, 9:38pm UTC](https://discuss.huggingface.co/t/which-parameter-is-causing-the-decrease-in-learning-rate-every-epoch/13015/2 "2021-12-21T21:38:15Z")

</div>

The `learning_rate` parameter is just the initial learning rate, but it is usually changed during training.

You can find the default values of `TrainingArguments` at [Trainer](https://huggingface.co/docs/transformers/master/main_classes/trainer#transformers.TrainingArguments.lr_scheduler_type). You can see that `lr_scheduler_type` is `linear` by default.

As specified in its [documentation([Optimization](https://huggingface.co/docs/transformers/master/en/main_classes/optimizer_schedules#transformers.get_linear_schedule_with_warmup)), linear creates a schedule with a learning rate that decreases linearly from the initial learning rate after an initial warmup period.

---

<div class="post-metadata">

**Author:** ![NDugar](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/ndugar/32/40501_2.png) [@NDugar](https://discuss.huggingface.co/u/NDugar)\
**Post date:** [December 21, 2021, 9:53pm UTC](https://discuss.huggingface.co/t/which-parameter-is-causing-the-decrease-in-learning-rate-every-epoch/13015/3 "2021-12-21T21:53:08Z")

</div>

Ok thank you for your answer. Can tell me whether the way  
mentioned here [microsoft/deberta-v2-xxlarge-mnli · Hugging Face](https://huggingface.co/microsoft/deberta-v2-xxlarge-mnli#fine-tuning-on-nlu-tasks) also uses linear as scheduler type?