# Regarding Training a Task Specific Knowledge Distillation model

**URL:** <https://discuss.huggingface.co/t/regarding-training-a-task-specific-knowledge-distillation-model/16798>\
**Category:** 🤗Transformers\
**Created:** [April 13, 2022, 2:20pm UTC](https://discuss.huggingface.co/t/regarding-training-a-task-specific-knowledge-distillation-model/16798 "2022-04-13T14:20:48Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![pchhapolika](https://avatars.discourse-cdn.com/v4/letter/p/b9e5f3/32.png) [@pchhapolika](https://discuss.huggingface.co/u/pchhapolika)\
**Post date:** [April 13, 2022, 2:20pm UTC](https://discuss.huggingface.co/t/regarding-training-a-task-specific-knowledge-distillation-model/16798/1 "2022-04-13T14:20:48Z")

</div>

I was referring to this code:

> <https://github.com/philschmid/knowledge-distillation-transformers-pytorch-sagemaker/blob/master/knowledge-distillation.ipynb>

From @philschmid

I could follow most of the code, but had few doubts. Please help me to clarify these doubts.

In this code below:

```auto
class DistillationTrainer(Trainer):
    def __init__ (self, *args, teacher_model=None, **kwargs):
        super(). __init__ (*args, **kwargs)
        
        self.teacher = teacher_model
        
        # place teacher on same device as student
        self._move_model_to_device(self.teacher,self.model.device)
        
        self.teacher.eval()

```

When I take fine-tuned `teacher model` it is never fine-tuned in the process of Task Specific Distillation training, as in line ` self.teacher.eval()` mentioned in the code.? Only the output of `teacher model` is considered for loss calculations.

I couldn’t follow this line `self._move_model_to_device(self.teacher,self.model.device)`. What it is actually doing?

In Task Specific Distillation training, I am fine tuning my student model, but in the `DistillationTrainer` I pass both models. **Where it’s making sure that only student model weights are learned and not the teacher?**

```auto
trainer = DistillationTrainer(
    student_model,
    training_args,
    teacher_model=teacher_model,
    train_dataset=train_data,
    eval_dataset=val_data,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics,
    
)

```

---

<div class="post-metadata">

**Author:** ![NimaBoscarino](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nimaboscarino/32/4358_2.png) [@NimaBoscarino](https://discuss.huggingface.co/u/NimaBoscarino)\
**Post date:** [April 13, 2022, 6:57pm UTC](https://discuss.huggingface.co/t/regarding-training-a-task-specific-knowledge-distillation-model/16798/2 "2022-04-13T18:57:32Z")

</div>

As far as I can tell, the student model is the one being passed to the `Trainer`'s `model` property, and when `trainer.train()` gets called the Trainer will only look into its `model` to adjust the weights. As you pointed out, the teacher model is being set to `eval` mode, and it’s only being used in the overridden `compute_loss` function. (More info about `compute_loss` here: [Trainer](https://huggingface.co/docs/transformers/main/en/main_classes/trainer#transformers.Trainer.compute_loss)) The `DistillationTrainer` class is just a custom subclass, and the `teacher_model` won’t actually get passed into the `Trainer`.

I think `self._move_model_to_device(self.teacher,self.model.device)` just sets the teacher model to use GPU if it’s available (or at least puts it on whatever device the student model is on), since the Trainer class already does that automatically for the student model when it’s passed in: [transformers/trainer.py at c4ad38e5ac69e6d96116f39df789a2369dd33c21 · huggingface/transformers · GitHub](https://github.com/huggingface/transformers/blob/c4ad38e5ac69e6d96116f39df789a2369dd33c21/src/transformers/trainer.py#L377)

Hope this helps!

---

<div class="post-metadata">

**Author:** ![pchhapolika](https://avatars.discourse-cdn.com/v4/letter/p/b9e5f3/32.png) [@pchhapolika](https://discuss.huggingface.co/u/pchhapolika)\
**Post date:** [April 14, 2022, 10:34am UTC](https://discuss.huggingface.co/t/regarding-training-a-task-specific-knowledge-distillation-model/16798/3 "2022-04-14T10:34:26Z")

</div>

Yes, it helps. Few more clarification on your answer!

But in the `class DistillationTrainer(Trainer):` both models are passed, if you see here

```auto
trainer = DistillationTrainer(
    **student_model,**
    training_args,
    **teacher_model=teacher_model,**
    train_dataset=train_data,
    eval_dataset=val_data,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics,
    
)

```

Why you say that \<\< the student model is the one being passed to the `Trainer` 's `model` property,\>\>

And teacher model is just for purpose to calculate loss?

---

<div class="post-metadata">

**Author:** ![pchhapolika](https://avatars.discourse-cdn.com/v4/letter/p/b9e5f3/32.png) [@pchhapolika](https://discuss.huggingface.co/u/pchhapolika)\
**Post date:** [April 14, 2022, 1:18pm UTC](https://discuss.huggingface.co/t/regarding-training-a-task-specific-knowledge-distillation-model/16798/4 "2022-04-14T13:18:58Z")

</div>

And one more basic doubt is:

In this code:

```auto
 loss_function = nn.KLDivLoss(reduction="batchmean")
        loss_logits = (loss_function
                       (
            F.log_softmax(outputs_student.logits / self.args.temperature, dim=-1),
            F.softmax(outputs_teacher.logits / self.args.temperature, dim=-1)
                       ) * (self.args.temperature ** 2)
                      )

```

Why are we using `log_softmax` for student and `softmax` for teacher?  
Also what is the purpose of `KLDivLoss?` What it is doing?

---

<div class="post-metadata">

**Author:** ![NimaBoscarino](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nimaboscarino/32/4358_2.png) [@NimaBoscarino](https://discuss.huggingface.co/u/NimaBoscarino)\
**Post date:** [April 14, 2022, 4:44pm UTC](https://discuss.huggingface.co/t/regarding-training-a-task-specific-knowledge-distillation-model/16798/5 "2022-04-14T16:44:04Z")

</div>

> [@pchhapolika](#):
>
> Why you say that \<\< the student model is the one being passed to the `Trainer` 's `model` property

The `DistillationTrainer` is a new custom class that’s being created in your notebook, which is subclassing the `Trainer` class (which is from Hugging Face’s `transformers`). So even though you pass both the `student_model` and `teacher_model` to the `DistillationTrainer`, note that this section of the code:

```python
class DistillationTrainer(Trainer):
    def __init__ (self, *args, teacher_model=None, **kwargs):
        super(). __init__ (*args, **kwargs)
        self.teacher = teacher_model
        # etc...

```

means that the teacher model is being saved in the `DistillationTrainer` as `self.teacher`, while the student model is being passed up through to the `Trainer`’s init function as the first parameter when you run:

```python
trainer = DistillationTrainer(
    student_model, # This is a positional argument, captured in *args and passed to super(). __init__
    training_args,
    teacher_model=teacher_model,
    train_dataset=train_data,
    eval_dataset=val_data,
    tokenizer=tokenizer,
    compute_metrics=compute_metrics,
)

```

If we look at where `self.teacher` gets used, it’s actually only called in the `compute_loss()` function. (The `Trainer` class doesn’t use a `self.teacher` property)

Are you familiar with [subclasses and `super` in Python](https://www.digitalocean.com/community/tutorials/understanding-class-inheritance-in-python-3)? If not, reading into that might help a bit!

As for the other questions, they’re out of my expertise but this GitHub issue seems to explain it pretty well! ([kd loss · Issue #2 · haitongli/knowledge-distillation-pytorch · GitHub](https://github.com/peterliht/knowledge-distillation-pytorch/issues/2)) Here’s a Reddit thread specifically about KLDivLoss, which seems to be commonly used in knowledge distillation: [Reddit - Dive into anything](https://www.reddit.com/r/MachineLearning/comments/ke1r2q/r_why_do_knowledge_distillation_papers_always_use/)

This is pretty neat, I’ve learned a lot through researching for your question 🤗

---

<div class="post-metadata">

**Author:** ![pchhapolika](https://avatars.discourse-cdn.com/v4/letter/p/b9e5f3/32.png) [@pchhapolika](https://discuss.huggingface.co/u/pchhapolika)\
**Post date:** [April 15, 2022, 8:00am UTC](https://discuss.huggingface.co/t/regarding-training-a-task-specific-knowledge-distillation-model/16798/6 "2022-04-15T08:00:13Z")

</div>

I learned a lot through your answers!! Thanks.

---

<div class="post-metadata">

**Author:** ![pchhapolika](https://avatars.discourse-cdn.com/v4/letter/p/b9e5f3/32.png) [@pchhapolika](https://discuss.huggingface.co/u/pchhapolika)\
**Post date:** [June 28, 2022, 7:38am UTC](https://discuss.huggingface.co/t/regarding-training-a-task-specific-knowledge-distillation-model/16798/7 "2022-06-28T07:38:20Z")

</div>

Hi @NimaBoscarino , **Why in the paper they mention that** \<\<In our experiments, we have observed that dis- tilled models do not work well when distilled to a different model type. Therefore, we restricted our setup to avoid distilling RoBERTa model to BERT or vice versa. The major difference between the two model groups is the input token (sub-word) em- bedding. We think that different input embedding spaces result in different output embedding spaces, and knowledge transfer with different spaces does not work well\>\>

At the end, we are only taking KLDiversion Loss between 2 dimensional logit vector, right ? So can’t we use **Roberta XLM large model as teacher** and **Bert base as student**. Both have different tokenization. Or we can still push our code to train whatever we want, but technically its not wise to do this?

---

<div class="post-metadata">

**Author:** ![NimaBoscarino](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nimaboscarino/32/4358_2.png) [@NimaBoscarino](https://discuss.huggingface.co/u/NimaBoscarino)\
**Post date:** [June 30, 2022, 8:39pm UTC](https://discuss.huggingface.co/t/regarding-training-a-task-specific-knowledge-distillation-model/16798/8 "2022-06-30T20:39:40Z")

</div>

Oooh, I don’t know enough about that to comment on it 😬 I think that what they mean in the paper is that you could use those as teacher + student, but you might not be super impressed with the results. That’s my intuition though, so I might be wrong!

---

<div class="post-metadata">

**Author:** ![Venkatesh4342](https://avatars.discourse-cdn.com/v4/letter/v/a9a28c/32.png) [@Venkatesh4342](https://discuss.huggingface.co/u/Venkatesh4342)\
**Post date:** [September 2, 2023, 1:15pm UTC](https://discuss.huggingface.co/t/regarding-training-a-task-specific-knowledge-distillation-model/16798/9 "2023-09-02T13:15:17Z")

</div>

can anyone help to perform task specific knowledge distillation on NER task.