# Carrying Gradients Through Generate

**URL:** <https://discuss.huggingface.co/t/carrying-gradients-through-generate/301>\
**Category:** Research\
**Created:** [July 15, 2020, 11:45am UTC](https://discuss.huggingface.co/t/carrying-gradients-through-generate/301 "2020-07-15T11:45:52Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![chrisdoyleIE](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/chrisdoyleie/32/2546_2.png) [@chrisdoyleIE](https://discuss.huggingface.co/u/chrisdoyleIE)\
**Post date:** [July 15, 2020, 11:45am UTC](https://discuss.huggingface.co/t/carrying-gradients-through-generate/301/1 "2020-07-15T11:45:52Z")

</div>

Hi folks,

How would you best recommend that I pass gradients through generate? below is a rough code snippet explaining the objective.

I am thinking that I could take the hypo\_ids directly from the model output (instead of from generate), but this is no longer natural because teacher-forcing is used to generate these.

Thoughts?

Context from Pytorch Lightning Implementation:

```python

# self.model = BartForConditionalGeneration("facebook/bart-base")

def forward(self, batch, batch_id):
    return self.model(input_ids = batch["x"], decoder_inputs=["decoder_inputs"], decoder_labels = ["decoder_labels"] )

def training_step(self, batch, batch_id)
   """Want two losses, language modelling loss and semantic similarity loss"""
    
    # language modelling loss
    outputs = self(batch)[0]
    language_modelling_loss = outputs[0]
    
    # semantic similarity loss
    target_ids = batch["target_ids"]
    hypo_ids = self.model.generate(batch["x"]) # no gradients passed of course
    semsim_loss = 1 - nn.CosineSimilarity(dim=0)(target_ids, hypo_ids)

   return {"loss": language_modelling_loss + semsim_loss}
    
    

```

---

<div class="post-metadata">

**Author:** ![chrisdoyleIE](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/chrisdoyleie/32/2546_2.png) [@chrisdoyleIE](https://discuss.huggingface.co/u/chrisdoyleIE)\
**Post date:** [July 16, 2020, 11:16am UTC](https://discuss.huggingface.co/t/carrying-gradients-through-generate/301/2 "2020-07-16T11:16:02Z")

</div>

EDIT: The only method seems to be to use RL to simulate the sampling that occurs.

see [https://papers.nips.cc/paper/8682-training-language-gans-from-scratch.pdf](https://papers.nips.cc/paper/8682-training-language-gans-from-scratch.pdf)

---

<div class="post-metadata">

**Author:** ![sshleifer](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/sshleifer/32/3304_2.png) [@sshleifer](https://discuss.huggingface.co/u/sshleifer)\
**Post date:** [July 16, 2020, 5:14pm UTC](https://discuss.huggingface.co/t/carrying-gradients-through-generate/301/3 "2020-07-16T17:14:36Z")

</div>

@yjernite is also interested in this line of work.  
I would write a method similar to parlai’s [`decode_forced`](https://github.com/facebookresearch/ParlAI/blob/0292f5765a50589d3de0afa90eab403ebcb967f8/parlai/core/torch_generator_agent.py#L147-L185)

that forces the model to decode the tgt sequence and estimates its probability, then backprob the sum of the GT sequence. I’m not sure if that will lead to super similar results to the current teacher-forcing training approach, but it would be interesting to test!

---

<div class="post-metadata">

**Author:** ![chrisdoyleIE](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/chrisdoyleie/32/2546_2.png) [@chrisdoyleIE](https://discuss.huggingface.co/u/chrisdoyleIE)\
**Post date:** [August 6, 2020, 5:06pm UTC](https://discuss.huggingface.co/t/carrying-gradients-through-generate/301/4 "2020-08-06T17:06:36Z")

</div>

I just tried a simple ffnn to replicate argmax, but found that the gradients are almost always zero which makes sense I guess - changing other vector values will almost never change the maximum value.

---

<div class="post-metadata">

**Author:** ![patrickvonplaten](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/patrickvonplaten/32/2171_2.png) [@patrickvonplaten](https://discuss.huggingface.co/u/patrickvonplaten)\
**Post date:** [November 2, 2020, 12:55pm UTC](https://discuss.huggingface.co/t/carrying-gradients-through-generate/301/5 "2020-11-02T12:55:18Z")

</div>

This should also be interesting: [Big `generate()` refactor](https://discuss.huggingface.co/t/big-generate-refactor/1857)

---

<div class="post-metadata">

**Author:** ![eduardoprea44](https://avatars.discourse-cdn.com/v4/letter/e/ea666f/32.png) [@eduardoprea44](https://discuss.huggingface.co/u/eduardoprea44)\
**Post date:** [January 29, 2023, 10:57pm UTC](https://discuss.huggingface.co/t/carrying-gradients-through-generate/301/7 "2023-01-29T22:57:54Z")

</div>

Hello,  
I’m trying to do something similar. Did you manage to implement something working?