# VisionEncoderDecoder X-Attn Question

**URL:** <https://discuss.huggingface.co/t/visionencoderdecoder-x-attn-question/19276>\
**Category:** 🤗Transformers\
**Created:** [June 17, 2022, 2:51am UTC](https://discuss.huggingface.co/t/visionencoderdecoder-x-attn-question/19276 "2022-06-17T02:51:12Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![prithivida](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prithivida/32/20872_2.png) [@prithivida](https://discuss.huggingface.co/u/prithivida)\
**Post date:** [June 17, 2022, 2:51am UTC](https://discuss.huggingface.co/t/visionencoderdecoder-x-attn-question/19276/1 "2022-06-17T02:51:12Z")

</div>

@nielsr - I am trying to understand the X-attn code in VisionEncoderDecoder architecture. I can see the comment in the code `" Cross-attention layers are automatically added to the decoder and should be fine-tuned on a downstream generative task, like image captioning."`

What is the best place to see the actual implementation of the X-attn between V Encoder and L Decoder?

Please advice.

Thanks  
Prithivi

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [June 18, 2022, 8:07am UTC](https://discuss.huggingface.co/t/visionencoderdecoder-x-attn-question/19276/2 "2022-06-18T08:07:33Z")

</div>

You can see [here](https://github.com/huggingface/transformers/blob/6589e510fa4e6c442059de2fab84752535de9b23/src/transformers/models/bert/modeling_bert.py#L294) that the `BertSelfAttention` class can be used both for self-attention and cross-attention, depending on whether or not `encoder_hidden_states` are provided.

---

<div class="post-metadata">

**Author:** ![prithivida](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prithivida/32/20872_2.png) [@prithivida](https://discuss.huggingface.co/u/prithivida)\
**Post date:** [June 20, 2022, 7:30am UTC](https://discuss.huggingface.co/t/visionencoderdecoder-x-attn-question/19276/3 "2022-06-20T07:30:31Z")

</div>

Thanks @nielsr. Just to close out the loading and fine-tuning models with `VisionEcoderDecoder` only finetunes the X-Attn layers - is my assumption right? if not is there an explicit way to freeze the pretrained vision and language models?

Please advice

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [June 20, 2022, 8:09am UTC](https://discuss.huggingface.co/t/visionencoderdecoder-x-attn-question/19276/4 "2022-06-20T08:09:41Z")

</div>

No, all parameters get updated. It’s just that the weights of the cross-attention layers are the only ones that start from scratch, all other weights are already pre-trained, and those get fine-tuned.

---

<div class="post-metadata">

**Author:** ![prithivida](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/prithivida/32/20872_2.png) [@prithivida](https://discuss.huggingface.co/u/prithivida)\
**Post date:** [June 20, 2022, 9:15am UTC](https://discuss.huggingface.co/t/visionencoderdecoder-x-attn-question/19276/5 "2022-06-20T09:15:16Z")

</div>

Ok thank you