# Self-attention masking for T5 encoder?

**URL:** <https://discuss.huggingface.co/t/self-attention-masking-for-t5-encoder/15182>\
**Category:** 🤗Transformers\
**Created:** [February 27, 2022, 6:00pm UTC](https://discuss.huggingface.co/t/self-attention-masking-for-t5-encoder/15182 "2022-02-27T18:00:19Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![ronen](https://avatars.discourse-cdn.com/v4/letter/r/838e76/32.png) [@ronen](https://discuss.huggingface.co/u/ronen)\
**Post date:** [February 27, 2022, 6:00pm UTC](https://discuss.huggingface.co/t/self-attention-masking-for-t5-encoder/15182/1 "2022-02-27T18:00:19Z")

</div>

Hi,  
I’m interested in applying self-attention masking in the T5 encoder, meaning a tensor of shape (batch, max\_input\_len, max\_input\_len), which is a binary matrix for each tensor in the batch, specifying which tokens (i,j) in the input can attend to each other.  
This idea is explored for a different Transformer architecture in the paper “Improving Compositional Generalization in Classification Tasks via Structure Annotations”. It’s implemented in TF, the figure below is from that paper:

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/2X/f/f83e202c75c2421095b44d67533f23c6257dacb9.png)

In contrast, I’d like to use the Transformers PyTorch T5 implementation if possible. The option seems to be available for BERT:

> <https://github.com/huggingface/transformers/blob/main/src/transformers/models/bert/modeling_bert.py#L261>

But for T5 I haven’t been able to find an equivalent- the `encoder_attention_mask` there is of dimension (batch\_size, encoder\_seq\_len):

> <https://github.com/huggingface/transformers/blob/84eaa6acf582206dba33135727dc3bfff05a7e9c/src/transformers/models/t5/modeling_t5.py#L945>

Anyone have experience with this?  
Thanks!