# Self-attention extraction from Long T5

**URL:** <https://discuss.huggingface.co/t/self-attention-extraction-from-long-t5/75982>\
**Category:** 🤗Transformers\
**Created:** [March 5, 2024, 8:05am UTC](https://discuss.huggingface.co/t/self-attention-extraction-from-long-t5/75982 "2024-03-05T08:05:17Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![giuseppe-trimigno](https://avatars.discourse-cdn.com/v4/letter/g/3e96dc/32.png) [@giuseppe-trimigno](https://discuss.huggingface.co/u/giuseppe-trimigno)\
**Post date:** [March 5, 2024, 8:05am UTC](https://discuss.huggingface.co/t/self-attention-extraction-from-long-t5/75982/1 "2024-03-05T08:05:17Z")

</div>

Hi there, I’m trying to extract the encoder self-attention weights from Long T5 (using the `LongT5Encoder` model).  
I can get encoder attention weights by adding the `output_attentions=True` arg in the forward method, but I’m having some problem in understanding the output, which is a tensor of shape `(batch_size, num_layers, num_heads, 128, 3*128 + K)`, where I think that K depends on the sequence length, while 128 should be the `relative_attention_max_distance`.  
When doing that for BERT-like models, last two dimensions are a squared matrix (for each column, i.e. for each token, we have the attention score respect to all other tokens), which is what I want. But, how can I achieve this with Long T5 model?  
Thanks in advance!