# Decoder Causal Masking \[Keras\]

**URL:** <https://discuss.huggingface.co/t/decoder-causal-masking-keras/160781>\
**Category:** Intermediate\
**Created:** [June 26, 2025, 1:52pm UTC](https://discuss.huggingface.co/t/decoder-causal-masking-keras/160781 "2025-06-26T13:52:55Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![ynaghibi](https://avatars.discourse-cdn.com/v4/letter/y/eb9ed0/32.png) [@ynaghibi](https://discuss.huggingface.co/u/ynaghibi)\
**Post date:** [June 26, 2025, 1:52pm UTC](https://discuss.huggingface.co/t/decoder-causal-masking-keras/160781/1 "2025-06-26T13:52:55Z")

</div>

[Link](https://github.com/fchollet/deep-learning-with-python-notebooks/blob/master/chapter11_part04_sequence-to-sequence-learning.ipynb)

The link shows Francois Chollet’s github implementation of the TransformerDecoder for a Seq2Seq model, which translates English phrases to Spanish ones.

The problem is that its call function of TransformerDecoder applies padding\_mask to the key/value sequence returned from the English TransformerEncoder, even though padding\_mask is based on the mask obtained from the Spanish embedding layer.  
I would have expected that the padding mask of the English embedding layer would have been used here as well.

```auto
def call(self, inputs, encoder_outputs, mask=None):
        causal_mask = self.get_causal_attention_mask(inputs)
        if mask is not None:
            padding_mask = tf.cast(
                mask[:, tf.newaxis, :], dtype="int32"
            )
            padding_mask = tf.minimum(padding_mask, causal_mask)
        else:
            padding_mask = mask
        attention_output_1 = self.attention_1(
            query=inputs, value=inputs, key=inputs, attention_mask=causal_mask
        )
        attention_output_1 = self.layernorm_1(inputs + attention_output_1)
        attention_output_2 = self.attention_2(
            query=attention_output_1, value=encoder_outputs, key=encoder_outputs, 
            attention_mask=padding_mask,
        )
        attention_output_2 = self.layernorm_2( attention_output_1 + attention_output_2 )
        proj_output = self.dense_proj(attention_output_2)
        return self.layernorm_3(attention_output_2 + proj_output)

```

---

<div class="post-metadata">

**Author:** ![Pimpcat-AU](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/pimpcat-au/32/48989_2.png) [@Pimpcat-AU](https://discuss.huggingface.co/u/Pimpcat-AU)\
**Post date:** [June 27, 2025, 12:13am UTC](https://discuss.huggingface.co/t/decoder-causal-masking-keras/160781/2 "2025-06-27T00:13:04Z")

</div>

It seems like the padding mask issue arises from the fact that the English padding mask is being overwritten by the Spanish mask. A solution could be to combine both masks instead of replacing one with the other. I would suggest trying this modification to make sure that both the English and Spanish padding masks are applied correctly.

```auto
def call(self, inputs, encoder_outputs, mask=None):
    causal_mask = self.get_causal_attention_mask(inputs)
    if mask is not None:
        padding_mask = tf.cast(mask, dtype="int32")
        combined_mask = tf.minimum(padding_mask, causal_mask)
    else:
        combined_mask = causal_mask

    attention_output_1 = self.self_attention_1(
        query=inputs, value=inputs, key=inputs, attention_mask=combined_mask
    )
    attention_output_2 = self.self_attention_2(
        inputs=attention_output_1
    )
    return self.layernorm_3(attention_output_2)

```

This should ensure both the English and Spanish masks are respected without overriding one another.

**Solution provided by Triskel Data Deterministic Ai**

---

<div class="post-metadata">

**Author:** ![ynaghibi](https://avatars.discourse-cdn.com/v4/letter/y/eb9ed0/32.png) [@ynaghibi](https://discuss.huggingface.co/u/ynaghibi)\
**Post date:** [June 27, 2025, 4:42pm UTC](https://discuss.huggingface.co/t/decoder-causal-masking-keras/160781/3 "2025-06-27T16:42:11Z")

</div>

Your code does not seem to be doing that. It also does not use any mask for the second attention layer.

I am more interested in the understanding the concept, so the solution does not have to be written with Keras. It can also be Pytorch or anything else. Finding detailed information about (causal) masks turned out to be surprisingly hard for me.

---

<div class="post-metadata">

**Author:** ![John6666](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/john6666/32/27664_2.png) [@John6666](https://discuss.huggingface.co/u/John6666)\
**Post date:** [June 28, 2025, 1:00am UTC](https://discuss.huggingface.co/t/decoder-causal-masking-keras/160781/4 "2025-06-28T01:00:00Z")

</div>

Hmm, that’s difficult… Even if it’s not word-for-word, is it okay because padding is applied or so…?🤔  
[https://ai.stackexchange.com/questions/42116/transformer-decoder-causal-masking-during-inference](https://ai.stackexchange.com/questions/42116/transformer-decoder-causal-masking-during-inference)

> **[Masking in Transformer Encoder/Decoder Models](https://sanjayasubedi.com.np/deeplearning/masking-in-attention/)**
>
> Understand why masking is needed in Transformer Encoder and Decoder networks and how they are used

---

<div class="post-metadata">

**Author:** ![ynaghibi](https://avatars.discourse-cdn.com/v4/letter/y/eb9ed0/32.png) [@ynaghibi](https://discuss.huggingface.co/u/ynaghibi)\
**Post date:** [June 30, 2025, 6:11pm UTC](https://discuss.huggingface.co/t/decoder-causal-masking-keras/160781/5 "2025-06-30T18:11:12Z")

</div>

Thanks. The blog you posted explains how causal masks work for self-attention, which is what I have already understood before. The question is how the mask looks like for the attention head that combines the encoder output with the decoder.  
I have my guesses how this should work, but I would also like to see explicit explanations about this specific topic.