# How to denoise text using T5?

**URL:** <https://discuss.huggingface.co/t/how-to-denoise-text-using-t5/38929>\
**Category:** 🤗Transformers\
**Created:** [May 6, 2023, 10:39pm UTC](https://discuss.huggingface.co/t/how-to-denoise-text-using-t5/38929 "2023-05-06T22:39:34Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Shaier](https://avatars.discourse-cdn.com/v4/letter/s/258eb7/32.png) [@Shaier](https://discuss.huggingface.co/u/Shaier)\
**Post date:** [May 6, 2023, 10:39pm UTC](https://discuss.huggingface.co/t/how-to-denoise-text-using-t5/38929/1 "2023-05-06T22:39:34Z")

</div>

I’m trying to denoise text using a T5 model following [the Huggingface doc:](https://huggingface.co/docs/transformers/v4.28.1/en/model_doc/t5#transformers.T5ForConditionalGeneration)

```auto
from transformers import T5Tokenizer, T5ForConditionalGeneration
tokenizer = T5Tokenizer.from_pretrained("t5-small")
model = T5ForConditionalGeneration.from_pretrained("t5-small")

input_ids = tokenizer("The <extra_id_0> walks in <extra_id_1> park", return_tensors="pt").input_ids
labels = tokenizer("<extra_id_0> cute dog <extra_id_1> the <extra_id_2>", return_tensors="pt").input_ids

# the forward function automatically creates the correct decoder_input_ids
loss = model(input_ids=input_ids, labels=labels).loss
loss.item()

```

But I can’t figure out how to get the actual text that corresponds to the masked input. They only show how to get the loss and mention

> the forward function automatically creates the correct decoder\_input\_ids

**I don’t care for the loss, nor do I have labels in my setting. I just have text with masked tokens that I need to fill:**

```auto
my_masked_text = [
"The kid went to the [MASK]",
"The dog likes [MASK] and also [MASK]"
]

```

---

<div class="post-metadata">

**Author:** ![nielsr](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/nielsr/32/39617_2.png) [@nielsr](https://discuss.huggingface.co/u/nielsr)\
**Post date:** [May 7, 2023, 5:31pm UTC](https://discuss.huggingface.co/t/how-to-denoise-text-using-t5/38929/2 "2023-05-07T17:31:57Z")

</div>

Hi,

That’s explained in the [docs](https://huggingface.co/docs/transformers/model_doc/t5):

```auto
from transformers import T5Tokenizer, T5ForConditionalGeneration

tokenizer = T5Tokenizer.from_pretrained("t5-small")
model = T5ForConditionalGeneration.from_pretrained("t5-small")

input_ids = tokenizer("The <extra_id_0> walks in <extra_id_1> park", return_tensors="pt").input_ids

sequence_ids = model.generate(input_ids)
sequences = tokenizer.batch_decode(sequence_ids)
sequences

```

Note that T5 doesn’t use a MASK token, but rather “sentinel tokens”, like the \<extra\_id\_0\>. See the [original T5 paper](https://arxiv.org/abs/1910.10683) for details.

---

<div class="post-metadata">

**Author:** ![Shaier](https://avatars.discourse-cdn.com/v4/letter/s/258eb7/32.png) [@Shaier](https://discuss.huggingface.co/u/Shaier)\
**Post date:** [May 8, 2023, 3:44pm UTC](https://discuss.huggingface.co/t/how-to-denoise-text-using-t5/38929/3 "2023-05-08T15:44:40Z")

</div>

The output doesn’t make any sense though:  
`['<pad><extra_id_0> park offers<extra_id_1> the<extra_id_2> park.</s>']`  
Does it thinks the sentence should be  
`The park offers walks in the park park`  
I tried other ones and it’s even worse:  
`input_ids = tokenizer("The kid went to the <extra_id_1> to buy chocolate", return_tensors="pt").input_ids`  
Output:  
`['<pad><extra_id_0> kiddie kid<extra_id_1> kiddie<extra_id_2> kiddie kid kid<extra_id_3> kiddie kid<extra_id_4>kid kid']`  
Where did all these extra IDs came from?