# Pre-trained DeBERTa

**URL:** <https://discuss.huggingface.co/t/pre-trained-deberta/43203>\
**Category:** 🤗Transformers\
**Created:** [June 14, 2023, 10:30am UTC](https://discuss.huggingface.co/t/pre-trained-deberta/43203 "2023-06-14T10:30:12Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mintbach](https://avatars.discourse-cdn.com/v4/letter/m/35a633/32.png) [@Mintbach](https://discuss.huggingface.co/u/Mintbach)\
**Post date:** [June 14, 2023, 10:30am UTC](https://discuss.huggingface.co/t/pre-trained-deberta/43203/1 "2023-06-14T10:30:12Z")

</div>

Hi,

I wanted to use DeBERTa.  
Somehow the preview of its unmasking abilities seems very bad.

 ![image](https://us1.discourse-cdn.com/hellohellohello/original/3X/b/2/b287438c935d61369dc9aa5f43a84d7ba358269f.png)

I looked at the source code and cannot see the addition of the absolute positions.

Can someone explain me why the model performs so bad at MLM preview.  
Maybe I overlooked the addition of the absolute positions in the source code.  
An explanation of the implementation would be really helpful aswell!

Thank you!  
Stephan