# How to pre-train a model using a custom mask strategy?

**URL:** <https://discuss.huggingface.co/t/how-to-pre-train-a-model-using-a-custom-mask-strategy/17447>\
**Category:** 🤗Transformers\
**Created:** [May 2, 2022, 8:44pm UTC](https://discuss.huggingface.co/t/how-to-pre-train-a-model-using-a-custom-mask-strategy/17447 "2022-05-02T20:44:26Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![vabatista](https://avatars.discourse-cdn.com/v4/letter/v/47e85d/32.png) [@vabatista](https://discuss.huggingface.co/u/vabatista)\
**Post date:** [May 2, 2022, 8:44pm UTC](https://discuss.huggingface.co/t/how-to-pre-train-a-model-using-a-custom-mask-strategy/17447/1 "2022-05-02T20:44:26Z")

</div>

Is there any way to pre-train a model (i.e. RoBERTa), using my own corpus, but with a custom mask strategy? For instance, I want to look up a dictionary to choose which words to mask.