# Clarify BERT model learnable parameters

**URL:** <https://discuss.huggingface.co/t/clarify-bert-model-learnable-parameters/11018>\
**Category:** 🤗Transformers\
**Created:** [October 23, 2021, 3:12pm UTC](https://discuss.huggingface.co/t/clarify-bert-model-learnable-parameters/11018 "2021-10-23T15:12:19Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![hfnlpmb](https://avatars.discourse-cdn.com/v4/letter/h/ecd19e/32.png) [@hfnlpmb](https://discuss.huggingface.co/u/hfnlpmb)\
**Post date:** [October 23, 2021, 3:12pm UTC](https://discuss.huggingface.co/t/clarify-bert-model-learnable-parameters/11018/1 "2021-10-23T15:12:19Z")

</div>

Hello,

I want to use HateBERT from the paper’s [repository](https://osf.io/tbd58/?view_only=d90e681c672a494bb555de99fc7ae780), which is a BERT model that its pre-training was extended in abusive language.

In order to do that, I created a BERT model (`bert-base-uncased` in `PyTorch`) and tried to load HateBERT’s weights with `load_state_dict()` (after having made minor changes in parameters names, to match BERT’s).

`load_state_dict()` throws the error:

```auto
RuntimeError: Error(s) in loading state_dict for BertModel:
	Missing key(s) in state_dict: "embeddings.position_ids".

```

which means that the BERT model requires `embeddings.position_ids`. I checked this tensor and it is not a `PyTorch` parameters tensor, just a tensor. From the error message, it is also evident that all other parameters match, otherwise their names would also be mentioned. Can someone explain?

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [October 23, 2021, 5:39pm UTC](https://discuss.huggingface.co/t/clarify-bert-model-learnable-parameters/11018/2 "2021-10-23T17:39:28Z")

</div>

HateBERT is available on the hub. [GroNLP/hateBERT · Hugging Face](https://huggingface.co/GroNLP/hateBERT)

```auto
from transformers import AutoTokenizer, AutoModelForMaskedLM
  
tokenizer = AutoTokenizer.from_pretrained("GroNLP/hateBERT")

model = AutoModelForMaskedLM.from_pretrained("GroNLP/hateBERT")

```

---

<div class="post-metadata">

**Author:** ![hfnlpmb](https://avatars.discourse-cdn.com/v4/letter/h/ecd19e/32.png) [@hfnlpmb](https://discuss.huggingface.co/u/hfnlpmb)\
**Post date:** [November 4, 2021, 10:11am UTC](https://discuss.huggingface.co/t/clarify-bert-model-learnable-parameters/11018/3 "2021-11-04T10:11:37Z")

</div>

Thank you @BramVanroy, and sorry for my late response.

Is there a way to know that the Hugging Face library has that model, and do not make the OP in the first place? The question is to avoid future posts, and make a better search in advance.

---

<div class="post-metadata">

**Author:** ![BramVanroy](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/bramvanroy/32/47360_2.png) [@BramVanroy](https://discuss.huggingface.co/u/BramVanroy)\
**Post date:** [November 4, 2021, 1:54pm UTC](https://discuss.huggingface.co/t/clarify-bert-model-learnable-parameters/11018/4 "2021-11-04T13:54:06Z")

</div>

Sure, you can look for a model on the hub.

> **[Models - Hugging Face](https://huggingface.co/models)**
>
> We’re on a journey to advance and democratize artificial intelligence through open source and open science.

---

<div class="post-metadata">

**Author:** ![hfnlpmb](https://avatars.discourse-cdn.com/v4/letter/h/ecd19e/32.png) [@hfnlpmb](https://discuss.huggingface.co/u/hfnlpmb)\
**Post date:** [November 4, 2021, 2:29pm UTC](https://discuss.huggingface.co/t/clarify-bert-model-learnable-parameters/11018/5 "2021-11-04T14:29:46Z")

</div>

Thank you @BramVanroy!