# Weights in BERT model

**URL:** <https://discuss.huggingface.co/t/weights-in-bert-model/26774>\
**Category:** 🤗Transformers\
**Created:** [November 26, 2022, 8:50pm UTC](https://discuss.huggingface.co/t/weights-in-bert-model/26774 "2022-11-26T20:50:30Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![minimario](https://avatars.discourse-cdn.com/v4/letter/m/e79b87/32.png) [@minimario](https://discuss.huggingface.co/u/minimario)\
**Post date:** [November 26, 2022, 8:50pm UTC](https://discuss.huggingface.co/t/weights-in-bert-model/26774/1 "2022-11-26T20:50:30Z")

</div>

I have a BERT model `model = BertForMaskedLM.from_pretrained('bert-base-uncased')`.

I notice that `model.state_dict().keys()` has 205 elements, while looking at the `[i[0] for i in list(model.named_parameters())]` only has 202 elements. The three missing elements are ‘bert.embeddings.position\_ids’, ‘cls.predictions.decoder.weight’, and ‘cls.predictions.decoder.bias’. The first is in `models.named_buffers()`, but how about the other two? What are these doing?

Another question: I noticed that `model.named_buffers()` also contains `bert.embeddings.token_type_ids`, but this is not in `model.state_dict()`. Why isn’t this the case? Shouldn’t it be important to store this in the state as well?

---

<div class="post-metadata">

**Author:** ![Kiva12138](https://avatars.discourse-cdn.com/v4/letter/k/d07c76/32.png) [@Kiva12138](https://discuss.huggingface.co/u/Kiva12138)\
**Post date:** [April 12, 2023, 3:06am UTC](https://discuss.huggingface.co/t/weights-in-bert-model/26774/2 "2023-04-12T03:06:11Z")

</div>

I have the same question.  
But I found that `cls.predictions.decoder.weight` and `bert.embeddings.word_embeddings.weight` have the same weights.