# Include more features per token while training the BERT model

**URL:** <https://discuss.huggingface.co/t/include-more-features-per-token-while-training-the-bert-model/103333>\
**Category:** Intermediate\
**Created:** [August 21, 2024, 1:34pm UTC](https://discuss.huggingface.co/t/include-more-features-per-token-while-training-the-bert-model/103333 "2024-08-21T13:34:12Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![gsarda](https://avatars.discourse-cdn.com/v4/letter/g/278dde/32.png) [@gsarda](https://discuss.huggingface.co/u/gsarda)\
**Post date:** [August 21, 2024, 1:34pm UTC](https://discuss.huggingface.co/t/include-more-features-per-token-while-training-the-bert-model/103333/1 "2024-08-21T13:34:12Z")

</div>

My data is in this format for representing journey of one user per row.

Item 1, Item 2, Item 3, Item 7, Item 10  
Item Description 1, Item Description 2…Item Descripiton 10  
Item 1 purchased on Monday, Item 2 purchased on Tuesday, …Item 10 purchased on Tuesday…  
Item 1 Price, Item 2 Price…Item 10 Price

Consider for 10 users, you shall have 10 such rows of journey which has Item list, Item Description list, Item Purchased day list, Item price list.

Currently, we have trained MLM on data of all users with only item list using BertForMaskedLM from huggingface transformers library  
Journey is transformed to: Item 1, Item 2, Item 3, [MASK], Item 10  
And then in inferencing we try to predict the next item the user should select by providing his history to the model.

Now, we want to include description, price, day of purchase etc as more features. These features are per token and not per user.

Need help to understand the architecture and code changes for custom BERT library.