# Tokenizing using JS

**URL:** <https://discuss.huggingface.co/t/tokenizing-using-js/17580>\
**Category:** 🤗Transformers\
**Created:** [May 6, 2022, 1:05am UTC](https://discuss.huggingface.co/t/tokenizing-using-js/17580 "2022-05-06T01:05:01Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![msamogh](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/msamogh/32/967_2.png) [@msamogh](https://discuss.huggingface.co/u/msamogh)\
**Post date:** [May 6, 2022, 1:05am UTC](https://discuss.huggingface.co/t/tokenizing-using-js/17580/1 "2022-05-06T01:05:01Z")

</div>

I’ve exported a custom PyTorch-based Transformer model into ONNX to run it on NodeJS. However, the exported model seems to expect `input_ids` directly (and not raw text).

**Is there any way I can perform tokenization in JS?**

Or is there something I’m missing, wherein the ONNX model itself is capable of performing the tokenization as well?

---

<div class="post-metadata">

**Author:** ![praeclarum](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/praeclarum/32/5720_2.png) [@praeclarum](https://discuss.huggingface.co/u/praeclarum)\
**Post date:** [August 17, 2022, 6:05am UTC](https://discuss.huggingface.co/t/tokenizing-using-js/17580/2 "2022-08-17T06:05:49Z")

</div>

I have the same problem. Seems converting to onnx is only half the battle. Maybe I’ll write a library. How hard could it be?

---

<div class="post-metadata">

**Author:** ![praeclarum](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/praeclarum/32/5720_2.png) [@praeclarum](https://discuss.huggingface.co/u/praeclarum)\
**Post date:** [August 19, 2022, 9:46am UTC](https://discuss.huggingface.co/t/tokenizing-using-js/17580/3 "2022-08-19T09:46:08Z")

</div>

I have written a JavaScript library that is capable of running the T5 tokenizer: [transformers-js/tokenizers.js at main · praeclarum/transformers-js · GitHub](https://github.com/praeclarum/transformers-js/blob/main/src/tokenizers.js)

---

<div class="post-metadata">

**Author:** ![Xenova](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/xenova/32/28089_2.png) [@Xenova](https://discuss.huggingface.co/u/Xenova)\
**Post date:** [February 22, 2023, 9:39pm UTC](https://discuss.huggingface.co/t/tokenizing-using-js/17580/4 "2023-02-22T21:39:50Z")

</div>

Awesome library @praeclarum! I used it as inspiration when developing [https://github.com/xenova/transformers.js](https://github.com/xenova/transformers.js), which supports BERT, DistilBERT, T5 and GPT2 tokenization and inference.

---

<div class="post-metadata">

**Author:** ![MatousAc](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/matousac/32/20292_2.png) [@MatousAc](https://discuss.huggingface.co/u/MatousAc)\
**Post date:** [May 5, 2023, 5:07pm UTC](https://discuss.huggingface.co/t/tokenizing-using-js/17580/5 "2023-05-05T17:07:24Z")

</div>

Thank you, so so much. This library finally made it possible for even a noob such as myself to execute my models in the browser. I’d probably get a C in my Data Mining class if I didn’t find this amazing tool.