# Tokenization: different results when tokenizing in one pass vs sample-by-sample

**URL:** <https://discuss.huggingface.co/t/tokenization-different-results-when-tokenizing-in-one-pass-vs-sample-by-sample/40825>\
**Category:** Intermediate\
**Created:** [May 24, 2023, 9:49am UTC](https://discuss.huggingface.co/t/tokenization-different-results-when-tokenizing-in-one-pass-vs-sample-by-sample/40825 "2023-05-24T09:49:39Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![TheBloke](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thebloke/32/15927_2.png) [@TheBloke](https://discuss.huggingface.co/u/TheBloke)\
**Post date:** [May 24, 2023, 9:49am UTC](https://discuss.huggingface.co/t/tokenization-different-results-when-tokenizing-in-one-pass-vs-sample-by-sample/40825/1 "2023-05-24T09:49:39Z")

</div>

Hi all

I have an issue that has been holding me up and driving me crazy for more than 24 hours.

**In a nutshell: tokenizing a long dataset in a single pass of `tokenizer()` produces different results compared to iterating through the dataset sample by sample, calling `tokenizer()` on each. I do not understand why, and this is impacting my results.**

I am working on a perplexity algorithm for quantised transformers models, copying the algorithm used by llama.cpp. I have the code working, but am now trying to integrate it with an LLM evaluation library that’s part of AutoGPTQ. In order to do this, I need my code to be able to work when the source dataset is iterated sample-by-sample.

I have so far tested only with Llama models, specifically Llama 7B.

The following simple code demonstrates the problem I am having:

```python
from transformers import AutoTokenizer
from datasets import load_dataset
import torch

tokenizer = AutoTokenizer.from_pretrained("huggyllama/llama-7b", use_fast=True)

wikidata = load_dataset('wikitext', 'wikitext-2-raw-v1', split='test')
wikilist = [' \n' if s == '' else s for s in wikidata['text']]

def tokenize_one_pass():
    # Tokenize the full text in one pass
    text = ''.join(wikilist)

    method1_tokens = tokenizer(text, truncation=False, add_special_tokens=False, return_tensors='pt').input_ids

    return method1_tokens[0]

def tokenize_by_sample():
    # Iterate through each sample of the dataset, tokenizing one by one. Then concat at the end.
    tokens = []
    for sample in wikilist:
        output = tokenizer(sample, truncation=False, add_special_tokens=False, return_tensors='pt').input_ids
        tokens.append(output)

    method2_tokens = torch.LongTensor()
    for item in tokens: # Concat tokenized samples into one tensor
        input_ids = item[0]
        method2_tokens = torch.cat((method2_tokens, input_ids), dim=0)
    return method2_tokens

m1_tokens = tokenize_one_pass()
m2_tokens = tokenize_by_sample()

# Method 2, sample-by-sample, returns 2881 more tokens
print("method 1 (tokenize in one pass) token len:", len(m1_tokens))
print("method 2 (tokenize sample-by-sample) token len:", len(m2_tokens)
print("m1 tokens:", end='')
for i, m1 in enumerate(m1_tokens):
    if i < 20:
        print(f"{m1}, ", end='')

print("\n\n\n")

print("m2 tokens:", end='')
for i, m2 in enumerate(m2_tokens):
    if i < 20:
        print(f"{m2}, ", end='')
print("\n")
# The tokens generated will be slightly different as well!

```

Here is the output it produces:

```auto
method 1 (tokenize in one pass) token len: 335687
method 2 (tokenize sample-by-sample) token len: 338568
m1 tokens:259, 13, 353, 4755, 350, 5059, 357, 353, 29871, 13, 29871, 13, 4755, 350, 5059, 357, 338, 385, 4223, 2706,

m2 tokens:259, 13, 29871, 353, 4755, 350, 5059, 357, 353, 29871, 13, 259, 13, 29871, 4755, 350, 5059, 357, 338, 385,

```

Notice that: method 2 returns more tokens, so it has added some extra tokens into the output somehow.

And we can see that in the sample of 20 tokens returned.  
m1 is `259, 13, 353, 4755, ..`  
m2 is `259, 13, 29871, 353, 4755, ..`

the token 29871 has been added.

Further debug showed me that 29871 was added at the beginning of sample 2:

```auto
sample1: tensor([[259, 13]])
sample2: tensor([[29871, 353, 4755, 350, 5059, 357, 353, 29871, 13]])
                  ^^^^^

```

But it is not the BOS token (which anyway I am suppressing with `add_special_tokens=False`).

Why is this happening? Why does tokenising in smaller samples result in extra tokens being added compared to tokenising in one long string? And is there anything I can do to avoid that (while still tokenising sample-by-sample)?

Thanks in advance for any help.

---

<div class="post-metadata">

**Author:** ![TheBloke](https://sea2.discourse-cdn.com/hellohellohello/user_avatar/discuss.huggingface.co/thebloke/32/15927_2.png) [@TheBloke](https://discuss.huggingface.co/u/TheBloke)\
**Post date:** [May 25, 2023, 6:36am UTC](https://discuss.huggingface.co/t/tokenization-different-results-when-tokenizing-in-one-pass-vs-sample-by-sample/40825/2 "2023-05-25T06:36:54Z")

</div>

Here is a really simple demonstration of the problem:

```auto
>>> tokenizer = AutoTokenizer.from_pretrained("huggyllama/llama-7b")
>>> tokenizer(" \n", add_special_tokens=False).input_ids
[259, 13]
>>> tokenizer(" \n = Robert Boulter = \n", add_special_tokens=False).input_ids
[259, 13, 353, 4755, 350, 5059, 357, 353, 29871, 13]
>>> tokenizer(" = Robert Boulter = \n", add_special_tokens=False).input_ids
[29871, 353, 4755, 350, 5059, 357, 353, 29871, 13]

```

Where does that 29871 come from at the start of line 3’s output? It is not there in line 2, even though line 3 is a subset of the text in line 2.

OK… I think I finally understand

```auto
>>> tokenizer.decode([13])
'\n'
>>> tokenizer.decode([353])
'='
>>> tokenizer.decode([13, 353])
'\n ='

```

13 on its own is `\n` and 353 on its own is `=`, but 1`3, 353` together has a space in-between - `\n =`

So I suppose the 29871 is needed at the start to add the space in which is not there otherwise.

Meaning, more tokens are needed to encode a given piece of text when it’s broken into smaller chunks than would be when it’s done in a single pass.

So I probably can’t even do what I need to do, given I need tokenization to be 100% identical to the source material, which apparently is impossible unless I tokenize it in one pass, not sample by sample.

---

<div class="post-metadata">

**Author:** ![NULLNode](https://avatars.discourse-cdn.com/v4/letter/n/6f9a4e/32.png) [@NULLNode](https://discuss.huggingface.co/u/NULLNode)\
**Post date:** [August 15, 2023, 6:30am UTC](https://discuss.huggingface.co/t/tokenization-different-results-when-tokenizing-in-one-pass-vs-sample-by-sample/40825/3 "2023-08-15T06:30:02Z")

</div>

bro, i get the same problem with you, it makes me grazy

---

<div class="post-metadata">

**Author:** ![joshc8c7](https://avatars.discourse-cdn.com/v4/letter/j/6bbea6/32.png) [@joshc8c7](https://discuss.huggingface.co/u/joshc8c7)\
**Post date:** [October 23, 2023, 5:15pm UTC](https://discuss.huggingface.co/t/tokenization-different-results-when-tokenizing-in-one-pass-vs-sample-by-sample/40825/4 "2023-10-23T17:15:56Z")

</div>

Anyone got an update on this?