tokenizer.json: post-processor wraps inputs as <unk> … <s> instead of <s> … </s>
#1
by GorkaUrbizu - opened
Kaixo,
The post_processor (TemplateProcessing) in tokenizer.json uses special-token ids that don't match the vocabulary:
| token | vocab id (added_tokens, config.json) |
post_processor.special_tokens |
|---|---|---|
<s> |
1 (bos_token_id, cls_token_id) |
0 (= <unk>) |
</s> |
2 (eos_token_id, sep_token_id) |
1 (= <s>) |
So with add_special_tokens=True (the default), every input gets wrapped as <unk> … <s> instead of <s> … </s>. This affects fine-tuning and inference through AutoTokenizer.
Reproduction (transformers 5.19.0, revision 94e9f46):
from transformers import AutoTokenizer
tok = AutoTokenizer.from_pretrained("HiTZ/JaunBERT")
ids = tok("Urteten jake urteten jakena.")["input_ids"]
print(ids) # [0, 69665, ..., 255655, 1]
print(tok.decode(ids)) # '<unk> Urteten jake urteten jakena.<s>'
Suggested fix: change the two ids values under post_processor.special_tokens in tokenizer.json:
"</s>": {
"id": "</s>",
"ids": [
- 1
+ 2
],
@@
"<s>": {
"id": "<s>",
"ids": [
- 0
+ 1
],
With this change the output becomes [1, 69665, ..., 255655, 2] / '<s> Urteten jake urteten jakena.</s>'.
Ondo segi,
Gorka