interpretable-finetuning/qwen2.5-1.5b-unaliased

Qwen2.5-1.5B, un-aliased ChatML turn markers

Qwen/Qwen2.5-1.5B with exactly two embedding rows replaced — the ChatML turn markers <|im_start|> (151644) and <|im_end|> (151645) — copied from Qwen/Qwen2.5-1.5B-Instruct. Everything else is bit-identical to the original.

337 of 338 tensors byte-identical to Qwen/Qwen2.5-1.5B
1 tensor differs: model.embed_tokens.weight, rows 151644 and 151645

Tokenizer, vocabulary, token ids, config and architecture are unchanged. The -Instruct rows are used because it shares this tokenizer and vocabulary, so the ids line up exactly.

Why

The base checkpoint ships 267 embedding rows that are bit-identical to one another, and both ChatML turn markers are inside that block:

token	id	rows bit-identical to it
<|im_start|>	151644	97
<|im_end|>	151645	267
<|endoftext|>	151643	1 (unique)

tie_word_embeddings: true, so those rows are also the output head. Identical rows produce identical logits for any residual stream. A model therefore cannot prefer <|im_end|> over its 266 twins, and if the embedding is frozen — as under LoRA, or any adapter method with modules_to_save: null — no amount of training can separate them.

The failure is quiet and easy to misread as undertraining. On a model fine-tuned against the unpatched base, we measured 10.7% of probability mass on the aliased block at the turn boundary — the turn-end intent was learned — but split 267 ways it gave p(<|im_end|>) = 0.00040, losing the argmax to an untrained token. Generations never terminated.

Note that <|endoftext|>, the tokenizer's declared eos_token, is unique and was never the problem; a check that only inspects eos_token will miss this entirely.

Effect

	Qwen/Qwen2.5-1.5B	this model
rows bit-identical to <|im_start|>	97	1
rows bit-identical to <|im_end|>	267	1
p(<|im_end|>) at the turn boundary¹	0.00040 (rank 6–9)	0.98702 (rank 1)
generations terminating before the token cap¹	0 / 12	200 / 200

¹ measured with a fixed LoRA fine-tune held constant across the two bases.

Caveats

• These rows are still low-norm — about 0.41× the median embedding row. The fix makes them addressable, not strong. A model that needs to emit them still has to learn to.
• Only the two ChatML markers were separated. The checkpoint still contains 2,210 rows in 81 mutually-aliased groups (largest: 473 rows; the remaining 266 of <|im_end|>'s original block is one of them). That is harmless for ordinary use — those are unused vocabulary slots — but any token you add or repurpose from them inherits the same problem. Run the check above on every special token your chat template emits.
• This is a patched base, not an instruction-tuned model. It has no chat behaviour of its own.

Checking a base model for this

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer

mid = "interpretable-finetuning/qwen2.5-1.5b-unaliased"
tok, m = AutoTokenizer.from_pretrained(mid), AutoModelForCausalLM.from_pretrained(mid)
E = m.get_input_embeddings().weight.data
for t in ("<|im_start|>", "<|im_end|>"):
    i = tok.convert_tokens_to_ids(t)
    print(t, "aliased with", int((E == E[i]).all(-1).sum()), "rows")   # must be 1

Any base whose chat-template special tokens report more than 1 will not learn to emit them.

Used by

interpretable-finetuning/topklora-qwen2.5-1.5b-v2 — 46 TopK-LoRA sleeper-agent organisms trained on this base. Those adapters are only correct on this model; loading them on vanilla Qwen/Qwen2.5-1.5B silently restores the broken behaviour.