Re-convert GGUF with embedded SentencePiece tokenizer

#32
NVIDIA org

The currently published GGUF predates NeMo-Speech.cpp's tokenizer-embedding conversion feature, so RNNT word-boosting (speech_contexts) silently has no effect -- confirmed via nemo-speech's own reference CLI (--speech-context --verbose logs "RNNT word boosting requested but no phrases could be tokenized (no embedded SentencePiece tokenizer in the GGUF?); boosting disabled."). The .nemo checkpoint at this same pinned revision already contains a SentencePiece tokenizer.model; re-running convert_model.py against it (current NeMo-Speech.cpp main) embeds it correctly. Verified word-boosting works end-to-end against this file (4/4 test cases recognizing an otherwise-mangled proper noun once boosted). No retraining, no config changes -- this is the same checkpoint, just re-converted.

NVIDIA org

For reviewers wondering whether this is "just" a tokenizer add or a wider re-conversion: diffed every KV key and every tensor between the currently-published GGUF and this one (via gguf.GGUFReader, comparing shape/dtype/sha256 per tensor).

655 of 657 tensors are byte-identical. The only two that differ are not learned weights -- both are computed analytically by the converter, not read from the checkpoint:

  • preprocessor.fb (mel filterbank): numerically 100% identical; the only byte-level difference is -0.0 vs 0.0 on already-zero values.
  • encoder.pos_enc.pe (sinusoidal positional encoding, via torch.sin/cos/exp): 88.3% of elements exactly equal, rest differ by up to 4.9e-4 (mean 1.6e-6) -- consistent with floating-point non-determinism from a different PyTorch/numpy version computing the same closed-form formula, not a logic change. Well under the model's own Q8_0 quantization noise floor elsewhere.

Metadata-wise, the only change besides the new asr.tokenizer.spm_model key is general.name losing a stray leading . and file extension it shouldn't have had (cosmetic).

So: no weights changed, this is purely "add the tokenizer that was already sitting in the .nemo checkpoint."

Jinhanw changed pull request status to merged

Sign up or log in to comment