Sagarmatha V5 — Nepali Speech Recognition

FLEURS ne_np 27.53 WER · Common Voice 29.53 · in-domain 15.78

Whisper large-v3-turbo, fully fine-tuned, vocabulary extended for Devanagari.

Release Note. Sagarmatha V5 is the predecessor foundation model to Sagarmatha V6. It serves as the acoustic base model and weight-averaged prior ($\eta = 0.80$) for Sagarmatha V6.

Results

Raw WER, NFC normalisation only. No orthographic folding — see Scoring.

benchmark	clips	V4	V5	best published
FLEURS `ne_np`	724	38.61	27.53	39.06 (whisper-medium)
Common Voice `ne-NP`	704	43.19	29.53	48.35 (large-v3-turbo)
internal, 60 unseen videos	300	31.39	15.78	—

CER: FLEURS 11.93 → 9.52, Common Voice 11.52 → 7.11, internal 10.99 → 5.20.

Read the published column carefully. Those baselines (arXiv:2608.12327 Table 4) were scored NFC-only and are fine-tuned on OpenSLR, not zero-shot. This is not a zero-shot comparison and should never be presented as one.

The internal 15.78 is the closest test to deployment: unseen videos from the training channel distribution. The 14.70 dev WER is neither out-of-domain nor unseen and must not be quoted.

Training


base	`openai/whisper-large-v3-turbo` (809 M)
method	full fine-tune, all layers
data	199.48 h, 37,466 train windows
schedule	4 epochs, 2,344 steps, lr 1e-5, warmup 500
batch	8 × 8 accumulation = 64 effective
regularisation	label smoothing 0.1, gain augment ±12 dB
hardware	1 × RTX 5090, 3.18 wall hours, 2.56 × 10²⁰ FLOPs

The full run is mirrored in wandb/run_8brft64m.json — 93 logged train points, 10 evals, and the complete config as plain JSON, so none of the numbers on this card depend on the W&B run staying reachable.

The run converged; it did not run out of budget

Eval WER went 51.09 → 14.70 and CER 22.93 → 4.64 across 10 evals. The shape is the finding: 81 % of the total gain is in by step 750, inside the second epoch. The last 250 steps bought 0.34 points and the final 94 bought 0.01.

Eval loss never rises, so this is not overfitting by the usual test. But the train–eval gap does open: −0.005 at step 1500, +0.068 at step 2000, +0.026 at the end — and it opens over exactly the span where WER stops moving. A fifth epoch on this corpus is not the lever.

The schedule reaches zero exactly at the last step, so part of that flat tail is the schedule rather than the data. The bend came earlier, at step 1750, with the rate still a third of peak.

The spikes are not instability. The norm peaks at 35.25 and exceeds 5 on 17 of 93 logged steps, but max_grad_norm is 1.0 and this series is logged before clipping — every spike was scaled to 1.0 on the way in. It measures how much batch difficulty varies, which is what a corpus mixing read prompts with spontaneous speech should look like.

One thing in the recipe did not happen

The recipe specifies group_by_length: on. The run logged train_sampling_strategy = random. transformers 5.15.1 removed the old name in favour of the new one and the setting was ignored rather than rejected, so the run paid the full padding cost it was written to avoid — 53 % on the internal half, 60 % on OpenSLR. It costs wall clock and not accuracy, and it is a free speedup on the next run.

What is in this repository

path	what it is
`model.safetensors` + `config.json` + `tokenizer.json`	the fine-tune, for `transformers`
`ct2-fp16-all80/`	the CTranslate2 fp16 conversion this is actually served from, with `alignment_heads` rewritten to all 80. `config.json` there is read at load, so the head set is a config edit and not a re-conversion
`figures/`	every figure on this card, exported from the project dashboard
`wandb/run_8brft64m.json`	the complete training run
`training_summary.json`	final metrics as written by the trainer

training_args.bin is deliberately not shipped: it is a pickle, and everything inside it is in wandb/run_8brft64m.json as plain JSON.

# transformers
from transformers import WhisperForConditionalGeneration, WhisperProcessor
m = WhisperForConditionalGeneration.from_pretrained("tonibirat/sagarmatha-v5")
p = WhisperProcessor.from_pretrained("tonibirat/sagarmatha-v5")

# CTranslate2, which is what the streaming server runs. Note: drive
# ctranslate2.models.Whisper directly, NOT faster_whisper.WhisperModel - see Serving.
import ctranslate2
w = ctranslate2.models.Whisper("ct2-fp16-all80", device="cuda", compute_type="float16")

CTranslate2 4.8.x is built against CUDA 12 while current torch wheels carry cu13. If the model loads on GPU and the first compute call raises Library libcublas.so.12 is not found, that is why; side-install nvidia-cublas-cu12 and nvidia-cudnn-cu12 and put them on LD_LIBRARY_PATH rather than into the same site-packages.

What is different from the base model

A 2,000-token Devanagari vocabulary extension. Stock turbo spends 6.65 tokens per Nepali word; this model spends 3.19. That was an accuracy decision and turned out to be a 2× decode-step win for streaming.

Whisper's embedding is tied, so LoRA cannot train new token rows — extending the vocabulary forces a full fine-tune. Vocabulary is 53,866; the added ids occupy 50,364–52,363.

That id range is load-bearing. See Serving.

Serving: do not use library defaults

Three settings, each measured. On library defaults this model is 29.94 WER from its own HuggingFace output and 17.5 points worse against reference.

setting	why
patch `faster_whisper.Tokenizer.decode`	it keeps only ids `< 50257`, and our 2,000 tokens live at 50364–52363. Unpatched output is 92.34 WER from HF and 34.6 % of the characters — still fluent Nepali, which is what makes it dangerous
`compression_ratio_threshold=4.51`	the 2.4 default flags 47.3 % of correct Nepali as a repetition loop, because Devanagari's UTF-8 is redundant before the words are. Degenerate Nepali starts at 5.45, so retune — never disable
`without_timestamps=True`	HF decodes with `<\|notimestamps\|>`; faster-whisper does not. Worth 1.64 WER of spurious difference

With all three, CTranslate2 sits 2.87 WER from HuggingFace and 0.36 against reference. The residual is CT2's own fp16 kernels and does not close, so numbers measured on HuggingFace do not transfer token-for-token to the served engine.

Also: decoder_start_token_id is <|startoftranscript|>, and forward prepends it when labels arrive without decoder_input_ids. Targets already starting with it must have it stripped in the collator, or training conditions on a doubled prefix, silently.

Real-time streaming

Measured end to end on CTranslate2, 24 windows, 500 ms step:

policy	streaming WER	median lag	p90 lag
offline, single pass	20.86	—	—
LocalAgreement-2	21.96	1.200 s	2.940 s
AlignAtt (recommended)	19.83	0.870 s	1.380 s

Latency is a computation-unaware word delay — the same convention as UFAL Table 3 and SimulStreaming Table 7.

Drive AlignAtt from all 80 cross-attention heads via ctranslate2.models.Whisper.align(). Not from generation_config.alignment_heads — see below.

A commit is not a promise, whichever policy you pick

Both published metrics — WER and lag — are computed on the finished transcript, and a streaming loop re-derives its working text whenever the model contradicts itself, so neither number can see a revision. Measured directly, on 180 s through the real serving loop:

	contradicts itself	prefix that survives	of the prefix revised
AlignAtt, 20 frames	30.4 % of steps	median 4 tokens	89.0 %
AlignAtt, 40 frames	25.9 %	median 36 tokens	58.6 %
LocalAgreement-2	19.0 %	median 4.5 tokens	91.9 %

LocalAgreement-2 only commits a prefix two successive decodes have already agreed on, and it still takes one back every fifth step — revising proportionally more than AlignAtt when it does. All three arms produce the same amount of text.

So do not present policy-committed text as final. The only irrevocable text is text whose audio has been dropped from the buffer, because that is the one thing the model cannot re-read. Our server sends three states — final (trim-harvested), stable (committed, audio still buffered) and volatile — and the UI draws them as three distinct shapes.

Alignment heads did not survive the fine-tune

If you want word timestamps or attention-guided streaming, this matters.

implementation	declared 6 better	all-80 better	sign test
torch DTW	12	34	p = 0.0016
CTranslate2 `align()`	13	35	p = 0.0021

Two independent implementations, the same answer. Use all heads, or at minimum the upper half of layers — which is CTranslate2's own default when a model declares none. The all-80 configuration gives median 0.060 s boundary error and 90.5 % within 250 ms.

Scoring

Never score Nepali with Whisper's BasicTextNormalizer. It replaces every Unicode category M character with a space — every Devanagari vowel sign and bindu — and inflates the FLEURS ne_np word count 2.52×.

Normalisation is split into a safe set and a diagnostic upper bound, and only the safe set may sit next to a published number. Blanket vowel-length, nasal-presence, ब/व and sibilant folds each merge real words (जून/जुन, छौ/छौँ, वन/बन, केश/केस).

Raw WER is the headline.

benchmark	raw	safe folds	+ numbers + boundary	CER after safe norm
internal	15.78	14.13	13.06	5.10
FLEURS	27.55	27.10	23.51	8.43
Common Voice	29.74	28.65	25.35	6.56

Known behaviour and limitations

What it does not do

It does not hallucinate long runs. 0 insertion runs of 5+ words and 0 repeated 5-grams over 300 in-domain clips; 88.4 % of insertion runs are a single word. For scale, WhisperX measured 131 five-gram duplicates for base Whisper on Kincaid46.
It does not truncate. Longest generation across 1,728 clips used 213 of 440 tokens.

What it does do, and you should know

It emits a word on 100 % of non-speech input. The corpus has zero empty targets, so the model has never seen audio whose correct transcription is nothing. Fine-tuning did remove the base model's runaway loop — 6 invented words over 6 non-speech inputs, against base turbo's 267 — but it never outputs silence.

Gate it, and not with no_speech_prob. Whisper's own no-speech signal is unusable on large-v3-turbo: measured AUC 0.333 on this checkpoint and 0.108 on stock turbo, both below chance, meaning silence scores lower than speech. It is prefix-invariant, so it is not an artefact of prompting, and the stock control says this is a property of turbo rather than of our fine-tune. Use a real VAD — silero separated the same inputs at AUC 1.0, with 0.0 voiced fraction on every non-speech input against a minimum of 0.6497 across 60 real clips. Our server gates at 0.10 and runs it before the decode.
It writes English in Devanagari. कम्प्युटर, never computer. The corpus contains zero Latin-script tokens. This is a deliberate convention, not a bug.
Beams collapse. All 8 beams return byte-identical sequences; beam search buys nothing over greedy on this checkpoint.

Where the errors are

word is…	error rate, FLEURS	share of all errors
seen in training, common in Nepali	15.5 %	54.0 %
rare in training, common in Nepali	41.9 %	16.1 %
never seen in training	61.7 %	32.2 %
absent even from 194M words of Nepali	76.5 %	10.8 %

Two things follow. More than half of all errors are on words the model has seen plenty of — that is acoustics, and no lexicon fixes it. And a word the model never heard is wrong roughly 3 times in 5; the vocabulary extension made such words cheap to spell, not recognisable. Use contextual biasing for names and jargon.

And more audio is the expensive way to fix it. Subsampling the corpus at nine sizes gives a Heaps exponent of 0.560, against the 0.4–0.5 typical of English: Nepali vocabulary does not saturate, because every noun crossed with every case marker is a distinct surface form. Doubling the audio to 400 h moves FLEURS out-of-vocabulary from 11.95 % to 9.80 %. Halving it would take roughly 5,000 hours. The same gap is cheap to close with text, because the missing knowledge is that a word exists and how it is spelled — and the decoder is a language model.

Across 11 unseen videos, WER falls 2.29 points per doubling of that channel's hours in training, r = −0.754. This is correlational — it cannot separate "more data on a channel helps" from "easier channels happened to get more data". What it says without ambiguity is that the channels we under-covered are the ones failing, which is a targeting instruction and not a volume one.

Degraded audio: measured, and it is the weakest part of this model

60 internal clips, offline, raw WER. Synthetic corruption — read it as an upper bound on the good news, not as a room test.

	WER	vs clean
clean	15.84	—
white noise, 20 dB SNR	18.01	+2.17
white noise, 10 dB SNR	25.95	+10.11
babble, 10 dB SNR	23.40	+7.56
reverb alone, rt60 0.4 s	24.84	+9.00
reverb + white 10 dB	54.45	+38.61

A room alone costs as much as 10 dB of broadband noise. Every training hour is broadcast material, close-mic or already dereverberated, so the model has never heard reverberation.

Reverb and noise are not additive — they are 2.02× additive. +9.00 and +10.11 apart, +19.11 if they summed, +38.61 together. Do not interpolate a deployment number from this table.

Babble is easier than white noise at matched SNR, consistently at both levels tested: the interferer is intermittent and the model reads the gaps.

Nothing produced an empty output in any condition, including 0 dB SNR. Degrading the input does not make this model stop — it makes it wrong. A VAD does not help there, because at 0 dB there really is speech.

Untested

Real microphones. Every number here is YouTube-derived or read-prompt audio. Phone and laptop capture differ in bandwidth, noise floor, AGC and codec, and this model has never been scored on any of them. The table above raises the prior that it goes badly.

Licence

Derived from a mixed-provenance corpus, part CC BY-SA 4.0 and part not redistributable. See the dataset card. Keep this repository private.

Citation

@misc{sagarmatha_v5_2026,
  title  = {Sagarmatha V5: Nepali speech recognition},
  author = {Birat},
  year   = {2026},
  note   = {Whisper large-v3-turbo fine-tune, vocabulary extended to 53,866}
}