nllb200-zh-formosan-spm8k

Direction: Traditional Chinese to Formosan
Base model: facebook/nllb-200-distilled-600M
Recipe: nllb200-spm8k-directional-v9
Release: 20260828-121226, validation-selected step 290,000

This is a directional model for 15 Formosan languages. It uses the private_no_bible leakage-controlled corpus, Formosan-aware 8k SentencePiece extension, and explicit direction, source-language, and dialect controls. It does not infer or encode domains from repository or path names. Training uses language-temperature sampling with alpha 0.5; sentence rows have weight 1.0 and explicit lexical rows, when present, have weight 0.25. Dialect tags use 25% dropout so default is a normal inference condition. Synthetic pivot rows are train-only. The model weights are public. The training corpus is distributed separately to authorized FormosanBank members through the access-controlled FormosanBank/formosan-mt-private dataset and is not included with the weights.

Model details

Item	Value
Base revision	f8d333a098d19b4fd9a8b18f94170487ad3f821d
Corpus release	20260824T042425Z_v3
Training code commit	a2cfed01277bc30bd9dea6a96af26564bb4f0718
Total corpus rows	796,522
Training rows	681,990
Human split policy	85% train / 5% validate / 10% test
Synthetic evaluation policy	train_only_after_human_split
Language sampling alpha	0.5
Sentence / lexical sampling weight	1.0 / 0.25
Dialect-tag dropout	25%
Training budget	300,000 updates
Effective batch size	64
Maximum sequence length	384
Learning rate	2e-05
Precision	bf16
Checkpoint selection	Validation chrF2
Formosan text	kindOf=standard, formosan-mt-standard-v3
Corpus SHA-256	b2592c7c450334f283f35078ce9cc1976121af8de63b52d16f699d9e902f298e
Training profile SHA-256	01637c79ae86f8cc9c149a61b4a24e808d18261af140afe4fb0508a5190ce62a

Usage

import torch
from transformers import AutoModelForSeq2SeqLM, NllbTokenizer


model_id = "FormosanBank/nllb200-zh-formosan-spm8k"
tokenizer = NllbTokenizer.from_pretrained(model_id, use_fast=False)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)
model.to("cuda" if torch.cuda.is_available() else "cpu")
NLLB_LIDS = {'ami': 'ami_Latn', 'bnn': 'bnn_Latn', 'ckv': 'ckv_Latn', 'dru': 'dru_Latn', 'pwn': 'pwn_Latn', 'pyu': 'pyu_Latn', 'ssf': 'ssf_Latn', 'sxr': 'sxr_Latn', 'szy': 'szy_Latn', 'tao': 'tao_Latn', 'tay': 'tay_Latn', 'trv': 'trv_Latn', 'tsu': 'tsu_Latn', 'xnb': 'xnb_Latn', 'xsy': 'xsy_Latn'}

def translate(text, lang_code, dialect="default"):

    tokenizer.src_lang = 'zho_Hant'
    dialect_tag = f"<dialect_{dialect}>"
    if tokenizer.convert_tokens_to_ids(dialect_tag) == tokenizer.unk_token_id:
        dialect_tag = "<dialect_default>"
    prompt = f"<to_{lang_code}> <src_zh> {dialect_tag} {text}"
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    output = model.generate(
        **inputs,
        decoder_start_token_id=tokenizer.eos_token_id,
        forced_bos_token_id=tokenizer.convert_tokens_to_ids(NLLB_LIDS[lang_code]),
        max_new_tokens=256,
        num_beams=4,
    )
    return tokenizer.batch_decode(output, skip_special_tokens=True)[0]

print(translate('他回家了。', "ami"))

The direction, source-language, and dialect tags are part of the training contract. Use default when dialect metadata is unavailable. This model does not use domain or repository tags.

Evaluation

The best checkpoint was selected on validation chrF2. Test and validation contain only eligible, human-translated sentence pairs. Synthetic pivots and lexical entries are train-only. The headline result uses default metadata controls (the default dialect only), so it does not assume access to test-set dialect labels or any inferred domain metadata.

Split	Rows
Train	681,990
Test	76,353
Validate	38,179

Scope	BLEU	chrF2	TER
Hard test	11.83	36.86	82.90
Selection validation	10.18	37.06	85.46

Test empty-output rate: 0.0013%.

Bootstrap confidence intervals were not requested for this release.

Metric signatures:

• BLEU: nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1
• chrF2: nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1
• TER: nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1

Language	Samples	BLEU	chrF2	TER
ami	14,248	9.37	32.82	81.69
bnn	6,276	8.61	37.76	90.04
ckv	2,887	25.80	50.66	60.17
dru	6,764	3.98	27.99	113.78
pwn	5,800	5.93	33.60	104.09
pyu	4,422	13.98	39.92	75.66
ssf	2,299	23.92	53.68	52.98
sxr	2,056	8.02	45.74	87.99
szy	2,520	18.37	44.53	67.52
tao	2,504	13.98	38.22	72.14
tay	7,163	5.19	25.66	100.50
trv	12,048	12.88	34.99	72.67
tsu	2,304	13.86	40.79	73.30
xnb	2,563	21.11	53.45	59.86
xsy	2,499	25.11	50.91	60.80

The corpus gate applies 85% train / 5% validate / 10% test to deduplicated human pairs within each language and source where capacity permits. Synthetic pivots are appended to training after that split. Evaluation is sentence-only and contains no lexical or synthetic rows. Standard-tier Formosan text is normalized with formosan-mt-standard-v3. The release requires zero exact, skeleton, one-edit, or configured high character n-gram train/evaluation conflicts. Document overlap is diagnostic. This release passed all leakage gates: exact 0, skeleton 0, one-edit 0, character n-gram 0. Document overlap: 1433.

See eval/metrics.json for sacreBLEU signatures, per-language, source-corpus, dialect, and length diagnostics. publication.json records the corpus, profile, run, and checkpoint hashes used for this release.

Intended use

This model supports research, corpus development, and assisted translation for the 15 included Formosan languages. It is designed for the exact prompt and generation contract shown above.

Limitations

Outputs require knowledgeable speaker review. Aggregate metrics hide large differences among languages and source collections. This model is not suitable for authoritative, medical, legal, or safety-critical translation.