Matcha-TTS-PL — Polish Matcha-TTS for a conversational robot

Non-autoregressive Polish text-to-speech (Matcha-TTS, optimal-transport conditional flow matching, 20.9 M parameters) built for a humanoid robot: real-time on a small GPU (≈ 23 ms to first audio on an NVIDIA GB10, 4 ODE steps), 8 blendable reader voices, 18 style tokens (pitch range × speaking rate × question), and a HiFi-GAN vocoder fine-tuned to this model, which removes the phasey layer a stock vocoder adds to predicted mels. Trained from a VCTK warm start on consistency-filtered Wolne Lektury audiobooks (prose only, text/audio agreement verified with Whisper) plus AZON spontaneous speech.

Files

path	what
model/matcha_pl_target.ckpt	the released acoustic model: Lightning checkpoint, 20 speaker rows (ids 0–7 = target readers), 18 style rows
vocoder/hifigan_pl.pt	HiFi-GAN generator fine-tuned on this model's mels (use this one); vocoder/g_02500000_universal = the stock universal vocoder for comparison
onnx/matcha_pl_voices_t2.onnx, onnx/matcha_pl_voices_t4.onnx	acoustic model + fine-tuned vocoder in one ONNX graph, 2 / 4 ODE steps; inputs x (phoneme ids), x_lengths, scales=[temperature, length_scale], spks; output wav, wav_lengths. Extra speaker rows with ready-made voices, see onnx/voices.json
data/speaker_map.json, data/speakers.json	speaker id → reader
data/style_map.json, data/style_centroids.json	style token definitions
samples/	synthesised test sentences (manifest.csv: file, voice, text)
RECIPE.md	the full training procedure (data, base, target, vocoder fine-tune)
ATTRIBUTION.md, LICENSE	data attribution (every book, reader, director) and CC BY-SA 4.0
(external)	UI, CLI tools and the Matcha-TTS patch the checkpoint needs: github.com/machinekind/tts-pl-playground

Quick start

git clone https://github.com/machinekind/tts-pl-playground.git && cd tts-pl-playground   # follow its README (Matcha-TTS clone + patch)
python -c "from huggingface_hub import snapshot_download as d; d('machinekind/Matcha-TTS-PL', local_dir='.')"
python playground.py --port 8771            # UI: voices, blends, style tokens, effects, sequence composer, video visualizer
python synth_samples.py --ckpt model/matcha_pl_target.ckpt --vocoder vocoder/hifigan_pl.pt \
    --sentences my_sentences.txt --voice "0.5*6+0.5*3" --steps 4 --temperature 0.5 --out out/

ONNX Runtime (CPU or GPU), no PyTorch needed at run time:

import onnxruntime as ort, numpy as np
sess = ort.InferenceSession("onnx/matcha_pl_voices_t4.onnx")
x = phonemes  # int64 [1, T]: espeak-ng "pl" ids from the playground's polish_cleaners (see RECIPE.md §1)
wav, n = sess.run(None, {"x": x, "x_lengths": np.array([x.shape[1]]), "scales": np.array([0.5, 0.95], np.float32), "spks": np.array([24])})

Voices

Speaker ids 0–7 are the target readers (0 Bartosz Bielenia, 1 Katarzyna Faszczewska, 2 Wojciech Masiak, 3 Bartosz Głogowski, 4 Jan Staszczyk, 5 Marek Proszek, 6 Piotr Kopa, 7 Radosław Krzyżowski); ids 8–19 are base-training speakers kept for completeness. Blend voices by averaging embedding rows (--voice "0.5*6+0.5*3"). The ONNX graphs have these blends baked in as extra speaker ids:

id	name	blend (weight × speaker id)
20	kopa	6
21	glogowski	3
22	faszczewska	1
23	bielenia	0
24	mix3	0.56+0.33+0.2*1
25	kopa_glogowski	0.56+0.53

mix3 (0.5 Kopa + 0.3 Głogowski + 0.2 Faszczewska) and kopa_glogowski are the recommended robot voices.

Style tokens

An extra embedding added to the speaker embedding (pass styles=<id> to synthesise, or bake a row for ONNX). Neutral = 7. Pitch-range tokens move the spread by about ±1.3 semitones; the question tokens give a rising terminal contour for yes/no questions (Polish wh-questions fall — write real punctuation, "?" drives intonation).

id	label
0	flat-range · slow
1	flat-range · slow · question
2	flat-range · normal
3	flat-range · normal · question
4	flat-range · fast
5	flat-range · fast · question
6	mid-range · slow
7	mid-range · slow · question
8	mid-range · normal
9	mid-range · normal · question
10	mid-range · fast
11	mid-range · fast · question
12	wide-range · slow
13	wide-range · slow · question
14	wide-range · normal
15	wide-range · normal · question
16	wide-range · fast
17	wide-range · fast · question

Quality

10 conversational test sentences per voice (4 ODE steps, T 0.5, fine-tuned vocoder; mix0 = Kopa+Głogowski 50/50, mix1 = 0.5 Kopa + 0.3 Głogowski + 0.2 Faszczewska)

voice	n	WER	CER	UTMOS	F0 spread [st]	chars/s	silence %
Bartosz Bielenia	10	0.012	0.004	3.24	2.40	11.1	21
Katarzyna Faszczewska	10	0.012	0.002	3.35	2.47	10.6	21
Wojciech Masiak	10	0.012	0.002	3.28	2.74	12.4	14
Bartosz Głogowski	10	0.000	0.000	3.29	3.55	11.5	16
Jan Staszczyk	10	0.047	0.063	3.06	3.30	11.3	15
Marek Proszek	10	0.047	0.061	3.00	4.23	11.6	15
Piotr Kopa	10	0.070	0.074	3.30	2.76	10.6	11
Radosław Krzyżowski	10	0.047	0.017	2.73	1.70	10.8	21
mix0	10	0.058	0.061	3.41	4.00	10.9	14
mix1	10	0.047	0.061	3.27	3.60	10.8	15

Whisper large-v3 WER/CER, UTMOS (tarepan/SpeechMOS), pitch spread. UTMOS does not capture the vocoder artefacts the fine-tune removes; the vocoder choice was made by listening (see RECIPE.md §4–5).

Latency: NVIDIA GB10, PyTorch bf16 + torch.compile, batch 1, 4 steps ≈ 23 ms to first audio, RTF ≈ 0.006. Apple M-series CPU, ONNX Runtime, 4 steps ≈ 0.4 s for a 4 s sentence. Recommended runtime settings: 4 ODE steps, temperature 0.5, length scale 0.9–1.0, plus a 10 kHz low-pass on the output.

Known limitations

Phrase endings are flatter than a human reader's (the model averages final contours); short one-word replies are less natural than full sentences; only readers' prose style is covered (no shouting, whispering or singing).

Training procedure (summary)

RECIPE.md has the complete, reproducible version. Data: Wolne Lektury prose + AZON, per-clip UTMOS/DNSMOS/F0 statistics, Whisper CER filter, reader ranking by consistency (top 15 → base, top 8 → target), question labels corrected from the measured final pitch, 18 designed style tokens. Base: 40k steps from matcha_vctk (batch 64, bf16, lr 1e-4). Target: 12k steps on the 8 readers (lr 5e-5). Vocoder: HiFi-GAN universal fine-tuned 30k steps on the target model's teacher-forced mels (generator lr 2e-5, discriminators 1e-5, 2k-step generator-only warm-up). About 6 GPU-hours on an RTX 4090.

Licence and attribution

• Weights: CC BY-SA 4.0 (LICENSE). The training audio is CC BY-SA 3.0 PL (Wolne Lektury) and CC BY-SA 4.0 (AZON); ShareAlike propagates to the weights. Every book, reader and director is listed in ATTRIBUTION.md — keep that file with any redistribution or derivative.
• Warm start: Matcha-TTS matcha_vctk.ckpt (MIT; VCTK corpus CC BY 4.0, CSTR, University of Edinburgh). Vocoder: HiFi-GAN universal v1 (MIT), fine-tuned here. Code: Matcha-TTS (MIT), jik876/hifi-gan (MIT), espeak-ng (GPL-3.0, runtime dependency).
• Voices are personal attributes. The CC licence covers the recordings, not the readers' personality rights. The recommended deployment is a blend of two or more readers under a neutral voice name (the ONNX exports ship such blends); using a single reader's voice commercially should be cleared with the reader or Wolne Lektury. Readers are named here only as data sources.
• Synthetic speech should be disclosed as such where the listener could otherwise take it for a person (EU AI Act, art. 50).