Chatterbox Nano voice encoders for Core AI

The encoders that turn a short recording into a Chatterbox Nano voice, for OpenReader and Apple's Core AI runtime. They are Chatterbox's own: the S3 speech tokenizer, CAM++ speaker encoder and voice encoder from the official ResembleAI/chatterbox-nano weights, revision 71ccd1d0081b430592cea481f4307e764e07bc64, with Chatterbox source revision 5de7a54aa4e5e2baadb0182dde554908b48b85c2. No recordings are included.

The pack is separate from the Nano runtime because only someone making a voice needs it. OpenReader runs it once per voice and stores the result with the reference, so narration never loads it.

voice_encoder.aimodel holds five functions:

Function Input Output
tokenizer_mel 10–15 s at 16 kHz S3 tokenizer log-mel
tokenizer that mel, padded to a multiple of four, and its valid length S3 speech tokens
speaker 10 s at 16 kHz CAM++ x-vector, 192
prompt_mel 10 s at 24 kHz S3 prompt features, 500 Γ— 80
voice_mel trimmed speech at 16 kHz voice-encoder mel, 40 bands

The voice encoder's LSTM, its projection and T3's speaker projection are FP32 host tensors in voice-encoder/, named with shapes and checksums in voice-encoder.json. An unrolled recurrent layer is slow to specialise on both Core AI backends, and this one is small enough to run on the host.

Weights are FP16 and compute is FP32, on CPU only. The FP32 program was 517 MB. Half-precision compute flipped one S3 token in 1,875 more often than the CPU's own FP32 noise did. The GPU lowering fails inside MPS. The pack expects upstream's preprocessing: a 24 kHz reference at -27 LUFS, a 16 kHz copy of it, and librosa's 20 dB silence trim before voice_mel.

Against upstream PyTorch on a Mac CPU, from identical inputs across five clips:

  • S3 tokens agree on at least 99.7%.
  • The x-vector cosine is at least 0.9999987.
  • Mel errors are at most 1.1e-4.

On an iPhone 15 Pro, OpenReader's Release build reproduced the Mac's conditionals exactly. See clone-results.json.

MIT licensed, as upstream; see LICENSE.txt.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for richardr1126/chatterbox-nano-voice-encoder-coreai

Finetuned
(9)
this model