lokutor-ai/oido-ctc-small-stream-int8

Oído streaming: low-latency speech recognition on the ESP32-S3

¡Oído! is Spanish kitchen slang for heard, got it.

This is the low-latency variant of Oído: NVIDIA's 13 M-parameter stt_en_conformer_ctc_small, fine-tuned to run on audio as it arrives and quantized to int8 for an ESP32-S3 (240 MHz dual core, 8 MB PSRAM, 16 MB flash, no neural accelerator). The encoder processes 1.28 s chunks (32 frames) with 5 s of left context while you speak, so when you stop only the last partial chunk is left to compute, and partial text appears while you talk. It was trained with noise, music, babble and reverberation augmentation, so it keeps the original model's robustness.

Paper: Oído: Open-Vocabulary Speech Recognition on a $5 Microcontroller without a Neural Accelerator

Code, firmware and tools: github.com/lokutor-ai/oido (GPLv3, commercial licenses available). Audio samples: lokutor-ai.github.io/oido.

LibriSpeech WER (%)	test-clean	test-other
This model, streaming (32-frame chunks), greedy	6.27	13.06
This model, streaming (32-frame chunks) + language model	4.86	11.03
This model, full-context mode, greedy	4.07	8.97
This model, full-context mode + language model	3.43	7.78
Oído int8, utterance mode (no streaming)	3.70	8.23
Espressif MultiNet7 on the same chip (ESP-SR benchmark)	8.5	21.3
Whisper tiny.en, fp32 on a laptop (500-utterance subsets)	6.3	15.9

Mean WER over 14 noise and reverberation conditions (300 sentences each, DEMAND noise, babble, simulated rooms; with the language model): 8.96 streaming, 6.22 full-context, against 7.45 for the utterance-mode Oído and 12.1 for Whisper tiny.en.

Latency (measured on a board, one core). The chip runs this model at 1.96× real time in streaming mode (2.04× in utterance mode), so it falls behind the speaker and the backlog grows with the utterance. Feeding six clips at the rate of speech, the final text appeared 4.1–11.1 s after the last sample (clips of 3.6–10.6 s, mean 6.7 s; about 4 s for a 3.7 s command), against 7.2–22.4 s for the same clips in utterance mode. The 0.8 s end-of-speech wait comes on top. When the chip has kept up, the step after the last chunk takes 1.1 s on average (0.8–1.6 s). Streaming roughly halves the wait; the 1.1–1.4 s we had estimated from instruction counts before we had a board needs a real-time factor below one.

Status (9 October 2026). WERs come from the host build of the firmware engine (same C code, same int8 arithmetic). On the board (ESP32-S3-WROOM-1-N16R8, one core) the model's transcripts equal the host build's on 12 of 12 clips in utterance mode and 10 of 12 in streaming mode. The two-core mode is not correct on silicon yet, which is why the chip is slower than real time. Details: github.com/lokutor-ai/oido.

Español: lokutor-ai/oido-es-ctc-small-int8 (CC-BY-4.0).

Which Oído model?

One model per language; the models that stream also run in full-context (utterance) mode.

Model	Language	Modes	Size	License
oido-ctc-small-int8	English	utterance only (best accuracy)	14.0 MB (+1.3 MB LM)	CC-BY-4.0
oido-ctc-small-int4	English	utterance only (smallest)	8.3 MB	CC-BY-SA-4.0
oido-ctc-small-stream-int8	English	streaming + full-context (low latency)	14.0 MB	CC-BY-SA-4.0
oido-es-ctc-small-int8	Spanish	streaming + full-context in one file	14.0 MB (+1.3 MB LM)	CC-BY-4.0

Streaming-capable files stream automatically in the firmware's microphone mode and in live_demo.py.

Use

git clone https://github.com/lokutor-ai/oido && cd oido
huggingface-cli download lokutor-ai/oido-ctc-small-stream-int8 oido_stream.tnm --local-dir models   # or use models/ from the repo
cd esp32/host && make && python live_demo.py --model stream        # laptop microphone, partial text while you speak
esp32/tools/flash.sh /dev/ttyUSB0 models/oido_stream.tnm           # ESP32-S3-DevKitC-1 N16R8 + INMP441 microphone

Files

• oido_stream.tnm: int8 weights in the TNM1 format (14.0 MB). Header flags mark fixed feature normalization and streaming; the default chunk is 32 frames with 128 frames of left context.
• tokenizer.model: the original SentencePiece tokenizer (1024 BPE pieces).
• The optional language model nemo_lm.tlm is in oido-ctc-small-int8.

Training

Two stages starting from NVIDIA's weights, 23,000 steps in total (train/train_nemo_stream.py in the repository): chunked self-attention (chunk sizes drawn from 8 to 32 frames and full context), chunk-limited depthwise convolution, fixed global feature normalization, and in the second stage noise, music and babble from MUSAN plus simulated reverberation. Data: LibriSpeech, Common Voice 17 English, VoxPopuli English, Multilingual LibriSpeech English, AMI, VCTK, People's Speech (transcripts regenerated by parakeet-tdt-0.6b-v2), OpenSLR 70 and 83.

License and attribution

Derived from NVIDIA's stt_en_conformer_ctc_small, licensed CC-BY-4.0. Changes: re-implemented in PyTorch, fine-tuned for streaming and robustness, quantized to int8 and repacked. Because the fine-tuning data includes share-alike corpora (People's Speech, OpenSLR 70 and 83), this model is released under CC-BY-SA-4.0. The engine and firmware are GPLv3, with commercial licenses from Lokutor.