ausculta

ausculta — "listen closely", imperative of auscultāre

Offline Classical Latin phoneme recognition and keyword spotting, built to run without a network connection and eventually on very small hardware.

Picking this up cold? Read HANDOFF.md first — it covers what currently works, the one thing that does not, and where the audio lives (it is not in git).

Status: Phases 1–3 implemented; keyword spotting runs on real Pico 2 hardware. Audio pipeline, phoneme inventory, G2P, corpus tooling, acoustic model, trie-constrained keyword spotting, and ONNX/int8 export.

125 clips of real recorded Latin now exist (4.2 min, one speaker, every phoneme ≥10 examples) — but the recognition figures below still come from synthetic audio, because fine-tuning on the real corpus does not yet work. See HANDOFF.md for that investigation. No trained checkpoint is in the repo.

---

Why Latin

This is the sibling of dxʷaʔɬ, a Lushootseed phoneme recogniser with the same architecture and the same end goal: audio keyword matching on tiny hardware.

That project ran into a wall that no amount of engineering solves. A survey of every publicly discoverable Lushootseed audio source found thirteen holders and zero with a written grant covering model training — the most conveniently reachable material is CC BY-NC-ND, where the NoDerivatives term lands squarely on fine-tuning. Lushootseed recordings belong to living communities who have not been asked, and asking properly takes time that shouldn't be rushed.

Latin removes that constraint entirely:

• No cultural encumbrance. No community holds Latin as heritage language property. Public-domain texts are abundant.
• Trivial data gathering. Read it into a microphone. The audio is yours.
• Deterministic labels. Latin spelling is close to phonemic, so g2p.py derives phoneme transcripts from text with a rule set. You never hand-write a label.

So the technology gets refined here, where the data question is a scripting problem, and the Lushootseed work resumes on the technical foundation this produces — at whatever pace the community conversations actually take.

What transfers back: everything except g2p.py and the phoneme config. audio.py, phonemes.py, manifest.py, model.py and dataset.py are language-agnostic and driven entirely by a JSON inventory.

---

Install

Requires Python ≥ 3.10 and ffmpeg on PATH.

uv venv
uv pip install -e '.[asr,fetch,dev]'

Extra	Pulls in	Needed for
asr	torch, transformers, datasets, accelerate	training and inference
fetch	requests	Wikimedia Commons audio
synth	espeakng-loader	synthetic audio
edge	onnx, onnxruntime, onnxscript	Phase 3 export
dev	pytest, mypy, ruff	development

---

Quick start

# What the model can output
ausculta phonemes

# Latin spelling -> phonemes, no lexicon required
ausculta g2p "Gallia est omnis dīvīsa in partēs trēs"
#   phonemes    g a l l i a | e s t | o m n i s | d iː w iː s a | i n | ...
#   confidence  high

# Recognise a 16 kHz WAV
ausculta transcribe clip.wav --checkpoint checkpoints/ausculta/final

---

Synthetic audio for pipeline development

You need audio to make a pipeline work long before you need good audio. synth generates a corpus with eSpeak NG so the plumbing can be built and debugged before anyone books a microphone.

uv pip install -e '.[synth]'
python scripts/fetch_latin_data.py synth -n 120 --voices 4 -o data/synth

eSpeak turns out to suit Classical Latin unusually well:

• Its la voice is the restored pronunciation — dīvīsa is [diːwiːsa] with [w], not Ecclesiastical [v].
• It honours macrons. malum and mālum synthesise differently, so the hardest label in a Latin corpus comes out right.
• It is a shared library, so there is no subprocess per clip. A few thousand prompts synthesise fast enough to sit inside a test run.

Audio and label are cross-checked. eSpeak phonemises independently of g2p.py, so synth maps its IPA onto our inventory and compares. A prompt where the two disagree is dropped rather than trained on, which guarantees the synthetic corpus is self-consistent. That same check is a free second opinion on our rules — it found three real bugs, including via being rendered [wja] instead of [ˈwi.a].

What it proved

Training a 2.4M-parameter model from scratch on 857 synthetic clips (714 train / 143 held-out, 31 min on CPU), with augmentation:

metric	held-out
phoneme error rate	10.1%
exact sequence match	49.7%
keyword recall	90.3% (112/124)
false alarms	1 in 15,252 chances
constrained decode exact	57.3%

That is the end-to-end Phase 2 result: audio → CTC → trie-constrained spotting, on a model with no pretraining at all. Confidence separates cleanly — target median 0.65, impostor median 0.0005 — which is what makes a single threshold workable:

threshold	recall	false alarms
0.10	99.2%	48
0.20	96.8%	14
0.30	90.3%	6
0.50	71.8%	3
0.70	38.7%	1

(Raw per-keyword scores; the spotter's overlap suppression removes most of those false alarms again — the end-to-end figure at 0.30 is 1, not 6.)

This is not a training corpus. eSpeak is formant synthesis: no breath, no room, one glottal source. A model trained only on it learns eSpeak, not Latin. Every record is tagged synthetic: true so it can never be confused with real audio. Use it to make the pipeline converge and to smoke-test Phase 2, then record.

---

Recording your own corpus

python scripts/record_server.py -o data/session1
# open http://127.0.0.1:8420

A local recording booth. For each prompt you can hear eSpeak read it first, then record your own take, check it, and move on.

That reference playback is the point of the tool rather than a convenience. Classical Latin vowel quantity is phonemic, and if you have not spoken Latin aloud a macron is a spelling rule rather than a sound. eSpeak honours macrons, so hearing mālum before reading it gives you something concrete to imitate. It is a robot and its rhythm is wrong — but its quantities are right, and quantity is the one thing that cannot be recovered if the recording gets it wrong. There is a speed control; slow (110 wpm) makes the long/short contrast much easier to hear than natural pace.

Stress: not a label, but a trap for the speaker

Latin stress is fully predictable from vowel length and syllable structure (the penultimate law), so it carries no information the phoneme labels lack. It is deliberately absent from the inventory — tokenising it would add symbols and subtract nothing.

It still matters, for the reader rather than the model. English marks stress mainly by lengthening the vowel; Latin uses length for a phonemic contrast. Read malum with an English stress instinct and you lengthen the stressed a — producing something acoustically close to mālum while the label still says short. That is a mislabelled example landing on exactly the contrast the inventory is built around.

So the booth shows the syllabification with the stressed syllable in bold, and warns on the two shapes where the habit bites:

• stress on a short vowel — stress it with pitch, not length;
• a long vowel that is not stressed — keep it long anyway.

ausculta.stress implements the penultimate law over phonemes, including muta cum liquida (tenebrae is TE-ne-brae, not te-NE-brae). It agrees with eSpeak on 30 of 31 test words; the one difference follows from the documented su- = [sw] disagreement rather than being a separate bug.

The booth also:

• highlights long vowels in the prompt, so you can see what to hold;
• flags a clipped or inaudible take immediately — caught at the microphone rather than at training time, when it would mean recording the session again;
• tracks phoneme coverage live, showing which of the 39 are still missing or thin, so you know when you can stop;
• writes prompts.tsv with the quantity-marked header, so the corpus builder trusts the vowel lengths.

Keyboard-driven, because a session is a hundred repetitions: Space record / stop, P hear the reference, K hear your take, ↵ next.

Then index it — phoneme labels are derived, never hand-written:

python scripts/fetch_latin_data.py local data/session1 -o data/ --speaker bethany

If you would rather record with your own tools, the prompt list alone is:

python scripts/fetch_latin_data.py prompts -n 80 -o data/session1
# read each line, save as <id>.wav in that directory

prompts runs a greedy set cover over the phoneme inventory rather than sampling at random. Eighty random words leave the rare phonemes — gʷ, ui̯, the Greek aspirates — with no examples at all; the cover reaches full coverage in far fewer recordings and then keeps topping up whatever is thinnest.

Every prompt is macronised, which is not cosmetic: vowel length is phonemic in Latin and unrecoverable from unmarked text. A prompt list without macrons would mislabel the length of every vowel in the corpus.

Other sources:

# Wikimedia Commons single-word pronunciations, with machine-readable licences
python scripts/fetch_latin_data.py commons --category "Latin pronunciation" -o data/

# Deduplicate, split and report coverage
python scripts/fetch_latin_data.py finalise -o data/

LibriVox has public-domain Latin readings; download them however you like and point local at the directory with a matching text file.

---

The phoneme inventory

config/latin_ipa.json defines 39 phonemes — 22 consonants, 12 vowels, 5 diphthongs — plus three CTC special tokens, for a vocab size of 42. This is the Classical restored pronunciation (Allen, Vox Latina), not Ecclesiastical.

"Easier tokenization" is only half true

Latin looks like it should tokenise one-character-per-phoneme. It does not:

Phoneme	Codepoints	Length
a	U+0061	1
aː	U+0061 U+02D0	2
kʷ	U+006B U+02B7	2
ae̯	U+0061 U+0065 U+032F	3

Vowel length is phonemic and carries real minimal pairs — malum "evil" vs mālum "apple", os "bone" vs ōs "mouth", levis "light" vs lēvis "smooth". Drop the length mark and you have merged two phonemes. So aː must be one atomic token, exactly as c̕ must be in Lushootseed, and the longest-match tokeniser earns its keep again.

That is a useful result rather than an annoyance: the palette swap validates the architecture instead of routing around it. The same PhonemeVocab code runs unmodified against a completely different inventory.

Geminates are phonemic (anus "old woman" vs annus "year") and are written as two identical tokens rather than given their own. CTC represents a doubled token by placing a blank between the two, which it learns from data.

---

Grapheme-to-phoneme

Latin spelling is regular enough for rules, but three things are genuinely ambiguous and are handled explicitly rather than guessed:

Vowel length is taken from macrons. Without them, every vowel is emitted short and the result is flagged length_unmarked so the corpus builder can decide whether to keep it. finalise reports how many clips are affected.

Consonantal i and u. iam opens with [j], ibi with [i]. Editions writing j/v resolve this; i/u-only editions do not, so positional rules apply and emit a warning.

eu and ui are diphthongs in only a handful of words. heu is one syllable, deus is two; cui is one, fuit is two. A blanket rule would be wrong far more often than right, so these are lexical.

ausculta g2p "poëta" --json    # diaeresis correctly blocks the oe diphthong

---

Phase 2: zero-hallucination keyword spotting

ausculta trie -k aqua -k aquila -k lingua -k philosophia
#   legal first phonemes (3/39): a l pʰ

ausculta spot clip.wav -k aqua -k lingua --checkpoint ckpt/ --decode

The trie is a finite-state automaton over phoneme token ids. Its job is to supply a hard mask: given a position in the automaton, only tokens that continue a real word are legal. A forbidden token is unreachable, not down-weighted.

That is the whole design. A free CTC decode over 39 phonemes can emit any sequence, so a decode-then-match pipeline has to decide how near a near-miss has to be before it counts. Constraining the search moves that decision into the score: the only sequences ever considered are real words, so the failure mode is a missed detection, never an invented one — the right trade for a device talking to a child learning the language.

Two consumers of the mask:

• KeywordSpotter — "was this said, and when?" Each keyword is scored by a free-start, free-end Viterbi CTC alignment, so a word may sit anywhere in the audio and the surrounding material is ignored rather than modelled.
• TrieConstrainedDecoder — "what known words are in this audio?" A beam search where the trie masks the vocabulary every frame.

Three things that took a bit of getting right:

• Geminates fall out of the CTC skip rule. Moving between identical neighbouring labels requires an intervening blank, which is exactly how CTC spells a doubled consonant — so annus and anus separate with no special casing.
• Overlap is measured as a fraction of the shorter detection. An absolute margin cannot work: subtract 200 ms from a 200 ms word and there is nothing left to compare.
• Longest match wins. A prefix always scores at least as well as the word extending it (extra tokens only add negative terms, and a free end makes leftover audio free), so confidence ranking alone reported ama for every utterance of amat. The threshold keeps maximal munch honest.

The guarantee is tested adversarially: 25 seeds of random posteriors asserting every decoded word is in the lexicon, plus a test that the same input decoded freely does not spell a word — so the constraint is doing real work.

---

Phase 3: fitting a Raspberry Pi Pico 2

ausculta budget ckpt/ --device pico2
ausculta export ckpt/ -o build/ -k aqua -k lingua   # onnx + int8 + trie.h

Target is the RP2350: dual Cortex-M33 at 150 MHz with a single-precision FPU, 520 KB SRAM, 4 MB flash.

The FPU matters more than it first appears. Quantisation is no longer needed for the device to do arithmetic at all — float32 is native — so int8 becomes a size and bandwidth decision rather than a prerequisite. Memory is what binds.

Estimated at fp32 with a 1.5 s working window:

model	params	flash	SRAM	RTF	fits
192d/4L	2380K	9298K	425K	1.63	no — flash
128d/4L	1063K	4151K	313K	0.73	no — flash
96d/3L	507K	1981K	257K	0.35	yes
64d/2L	186K	726K	200K	0.13	yes

So a ~500K-parameter encoder runs in plain float32 with room to spare; int8 is available to buy headroom for a larger model rather than being required. The same model on an RP2040 scores RTF 6.63 — the FPU is worth about 19×, and is the reason the Pico 2 is viable where the Pico 1 is not.

Sizing the window to 1.5 s rather than 30 s is the other reason a transformer is even considerable here: attention is quadratic in frames, and 1.5 s is only ~75 frames.

The trie crosses over unchanged. to_c_header() emits a binary-searchable edge table with no pointers and no allocator, and the CTC decode is an O(T×L) dynamic program. The automaton is deployable today even where the acoustic model is not.

The edge artefact preserves the guarantee

Exporting the trained model and re-running keyword spotting through ONNX, on the same held-out clips:

runtime	size	recall	false alarms	RTF (1 thread)
torch	—	90.3%	1	—
ONNX fp32	9.73 MB	90.3%	1	0.050
ONNX int8	3.74 MB	88.7%	0	0.071

fp32 reproduces the torch model exactly. int8 costs 1.6 points of recall and, on this run, removed the one false alarm — quantisation noise pushing a marginal score below threshold, which is luck rather than a benefit.

Note int8 is slower than fp32 on x86 (RTF 0.071 vs 0.050): dynamic quantisation adds quantise/dequantise overhead that only pays off where int8 kernels are native. On a Pico 2 with an FPU, fp32 is the sensible default and int8 is for buying flash headroom.

Quantisation is dynamic rather than static on purpose: static needs a calibration set whose activations match deployment, and calibrating on eSpeak output would bake synthetic statistics into the model.

A device bundle is one directory:

model.int8.onnx   3831.3 KB
trie.h              13.7 KB     124 keywords, 503 nodes, no allocator
trie.json           24.8 KB
vocab.json           0.5 KB
manifest.json        1.8 KB

14 KB of automaton for 124 keywords is the number that makes the two-tier plan work — the trie is deployable anywhere, whatever the encoder ends up being.

All budget figures above are estimates from parameter counts and clock speed. The decoder half has since been measured — see below.

Measured on a real Pico 2

firmware/ builds the Phase 2 decoder as RP2350 firmware: the trie, the CTC Viterbi aligner and the confidence scoring, in C, with no allocator and no dynamic memory at all. Flashed over SWD to actual hardware:

clock          150000000 Hz
lexicon        41 keywords, 178 nodes, 177 edges
trie tables    1418 bytes flash
posterior buf  12600 bytes SRAM (75 frames x 42)

hard mask      20/42 tokens legal at root
reachability   41/41 keywords reach their terminal
random walks   2000 rejected, 0 stopped mid-word, 0 spelled a real keyword
               (0 of 2000 trials invented a word -- the guarantee)

accuracy       41/41 keywords correctly identified
full scan      6820 us for 41 keywords over 75 frames
audio window   1500 ms
headroom       220x faster than realtime
single keyword 113 us

Whole binary: 24.9 KB flash, 14.0 KB SRAM — of 4 MB and 520 KB available.

Three things worth taking from that:

1. The decoder is effectively free. 220× real-time headroom for a 41-keyword lexicon means the entire device budget belongs to the encoder. A single wake word costs 113 µs.
2. 1418 bytes of trie for 41 keywords. The automaton scales with shared prefixes, not with vocabulary size, so a much larger lexicon stays cheap.
3. The zero-hallucination guarantee holds on the device, not just in the Python tests — 2000 random walks over the automaton, zero invented words. It is a structural property of the trie, so it survives the port.

What the encoder costs at Pico size

Training the 96d/3L model that does fit (507,146 parameters) on the same corpus, against the 192d/4L one that does not:

	2.4M params	507K params (fits)
phoneme error rate	10.1%	27.2%
keyword recall	90.3%	60.5%
false alarms	1	1
Pico 2 budget	flash-limited	RTF 0.35, fits

So the model that fits is meaningfully worse, and that is the real Phase 3 tension stated plainly rather than hidden behind the budget table.

But this is a floor, not a ceiling. That 507K model was trained from scratch on 11 minutes of robot speech. The two-tier plan exists precisely because it should instead be distilled from a fine-tuned XLS-R teacher, which is a different and much better-posed problem — and one that needs real audio to be worth doing at all.

The remaining unknown is running the encoder itself on the RP2350, which needs an inference runtime. That is the honest state of Phase 3: decoder proven on hardware, encoder sized and trained but not yet ported.

---

Architecture

audio file
    ↓  audio.py      decode → 16 kHz mono float32 → VAD → segments
    ↓  model.py      wav2vec2/XLS-R encoder → CTC head (42 outputs)
    ↓
log_probs [frames × 42] + frame_times [frames]     ← Phase 2 consumes this
    ↓  greedy_ctc_decode
[(phoneme, confidence, start, end), ...]

predict_phonemes(path) -> list[tuple[str, float]] is the Phase 1 contract. predict() returns the full RecognitionResult including the dense posterior matrix and per-frame timestamps.

The posterior matrix is the seam for Phase 2. A trie-constrained keyword spotter needs the full distribution to mask against, not a collapsed string — by the time you have ["a", "kʷ", "a"] the information needed to reject a near-miss is gone. log_probs and frame_times are parallel arrays because VAD removes silence, so frame index is not proportional to wall-clock time.

Layout

config/latin_ipa.json         39-phoneme spec, token ids, normalisation table
src/ausculta/
    phonemes.py               inventory, normalisation, longest-match tokeniser
    g2p.py                    Latin spelling → phonemes       (Latin-specific)
    audio.py                  decode, resample, VAD, chunking (no torch)
    manifest.py               dataset.jsonl schema and IO     (no torch)
    augment.py                noise/gain/speed/reverb         (no torch)
    tts.py                    eSpeak NG synthesis + G2P cross-check
    model.py                  wav2vec2 + CTC head, inference
    dataset.py                torch Dataset and padding collator
    trie.py                   Phase 2: the automaton + C export
    kws.py                    Phase 2: spotter + constrained decoder
    device.py                 Phase 3: flash/SRAM/RTF budgets
    export.py                 Phase 3: ONNX and int8
    cli.py                    the ausculta command
scripts/fetch_latin_data.py   corpus building
scripts/evaluate.py           phoneme error rate + KWS metrics
tests/

---

Development

pytest          # audio and G2P tests run without torch; model tests skip
mypy            # strict
ruff check .

Model tests build a tiny wav2vec2 from config rather than downloading a 1.2 GB backbone, keeping the convolutional stride at its pretrained values so the 20 ms frame rate that timestamps depend on is exercised for real. The suite runs in seconds and needs no network.

---

Training

ausculta train data/dataset.jsonl -o checkpoints/ausculta --epochs 30

Defaults suit a small corpus: the convolutional feature encoder is frozen, dropout and layerdrop are on, and CTC uses zero_infinity so one misaligned clip cannot poison a run. Training targets a single 24 GB card — part of why the backbone is XLS-R 300M rather than something larger.

---

Roadmap

• Phase 1 — done. Audio pipeline, phoneme inventory, G2P, corpus tooling, fine-tunable acoustic model.
• Phase 2 — done. PhonemeTrie over the posterior matrix doing hard-masked, zero-hallucination keyword matching.
• Phase 3 — done in estimate. ONNX export, int8 quantisation, and Pico 2 flash/SRAM/real-time budgets. Not yet run on hardware.

What needs a human next

Everything above is validated on synthesised speech, which is the one thing synthesis cannot validate. Specifically:

1. Recorded audio. ausculta's own prompt list reaches full phoneme coverage in ~40 reads. That is an afternoon, and it is the blocker on every number in this README meaning anything about real speech.
2. A Pico 2 on a desk. The budgets are arithmetic. Memory bandwidth is what usually differs from the estimate.
3. A distilled 500K encoder. Straightforward once there is real audio to distil against — pointless before.

Licence

MIT. Recorded audio you contribute carries whatever licence you choose; Wikimedia Commons material keeps the licence recorded per clip in the manifest.

A note on the reconstruction

Latin has no living native speakers. Every recording is one modern reader's interpretation of a scholarly reconstruction, so "speaker variation" here is partly a difference of opinion about the reconstruction itself. Keep the reader's identity in the corpus metadata — local --speaker does this — and treat a model trained on one reader as having learned that reader's Latin.