Complete 72 allophone binary for testing

Previous topic - Next topic

0 Members and 1 Guest are viewing this topic.

Andrew Davie

And with this update, I'm out of quota for Claude. I've burned through a week's quote in 2 days, so unless I decide to update my subscription that will be it for 4 or 5 days.



**SpeakJet library: the effects now follow Pitch and Bend the way the chip does**

I recorded both AtariVoxes playing a fourth probe ROM, VoxProbe 4. It plays the effects at pitch 60 and 150 and at bend 0 and 15, so it shows which parts of each effect follow which control. I fitted on some of those settings and kept the rest back to check against, so the numbers below are for data the library never saw.

What the chip does, and what the library does now:

- **The "biological" effects' pulsing follows Pitch.** C0 to C6 chop their tones on and off. The chop rate is the voicing frequency plus a fixed offset per effect (C2 about 20 Hz under it, C4 right on it, C6 about 43 Hz over it). Each pulse also lasts the same time at every rate: the tone starts at full level and fades out like a voicing pulse does. On the held-out settings, C2, C4 and C6 now play that way:
  - C2: 73 → 77%
  - C4: 65 → 71%
  - C6: 65 → 67%
 
  C0 and C3 stay as they were, because changing them came out worse.
- **Bend moves the noise effects as a block.** C8 and C9 are band-limited noise. On the chip Bend slides the whole band up or down without making it wider. The library used to widen it as it moved, and at bend 15 it leaked noise down to 300 Hz where the chip has nothing below about 560 Hz. With the band moving as a block, plus a change to how the noise is made, C8 and C9 at bend 15 went from 37% to 62%. An older recording I hadn't used for this agrees.
- **M2 ignores Bend completely.** It's identical at bend 0 and bend 15. One oddity: M2's starting formants differ from one recording to the next. In VoxProbe 4 it has no low formant at all. I don't know what sets them yet.
- **A small fix for some traced effects.** Some used to sweep in from a leftover low frequency. C3 had a false sweep near 660 Hz at bend 15; that's gone.

Speech is unchanged; only the effects sound different. The C library is on GitHub (andrew-davie/SpeakJet), and the Go version that Gopher2600 uses is on its way upstream.

Still open: A1 needs re-tracing. On the chip it's a steady low formant plus a straight sweep up to 2.5 kHz, and the library has something quite different. R8 and R9 sit a little lower on this recording than on the older one.

JetSetIlly

Quote from: JetSetIlly on 28 Oct 2026, 03:31 AMI've increased the volume gain because it was hard to hear the atarivox against the TIA output (I suppose that should be made a user option). It sounds closer to the recording but the G in "goes" sounds too soft to me. Also, the screech when you fall off the screen needs work.

I've added a control for speakjet volume gain.
https://github.com/JetSetIlly/Gopher2600
@JetSetIlly@mastodon.gamedev.place
@jetsetilly.bsky.social

SpiceWare

Description for this device says it captures 24-bit audio at 96 kHz - might that help with the analysis?

Andrew Davie

Quote from: SpiceWare on 28 Oct 2026, 04:47 AMDescription for this device says it captures 24-bit audio at 96 kHz - might that help with the analysis?

I think we're doing OK with the current process, actually.  So far Claude has identified several internal mechanisms of the SpeakJet and it's been fascinating to watch. I'm interested to see where this ends up. That it was able, from the manual alone, to get a functional first-pass that actually made legible speech was amazing to me.

The process of making probe ROMS and analysing the recordings you're doing have produced improvements and better understanding of the SpeakJet functionality every round. I'm confident that throwing more analysis at these can still improve the emulation.

We're now close to "good enough" and for all intents and purposes in emulation that would be all we really need. It's replaced Festival for Gopher. I don't think the Stella guys are going to switch, no matter how good it gets; they are going with a different approach.

For me, I have a stella build using this library and of course Gopher uses this library so I'm set with the tools I really wanted/needed. I'm delighted Gopher is using it by default; whatever this "research" contributes to improvements in Stella through additional knowledge/understanding of SpeakJet and supporting their implementation... well, that's great too.


Andrew Davie

I wanted to keep working on this, so I bit the bullet and upgraded my Claude subscription.

**SpeakJet emulation: checked against a real game**

So far the SpeakJet library has been fitted to test ROMs that play one sound at a time. This round I recorded Man Goes Down on two AtariVoxes (an original and a later unit) and compared them with the same game running in Gopher2600 on the library. To make that repeatable, Gopher's HEADLESS mode can now plug in an AtariVox and record just the speech to a WAV. The game says "man goes down", then "may the fruit be with you", then plays effect C1 slowed right down, then "you suck". That's connected speech, with Slow, Bend, Pitch and pauses all mixed in, which the probe ROMs never exercised.

(A small surprise: which phrase you get after pressing fire depends on whether the frame counter is odd or even, `LDA $88 / AND #1`. Odd gives "may the fruit be with you", even gives "run, run, run".)

What the game showed:

- **C1 isn't what we had.** Slowed down, every line in it is a harmonic of a pitch that moves one voicing tick at a time: it holds at 333 Hz, rises to 570 Hz, then falls a tick at a time to 200 Hz. The Pitch setting doesn't change that sweep. Three formants move through it, and oscillator 3 runs free of the voicing as a tone rising to 2.9 kHz. The old version, traced from strongest peaks, was noise in the game and 4.3 dB too loud. It's now rebuilt from these measurements.
- **Slow doesn't stretch S, T, TH and the other voiceless consonants**, only the vowels and voiced sounds. "S S" took 290 ms in the library against 200 ms on the chip.
- **A pause before a stop counts towards the stop's silent closure.** "UX, pause, EK" leaves 140 ms before the K burst on both units, not the 226 ms you'd get by adding the pause and the closure.
- **Sounds played from the chip's own recorded frames no longer fade out and back in** where they join the next sound. OWWW into ZZ had a 30 ms hole the chip doesn't have.
- **ZZ and ZH have hiss at 3.5–4.5 kHz**, which the frame recordings left out.

That last one came from a problem with the measuring rather than the model: the frame-by-frame comparisons had been cutting everything above 4 kHz. There's now one scorer for every recording, covering 150 Hz to 16 kHz. Scores in this post use it, so they don't line up with the numbers in earlier posts.

Results on the game, both units (1.0 would be identical to the chip):

| Phrase | Before | After |
|---|---|---|
| "man goes down" | 0.82 | 0.83 |
| "may the fruit be with you" | 0.75 | 0.875 |
| C1 | 0.60 | 0.85 |
| "you suck" | 0.63 | 0.875 |

Nothing else on the VoxProbe or VoxRecord recordings got worse, apart from C1 at pitch 60 (0.81 → 0.79). The attached graph shows every sound from the original library to now.

Still off:
- IY IY sliding into WW: the chip glides over about 80 ms, the library steps.
- TT's burst is about 40 ms too long.
- GO OWWW is a bit short.
- C1 is a tick off at pitch 150.


Andrew Davie

**SpeakJet emulation: every sound refitted, worst first**

After the Man Goes Down round, I scored every sound in all three AtariVox recordings against the library: both units, the full band from 150 Hz to 16 kHz, with every pitch, bend, speed and repeat variant. I ranked them by how far each one falls short of how well the two real AtariVoxes agree with each other, then worked up from the bottom. Five things came out of it. The rest of the list was either already close or limited by the recordings themselves.

**C3 pulses, and the pulses slow down as it plays.** On both units the gaps go 15, 15, 16, 16, 17, 19, 19, 22, 22 ms. Slow the speed down and the slowing stretches with it; speed it up and it squeezes. So the gate follows the sound's own trace, not the clock. Change the pitch and the pulses keep the same distance below the voicing frequency. The library had a fixed 59 Hz gate, which at pitch 60 pulsed nearly twice as fast as the chip. C3 across all 18 recorded variants went from 67% to 77%, and at pitch 60 from 50% to 72%.

**Bend can wrap a tone round.** At Bend 0, A1 does something strange on the real chip: one of its formants runs backwards, a line falling from 2.4 kHz while the formant itself is rising, and a steady tone appears near 2.4 kHz. Both are the same effect. The formant starts at 120 Hz, Bend 0 takes about 200 Hz off, and the chip doesn't stop at zero: the tone wraps up by about 5.7 kHz and folds back down at the chip's sample rate. That predicts the line to within 10 Hz at every point measured. A1 at Bend 0 went from 59% to 82%.

**C0 pulses with the voicing too.** It runs 4 to 6 Hz under the voicing frequency at every pitch, not at a fixed 85 Hz. It went from 76% to 80%, and at pitch 60 from 73% to 80%.

**C8 and C9's flutter keeps time with Speed, not Pitch.** It measured 19.1 ms between pulses at the default speed, 32.7 ms at speed 90 and 11.2 ms at speed 127, which is exactly how much Speed stretches the sound. C8 at speed 90 went from 62% to 68%.

**The chip hisses under its tones.** Under every pure tone (the DTMF tones, the beeps, the robots) both units have a floor of noise 40 to 55 dB down, from 100 Hz to 3 kHz. The library's tones were perfectly clean, so every band between the tones counted as wrong. A noise floor that follows the output level fixes that: the effects' spectral error dropped from 6.0 to 4.4 dB. Speech doesn't change, because its harmonics hide the floor.

Overall, effects at other pitches and bends went from 79.8% to 81.2%, and at other speeds from 77.8% to 81.0%. Speech and Man Goes Down are unchanged.

Things I tried that didn't help, so they're not in: having C3's low rumble ignore Bend, giving R8/R9's pulse rate a pitch slope, cutting a consonant's hiss the moment the vowel's voicing starts, and gliding repeated copies into each other.

Still open:

- **Repeated sounds.** On the chip each copy glides from the previous one over about 40 ms, and some copies come out shorter than the sound alone. I haven't found a rule for which.
- **M2.** It starts from whatever formants the sound before it left, and these recordings can't tell me which sounds set them. That needs a dedicated test ROM.
- **B9.** Its middle section pulses and has tones the trace is missing. It needs a proper retrace.
- **AW after TT.** On the chip that AW is about 50 ms longer than everywhere else.

The C library (andrew-davie/SpeakJet) and the Go port both have the changes, and they match each other to 3e-8. The measurements are in docs/design.md under "Every sound, worst first". Gopher2600 builds against it.


Andrew Davie

**SpeakJet emulation: a CI hiccup, and the quiet consonants back on the graph**

Two small follow-ups to the consonant shaping.

**CI went red on Linux, now green.** The new consonant filters remade their settings only after a sound's gain had moved by 0.02 dB. That turned out to depend on the compiler. On the Mac, clang fuses a multiply and an add into one step; gcc and clang on Linux x86 don't. That changes the last bit of a number while one sound glides into the next. The 0.02 dB step then tripped one sample earlier or later, and the filters' feedback carried the difference on: the pangram came out 21 steps off on Linux, where the test allows 2. I could reproduce it exactly on the Mac by turning the fusing off. The filters now remake their settings whenever the gain changes at all, so the two builds only ever differ in the last bit, which never reaches the output. The sound moves by less than 0.02 dB, and all four CI jobs pass.

**HE, HO, WH, FF, TH, PE and PO are on the graph now.** They always showed as "left out", and the reason wasn't the chip. They're the quietest sounds it makes, peaking only 7 to 12 dB above the recordings' mains hum. The scorer measured loudness across the whole band, so the hum hid them. Between 1 and 8 kHz, where their hiss actually is, they stand 11 to 26 dB clear. The scorer now retries there when a sound fails the whole-band test, and nothing that already passed changes score. All seven come in at about 84 to 91%, much like the louder fricatives. M1 is the only sound still off the graph, because it clips on every recording.

Andrew Davie

Here are spectral comparisons.  Really converging well at the moment.


Andrew Davie

#53
**SpeakJet emulation: a new yardstick, and everything refitted to it**

NOTE: scoring chart uses new score method, so the accuracy *appears* to have decreased. In reality it's improved, but we have a much harsher scoring criteria.

I'd been comparing the library against recordings of real AtariVoxes by eye, using the spectrum graphs, and kept seeing obvious mismatches that the old score rated as fine. So the scoring came first.

The new score compares the library against two real AtariVoxes at once. The library's output gets each recording's actual background noise (hum, hiss, the 15.8 kHz line) mixed in, and then both are measured by exactly the same rule. It looks at seven things:

- length
- level over time
- the time-frequency picture after alignment
- timing inside the sound
- sideways shifts of the spectrum
- the whole-sound spectrum
- pitch

Each one is judged against how closely the two real chips agree with each other. A score of 0 means "as close as a second chip would be." It covers 752 items: VoxRecord's sounds alone plus its connected groups (Pitch, Speed, Bend, Fast/Slow/Stress, Volume, Repeat, syllables), the fourth VoxProbe, and Man Goes Down.

Then I refitted everything to it. A change was only kept if no kind of recording got worse.

Overall it goes from 1.09 to 0.93:

| | Before | After |
|---|---|---|
| Allophones alone | 0.81 | 0.49 |
| Effects alone | 1.05 | 0.94 |
| Connected groups | 1.08 | 0.97 |
| Man Goes Down | 1.34 | 1.27 |

What changed:

- **Phrase-final lengthening.** This is the interesting one. A sound that's last before a pause plays at a different length on the chip from the same sound mid-phrase. Final IY and AW are 1.25 times longer, while final TT and FF are shorter (0.64 and 0.79 times). No single length per sound could match both, so the library now gives a final sound its own length.
- **Lengths.** Vowels played alone were about 20% short, and voiceless fricatives a bit long. 46 sounds are refitted.
- **Levels.** Eight sounds have new levels where they sit next to others in a phrase.
- **Spectral shaping.** Each sound's shape is now fitted for 98 sounds, effects included (R2, A3 and A8 improved a lot).

Still open:

- The chip has a raised noise floor close to each tone in the tonal effects, and I don't yet know what causes it.
- A handful of traced effects are still well off: B9, A9, C9, A5 and A0.

Andrew Davie

The next probe ROM is ready: It plays for about 4 minutes.

**What's new: items sent on their own.** The sequence is now a list of chunks. Within a chunk, bytes stream as before, so the next bytes are always waiting in the chip's buffer. After a chunk, the ROM waits before sending the next one, so an item sent as its own chunk plays with an empty buffer, the way a game sends a phrase. All earlier probes only ever streamed. Alex Herbert's driver is unchanged; the sequencing is added around it.

**What it tests**

Sections  What                                                Open question
--------  --------------------------------------------------  ----------------------------------
1         AW, IY, SH, B3, D1, A4                              level setting and reference
2-4       SH, UW, UX, BE and 6 others, three ways:            the 15% length gap between
          streamed, alone, alone followed by pauses           VoxProbe and VoxRecord
5         AW, IY, TT, FF followed by pause 0/1/4/5/6,         what triggers phrase-final
          a Pitch command, a Delay, or themselves             lengthening
6         12 sounds repeated 2 and 3 times                    repeat copy lengths, beyond
                                                              VoxProbe 4's 5 and 10
7         D1, R3, B3, B0, A9 at volume 32/64/127              whether the sum/difference tones
                                                              and onset clicks scale with volume
8-9       the 12 worst effects and 8 worst allophones,        clean data to re-trace A9, B9,
          3 times each                                        M2, A5, A0, C8 ...
10        SH, UW, UX, AW at speed 60/90/127                   Speed
11-12     Man Goes Down's four phrases, streamed and then     real game phrases in both
          each alone                                          buffer states

Speeds below 90 come in the last sections, because slow groups have made one unit reset partway through before.

**Checks done**
- It assembles cleanly into 4K.
- All 56 chunks and their waits, read back out of the binary, match the plan in `probe5.json`.
- The reset vector is right, and the sequencer runs before the display's timed code.
- In Gopher2600 headless, the SpeakJet received all 940 commands in the planned order, and every gap between chunks was within a frame of its planned wait.
- Every item sent alone arrived with the chip silent and its buffer empty.
- The emulated run is saved in `VoxProbe5/emulated/` as the reference for the hardware takes.

**Notes for whoever records**
- **Level:** set it lower than last time, peaks around −6 dBFS, because loud effects clipped before. Section 1 plays the loudest sounds first, so it doubles as a level check.
- **The silences are part of the test:** the screen stays red through them.
- **Keep recording 15 seconds after the screen turns green.**
- **Several units are welcome.** Please note which unit (original AtariVox or AtariVox+) was used.

JetSetIlly

Quote from: Andrew Davie on 28 Oct 2026, 12:52 PMI think we're doing OK with the current process, actually.  So far Claude has identified several internal mechanisms of the SpeakJet and it's been fascinating to watch. I'm interested to see where this ends up. That it was able, from the manual alone, to get a functional first-pass that actually made legible speech was amazing to me.
I think the process is getting carried away. What's being added now are effectively special conditions. The size of tables.go is now 333k. It was originally 18k. For a really good emulation, I would expect those values to be calculable at run time from first principles.

I don't think you should spend any more money on this. It's fine as it is.
https://github.com/JetSetIlly/Gopher2600
@JetSetIlly@mastodon.gamedev.place
@jetsetilly.bsky.social

Andrew Davie

Quote from: JetSetIlly on 28 Oct 2026, 09:40 PMI think the process is getting carried away. What's being added now are effectively special conditions. The size of tables.go is now 333k. It was originally 18k. For a really good emulation, I would expect those values to be calculable at run time from first principles.


Fair point, but what I'm trying to do is to *characterise* exactly what is being done by the chip to form all of the various sounds. Discovering where there's fades, where there are oscillators, onsets, delays, whatever.  Then an optimisation phase.  But I agree 333K is ridiculous.   I expect a latter refinement to be programmatic rather than table driven.

Andrew Davie

Claude says...

I agree the tables have grown far too big, but the growth isn't where it looks. Most of it is measured effect traces, and some of that is dead weight that can go with no change to the sound.

```
tables.go, 340 KB       bytes    share   what it is
---------------------  -------  ------  ---------------------------------------
fx2Points               119 K     35%   per-ms tone traces, 40 effects (1390 pts)
fxPoints                108 K     32%   older 5-20 ms traces
  of which never played  50 K     15%   29 effects whose fx2/fxv trace replaces it
vfPoints                 26 K      8%   voiced sounds, frame by frame
c1Points                 12 K      3%   C1
comments                 42 K     12%   (spread through the above)
everything else          ~35 K    10%   formants, lengths, EQ, code
```

**How it got here**
- **It was already 284 KB** when jetsetilly merged PR #6. My recent sessions added about 56 KB, mostly the re-traces from the last two days.
- **Two-thirds is traces:** sampled measurements of the chip's output, 1 to 20 ms apart, one table entry per point.

**On "calculable from first principles"**
- **The allophones mostly are already:** formants, volumes and lengths per sound, played by the synth.
- **The 55 effects can't be.** Each one is an arbitrary sound stored in the chip's ROM, so some per-effect data is unavoidable.
- **But the chip almost certainly stores them compactly**, as short lists of targets and glides on its 4.38 Hz frequency grid, not 1 ms samples. A4, for instance, is basically one sweep plus its harmonic, and we're storing it as about 40 points.

**Proposed consolidation, each step checked against the score:**

1. **Delete the dead traces.** That's 50 KB with no change to the output, verified sample for sample.
2. **Re-encode every trace the way the chip probably stores it:** a few linear glide segments per tone, on the chip's frequency grid. An effect's encoding is accepted only if its score worsens by at most about 0.02. I'd expect the traces to shrink from about 230 KB to 20–30 KB.
3. **Merge the stacked per-sound adjustments.** Fold `refit_length` into the base length table. Then test whether the phrase-final, copy-length and gain adjustments can become class rules (vowels, fricatives, stops) instead of 72 values each, keeping a rule only if the score loss is small. These are only a few KB, but they're the "special conditions" being objected to.
4. **Move the history out of the comments** into the changelog and design doc, keeping only what explains the code.

**Expected result:** tables.go well under 80 KB, the C file shrinking the same way, and the overall score within about 0.02 of today's 0.856. Getting back to the original 18 KB isn't realistic: that version had invented effects, and matching the real chip's 55 effects takes some data.

JetSetIlly

Quote from: Andrew Davie on 28 Oct 2026, 10:24 PMFair point, but what I'm trying to do is to *characterise* exactly what is being done by the chip to form all of the various sounds. Discovering where there's fades, where there are oscillators, onsets, delays, whatever.  Then an optimisation phase.  But I agree 333K is ridiculous.  I expect a latter refinement to be programmatic rather than table driven.
Agreed. The character of the sound is important and it's very difficult to capture. It's much like how we deal with television colour. We can construct the colour mathematically, but people seem to (understandably) prefer precomputed colours that match some hypothetical TV with all the imperfections.

It's similar with SpeakJet I suspect. The SpeakJet chip will create a waveform but it is then given "character" by the DAC and the type of speaker people are used to hearing the AtariVox from.

If Claude can work on cutting the amount of data in the tables that would be a good next step.
https://github.com/JetSetIlly/Gopher2600
@JetSetIlly@mastodon.gamedev.place
@jetsetilly.bsky.social

Andrew Davie

**SpeakJet emulation: the tables in the chip's own numbers**

Following the slim-down, the emulator's data tables now hold the small integers a chip like the SpeakJet would use, not decimal values:

- **Frequencies** are in the chip's own 4.38 Hz steps: its oscillators run on that grid, and the emulator already rounded to it as it played.
- **Volumes** are in eighths of a level, and levels in tenths of a dB.
- **C1's pitch** is in whole voicing ticks, which is how the chip counts it.
- **Times** are in half milliseconds for tone effects, and in hundredths for the sounds played pulse by pulse, which need that precision to stay in step with the voicing.

                before   now
Go tables.go    100 KB   92 KB    (340 KB before the slim-down)
C speakjet.c    163 KB   155 KB
match score     0.855    0.855    (0 = as close as a second real chip)