Raw audio has no packet-loss concealment

[mcfnord]

🤖 AI: A lost raw-audio packet and a lost Opus packet are handled differently a few lines apart in the same loop: a missing Opus frame goes to opus_custom_decode with a null pointer, which is packet loss concealment, while a missing raw frame is memset to zero — a hard digital gap. The consequence is measurable, audible, and present with no loss injected at all.

Measured with one client sending a 440 Hz sine, once as raw PCM and once as Opus, loss injected at the sender, the server's own -R recording as the instrument. Silence gaps of 64 samples or more, counted over 17 s of steady state:

injected loss	raw: gaps/s	raw: zero samples	Opus: gaps/s	Opus: zero samples
0%	2.1	0.81%	0.0	0.01%
2%	9.2	3.20%	0.0	0.01%
5%	16.8	4.85%	0.0	0.01%

At 2% loss the longest single raw gap ran 1664 samples, 35 ms.

Repeating it with a synthesised musical phrase, and with the Opus arm at AQ_HIGH's real 82 coded bytes rather than a size picked to avoid a collision, gives the same split for raw: 0.6 discontinuities/s at 0% injected loss, 7.9/s at 2%, 18.5/s at 5%, with the jump to zero reaching 3.0–3.8× the signal RMS. A listener judging the recordings by ear counted 8 audible clicks in the 0%-loss raw recording (the detector found 9), "many" at 2%, "very many" at 5%.

Both modes on one scale

Counting discontinuities favours raw's failure mode: a zero-fill leaves a sample-level jump, concealment leaves a smooth but wrong signal, and the same listener reported weak artifacts in the AQ_HIGH recordings that no discontinuity detector scored above zero. Comparing against the signal that was sent measures both. Per 20 ms frame, log-spectral distance across 24 bands, aligned on chirp markers embedded once per second, a frame counted bad at 8 dB. Two runs of each arm:

injected loss	raw: bad frames	AQ_HIGH: bad frames
0%	4.5%, 3.6%	1.4%, 1.1%
2%	12.0%, 8.3%	5.3%, 5.0%
5%	21.3%, 19.6%	12.3%, 13.2%

Raw is worse at every rate in both runs, including with nothing injected. AQ_HIGH is not clean either — 1.4% and 1.1% of frames are wrong with no loss injected, which is what the listener described for those same files. The difference is in kind as well as degree, and only the discontinuity is what a listener called jarring.

The same link loses more raw packets in the first place

The two modes send at the same packet rate — one packet per 2.67 ms, ~375/s. Only the size differs: 257 bytes for raw mono against 83 for AQ_HIGH mono. So this is not "more packets to lose", it is "more bytes to fit", and it only bites below a threshold. Measured on a shaped link (tbf, client→server, server in its own network namespace), counting audio datagrams that arrived:

shaped link	raw, 257 B/pkt	AQ_HIGH, 83 B/pkt
400 kbit	52.0% loss	0.0%
600 kbit	28.7% loss	0.0%
900 kbit	0.0%	0.0%
1500 kbit	0.0%	0.0%

At 400 and 600 kbit the choice of quality setting decides whether audio arrives at all; by 900 kbit both fit. A repeat of the two shaped points returned 51.1% and 27.5%, so the run-to-run spread is about one point. That is the range where the two effects compound: raw loses the packets and has no concealment for the ones it loses, while AQ_HIGH on the same link loses none.

Concealment for the raw path is the obvious question, and it is not free — the decoder has no model of the signal to conceal with, which is exactly what Opus provides. Repeating the previous frame, or a short fade to silence instead of an instant memset, would remove the discontinuity without one.

Worth noting for anyone reproducing this: a discontinuity detector cannot compare the two modes, because it can only see one of them. Three detectors here scored the AQ_HIGH recordings at zero while a listener heard artifacts in them, and the reference comparison above is what resolved the disagreement.

Rig, harness and the full method are available if useful. How a server tells raw from Opus in the first place is a separate matter, covered in #3894.

---

🤖 This message was written by AI and reviewed by @mcfnord.

[dingodoppelt]

This is a known fact. In practice a user would simply switch servers, if the noise is too bad. In practice raw audio sounds better than opus.
An artificial test to prove a point doesn't help in solving an artificial issue.

Repeating the previous frame, or a short fade to silence instead of an instant memset, would remove the discontinuity without one.

The term "discontinuity" is used in the wrong context here. Yes, introducing latency to being able to crossfade to and from silence after packet loss would reduce audible clicks. Repeating a frame would still produce clicks because it would itself produce a discontinuity at a known rate producing a beep instead of a click. Removing a discontinuity in the mathematical sense requires a lot more.
Most PLC algorithms are optimised for speech and don't work too well on polyphonic signals (piano, guitar) or transients (drums). If we wanted to get this right, we'd be publishing academic papers about our findings, really.

[softins]

I had a useful and detailed conversation with Google AI about this a month ago, but haven't had the time to follow up on it. I don't have it as Markdown to paste here, but did save it as html. Which is just as well, as the original conversation won't reload for me.

I have uploaded the saved conversation to https://softins.co.uk/jamulus-plc.html - ignore the first third (except for background), and scroll down to Alternative Architecture: Lightweight, Stateless PCM PLC. Its suggestions look promising and worth exploring, but it hasn't got to the top of my pile yet! Happy if anyone else wants to pick it up.

[dingodoppelt]

I have uploaded the saved conversation to https://softins.co.uk/jamulus-plc.html - ignore the first third (except for background), and scroll down to Alternative Architecture: Lightweight, Stateless PCM PLC. Its suggestions look promising and worth exploring, but it hasn't got to the top of my pile yet! Happy if anyone else wants to pick it up.

It proposes "PSOLA" but I can't see how that would work. The server would need to track the pitch of every client's signal (either by autocorrelation or FFT) and that only works for monophonic, periodic signals (no drums, piano, guitar).
The only time-domain algorithm I can think of is introducing latency and crossfade to silence to mitigate some of the clicks. What the AI proposes is basically a pitch shifter or phase vocoder and doesn't seem fit here.

[softins]

It proposes "PSOLA" but I can't see how that would work. The server would need to track the pitch of every client's signal (either by autocorrelation or FFT) and that only works for monophonic, periodic signals (no drums, piano, guitar). The only time-domain algorithm I can think of is introducing latency and crossfade to silence to mitigate some of the clicks. What the AI proposes is basically a pitch shifter or phase vocoder and doesn't seem fit here.

Oh, right. As I said, I haven't yet studied its suggestions at all, and just wanted to add it to the conversation.

[softins]

I wonder if clicks on missing packets might be reduced by just smoothing discontinuities. So instead of inserting a frame of zeroes when there is a lost packet, it constructs a frame containing a straight line from the last sample of the previous packet to the first sample of the next available packet? That would assume we don't insert a missing packet until we have the valid packet that follows it.

Just thinking out loud, and don't know if that would really work.

[dingodoppelt]

I wonder if clicks on missing packets might be reduced by just smoothing discontinuities. So instead of inserting a frame of zeroes when there is a lost packet, it constructs a frame containing a straight line from the last sample of the previous packet to the first sample of the next available packet? That would assume we don't insert a missing packet until we have the valid packet that follows it.

Just thinking out loud, and don't know if that would really work.

This is still a discontinuity (first derivative undefined). This is why we need to know more about the signal than the last known sample value.

[mcfnord]

🤖 AI: The four ideas here were tested offline against the sent signal — piano, guitar and drums, 2% and 5% loss, mean burst 1 and 4, 12 cells, scored as log-spectral distance per 20 ms frame.

Frame repetition and fading to silence are both worse than the current zero-fill in all 12. A straight line is better in all 12 — on piano at 5% it cuts the distance from 4.10 to 1.62 — and needs no new machinery. Its second endpoint is the first sample after the gap, so where that packet is missing too the arm ramps to zero instead: 74% of concealed frames in the mean-burst-4 cells, 6% at burst 1. The client's raw branch has the same gap.

The linked page's PSOLA is worse than zero-fill in 11 of 12, but its crossfade is why: it blends against the 48 samples before the gap, 48 out of phase with the frame being written, overwriting the part the extrapolation got right. Delete that loop and it beats zero-fill in all 8 pitched cells and is best of everything on guitar — while staying worse on drums in 3 of 4, where its pitch search rails at its bounds in 20 of 80 windows.

Caveat: this metric ranks how wrong a frame is, not how much it annoys, and it puts zero-fill ahead of three of the four proposals. Ranking by ear is the next step.

[dingodoppelt]

🤖 AI: The four ideas here were tested offline against the sent signal — piano, guitar and drums, 2% and 5% loss, burst 1 and 4, 12 cells.

Frame repetition and fading to silence are both worse than the current zero-fill in all 12. A straight line is better in all 12, by 1.07 to 11.6 times, and needs no new machinery — both endpoints are already in hand, and the client's raw branch has the same gap.

The linked page's PSOLA is worse than zero-fill in all 12, but its crossfade is why: it blends against the 48 samples before the gap, 48 out of phase with the frame being written, overwriting the part the extrapolation got right. Delete that loop and it beats zero-fill in all 8 pitched cells and is best of everything on guitar — while staying worse on drums in 3 of 4, where its pitch search rails at its bounds in 20 of 80 windows.

Caveat: this metric ranks how wrong a frame is, not how much it annoys, and it puts zero-fill ahead of three of the four proposals. Ranking by ear is the next step.

I don't actually believe the AI got a working implementation for anything here. Better not waste any more tokens on this.

[mcfnord]

I don't actually believe the AI got a working implementation for anything here. Better not waste any more tokens on this.

AI has a working implementation for everything here if you wish to use it.

[dingodoppelt]

I don't actually believe the AI got a working implementation for anything here. Better not waste any more tokens on this.

AI has a working implementation for everything here if you wish to use it.

Yeah, I'd like to see the PSOLA algorithm and how it did the pitch recognition and sample alignment. Though I think the PSOLA approach isn't the way to go at all I'm interested in the AI's solution.

[mcfnord]

🤖 AI: Code here — it runs on any 48 kHz stereo raw file. The PSOLA arm is the page's JamulusPcmPlc transcribed to numpy, constants unchanged, so it is that algorithm rather than a new one.

Pitch recognition: AMDF over lags 40 to 600 samples (1200 Hz to 80 Hz), minimising the sum of |x(n) − x(n−τ)| over the 480 samples ending at the gap. The argmin wins — no interpolation, no voicing test, no confidence threshold. Positive control: on a 220 Hz sine it returns 218, against a true 218.18.

Sample alignment: none beyond that period. output[i] = history[HIST_SIZE - pitch + (i % pitch)] is pinned exactly one period back, and nothing searches for a matching start sample. Which is where your percussion point lands: on drums the search returns one of its own bounds, meaning it found no period at all, in 20 of 80 windows, against 0 of 61 on guitar.

[dingodoppelt]

🤖 AI: Code here — it runs on any 48 kHz stereo raw file. The PSOLA arm is the page's JamulusPcmPlc transcribed to numpy, constants unchanged, so it is that algorithm rather than a new one.

Pitch recognition: AMDF over lags 40 to 600 samples (1200 Hz to 80 Hz), minimising the sum of |x(n) − x(n−τ)| over the 480 samples ending at the gap. The argmin wins — no interpolation, no voicing test, no confidence threshold. Positive control: on a 220 Hz sine it returns 218, against a true 218.18.

Sample alignment: none beyond that period. output[i] = history[HIST_SIZE - pitch + (i % pitch)] is pinned exactly one period back, and nothing searches for a matching start sample. Which is where your percussion point lands: on drums the search returns one of its own bounds, meaning it found no period at all, in 20 of 80 windows, against 0 of 61 on guitar.

Nice, but all this works (if at all) on monophonic and periodic signals only and introduces latency of around 12 ms for the analysis window. No windowing takes place which introduces new discontinuities. This looks like a PSOLA implementation, but it wouldn't work well (if at all).
Funnily the most promising all purpose PLC algorithms seems to be stochastic models (very small neural networks) and not the traditional DSP algorithms.

[pljones]

If three people are on a server, is it better, when on of them has a missed packet for a certain mix point, to muddy the sound of the two existing packets with something "made up" (any algorithm) or simply keep the mix "clean"? Does it "self-heal" to the participants?

If there are 150 stereo clients at raw audio getting mixed, what's the lowest CPU approach to avoid mixing time exceeding packet time (64 samples)?