Sound mixing makes nasty noises

> On 2020-12-27, at 10:49 AM, Herbert König <[hidden email]> wrote:
>
> I searched all method sources for 3276 and only found one suspicious
> place in :
> adjustVolumeTo: vol overMSecs: mSecs

I even tried not using the adjustVol... at all in case that was a cause - no change whatsoever.

Next stop, using GDB. I hate using GDB.

I suppose one option is also faking a nice clean sound buffer and finding a way to feed it as directly as possible to various stages. Maybe it's even a problem in the audio libraries! Not like we haven't discovered similar issues in the past.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Hipatitis (n): Terminal coolness

I changed SoundPlayer class playLoop which I figured is what passes the samples to the OS like attached, writing 1 mega samples to an aiff file. TSTTCPW.
I must have made a mistake because I suddenly get lost buffers but frankly I had some wine :-).

I used Tim's first snippet:

"awful noise when mixing"
|snd|
snd := FMSound organ1.
snd setPitch: 440 dur: 10 loudness: 0.9;
    play.
1 second wait.

FMSound brass1 setPitch: 470 // 2 dur:2 loudness: 0.9;
    play.
1 second wait.
snd stopGracefully

It's repeatable the sound seems to have the right length but the discontinuity marked red usually happens if the sound system of the OS looses a buffer. I had my laptop run at 4 GHZ, the audio output sounded normally bad and the loss is in the first second, where the audio is still ok. Playing the recorded file reveals the additional distortions.

I just want to save another trip through the time zones so I put it out buggy as it is. Maybe I just used the wrong place to do my recording, maybe I recorded mono from a stereo buffer, whatever.

Be mild :-)

Herbert

Next step is to try to record the output before it goes to the OS and
see if the image or some primitive creates the problem.

Cheers,

Herbert

I tested Tim's snippet (modified to use Delay forSeconds: ) on Squeak 3.6 (Windows) and it has the same problem. But it's only clipping and no wrapping on Windows! Still very ugly.

Tomorrow I'll switch to a Pi to see if my recording to file also shows wrapping. Then we'll know if it's the primitive handling audio to the OS in Linux or Rapbian or if it happens inside Squeak's sound architecture.

Cheers,

Herbert

I changed SoundPlayer class playLoop which I figured is what passes the samples to the OS like attached, writing 1 mega samples to an aiff file. TSTTCPW.
I must have made a mistake because I suddenly get lost buffers but frankly I had some wine :-).

I used Tim's first snippet:

"awful noise when mixing"
|snd|
snd := FMSound organ1.
snd setPitch: 440 dur: 10 loudness: 0.9;
    play.
1 second wait.

FMSound brass1 setPitch: 470 // 2 dur:2 loudness: 0.9;
    play.
1 second wait.
snd stopGracefully

It's repeatable the sound seems to have the right length but the discontinuity marked red usually happens if the sound system of the OS looses a buffer. I had my laptop run at 4 GHZ, the audio output sounded normally bad and the loss is in the first second, where the audio is still ok. Playing the recorded file reveals the additional distortions.

I just want to save another trip through the time zones so I put it out buggy as it is. Maybe I just used the wrong place to do my recording, maybe I recorded mono from a stereo buffer, whatever.

Be mild :-)

Herbert

Am 27.12.2020 um 19:49 schrieb Herbert König:

Next step is to try to record the output before it goes to the OS and
see if the image or some primitive creates the problem.

Cheers,

Herbert

Thanks for banging on this Herbert - I suspect a few drinks might actually help with this one...

GDB is such fun, not.

So far as I can see the algorithm is doing correct things - the 16bit values get read & written without whacking other values, we get suitably matching left & right channel values, when the value over/under flows the max value is properly written, etc etc. Of course, it's extra fun to mentally deal with the 'short' addressing.

I've double checked the 'FMSound new' wavetable values in case they changed across some release but they appear to be the same.

Anybody got any more ideas? In particular, a recipe for recording the output with alsa would be interesting so at least I could add the record of the old-alsa-pi getting it right!

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: KFP: Kindle Fire in Printer

Hi,

after a lot of trying I managed to have

parec --file-format=wav
--device=alsa_output.pci-0000_00_1b.0.analog-stereo.monitor test.wav

record the audio output of Squeak. Device is found via pactl list sinks
and Squeak is started via

exec padsp "${VM}" "${IMAGE}"

 From there on I used my slightly improved but still not working
playLoop to record inside Squeak and the above to record outside Squeak.

Inside Squeak I saw a lot of clipping as expected, outside I saw
wrapping whenever it clipped inside. I saw no buffer loss in the
recording outside of Squeak up to the 6 voices I checked. My recording
inside playLoop still looses most of the buffers.

I used:

SoundPlayer startReverb.
FMSound organ1 setPitch: 261.6 dur: 10 loudness: 0.3 ; play.
FMSound organ1 setPitch: 329.6 dur: 10 loudness: 0.3 ; play.
FMSound organ1 setPitch: 392 dur: 10 loudness: 0.3 ; play.
FMSound organ1 setPitch: 523.2 dur: 10 loudness: 0.3 ; play.
FMSound organ1 setPitch: 659.3 dur: 10 loudness: 0.3 ; play.
FMSound organ1 setPitch: 784 dur: 10 loudness: 0.3 ; play.

C major chord plus an octave above.

With loudness 0.2 nearly no clipping occurred up to 0.3 with constant
clipping. As playing even one voice sets the CPU to 100% load on the Pi
(as per top). I checked up to 6 voices to see if it was a matter of CPU
load which it seems to be not on a Pi3.


Conclusion: It seems to happen somewhere between

primSoundPlaySamples: count from: Buffer startingAt: 1 (Before that I
recorded inside Squeak to a file)

and the output to the os. primSoundPlay... is sent in SoundPlayer class
playLoop.

quoting Tim:

Staring at the generated C code it looks possibly suspicious that we have

short int *aSoundBuffer;
sqInt s;
                if (leftVol > 0) {
                        i = (2 * sliceIndex) - 1;
                        s = (aSoundBuffer[i]) + ((sample * leftVol) / ScaleFactor);
                        if (s > 0x7FFF) {
                                s = 0x7FFF;
                        }
                        if (s < -32767) {
                                s = -32767;
                        }
                        aSoundBuffer[i] = s;
                }
But surely no C compiler screws that up these days?

if that's from the primitive, why do we clip here? Clipping is done
inside Squeak.

Cheers,


Herbert



Am 25.12.20 um 14:54 schrieb K K Subbu:

> On 2020-12-28, at 10:36 AM, Herbert <[hidden email]> wrote:
>
> Inside Squeak I saw a lot of clipping as expected, outside I saw
> wrapping whenever it clipped inside.

That would be an interesting thing to prove conclusively. It's not unimagineable that there could be a bug in a unix sound library. Just ofr a moment I got excited looking at the pulse driver code when I spotted it is using
audioIn.pa_spec.format = PA_SAMPLE_S16LE;
... because I could have sworn that squeak is generating 16 bit BIG endian samples. But no...

Maybe if we make a crafted sound buffer with a clean waveform that we know never gets clipped but does hit the maxval/minval and play it directly with no mixing related calls (because the mixer code does volume scaling etc and internal clipping) we might see what comes out of pulseaudio and indeed alsa.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
MIPS:  Meaningless Indicator of Processor Speed.

Am 28.12.2020 um 22:48 schrieb tim Rowledge:
>> On 2020-12-28, at 10:36 AM, Herbert <[hidden email]> wrote:
>>
>> Inside Squeak I saw a lot of clipping as expected, outside I saw
>> wrapping whenever it clipped inside.
> That would be an interesting thing to prove conclusively. It's not unimagineable that there could be a bug in a unix sound library.
I'd suspect the Squeak primitive that hands the buffers to the OS before
looking into OS code.

Would it be conclusive to record the outside for a minute or so, read
that file and count wrapping
(sample(n) = +fullscale and sample(n+1) = -fullscale) or (sample(n) =
-fullscale and sample(n+1) = +fullscale)
and do the same for all SoundBuffers we get hold of inside Squeak during
the recording? (repeatedly checking SoundBuffer allInstances)

> Maybe if we make a crafted sound buffer with a clean waveform that we know never gets clipped but does hit the maxval/minval and play it directly with no mixing related calls (because the mixer code does volume scaling etc and internal clipping) we might see what comes out of pulseaudio and indeed alsa.
AbstractSound or the like has a class var Sine with that but I feel
that's directed at finding a bug in the OS and discards the possibility
of Squeak messing it up. And the organ sounds used are predictable
enough re clipping but I hope the individual envelopes add some
randomness to cover more cases.

Cheers,

Herbert

I changed playLoop to just copy the current SoundBuffer into a collection. I got some 11 seconds sound (2 below)
Each buffer I differentiated and took the abs value. (see attached st)
Of those I took the max. In case of wraparound that would be close to 64k. 
The zeroes in these buffers are the clippings. (3 below) Again I took the max. (4 below).

I can't email screenshots from the Pi but I did it on a Pi too. I used Tim's first snippet. I found a lot of clipping inside Squeak and not a single wraparound in several attempts.

Same observation id waggling the Keyboard or playing my 6 voiced C major while waggling the keyboard morph.

On the Pi3 I still lost buffers (getting 7 seconds worth of buffers from 10+ seconds of Audio). I would not search in the Squeak Smalltalk code for the problem but would suspect the primitive where Squeak hands the output to Raspbian.

That said clipping still makes for horrible sound (SCNR :-).

Cheers,

Herbert

Am 28.12.2020 um 22:48 schrieb tim Rowledge:

On 2020-12-28, at 10:36 AM, Herbert [hidden email] wrote:

Inside Squeak I saw a lot of clipping as expected, outside I saw
wrapping whenever it clipped inside.

That would be an interesting thing to prove conclusively. It's not unimagineable that there could be a bug in a unix sound library.

I'd suspect the Squeak primitive that hands the buffers to the OS before
looking into OS code.

Would it be conclusive to record the outside for a minute or so, read
that file and count wrapping
(sample(n) = +fullscale and sample(n+1) = -fullscale) or (sample(n) =
-fullscale and sample(n+1) = +fullscale)
and do the same for all SoundBuffers we get hold of inside Squeak during
the recording? (repeatedly checking SoundBuffer allInstances)

Maybe if we make a crafted sound buffer with a clean waveform that we know never gets clipped but does hit the maxval/minval and play it directly with no mixing related calls (because the mixer code does volume scaling etc and internal clipping) we might see what comes out of pulseaudio and indeed alsa.

AbstractSound or the like has a class var Sine with that but I feel
that's directed at finding a bug in the OS and discards the possibility
of Squeak messing it up. And the organ sounds used are predictable
enough re clipping but I hope the individual envelopes add some
randomness to cover more cases.

Cheers,

Herbert

--

> On 2020-12-29, at 11:03 AM, Eliot Miranda <[hidden email]> wrote:
>
> Hi Herbert, Hi Tim,
>
>     I suspect I've made a mistake somewhere.

Well, I guess that happens occasionally - but I should note that my experiments are using a 5.3 release image (though I have tried the fairly up to date 6alpha stream as well). and so don't have your change involved.

Herbert - the  prim to play the sound simply passes pointers around and I can't find any place where values might get changed. Given that both ALSA and pulseaudio allow setting of the sample format in order to handle 8, 16, 32 bt, integer and float, compressed, spindled, folded and mutilated, it is certainly possible there is an issue, though finding it will be 'fun'.

One thing I should do is dig out another Pi and set it up with a clean 'last before pulse' copy of the OS and make sure we know what happens in that case.

So many things to break, so little time...

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
The world is not magic

Hi Tim, Hi Eliot,

Eliot our problem is on the Pi only, not on Windows10 not on Linux Mint
and I think Tim uses Mac Os w/o these problems. 5.3 release Squeak here too.

I created a fullscale sine and a 1.01 * fullscale sine in Audacity
(cross platform audio editor) on the Pi, saved them as 16 Bit Wav and
read them again -> all as expected. But Audacity was a bit unclear if it
used Pulse or Alsa.

Next I'll create these sines in Python where I hope to have more control
over the audio drivers.

Cheers,

Herbert

Am 29.12.2020 um 20:49 schrieb tim Rowledge:

The Great Saga marches onward...

I've hacked the playLoop to also save the samples sent to SoundPlayer class>>#primSoundPlaySamples:from:startingAt: in a SamplesSound with a big buffer. Running sample sound snippets results in a saveable sample that I can then play in various ways on other machines.

Right now it appears to strongly suggest that the samples being created by Squeak are just fine; make horrible noises on the pulse-Pi, copy the generated .wav to an alsa-Pi and it sounds fine. Play the .wav on my iMac via iTunes and it sounds fine. try to play the .wav via VLC media player on the pulse-Pi and if anything it sounds worse. VLC player on the alsa Pi sounds very similar to the iTunes/iMac.

More testing to try but I'm leaning towards the pulse system on the Pi being borked.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
To err is human; to really foul things up requires a computer.

> On 2021-01-01, at 9:38 PM, tim Rowledge <[hidden email]> wrote:
>
> The Great Saga marches onward...

Attached is a changeset that has the hacked playLoop and a rather long preamble that *should* explain it.

Basically, make a global BigBuffer, change playLoop to write samples to BigBuffer if it is not nil and not full.
Inspect the SampledSound that owns a big buffer and use the included code snippets (that magically appear in the inspector, woo-hoo!) to point BigBuffer at a .. buffer... make some noises and the set BigBuffer to nil to stop recording.

Save the processed samples as a WAV file and try them on other machines and setups. You can load WAV files 'as sound' from the Squeak FileList, or play them with iTunes, VLC, whatever.




tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
"How many Motie Mediators does it take to chage a lightbulb?”
"Are you insane? Only Crazy Eddie would want to change *anything*!"

> On 2021-01-02, at 2:14 PM, tim Rowledge <[hidden email]> wrote:
>
>
>
>> On 2021-01-01, at 9:38 PM, tim Rowledge <[hidden email]> wrote:
>>
>> The Great Saga marches onward...

The recently update pulse libraries for Pi seem to have removed a lot of the especially ugly graunching, which is nice to hear. It's still quite nasty when using full volume range sounds and playing multiple sounds simultaneously though; maybe some use of MixedSound might help but it remains true that the alsa setup doesn't do this.

I've also realised that there is a definite buffer-size aliasing problem regardless of the alsa/pulse setup.

Given that the normal buffer size is set to 120mS worth of samples, consider this snippet


|snd volume dur|
volume:= 100.
dur := 120.
10 timesRepeat: [
        snd := FMSound new.
        snd
                setPitch: (AbstractSound pitchForMIDIKey: 60)
                dur: 1000
                loudness: ((volume / 100.0) within: 0.0 and: 1.0).
        snd play.
        dur milliSeconds wait.
        snd stopGracefully.
        snd := FMSound new.
        snd
                setPitch: (AbstractSound pitchForMIDIKey: 58)
                dur: 1000
                loudness: ((volume / 100.0) within: 0.0 and: 1.0).
        snd play.
        dur milliSeconds wait.
        snd stopGracefully]

This makes a nice dee-dah-dee-dah... sequence, very cleanly.

Change 'dur' to 150 and try again...on either sound library the second version is incorrect and more dee-dah-de-da-dee-daaah-d-dah.. sort of choppy. 300 is less awful but still wrong. 240 is fine, as one might imagine.

Just whose idea was generating sounds with computers? I want a word...


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Flabbergasted (adj.), appalled over how much weight you have gained.

On 03/01/21 6:23 am, tim Rowledge wrote:
> I've also realised that there is a definite buffer-size aliasing
> problem regardless of the alsa/pulse setup.
>
> Given that the normal buffer size is set to 120mS worth of samples,
> consider this snippet

On Pi, the cpu gets throttled based on temperature and power rail
voltage levels. This may cause variations in sound latency.

You can monitor actual latency on your sink/playback/output device with:
----
~$ pactl list sinks | grep -i 'sink\|latency'
Sink #0
        Latency: 75611 usec, configured 76000 usec
        Flags: HARDWARE HW_MUTE_CTRL HW_VOLUME_CTRL DECIBEL_VOLUME LATENCY
----

HTH .. Subbu

Hi,
so yes, a Pi does indeed throttle when it gets hot - but hot is > 60C core temperature and believe me, none of my Pi ever get anywhere near that. Nor are they suffering low-voltage problems right now; I had a nasty issue with a failing PSU some time last year, or last century, whenever we think 2020 was. All fixed now with bigger margins on the supply.

It really does appear to be an issue with the basic design of the unix sound handling prims. Without digging too deep (because I'm knackered this weekend)  it looks to me as if the unix sound prims are all single buffer stuff, whereas the Mac one appears double-buffered. For Windows I can't tell because none of the code makes any sense to me. For RISC OS there is a curious OS level buffering system that I vaguely understood for about 10 minutes 10 years ago.

Currently we by default create 120mSecs worth of buffer, fill it up and send it out to be played. Supposedly we can work out how many slots are free to refilled at any time but what does that even mean? It depends upon the details of the OS sound handling; if the OS takes a copy of the squeak buffer then in a sense said squeak buffer is now completely empty and ready to be filled completely. But what if the next attempt to pass a buffer finds that the OS cannot accept any more samples just yet? How complicated can we make this?

Even on a Mac you can trivially see the effect of some level of aliasing in the combination of the size of the buffer and the lengths of the 'notes' played by that snippet. Just try playing it with 'dur' set to 120 and then 150. Then edit the SoundPlayer class>>#initialize method to change the BufferMSecs value, then try again. It's really audible. But be careful - it appears to be possible to break the sound player completely by changing the BufferMSecs by much.

The problem seems to me that the interface is really set up for streaming not short chunk work. The Squeak side of it was designed 20 years ago on the basis of machines barely 1% as fast as the slowest machines we generally use these days (I'd guess a Raspberry Pi Zero would count for that) and has barely changed. We probably ought to do a lot better now.


> On 2021-01-02, at 11:02 PM, K K Subbu <[hidden email]> wrote:
>
> You can monitor actual latency on your sink/playback/output device with:
> ----
> ~$ pactl list sinks | grep -i 'sink\|latency'

That's seriously cool, thanks.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Useful Latin Phrases:- Ne auderis delere orbem rigidum meum! = Don't you dare erase my hard disk!

On 04/01/21 7:01 am, tim Rowledge wrote:
> It depends upon the details of the OS sound handling; if the OS takes
> a copy of the squeak buffer then in a sense said squeak buffer is now
> completely empty and ready to be filled completely. But what if the
> next attempt to pass a buffer finds that the OS cannot accept any
> more samples just yet? How complicated can we make this?

Latency could be introduced in any layer - driver, kernel, library, app.

* aplay plays local files well that rules out any latency issues in
lib/kernel/driver layers.
* PulseAudio introduces another buffering layer with its audio server
app. pactl should reveal this.
* Squeak also uses its own buffers. As an app, it could be subject to
fair-share scheduling. As an interpreter, Squeak would be CPU intensive
workload. If there are other CPU intensive loads, then its scheduling
latency may impact audio play.

In addition to pactl, I would also have vmstat running in a separate
terminal to reveal CPU/RAM/IO spikes.
  $ vmstat 5

HTH .. Subbu

On Mon, Jan 04, 2021 at 10:46:45AM +0530, K K Subbu wrote:
<OT>
Hi Subbu,

I have been using Linux for many years, but I am always learning new
things from the tips and explanations that you provide here.

So thank you :-)

Dave
</OT>