Smalltalk › Squeak › Squeak - Dev

MixedSound (& consequently stereo) is badly broken in both 32-bit and 64-bit versions

_‹ Previous Topic Next Topic _›

Classic

List

Threaded

16 messages Options

Eliot Miranda-2

MixedSound (& consequently stereo) is badly broken in both 32-bit and 64-bit versions

Hi All,

the SampledSound>>#mixSampleCount:into:startingAt:leftVol:rightVol: method looks to be badly broken. I don't understand the index scaling (yet) so I thought I'd throw this out there and see if anyone can spot when and why this broke.

Here's a simple example that generates a second long A below middle C (220Hz), smoothing start and stop, and then plays it, so far so good:

| samples sineTable sound |

"1 second of A below middle C (220Hz). 16000 / 220 is 72.72 recurring"

sineTable := SoundPlayer sineTable: 73.

sineTable doWithIndex: "And let's not deafen anyone..."

[:sample :index| sineTable at: index put: sample // 4].

samples := SoundBuffer new: 16000.

1 to: samples size by: sineTable size do:

[:i| samples replaceFrom: i to: (i + sineTable size - 1 min: 16000) with: sineTable startingAt: 1].

1 to: 146 do: "smooth start and end of the sound"

[:i|

samples at: i put: ((samples at: i) * i / 146) asInteger.

samples at: 16001 - i put: ((samples at: 16001 - i) * i / 146) asInteger].

(SampledSound samples: samples samplingRate: 16000) play

Now let's create a stereo sound image and try and play that:

| samples sineTable sound |

"1 second of A below middle C (220Hz). 16000 / 220 is 72.72 recurring"

sineTable := SoundPlayer sineTable: 73.

sineTable doWithIndex:

[:sample :index| sineTable at: index put: sample // 4].

samples := SoundBuffer new: 16000.

1 to: samples size by: sineTable size do:

[:i| samples replaceFrom: i to: (i + sineTable size - 1 min: 16000) with: sineTable startingAt: 1].

1 to: 146 do:

[:i|

samples at: i put: ((samples at: i) * i / 146) asInteger.

samples at: 16001 - i put: ((samples at: 16001 - i) * i / 146) asInteger].

sound := SampledSound samples: samples samplingRate: 16000.

sound := MixedSound new

add: sound pan: 0.25;

add: sound pan: 0.75;

yourself.

sound play

Eek!!

You can examine the mixing without hurting your ears by replacing

"sound play"

with

"sound computeSamplesForSeconds: sound duration"

which invokes the mixing directly.

Any help making sense of this gratefully received. Some comments around the

scaledIndex := scaledIndex + scaledIncrement.

scaledIndex >= ScaledIndexOverflow ifTrue: [

overflow := scaledIndex >> IncrementFractionBits.

indexHighBits := indexHighBits + overflow.

scaledIndex := scaledIndex - (overflow << IncrementFractionBits)].

in SampledSound>>#mixSampleCount:into:startingAt:leftVol:rightVol: would be most welcome.

And if you simply comment out the primitive teh Smaklltalk code produces garbage also, so the problem does not seem to be Slang but the algorithm itself.

_,,,^..^,,,_

best, Eliot

Stéphane Rollandin

Re: MixedSound (& consequently stereo) is badly broken in both 32-bit and 64-bit versions

As a first and fast answer, let me say that I did not notice anything
broken there in years of doing sound and music in Squeak.

Did you check if the way you populate the samples buffers is the same as
in #fromWaveStream: which is used to load wav files and works correctly
AFAIK?

There are some useful tools in muO. For example, fetch the (32 bit)
image in the rogue352.zip here:
http://www.zogotounga.net/comp/squeak/rogue.htm , then copy the attached
sound file in the root folder and do

(SampledSound fromWaveFileNamed: 'sound.wav') edit
(SampledSound fromWaveFileNamed: 'sound.wav') play

Works fine here.

Stef

sound.wav (448K) Download Attachment

Stéphane Rollandin

Re: MixedSound (& consequently stereo) is badly broken in both 32-bit and 64-bit versions

> (SampledSound fromWaveFileNamed: 'sound.wav') edit

A note about the editor:

Left-click in one of the two displayed channel and select 'edit
[element]' to see the waveform and spectrogram (which is enabled via the
'spectrogram' checkbox in the contextual menu).

You can scale the display smoothly to an arbitrary precision by holding
SHIFT and click-dragging in the window.

Stef

Eliot Miranda-2

Re: MixedSound (& consequently stereo) is badly broken in both 32-bit and 64-bit versions

In reply to this post by Stéphane Rollandin

Stef,

if you play the example you’ll see that the first version works fine. The second version, which adds MixedSound yo create stereo doesn’t. That proves that the issue is in MixedSound. Please just try the two examples.

_,,,^..^,,,_ (phone)

> On Sep 19, 2020, at 12:46 AM, Stéphane Rollandin <[hidden email]> wrote:
>
> As a first and fast answer, let me say that I did not notice anything broken there in years of doing sound and music in Squeak.
>
> Did you check if the way you populate the samples buffers is the same as in #fromWaveStream: which is used to load wav files and works correctly AFAIK?
>
>
> There are some useful tools in muO. For example, fetch the (32 bit) image in the rogue352.zip here: http://www.zogotounga.net/comp/squeak/rogue.htm , then copy the attached sound file in the root folder and do
>
> (SampledSound fromWaveFileNamed: 'sound.wav') edit
> (SampledSound fromWaveFileNamed: 'sound.wav') play
>
>
> Works fine here.
>
> Stef
> <sound.wav>
>

Stéphane Rollandin

Re: MixedSound (& consequently stereo) is badly broken in both 32-bit and 64-bit versions

> if you play the example you’ll see that the first version works fine. The second version, which adds MixedSound yo create stereo doesn’t. That proves that the issue is in MixedSound. Please just try the two examples.

Right. What I can see If I do

| samples sineTable sound |
"1 second of A below middle C (220Hz). 16000 / 220 is 72.72 recurring"
sineTable := SoundPlayer sineTable: 73.
sineTable doWithIndex:
[:sample :index| sineTable at: index put: sample // 4].
samples := SoundBuffer new: 16000.
1 to: samples size by: sineTable size do:
[:i| samples replaceFrom: i to: (i + sineTable size - 1 min: 16000)
with: sineTable startingAt: 1].
1 to: 146 do:
[:i|
samples at: i put: ((samples at: i) * i / 146) asInteger.
samples at: 16001 - i put: ((samples at: 16001 - i) * i / 146) asInteger].
sound := SampledSound samples: samples samplingRate: 16000.
sound := MixedSound new
add: sound pan: 0.25;
add: sound pan: 0.75;
yourself.
(SampledSound samples: sound monoSamples samplingRate: sound
originalSamplingRate) edit

in the muO image I pointed to in my previous message, is a phasing
problem leading to periodic glitches (the attached image is directly
taken from the muO editor)

I'm looking at this..

Stef

SoundElementEditor.png (15K) Download Attachment

Stéphane Rollandin

Re: MixedSound (& consequently stereo) is badly broken in both 32-bit and 64-bit versions

In reply to this post by Eliot Miranda-2

Ok, it looks like your choice of frequency let a bug surface.

In SampledSound>>#reset, the computation of scaledIncrement depends on
the sampling rate. But scaledIncrement is also supposed to be somewhat
commensurate with ScaledIndexOverflow which has a fixed value set in the
class-side initialize.

So I guess the whole LargePositiveIntegers avoidance scheme is bogus. If
you get rid of it altogether, which in
SampledSound>>#mixSampleCount:into:startingAt:leftVol:rightVol:
translates as follow (primitive uncommented):

-----

mixSampleCount: n into: aSoundBuffer startingAt: startIndex leftVol:
leftVol rightVol: rightVol
"Mix the given number of samples with the samples already in the given
buffer starting at the given index. Assume that the buffer size is at
least (index + count) - 1."

| lastIndex outIndex sample i s |
" <primitive:'primitiveMixSampledSound' module:'SoundGenerationPlugin'>
<var: #aSoundBuffer declareC: 'short int *aSoundBuffer'>
<var: #samples declareC: 'short int *samples'>
"
lastIndex := (startIndex + n) - 1.
outIndex := startIndex. "index of next stereo output sample pair"

[(outIndex <= samplesSize) and: [outIndex <= lastIndex]] whileTrue: [
sample := ((samples at: outIndex) * scaledVol) // ScaleFactor.
leftVol > 0 ifTrue: [
i := (2 * outIndex) - 1.
s := (aSoundBuffer at: i) + ((sample * leftVol) // ScaleFactor).
s > 32767 ifTrue: [s := 32767]. "clipping!"
s < -32767 ifTrue: [s := -32767]. "clipping!"
aSoundBuffer at: i put: s].
rightVol > 0 ifTrue: [
i := 2 * outIndex.
s := (aSoundBuffer at: i) + ((sample * rightVol) // ScaleFactor).
s > 32767 ifTrue: [s := 32767]. "clipping!"
s < -32767 ifTrue: [s := -32767]. "clipping!"
aSoundBuffer at: i put: s].

scaledVolIncr ~= 0 ifTrue: [
scaledVol := scaledVol + scaledVolIncr.
((scaledVolIncr > 0 and: [scaledVol >= scaledVolLimit]) or:
[scaledVolIncr < 0 and: [scaledVol <= scaledVolLimit]])
ifTrue: [ "reached the limit; stop incrementing"
scaledVol := scaledVolLimit.
scaledVolIncr := 0]].

outIndex := outIndex + 1].
count := count - n.

-----

then you get a nice sine wave (which still play badly in my Squeak image
but now only because it is too slow to generate without a primitive).

Stef

Stéphane Rollandin

Re: MixedSound (& consequently stereo) is badly broken in both 32-bit and 64-bit versions

> Ok, it looks like your choice of frequency let a bug surface.
I meant choice of *sampling rate*, sorry.

Stef

Stéphane Rollandin

Re: MixedSound (& consequently stereo) is badly broken in both 32-bit and 64-bit versions

In reply to this post by Stéphane Rollandin

> SampledSound>>#mixSampleCount:into:startingAt:leftVol:rightVol:
> translates as follow (primitive uncommented):

...and here I meant "primitive commented out", sorry again.

Stef

Levente Uzonyi

Re: MixedSound (& consequently stereo) is badly broken in both 32-bit and 64-bit versions

In reply to this post by Eliot Miranda-2

Hi Eliot,

On Fri, 18 Sep 2020, Eliot Miranda wrote:

> Hi All,
>
> the SampledSound>>#mixSampleCount:into:startingAt:leftVol:rightVol: method looks to be badly broken. I don't understand the index scaling (yet) so I thought I'd throw this out there and see if anyone can spot when and
> why this broke.
>
> Here's a simple example that generates a second long A below middle C (220Hz), smoothing start and stop, and then plays it, so far so good:
>
> | samples sineTable sound |
> "1 second of A below middle C (220Hz). 16000 / 220 is 72.72 recurring"
> sineTable := SoundPlayer sineTable: 73.
> sineTable doWithIndex: "And let's not deafen anyone..."
> [:sample :index| sineTable at: index put: sample // 4].
> samples := SoundBuffer new: 16000.
> 1 to: samples size by: sineTable size do:
> [:i| samples replaceFrom: i to: (i + sineTable size - 1 min: 16000) with: sineTable startingAt: 1].
> 1 to: 146 do: "smooth start and end of the sound"
> [:i|
> samples at: i put: ((samples at: i) * i / 146) asInteger.
> samples at: 16001 - i put: ((samples at: 16001 - i) * i / 146) asInteger].
> (SampledSound samples: samples samplingRate: 16000) play
>
> Now let's create a stereo sound image and try and play that:
>
> | samples sineTable sound |
> "1 second of A below middle C (220Hz). 16000 / 220 is 72.72 recurring"
> sineTable := SoundPlayer sineTable: 73.
> sineTable doWithIndex:
> [:sample :index| sineTable at: index put: sample // 4].
> samples := SoundBuffer new: 16000.
> 1 to: samples size by: sineTable size do:
> [:i| samples replaceFrom: i to: (i + sineTable size - 1 min: 16000) with: sineTable startingAt: 1].
> 1 to: 146 do:
> [:i|
> samples at: i put: ((samples at: i) * i / 146) asInteger.
> samples at: 16001 - i put: ((samples at: 16001 - i) * i / 146) asInteger].
> sound := SampledSound samples: samples samplingRate: 16000.
> sound := MixedSound new
> add: sound pan: 0.25;
> add: sound pan: 0.75;

If you replace the above line with

> add: sound copy pan: 0.75;

then the produced sound should be okay. Sound objects have state, and if
you pass the same object twice, the state will be messed up.

Levente

> yourself.
> sound play
>
> Eek!!
>
> You can examine the mixing without hurting your ears by replacing
> "sound play"
> with
> "sound computeSamplesForSeconds: sound duration"
> which invokes the mixing directly.
>
> Any help making sense of this gratefully received. Some comments around the
>
> scaledIndex := scaledIndex + scaledIncrement.
> scaledIndex >= ScaledIndexOverflow ifTrue: [
> overflow := scaledIndex >> IncrementFractionBits.
> indexHighBits := indexHighBits + overflow.
> scaledIndex := scaledIndex - (overflow << IncrementFractionBits)].
>
> in SampledSound>>#mixSampleCount:into:startingAt:leftVol:rightVol: would be most welcome.
>
> And if you simply comment out the primitive teh Smaklltalk code produces garbage also, so the problem does not seem to be Slang but the algorithm itself.
>
> _,,,^..^,,,_
> best, Eliot
>
>

Stéphane Rollandin

Re: MixedSound (& consequently stereo) is badly broken in both 32-bit and 64-bit versions

> If you replace the above line with
>
>> add: sound copy pan: 0.75;
>
> then the produced sound should be okay. Sound objects have state, and if
> you pass the same object twice, the state will be messed up.

Hmm... this is much simpler than my "fix" :)

Stef

Eliot Miranda-2

Re: MixedSound (& consequently stereo) is badly broken in both 32-bit and 64-bit versions

In reply to this post by Levente Uzonyi

Hi Levente, Hi Stef,

On Sat, Sep 19, 2020 at 4:47 AM Levente Uzonyi <[hidden email]> wrote:

Hi Eliot,

On Fri, 18 Sep 2020, Eliot Miranda wrote:

> Hi All,
>
> the SampledSound>>#mixSampleCount:into:startingAt:leftVol:rightVol: method looks to be badly broken. I don't understand the index scaling (yet) so I thought I'd throw this out there and see if anyone can spot when and
> why this broke.
>
> Here's a simple example that generates a second long A below middle C (220Hz), smoothing start and stop, and then plays it, so far so good:
>
> | samples sineTable sound |
> "1 second of A below middle C (220Hz). 16000 / 220 is 72.72 recurring"
> sineTable := SoundPlayer sineTable: 73.
> sineTable doWithIndex: "And let's not deafen anyone..."
> [:sample :index| sineTable at: index put: sample // 4].
> samples := SoundBuffer new: 16000.
> 1 to: samples size by: sineTable size do:
> [:i| samples replaceFrom: i to: (i + sineTable size - 1 min: 16000) with: sineTable startingAt: 1].
> 1 to: 146 do: "smooth start and end of the sound"
> [:i|
> samples at: i put: ((samples at: i) * i / 146) asInteger.
> samples at: 16001 - i put: ((samples at: 16001 - i) * i / 146) asInteger].
> (SampledSound samples: samples samplingRate: 16000) play
>
> Now let's create a stereo sound image and try and play that:
>
> | samples sineTable sound |
> "1 second of A below middle C (220Hz). 16000 / 220 is 72.72 recurring"
> sineTable := SoundPlayer sineTable: 73.
> sineTable doWithIndex:
> [:sample :index| sineTable at: index put: sample // 4].
> samples := SoundBuffer new: 16000.
> 1 to: samples size by: sineTable size do:
> [:i| samples replaceFrom: i to: (i + sineTable size - 1 min: 16000) with: sineTable startingAt: 1].
> 1 to: 146 do:
> [:i|
> samples at: i put: ((samples at: i) * i / 146) asInteger.
> samples at: 16001 - i put: ((samples at: 16001 - i) * i / 146) asInteger].
> sound := SampledSound samples: samples samplingRate: 16000.
> sound := MixedSound new
> add: sound pan: 0.25;
> add: sound pan: 0.75;

If you replace the above line with

> add: sound copy pan: 0.75;

then the produced sound should be okay. Sound objects have state, and if
you pass the same object twice, the state will be messed up.

Thank you! This has cost me a day. I'd say that it fails the principle of least astonishment. I'm going to see if I can add a check to add:pan: or the mixdown step that would warn of this requirement.

Levente

> yourself.
> sound play
>
> Eek!!
>
> You can examine the mixing without hurting your ears by replacing
> "sound play"
> with
> "sound computeSamplesForSeconds: sound duration"
> which invokes the mixing directly.
>
> Any help making sense of this gratefully received. Some comments around the
>
> scaledIndex := scaledIndex + scaledIncrement.
> scaledIndex >= ScaledIndexOverflow ifTrue: [
> overflow := scaledIndex >> IncrementFractionBits.
> indexHighBits := indexHighBits + overflow.
> scaledIndex := scaledIndex - (overflow << IncrementFractionBits)].
>
> in SampledSound>>#mixSampleCount:into:startingAt:leftVol:rightVol: would be most welcome.
>
> And if you simply comment out the primitive teh Smaklltalk code produces garbage also, so the problem does not seem to be Slang but the algorithm itself.
>
> _,,,^..^,,,_
> best, Eliot
>
>

_,,,^..^,,,_

best, Eliot

Eliot Miranda-2

Re: MixedSound (& consequently stereo) is badly broken in both 32-bit and 64-bit versions

In reply to this post by Stéphane Rollandin

Hi Stef, Hi Levente,

On Sat, Sep 19, 2020 at 2:27 AM Stéphane Rollandin <[hidden email]> wrote:

Ok, it looks like your choice of frequency let a bug surface.

In SampledSound>>#reset, the computation of scaledIncrement depends on
the sampling rate. But scaledIncrement is also supposed to be somewhat
commensurate with ScaledIndexOverflow which has a fixed value set in the
class-side initialize.

So I guess the whole LargePositiveIntegers avoidance scheme is bogus. If
you get rid of it altogether, which in
SampledSound>>#mixSampleCount:into:startingAt:leftVol:rightVol:
translates as follow (primitive uncommented):

Stef, thanks for looking at the overflow arithmetic. In 64-bits we may not need it at all, and so I could add an ifTrue:ifFalse: based on the value of SmallInteger maxVal that would avoid the complex arithmetic on 64-bits. What do you think?

-----

mixSampleCount: n into: aSoundBuffer startingAt: startIndex leftVol:
leftVol rightVol: rightVol
"Mix the given number of samples with the samples already in the given
buffer starting at the given index. Assume that the buffer size is at
least (index + count) - 1."

| lastIndex outIndex sample i s |
" <primitive:'primitiveMixSampledSound' module:'SoundGenerationPlugin'>
<var: #aSoundBuffer declareC: 'short int *aSoundBuffer'>
<var: #samples declareC: 'short int *samples'>
"
lastIndex := (startIndex + n) - 1.
outIndex := startIndex. "index of next stereo output sample pair"

[(outIndex <= samplesSize) and: [outIndex <= lastIndex]] whileTrue: [
sample := ((samples at: outIndex) * scaledVol) // ScaleFactor.
leftVol > 0 ifTrue: [
i := (2 * outIndex) - 1.
s := (aSoundBuffer at: i) + ((sample * leftVol) // ScaleFactor).
s > 32767 ifTrue: [s := 32767]. "clipping!"
s < -32767 ifTrue: [s := -32767]. "clipping!"
aSoundBuffer at: i put: s].
rightVol > 0 ifTrue: [
i := 2 * outIndex.
s := (aSoundBuffer at: i) + ((sample * rightVol) // ScaleFactor).
s > 32767 ifTrue: [s := 32767]. "clipping!"
s < -32767 ifTrue: [s := -32767]. "clipping!"
aSoundBuffer at: i put: s].

scaledVolIncr ~= 0 ifTrue: [
scaledVol := scaledVol + scaledVolIncr.
((scaledVolIncr > 0 and: [scaledVol >= scaledVolLimit]) or:
[scaledVolIncr < 0 and: [scaledVol <= scaledVolLimit]])
ifTrue: [ "reached the limit; stop incrementing"
scaledVol := scaledVolLimit.
scaledVolIncr := 0]].

outIndex := outIndex + 1].
count := count - n.

-----

then you get a nice sine wave (which still play badly in my Squeak image
but now only because it is too slow to generate without a primitive).

Stef

_,,,^..^,,,_

best, Eliot

Stéphane Rollandin

Re: MixedSound (& consequently stereo) is badly broken in both 32-bit and 64-bit versions

> Stef, thanks for looking at the overflow arithmetic. In 64-bits we may
> not need it at all, and so I could add an ifTrue:ifFalse: based on the
> value of SmallInteger maxVal that would avoid the complex arithmetic on
> 64-bits. What do you think?

Well my analysis of the problem was way off, as Levente showed, so I
will pass on this one:)

Stef

Eliot Miranda-2

Re: MixedSound (& consequently stereo) is badly broken in both 32-bit and 64-bit versions

In reply to this post by Stéphane Rollandin

Hi Stef,

On Sat, Sep 19, 2020 at 1:17 AM Stéphane Rollandin <[hidden email]> wrote:

> if you play the example you’ll see that the first version works fine. The second version, which adds MixedSound yo create stereo doesn’t. That proves that the issue is in MixedSound. Please just try the two examples.

Right. What I can see If I do

| samples sineTable sound |
"1 second of A below middle C (220Hz). 16000 / 220 is 72.72 recurring"
sineTable := SoundPlayer sineTable: 73.
sineTable doWithIndex:
[:sample :index| sineTable at: index put: sample // 4].
samples := SoundBuffer new: 16000.
1 to: samples size by: sineTable size do:
[:i| samples replaceFrom: i to: (i + sineTable size - 1 min: 16000)
with: sineTable startingAt: 1].
1 to: 146 do:
[:i|
samples at: i put: ((samples at: i) * i / 146) asInteger.
samples at: 16001 - i put: ((samples at: 16001 - i) * i / 146) asInteger].
sound := SampledSound samples: samples samplingRate: 16000.
sound := MixedSound new
add: sound pan: 0.25;
add: sound pan: 0.75;
yourself.
(SampledSound samples: sound monoSamples samplingRate: sound
originalSamplingRate) edit

in the muO image I pointed to in my previous message, is a phasing
problem leading to periodic glitches (the attached image is directly
taken from the muO editor)

Interesting!! I don't see this in my data, but I do see evidence of an off-by-one error in misdown. Here's the start, first zero crossing, and end of the mono sound (the original samples sound buffer)

{ samples first: 8. samples copyFrom: 70 to: 77. samples last: 8 }

{a SoundBuffer(4 19 42 75 117 166 222 285) .

a SoundBuffer(-1003 -682 -347 0 356 720 1088 1457) .

a SoundBuffer(221 222 213 196 170 136 96 50)}

So these look fine and smooth.

But here's the corresponding data from the computeSamplesForSeconds: result of the MixedSound:

{ self first: 16. self copyFrom: 70 * 2 to: 77 * 2 + 1. self last: 16 }

{a SoundBuffer(4 4 4 4 18 18 41 41 41 41 74 74 116 116 165 165) .

a SoundBuffer(-2712 -2712 -2712 -2836 -2836 -2940 -2940 -2940 -2940 -3022 -3022 -3084 -3084 -3124 -3124 -3214) .

a SoundBuffer(212 212 212 212 196 196 169 169 169 169 136 136 96 96 49 49)}

The data *should* start with a SoundBuffer(4 4 18 18...

The data ends well. But the first crossing is completely wrong.

Strange, because it sounds ok...

Let me count the crossings using:

| sign crossings |

crossings := 0.

sign := self first sign.

self do:

[:each|

each sign = sign negated ifTrue:

[crossings := crossings + 1.

sign := each sign]].

crossings

Both give 438 as expected. Strange. Why is there no crossing between 140 and 154 in the mixed down sound?

Ah, ok, the sample rate has changed:

| sign crossings ncrossings |

ncrossings := 0.

crossings := OrderedCollection new.

sign := self first sign.

self doWithIndex:

[:each :idx|

each sign = sign negated ifTrue:

[crossings add: idx - 1 -> idx.

ncrossings := ncrossings + 1.

sign := each sign]].

{ crossings first: 4. ncrossings } {an OrderedCollection(100->101 202->203 302->303 404->405) . 438}

So everything is fine *except for* the extra sample at the beginning. That's clearly an error, right?

I'm looking at this..

Stef

_,,,^..^,,,_

best, Eliot

Eliot Miranda-2

Re: MixedSound (& consequently stereo) is badly broken in both 32-bit and 64-bit versions

Hi Dan, Hi Stef, Hi Levente, Hi Christoph,

this is not a bug. I'm just noting something for our collective understanding and pointing to a potential improvement in the sound system. This is concerning the code in SampledSound>>#mixSampleCount:into:startingAt:leftVol:rightVol:. If you look at the results of mixing down a 16KHz sound into e.g. a 22.1KHz sound (the default sampling rate for a mix down) then if the input is a 220Hz sine wave at 16KHz (A below middle C, computed in the example below) then the first few samples look like this

a SoundBuffer(4 19 42 75 117 166 222 285 353 425 ...

When we mix this down the corresponding range of samples in the stereo 22.1Khz mixed-down sound is

a SoundBuffer(4 4 4 4 18 18 41 41 41 41 74 74 116 116 165 165 165 165 221 221 284 284 284 284 352 352 424 424 500 500 ...

Now this comes because of the scaled indexing in SampledSound>>#mixSampleCount:into:startingAt:leftVol:rightVol: and is to be expected. The input sample rate is less than the output sample rate so the algorithm smears mixed output samples until the next matching input sample is reached.

The improvement would be not to smear, but to interpolate. If we added (say, simple linear) interpolation then the algorithm would instead produce

a SoundBuffer(4 4 11 11 18 18 41 41 57 57 74 74 116 116 165 165 193 193 221 221 284 284 318 318 352 352 424 424 500 500 ...

Now I (we?) just have to find time to work on this ;-)

On Sat, Sep 19, 2020 at 10:56 AM Eliot Miranda <[hidden email]> wrote:

Hi Stef,

On Sat, Sep 19, 2020 at 1:17 AM Stéphane Rollandin <[hidden email]> wrote:
> if you play the example you’ll see that the first version works fine. The second version, which adds MixedSound yo create stereo doesn’t. That proves that the issue is in MixedSound. Please just try the two examples.

Right. What I can see If I do

| samples sineTable sound |
"1 second of A below middle C (220Hz). 16000 / 220 is 72.72 recurring"
sineTable := SoundPlayer sineTable: 73.
sineTable doWithIndex:
[:sample :index| sineTable at: index put: sample // 4].
samples := SoundBuffer new: 16000.
1 to: samples size by: sineTable size do:
[:i| samples replaceFrom: i to: (i + sineTable size - 1 min: 16000)
with: sineTable startingAt: 1].
1 to: 146 do:
[:i|
samples at: i put: ((samples at: i) * i / 146) asInteger.
samples at: 16001 - i put: ((samples at: 16001 - i) * i / 146) asInteger].
sound := SampledSound samples: samples samplingRate: 16000.
sound := MixedSound new
add: sound pan: 0.25;
add: sound pan: 0.75;
yourself.
(SampledSound samples: sound monoSamples samplingRate: sound
originalSamplingRate) edit

in the muO image I pointed to in my previous message, is a phasing
problem leading to periodic glitches (the attached image is directly
taken from the muO editor)

Interesting!! I don't see this in my data, but I do see evidence of an off-by-one error in misdown. Here's the start, first zero crossing, and end of the mono sound (the original samples sound buffer)

{ samples first: 8. samples copyFrom: 70 to: 77. samples last: 8 }
{a SoundBuffer(4 19 42 75 117 166 222 285) .
a SoundBuffer(-1003 -682 -347 0 356 720 1088 1457) .
a SoundBuffer(221 222 213 196 170 136 96 50)}

So these look fine and smooth.

But here's the corresponding data from the computeSamplesForSeconds: result of the MixedSound:
{ self first: 16. self copyFrom: 70 * 2 to: 77 * 2 + 1. self last: 16 }
{a SoundBuffer(4 4 4 4 18 18 41 41 41 41 74 74 116 116 165 165) .
a SoundBuffer(-2712 -2712 -2712 -2836 -2836 -2940 -2940 -2940 -2940 -3022 -3022 -3084 -3084 -3124 -3124 -3214) .
a SoundBuffer(212 212 212 212 196 196 169 169 169 169 136 136 96 96 49 49)}

The data *should* start with a SoundBuffer(4 4 18 18...
The data ends well. But the first crossing is completely wrong.

Strange, because it sounds ok...

Let me count the crossings using:
| sign crossings |
crossings := 0.
sign := self first sign.
self do:
[:each|
each sign = sign negated ifTrue:
[crossings := crossings + 1.
sign := each sign]].
crossings

Both give 438 as expected. Strange. Why is there no crossing between 140 and 154 in the mixed down sound?

Ah, ok, the sample rate has changed:

| sign crossings ncrossings |
ncrossings := 0.
crossings := OrderedCollection new.
sign := self first sign.
self doWithIndex:
[:each :idx|
each sign = sign negated ifTrue:
[crossings add: idx - 1 -> idx.
ncrossings := ncrossings + 1.
sign := each sign]].
{ crossings first: 4. ncrossings } {an OrderedCollection(100->101 202->203 302->303 404->405) . 438}

So everything is fine *except for* the extra sample at the beginning. That's clearly an error, right?

I'm looking at this..

Stef

_,,,^..^,,,_
best, Eliot

_,,,^..^,,,_

best, Eliot

Eliot Miranda-2

Interpolation on mix down in MixedSound

In reply to this post by Eliot Miranda-2

Hi Dan, Hi Stef, Hi Levente, Hi Christoph,

(duplicating to change the subject line)

a SoundBuffer(4 19 42 75 117 166 222 285 353 425 ...

When we mix this down the corresponding range of samples in the stereo 22.1Khz mixed-down sound is

a SoundBuffer(4 4 4 4 18 18 41 41 41 41 74 74 116 116 165 165 165 165 221 221 284 284 284 284 352 352 424 424 500 500 ...

The improvement would be not to smear, but to interpolate. If we added (say, simple linear) interpolation then the algorithm would instead produce

a SoundBuffer(4 4 11 11 18 18 41 41 57 57 74 74 116 116 165 165 193 193 221 221 284 284 318 318 352 352 424 424 500 500 ...

Now I (we?) just have to find time to work on this ;-)

On Sat, Sep 19, 2020 at 10:56 AM Eliot Miranda <[hidden email]> wrote:

Hi Stef,

On Sat, Sep 19, 2020 at 1:17 AM Stéphane Rollandin <[hidden email]> wrote:
> if you play the example you’ll see that the first version works fine. The second version, which adds MixedSound yo create stereo doesn’t. That proves that the issue is in MixedSound. Please just try the two examples.

Right. What I can see If I do

| samples sineTable sound |
"1 second of A below middle C (220Hz). 16000 / 220 is 72.72 recurring"
sineTable := SoundPlayer sineTable: 73.
sineTable doWithIndex:
[:sample :index| sineTable at: index put: sample // 4].
samples := SoundBuffer new: 16000.
1 to: samples size by: sineTable size do:
[:i| samples replaceFrom: i to: (i + sineTable size - 1 min: 16000)
with: sineTable startingAt: 1].
1 to: 146 do:
[:i|
samples at: i put: ((samples at: i) * i / 146) asInteger.
samples at: 16001 - i put: ((samples at: 16001 - i) * i / 146) asInteger].
sound := SampledSound samples: samples samplingRate: 16000.
sound := MixedSound new
add: sound pan: 0.25;
add: sound pan: 0.75;
yourself.
(SampledSound samples: sound monoSamples samplingRate: sound
originalSamplingRate) edit

in the muO image I pointed to in my previous message, is a phasing
problem leading to periodic glitches (the attached image is directly
taken from the muO editor)

Interesting!! I don't see this in my data, but I do see evidence of an off-by-one error in misdown. Here's the start, first zero crossing, and end of the mono sound (the original samples sound buffer)

{ samples first: 8. samples copyFrom: 70 to: 77. samples last: 8 }
{a SoundBuffer(4 19 42 75 117 166 222 285) .
a SoundBuffer(-1003 -682 -347 0 356 720 1088 1457) .
a SoundBuffer(221 222 213 196 170 136 96 50)}

So these look fine and smooth.

But here's the corresponding data from the computeSamplesForSeconds: result of the MixedSound:
{ self first: 16. self copyFrom: 70 * 2 to: 77 * 2 + 1. self last: 16 }
{a SoundBuffer(4 4 4 4 18 18 41 41 41 41 74 74 116 116 165 165) .
a SoundBuffer(-2712 -2712 -2712 -2836 -2836 -2940 -2940 -2940 -2940 -3022 -3022 -3084 -3084 -3124 -3124 -3214) .
a SoundBuffer(212 212 212 212 196 196 169 169 169 169 136 136 96 96 49 49)}

The data *should* start with a SoundBuffer(4 4 18 18...
The data ends well. But the first crossing is completely wrong.

Strange, because it sounds ok...

Let me count the crossings using:
| sign crossings |
crossings := 0.
sign := self first sign.
self do:
[:each|
each sign = sign negated ifTrue:
[crossings := crossings + 1.
sign := each sign]].
crossings

Both give 438 as expected. Strange. Why is there no crossing between 140 and 154 in the mixed down sound?

Ah, ok, the sample rate has changed:

| sign crossings ncrossings |
ncrossings := 0.
crossings := OrderedCollection new.
sign := self first sign.
self doWithIndex:
[:each :idx|
each sign = sign negated ifTrue:
[crossings add: idx - 1 -> idx.
ncrossings := ncrossings + 1.
sign := each sign]].
{ crossings first: 4. ncrossings } {an OrderedCollection(100->101 202->203 302->303 404->405) . 438}

So everything is fine *except for* the extra sample at the beginning. That's clearly an error, right?

I'm looking at this..

Stef

_,,,^..^,,,_
best, Eliot

_,,,^..^,,,_

best, Eliot