Sound mixing makes nasty noises
Locked
123
123
 
|
I've discovered that playing two or more sounds simultaneously can make truly appalling noises. I'm hoping somebody has a good solution.
The problem started with playing a sound and using the pianokeyboardmorph at the same time. Ouch! Also, swizzling the mouse across the keyboard causes occasional quite audible 'scratchy-clicks' somehow. There can also be a quite noticeable sound between notes, not easily describable but sort of a 'dull thump'. I thought it was some issue with how the sound was played, maybe the note-end process as each piano key was released and the next pressed, timing issues.. but none of that had very much effect. The AbstractSound>>#stopGracefully method does seem to have a small bug, in that the decayInMs value surely ought not be the sum of the attack & decay times from the envelope? But carefully fudging the note shutdown to be faster didn't help a lot. After too much messing around I discovered that at least some of the problem is the mixing of sounds. Now, I know we did a small change to MixedSound>>#add:pan:volume: since 5.3, and that we made a small change to the SampledSound>>#mixSampleCount:into:startingAt:leftVol:rightVol: primitive. However, I'm running on a VM that has the prim change, and I've ported across the add:pan... change, with no real difference. This is, of course, on a Pi, but the same issue is audible on my iMac as well. Eventually I cottoned on to the fact that I was mixing two sounds with volumes set to 100%. Changing to 50% for both actually results in a perfectly fine end result - Yay! This doesn't seem like a robust solution though. What if I have four sounds playing each set to 40%? Why does it seem to work perfectly well for AbstractSound>>#stereoBachFugue play, which uses four? To make life even more fun, it doesn't seem like there is any volume control once a sound is playing. The AbstractSound>>#adjustVolumeTo:overMSecs: method appears to be the one to use but it has no audible effect. There is also #loudness: which does work, but produces pretty much exactly the glitch I hear from glissandoing(?) the keyboard. So - some workspace code to illustrate "awful noise when mixing" |snd| snd := FMSound organ1. snd setPitch: 440 dur: 10 loudness: 0.9; play. 1 second wait. FMSound brass1 setPitch: 470 dur:2 loudness: 0.9; play. 1 second wait. snd stopGracefully "OK, because both sound defualt to quieter" |snd | snd := FMSound organ1. snd duration: 5; play. 1 second wait. FMSound brass1 play. 1 second wait. snd stopGracefully. "nasty glitch using #loudness: plus mixing graunch" |snd | snd := FMSound organ1. snd duration: 5; play. 1 second wait. snd loudness: 1.0 . 1 second wait. FMSound brass1 play. 1 second wait. snd stopGracefully. "no effect from adjustVol..." |snd | snd := FMSound organ1. snd duration: 5; play. 1 second wait. snd adjustVolumeTo: 1.0 overMSecs: 500. 1 second wait. FMSound brass1 play. 1 second wait. snd stopGracefully. Aargh! tim -- tim Rowledge; [hidden email]; http://www.rowledge.org/tim A computer program does what you tell it to do, not what you want it to do. |
Re: Sound mixing makes nasty noises
|
|
Hi Tim,
tl,dr Let's build a limiter. I stopped using Squeak for Audio with 3.8 :-) With that in mind: I believe that all sound goes out through SoundPlayer playing SoundBuffers. The samples get created via mixSsampleCountInto.... which seems to be implemented by many classes and there they all seem to do clipping. After running some of your examples I found several SoundBuffers containing samples 32767 and -32768 which indicates clipping. I didn't find enough consecutive clipped samples (you shouldn't hear a single one) but that's because I only found a few SoundBuffers. BTW running your example labelled "OK because.." also clipped when run several times. I didn't find the place where float values are changed to Integers for output. I did not find any limiter. --> baaad! So if the above is true and the problem is really the clipping (I had the impression I also heard some wrapping where 32768 uses the top bit as a sign) I suggest: -No clipping during the mixing -Find the place where the SoundBuffers go to the hardware -Implement a limiter there Basically you need a controlled gain. You look ahead in your SoundBuffer for min and max. In case of overflow you calculate the necessary gain and gradually reduce your actual gain in the time you have until that sample really gets output. (attack phase). If no over/underflow is found you (more) gradually raise your gain back to 1 (release phase). Maybe introduce some delay before raising your gain again (hold phase). For live real time audio you may not have the luxury to look ahead. You still reduce your gain gradually (let's say in 10 ms) and clip until you reached your final gain. Human's ears tend to tolerate this. There's tons of compromises to evaluate with a limiter we should go with a bit over TSTTCPW to save CPU. Maybe have another preference to choose the system wide limiter. SCNR :-))) I can help with rough implementations and audiowise discussions but have no idea where to hook it in and cannot convert it to a primitive or optimize it. Let's wait until Stef chimes in, I'm sure he has solved that for muO already. Best regards, Herbert Am 16.12.2020 um 23:52 schrieb tim Rowledge: > I've discovered that playing two or more sounds simultaneously can make truly appalling noises. I'm hoping somebody has a good solution. > > The problem started with playing a sound and using the pianokeyboardmorph at the same time. Ouch! Also, swizzling the mouse across the keyboard causes occasional quite audible 'scratchy-clicks' somehow. There can also be a quite noticeable sound between notes, not easily describable but sort of a 'dull thump'. > > I thought it was some issue with how the sound was played, maybe the note-end process as each piano key was released and the next pressed, timing issues.. but none of that had very much effect. > > The AbstractSound>>#stopGracefully method does seem to have a small bug, in that the decayInMs value surely ought not be the sum of the attack & decay times from the envelope? But carefully fudging the note shutdown to be faster didn't help a lot. > > After too much messing around I discovered that at least some of the problem is the mixing of sounds. > > Now, I know we did a small change to MixedSound>>#add:pan:volume: since 5.3, and that we made a small change to the SampledSound>>#mixSampleCount:into:startingAt:leftVol:rightVol: primitive. However, I'm running on a VM that has the prim change, and I've ported across the add:pan... change, with no real difference. This is, of course, on a Pi, but the same issue is audible on my iMac as well. > > Eventually I cottoned on to the fact that I was mixing two sounds with volumes set to 100%. Changing to 50% for both actually results in a perfectly fine end result - Yay! This doesn't seem like a robust solution though. What if I have four sounds playing each set to 40%? Why does it seem to work perfectly well for AbstractSound>>#stereoBachFugue play, which uses four? > > To make life even more fun, it doesn't seem like there is any volume control once a sound is playing. The AbstractSound>>#adjustVolumeTo:overMSecs: method appears to be the one to use but it has no audible effect. There is also #loudness: which does work, but produces pretty much exactly the glitch I hear from glissandoing(?) the keyboard. > > So - some workspace code to illustrate > > "awful noise when mixing" > |snd| > snd := FMSound organ1. > snd setPitch: 440 dur: 10 loudness: 0.9; > play. > 1 second wait. > > FMSound brass1 setPitch: 470 dur:2 loudness: 0.9; > play. > 1 second wait. > snd stopGracefully > > "OK, because both sound defualt to quieter" > |snd | > snd := FMSound organ1. > snd > duration: 5; > play. > 1 second wait. > > FMSound brass1 > play. > > 1 second wait. > snd stopGracefully. > > "nasty glitch using #loudness: plus mixing graunch" > |snd | > snd := FMSound organ1. > snd duration: 5; > play. > 1 second wait. > snd loudness: 1.0 . > > 1 second wait. > FMSound brass1 > play. > > 1 second wait. > snd stopGracefully. > > > "no effect from adjustVol..." > |snd | > snd := FMSound organ1. > snd > duration: 5; > play. > 1 second wait. > snd adjustVolumeTo: 1.0 overMSecs: 500. > 1 second wait. > FMSound brass1 > play. > > 1 second wait. > snd stopGracefully. > > Aargh! > > tim > -- > tim Rowledge; [hidden email]; http://www.rowledge.org/tim > A computer program does what you tell it to do, not what you want it to do. > > > |
Re: Sound mixing makes nasty noises
|
|
>
> Let's wait until Stef chimes in, I'm sure he has solved that for muO > already. :) Well, indeed, the sound for the first example from Tim is fine in a muO image - I could not explain why off the top of my head, actually, I kind of forgot a lot of what I did. Stef |
Re: Sound mixing makes nasty noises
|
|
In reply to this post by timrowledge
> "no effect from adjustVol..."
> |snd | > snd := FMSound organ1. > snd > duration: 5; > play. > 1 second wait. > snd adjustVolumeTo: 1.0 overMSecs: 500. > 1 second wait. > FMSound brass1 > play. > > 1 second wait. > snd stopGracefully. There is no effect because amplitude change is supposed to be driven by an AmplitudeEnvelope (these are the ones that send #adjustVolumeTo:overMSecs:). Unfortunately, there is already one such envelope active in FMSound organ1, which uses amplitude modulation, so a second one would be ignored anyway. What we need is a way to combine several layers of amplitude control; we could start by supporting several AmplitudeEnvelopes in a single AbstractSound. Stef |
Re: Sound mixing makes nasty noises
|
|
I'd think if nobody clips early we can just add all simultaneous sounds
(each one after using their own envelope), do that all in Float32 and then add a limiter like I described. It just must be put at the end of the audio chain. Either a clever one with lookahed and no clipping at all or (for live recording and playback with little latency) a conventional one which changes the gain with a certain attack rate and has some clipping until the attack is over. Music industry lived with the latter for decades. No idea if Squeak's sound architecture is suitable for that, it seems the conversion to Integer happens too early. But it was done with 16 Bit DSPs so some search or thinking should uncover that. So first question: Where is the place, where the audio samples are handed to the OS? Second question: Are there more places than the diverse mixSampleCount:into:startingAt:leftVol:rightVol: which are responsible for clipping. Cheers, Herbert Am 17.12.2020 um 20:00 schrieb Stéphane Rollandin: >> "no effect from adjustVol..." >> |snd | >> snd := FMSound organ1. >> snd >> duration: 5; >> play. >> 1 second wait. >> snd adjustVolumeTo: 1.0 overMSecs: 500. >> 1 second wait. >> FMSound brass1 >> play. >> >> 1 second wait. >> snd stopGracefully. > > There is no effect because amplitude change is supposed to be driven > by an AmplitudeEnvelope (these are the ones that send > #adjustVolumeTo:overMSecs:). > > Unfortunately, there is already one such envelope active in FMSound > organ1, which uses amplitude modulation, so a second one would be > ignored anyway. > > What we need is a way to combine several layers of amplitude control; > we could start by supporting several AmplitudeEnvelopes in a single > AbstractSound. > > Stef > |
Re: Sound mixing makes nasty noises
|
|
In reply to this post by Stéphane Rollandin
Thanks for chiming in (see what I did there?).
> On 2020-12-17, at 11:00 AM, Stéphane Rollandin <[hidden email]> wrote: > >> "no effect from adjustVol..." >> |snd | >> snd := FMSound organ1. >> snd >> duration: 5; >> play. >> 1 second wait. >> snd adjustVolumeTo: 1.0 overMSecs: 500. >> 1 second wait. >> FMSound brass1 >> play. >> >> 1 second wait. >> snd stopGracefully. > > There is no effect because amplitude change is supposed to be driven by an AmplitudeEnvelope (these are the ones that send #adjustVolumeTo:overMSecs:). That's extra-puzzling because it is the mechanism used in the original Scratch code to control volume and I'd swear it used to work. The other mechanism #loudness: is rather more brutal and makes glitchy noises if used whilst playing. > Unfortunately, there is already one such envelope active in FMSound organ1, which uses amplitude modulation, so a second one would be ignored anyway. > > What we need is a way to combine several layers of amplitude control; we could start by supporting several AmplitudeEnvelopes in a single AbstractSound. I'm not very familiar with the workings of the sound system and happy to keep it that way if possible. There is the futzing with the 'scaledVol' value built in to the assorted mix prims & methods, so surely that ought to work? What would be the 'proper' way to merge multiple sounds with varying volumes? Do you scale all of them? Wouldn't that mean that starting a new sound would make the others go quieter in an odd sounding manner? Do you just clip the final total for every sample? tim -- tim Rowledge; [hidden email]; http://www.rowledge.org/tim "How many Slavers does it take to change a lightbulb?” "Dunno. How susceptible are lightbulbs to telepathy?" |
Re: Sound mixing makes nasty noises
|
|
> There is the futzing with the 'scaledVol' value built in to the assorted mix prims & methods, so surely that ought to work? Yes, but that scaledVol value is already set to change cyclicly (is that a word?) in FMSound organ1 in order to modulate the base sine wave and give it some timbre. If we attempt to use that same value to control the overall loudness, we will lose the timbre. In fact we would need to multiply the timbral volume envelope with the loudness one, but this is not trivial because the first one uses a loop. > What would be the 'proper' way to merge multiple sounds with varying volumes? Do you scale all of them? Yes, and whether or not their volumes are varying, the sounds must have been scaled beforehand to ensure there is enough headroom for their superposition. This again is not trivial, because of the way interference works: adding two sounds can potentially double the height of their amplitude peaks, where they interfere constructively, but can also, where they interfere destructively, lead to a lower volume (for an extreme example, if you mix a signal with its exact mirror you will get a plain silence). Stef |
Re: Sound mixing makes nasty noises
|
|
Well...
> On 2020-12-17, at 5:12 PM, Stéphane Rollandin <[hidden email]> wrote: > > >> There is the futzing with the 'scaledVol' value built in to the assorted mix prims & methods, so surely that ought to work? > > Yes, but that scaledVol value is already set to change cyclicly (is that a word?) in FMSound organ1 in order to modulate the base sine wave and give it some timbre. If we attempt to use that same value to control the overall loudness, we will lose the timbre. In fact we would need to multiply the timbral volume envelope with the loudness one, but this is not trivial because the first one uses a loop. I *think* that the FMSound>>#primitiveMixFMSound code handles this; at least it looks like it pays attention to the waveTable etc. I've just taken a quick look and the code that changes the scaleVol if the scaledVolIncr value is non-0 is there in the prim too, so there hasn't been a major regression in the code. Probably. > > >> What would be the 'proper' way to merge multiple sounds with varying volumes? Do you scale all of them? > > Yes, and whether or not their volumes are varying, the sounds must have been scaled beforehand to ensure there is enough headroom for their superposition. Yikes. That smacks of needing a lot of advance information. How on earth do we handle sometihng like a true ten-finger keyboard? Make each sound only allow volume 0.1 ? > This again is not trivial, because of the way interference works: adding two sounds can potentially double the height of their amplitude peaks, where they interfere constructively, but can also, where they interfere destructively, lead to a lower volume (for an extreme example, if you mix a signal with its exact mirror you will get a plain silence). Oh yes. Wave interference maths is great fun. I was just reading a couple of nights ago about using the positive Grassmanian and Kodama-Williams equations to model soliton waves. I hope we don't need to worry about that to solve this tiny problem! tim -- tim Rowledge; [hidden email]; http://www.rowledge.org/tim If at first you don't succeed, destroy all evidence that you tried. |
Re: Sound mixing makes nasty noises
|
|
Am 18.12.2020 um 02:52 schrieb tim
Rowledge:
That is just very unlikely, all tones you press on your keyboard must start with that exact delay that they reach their maxima at the same time. So the higher tone must start later than the lower one because it reaches its max quicker when both start from zero. But yes, if you look at the waveform of some classical music you see a lot of headroom for the few peaks. Just not a factor 10. Different story for current popular music. Or music to be listened to while driving.Yikes. That smacks of needing a lot of advance information. How on earth do we handle sometihng like a true ten-finger keyboard? Make each sound only allow volume 0.1 ?This again is not trivial, because of the way interference works: adding two sounds can potentially double the height of their amplitude peaks, where they interfere constructively, but can also, where they interfere destructively, lead to a lower volume (for an extreme example, if you mix a signal with its exact mirror you will get a plain silence). Cheers, Herbert 
|
Re: Sound mixing makes nasty noises
|
|
In reply to this post by timrowledge
>> Yes, but that scaledVol value is already set to change cyclicly (is that a word?) in FMSound organ1 in order to modulate the base sine wave and give it some timbre. If we attempt to use that same value to control the overall loudness, we will lose the timbre. In fact we would need to multiply the timbral volume envelope with the loudness one, but this is not trivial because the first one uses a loop.
> > I *think* that the FMSound>>#primitiveMixFMSound code handles this; at least it looks like it pays attention to the waveTable etc. I've just taken a quick look and the code that changes the scaleVol if the scaledVolIncr value is non-0 is there in the prim too, so there hasn't been a major regression in the code. Probably. That's what I am talking about. Because there is only one scaledVol instance, its evolution in time can be (and actually *is*) described by a *single* envelope. Now see the code in FMSound>>#organ1 There, that envelope is defined in snd addEnvelope: (VolumeEnvelope points: p loopStart: 2 loopEnd: 4). So that is what drives the time evolution of scaledVol. It is a short envelope constantly looping over break points 2 to 4. If you now want to control the overall loudness of the sound, let's say for a crescendo, you have to surimpose to this scaledVol evolution a gradual increase of its max value - this cannot anymore be defined by a looping envelope. Stef |
Re: Sound mixing makes nasty noises
|
|
In reply to this post by Herbert König
By the way, it is possible in muO to have that kind of waveform display,
along with the spectrogram. Try for example (AbstractSound bachFugueVoice1On: PluckedSound default) viewSamples This will yield a zoommable-field morph representing the waveform, where click-dragging smoothly scrolls the display, a shift+click-drag smoothly scales it (images attached). The right-click menu gives access to the FFT settings. Best, Stef  waveform1.png (12K) Download Attachment waveform2.png (33K) Download Attachment waveform3.png (24K) Download Attachment waveform4.png (246K) Download Attachment |
Re: Sound mixing makes nasty noises
|
|
In reply to this post by Herbert König
Hi all,
just noticed SoundBuffer is 16 bits so my suggestions are not practicable. Cheers, Herbert Am 17.12.2020 um 14:14 schrieb Herbert König: > Hi Tim, > > tl,dr Let's build a limiter. > > I stopped using Squeak for Audio with 3.8 :-) With that in mind: > > I believe that all sound goes out through SoundPlayer playing > SoundBuffers. The samples get created via mixSsampleCountInto.... which > seems to be implemented by many classes and there they all seem to do > clipping. > > After running some of your examples I found several SoundBuffers > containing samples 32767 and -32768 which indicates clipping. I didn't > find enough consecutive clipped samples (you shouldn't hear a single > one) but that's because I only found a few SoundBuffers. > BTW running your example labelled "OK because.." also clipped when run > several times. > > I didn't find the place where float values are changed to Integers for > output. > > I did not find any limiter. --> baaad! > > So if the above is true and the problem is really the clipping (I had > the impression I also heard some wrapping where 32768 uses the top bit > as a sign) I suggest: > -No clipping during the mixing > -Find the place where the SoundBuffers go to the hardware > -Implement a limiter there > > Basically you need a controlled gain. You look ahead in your SoundBuffer > for min and max. In case of overflow you calculate the necessary gain > and gradually reduce your actual gain in the time you have until that > sample really gets output. (attack phase). If no over/underflow is found > you (more) gradually raise your gain back to 1 (release phase). Maybe > introduce some delay before raising your gain again (hold phase). > > For live real time audio you may not have the luxury to look ahead. You > still reduce your gain gradually (let's say in 10 ms) and clip until you > reached your final gain. Human's ears tend to tolerate this. > > There's tons of compromises to evaluate with a limiter we should go with > a bit over TSTTCPW to save CPU. Maybe have another preference to choose > the system wide limiter. SCNR :-))) > > I can help with rough implementations and audiowise discussions but have > no idea where to hook it in and cannot convert it to a primitive or > optimize it. > > Let's wait until Stef chimes in, I'm sure he has solved that for muO > already. > > Best regards, > > Herbert > > > Am 16.12.2020 um 23:52 schrieb tim Rowledge: >> I've discovered that playing two or more sounds simultaneously can >> make truly appalling noises. I'm hoping somebody has a good solution. >> >> The problem started with playing a sound and using the >> pianokeyboardmorph at the same time. Ouch! Also, swizzling the mouse >> across the keyboard causes occasional quite audible 'scratchy-clicks' >> somehow. There can also be a quite noticeable sound between notes, >> not easily describable but sort of a 'dull thump'. >> >> I thought it was some issue with how the sound was played, maybe the >> note-end process as each piano key was released and the next pressed, >> timing issues.. but none of that had very much effect. >> >> The AbstractSound>>#stopGracefully method does seem to have a small >> bug, in that the decayInMs value surely ought not be the sum of the >> attack & decay times from the envelope? But carefully fudging the >> note shutdown to be faster didn't help a lot. >> >> After too much messing around I discovered that at least some of the >> problem is the mixing of sounds. >> >> Now, I know we did a small change to MixedSound>>#add:pan:volume: >> since 5.3, and that we made a small change to the >> SampledSound>>#mixSampleCount:into:startingAt:leftVol:rightVol: >> primitive. However, I'm running on a VM that has the prim change, and >> I've ported across the add:pan... change, with no real difference. >> This is, of course, on a Pi, but the same issue is audible on my iMac >> as well. >> >> Eventually I cottoned on to the fact that I was mixing two sounds >> with volumes set to 100%. Changing to 50% for both actually results >> in a perfectly fine end result - Yay! This doesn't seem like a robust >> solution though. What if I have four sounds playing each set to 40%? >> Why does it seem to work perfectly well for >> AbstractSound>>#stereoBachFugue play, which uses four? >> >> To make life even more fun, it doesn't seem like there is any volume >> control once a sound is playing. The >> AbstractSound>>#adjustVolumeTo:overMSecs: method appears to be the >> one to use but it has no audible effect. There is also #loudness: >> which does work, but produces pretty much exactly the glitch I hear >> from glissandoing(?) the keyboard. >> >> So - some workspace code to illustrate >> >> "awful noise when mixing" >> |snd| >> snd := FMSound organ1. >> snd setPitch: 440 dur: 10 loudness: 0.9; >> play. >> 1 second wait. >> >> FMSound brass1 setPitch: 470 dur:2 loudness: 0.9; >> play. >> 1 second wait. >> snd stopGracefully >> >> "OK, because both sound defualt to quieter" >> |snd | >> snd := FMSound organ1. >> snd >> duration: 5; >> play. >> 1 second wait. >> >> FMSound brass1 >> play. >> >> 1 second wait. >> snd stopGracefully. >> >> "nasty glitch using #loudness: plus mixing graunch" >> |snd | >> snd := FMSound organ1. >> snd duration: 5; >> play. >> 1 second wait. >> snd loudness: 1.0 . >> >> 1 second wait. >> FMSound brass1 >> play. >> >> 1 second wait. >> snd stopGracefully. >> >> >> "no effect from adjustVol..." >> |snd | >> snd := FMSound organ1. >> snd >> duration: 5; >> play. >> 1 second wait. >> snd adjustVolumeTo: 1.0 overMSecs: 500. >> 1 second wait. >> FMSound brass1 >> play. >> >> 1 second wait. >> snd stopGracefully. >> >> Aargh! >> >> tim >> -- >> tim Rowledge; [hidden email]; http://www.rowledge.org/tim >> A computer program does what you tell it to do, not what you want it >> to do. >> >> >> > > |
Re: Sound mixing makes nasty noises
|
|
Hi Herbert,
_,,,^..^,,,_ (phone) > On Dec 18, 2020, at 4:46 AM, Herbert König <[hidden email]> wrote: > > Hi all, > > just noticed SoundBuffer is 16 bits so my suggestions are not practicable. But adding a signed 32-bit, or even a signed 64-bit collection for mixing would be straight forward, except for ensuring it mixes down to 16 bits. A signed 32-bit collection would allow 32k 16-bit sources to be additively mixed without clipping, right? So an underlying 64-bit collection is unnecessary. So a MixedSound that adds into an underlying 32-bit signed sequence, and updates a count of added sounds, and does interpolation when adding sounds of different sample rates, and outputs 16-bit samples derived by dividing the 32-bit sums by the count of added sounds would be a solution right? > > Cheers, > > Herbert > >> Am 17.12.2020 um 14:14 schrieb Herbert König: >> Hi Tim, >> >> tl,dr Let's build a limiter. >> >> I stopped using Squeak for Audio with 3.8 :-) With that in mind: >> >> I believe that all sound goes out through SoundPlayer playing >> SoundBuffers. The samples get created via mixSsampleCountInto.... which >> seems to be implemented by many classes and there they all seem to do >> clipping. >> >> After running some of your examples I found several SoundBuffers >> containing samples 32767 and -32768 which indicates clipping. I didn't >> find enough consecutive clipped samples (you shouldn't hear a single >> one) but that's because I only found a few SoundBuffers. >> BTW running your example labelled "OK because.." also clipped when run >> several times. >> >> I didn't find the place where float values are changed to Integers for >> output. >> >> I did not find any limiter. --> baaad! >> >> So if the above is true and the problem is really the clipping (I had >> the impression I also heard some wrapping where 32768 uses the top bit >> as a sign) I suggest: >> -No clipping during the mixing >> -Find the place where the SoundBuffers go to the hardware >> -Implement a limiter there >> >> Basically you need a controlled gain. You look ahead in your SoundBuffer >> for min and max. In case of overflow you calculate the necessary gain >> and gradually reduce your actual gain in the time you have until that >> sample really gets output. (attack phase). If no over/underflow is found >> you (more) gradually raise your gain back to 1 (release phase). Maybe >> introduce some delay before raising your gain again (hold phase). >> >> For live real time audio you may not have the luxury to look ahead. You >> still reduce your gain gradually (let's say in 10 ms) and clip until you >> reached your final gain. Human's ears tend to tolerate this. >> >> There's tons of compromises to evaluate with a limiter we should go with >> a bit over TSTTCPW to save CPU. Maybe have another preference to choose >> the system wide limiter. SCNR :-))) >> >> I can help with rough implementations and audiowise discussions but have >> no idea where to hook it in and cannot convert it to a primitive or >> optimize it. >> >> Let's wait until Stef chimes in, I'm sure he has solved that for muO >> already. >> >> Best regards, >> >> Herbert >> >> >>> Am 16.12.2020 um 23:52 schrieb tim Rowledge: >>> I've discovered that playing two or more sounds simultaneously can >>> make truly appalling noises. I'm hoping somebody has a good solution. >>> >>> The problem started with playing a sound and using the >>> pianokeyboardmorph at the same time. Ouch! Also, swizzling the mouse >>> across the keyboard causes occasional quite audible 'scratchy-clicks' >>> somehow. There can also be a quite noticeable sound between notes, >>> not easily describable but sort of a 'dull thump'. >>> >>> I thought it was some issue with how the sound was played, maybe the >>> note-end process as each piano key was released and the next pressed, >>> timing issues.. but none of that had very much effect. >>> >>> The AbstractSound>>#stopGracefully method does seem to have a small >>> bug, in that the decayInMs value surely ought not be the sum of the >>> attack & decay times from the envelope? But carefully fudging the >>> note shutdown to be faster didn't help a lot. >>> >>> After too much messing around I discovered that at least some of the >>> problem is the mixing of sounds. >>> >>> Now, I know we did a small change to MixedSound>>#add:pan:volume: >>> since 5.3, and that we made a small change to the >>> SampledSound>>#mixSampleCount:into:startingAt:leftVol:rightVol: >>> primitive. However, I'm running on a VM that has the prim change, and >>> I've ported across the add:pan... change, with no real difference. >>> This is, of course, on a Pi, but the same issue is audible on my iMac >>> as well. >>> >>> Eventually I cottoned on to the fact that I was mixing two sounds >>> with volumes set to 100%. Changing to 50% for both actually results >>> in a perfectly fine end result - Yay! This doesn't seem like a robust >>> solution though. What if I have four sounds playing each set to 40%? >>> Why does it seem to work perfectly well for >>> AbstractSound>>#stereoBachFugue play, which uses four? >>> >>> To make life even more fun, it doesn't seem like there is any volume >>> control once a sound is playing. The >>> AbstractSound>>#adjustVolumeTo:overMSecs: method appears to be the >>> one to use but it has no audible effect. There is also #loudness: >>> which does work, but produces pretty much exactly the glitch I hear >>> from glissandoing(?) the keyboard. >>> >>> So - some workspace code to illustrate >>> >>> "awful noise when mixing" >>> |snd| >>> snd := FMSound organ1. >>> snd setPitch: 440 dur: 10 loudness: 0.9; >>> play. >>> 1 second wait. >>> >>> FMSound brass1 setPitch: 470 dur:2 loudness: 0.9; >>> play. >>> 1 second wait. >>> snd stopGracefully >>> >>> "OK, because both sound defualt to quieter" >>> |snd | >>> snd := FMSound organ1. >>> snd >>> duration: 5; >>> play. >>> 1 second wait. >>> >>> FMSound brass1 >>> play. >>> >>> 1 second wait. >>> snd stopGracefully. >>> >>> "nasty glitch using #loudness: plus mixing graunch" >>> |snd | >>> snd := FMSound organ1. >>> snd duration: 5; >>> play. >>> 1 second wait. >>> snd loudness: 1.0 . >>> >>> 1 second wait. >>> FMSound brass1 >>> play. >>> >>> 1 second wait. >>> snd stopGracefully. >>> >>> >>> "no effect from adjustVol..." >>> |snd | >>> snd := FMSound organ1. >>> snd >>> duration: 5; >>> play. >>> 1 second wait. >>> snd adjustVolumeTo: 1.0 overMSecs: 500. >>> 1 second wait. >>> FMSound brass1 >>> play. >>> >>> 1 second wait. >>> snd stopGracefully. >>> >>> Aargh! >>> >>> tim >>> -- >>> tim Rowledge; [hidden email]; http://www.rowledge.org/tim >>> A computer program does what you tell it to do, not what you want it >>> to do. >>> >>> >>> >> >> > > |
Re: Sound mixing makes nasty noises
|
|
I've made a tiny bit of progress on fighting this and I've discovered some points that might help with understanding why a VolumeEnvelope stops the volume adjustment working.
So first, using a simple FMSound with the base volume set to 40% solves a lot of problems in practice when it comes to the note-chooser in Scratch - simply keeping the volume lower avoids any clamping. The simple FMSound has not volume envelope and so the volume control actually works. Looking at how the volume envelope works it seems like it shouldn't cause any issues; after all the operation of the envelope uses the #adjustVolume... method. I think that we're looking at the attempt to set the volume with it is clashing with the VolumeEnvelope>>#updateTargetAt: stomping on the related variables. That happens every time the #doControl method runs, which is nominally 100Hz. Somewhere there is a way to composite the two and get the desired result. I just don't see it right now because I'm trying too many things at once. tim -- tim Rowledge; [hidden email]; http://www.rowledge.org/tim "How many Motie Warriors does it take to change a lightbulb?” "None. One of the dead ones will do it." |
Re: Sound mixing makes nasty noises
|
|
Well nothing to do with sound ever seems as simple as one would like.
Remember, the waggling keyboard thing makes horrific scratchy noises BUT it doesn't do that on a Mac running exactly the same Smalltalk code, so I got suspicious. I now have three Pi setups that make completely different sounds. 1) an old Pi-top CEED with a Pi 3 and a May 2017 OS, a)with the as-provided NuScratch. This makes perfectly ok sounds under the piano-waggle regime, event though the volume is 100%. b) same Pi-top but using a newer (but still 2017 vintage) Squeak system, where the sound seems fine. Both cases are using the old release of alsa. 2) my main daily use Pi4 with the last non-pulse version of OS. a) using the supplied NuScratch it makes barely audible noises that are terrible when actually audible. Even the cat meow is terrible! b) using an early-this-year Squeak VM there is just no sound at all 3) a fresh Pi4 with the release candidate OS Simon pointed me to. a) in as-supplied NuScratch with a pulse audio sound lib added the sound is bad unless the internal volume is turned way down, when there are still nasty scratchy-glitches b) in a new system the sound is still bad but much nearer to ok with the internal volume reduced. As mentioned, the Mac runs the exact same Smalltalk image just fine. I don't have Windows so I can't opine on that option. Other than "Eww, Windows?" The obvious thing to dig into now is what changes have been made to the lowest-level unix sound stuff since 2016. Anybody got ideas? Could it even be changes to the unix libraries in the pulse/alsa world? tim -- tim Rowledge; [hidden email]; http://www.rowledge.org/tim Strange OpCodes: MET: Misread and Eat Tape |
Re: Sound mixing makes nasty noises
|
|
There's always a chance that the OS meddles with it too. Notably on
Windows if you simultaneously play and record it looses buffers sometimes. And often it is the combination of application and OS. I didn't find the place where Squeak audio goes out to the OS but writing this to a (preferably .wav) file would give a chance to analyse what happens inside Squeak. If anybody can post the necessary changes I can do that and check. I will set up a Pi and record the audio (hardware) output with another computer to see the sum of the effects. Are the snippets in your original post those which produce the problem on a Pi? Would be more comparable then waggling the keyboard. Cheers, Herbert Am 24.12.2020 um 04:32 schrieb tim Rowledge: > Well nothing to do with sound ever seems as simple as one would like. > > Remember, the waggling keyboard thing makes horrific scratchy noises BUT it doesn't do that on a Mac running exactly the same Smalltalk code, so I got suspicious. > > I now have three Pi setups that make completely different sounds. > > 1) an old Pi-top CEED with a Pi 3 and a May 2017 OS, > a)with the as-provided NuScratch. This makes perfectly ok sounds under the piano-waggle regime, event though the volume is 100%. > b) same Pi-top but using a newer (but still 2017 vintage) Squeak system, where the sound seems fine. > Both cases are using the old release of alsa. > > 2) my main daily use Pi4 with the last non-pulse version of OS. > a) using the supplied NuScratch it makes barely audible noises that are terrible when actually audible. Even the cat meow is terrible! > b) using an early-this-year Squeak VM there is just no sound at all > > 3) a fresh Pi4 with the release candidate OS Simon pointed me to. > a) in as-supplied NuScratch with a pulse audio sound lib added the sound is bad unless the internal volume is turned way down, when there are still nasty scratchy-glitches > b) in a new system the sound is still bad but much nearer to ok with the internal volume reduced. > > As mentioned, the Mac runs the exact same Smalltalk image just fine. I don't have Windows so I can't opine on that option. Other than "Eww, Windows?" > > The obvious thing to dig into now is what changes have been made to the lowest-level unix sound stuff since 2016. Anybody got ideas? Could it even be changes to the unix libraries in the pulse/alsa world? > > tim > -- > tim Rowledge; [hidden email]; http://www.rowledge.org/tim > Strange OpCodes: MET: Misread and Eat Tape > > > |
Re: Sound mixing makes nasty noises
|
|
> On 2020-12-24, at 2:35 AM, Herbert König <[hidden email]> wrote: > > There's always a chance that the OS meddles with it too. Notably on > Windows if you simultaneously play and record it looses buffers > sometimes. And often it is the combination of application and OS. Yah. It certainly isn't impossible that changes to the underlying libraries are causing issues; after, my original work on this was done six years ago and the basic sound related stuff may be 20 years old. > > I didn't find the place where Squeak audio goes out to the OS but > writing this to a (preferably .wav) file would give a chance to analyse > what happens inside Squeak. If anybody can post the necessary changes I > can do that and check. I can see three plausible places where it might be smart to record things a) copy the sound buffer samples as sent to the prim in the #playLoop method b) copy the samples actually sent to the OS by the VM c) copy the sound output by the OS I suspect that somebody can point to an application to record the final output cleanly. > > I will set up a Pi and record the audio (hardware) output with another > computer to see the sum of the effects. I was almost at the point of just aiming my iPhone at the screen. > > Are the snippets in your original post those which produce the problem > on a Pi? Would be more comparable then waggling the keyboard. Yes and no; that code shows off how the volume level can drive things into a screeching mess. Here is another snippet that seems to illustrate things a bit better - |snd volume | volume:= 40. 10 timesRepeat: [ snd := FMSound new. "250 milliSeconds wait." snd setPitch: (AbstractSound pitchForMIDIKey: 60) dur: 1000 loudness: ((volume / 100.0) min: 1.0 max: 0.0). snd play. 50 milliSeconds wait. snd stopGracefully. "250 milliSeconds wait." snd := FMSound new. snd setPitch: (AbstractSound pitchForMIDIKey: 58) dur: 1000 loudness: ((volume / 100.0) min: 1.0 max: 0.0). snd play. 50 milliSeconds wait. snd stopGracefully] You might experiment with - the volume setting - the variety of FMSound (organ1 includes an envelope or two to mix things up a bit) - the commented out between-sound delay, which I used to try to avoid two sounds overlapping in order to test the hypothesis that mixing was messing up And depending on the update level of your image, the "X milliSeconds wait" might need to be changed back to "(Delay forMIlliSeconds: X) wait" etc. I get a strangely interesting effect that is probably suggesting something important (and that I'm not yet getting) with the between-sound delay used and set to 150 mS. It's as if there is some mangling of buffers going on. At 100mS it goes berserk; at 200 it sounds perfectly regular. I'm going to indulge in some code scanning to try to spot changes made that might affect this. I'll be interested how you see, or rather hear, things. tim -- tim Rowledge; [hidden email]; http://www.rowledge.org/tim "Bother!" said Pooh, as Piglet pressed <START> on the Microwave... |
Re: Sound mixing makes nasty noises
|
|
Hi Tim,
ATM I get no sound from Squeak or Scratch on my Raspi3. I updated the OS but it still doesn't say "Raspberry Pi OS" With some googling I got Audio working from Audacity (simple Audio Editor I use on Raspi, Linux and Windows). I use Squeak5.3-19439. Recording with a phone will show some glitches but the bandwidth of phone (and monitor speakers) mellows down (err filters) the waveform. Surely only second choice. I would not mind downloading and using the proper Squeak to get sound going. Any quick help? Cheers, Herbert Am 24.12.2020 um 20:58 schrieb tim Rowledge: > >> On 2020-12-24, at 2:35 AM, Herbert König <[hidden email]> wrote: >> >> There's always a chance that the OS meddles with it too. Notably on >> Windows if you simultaneously play and record it looses buffers >> sometimes. And often it is the combination of application and OS. > Yah. It certainly isn't impossible that changes to the underlying libraries are causing issues; after, my original work on this was done six years ago and the basic sound related stuff may be 20 years old. > >> I didn't find the place where Squeak audio goes out to the OS but >> writing this to a (preferably .wav) file would give a chance to analyse >> what happens inside Squeak. If anybody can post the necessary changes I >> can do that and check. > I can see three plausible places where it might be smart to record things > a) copy the sound buffer samples as sent to the prim in the #playLoop method > b) copy the samples actually sent to the OS by the VM > c) copy the sound output by the OS > > I suspect that somebody can point to an application to record the final output cleanly. > >> I will set up a Pi and record the audio (hardware) output with another >> computer to see the sum of the effects. > I was almost at the point of just aiming my iPhone at the screen. > >> Are the snippets in your original post those which produce the problem >> on a Pi? Would be more comparable then waggling the keyboard. > Yes and no; that code shows off how the volume level can drive things into a screeching mess. > > Here is another snippet that seems to illustrate things a bit better - > > |snd volume | > volume:= 40. > 10 timesRepeat: [ > snd := FMSound new. > "250 milliSeconds wait." > snd > setPitch: (AbstractSound pitchForMIDIKey: 60) > dur: 1000 > loudness: ((volume / 100.0) min: 1.0 max: 0.0). > snd play. > 50 milliSeconds wait. > snd stopGracefully. > "250 milliSeconds wait." > snd := FMSound new. > snd > setPitch: (AbstractSound pitchForMIDIKey: 58) > dur: 1000 > loudness: ((volume / 100.0) min: 1.0 max: 0.0). > snd play. > 50 milliSeconds wait. > snd stopGracefully] > > You might experiment with > - the volume setting > - the variety of FMSound (organ1 includes an envelope or two to mix things up a bit) > - the commented out between-sound delay, which I used to try to avoid two sounds overlapping in order to test the hypothesis that mixing was messing up > > And depending on the update level of your image, the "X milliSeconds wait" might need to be changed back to "(Delay forMIlliSeconds: X) wait" etc. > > I get a strangely interesting effect that is probably suggesting something important (and that I'm not yet getting) with the between-sound delay used and set to 150 mS. It's as if there is some mangling of buffers going on. At 100mS it goes berserk; at 200 it sounds perfectly regular. > > I'm going to indulge in some code scanning to try to spot changes made that might affect this. I'll be interested how you see, or rather hear, things. > > > tim > -- > tim Rowledge; [hidden email]; http://www.rowledge.org/tim > "Bother!" said Pooh, as Piglet pressed <START> on the Microwave... > > > |
Re: Sound mixing makes nasty noises
|
|
> On 2020-12-24, at 12:18 PM, Herbert König <[hidden email]> wrote: > > Hi Tim, > > ATM I get no sound from Squeak or Scratch on my Raspi3. I updated the OS > but it still doesn't say "Raspberry Pi OS" Hmm, updating a Pi can be confusing at times. If you do `uname -a` in a terminal, what do you see? On my still-alsa machine I get Linux Goldskin 5.4.51-v7l+ #1333 SMP Mon Aug 10 16:51:40 BST 2020 armv7l GNU/Linux On my test-pulse I get Linux Pi-4-1-pulse 5.4.79-v7l+ #1373 SMP Mon Nov 23 13:27:40 GMT 2020 armv7l GNU/Linux Remember that you need to include the vm sound flag when you start up squeak! I keep forgetting that. Though.. if you're start scratch with the standard menu entry, that should do it. Let's see - there is also the possibility you are not directing sound to the right output. The Pi sound-volume control widget on the menubar can set where the sound goes - right-click on it and you should see a menu with at least Analogue HDMI entries. I use the analogue and headphones since none of my Pi have a direct HDMI display attached but go via VNC. If you have an HDMI monitor attached it may not have speakers or you might not have them turned on! DAMHIKT. > > With some googling I got Audio working from Audacity (simple Audio > Editor I use on Raspi, Linux and Windows). I use Squeak5.3-19439. > > Recording with a phone will show some glitches but the bandwidth of > phone (and monitor speakers) mellows down (err filters) the waveform. > Surely only second choice. > > I would not mind downloading and using the proper Squeak to get sound going. The 5.3 armlinux package from squeak.org ought to do ok for this. Remembering of course to include that '-vm-sound-alsa' (or pulse) in your commandline tim -- tim Rowledge; [hidden email]; http://www.rowledge.org/tim Strange OpCodes: PRM: PRint Money |
Re: Sound mixing makes nasty noises
|
|
Am 24.12.2020 um 21:36 schrieb tim Rowledge: > >> On 2020-12-24, at 12:18 PM, Herbert König <[hidden email]> wrote: >> >> Hi Tim, >> >> ATM I get no sound from Squeak or Scratch on my Raspi3. I updated the OS >> but it still doesn't say "Raspberry Pi OS" > Hmm, updating a Pi can be confusing at times. > > If you do > `uname -a` in a terminal, what do you see? > > On my still-alsa machine I get > Linux Goldskin 5.4.51-v7l+ #1333 SMP Mon Aug 10 16:51:40 BST 2020 armv7l GNU/Linux > > On my test-pulse I get > Linux Pi-4-1-pulse 5.4.79-v7l+ #1373 SMP Mon Nov 23 13:27:40 GMT 2020 armv7l GNU/Linux > > Remember that you need to include the vm sound flag when you start up squeak! I keep forgetting that. Though.. if you're start scratch with the standard menu entry, that should do it. Let's see - there is also the possibility you are not directing sound to the right output. The Pi sound-volume control widget on the menubar can set where the sound goes - right-click on it and you should see a menu with at least > Analogue > HDMI > entries. I use the analogue and headphones since none of my Pi have a direct HDMI display attached but go via VNC. If you have an HDMI monitor attached it may not have speakers or you might not have them turned on! DAMHIKT. I use the 3.5mm headphone output and verified that via Audacity and VLC. I have a cheap active speaker connected. Just checked: Scratch 2 plays sound Scratch1 started via the menu doesn't. Now downloading a fresh Squeak. I tried the -vmsound alsa and pulse but I'm not sure If I used the right place in the script. I tried editing the last line to: exec "${VM}" "-vm-sound-alsa" "${IMAGE}" (and pulse and w/o the "" but my Linux fu is not enough to know if that's ok. Cheers, Herbert > > > >> With some googling I got Audio working from Audacity (simple Audio >> Editor I use on Raspi, Linux and Windows). I use Squeak5.3-19439. >> >> Recording with a phone will show some glitches but the bandwidth of >> phone (and monitor speakers) mellows down (err filters) the waveform. >> Surely only second choice. >> >> I would not mind downloading and using the proper Squeak to get sound going. > The 5.3 armlinux package from squeak.org ought to do ok for this. Remembering of course to include that '-vm-sound-alsa' (or pulse) in your commandline > > > tim > -- > tim Rowledge; [hidden email]; http://www.rowledge.org/tim > Strange OpCodes: PRM: PRint Money > > > |
«
Return to Squeak - Dev
|
1 view|%1 views
| Free forum by Nabble | Edit this page |