Weird Pi Squeak slowdown alert

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Weird Pi Squeak slowdown alert

timrowledge
This is weird, annoying and driving me nuts. I'm trying everything I can think of to work out what is happening but just in case anybody else here is using Squeak on a Pi4 ...

Since January my Pi 4 has been returning 810 million bc/sc and 60 million-ish sends/sec for the tinyBenchmarks. Recently it has seemed to slow down oddly , back down to Pi 3+ values of 330/29-ish. I haven't found any rhyme or reason for it so far. I don't think it is simply some artefact messing the benchmark either; it definitely feels slower.

A while back I noticed this, did some fiddling and set up FSTRIM on the SSD I use; that seemed to solve the problem at the time but clearly didn't really solve it.

I've done the trivial - rebooting, different vintage of VM, different images, different screen resolution. I've done more involved - trying assorted FSTRIM stuff for the SSD, running from uSD card again, cooling harder. crossing my legs differently.

Nothing seems to make sense. Is anybody else running a Pi 4 with plain old Raspberry Pi OS 32 bit and Squeak?

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
When all else fails, let a = 7.  If that doesn't help, then read the manual.



Reply | Threaded
Open this post in threaded view
|

Re: Weird Pi Squeak slowdown alert

timrowledge


> On 2020-10-01, at 3:19 PM, tim Rowledge <[hidden email]> wrote:
>
> This is weird, annoying and driving me nuts. I'm trying everything I can think of to work out what is happening but just in case anybody else here is using Squeak on a Pi4 ...

Power supply. Some slow fading of the output voltage lead to the Pi throttling , swap PSU and all seems well. Fetching 27,000 records / sec from a postgreSQL DB on a different machine :-)


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Useful random insult:- His head whistles in a cross wind.



Reply | Threaded
Open this post in threaded view
|

Re: Weird Pi Squeak slowdown alert

Herbert König
Am 02.10.2020 um 22:52 schrieb tim Rowledge:
> Power supply. Some slow fading of the output voltage lead to the Pi throttling , swap PSU and all seems well. Fetching 27,000 records / sec from a postgreSQL DB on a different machine :-)
>
>
That's HARDWARE :-))

Herbert

Reply | Threaded
Open this post in threaded view
|

Re: Weird Pi Squeak slowdown alert

Eliot Miranda-2
In reply to this post by timrowledge
Hi Tim,


> On Oct 2, 2020, at 1:52 PM, tim Rowledge <[hidden email]> wrote:
>
> 
>
>> On 2020-10-01, at 3:19 PM, tim Rowledge <[hidden email]> wrote:
>>
>> This is weird, annoying and driving me nuts. I'm trying everything I can think of to work out what is happening but just in case anybody else here is using Squeak on a Pi4 ...
>
> Power supply. Some slow fading of the output voltage lead to the Pi throttling , swap PSU and all seems well. Fetching 27,000 records / sec from a postgreSQL DB on a different machine :-)

So two questions, a) is there a way from software or GUI to find out if the pi is throttling? b) is there a way from software or GUI to find out if the input voltage is dripping

oh, and c) how did you debug this?



_,,,^..^,,,_ (phone)

Reply | Threaded
Open this post in threaded view
|

Re: Weird Pi Squeak slowdown alert

Mariano Martinez Peck
Hi Tim,

Last week Instantiations presented at RIoT Developer month "Faster & More Efficient: JITs for IoT", an experiment with VA Smalltalk and other languages like Python, Java, etc... the context was how a JIT compiler can also help you reduce energy consumption...which could be very important for IoT. We did experiments on a Raspberry Pi, run some benchmarks and with some hardware tool we were measuring the wattage. 

You can see the whole presentation here: https://youtu.be/2xO0ohUNnug
And here the exact moment that shows the tool used: https://youtu.be/2xO0ohUNnug?t=1818


I guess this is not exactly the same as detecting if it is throttling, but maybe it helps.

Cheers,

On Fri, Oct 2, 2020 at 7:14 PM Eliot Miranda <[hidden email]> wrote:
Hi Tim,


> On Oct 2, 2020, at 1:52 PM, tim Rowledge <[hidden email]> wrote:
>
> 
>
>> On 2020-10-01, at 3:19 PM, tim Rowledge <[hidden email]> wrote:
>>
>> This is weird, annoying and driving me nuts. I'm trying everything I can think of to work out what is happening but just in case anybody else here is using Squeak on a Pi4 ...
>
> Power supply. Some slow fading of the output voltage lead to the Pi throttling , swap PSU and all seems well. Fetching 27,000 records / sec from a postgreSQL DB on a different machine :-)

So two questions, a) is there a way from software or GUI to find out if the pi is throttling? b) is there a way from software or GUI to find out if the input voltage is dripping

oh, and c) how did you debug this?



_,,,^..^,,,_ (phone)



--
Mariano Martinez Peck


Reply | Threaded
Open this post in threaded view
|

Re: Weird Pi Squeak slowdown alert

timrowledge
In reply to this post by Eliot Miranda-2


> On 2020-10-02, at 3:14 PM, Eliot Miranda <[hidden email]> wrote:
>
> So two questions, a) is there a way from software or GUI to find out if the pi is throttling? b) is there a way from software or GUI to find out if the input voltage is dripping

There is existence proof of this in that the menubar has a widget (in Raspbian, at least) that shows the cpu temperature and changes color when it is throttling. I'm making a possibly naive assumption that it actually tells the truth.

>
> oh, and c) how did you debug this?

Oh, the usual way; curse, swear, try ludicrous ideas that make no sense just moments later. Since I have two Pi4, one with an SSD and one without I was able to swap the SSD out and try the uSD to see if it was anything to do with that. I ran an eeprom state updater/checker (remember, unlike the earlier models the Pi 4 has a chunk of eeprom for the hardware boot code) and showed that they were claimed identical. I made sure to run the exact same vm/image on both.

Eventually the Pi told me what the problem was itself - a widget I'd never seen before appeared saying 'voltage low. check your power supply'. Well, yes, there you go. I'd not previously considered that throttling was caused by anything other than temperature and was in the middle of writing a post to the Pi forums to ask why the problem Pi was indicating throttled when it was only at 30C, whereas the other was happy up to 50C with no throttling. Lesson learned.
Luckily I had another suitable PSU on my desk awaiting a case build for the other Pi 4 so I had one to swap in.

All in all, just like debugging an errant Cog. But swearier.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Java:  the best argument for Smalltalk since C++



Reply | Threaded
Open this post in threaded view
|

Re: Weird Pi Squeak slowdown alert

timrowledge
In reply to this post by Herbert König


> On 2020-10-02, at 2:42 PM, Herbert König <[hidden email]> wrote:
>
> Am 02.10.2020 um 22:52 schrieb tim Rowledge:
>> Power supply. Some slow fading of the output voltage lead to the Pi throttling , swap PSU and all seems well. Fetching 27,000 records / sec from a postgreSQL DB on a different machine :-)
>>
>>
> That's HARDWARE :-))

That's cool hardware with cool software running a very cool postgreSQL package from levente & co. Now all I have to do is finish porting a multithousand class VW package... with some very 'interesting' design decisions to unwrinkle.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Klingon Code Warrior:- 6) "Our competitors are without honor!"



Reply | Threaded
Open this post in threaded view
|

Re: Weird Pi Squeak slowdown alert

timrowledge
In reply to this post by Mariano Martinez Peck


> On 2020-10-02, at 3:52 PM, Mariano Martinez Peck <[hidden email]> wrote:
>
> Hi Tim,
>
> Last week Instantiations presented at RIoT Developer month "Faster & More Efficient: JITs for IoT",

You guys are having a lot of fun. Nice.

The real trick for IoT stuff would be to make a system that can run on an ESP32 class chip. I'm pretty sure it's doable, but not easy. The good news is that with built-in WiFi one could do a sort of OOZE/LOOM/network virtual memory thing. And really, just how much development software does a light switch need to load by default?


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: DTF: Dump Tape to Floor



Reply | Threaded
Open this post in threaded view
|

Re: Weird Pi Squeak slowdown alert

Herbert König
In reply to this post by timrowledge


Am 03.10.2020 um 03:06 schrieb tim Rowledge:

>
>> On 2020-10-02, at 2:42 PM, Herbert König <[hidden email]> wrote:
>>
>> Am 02.10.2020 um 22:52 schrieb tim Rowledge:
>>> Power supply. Some slow fading of the output voltage lead to the Pi throttling , swap PSU and all seems well. Fetching 27,000 records / sec from a postgreSQL DB on a different machine :-)
>>>
>>>
>> That's HARDWARE :-))
> That's cool hardware with cool software running a very cool postgreSQL package from levente & co. Now all I have to do is finish porting a multithousand class VW package... with some very 'interesting' design decisions to unwrinkle.
>
>
> tim
> --
> tim Rowledge; [hidden email]; http://www.rowledge.org/tim
> Klingon Code Warrior:- 6) "Our competitors are without honor!"
>
>
>
Was a joke, me owning 7 up to now. One A+ running Squeak 24/7 with 60%
load. Living outside since 2014. Sometimes cool sometimes hot. When done
in some years it will send a Bobby Car to the pooing cat.

Cheers,

Herbert



2020-10-02T01 21 30.80743702 00Max 11.4.jpg (38K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Weird Pi Squeak slowdown alert

Bruce O'Neel-2
In reply to this post by timrowledge

From the command line...  Do note that I log in over X so nothing gets plugged in or removed from the USB ports.  

1.  dmesg should be pretty clean.  Those times on the left are in seconds since boot.

[   23.088460] Bluetooth: BNEP (Ethernet Emulation) ver 1.3

[   23.088467] Bluetooth: BNEP filters: protocol multicast

[   23.088479] Bluetooth: BNEP socket layer initialized

[   23.124540] Bluetooth: RFCOMM TTY layer initialized

[   23.124559] Bluetooth: RFCOMM socket layer initialized

[   23.124577] Bluetooth: RFCOMM ver 1.11

 10:54:29 up 18 days, 15:37,  1 user,  load average: 0.00, 0.00, 0.00


2. Mine is not throttling now but my memory is that one of the following commands will give you some lines

sudo zgrep -i throt  /var/log/messages*.gz

sudo zgrep -i throt  /var/log/syslog*.gz


3.  There is a vcgencmd get_throttled


vcgencmd is an interesting command for many of these sorts of things.  You can measure the CPU and GPU temp.

cheers

bruce

03 October 2020 03:04 tim Rowledge <[hidden email]> wrote:


> On 2020-10-02, at 3:14 PM, Eliot Miranda wrote:
>
> So two questions, a) is there a way from software or GUI to find out if the pi is throttling? b) is there a way from software or GUI to find out if the input voltage is dripping

There is existence proof of this in that the menubar has a widget (in Raspbian, at least) that shows the cpu temperature and changes color when it is throttling. I'm making a possibly naive assumption that it actually tells the truth.

>
> oh, and c) how did you debug this?

Oh, the usual way; curse, swear, try ludicrous ideas that make no sense just moments later. Since I have two Pi4, one with an SSD and one without I was able to swap the SSD out and try the uSD to see if it was anything to do with that. I ran an eeprom state updater/checker (remember, unlike the earlier models the Pi 4 has a chunk of eeprom for the hardware boot code) and showed that they were claimed identical. I made sure to run the exact same vm/image on both.

Eventually the Pi told me what the problem was itself - a widget I'd never seen before appeared saying 'voltage low. check your power supply'. Well, yes, there you go. I'd not previously considered that throttling was caused by anything other than temperature and was in the middle of writing a post to the Pi forums to ask why the problem Pi was indicating throttled when it was only at 30C, whereas the other was happy up to 50C with no throttling. Lesson learned.
Luckily I had another suitable PSU on my desk awaiting a case build for the other Pi 4 so I had one to swap in.

All in all, just like debugging an errant Cog. But swearier.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Java: the best argument for Smalltalk since C++






Reply | Threaded
Open this post in threaded view
|

Re: Weird Pi Squeak slowdown alert

Bruce O'Neel-2
In reply to this post by timrowledge

And really, just how much development software does a light switch need to load by default?

Given the number of updates they seem to need, quite a bit...

03 October 2020 03:16 tim Rowledge <[hidden email]> wrote:


> On 2020-10-02, at 3:52 PM, Mariano Martinez Peck wrote:
>
> Hi Tim,
>
> Last week Instantiations presented at RIoT Developer month "Faster & More Efficient: JITs for IoT",

You guys are having a lot of fun. Nice.

The real trick for IoT stuff would be to make a system that can run on an ESP32 class chip. I'm pretty sure it's doable, but not easy. The good news is that with built-in WiFi one could do a sort of OOZE/LOOM/network virtual memory thing. And really, just how much development software does a light switch need to load by default?


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: DTF: Dump Tape to Floor






Reply | Threaded
Open this post in threaded view
|

Re: Weird Pi Squeak slowdown alert

timrowledge
In reply to this post by Bruce O'Neel-2


> On 2020-10-03, at 2:03 AM, Bruce O'Neel <[hidden email]> wrote:
>
> 3.  There is a vcgencmd get_throttled
>
> http://raspberrypi.stackexchange.com/questions/83184/ddg#83185
>
> vcgencmd is an interesting command for many of these sorts of things.  You can measure the CPU and GPU temp.

Ah, yes. I vaguely recall having to use it some years ago to set a bit to allow usd boot on early Pi 3 models.
I found a useful script from RonR (who provides many such on the PI forums, including a v.useful "set up usb boot" script) to report the relevant status info -
https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=254024&p=1550160&hilit=pistatus#p1550160

Also finally found the doc for vcgencmd at https://www.raspberrypi.org/documentation/raspbian/applications/vcgencmd.md
and boy, does it do a lot.  Looking at the output of the vcgencmd commands command, there are more commands than currently documented. The source is on github though, so one could work it out eventually!

Be cool to have a morph that displays the important status occasionally. Or I could hook up one of the dials I wrote for the weather station for that SteamPunk style :-)

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: XI: Execute Invalid op-code



Reply | Threaded
Open this post in threaded view
|

Re: Weird Pi Squeak slowdown alert

Mariano Martinez Peck
That vcgencmd would be a great addition to the rpi monitor I normally use: https://rpi-experiences.blogspot.com/p/rpi-monitor.html

On Sat, Oct 3, 2020 at 2:32 PM tim Rowledge <[hidden email]> wrote:


> On 2020-10-03, at 2:03 AM, Bruce O'Neel <[hidden email]> wrote:
>
> 3.  There is a vcgencmd get_throttled
>
> http://raspberrypi.stackexchange.com/questions/83184/ddg#83185
>
> vcgencmd is an interesting command for many of these sorts of things.  You can measure the CPU and GPU temp.

Ah, yes. I vaguely recall having to use it some years ago to set a bit to allow usd boot on early Pi 3 models.
I found a useful script from RonR (who provides many such on the PI forums, including a v.useful "set up usb boot" script) to report the relevant status info -
https://www.raspberrypi.org/forums/viewtopic.php?f=63&t=254024&p=1550160&hilit=pistatus#p1550160

Also finally found the doc for vcgencmd at https://www.raspberrypi.org/documentation/raspbian/applications/vcgencmd.md
and boy, does it do a lot.  Looking at the output of the vcgencmd commands command, there are more commands than currently documented. The source is on github though, so one could work it out eventually!

Be cool to have a morph that displays the important status occasionally. Or I could hook up one of the dials I wrote for the weather station for that SteamPunk style :-)

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: XI: Execute Invalid op-code





--
Mariano Martinez Peck


Reply | Threaded
Open this post in threaded view
|

Re: Weird Pi Squeak slowdown alert

Mariano Martinez Peck
In reply to this post by timrowledge


On Fri, Oct 2, 2020 at 10:17 PM tim Rowledge <[hidden email]> wrote:


> On 2020-10-02, at 3:52 PM, Mariano Martinez Peck <[hidden email]> wrote:
>
> Hi Tim,
>
> Last week Instantiations presented at RIoT Developer month "Faster & More Efficient: JITs for IoT",

You guys are having a lot of fun. Nice.

The real trick for IoT stuff would be to make a system that can run on an ESP32 class chip. I'm pretty sure it's doable, but not easy. The good news is that with built-in WiFi one could do a sort of OOZE/LOOM/network virtual memory thing. And really, just how much development software does a light switch need to load by default?



Being that also the topic of my PhD with Marea, I would say that I love the idea hahaha. Something that VAST has that is very cool is that you can bootstrap images from scratch with what you need and then use the "Packager" to reduce even the "unused" code. A few examples we have:
1) Seaside running in 3MB image
2) TCP/HTTP server running in 1MB image
3) VM benchmarks running in a 500KB image (used in the above JIT presentation)

So yeah... so many nice research projects :)