stuck headless server running as a Windows Service
Asking the community for ideas on the cause or how to debug....
For many years, we have had a headless server application running as a Windows Service. Fairly recently we upgraded from VA Smalltalk 8.5.2 to 8.6.3. In the past 3 months, one customer of ours has had a problem in just one environment (production).
Users on the customer network connect to our server via SST from our fat client application. Our server application logs things, including some things periodically even if no users are doing anything.
The problem is that maybe twice a week, our server seems to not be responsive. It is at apparently zero CPU usage. Clients connected get no response. New clients can't connect to the server. Our log file doesn't get written. But a separate shell script that we built to log the time every minute into a file - still logs fine - so it's not like the filesystem went away.
Amazingly, the server stops gracefully when stopped via the Services window. Normally this would involve more logging (not seen) and a Windows Application Event Log entry (not seen). This is so puzzling to me, how we can seem so stuck, not logging, yet shut down gracefully.
Our server makes callouts to the filesystem, DB2, MQ, and a web service via HTTPS (OpenSSL) (though from log evidence the latter seems to not be happening at the time of the problem). My memory from the past is that a callout to the filesystem is synchronous (locks up the image) so could explain the stuck problem - but I suspect the server Windows Service would then be blind from the stop request coming from Windows. If we had interrupts disabled, I believe you wouldn't be able to gracefully stop the Service.