Using squeak rfb/vnc server with unix server stuff

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|

Using squeak rfb/vnc server with unix server stuff

timrowledge
I've been doing some work to use the personal squeaksource package and having some problems that Chris has been helping me with-

BUT

I simply can't work out how to get a connection from one squeak image running the rfb/vnc viewer to another running the server *when that image is run via daemontools*. If you run the server image directly (as in ./vm/squeak -vm display-none ss.image run.st where the run.st includes starting up the vnc server etc) then you get the expected no window running of the image but the vnc viewer connects to it easily. Run the same command as part of a daemontools service and the viewer cannot connect.

Somebody out there must know why....

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
A fool and his money are soon partying



Reply | Threaded
Open this post in threaded view
|

Re: Using squeak rfb/vnc server with unix server stuff

Nicola Mingotti

I have one Squeak running in a server and I access it through VNC.
But, I am using Unix VNC client/server programs, not Squeak native.
 
If you get it all working in Squeak please let me know.

bye
Nicola




On 9/10/19 12:49 PM, tim Rowledge wrote:
I've been doing some work to use the personal squeaksource package and having some problems that Chris has been helping me with- 

BUT

I simply can't work out how to get a connection from one squeak image running the rfb/vnc viewer to another running the server *when that image is run via daemontools*. If you run the server image directly (as in ./vm/squeak -vm display-none ss.image run.st where the run.st includes starting up the vnc server etc) then you get the expected no window running of the image but the vnc viewer connects to it easily. Run the same command as part of a daemontools service and the viewer cannot connect.

Somebody out there must know why....

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
A fool and his money are soon partying






Reply | Threaded
Open this post in threaded view
|

Re: Using squeak rfb/vnc server with unix server stuff

Levente Uzonyi
In reply to this post by timrowledge
Hi Tim,

On Tue, 10 Sep 2019, tim Rowledge wrote:

> I've been doing some work to use the personal squeaksource package and having some problems that Chris has been helping me with-
>
> BUT
>
> I simply can't work out how to get a connection from one squeak image running the rfb/vnc viewer to another running the server *when that image is run via daemontools*. If you run the server image directly (as in ./vm/squeak -vm display-none ss.image run.st where the run.st includes starting up the vnc server etc) then you get the expected no window running of the image but the vnc viewer connects to it easily. Run the same command as part of a daemontools service and the viewer cannot connect.
>
> Somebody out there must know why....

Try

lsof -p <PID_OF_THE_SQUEAK_VM_RUNNING_YOUR_IMAGE>

It should tell you whether the RFB server is really listening on the tcp
port you defined or not.
If not, there are some possibilities
- missing sources file hangs the startup process, thus RFBServer hasn't
been started
- another program is using the port RFBServer is trying to open
- the port number you defined is less than 1024, thus you can't listen on
it unless you're root. But you don't want to run the VM as root.

Also, if you use RFBServer on any machine, make sure that you have a
firewall blocking its port from the outside. You should only be able to
connect to it through an ssh tunnel.

Levente

>
> tim
> --
> tim Rowledge; [hidden email]; http://www.rowledge.org/tim
> A fool and his money are soon partying

Reply | Threaded
Open this post in threaded view
|

Re: Using squeak rfb/vnc server with unix server stuff

Edgar J. De Cleene-3
In reply to this post by Nicola Mingotti
Re: [squeak-dev] Using squeak rfb/vnc server with unix server stuff Like know your setup.
Could tell more ?


On 10/09/2019, 19:05, "Nicola Mingotti" <[hidden email]> wrote:

  
 I have one Squeak running in a server and I access it through VNC.
 But, I am using Unix VNC client/server programs, not Squeak native.
  
 If you get it all working in Squeak please let me know.
 
 bye
 Nicola


Reply | Threaded
Open this post in threaded view
|

Re: Using squeak rfb/vnc server with unix server stuff

timrowledge
In reply to this post by Levente Uzonyi
Well, this is all very curious. Some good news at the end, but first some details of testing for the record because maybe, just maybe it will help somebody else one day.

> On 2019-09-10, at 5:41 PM, Levente Uzonyi <[hidden email]> wrote:
>

> Try
>
> lsof -p <PID_OF_THE_SQUEAK_VM_RUNNING_YOUR_IMAGE>


OK, after installing it I get 'interesting' results.

The image as run via daemontools (let's acall it the 'background image') results in no report from `lsof -p ` other than a warning about "can't stat() fuse.gvfsd-fuse file system /run/user/1000/gvfs". Except just now on the n'th attempt (where n is probably around 10) at restarting the image I also actually got a listing of a dozen or so open files like libasound, the directory and a couple of pipes.
`lsof -i:5900-5950` shows the single vnc connection to my iMac vnc client, so we kow it is working at that level, at least.

Using lsof -i  when the *same* image is run directly from the commandline, with the same input .st file etc shows the new 5901 port open, and after I connect it shows the ESTABLISHED status.

>
> It should tell you whether the RFB server is really listening on the tcp port you defined or not.
> If not, there are some possibilities
> - missing sources file hangs the startup process, thus RFBServer hasn't been started

That's an interesting idea and I'd hope that we'd get an error log in that case but so the the daemontools 'run' script isn't giving me any output all despite claiming to redirect stdout/err to a log.  The directly run image happily copies all Transcript output to stdout and it appears in the terminal.

OK, this is a bit different; lsof is now showing me that the background image is listening on 5901 as hoped and after connecting from the viewer image it is apparently ESTABLISHED. But no actual display appears...

I added a couple of very deliberate write-to-known-file logging lines and both output ok, demonstrating that the background image has got past the VNC setup. I guess this is good. I added the RFB log info to my file and that seems ok too. But still no display.

I tried out `kill -10 PID` to get the status dump that Eliot added some time ago. In the background image it seemed t obe stuck in attemtping to write to a file and it seemed to be as a result of the Transcript redirecting to stdout that I had previously enabled in the hope of getting some useful information! So, turn off that preference in the image.. .and we get a display in the VNC viewer. Yay!

So; after all that annoyance it appears that some aspect of the daemontools related script used to start the image is causing attempts to write to stdout/err to block. I can't say I'm hugely surprised at this, pretty much everything I've ever had problems with seems to come down to some weird crap in unix (or Windows, etc).

I guess the question has now become something like
"does anyone understand daemontools related usage (or systemd etc) to be able to send stdout/err output to somewhere helpful?

The current 'run' script I have is -

#!/bin/bash
# cd /home/pi/SqSrcDeploy/ss
# redirect stderr to stdout so error messages will show up in log/main
exec 2>&1

# remove Magma _open indicator (same as a "lock" file),
# this subverts Magma's safety check, but if controlled by daemontools
# then we know it shouldn't be due to duplicate run
setuidgid pi rm -f /ss.magma/_open

# permission for root to run the HT version of the vm
ulimit -r 2
exec setuidgid pi ./vm/bin/squeak -vm display=none ss.image run.st `cat webPort`



>
> Also, if you use RFBServer on any machine, make sure that you have a firewall blocking its port from the outside. You should only be able to
> connect to it through an ssh tunnel.

Yeah, the security thing will be important. The VNC server does at least have DES encrypted passwords available. since all this 'fun' is related to running a private squeaksource server I think we'll be wanting to run it behind a firewall, use a VPN or similar and probably change the user registration stuff to only be usable by some admin etc.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: DPN: Double Precision No-op



Reply | Threaded
Open this post in threaded view
|

Re: Using squeak rfb/vnc server with unix server stuff

Chris Muller-3
Hi Tim,

I don't have a definitive answer, but some tidbits not yet mentioned in this thread.

daemontools runs programs in a *minimal environment*.  It uses setuidgid to set the user, so it needs to have permissions on the directory and ./log subdirectories,

and... I seem to remember something about the "environment" that daemontools runs in, that even though as your user "pi", it does not include a :1 display -- or something like that.  That may be why you're experiencing the difference between connecting to RFB when its running in the daemontools environment -- but I could be wrong about that -- since its Squeak's RFB, it seems like it should just be an open port with data...   Very strange.  Levente is the expert on this.


>
> It should tell you whether the RFB server is really listening on the tcp port you defined or not.
> If not, there are some possibilities
> - missing sources file hangs the startup process, thus RFBServer hasn't been started

That's an interesting idea and I'd hope that we'd get an error log in that case but so the the daemontools 'run' script isn't giving me any output all despite claiming to redirect stdout/err to a log.

daemontools runs the log as a separate service, I can't remember if you have to start it separately, if so, the command would be the same as your service, just add /log to the end:

     svc -u /service/squeaksource/log

Beyond that, double check those permissions...

There is also the application-level log, "ss.log" in the image directory, but this is just app-level stuff.
 
  The directly run image happily copies all Transcript output to stdout and it appears in the terminal.

OK, this is a bit different; lsof is now showing me that the background image is listening on 5901 as hoped and after connecting from the viewer image it is apparently ESTABLISHED. But no actual display appears...

I found that Squeak's RFB seems to have a bug where, when it starts, it's blank, but if you RESIZE the window, it'll "refresh" and display the desktop...
 

I added a couple of very deliberate write-to-known-file logging lines and both output ok, demonstrating that the background image has got past the VNC setup. I guess this is good. I added the RFB log info to my file and that seems ok too. But still no display.

I tried out `kill -10 PID` to get the status dump that Eliot added some time ago. In the background image it seemed t obe stuck in attemtping to write to a file and it seemed to be as a result of the Transcript redirecting to stdout that I had previously enabled

This makes me wonder whether FileStream class>>stdout needs to be guarded with a Mutex or something.
 
in the hope of getting some useful information! So, turn off that preference in the image.. .and we get a display in the VNC viewer. Yay!

So; after all that annoyance it appears that some aspect of the daemontools related script used to start the image is causing attempts to write to stdout/err to block. I can't say I'm hugely surprised at this, pretty much everything I've ever had problems with seems to come down to some weird crap in unix (or Windows, etc).

I actually think its because daemontools is strict in order to be secure.  You may simply have a permissions issue.

From your image directory, you could try a sudo setuidgid pi echo test > './log/main/test.out'  and see if you get any revealing error messages?
 


I guess the question has now become something like
"does anyone understand daemontools related usage (or systemd etc) to be able to send stdout/err output to somewhere helpful?

The logging is definitely working on source.squeak.org and my laptops.  Assuming all your directories are owned by "pi" all the way down, I don't see any problems with your setup...   just maybe try that sudo svc -u /service/ss/log ...
 

The current 'run' script I have is -

#!/bin/bash
# cd /home/pi/SqSrcDeploy/ss

The ./ss subdirectory is supposed to be a soft-link to your main MC directory, with one project subdirectory per project.

The directory you want to cd to is the directory ABOVE that.

But, I have my cd commented out too, and it still works (on Ubuntu), so maybe not an issue then.
 
# redirect stderr to stdout so error messages will show up in log/main
exec 2>&1

# remove Magma _open indicator (same as a "lock" file),
# this subverts Magma's safety check, but if controlled by daemontools
# then we know it shouldn't be due to duplicate run
setuidgid pi rm -f /ss.magma/_open

# permission for root to run the HT version of the vm
ulimit -r 2
exec setuidgid pi ./vm/bin/squeak -vm display=none ss.image run.st `cat webPort`



>
> Also, if you use RFBServer on any machine, make sure that you have a firewall blocking its port from the outside. You should only be able to
> connect to it through an ssh tunnel.

This is how its done for source.squeak.org.  Thanks for this ingenius setup Levente!

 - Chris
 

Yeah, the security thing will be important. The VNC server does at least have DES encrypted passwords available. since all this 'fun' is related to running a private squeaksource server I think we'll be wanting to run it behind a firewall, use a VPN or similar and probably change the user registration stuff to only be usable by some admin etc.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: DPN: Double Precision No-op





Reply | Threaded
Open this post in threaded view
|

[almost fixed] Re: Using squeak rfb/vnc server with unix server stuff

timrowledge
Tl;dr - The good news is that the RFB/VNC connection pretty much works now, but the logging doesn't.

> On 2019-09-11, at 5:05 PM, Chris Muller <[hidden email]> wrote:
>
> daemontools runs programs in a *minimal environment*.  It uses setuidgid to set the user, so it needs to have permissions on the directory and ./log subdirectories,

There doesn't appear to be a permissions issue for the general running stuff. The only 'locked up' dir is the log/supervise branch.

>
> and... I seem to remember something about the "environment" that daemontools runs in, that even though as your user "pi", it does not include a :1 display -- or something like that.  That may be why you're experiencing the difference between connecting to RFB when its running in the daemontools environment -- but I could be wrong about that -- since its Squeak's RFB, it seems like it should just be an open port with data...   Very strange.  Levente is the expert on this.

It actually does a good job of working out what the display number is from the rfbPort number, which is nice.

One of the initial things blocking me from connecting was the lost password you currently have set (reminder: needs changing!) combined with the problem in the RFBServer >hasPassword method that means any set password overrides the AllowEmptyPasswords setting.  I suspect that it was whilst finding this and changing it that I might have enabled the 'redirect transcript to stdout' and caused so many later problems  :-( We can probably improve the passwords situation in the run.st script you use for personal squeaksource at some point.


>
> daemontools runs the log as a separate service, I can't remember if you have to start it separately, if so, the command would be the same as your service, just add /log to the end:
>
>      svc -u /service/squeaksource/log

Ah. Nothing was mentioned about that before. Bad news - starting it doesn't seem to make any difference so far. I've tried starting it with and without a sudo, it doesn't make any complaints but checking the status return either 0 seconds or 1 second, which makes it seem it isn't feeling too happy. But there isn't any log for the log logger log...


> OK, this is a bit different; lsof is now showing me that the background image is listening on 5901 as hoped and after connecting from the viewer image it is apparently ESTABLISHED. But no actual display appears...
>
> I found that Squeak's RFB seems to have a bug where, when it starts, it's blank, but if you RESIZE the window, it'll "refresh" and display the desktop...

Interesting; that's not happening to me in this case. I think I have pretty solid evidence that my problem with no display was a side effect of the transcript redirection setting. With that disabled I get a connection immediately, which is a big relief.

> From your image directory, you could try a sudo setuidgid pi echo test > './log/main/test.out'  and see if you get any revealing error messages?

That works fine, which I hope means permissions are ok

> The logging is definitely working on source.squeak.org and my laptops.  Assuming all your directories are owned by "pi" all the way down, I don't see any problems with your setup...   just maybe try that sudo svc -u /service/ss/log ...

Right now it just doesn't do anything. Tried taking both the log and ss down and the restarting, no change. Still getting the repeated 0 seconds uptime for the logging service; surely an error signifier?

> The current 'run' script I have is -
>
> #!/bin/bash
> # cd /home/pi/SqSrcDeploy/ss
>
> The ./ss subdirectory is supposed to be a soft-link to your main MC directory, with one project subdirectory per project.
>
> The directory you want to cd to is the directory ABOVE that.
>
> But, I have my cd commented out too, and it still works (on Ubuntu), so maybe not an issue then.

Yup, that's just as your deploy process made it. I haven't been brave enough to change that yet.

One bit of weird is the way I ended up with
/service -> /etc/service
/etc/service contains
   service -> /etc/service
  ss -> /home/pi/SqSrcDeploy/ss
  supervise

It's hard to imagine requiring a link to yourself in a directory? I can't see any way it would get created in your deployment scripts. Deleting it doesn't appear to make any difference though.

tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: SDS: Sort of Do Something



Reply | Threaded
Open this post in threaded view
|

Re: [almost fixed] Re: Using squeak rfb/vnc server with unix server stuff

Chris Muller-4
Ah. Nothing was mentioned about that before. Bad news - starting it doesn't seem to make any difference so far. I've tried starting it with and without a sudo, it doesn't make any complaints but checking the status return either 0 seconds or 1 second, which makes it seem it isn't feeling too happy. But there isn't any log for the log logger log...

Heh, that's funny.  :)  Recall, daemontools simply runs the "run" script in the service directory, so what happens if you try running it in the foreground tty?

   sudo ./log/run

> From your image directory, you could try a sudo setuidgid pi echo test > './log/main/test.out'  and see if you get any revealing error messages?

That works fine, which I hope means permissions are ok 
 
Not necessarily the execute permission needed on log/run.


One bit of weird is the way I ended up with
/service -> /etc/service
/etc/service contains
   service -> /etc/service
  ss -> /home/pi/SqSrcDeploy/ss
  supervise

It's hard to imagine requiring a link to yourself in a directory? I can't see any way it would get created in your deployment scripts. Deleting it doesn't appear to make any difference though.

I think daemontools makes that, no idea why.  I deleted mine, too.

 


Reply | Threaded
Open this post in threaded view
|

Re: [almost fixed] Re: Using squeak rfb/vnc server with unix server stuff

timrowledge


> On 2019-09-11, at 6:59 PM, Chris Muller <[hidden email]> wrote:
>
> Heh, that's funny.  :)  Recall, daemontools simply runs the "run" script in the service directory, so what happens if you try running it in the foreground tty?
>
>    sudo ./log/run

Hmm, well it complains about the non-existence of user 'multilog', which tallies with a suggestion I found online to run
 `ps -aux | grep readproctitle`
which resulted in the barely readable -

root       386  0.0  0.0   1668   304 ?        S    Sep11   1:13 readproctitle service errors: ...start service/run: file does not exist supervise: fatal: unable to start supervise/run: file does not exist supervise: fatal: unable to start service/run: file does not exist setuidgid: fatal: unknown account multilog supervise: fatal: unable to start supervise/run: file does not exist setuidgid: fatal: unknown account multilog supervise: fatal: unable to start service/run: file does not exist

The first weird bit in this is that since multilog is a program and not a user account we have to suspect some aspect of the script ie
"exec setuidgid $WHOAMI multilog t ./main"
from ./log/run

The weird bit *here* is that your configsvc script is clearly expecting to create the log/run script with the $WHOAMI already substituted (in my case with 'pi'). The next line in configsvc correctly uses the $WHOAMI value to set the file ownership.

It looks like the export of the configsvc script is not doing what is expected and missing quotes around the $WHOAMI. Indeed if I manually surround the $WHOAMI with single quotes it now magically creates a correct log/run file. As best I can tell, fixing SSRepository>configsvc solves this particular issue. See a) below.

OK, part b) Why is there not yet a visible log in log/main even after making the multilog stuff run without whining?
From man multilog it seems that
mulitilog t ./main
is supposed to handle auto-rotated log files named 'current' in the directory .log/main and stick a timestamp in front of each entry.
Ah; good old file caching nonsense. Simply using
FileStream stdout nextPutAll: 'hello2'; cr
is not good enough.
FileStream stdout nextPutAll: 'hello2'; cr; flush
actually flushes  something to the log file. Hooray.

We still have a problem with `ps -aux | grep readproctitle` claiming -
supervise/run: file does not exist
It's not the most helpful error message I've ever seen since it doesn't give a full path but if we assume it refers to the 'supervise' directory within the squeaksource working directory then yeah, there is no run file. Is there supposed to be? I'm not getting a clear picture from whacking google around.

Time for a break. My brains are melting.

tim
----------------------
a) fix for
SSRepository>configsvc
        "One time setup creates daemontools service and starts it."
        ^ '#!/bin/bash
WHOAMI=`whoami`

echo creating rotated log/main directory...
mkdir -p log/main
chmod o-rwx log/main

echo --- generate log/run script
echo ''#!/bin/bash
exec 2>&1
exec setuidgid ''$WHOAMI'' multilog t ./main'' > $PWD/log/run
sudo chown -R $WHOAMI log
sudo chmod 750 log/run

echo --- create Repository if not already existing...
./vm/bin/squeak -vm display=none ss.image configsvc.st $1 $WHOAMI > configsvc.out 2>&1

echo installing daemon, it will start...
sudo ln -s $PWD /etc/service/`basename $PWD`
sleep 7
sudo svstat /service/`basename $PWD`' ->
                [ Smalltalk runAndQuit:
                        [ : whoami | SSRepository
                                ensureRepositoryIfNoneCreateNewNamed: 'Personal SqueakSource'
                                administeredBy: whoami ]
{it's just the doubled-up single quotes around $WHOAMI}

--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Fractured Idiom:- HASTE CUISINE - Fast French food



Reply | Threaded
Open this post in threaded view
|

Re: [almost fixed] Re: Using squeak rfb/vnc server with unix server stuff

K K Subbu
On 13/09/19 2:04 AM, tim Rowledge wrote:

> WHOAMI=`whoami`
>
> echo creating rotated log/main directory...
> mkdir -p log/main
> chmod o-rwx log/main
>
> echo --- generate log/run script
> echo ''#!/bin/bash
> exec 2>&1
> exec setuidgid ''$WHOAMI'' multilog t ./main'' > $PWD/log/run
> sudo chown -R $WHOAMI log

WHOAMI is local to the script, so it won't be set in the run script
leading to multilog being treated as a uid argument.

You could change this to the more readable:

echo -- generate log/run script
cat >$PWD/log/run <<EOF
#!/bin/bash
exec 2>&1
exec setuidgid $WHOAMI multilog t ./main
EOF
sudo chown -R $WHOAMI log

HTH .. Subbu

Reply | Threaded
Open this post in threaded view
|

Re: [almost fixed] Re: Using squeak rfb/vnc server with unix server stuff

Chris Muller-4
On Sun, Sep 15, 2019 at 5:17 AM K K Subbu <[hidden email]> wrote:
On 13/09/19 2:04 AM, tim Rowledge wrote:
> WHOAMI=`whoami`
>
> echo creating rotated log/main directory...
> mkdir -p log/main
> chmod o-rwx log/main
>
> echo --- generate log/run script
> echo ''#!/bin/bash
> exec 2>&1
> exec setuidgid ''$WHOAMI'' multilog t ./main'' > $PWD/log/run
> sudo chown -R $WHOAMI log

WHOAMI is local to the script, so it won't be set in the run script

Correct.  It's for the installation script only.
 
leading to multilog being treated as a uid argument.

My undestanding is that Tim's fix, to put double-quotations around it, will cause "$WHOAMI" to resolve during the run of this installation script, hardcoding the current user as the uid argument.
 

You could change this to the more readable:

echo -- generate log/run script
cat >$PWD/log/run <<EOF
#!/bin/bash
exec 2>&1
exec setuidgid $WHOAMI multilog t ./main
EOF
sudo chown -R $WHOAMI log

Thanks!

  - Chris
 

HTH .. Subbu


Reply | Threaded
Open this post in threaded view
|

Re: [almost fixed] Re: Using squeak rfb/vnc server with unix server stuff

timrowledge


> On 2019-09-16, at 12:42 PM, Chris Muller <[hidden email]> wrote:
>
> My undestanding is that Tim's fix, to put double-quotations around it, will cause "$WHOAMI" to resolve during the run of this installation script, hardcoding the current user as the uid argument.
>  

It is intended to do that and, so far, seems to work.


tim
--
tim Rowledge; [hidden email]; http://www.rowledge.org/tim
Strange OpCodes: PWB: Put to Waste Basket