[Tut] GemStone/Seaside and Daemontools

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

[Tut] GemStone/Seaside and Daemontools

Tobias Pape
Hi —,

some time ago, there was a discussion on the list about using daemontools for
the serving gems of GS/SS. After a brief discussion with Dale, I gave it a try
an—Io and behold—it’s working like a charm.[1]
  Now, what is to be done? Read the small tutorial appended.

So Long,
        -Tobias




                               GemStone/Seaside  
                                      and
                                  Daemontools



                                  I PRINCIPLES.
I.1 Intro
(If you’re familiar with daemontools, skip to II)
Daemontools by D.J.Bernstein[2] is a set of tools for managing services or pro-
grams that normally have a non-forking behaviour. The tools then take care of
concerns like logging (given log information is printed to the terminal, by de-
fault), re-starting of dead processes &c. As per the intention of the author,
the daemontools installation and usage is often dissimilar to normal Unix
/Linux or even MacOS.[3]
  The daemontools work by the notion of a ‘service’ that, essentially, is a
program in the aforementioned fashion. Each such service has a directory in a
well-known place in the filesystem where a ‘supervisor’ is able to find it.
From this moment on, the daemontools can be used to control the service; i.e.
shutting down, starting or restarting the service. The key point, in fact, is
that the supervisor takes care of restarting the service whenever it dies.
This is my motivation of taking the gems to the daemontools.



I.2 Installations
Following the guides on the web site of D.J.Bernstein is usually the best idea
to install the daemontools on any Unix/Linux. However, Debian provides some
packages for installation with APT. These packages introduce some differences
to the original installation. Debian specifics are marked D< > in the follow-
ing.



I.3 Directories
As said before, the daemontools, expect services to have a certain directory
in a well-known place. By default, this place would be /service
D<
/etc/service
>. Every directory beneath this place is checked regularly by the supervisor
for new services. If the supervisor finds a file called ‘run’ that is execut-
able, this directory is treated as a service. N.B. it seems to be a common pat-
tern to use only symlinks in the supervisors directory in order to maintain a
separate repository of services and/or service templates. D<There even is a
default directory, /etc/sv to put the original services in. See III.5> We will
use this in II.3.1.



I.4 Logging.
If a service directory contains a directory called ‘log’ that looks like a ser-
vice directory, this is treated as the services logging service.[4] Hence,
there has to be a ‘run’ file to make that working. Every output of the ‘main’
service (stdout) is then piped to the logging services input (stdin).






                                  II SCRIPTS.
II.1 Patching
In order to behave as the daemontools expect, the scripts for the GS/SS gems,
i.e. the maintenance gem and the actual seaside gem, have to be patched, espe-
cially for logging handled by the daemontools.[5] For the seaside gem, I will
only give the patches for the FastCGI version, but the Hyper version should be
similar.



II.1.1 ‘nohup’ and ‘exec’
Currently, GS/SS tries to avoid unexpected hangups by using the nohup(1) tool.
However, this conflicts with daemontools which are managing hangups themselves.
As I said above, the daemontools expect a foreground program, thus, the cur-
rently used backgrounding  must be removed.
  Additionally, daemontools require the supervised service to be a singe pro-
cess. That is, a shell script that has child processes would not work correct-
ly[6]. Unfortunately, that is exactly how the GS/SS gems work. To overcome
this, we use the exec(1) command which will replace the shell with the started
child.
  Now the patching. For the maintenance gem that resides in $GEMSTONE/seaside/
bin/startMaintenance, the line

    cat << EOF | nohup $GEMSTONE/bin/topaz -l -T200000 2>&1 > \
    $GEMSTONE_LOGDIR/maintenance_gem.log &

should become

    exec $GEMSTONE/bin/topaz -l -T200000 2>&1 > \
    $GEMSTONE_LOGDIR/maintenance_gem.log << EOF

and for the seaside gem in $GEMSTONE/seaside/bin/seasideGem_FastCGI, change

    cat << EOF | nohup $GEMSTONE/bin/topaz -l -T50000 2>&1 \
    >> $GEMSTONE_LOGDIR/FastCGI_server-${1}.log &

for

    exec $GEMSTONE/bin/topaz -l -T50000 2>&1 \
    >> $GEMSTONE_LOGDIR/FastCGI_server-${1}.log  << EOF

I did not alter the original files but named them differently,
‘maintenanceGem’ and ‘seasideGem_FastCGI’, respectively; and put them besides
their originals. Note that these new scripts will
• not go into background when started and
• replace the shell they are started in
which was exactly the intention.



II.1.2 Logging
To provide the output to the logging service, the output of the gem should be
simply printed and not be redirected to a file. Hence, for the maintenance gem,
change the line from II.1.1

    exec $GEMSTONE/bin/topaz -l -T200000 2>&1 > \
    $GEMSTONE_LOGDIR/maintenance_gem.log << EOF

to

    exec $GEMSTONE/bin/topaz -l -T200000 2>&1  << EOF

and for the seaside gem from

    exec $GEMSTONE/bin/topaz -l -T50000 2>&1 \
    >> $GEMSTONE_LOGDIR/FastCGI_server-${1}.log  << EOF

to

    exec $GEMSTONE/bin/topaz -l -T50000 2>&1 << EOF

Thus, all requirements are met.



II.2 The ‘run’ Scripts
Essentially, all run scripts to follow are a variant of ‘run that gem,’ more
or less easy to configure. Thus, the simplest seaside gem run script would be

-->8--
source /opt/gemstone/product/seaside/defSeaside
exec $GEMSTONE/seaside/bin/seasideGem_FastCGI 9001

--8<--

However, there is a security problem, as this script is executed as root and,
subsequently, the gem is, too. To overcome this problem, daemontools provide
the setuidgid(8) tool which executes a command under a given user. Hence, the
improved version:

-->8--
source /opt/gemstone/product/seaside/defSeaside
exec setuidgid gemstone $GEMSTONE/seaside/bin/seasideGem_FastCGI 9001

--8<--

Now, let’s put it together: We create the directory

    /service/gs_fastcgi         D</etc/service/gs_fastcgi>

and put the run script there. Also, we create the directory

    /service/gs_maintenance     D</etc/serivce/gs_maintenance>

and put the run script there, replacing the actual command from
‘seasideGem_FastCGI 9001’ to ‘maintenanceGem’. For simple setups, this should
suffice, skip to III. For more complex setups, read on.



II.3 Multiple Seaside Gems
When we want to run multiple seaside gems in parallel, it is desirable to not
have to write the run script for every gem and just change the port. When we
decide to change something, we don’t want to touch every run file; this is er-
ror prone.



II.3.1 New Directory Layout
  As said in I.3, it might be useful to have a template for various services.
We use the service template directory[7] and create the service ‘gs_fastcgi.’
However, unlike the setup in II.2, this service is never actually run. Instead,
it serves as template for out, say three, actual services. For each actual
service, we will have a directory in the service template directory, too. In
accordance with the Debian way, we will create links in the service directory
to enable such service (see III. for a Debian-provided tool assisting this pro-
cedure). Hence, when finished, the layout of our service template directory
should be:

D<# ls -l /etc/sv >
  # ls -l /service_tmpl
  total 0
  drwxr-xr-x 3 root root 4096  13:52 gs_fastcgi
  drwxr-xr-x 3 root root 4096  14:07 gs_fastcgi-1
  drwxr-xr-x 3 root root 4096  14:13 gs_fastcgi-2
  drwxr-xr-x 3 root root 4096  14:13 gs_fastcgi-3
  drwxr-xr-x 3 root root 4096  14:17 gs_maintainance

and of the service directory:

D<
  # ls -l /etc/service
  total 0
  lrwxrwxrwx 1 root root 20  14:07 gs_fastcgi-1 -> /etc/sv/gs_fastcgi-1
  lrwxrwxrwx 1 root root 20  14:11 gs_fastcgi-2 -> /etc/sv/gs_fastcgi-2
  lrwxrwxrwx 1 root root 20  14:11 gs_fastcgi-3 -> /etc/sv/gs_fastcgi-3
  lrwxrwxrwx 1 root root 23  14:17 gs_maintainance -> /etc/sv/gs_maintainance
>
  # ls -l /service
  total 0
  lrwxrwxrwx 1 root root 20  14:07 gs_fastcgi-1 -> /service_tmpl/gs_fastcgi-1
  lrwxrwxrwx 1 root root 20  14:11 gs_fastcgi-2 -> /service_tmpl/gs_fastcgi-2
  lrwxrwxrwx 1 root root 20  14:11 gs_fastcgi-3 -> /service_tmpl/gs_fastcgi-3
  lrwxrwxrwx 1 root root 23  14:17 gs_maintainance
    -> /service_tmpl/gs_maintainance

Note that the gs_fastcgi service itself is not linked into the service direc-
tory.



II.3.2 Common and Local Configuration
I decided to differentiate between a common configuration shared between all
services (such as the user that should run the gem) and ‘local’ configuration
that is unique to a certain service (such as the gem’s port). To enable this,
we will use a small shell trick. As the actual run script will reside in
D</etc/sv/gs_fastcgi> /service_tmpl/gs_fastcgi, we will put the common configu-
ration there. The local configuration will reside in the actual service’s temp-
late directory. This looks like this:

D<
  # ls -l /etc/sv/*fastcgi*/config*
  -rw-r--r-- 1 root root 32  14:07 /etc/sv/gs_fastcgi-1/config.local
  -rw-r--r-- 1 root root 32  14:10 /etc/sv/gs_fastcgi-2/config.local
  -rw-r--r-- 1 root root 32  14:10 /etc/sv/gs_fastcgi-3/config.local
  -rw-r--r-- 1 root root 86  13:42 /etc/sv/gs_fastcgi/config
 >
  # ls -l /service_tmpl/*fastcgi*/config*
  -rw-r--r-- 1 root root 32  14:07 /service_tmpl/gs_fastcgi-1/config.local
  -rw-r--r-- 1 root root 32  14:10 /service_tmpl/gs_fastcgi-2/config.local
  -rw-r--r-- 1 root root 32  14:10 /service_tmpl/gs_fastcgi-3/config.local
  -rw-r--r-- 1 root root 86  13:42 /service_tmpl/gs_fastcgi/config

What should go into the configuration

[config] As stated above, the common configuration can include the user that
    should run the gems. Also, I found it valuable to put the standard GS/SS
    stanza there that sets up the GS/SS shell environment:

    -->8--
    #!/bin/sh

    source /opt/gemstone/product/seaside/defSeaside

    export RUNASUSER=gemstone

    --8<--
[config.local] The only thing that is needed locally by each individual gem is
    its port, hence the local configuration becomes (for gs_fastcgi-1)
   
    -->8--
    #!/bin/sh

    export GEM_PORT=9001
   
    --8<--
    (with other port numbers for the other services)



II.3.3 Adapting the ‘run’ Script
In order to make the run script reusable, we introduced configurations. To ac-
tually use these configuration the run script must be adapted:

-->8--
#!/bin/sh

source /opt/gemstone/product/seaside/defSeaside
exec setuidgid $RUNASUSER $GEMSTONE/seaside/bin/seasideGem_FastCGI $GEM_PORT

--8<--

However, to access the configuration variables, we have to include the config-
gurations. Not the facts that
•   the common configuration is in the directory the run script is actually
    residing in;
•   the local configuration in in the directory the run script is linked to.
Knowing this, a small shell trick that provides both directories mentioned
enables us to include both configurations. E.g. for the gs_fastcgi-1 service
in the following script (the run script I currently use) ‘scriptpath’ would be
/service_tmpl/gs_fastcgi-1 D</etc/sv/gs_fastcgi-1> whereas ‘orig_scriptpath’
would be /service_tmpl/gs_fastcgi D</etc/sv/gs_fastcgi>.

-->8--
#!/bin/sh

old_pwd=$(pwd)
cd $(dirname $0)
scriptpath=$(pwd)
cd $old_pwd

orig_scriptpath=$(dirname "$(readlink -f $0)")

source $orig_scriptpath/config

[ -r $scriptpath/config.local ] && source $scriptpath/config.local


exec setuidgid $RUNASUSER $GEMSTONE/seaside/bin/seasideGem_FastCGI $GEM_PORT

--8<--

This way, both the common and the local configurations are included. It is
possible to use this script for the maintenance gem, too (adapted as descibed
above)



II.4 The Logging Scripts
For all services, the logging ‘run’ script simply is
-->8--
#!/bin/sh
exec multilog t ./main

--8<--

When this ‘run’ script is put into the log directory of a service and made exe-
cutable, the daemontools automatically will start use this logging facility.
Note that the ‘multilog’ command provided by daemontools uses timestamps with
a special format (cf. tai64n(8)). To read multilog output more human-readable
refer to the ml* tools (cf. III.5).
  Note that if we are using the Multiple Gems layout from II.3, there is
no need to copy that script over and over. Simply put it into D</etc/sv/
gs_fastcgi/log/run> /service_tmpl/gs_fastcgi/log/run and link to it from the
different actual service:

D<
  # ls -l /etc/sv/*/log/run
  lrwxrwxrwx 1 root root 26  13:55 /etc/sv/gs_fastcgi-1/log/run
    -> /etc/sv/gs_fastcgi/log/run
  lrwxrwxrwx 1 root root 26  13:55 /etc/sv/gs_fastcgi-2/log/run
    -> /etc/sv/gs_fastcgi/log/run
  lrwxrwxrwx 1 root root 26  13:55 /etc/sv/gs_fastcgi-3/log/run
    -> /etc/sv/gs_fastcgi/log/run
  -rwxr-xr-x 1 root root 33  14:02 /etc/sv/gs_fastcgi/log/run
  lrwxrwxrwx 1 root root 26  13:55 /etc/sv/gs_maintainance/log/run
    -> /etc/sv/gs_fastcgi/log/run
>
  # ls -l /etc/sv/*/log/run
  lrwxrwxrwx 1 root root 26  13:55 /etc/sv/gs_fastcgi-1/log/run
    -> /etc/sv/gs_fastcgi/log/run
  lrwxrwxrwx 1 root root 26  13:55 /etc/sv/gs_fastcgi-2/log/run
    -> /etc/sv/gs_fastcgi/log/run
  lrwxrwxrwx 1 root root 26  13:55 /etc/sv/gs_fastcgi-3/log/run
    -> /etc/sv/gs_fastcgi/log/run
  -rwxr-xr-x 1 root root 33  14:02 /etc/sv/gs_fastcgi/log/run
  lrwxrwxrwx 1 root root 26  13:55 /etc/sv/gs_maintainance/log/run
    -> /etc/sv/gs_fastcgi/log/run






                                III COMMANDS.
III.0
Most of the commands explained here use the daemontools svc tool. This tool
takes an _directory_ as argument. Later, some Debian specific tools will be
mentioned, e.g. mlcat or mltail. These tools take a _service name_ as argument.
If your current working directory is /service D</etc/service> this makes no
difference. There are more tools in daemontools; see [2] or [8]



III.1 Starting
Whenever you put a service into the service directory, whether by creating a
directory or linking into it, the supervisor will try to start it by running
its ‘run’ script. Hence, using the previously described method will already
have the service(s) started.
  Should a service have been stopped in any way, use

  # svc -u /service/gs_yourservice
D<# svc -u /etc/service/gs_yourservice>
 
to start it.



III.2 Restarting
Daemontools primary aim is to keep its services running. Thus, simply killing
your service by using

  # kill PIDOFYOURSERVICE
  # killall NAMEOFYOURSERVICE
   
would not stop your service completely but rather kill the application and
have it restarted by the supervisor. The svc tool can be used to do this, too:

  # svc -t /service/gs_yourservice
D<# svc -t /etc/service/gs_yourservice>

to terminate (signal(7) number 15, similar to kill),

  # svc -k /service/gs_yourservice
D<# svc -k /etc/service/gs_yourservice>

to kill (signal(7) number 9, similar to kill -9), or

  # svc -h /service/gs_yourservice
D<# svc -h /etc/service/gs_yourservice>

to send hangup (signal(7) number 1) to the service.



III.3 Stopping
As mentioned above, terminating/killing a service does not keep it shut down.
To keep your service shut down, use

  # svc -d /service/gs_yourservice
D<# svc -d /etc/service/gs_yourservice>

When you issued this command, you might want to use svc -u as in III.1 to
bring it up again.



III.4 Status
In order to see whether the services are running you might issue

  # svstat /service/gs_yourservice
D<# svstat /etc/service/gs_yourservice>

and it will print the status of the service. This is really useful if you do
this in the service directory /service D</etc/service>:

  # svstat *

as this provides an overview of all services. A healthy system might look like

  # svstat *
  gs_fastcgi-1: up (pid 8945) 1094 seconds
  gs_fastcgi-2: up (pid 31925) 14185 seconds
  gs_fastcgi-3: up (pid 31929) 14185 seconds
  gs_maintainance: up (pid 31909) 14199 seconds

but with several services not running, this might look like

  # svstat *
  gs_fastcgi-1: up (pid 8945) 1385 seconds
  gs_fastcgi-2: down 6 seconds, normally up
  gs_fastcgi-3: up (pid 31929) 14476 seconds
  gs_maintainance: down 3 seconds, normally up



III.5 Helpers
When using the Debian daemontools-run package, there is an additional command
that should help you enabling or disabling services from your service template
directory. That way, we might have enabled the maintenance service by issuing

  # update-service --add /etc/sv/gs_maintenance

(where we could have specified an optional name for the link in the
/etc/service directory). Similar, the following command will remove the en-
abled service

  # update-service --remove /etc/sv/gs_maintenance

We could also list all services currently enabled:

  # update-service --list
  gs_fastcgi-1
  gs_fastcgi-2
  gs_fastcgi-3
  gs_maintainance

There is a Debian package called svtools that provides various tools for
•   managing/viewing multilog files (mlcat, mltail, mltac…)
•   creating services and init.d wrappers (svsetup, svinid-create)
•   various information (svdir, svinfo)
This package is also responsible for the creation of the /etc/sv directory.






                               IV IMPROVEMENTS.
IV.1 Restrictions.
Now for the restriction mentioned initially. The runSeasideGems command no
longer works. It is responsible for starting/stopping/restarting the seaside
gems. Also, it is used from within the GemTools to perform these very tasks.
However, the techniques of the script interfere with the ones of daemontools.
Thus, I put an

    exit 0;
   
near the top of $GEMSTONE/seaside/bin/runSeasideGems. Unfortunately, it is not
possible to start/stop/restart the gems from within the GemTools with this
patch.
  Another possibility is to change the script to use the svc tool. Thus, it
would be possible to use the GemTools for the start/stop/restarting, again.



IV.2 Daemontools Everywhere.
It is possible to use the daemontools to control the stone itself rather than
only the gems. However, you should be aware, that you have to
•   deal with the existing init.d scripts,
•   take care of the start/stop order of the netldi and the Stone.



                                                            That’s all, folks.



[1] Well, almost, the limitation is described later.
[2] http://cr.yp.to/daemontools.html
[3] I will focus on Linux here, with special respect to Debian. I installed
    daemontools, daemontools-run, and svtools.
[4] Meta, isn’t it?
[5] Note that this is absolutely optional. The default GS/SS logging works
    when using daemontools, too.
[6] I.e. when using daemontools to stop a service, the shell script is termina-
    ted and not the children. This actually happened to me.
[7] Note that this directory is provided by default only by the Debian
    daemontools-run package. I will use /service_tmpl/ here for the non-Debian
    case.
[8] supervise(8), svok(8), svstat(8), svscanboot(8), svscan(8),
    readproctitle(8), fghack(8), pgrphack(8), multilog(8), tai64n(8),
    tai64nlocal(8), setuidgid(8), envuidgid(8), envdir(8), softlimit(8),
    setlock(8)

Reply | Threaded
Open this post in threaded view
|

Re: [Tut] GemStone/Seaside and Daemontools

Dale Henrichs
Tobias,

Thanks very much for this excellent writeup. I've copied your writeup
onto the glass db site along with edits to make the text wiki friendly:

   http://code.google.com/p/glassdb/wiki/GLASSDaemonTools

Dale

Tobias Pape wrote:

> Hi —,
>
> some time ago, there was a discussion on the list about using daemontools for
> the serving gems of GS/SS. After a brief discussion with Dale, I gave it a try
> an—Io and behold—it’s working like a charm.[1]
>   Now, what is to be done? Read the small tutorial appended.
>
> So Long,
>         -Tobias
>
>
>
>
>                                GemStone/Seaside
>                                       and
>                                   Daemontools
>
>
>
>                                   I PRINCIPLES.
> I.1 Intro
> (If you’re familiar with daemontools, skip to II)
> Daemontools by D.J.Bernstein[2] is a set of tools for managing services or pro-
> grams that normally have a non-forking behaviour. The tools then take care of
> concerns like logging (given log information is printed to the terminal, by de-
> fault), re-starting of dead processes &c. As per the intention of the author,
> the daemontools installation and usage is often dissimilar to normal Unix
> /Linux or even MacOS.[3]
>   The daemontools work by the notion of a ‘service’ that, essentially, is a
> program in the aforementioned fashion. Each such service has a directory in a
> well-known place in the filesystem where a ‘supervisor’ is able to find it.
> From this moment on, the daemontools can be used to control the service; i.e.
> shutting down, starting or restarting the service. The key point, in fact, is
> that the supervisor takes care of restarting the service whenever it dies.
> This is my motivation of taking the gems to the daemontools.
>
>
>
> I.2 Installations
> Following the guides on the web site of D.J.Bernstein is usually the best idea
> to install the daemontools on any Unix/Linux. However, Debian provides some
> packages for installation with APT. These packages introduce some differences
> to the original installation. Debian specifics are marked D< > in the follow-
> ing.
>
>
>
> I.3 Directories
> As said before, the daemontools, expect services to have a certain directory
> in a well-known place. By default, this place would be /service
> D<
> /etc/service
>> . Every directory beneath this place is checked regularly by the supervisor
> for new services. If the supervisor finds a file called ‘run’ that is execut-
> able, this directory is treated as a service. N.B. it seems to be a common pat-
> tern to use only symlinks in the supervisors directory in order to maintain a
> separate repository of services and/or service templates. D<There even is a
> default directory, /etc/sv to put the original services in. See III.5> We will
> use this in II.3.1.
>
>
>
> I.4 Logging.
> If a service directory contains a directory called ‘log’ that looks like a ser-
> vice directory, this is treated as the services logging service.[4] Hence,
> there has to be a ‘run’ file to make that working. Every output of the ‘main’
> service (stdout) is then piped to the logging services input (stdin).
>
>
>
>
>
>
>                                   II SCRIPTS.
> II.1 Patching
> In order to behave as the daemontools expect, the scripts for the GS/SS gems,
> i.e. the maintenance gem and the actual seaside gem, have to be patched, espe-
> cially for logging handled by the daemontools.[5] For the seaside gem, I will
> only give the patches for the FastCGI version, but the Hyper version should be
> similar.
>
>
>
> II.1.1 ‘nohup’ and ‘exec’
> Currently, GS/SS tries to avoid unexpected hangups by using the nohup(1) tool.
> However, this conflicts with daemontools which are managing hangups themselves.
> As I said above, the daemontools expect a foreground program, thus, the cur-
> rently used backgrounding  must be removed.
>   Additionally, daemontools require the supervised service to be a singe pro-
> cess. That is, a shell script that has child processes would not work correct-
> ly[6]. Unfortunately, that is exactly how the GS/SS gems work. To overcome
> this, we use the exec(1) command which will replace the shell with the started
> child.
>   Now the patching. For the maintenance gem that resides in $GEMSTONE/seaside/
> bin/startMaintenance, the line
>
>     cat << EOF | nohup $GEMSTONE/bin/topaz -l -T200000 2>&1 > \
>     $GEMSTONE_LOGDIR/maintenance_gem.log &
>
> should become
>
>     exec $GEMSTONE/bin/topaz -l -T200000 2>&1 > \
>     $GEMSTONE_LOGDIR/maintenance_gem.log << EOF
>
> and for the seaside gem in $GEMSTONE/seaside/bin/seasideGem_FastCGI, change
>
>     cat << EOF | nohup $GEMSTONE/bin/topaz -l -T50000 2>&1 \
>     >> $GEMSTONE_LOGDIR/FastCGI_server-${1}.log &
>
> for
>
>     exec $GEMSTONE/bin/topaz -l -T50000 2>&1 \
>     >> $GEMSTONE_LOGDIR/FastCGI_server-${1}.log  << EOF
>
> I did not alter the original files but named them differently,
> ‘maintenanceGem’ and ‘seasideGem_FastCGI’, respectively; and put them besides
> their originals. Note that these new scripts will
> •       not go into background when started and
> •       replace the shell they are started in
> which was exactly the intention.
>
>
>
> II.1.2 Logging
> To provide the output to the logging service, the output of the gem should be
> simply printed and not be redirected to a file. Hence, for the maintenance gem,
> change the line from II.1.1
>
>     exec $GEMSTONE/bin/topaz -l -T200000 2>&1 > \
>     $GEMSTONE_LOGDIR/maintenance_gem.log << EOF
>
> to
>
>     exec $GEMSTONE/bin/topaz -l -T200000 2>&1  << EOF
>
> and for the seaside gem from
>
>     exec $GEMSTONE/bin/topaz -l -T50000 2>&1 \
>     >> $GEMSTONE_LOGDIR/FastCGI_server-${1}.log  << EOF
>
> to
>
>     exec $GEMSTONE/bin/topaz -l -T50000 2>&1 << EOF
>
> Thus, all requirements are met.
>
>
>
> II.2 The ‘run’ Scripts
> Essentially, all run scripts to follow are a variant of ‘run that gem,’ more
> or less easy to configure. Thus, the simplest seaside gem run script would be
>
> -->8--
> source /opt/gemstone/product/seaside/defSeaside
> exec $GEMSTONE/seaside/bin/seasideGem_FastCGI 9001
>
> --8<--
>
> However, there is a security problem, as this script is executed as root and,
> subsequently, the gem is, too. To overcome this problem, daemontools provide
> the setuidgid(8) tool which executes a command under a given user. Hence, the
> improved version:
>
> -->8--
> source /opt/gemstone/product/seaside/defSeaside
> exec setuidgid gemstone $GEMSTONE/seaside/bin/seasideGem_FastCGI 9001
>
> --8<--
>
> Now, let’s put it together: We create the directory
>
>     /service/gs_fastcgi         D</etc/service/gs_fastcgi>
>
> and put the run script there. Also, we create the directory
>
>     /service/gs_maintenance     D</etc/serivce/gs_maintenance>
>
> and put the run script there, replacing the actual command from
> ‘seasideGem_FastCGI 9001’ to ‘maintenanceGem’. For simple setups, this should
> suffice, skip to III. For more complex setups, read on.
>
>
>
> II.3 Multiple Seaside Gems
> When we want to run multiple seaside gems in parallel, it is desirable to not
> have to write the run script for every gem and just change the port. When we
> decide to change something, we don’t want to touch every run file; this is er-
> ror prone.
>
>
>
> II.3.1 New Directory Layout
>   As said in I.3, it might be useful to have a template for various services.
> We use the service template directory[7] and create the service ‘gs_fastcgi.’
> However, unlike the setup in II.2, this service is never actually run. Instead,
> it serves as template for out, say three, actual services. For each actual
> service, we will have a directory in the service template directory, too. In
> accordance with the Debian way, we will create links in the service directory
> to enable such service (see III. for a Debian-provided tool assisting this pro-
> cedure). Hence, when finished, the layout of our service template directory
> should be:
>
> D<# ls -l /etc/sv >
>   # ls -l /service_tmpl
>   total 0
>   drwxr-xr-x 3 root root 4096  13:52 gs_fastcgi
>   drwxr-xr-x 3 root root 4096  14:07 gs_fastcgi-1
>   drwxr-xr-x 3 root root 4096  14:13 gs_fastcgi-2
>   drwxr-xr-x 3 root root 4096  14:13 gs_fastcgi-3
>   drwxr-xr-x 3 root root 4096  14:17 gs_maintainance
>
> and of the service directory:
>
> D<
>   # ls -l /etc/service
>   total 0
>   lrwxrwxrwx 1 root root 20  14:07 gs_fastcgi-1 -> /etc/sv/gs_fastcgi-1
>   lrwxrwxrwx 1 root root 20  14:11 gs_fastcgi-2 -> /etc/sv/gs_fastcgi-2
>   lrwxrwxrwx 1 root root 20  14:11 gs_fastcgi-3 -> /etc/sv/gs_fastcgi-3
>   lrwxrwxrwx 1 root root 23  14:17 gs_maintainance -> /etc/sv/gs_maintainance
>   # ls -l /service
>   total 0
>   lrwxrwxrwx 1 root root 20  14:07 gs_fastcgi-1 -> /service_tmpl/gs_fastcgi-1
>   lrwxrwxrwx 1 root root 20  14:11 gs_fastcgi-2 -> /service_tmpl/gs_fastcgi-2
>   lrwxrwxrwx 1 root root 20  14:11 gs_fastcgi-3 -> /service_tmpl/gs_fastcgi-3
>   lrwxrwxrwx 1 root root 23  14:17 gs_maintainance
>     -> /service_tmpl/gs_maintainance
>
> Note that the gs_fastcgi service itself is not linked into the service direc-
> tory.
>
>
>
> II.3.2 Common and Local Configuration
> I decided to differentiate between a common configuration shared between all
> services (such as the user that should run the gem) and ‘local’ configuration
> that is unique to a certain service (such as the gem’s port). To enable this,
> we will use a small shell trick. As the actual run script will reside in
> D</etc/sv/gs_fastcgi> /service_tmpl/gs_fastcgi, we will put the common configu-
> ration there. The local configuration will reside in the actual service’s temp-
> late directory. This looks like this:
>
> D<
>   # ls -l /etc/sv/*fastcgi*/config*
>   -rw-r--r-- 1 root root 32  14:07 /etc/sv/gs_fastcgi-1/config.local
>   -rw-r--r-- 1 root root 32  14:10 /etc/sv/gs_fastcgi-2/config.local
>   -rw-r--r-- 1 root root 32  14:10 /etc/sv/gs_fastcgi-3/config.local
>   -rw-r--r-- 1 root root 86  13:42 /etc/sv/gs_fastcgi/config
>  >
>   # ls -l /service_tmpl/*fastcgi*/config*
>   -rw-r--r-- 1 root root 32  14:07 /service_tmpl/gs_fastcgi-1/config.local
>   -rw-r--r-- 1 root root 32  14:10 /service_tmpl/gs_fastcgi-2/config.local
>   -rw-r--r-- 1 root root 32  14:10 /service_tmpl/gs_fastcgi-3/config.local
>   -rw-r--r-- 1 root root 86  13:42 /service_tmpl/gs_fastcgi/config
>
> What should go into the configuration
>
> [config] As stated above, the common configuration can include the user that
>     should run the gems. Also, I found it valuable to put the standard GS/SS
>     stanza there that sets up the GS/SS shell environment:
>
>     -->8--
>     #!/bin/sh
>
>     source /opt/gemstone/product/seaside/defSeaside
>
>     export RUNASUSER=gemstone
>
>     --8<--
> [config.local] The only thing that is needed locally by each individual gem is
>     its port, hence the local configuration becomes (for gs_fastcgi-1)
>
>     -->8--
>     #!/bin/sh
>
>     export GEM_PORT=9001
>
>     --8<--
>     (with other port numbers for the other services)
>
>
>
> II.3.3 Adapting the ‘run’ Script
> In order to make the run script reusable, we introduced configurations. To ac-
> tually use these configuration the run script must be adapted:
>
> -->8--
> #!/bin/sh
>
> source /opt/gemstone/product/seaside/defSeaside
> exec setuidgid $RUNASUSER $GEMSTONE/seaside/bin/seasideGem_FastCGI $GEM_PORT
>
> --8<--
>
> However, to access the configuration variables, we have to include the config-
> gurations. Not the facts that
> •   the common configuration is in the directory the run script is actually
>     residing in;
> •   the local configuration in in the directory the run script is linked to.
> Knowing this, a small shell trick that provides both directories mentioned
> enables us to include both configurations. E.g. for the gs_fastcgi-1 service
> in the following script (the run script I currently use) ‘scriptpath’ would be
> /service_tmpl/gs_fastcgi-1 D</etc/sv/gs_fastcgi-1> whereas ‘orig_scriptpath’
> would be /service_tmpl/gs_fastcgi D</etc/sv/gs_fastcgi>.
>
> -->8--
> #!/bin/sh
>
> old_pwd=$(pwd)
> cd $(dirname $0)
> scriptpath=$(pwd)
> cd $old_pwd
>
> orig_scriptpath=$(dirname "$(readlink -f $0)")
>
> source $orig_scriptpath/config
>
> [ -r $scriptpath/config.local ] && source $scriptpath/config.local
>
>
> exec setuidgid $RUNASUSER $GEMSTONE/seaside/bin/seasideGem_FastCGI $GEM_PORT
>
> --8<--
>
> This way, both the common and the local configurations are included. It is
> possible to use this script for the maintenance gem, too (adapted as descibed
> above)
>
>
>
> II.4 The Logging Scripts
> For all services, the logging ‘run’ script simply is
> -->8--
> #!/bin/sh
> exec multilog t ./main
>
> --8<--
>
> When this ‘run’ script is put into the log directory of a service and made exe-
> cutable, the daemontools automatically will start use this logging facility.
> Note that the ‘multilog’ command provided by daemontools uses timestamps with
> a special format (cf. tai64n(8)). To read multilog output more human-readable
> refer to the ml* tools (cf. III.5).
>   Note that if we are using the Multiple Gems layout from II.3, there is
> no need to copy that script over and over. Simply put it into D</etc/sv/
> gs_fastcgi/log/run> /service_tmpl/gs_fastcgi/log/run and link to it from the
> different actual service:
>
> D<
>   # ls -l /etc/sv/*/log/run
>   lrwxrwxrwx 1 root root 26  13:55 /etc/sv/gs_fastcgi-1/log/run
>     -> /etc/sv/gs_fastcgi/log/run
>   lrwxrwxrwx 1 root root 26  13:55 /etc/sv/gs_fastcgi-2/log/run
>     -> /etc/sv/gs_fastcgi/log/run
>   lrwxrwxrwx 1 root root 26  13:55 /etc/sv/gs_fastcgi-3/log/run
>     -> /etc/sv/gs_fastcgi/log/run
>   -rwxr-xr-x 1 root root 33  14:02 /etc/sv/gs_fastcgi/log/run
>   lrwxrwxrwx 1 root root 26  13:55 /etc/sv/gs_maintainance/log/run
>     -> /etc/sv/gs_fastcgi/log/run
>   # ls -l /etc/sv/*/log/run
>   lrwxrwxrwx 1 root root 26  13:55 /etc/sv/gs_fastcgi-1/log/run
>     -> /etc/sv/gs_fastcgi/log/run
>   lrwxrwxrwx 1 root root 26  13:55 /etc/sv/gs_fastcgi-2/log/run
>     -> /etc/sv/gs_fastcgi/log/run
>   lrwxrwxrwx 1 root root 26  13:55 /etc/sv/gs_fastcgi-3/log/run
>     -> /etc/sv/gs_fastcgi/log/run
>   -rwxr-xr-x 1 root root 33  14:02 /etc/sv/gs_fastcgi/log/run
>   lrwxrwxrwx 1 root root 26  13:55 /etc/sv/gs_maintainance/log/run
>     -> /etc/sv/gs_fastcgi/log/run
>
>
>
>
>
>
>                                 III COMMANDS.
> III.0
> Most of the commands explained here use the daemontools svc tool. This tool
> takes an _directory_ as argument. Later, some Debian specific tools will be
> mentioned, e.g. mlcat or mltail. These tools take a _service name_ as argument.
> If your current working directory is /service D</etc/service> this makes no
> difference. There are more tools in daemontools; see [2] or [8]
>
>
>
> III.1 Starting
> Whenever you put a service into the service directory, whether by creating a
> directory or linking into it, the supervisor will try to start it by running
> its ‘run’ script. Hence, using the previously described method will already
> have the service(s) started.
>   Should a service have been stopped in any way, use
>
>   # svc -u /service/gs_yourservice
> D<# svc -u /etc/service/gs_yourservice>
>
> to start it.
>
>
>
> III.2 Restarting
> Daemontools primary aim is to keep its services running. Thus, simply killing
> your service by using
>
>   # kill PIDOFYOURSERVICE
>   # killall NAMEOFYOURSERVICE
>
> would not stop your service completely but rather kill the application and
> have it restarted by the supervisor. The svc tool can be used to do this, too:
>
>   # svc -t /service/gs_yourservice
> D<# svc -t /etc/service/gs_yourservice>
>
> to terminate (signal(7) number 15, similar to kill),
>
>   # svc -k /service/gs_yourservice
> D<# svc -k /etc/service/gs_yourservice>
>
> to kill (signal(7) number 9, similar to kill -9), or
>
>   # svc -h /service/gs_yourservice
> D<# svc -h /etc/service/gs_yourservice>
>
> to send hangup (signal(7) number 1) to the service.
>
>
>
> III.3 Stopping
> As mentioned above, terminating/killing a service does not keep it shut down.
> To keep your service shut down, use
>
>   # svc -d /service/gs_yourservice
> D<# svc -d /etc/service/gs_yourservice>
>
> When you issued this command, you might want to use svc -u as in III.1 to
> bring it up again.
>
>
>
> III.4 Status
> In order to see whether the services are running you might issue
>
>   # svstat /service/gs_yourservice
> D<# svstat /etc/service/gs_yourservice>
>
> and it will print the status of the service. This is really useful if you do
> this in the service directory /service D</etc/service>:
>
>   # svstat *
>
> as this provides an overview of all services. A healthy system might look like
>
>   # svstat *
>   gs_fastcgi-1: up (pid 8945) 1094 seconds
>   gs_fastcgi-2: up (pid 31925) 14185 seconds
>   gs_fastcgi-3: up (pid 31929) 14185 seconds
>   gs_maintainance: up (pid 31909) 14199 seconds
>
> but with several services not running, this might look like
>
>   # svstat *
>   gs_fastcgi-1: up (pid 8945) 1385 seconds
>   gs_fastcgi-2: down 6 seconds, normally up
>   gs_fastcgi-3: up (pid 31929) 14476 seconds
>   gs_maintainance: down 3 seconds, normally up
>
>
>
> III.5 Helpers
> When using the Debian daemontools-run package, there is an additional command
> that should help you enabling or disabling services from your service template
> directory. That way, we might have enabled the maintenance service by issuing
>
>   # update-service --add /etc/sv/gs_maintenance
>
> (where we could have specified an optional name for the link in the
> /etc/service directory). Similar, the following command will remove the en-
> abled service
>
>   # update-service --remove /etc/sv/gs_maintenance
>
> We could also list all services currently enabled:
>
>   # update-service --list
>   gs_fastcgi-1
>   gs_fastcgi-2
>   gs_fastcgi-3
>   gs_maintainance
>
> There is a Debian package called svtools that provides various tools for
> •   managing/viewing multilog files (mlcat, mltail, mltac…)
> •   creating services and init.d wrappers (svsetup, svinid-create)
> •   various information (svdir, svinfo)
> This package is also responsible for the creation of the /etc/sv directory.
>
>
>
>
>
>
>                                IV IMPROVEMENTS.
> IV.1 Restrictions.
> Now for the restriction mentioned initially. The runSeasideGems command no
> longer works. It is responsible for starting/stopping/restarting the seaside
> gems. Also, it is used from within the GemTools to perform these very tasks.
> However, the techniques of the script interfere with the ones of daemontools.
> Thus, I put an
>
>     exit 0;
>
> near the top of $GEMSTONE/seaside/bin/runSeasideGems. Unfortunately, it is not
> possible to start/stop/restart the gems from within the GemTools with this
> patch.
>   Another possibility is to change the script to use the svc tool. Thus, it
> would be possible to use the GemTools for the start/stop/restarting, again.
>
>
>
> IV.2 Daemontools Everywhere.
> It is possible to use the daemontools to control the stone itself rather than
> only the gems. However, you should be aware, that you have to
> •   deal with the existing init.d scripts,
> •   take care of the start/stop order of the netldi and the Stone.
>
>
>
>                                                             That’s all, folks.
>
>
>
> [1] Well, almost, the limitation is described later.
> [2] http://cr.yp.to/daemontools.html
> [3] I will focus on Linux here, with special respect to Debian. I installed
>     daemontools, daemontools-run, and svtools.
> [4] Meta, isn’t it?
> [5] Note that this is absolutely optional. The default GS/SS logging works
>     when using daemontools, too.
> [6] I.e. when using daemontools to stop a service, the shell script is termina-
>     ted and not the children. This actually happened to me.
> [7] Note that this directory is provided by default only by the Debian
>     daemontools-run package. I will use /service_tmpl/ here for the non-Debian
>     case.
> [8] supervise(8), svok(8), svstat(8), svscanboot(8), svscan(8),
>     readproctitle(8), fghack(8), pgrphack(8), multilog(8), tai64n(8),
>     tai64nlocal(8), setuidgid(8), envuidgid(8), envdir(8), softlimit(8),
>     setlock(8)
>

Reply | Threaded
Open this post in threaded view
|

Re: [Tut] GemStone/Seaside and Daemontools

Monty Williams-3
Thanks indeed, Tobias,

I scripted your instructions and made the script work for either Seaside 2.8 or Seaside 3.0. The script also creates /etc/init.d scripts and links them into /etc/rc<n>.d directories to automatically start GemStone, the Seaside gems and maintenance gem when you bring the system up.

You can grab them from: http://github.com/Monty/GemStone_daemontools_setup

The basics: (assuming you already have a system that will run GemStone/Seaside, lighttpd or alternative, etc. and you have root access)

1) Install daemontools
    sudo apt-get install daemontools daemontools-run svtools

2) Grab the setup code:
    git clone git://github.com/Monty/GemStone_daemontools_setup.git

3) cd GemStone_daemontools_setup
    edit the "config" file (only if you need to customize the setup)

4) sudo setupAll.sh
    (it will allow you to skip any step)

5) sudo reboot
    or
    sudo /etc/init.d/netldi start; sudo /etc/init.d/gemstone start; sudo /etc/init.d/topaz start

6) check status of gems being run by daemontools
    sudo /etc/init.d/topaz status

Please fork it on github and enhance it. We'll grab your improvements.

-- Monty

On 05/20/2010 11:58 AM, Dale Henrichs wrote:
Tobias,

Thanks very much for this excellent writeup. I've copied your writeup
onto the glass db site along with edits to make the text wiki friendly:

   http://code.google.com/p/glassdb/wiki/GLASSDaemonTools

Dale

Tobias Pape wrote:
Hi —,

some time ago, there was a discussion on the list about using daemontools for
the serving gems of GS/SS. After a brief discussion with Dale, I gave it a try
an—Io and behold—it’s working like a charm.[1]
  Now, what is to be done? Read the small tutorial appended.

So Long,
        -Tobias




                               GemStone/Seaside
                                      and
                                  Daemontools



                                  I PRINCIPLES.
I.1 Intro
(If you’re familiar with daemontools, skip to II)
Daemontools by D.J.Bernstein[2] is a set of tools for managing services or pro-
grams that normally have a non-forking behaviour. The tools then take care of
concerns like logging (given log information is printed to the terminal, by de-
fault), re-starting of dead processes &c. As per the intention of the author,
the daemontools installation and usage is often dissimilar to normal Unix
/Linux or even MacOS.[3]
  The daemontools work by the notion of a ‘service’ that, essentially, is a
program in the aforementioned fashion. Each such service has a directory in a
well-known place in the filesystem where a ‘supervisor’ is able to find it.
From this moment on, the daemontools can be used to control the service; i.e.
shutting down, starting or restarting the service. The key point, in fact, is
that the supervisor takes care of restarting the service whenever it dies.
This is my motivation of taking the gems to the daemontools.



I.2 Installations
Following the guides on the web site of D.J.Bernstein is usually the best idea
to install the daemontools on any Unix/Linux. However, Debian provides some
packages for installation with APT. These packages introduce some differences
to the original installation. Debian specifics are marked D< > in the follow-
ing.



I.3 Directories
As said before, the daemontools, expect services to have a certain directory
in a well-known place. By default, this place would be /service
D<
/etc/service
. Every directory beneath this place is checked regularly by the supervisor
for new services. If the supervisor finds a file called ‘run’ that is execut-
able, this directory is treated as a service. N.B. it seems to be a common pat-
tern to use only symlinks in the supervisors directory in order to maintain a
separate repository of services and/or service templates. D<There even is a
default directory, /etc/sv to put the original services in. See III.5> We will
use this in II.3.1.



I.4 Logging.
If a service directory contains a directory called ‘log’ that looks like a ser-
vice directory, this is treated as the services logging service.[4] Hence,
there has to be a ‘run’ file to make that working. Every output of the ‘main’
service (stdout) is then piped to the logging services input (stdin).






                                  II SCRIPTS.
II.1 Patching
In order to behave as the daemontools expect, the scripts for the GS/SS gems,
i.e. the maintenance gem and the actual seaside gem, have to be patched, espe-
cially for logging handled by the daemontools.[5] For the seaside gem, I will
only give the patches for the FastCGI version, but the Hyper version should be
similar.



II.1.1 ‘nohup’ and ‘exec’
Currently, GS/SS tries to avoid unexpected hangups by using the nohup(1) tool.
However, this conflicts with daemontools which are managing hangups themselves.
As I said above, the daemontools expect a foreground program, thus, the cur-
rently used backgrounding  must be removed.
  Additionally, daemontools require the supervised service to be a singe pro-
cess. That is, a shell script that has child processes would not work correct-
ly[6]. Unfortunately, that is exactly how the GS/SS gems work. To overcome
this, we use the exec(1) command which will replace the shell with the started
child.
  Now the patching. For the maintenance gem that resides in $GEMSTONE/seaside/
bin/startMaintenance, the line

    cat << EOF | nohup $GEMSTONE/bin/topaz -l -T200000 2>&1 > \
    $GEMSTONE_LOGDIR/maintenance_gem.log &

should become

    exec $GEMSTONE/bin/topaz -l -T200000 2>&1 > \
    $GEMSTONE_LOGDIR/maintenance_gem.log << EOF

and for the seaside gem in $GEMSTONE/seaside/bin/seasideGem_FastCGI, change

    cat << EOF | nohup $GEMSTONE/bin/topaz -l -T50000 2>&1 \
    >> $GEMSTONE_LOGDIR/FastCGI_server-${1}.log &

for

    exec $GEMSTONE/bin/topaz -l -T50000 2>&1 \
    >> $GEMSTONE_LOGDIR/FastCGI_server-${1}.log  << EOF

I did not alter the original files but named them differently,
‘maintenanceGem’ and ‘seasideGem_FastCGI’, respectively; and put them besides
their originals. Note that these new scripts will
•       not go into background when started and
•       replace the shell they are started in
which was exactly the intention.



II.1.2 Logging
To provide the output to the logging service, the output of the gem should be
simply printed and not be redirected to a file. Hence, for the maintenance gem,
change the line from II.1.1

    exec $GEMSTONE/bin/topaz -l -T200000 2>&1 > \
    $GEMSTONE_LOGDIR/maintenance_gem.log << EOF

to

    exec $GEMSTONE/bin/topaz -l -T200000 2>&1  << EOF

and for the seaside gem from

    exec $GEMSTONE/bin/topaz -l -T50000 2>&1 \
    >> $GEMSTONE_LOGDIR/FastCGI_server-${1}.log  << EOF

to

    exec $GEMSTONE/bin/topaz -l -T50000 2>&1 << EOF

Thus, all requirements are met.



II.2 The ‘run’ Scripts
Essentially, all run scripts to follow are a variant of ‘run that gem,’ more
or less easy to configure. Thus, the simplest seaside gem run script would be

-->8--
source /opt/gemstone/product/seaside/defSeaside
exec $GEMSTONE/seaside/bin/seasideGem_FastCGI 9001

--8<--

However, there is a security problem, as this script is executed as root and,
subsequently, the gem is, too. To overcome this problem, daemontools provide
the setuidgid(8) tool which executes a command under a given user. Hence, the
improved version:

-->8--
source /opt/gemstone/product/seaside/defSeaside
exec setuidgid gemstone $GEMSTONE/seaside/bin/seasideGem_FastCGI 9001

--8<--

Now, let’s put it together: We create the directory

    /service/gs_fastcgi         D</etc/service/gs_fastcgi>

and put the run script there. Also, we create the directory

    /service/gs_maintenance     D</etc/serivce/gs_maintenance>

and put the run script there, replacing the actual command from
‘seasideGem_FastCGI 9001’ to ‘maintenanceGem’. For simple setups, this should
suffice, skip to III. For more complex setups, read on.



II.3 Multiple Seaside Gems
When we want to run multiple seaside gems in parallel, it is desirable to not
have to write the run script for every gem and just change the port. When we
decide to change something, we don’t want to touch every run file; this is er-
ror prone.



II.3.1 New Directory Layout
  As said in I.3, it might be useful to have a template for various services.
We use the service template directory[7] and create the service ‘gs_fastcgi.’
However, unlike the setup in II.2, this service is never actually run. Instead,
it serves as template for out, say three, actual services. For each actual
service, we will have a directory in the service template directory, too. In
accordance with the Debian way, we will create links in the service directory
to enable such service (see III. for a Debian-provided tool assisting this pro-
cedure). Hence, when finished, the layout of our service template directory
should be:

D<# ls -l /etc/sv >
  # ls -l /service_tmpl
  total 0
  drwxr-xr-x 3 root root 4096  13:52 gs_fastcgi
  drwxr-xr-x 3 root root 4096  14:07 gs_fastcgi-1
  drwxr-xr-x 3 root root 4096  14:13 gs_fastcgi-2
  drwxr-xr-x 3 root root 4096  14:13 gs_fastcgi-3
  drwxr-xr-x 3 root root 4096  14:17 gs_maintainance

and of the service directory:

D<
  # ls -l /etc/service
  total 0
  lrwxrwxrwx 1 root root 20  14:07 gs_fastcgi-1 -> /etc/sv/gs_fastcgi-1
  lrwxrwxrwx 1 root root 20  14:11 gs_fastcgi-2 -> /etc/sv/gs_fastcgi-2
  lrwxrwxrwx 1 root root 20  14:11 gs_fastcgi-3 -> /etc/sv/gs_fastcgi-3
  lrwxrwxrwx 1 root root 23  14:17 gs_maintainance -> /etc/sv/gs_maintainance
  # ls -l /service
  total 0
  lrwxrwxrwx 1 root root 20  14:07 gs_fastcgi-1 -> /service_tmpl/gs_fastcgi-1
  lrwxrwxrwx 1 root root 20  14:11 gs_fastcgi-2 -> /service_tmpl/gs_fastcgi-2
  lrwxrwxrwx 1 root root 20  14:11 gs_fastcgi-3 -> /service_tmpl/gs_fastcgi-3
  lrwxrwxrwx 1 root root 23  14:17 gs_maintainance
    -> /service_tmpl/gs_maintainance

Note that the gs_fastcgi service itself is not linked into the service direc-
tory.



II.3.2 Common and Local Configuration
I decided to differentiate between a common configuration shared between all
services (such as the user that should run the gem) and ‘local’ configuration
that is unique to a certain service (such as the gem’s port). To enable this,
we will use a small shell trick. As the actual run script will reside in
D</etc/sv/gs_fastcgi> /service_tmpl/gs_fastcgi, we will put the common configu-
ration there. The local configuration will reside in the actual service’s temp-
late directory. This looks like this:

D<
  # ls -l /etc/sv/*fastcgi*/config*
  -rw-r--r-- 1 root root 32  14:07 /etc/sv/gs_fastcgi-1/config.local
  -rw-r--r-- 1 root root 32  14:10 /etc/sv/gs_fastcgi-2/config.local
  -rw-r--r-- 1 root root 32  14:10 /etc/sv/gs_fastcgi-3/config.local
  -rw-r--r-- 1 root root 86  13:42 /etc/sv/gs_fastcgi/config
 >
  # ls -l /service_tmpl/*fastcgi*/config*
  -rw-r--r-- 1 root root 32  14:07 /service_tmpl/gs_fastcgi-1/config.local
  -rw-r--r-- 1 root root 32  14:10 /service_tmpl/gs_fastcgi-2/config.local
  -rw-r--r-- 1 root root 32  14:10 /service_tmpl/gs_fastcgi-3/config.local
  -rw-r--r-- 1 root root 86  13:42 /service_tmpl/gs_fastcgi/config

What should go into the configuration

[config] As stated above, the common configuration can include the user that
    should run the gems. Also, I found it valuable to put the standard GS/SS
    stanza there that sets up the GS/SS shell environment:

    -->8--
    #!/bin/sh

    source /opt/gemstone/product/seaside/defSeaside

    export RUNASUSER=gemstone

    --8<--
[config.local] The only thing that is needed locally by each individual gem is
    its port, hence the local configuration becomes (for gs_fastcgi-1)

    -->8--
    #!/bin/sh

    export GEM_PORT=9001

    --8<--
    (with other port numbers for the other services)



II.3.3 Adapting the ‘run’ Script
In order to make the run script reusable, we introduced configurations. To ac-
tually use these configuration the run script must be adapted:

-->8--
#!/bin/sh

source /opt/gemstone/product/seaside/defSeaside
exec setuidgid $RUNASUSER $GEMSTONE/seaside/bin/seasideGem_FastCGI $GEM_PORT

--8<--

However, to access the configuration variables, we have to include the config-
gurations. Not the facts that
•   the common configuration is in the directory the run script is actually
    residing in;
•   the local configuration in in the directory the run script is linked to.
Knowing this, a small shell trick that provides both directories mentioned
enables us to include both configurations. E.g. for the gs_fastcgi-1 service
in the following script (the run script I currently use) ‘scriptpath’ would be
/service_tmpl/gs_fastcgi-1 D</etc/sv/gs_fastcgi-1> whereas ‘orig_scriptpath’
would be /service_tmpl/gs_fastcgi D</etc/sv/gs_fastcgi>.

-->8--
#!/bin/sh

old_pwd=$(pwd)
cd $(dirname $0)
scriptpath=$(pwd)
cd $old_pwd

orig_scriptpath=$(dirname "$(readlink -f $0)")

source $orig_scriptpath/config

[ -r $scriptpath/config.local ] && source $scriptpath/config.local


exec setuidgid $RUNASUSER $GEMSTONE/seaside/bin/seasideGem_FastCGI $GEM_PORT

--8<--

This way, both the common and the local configurations are included. It is
possible to use this script for the maintenance gem, too (adapted as descibed
above)



II.4 The Logging Scripts
For all services, the logging ‘run’ script simply is
-->8--
#!/bin/sh
exec multilog t ./main

--8<--

When this ‘run’ script is put into the log directory of a service and made exe-
cutable, the daemontools automatically will start use this logging facility.
Note that the ‘multilog’ command provided by daemontools uses timestamps with
a special format (cf. tai64n(8)). To read multilog output more human-readable
refer to the ml* tools (cf. III.5).
  Note that if we are using the Multiple Gems layout from II.3, there is
no need to copy that script over and over. Simply put it into D</etc/sv/
gs_fastcgi/log/run> /service_tmpl/gs_fastcgi/log/run and link to it from the
different actual service:

D<
  # ls -l /etc/sv/*/log/run
  lrwxrwxrwx 1 root root 26  13:55 /etc/sv/gs_fastcgi-1/log/run
    -> /etc/sv/gs_fastcgi/log/run
  lrwxrwxrwx 1 root root 26  13:55 /etc/sv/gs_fastcgi-2/log/run
    -> /etc/sv/gs_fastcgi/log/run
  lrwxrwxrwx 1 root root 26  13:55 /etc/sv/gs_fastcgi-3/log/run
    -> /etc/sv/gs_fastcgi/log/run
  -rwxr-xr-x 1 root root 33  14:02 /etc/sv/gs_fastcgi/log/run
  lrwxrwxrwx 1 root root 26  13:55 /etc/sv/gs_maintainance/log/run
    -> /etc/sv/gs_fastcgi/log/run
  # ls -l /etc/sv/*/log/run
  lrwxrwxrwx 1 root root 26  13:55 /etc/sv/gs_fastcgi-1/log/run
    -> /etc/sv/gs_fastcgi/log/run
  lrwxrwxrwx 1 root root 26  13:55 /etc/sv/gs_fastcgi-2/log/run
    -> /etc/sv/gs_fastcgi/log/run
  lrwxrwxrwx 1 root root 26  13:55 /etc/sv/gs_fastcgi-3/log/run
    -> /etc/sv/gs_fastcgi/log/run
  -rwxr-xr-x 1 root root 33  14:02 /etc/sv/gs_fastcgi/log/run
  lrwxrwxrwx 1 root root 26  13:55 /etc/sv/gs_maintainance/log/run
    -> /etc/sv/gs_fastcgi/log/run






                                III COMMANDS.
III.0
Most of the commands explained here use the daemontools svc tool. This tool
takes an _directory_ as argument. Later, some Debian specific tools will be
mentioned, e.g. mlcat or mltail. These tools take a _service name_ as argument.
If your current working directory is /service D</etc/service> this makes no
difference. There are more tools in daemontools; see [2] or [8]



III.1 Starting
Whenever you put a service into the service directory, whether by creating a
directory or linking into it, the supervisor will try to start it by running
its ‘run’ script. Hence, using the previously described method will already
have the service(s) started.
  Should a service have been stopped in any way, use

  # svc -u /service/gs_yourservice
D<# svc -u /etc/service/gs_yourservice>

to start it.



III.2 Restarting
Daemontools primary aim is to keep its services running. Thus, simply killing
your service by using

  # kill PIDOFYOURSERVICE
  # killall NAMEOFYOURSERVICE

would not stop your service completely but rather kill the application and
have it restarted by the supervisor. The svc tool can be used to do this, too:

  # svc -t /service/gs_yourservice
D<# svc -t /etc/service/gs_yourservice>

to terminate (signal(7) number 15, similar to kill),

  # svc -k /service/gs_yourservice
D<# svc -k /etc/service/gs_yourservice>

to kill (signal(7) number 9, similar to kill -9), or

  # svc -h /service/gs_yourservice
D<# svc -h /etc/service/gs_yourservice>

to send hangup (signal(7) number 1) to the service.



III.3 Stopping
As mentioned above, terminating/killing a service does not keep it shut down.
To keep your service shut down, use

  # svc -d /service/gs_yourservice
D<# svc -d /etc/service/gs_yourservice>

When you issued this command, you might want to use svc -u as in III.1 to
bring it up again.



III.4 Status
In order to see whether the services are running you might issue

  # svstat /service/gs_yourservice
D<# svstat /etc/service/gs_yourservice>

and it will print the status of the service. This is really useful if you do
this in the service directory /service D</etc/service>:

  # svstat *

as this provides an overview of all services. A healthy system might look like

  # svstat *
  gs_fastcgi-1: up (pid 8945) 1094 seconds
  gs_fastcgi-2: up (pid 31925) 14185 seconds
  gs_fastcgi-3: up (pid 31929) 14185 seconds
  gs_maintainance: up (pid 31909) 14199 seconds

but with several services not running, this might look like

  # svstat *
  gs_fastcgi-1: up (pid 8945) 1385 seconds
  gs_fastcgi-2: down 6 seconds, normally up
  gs_fastcgi-3: up (pid 31929) 14476 seconds
  gs_maintainance: down 3 seconds, normally up



III.5 Helpers
When using the Debian daemontools-run package, there is an additional command
that should help you enabling or disabling services from your service template
directory. That way, we might have enabled the maintenance service by issuing

  # update-service --add /etc/sv/gs_maintenance

(where we could have specified an optional name for the link in the
/etc/service directory). Similar, the following command will remove the en-
abled service

  # update-service --remove /etc/sv/gs_maintenance

We could also list all services currently enabled:

  # update-service --list
  gs_fastcgi-1
  gs_fastcgi-2
  gs_fastcgi-3
  gs_maintainance

There is a Debian package called svtools that provides various tools for
•   managing/viewing multilog files (mlcat, mltail, mltac…)
•   creating services and init.d wrappers (svsetup, svinid-create)
•   various information (svdir, svinfo)
This package is also responsible for the creation of the /etc/sv directory.






                               IV IMPROVEMENTS.
IV.1 Restrictions.
Now for the restriction mentioned initially. The runSeasideGems command no
longer works. It is responsible for starting/stopping/restarting the seaside
gems. Also, it is used from within the GemTools to perform these very tasks.
However, the techniques of the script interfere with the ones of daemontools.
Thus, I put an

    exit 0;

near the top of $GEMSTONE/seaside/bin/runSeasideGems. Unfortunately, it is not
possible to start/stop/restart the gems from within the GemTools with this
patch.
  Another possibility is to change the script to use the svc tool. Thus, it
would be possible to use the GemTools for the start/stop/restarting, again.



IV.2 Daemontools Everywhere.
It is possible to use the daemontools to control the stone itself rather than
only the gems. However, you should be aware, that you have to
•   deal with the existing init.d scripts,
•   take care of the start/stop order of the netldi and the Stone.



                                                            That’s all, folks.



[1] Well, almost, the limitation is described later.
[2] http://cr.yp.to/daemontools.html
[3] I will focus on Linux here, with special respect to Debian. I installed
    daemontools, daemontools-run, and svtools.
[4] Meta, isn’t it?
[5] Note that this is absolutely optional. The default GS/SS logging works
    when using daemontools, too.
[6] I.e. when using daemontools to stop a service, the shell script is termina-
    ted and not the children. This actually happened to me.
[7] Note that this directory is provided by default only by the Debian
    daemontools-run package. I will use /service_tmpl/ here for the non-Debian
    case.
[8] supervise(8), svok(8), svstat(8), svscanboot(8), svscan(8),
    readproctitle(8), fghack(8), pgrphack(8), multilog(8), tai64n(8),
    tai64nlocal(8), setuidgid(8), envuidgid(8), envdir(8), softlimit(8),
    setlock(8)


    


-- 
Monty Williams
Director, Product Marketing, R&D
VMware
+1 503 533-3506 (office)
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: [Tut] GemStone/Seaside and Daemontools

Monty Williams-3
  I updated my scripts that setup GemStone/Seaside to run under
daemontools. Now they run two statmons
with a 1-second and a 60-second sampling interval:
http://github.com/Monty/GemStone_daemontools_setup

I'm using these scripts to build a new GLASS appliance that I expect to
publish next week.
The scripts make it easy to build your own custom appliance if you
already have a 2.4.4.1
based GemStone system.

-- Monty