VO-box HowTo - description, installation, testing

General description

The VO-box is a type of node where experiments can run specific agents and services to provide a reliable mechanism to accomplish various tasks. It is provided as an interim solution in order to allow experiments to provide their own services whenever the middleware still does not provide the required functionality. The access to the VO-box (or VO node) is restricted to the Software Group Manager (SGM) of the Virtual Organisation (VO).
There is not a proper description for the general requirements for a VO-box as each experiment tailored its own specific requirements. ALICE maintains its info on the AliEn twiki.

The rest of the data on this page dates from 2005!(See the LCG Service Challenge Wiki). However:

OS: Scientific Linux 3 (usually);
hardware: WN-type host + experiment specific requirements;
outbound connectivity - as for a UI-type node;
inbound connectivity - as for a CE-type node, plus additional requirements from the experiments i.e. from specific sources (local Worker Nodes, CERN, World) on specific fixed network ports;
access to local user accounts via ssh/gsissh depending on experiments;
a writable directory shared among the worker nodes to install application software (disk space size depending on experiment);
service certificates will be used for secure operations.

The VO-box consists in the following:
- a GSISSH server (running by default on port 1975) which allows ssh connection authorized through X509 proxies and proxy delegation;
- a GRIS (registering to the site GIIS) which publishes the GSISSH service and port;
- a Proxy Renewal Service (together with a user level tool) to ensure automatic refresh of user credentials;
- a GSSKLOG client to grant Kerberos tokens from X509 proxies; this is necessary only if the shared area of the experiment is under AFS and, also, it implies the existence of a GSI->KERBEROS mapping service at the site;
- the WP1 CLIs and APIs which allow job submission to the local CE, even if this is strongly discouraged and should be used only for very specific jobs, not for regular job submission;
- in addition, the software installation area is mounted and accessible in the filesystem.
Experiments require a dedicated VO node to be set up on each site and each experiment usually has its own requirements for its VO-box (again, see the LCG Service Challenge Wiki).

Installation

The LCG Generic Installation Guide provides full information regarding LCG middleware installation and configuration on the various LCG node types including VO-box. This chapter will provide details on a specific VO-box installation.

RPM Installation

The VO-box installation described here used a kickstart procedure to install Scientific Linux 3. Java SDK 1.4.2 (in RPM format - i.e. from this YUM repository at RAL) has to be installed on the system before installing the middleware (see the LCG 2 Installation Manual). Also the lcg-CA, lcg-yaim and lcg-VOBOX .rpm packages have to be installed (from this YUM repository or this one).

Note1: At the moment, lcg-VOBOX package creates the dependencies for and install the lcg-vobox-1.0.0-5.noarch.rpm package. Because of a script error within the latter .rpm, a new package was released and lcg-vobox has to be updated. The following command can be run (as root) on the VO-box:

rpm -Uvh /afs/cern.ch/project/gd/RpmDir_i386-sl3/external/lcg-vobox-1.0.1-3.noarch.rpm

In order to accomplish the above request, it might be worth to previously have installed (maybe included within the kickstart) the support for the AFS distributed filesystem (openafs, openafs-client, openafs-client and kernel-module-openafs RPMs - i.e. from the RAL YUM repository)

Also, the AFS environment installation could be useful if Software Installation area is under AFS i.e. under /afs directory (details in "Accessing the software installation area").

Note2: The new update can also be found at this link.

YAIM Installation

The lcg-VOBOX metapackage can be installed using YAIM as well. Before you proceed further make sure that Java SDK 1.4.2 (in RPM format) is installed on the node. The YAIM package has to be already installed on the node, as well as the 'apt' RPM package (RAL YUM Repository). The <site-configuration-file> needs a proper configuration of the following variables i.e.

LCG_REPOSITORY="'rpm http://linuxsoft.cern.ch LCG/apt/LCG-2_6_0/sl3/en/i386 lcg_sl3 lcg_sl3.updates' 'rpm http://grid-deployment.web.cern.ch/grid-deployment/gis apt/LCG-2_6_0/sl3/en/i386 lcg_sl3 lcg_sl3.updates'" 

CA_REPOSITORY="rpm http://grid-deployment.web.cern.ch/grid-deployment/gis apt/LCG_CA/en/i386 lcg"

in order to perform the Middleware and CA installation.

Then run the command: /opt/lcg/yaim/scripts/install_node /opt/lcg/yaim/etc/site_info.def lcg-VOBOX

Host Certificates

Valid host certificate/key files (see LCG Installation Manual) need to be obtained from a Certification Authority. Once they are available (hostcert.pem - the machine public key and hostkey.pem - the machine private key), they should be copied on the VO-box in the /etc/grid-security directory (hostkey.pem - only readable by root and hostcert.pem - readable by everybody).

Also the installation must ensure that the experiment software installation area is accessible (i.e. mounted) from the VOBOX.

Configuration

The Site Configuration file

Prior to start to configure the middleware packages that have been installed on the VO node, the Site Configuration file (/opt/lcg/yaim/etc/site-info.def) has to be checked, particularly for some environment variables:

VOBOX_HOST - the VO-box hostname (i.e. VOBOX_HOST=lcgvo0337.gridpp.rl.ac.uk)

PX_HOST - has to be set to the CERN myproxy server (i.e. PX_HOST=myproxy.cern.ch)

JAVA_LOCATION - path to Java VM installation (i.e. JAVA_LOCATION="/usr/java/j2sdk1.4.2_08").

Even if there is a private myproxy server for a site, the PX_HOST variable should point to the CERN one (myproxy.cern.ch)

Special attention should be taken regarding the experiments that are going to be supported by the VO-box (only one or more experiments). Usually, the default Site Configuration File is a comprehensive one i.e. it contains proper settings for all the experiments. The VOS and QUEUES variables have to be set accordingly with the supported experiment(s) and VO_<vo_name>_{SW_DIR, DEFAULT_SE, SGM, USERS, STORAGE_DIR, QUEUES} entries in the site-info.def file have to be appropriately enabled or disabled. A good idea would be to have always 'dteam' configured as a supported VO, this would help to make tests.

After the configuration of the site_info.def file is done, the following command has to be run to configure the VO:

/opt/lcg/yaim/scripts/configure_node /opt/lcg/yaim/etc/site-info.def VOBOX

The output of the above command will show some error messages during the Globus initialization scripts, but this is perfectly fine for the time being. A successful configuration should terminate as below:

... 
 Process  does not exist ...                               [FAILED]
 Starting ProxyRenewal Daemon: vobox-renewd                [  OK  ]
Configuration Complete
[root@lcgvo0337 etc]#

The site administrator must communicate the name and the DN of the VOBOX to the myproxy.cern.ch service administrator (email both hep-project-grid-cern-testbed-managers@cern.ch and support-eis@cern.ch) so that it is included in the list of authorized renewers. If this is not done, the renewal agent of the VOBOX will not work. (see LCG Installation Manual)

Firewall configuration

The site administrator has to ensure a proper firewall configuration for the VO-box. Thus, port 1975/tcp has to be open locally for inbound connectivity (the GSISSH server runs by default on port 1975). Other ports to be open for inbound connectivity are ntpd (port 123/udp from other ntp servers on 123/udp) and those specified for a CE, except that in the VO-box there is no service running on the GRAM port (see the LCG Ports Table document). Thus the VO-box needs to be accessed through GridFTP (2811/tcp), LDAP (389/tcp) and Globus tcp port range (see below).

The ports for outbound connectivity are the ones of a UI-node, basically the Globus tcp port range for the Globus callbacks (see /etc/sysconfig/globus file, usually 50000 to 52000/tcp).

In addition there are other specific requirements regarding firewall settings that come from individual VOs (see LCG Service Challenge Wiki).

Please note that the same set of inbound and outbound firewall rules might be needed to be open on the site main firewall having as destination or source the VO-box hostname (or IP).

Note: The entire firewall issue is currently under discussion as a security matter regarding the VO_box future. This chapter will be updated accordingly.

Testing the VO-box

Accessing the VOBOX (lcgvo0337.gridpp.rl.ac.uk)

Login as root on the VO node and change the file /etc/grid-security/grid-mapfile manually to map your DN to the SGM of your VO (i.e. "/C=UK/O=eScience/OU=CLRC/L=RAL/CN=catalin condurache" dteamsgm). This is just for doing some tests, as the above file will be overwritten every six hours. A long standing solution would be to add the mapping between the DN and SGM of the VO to /opt/edg/etc/grid-mapfile-local file and then execute
```
/opt/edg/sbin/edg-mkgridmap --output /etc/grid-security/grid-mapfile --safe
```
(as in /etc/cron.d/edg-mkgridmap cron file). Without any of the above mentioned actions you will not be granted the access to the VO-box.
In order to access the VO-box one needs a GSISSH client. This is installed automatically in every UI or WN with LCG 2.6.0 (i.e. lcgui02.gridpp.rl.ac.uk). Otherwise one can use a machine with SL3 and an AFS client installed (i.e. csfa.rl.ac.uk) and source the grid_env.sh (or grid_env.csh) file in order to set the User Interface environment.
```
[csfa] /home/csf/catalin > source /afs/cern.ch/project/gd/LCG-share/2.6.0/sl3/etc/profile.d/grid_env.sh
```

Obtain the necessary credentials. If you only want to access the VO-box and run some commands, you need a proxy.

[lcgui01] /home/csf/catalin > grid-proxy-init
Your identity: /C=UK/O=eScience/OU=CLRC/L=RAL/CN=catalin condurache
Enter GRID pass phrase for this identity:
Creating proxy ........................................... Done
Your proxy is valid until: Fri Sep 16 23:53:45 2005

GSISSHD test You can connect to the VO-box using the GSISSH client and you will be mapped to the SGM of your VO.

[lcgui01] /home/csf/catalin > gsissh -p 1975 lcgvo0337.gridpp.rl.ac.uk
[dteamsgm@lcgvo0337 dteamsgm]$ ls -al /tmp
total 72
drwxrwxrwt    7 root     root         4096 Sep 16 13:23 .
drwxr-xr-x   23 root     root         4096 Sep 15 12:45 ..
drwxr-xr-x    2 root     root         4096 Sep 15 12:46 fmon
drwxrwxrwt    2 xfs      xfs          4096 Sep 15 12:12 .font-unix
-rw-r--r--    1 root     root            7 Sep 15 12:45 grid-info-soft-register.pids.23127
drwxr-xr-x    2 rgma     rgma         4096 Sep 16 10:01 hsperfdata_rgma
drwxrwxrwt    2 root     root         4096 Sep 15 12:12 .ICE-unix
drwx------    2 root     root        16384 Sep 15 11:54 lost+found
-rw-r--r--    1 dteamsgm dteam         571 Sep 16 12:27 new_file.txt
-rw-r--r--    1 root     root          633 Sep 15 11:59 post.log
-rw-------    1 root     root         1024 Sep 15 11:59 .rnd
-rw-------    1 dteamsgm dteam           0 Sep 16 00:46 tmp.BBEJd19785
-rw-------    1 dteamsgm dteam           0 Sep 15 18:46 tmp.CmyzRl9005
-rw-------    1 dteamsgm dteam           0 Sep 16 03:46 tmp.gFLXZ25746
-rw-------    1 dteamsgm dteam           0 Sep 15 22:46 tmp.lhGcp16663
-rw-------    1 dteamsgm dteam           0 Sep 15 14:46 tmp.PoWYDz2718
-rw-------    1 dteamsgm dteam           0 Sep 16 01:46 tmp.srcxz21336
-rw-------    1 dteamsgm dteam           0 Sep 16 10:46 tmp.TrsBm12126
-rw-------    1 dteamsgm dteam           0 Sep 16 09:46 tmp.vdvsu10423
-rw-------    1 dteamsgm dteam           0 Sep 15 13:46 tmp.ZlbCW32015
-rw-r--r--    1 root     root         1863 Sep 15 12:06 unpack.log
-rw-------    1 dteamsgm dteam        5858 Sep 16 13:23 x509up_p16525.fileEZktlY.1
-rw-r--r--    1 root     root         1134 Sep 15 12:45 yaim.log

If you enabled the credential delegation in your GSISSH client, you will find a delegation of your proxy under /tmp and the env variable X509_USER_PROXY will point to it:

[dteamsgm@lcgvo0337 dteamsgm]$ echo $X509_USER_PROXY
/tmp/x509up_p14593.file2t0ogT.1

[dteamsgm@lcgvo0337 dteamsgm]$ grid-proxy-info
subject  : /C=UK/O=eScience/OU=CLRC/L=RAL/CN=catalin condurache/CN=proxy/CN=proxy
issuer   : /C=UK/O=eScience/OU=CLRC/L=RAL/CN=catalin condurache/CN=proxy
identity : /C=UK/O=eScience/OU=CLRC/L=RAL/CN=catalin condurache
type     : full legacy globus proxy
strength : 512 bits
path     : /tmp/x509up_p14593.file2t0ogT.1
timeleft : 11:31:14

You can also run commands from remote and take advantage of the proxy delegation:

[lcgui01] /home/csf/catalin > grid-proxy-info -subject
/C=UK/O=eScience/OU=CLRC/L=RAL/CN=catalin condurache/CN=proxy
[lcgui01] /home/csf/catalin > gsissh -p 1975 lcgvo0337.gridpp.rl.ac.uk "grid-proxy-info -subject"
/C=UK/O=eScience/OU=CLRC/L=RAL/CN=catalin condurache/CN=proxy/CN=proxy
[lcgui01] /home/csf/catalin >

You can also copy files in and out with 'gsiscp':

[lcgui01] /home/csf/catalin > gsiscp -P 1975 /etc/group lcgvo0337.gridpp.rl.ac.uk:/tmp/new_file.txt
group                                         100%  571     0.6KB/s   00:00
[lcgui01] /home/csf/catalin > gsissh -p 1975 lcgvo0337.gridpp.rl.ac.uk "ls -l /tmp/new_file.txt"
-rw-r--r--    1 dteamsgm dteam         571 Sep 16 12:27 /tmp/new_file.txt
[lcgui01] /home/csf/catalin >

GridFTP Test

[lcgui01] /home/csf/catalin > edg-gridftp-ls gsiftp://lcgvo0337.gridpp.rl.ac.uk/tmp
new_file.txt
tmp.TrsBm12126
hsperfdata_rgma
tmp.vdvsu10423
fmon
tmp.gFLXZ25746
tmp.srcxz21336
tmp.BBEJd19785
tmp.lhGcp16663
tmp.CmyzRl9005
tmp.PoWYDz2718
tmp.ZlbCW32015
grid-info-soft-register.pids.23127
yaim.log
unpack.log
post.log
lost+found

Proxy renewal test If you want to register a proxy for automatic renewal in the VO-box, then you must upload a proxy in the myproxy server (it should be myproxy.cern.ch)

[lcgui01] /home/csf/catalin > myproxy-init -s myproxy.cern.ch -d -n
Your identity: /C=UK/O=eScience/OU=CLRC/L=RAL/CN=catalin condurache
Enter GRID pass phrase for this identity:
Creating proxy ................................................ Done
Proxy Verify OK
Your proxy is valid until: Fri Sep 23 13:06:41 2005
A proxy valid for 168 hours (7.0 days) for user /C=UK/O=eScience/OU=CLRC/L=RAL/CN=catalin condurache now exists on myproxy.cern.ch.

The -n option is important in order to register a proxy without a passphrase. You can register, unregister, update information and query the status of registered proxies, using the vobox-proxy tool:

[dteamsgm@lcgvo0337 dteamsgm]$ /opt/lcg/bin/vobox-proxy --vo dteam --proxy-safe 10000 --myproxy-safe 20000 --email c.condurache@rl.ac.uk register
Registration was successful:

DN: /C=UK/O=eScience/OU=CLRC/L=RAL/CN=catalin condurache/CN=proxy/CN=proxy/CN=proxy
File: /opt/vobox/dteam/proxy_repository/+2fC+3dUK+2fO+3deScience+2fOU+3dCLRC+2fL+3dRAL+2fCN+3dcatalin+20condurache+2fCN+3dproxy
Proxy Expiration Trigger (seconds): 10000
Myproxy Expiration Trigger (seconds): 20000
Email Address: c.condurache@rl.ac.uk

A proxy has been registered for renewal. The service will notify the user c.condurache@rl.ac.uk if the proxy becomes shorter than 10000 seconds and the myproxy becomes shorter than 20000 seconds.

The next query provides various information concerning the proxy:

[dteamsgm@lcgvo0337 dteamsgm]$ /opt/lcg/bin/vobox-proxy --vo dteam query
*******************************************************************************
DN: /C=UK/O=eScience/OU=CLRC/L=RAL/CN=catalin condurache/CN=proxy/CN=proxy/CN=proxy
File: /opt/vobox/dteam/proxy_repository/+2fC+3dUK+2fO+3deScience+2fOU+3dCLRC+2fL+3dRAL+2fCN+3dcatalin+20condurache+2fCN+3dproxy
Proxy Expiration Trigger (seconds): 10000
Myproxy Expiration Trigger (seconds): 20000
Email Address: c.condurache@rl.ac.uk
Proxy Time Left (seconds): 42962
Myproxy Time Left (seconds): 515948
Status: OK
*******************************************************************************

Also the proxy can be unregistered (using the unregister option), or the trigger times and the email for notification can be changed (using the update option).

Installing services on the VO-box

This chapter contains instructions for VO Software Managers on how to install and run services on the VO-box. The way to run a service (vo-agent) in the VO-box is to put the startup agent in the directory /opt/vobox/<voname>/agents. Then one should create two symbolic links to this file, one under /opt/vobox/<voname>/start directory, the other under /opt/vobox/<voname>/stop directory.

ln -s /opt/vobox/dteam/agents/vo-agent /opt/vobox/dteam/start/S50-vo-agent
ln -s /opt/vobox/dteam/agents/vo-agent /opt/vobox/dteam/stop/K50-vo-agent

At startup time the renewal service will go through all the agents in /opt/vobox/<voname>/start/ directory and start them, i.e. it will run the command

/opt/vobox/dteam/start/S50-vo-agent start

At shutdown time the renewal service will go through all the agents in /opt/vobox/<voname>/stop/ directory and shut them down, i.e. it will run the command

/opt/vobox/dteam/start/K50-vo-agent stop

Note: The scripts are called in alphabetical order. Be sure you label correctly the links in case the order is important.

The script must be able to understand at least the 'start' and 'stop' options. A "Hello world" script which does nothing except discovering which proxy to use and setting the correct environment is reported below.

#!/bin/bash

export X509_USER_PROXY=`/opt/lcg/bin/vobox-proxy --vo dteam --dn \
"/C=UK/O=eScience/OU=CLRC/L=RAL/CN=catalin condurache" query-proxy-filename`

start() {
  echo "Script is starting with the proxy:" grid-proxy-info
}

stop() {
  echo "Script is stopping..."
}

case "$1" in
  start)
        start
                ;;
  stop)
        stop
                ;;
  *)
        echo "Usage: $0 {start|stop}"
                exit 1
esac

Accessing the software installation area

The software installation area is accessible from the VO-box. The location of such area is specified by the env variable VO_<VONAME>_SW_DIR.

It is the site manager responsibility to make sure this area is correctly mounted and accessible to the SGM. If you encounter problems accessing the area, contact the site administrator.

The software area could be under AFS or mounted in some other way (tipically through NFS). If the area is not under AFS, write permission to the area for the SGM is enforced automatically (if this does not happen contact the site admin).

If the area is mounted through NFS, one can just 'cd' to the directory pointed by VO_<VONAME>_SW_DIR. In case the area is under AFS, one has to get KERBEROS credentials in order to be able to access the area. This is possible if the site provides a X509->KERBEROS authetication server, which grants KERBEROS tokens starting from X509 credentials (the proxy certificate). If such server exists, it is specified in the VO-box by the env variable GSSKLOG_SERVER. From the VO-box one can obtain the KERBEROS token using the gssklog command line tool:

$ gssklog -server $GSSKLOG_SERVER

See 'gssklog --help' for more options.

Backing up the VO-box

A proper backup mechanism in place for the VO-box could save time resources and not only in the case of a disaster.

Usually the /opt/vobox directory, together with the home directory of the SGM of the VO (i.e. dteamsgm, alicesgm, atlassgm) on the VO-box should be backed up, but each experiment may have its own requirements regarding the backup procedure (see LCG Service Challenge Wiki).

For what concerns the VO-box itself, the log files of the middleware services (gsisshd, etc.) might be of interest for backup, as it could help to track down what had happened prior to an incident.