Globus Installation in the Astrogrid-D Standard (GACSI)

(AstroGrid-D / AIP. Based on Globus-Quickstart)

System Requirements and Preparation
Set Up User 'globus'
iODBC installation
Setting the PATH and Globus defaults
Configuring and Compiling the Globus-Toolkit
Linking in the Globus-Advisories
Installing the Service- and Host-Certificates
Setup for gsiftp and gsissh
Configuring the Database for RFT
Starting the Java Web Services Core Container and gsissh
Grid-User Creation
Ganglia and Cluster Integration
GRAM Audit Logging
Appendix: Overview of the Scripts in the Globus-Helper Package (New page)

As a short general introduction the following are recommended: the Globus Quickstart or the Presentations from the LRZ Garching:

Globus Workshop
GT4_Installation (PDF, German)

or the IBM redbook SG246778, "Introduction to Grid Computing".

For further information and debugging please see the Globus Admin Guide, the Globus Mailing lists and the Globus Bugzilla

[top]
1. System Requirements and Preparation

This guide is intended for the Astrogrid-D Standard Installation with Scientific Linux (Redhat Enterprise Unix, 4.2 or higher). Root access to the machine is essential, incl. sudo: for the most part the installation is done as user root. System administration experience would be useful, but is not required to complete the installation using this guide.

At many places in this document a text editor is required. Here vi is used for examples. Other alternatives are nedit (graphical Editor) or pico and nano.

Before continuing this guide, check in the Guide to required Unix Software and Updates that the required software versions are correctly installed and that they run.

The following table gives the links to the source archives for (Globus, iODBC) and the Globus-Helper package, which need to be downloaded:

Source code	Version	Target binary directory
Globus toolkit - Full Toolkit Source Download	gt4.0.8-all-source-installer.tar.gz	`/usr/local/globus/gtk/` [link]
iODBC source files	3.5	`~globus/iodbc/`
Astrogrid-D Globus Helper package	globus-helper-v1.3.tgz	`~globus/globus-helper/`

The installation can be done using a virtual machine (VMware 5.5 or higher recommended). Recommendations for the firewall settings are given by D-Grid at: Configuration of static Firewalls (pdf) (76KB).

The IP number and Fully Qualified Domain-Name (FQDN) need to be entered correctly into /etc/hosts, e. g.:

more /etc/host 141.33.4.98 cashmere.aip.de

The relevant IP address can be found using ifconfig. If the FQDN is not present, /etc/hosts should be edited and a line added.
If the machine name is not listed in the DNS, then it must be explicitly entered into /etc/hosts on the grid resources from which it will be accessed.

In the following examples at different places names and version numbers are used which are correct at the time of creating this document. During installation these should be replaced if required. In this guide a standard directory structure is used, which is listed below:

Home directory for user Globus: /work1/globus/ [= "~globus"]
Build: BUILDDIR=/work1/build/
$GLOBUS_LOCATION: /usr/local/globus/gtk [=link to ~globus/gt408]
Certificates: /etc/grid-security/, /root/AstroCert/ and ~/.globus (for the User Certificates)

This structure is used in the following (especially Chapter 4) and is implemented in the scripts belonging to "globus-helper" but it does not have to be adhered to as long as the scripts are adjusted accordingly.

[top]
2. Set up user `globus`

As user root, the user "globus" with group "globus" is set up. The login shell should be bash. The id's can be chosen (here: 10001).

/usr/sbin/groupadd -g 10001 globus /usr/sbin/useradd -u 10001 -g globus -s /bin/bash -d /work1/globus globus

Then the directory for the Globus Toolkit is set up and a soft link created:

cd ~globus mkdir gt408 mkdir -p /usr/local/globus chown globus:globus gt408 /usr/local/globus
ln -s ~globus/gt408 /usr/local/globus/gtk

In later upgrades to new version this link is updated, but the configuration data, .bashrc and $PATH variables remain the same. Finally the globus-helper package is copied into the directory and the update directory created:

wget -q -O - http://www.gac-grid.net/project-products/Software/Globus-Helper/globus-helper-v1.3.tgz | tar xfz - ln -s globus-helper-v1.3 globus-helper mkdir globus-update

[top]
3. iODBC installation

The relevant directories are created the source files unpacked:

BUILDDIR= /work1/build mkdir -p $BUILDDIR
chown globus:globus $BUILDDIR mv libiodbc-3.52.2.tar.gz $BUILDDIR cd $BUILDDIR tar xvzf libiodbc-3.52.2.tar.gz cp ~globus/globus-helper/globus-install/iodbc.cfg $BUILDDIR /libiodbc-3.52.2/ cd $BUILDDIR /libiodbc-3.52.2

The file iodbc.cfg contains the correct Globus settings. If other paths are to be used these must be set in iodbc.cfg. The file contains the following call to configure:

./configure --prefix=~globus/iodbc --disable-gtktest --with-pthreads --disable-gui \ --with-iodbc-inidir=~globus/iodbc/etc

sh -x iodbc.cfg make

Now iodbc can be installed (as user root):

make install

[top]
4. Setting the PATH and Globus defaults

It is important to have the shell variables for the Java Software Development Kit (JDK), ant and Tomcat directories set correctly. These are not Globus specific and often follow from the package installation (see the " Guide to Required Unix Software and Updates"). It is also useful, if not already in place, to create symbolic links from a standard location to the actual binaries in the particular directory versions e.g.:

mkdir -p /usr/local/jdk ln -s /opt/jsdk-xx/ant-1.6.5/ /usr/local/jdk/ant

Similar links should be created for JSDK - /usr/local/jdk/jsdk and Tomcat to /usr/local/jdk/tomcat.
Then these standard directories can be put into the standard path:

cat ~globus/globus-helper/griduser/etc_profile.add >> /etc/profile

The contents of globus_defs/griduser/etc_profile.add is then:

JDK="/usr/local/jdk"
JAVA_HOME="$JDK/jsdk"
CATALINA_HOME="$JDK/tomcat"
ANT_HOME="$JDK/ant"
J_BIN="$ANT_HOME/bin:$JAVA_HOME/bin:$JAVA_HOME/jre/bin:$CATALINA_HOME/bin"
PATH="$J_BIN:$PATH"
export PATH USER LOGNAME MAIL HOSTNAME HISTSIZE INPUTRC JAVA_HOME ANT_HOME CATALINA_HOME

Duplicate 'export' commands should be removed so that the path does not become too long. The given syntax is valid for bash. For csh and tcsh the relevant command for setting variables 'setenv' should be used.

After the standard directories the Globus-specific variables for the users root and globus are set (and respectively for additional users). For this purpose the globus-helper/griduser/grid-env.sh is appended to the .bashrc (for csh/tcsh: globus-helper/griduser/grid-env.csh):

cat ~globus/globus-helper/griduser/grid-env.sh >> ~globus/.bashrc cat ~globus/globus-helper/griduser/grid-env.sh >> /root/.bashrc

In the grid-env.sh the target pointed to by the symbolic link is set as the main Globus directory. The standard content is:

GLOBUS_LOCATION=/usr/local/globus/gtk GLOBUS_PATH=$GLOBUS_LOCATION/sbin PATH=$GLOBUS_PATH:$PATH GLOBUS_TCP_PORT_RANGE=20000,25000 export PATH GLOBUS_LOCATION GLOBUS_TCP_PORT_RANGE source $GLOBUS_LOCATION/etc/globus-user-env.sh

[top]
5. Configuring and Compiling the Globus Toolkit

Change to user globus now to avoid permissions issues. If the source is unpacked as user root then run the following command afterward in the expanded directory : 'chown -R globus *'. Because some scripts have not yet been copied for the Globus Toolkit there is an error message on switching to user globus - su globus:
/usr/local/globus/gtk/etc/globus-user-env.sh: No such file or directory.
This can be ignored.

cd $BUILDDIR chmod 777 * su - globus # blue letters indicate user globus cd $BUILDDIR tar xvfj ../gt4.0.8-all-source-installer.tar.bz2 cd gt4.0.8-all-source-installer cp ~globus/globus-helper/globus-install/gt4.cfg .

The file gt4.cfg should be edited as follows.

The variable GLOBUS_LOCATION is to specify the absolute path (no symbolic links!) of the directory in which Globus is to be installed e.g. GLOBUS_LOCATION=/work1/globus/gt408. This directory must be in a local file system -- not a network file system

For older machines with 32-bit processors the 32-bit Support is set using the configure option --with-flavor="gcc32dbg" (in the 64-bit script).

For PBS Support (compute-clusters) the configure option --enable-wsgram-pbs Home-Variable (e. g. PBS_HOME=/work1/PBS) must be set, otherwise this should be commented out PBS = "Portable Batch System" for compute-clusters.

ANT_HOME and JAVA_HOME have to be defined correctly and exported and the their bin directories must be in the $PATH.

It should be verified that the configure options (given in the last line ./configure ...) are not separated by a blank line or a commented line (as the following options would then be ignored).

Now the Makefile can be generated:

ls $ANT_HOME $JAVA_HOME # Test sh -x gt4.cfg

Warnings should not be ignored - instead the causes need to be resolved. Also, attempts to do a parallel build (with make -j3) have so far been unsuccessful.

Only after a configuration without Warnings is the Makefile to be called:

make 2>&1 | tee globus_install.log

Because 'make' calls 'ant' it is not certain that all errors in the make dependencies are correctly reported in the case paths are not set correctly. If "make" fails later due to Java errors and the problem is resolved then before starting a new 'make' remove the entire directory-tree under ~globus/gtk . 'Make' also stops if a version-number of the required program is insufficient (see the Guide to Required Unix Software and Updates) e.g. for libz.so (zlib).

If the compile is successful (on current systems this may take 2 hours) the program is then installed:

make install

A successful install finishes with the message "..Done". If this does not complete normally the options should be checked and the compilation restarted. If it is not present in the make script the last call is to 'gpt-postinstall' as a final check and completion of the installation.

[top]
6. Linking in the Globus Advisories

The Globus Advisories (i.e. the updates and patch fixes for a particular version number) can be downloaded into the directory "~globus/globus-updates/". Here it needs to be checked which parts of Globus are installed e.g. no 'rls' updates reduces the effort. The linking in can be done using the script update_globus.pl from the globus-helper package:

cd ~globus/globus-updates cp ~globus/globus-helper/globus-install/update_globus.pl . ./update_globus.pl -d . export GPT_LOCATION=$GLOBUS_LOCATION gpt-postinstall

For full updates between two version numbers there is a separate guide in the section on Globus Installation on the AstroGrid-D website.

[top]
7. Installation of the Service and Host Certificates

Generate a Globus default certificate

As user globus, generate an initial Globus default certificate

$GLOBUS_LOCATION/setup/globus/setup-simple-ca -noint

Then as user root call, the command printed from output of the above script. ( The GLOBUS_LOCATION variable must be set) e.g. :

$GLOBUS_LOCATION/setup/globus_simple_ca_{CA-HASH}_setup/setup-gsi

These default certificates are not required for working in the AstroGrid and are for backup only. They will be replaced by the certificates from the root CA of the Forschungszentrum Karlsruhe. The script which does this work, installs the public FZK certificates and configures the Organisational Unit, or OU, to which the machine belongs. This is passed using the option -i. The list of all the OU's registered in the Astrogrid-D is given in http://grid.fzk.de/ca/RA.html

cd /etc/grid-security tar cfz orig.tgz * cd ~globus/globus-helper/security/configrootCA/ ./ConfigureFZK-CA.pl -i {AEI|AIP| ...} -g

If the certificate authority used is DFN instead of FZK, check with the local registration authority in case there are any differences.

Generate and mail certificate request

A domain name for the host is required in the request e.g. "example.aip.de".

mkdir -p /root/AstroCert ~globus/globus-helper/security/service/generate_service_req.pl -h example.aip.de -d /root/AstroCert/

This will generate several files in /root/AstroCert. Keep the hostkey.pem file for later use. Send the request file "example.aip.de_hostcert_request.pem" as an attachment to the local Registration Authority (the one for institute can be found in the RA-List of VOMRS), e.g. by using

cat /root/AstroCert/nonexistent.aip.de_hostcert_request.pem | mail

The request will be signed and passed on to the CA, who will validate it and return it to you. (It is also possible in principle to submit the request oneself but this is very unusual.)

It can take a few days until the e-mail response with the signed and valid certificate arrives. Some of the steps from the following sections can already be completed without the certificate.

Install signed certificates

Once you get the signed certificate, install it:

mkdir -p /root/AstroCert cp certificates_from_mail /root/AstroCert/ cd /root/AstroCert/ ~globus/globus-helper/security/service/install_service_cert.pl -h nonexistent.aip.de

The last script moves and renames the certificates to /etc/grid-security/. Copy the Globus container certificate and key from the host certificate and key. After installation using 'ls -l /etc/grid-security/*.pem' check that the certificates have the correct permissions (owner root , instead of globus is also possible):

  -r--------  1 globus globus containerkey.pem
  -rw-r--r--  1 globus globus containercert.pem
  -rw-r--r--  1 root   root   hostcert.pem
  -r--------  1 root   root   hostkey.pem

Import the CA public key

Finally, import the public key of the DFN Certification Authority, for use by the Astrogrid-D:

tar xfz ~globus/globus-helper/security/configrootCA/dfncerts.tgz --directory /etc/grid-security/certificates

At this point, users with valid grid certificates should be able to log to the host in via gsissh.

Install CRL update script

To refresh daily the list of valid user certificates, install the program fetch-crl. It updates the list of certificates revoked by the certificate authority (the Certificate Revocation List, CRL). This program has been adopted by the 'Astrogrid-D Standard Installation'. Install it as user root using:

cd ~globus/globus-helper/security/astro-fetch-crl/ ./install_fetch-crl.pl

The script enters a daily update of the CRL into the cron tables.

Before grid users can log in, accounts have to be set up for them. Grid user management is handled below in the section User-Creation.

The process for getting a user certificate, with which a user can access the grid, is described in the GAC-Userguide.

[top]
8. Setup for `gsiftp` and `gsissh`

The required gsiftp ports have to be entered into /etc/services using an editor such e.g. vi /etc/services. the relevant ports are entered as new lines:

gsigatekeeper     2119/tcp

...

gsiftp            2811/tcp

...

Registering the services to xinitd.d is done using the below script :

cd ~globus/globus-helper/xinetd.d ./install_xinet-conf.pl

The Globus standard port for gsissh is the ssh port 22. if the usual ssh daemon is still to be used, then the gsissh daemon needs to be set to a different port. The port 2222 is recommended as more-or-less standard.

The service is added to init.d :

ln -s $GLOBUS_LOCATION/sbin/SXXsshd /etc/init.d/gsisshd /sbin/chkconfig --add gsisshd

To reconfigure the daemon,

the entry $GLOBUS_LOCATION/etc/ssh/sshd_config for "Port 22" is to be commented out (if not already commented out) and the new entry " Port 2222 " should be entered.
edit the file $GLOBUS_LOCATION/etc/ssh/ssh_config and the change to "Port 2222". It is also necessary to chmod 664 $GLOBUS_LOCATION/etc/ssh/ssh_config.
The new service is entered by editing the /etc/services file with the new line " gsissh 2222/tcp ".

Then the daemon is started to test the configuration:

/etc/init.d/gsisshd start

Anyone who is registered using grid-proxy-init (or my-proxy-logon) in the Grid and is listed in the grid-mapfile on the relevant machines, can then acess the Grid-Resources without a further password request. Otherwise gsissh behaves like ssh and asks for a local password. In case the client and server use different ports, use the option -p in the call e. g. gsissh -p 22 noexist.aip.de.

[top]
9. Configuring the database for Globus File Transfer (RFT-Database)

PostgreSQL is used for the database. For installation and checking PostgreSQL, see the System Requirements.

The database creation for Globus Reliable File Transfer (RFT) and access to it is done as user postgres, using an existing SQL script.

sudo su - postgres postgres> createdb rftDatabase CREATE DATABASE postgres> psql -d rftDatabase -f /usr/local/globus/gtk/share/globus_wsrf_rft/rft_schema.sql 6: NOTICE: CREATE TABLE / PRIMARY KEY will create implicit index "requestid_pkey" for table "requestid" ... CREATE INDEX postgres> psql -d rftDatabase -c "create role globus with superuser login encrypted password 'foo'" postgres CREATE ROLE postgres> exit

If the postgresql.conf file does not exist, e,g. if postgresql had never been started, then this is created as follows: /etc/init.d/postgresql start.

Also note: the above is for PostgreSQL 8.0 or greater. For PostgreSQL 7.x, use instead the command

postgres> psql -d rftDatabase -c "create user globus with createuser encrypted password 'foo'" postgres

If you are using PostgreSQL 8.0 or greater, to allow local access over TCP/IP sockets, the first line of Connection Settings in
/var/lib/pgsql/data/postgresql.conf has to be changed:

# - Connection Settings - listen_addresses = '*' ...

In any case, the /var/lib/pgsql/data/pg_hba.conf is updated to contain the local host.

# TYPE	DATABASE	USER	IP-ADDRESS	IP-MASK		METHOD
host	rftDatabase	globus	<IP-Number>	255.255.255.255	md5
...

In $GLOBUS_LOCATION/etc/globus_wsrf_rft/jndi-config.xml set the parameters:

<parameter> <name>userName</name> <value>globus</value> </parameter> <parameter> <name>password</name> <value>foo</value> </parameter> ...

Then restart the database with sudo /etc/init.d/postgresql restart!
As user globus, test:
psql -d rftDatabase -c "\d" globus
shows the database tables.

[top]
10. Starting the Java Web-Services Core Container and `gsissh`

To start the container the Certification process must be completed. The Web-Service of the Globus Container can then be added to the init.d directory:

cd ~globus/globus-helper/init.d/ ./install_init-conf.pl -e

The service will then be started automatically on system start on init level 3. This can be tested using:

/etc/init.d/globus start

(this includes the call to "$GLOBUS_LOCATION/sbin/globus-start-container-detached -p 8443" ). The startup can take several seconds.

To check the successful startup and for debugging the following commands are available:

/etc/init.d/globus status more $GLOBUS_LOCATION/var/container.log ps aux | grep java # is the Java container running? $GLOBUS_LOCATION/bin/globus-start-container -debug -p 8443

[top]
11. Creating the Grid Users

The registered Astrogrid-D users are selected from the VORMS database and are entered in the grid-mapfile. This is handled by the script ManageLocalGridUser.pl in the directory ~globus/globus-helper/manage-griduser/.

In the configuration file conf/manage-grid-user.conf, the setting gtk_path must be the absolute path of the Globus installation. Be sure to adapt the configuration for your environment. For security reasons to do with the command sudo, no symbolic links are allowed in the file, only the absolute path of the original directory e.g. /work1/globus/gt408/.

The script also expects the file

/root/AstroCert/VOMRS/volist_secrets

to be present. Copy the default file from ~globus/globus-helper/manage-griduser/, and ask your local VOMRS authority for the correct user and password fields.

cd ~globus/globus-helper/manage-griduser/ ./ManageLocalGridUser.pl -c conf/manage-grid-user.conf

The enabled set of users and groups are entered in the /etc/sudoers file, so that the WS-GRAM allows the user globus password-free access to all the grid user accounts. The required lines are created by ManageLocalGridUser.pl and can be found in the files sudoers-alias and sudoers-command. They can be copied into the /etc/sudoers files using copy & paste:

cat sudoers-* xterm -e visudo

Note the difference between the unix user globus as which the toolkit runs, and the Globus users. The latter uses its certificate to identify itself and to execute Globus programs.

To permanently install the user management software, copy the directory ~globus/globus-helper/manage-griduser/ somewhere convenient (we usually call this /root/AstroCert/VOMRS/Manage/). Then make sure the script update-local-griduser specifies this directory. Finally, arrange for this script to be run regularly, for instance by copying it into /etc/cron.daily/.

Log files are to be found in the log/ subdirectory of the folder.

[top]
12. Ganglia and Cluster Integration

For the Ganglia installations please follow the instructions provided at the Ganglia and MDS4 page.

[top]
13. GRAM Audit Logging

see installation guide
[top]

Globus Installation in the Astrogrid-D Standard (GACSI)

Contents:

[top] 1. System Requirements and Preparation

[top] 2. Set up user globus

[top] 3. iODBC installation

[top] 4. Setting the PATH and Globus defaults

[top] 5. Configuring and Compiling the Globus Toolkit

[top] 6. Linking in the Globus Advisories

[top] 7. Installation of the Service and Host Certificates