Server side installation instructions: setting up a NorduGrid resource

General notes:

Pre-installation steps:

Clusters:

  1. Creating Grid accounts: First, you have to create (some) UNIX accounts on your cluster dedicated for the Grid. These local UNIX accounts will be used to map the Grid users locally, every Grid job or Grid activity will take place via these accounts. In the simplest scenario, it is enough to create a single account, i.e. a user called grid, or you can have separate accounts for the different Grid user groups. You may group the created Grid accounts into UNIX groups and use the local UNIX authorization methods to restrict the Grid accounts. ---someone, please write here about the DYNAMIC accounts, how to set them up with the nordugrid---
  2. Create the disk areas on the frontend which will be used by the Grid services. A typical setup is given in the table below; example locations are indicated, too. Location "NFS" means that the directory has to be available on the nodes. It is recommended to put the "grid area" and the "cache directory" onto separate volumes (partitions, disks).

  3. Function Location Description Example
    grid area (required) NFS the directory which accomodates the session directories of the Grid jobs /scratch/grid
    cache directory (optional) NFS the place where the shared input files of the Grid jobs are kept /scratch/cache
    runtime environment scripts (optional) NFS the place for the initialization scripts of the pre-installed software environments /SOFTWARE/runtime
    control directory (required) local to the front-end the directory for the internal control files of the Grid Manager /var/spool/nordugrid/jobstatus

    Further notes on the Grid directories: Some of the NFS requirements can be relaxed with special cluster setup & configuration. For the possible special setups please consult the Grid Manager documentation.
    1. For example, the cache directory does not necessarily need to be available on computing nodes. If it is not avaialble the Grid Manager need to be configured to copy cached files to session directories.
    2. Cache can be divided into 2 subdirectories: a) a "control" subdirectory for control files and b) a "data" subdirectory for the cached data itself. It is advisable _not_ to put the "control" cache directory on remote NFS volume because some NFS implementations have problems with implementation of file locking. Otherwise You are safe to export the local volume containing "control" cache directory over NFS. In case you are really serious about security you should also use cache data copy instead of soft links. ---someone, please check it, I am not sure I got it right---
    3. Instead of sharing the directory of the runtime environment scripts, the small scripts could be installed on every nodes.
  4. Check the "network connectivity" of the computing nodes. In the NorduGrid it is NOT a requirement to have the internal cluster nodes fully available on the public internet. Nodes can have "inbound", "outbound", both or no network connectivity (or "nodeaccess"). The "nodeaccess" property should be set in the configuration (see below).
  5. Make your firewall Grid-friendly: There are certain ports & port ranges which need to be open in case your Grid resource sits behind a firewall. All of the requirements are comming from the Globus internals (you can read more on Globus & firewalls). In the NorduGrid we require the following ports to be opened: 2135 (MDS), 2811 (Gridftp) plus a range of ports for Gridftp data channels. If you are using the globus-config package you should set GLOBUS_TCP_PORT_RANGE=9000,9200 and GLOBUS_UDP_PORT_RANGE=9000,9200 in /etc/sysconfig/globus and open that range together with the 2135 and 2811 as well. The above ports for MDS, Gridftp services correspond to the recommended default settings which can be modified in the nordugrid.conf & globus.conf and /etc/sysconfig/globus. ---someone, what is the recommended PORT_RANGE? ----
  6. Configure the PBS batch system in order to fit the Grid. In a typical scenario a Grid queue (or queues) is created, all or some of the cluster nodes are assigned to the Grid queues, PBS queue & user limits are set for the Grid queue and the Grid accounts. The NorduGrid PBS configuration instructions is a good starting point.DO NOT use routing queues, they are not supported.

Storage Element:

  1. Install a standard linux box with a dedicated storage area. In case the SE wants to serve several Grid user groups (or "VO"s) it is preferable to dedicate separate disks (volumes, partitions, etc.) for the different Grid user groups.
  2. Creating Grid accounts: First, you have to create (some) UNIX accounts dedicated for the Grid. These local UNIX accounts will be used to map the Grid users locally, the data stored on the storage element will be owned by these accounts. In the simplest scenario, it is enough to create a single account, i.e. a user called grid, or you can have separate accounts for the different Grid user groups. It is recommended to put all the Grid accounts into the same UNIX group.
  3. Make your firewall Grid-friendly: There are certain ports & port ranges which need to be open in case your Grid resource sits behind a firewall. All of the requirements are comming from the Globus internals (you can read more on Globus & firewalls). In the NorduGrid we require the following ports to be opened: 2135 (MDS), 2811 (Gridftp) plus a range of ports for Gridftp data channels. If you are using the globus-config package you should set GLOBUS_TCP_PORT_RANGE=9000,9200 and GLOBUS_UDP_PORT_RANGE=9000,9200 in /etc/sysconfig/globus and open that range together with the 2135 and 2811 as well. The above ports for MDS, Gridftp services correspond to the recommended default settings which can be modified in the nordugrid.conf & globus.conf and /etc/sysconfig/globus. ---someone, what is the recommended PORT_RANGE? ---

Collecting & Installing the Grid software (middleware):

  1. In principal, the same software is needed for both a cluster and a storage resource. The NorduGrid download area contains all the required software including external dependencies, precompiled binaries and source versions.
  2. Download & install the NorduGrid-version of the Globus toolkit from the NorduGrid download area "External software" block, get the following packages and install them in this order: gpt, globus, globus-config . Unless it is relocated GPT and Globus is installed under /opt/gpt, /opt/globus respectively. On some systems some perl packages may be required which are also available from the "download area -> External software" block.

    Although you can use any other Globus distribution, we recommend to use the NorduGrid-distributed Globus since it contains some critical fixes, and allows you to install the NorduGrid from binary packages. The modifications within the NorduGrid Globus distribution are collected and described here: Specifics of the NorduGrid release of the Globus Toolkit 2 Furthermore, you may bump into unforseen problems because of not using the globus-config configuration accepted within the NorduGrid. If you choose to use another Globus installation you will need to get the NorduGrid source and recompile it against your Globus plus you will need to have recourse to the original Globus configuration files and documentation.

  3. Download & install the required NorduGrid packages from the "download area -> NorduGrid toolkit ->latest release" block (The same block contains the nightly & nightly_experimental packages, for a production site use ONLY tagged releases). You definitely need the nordugrid-server while the nordugrid-client and the nordugrid-doc is optional but recommended. It is useful to have the client installed on a server for testing purposes. You may be requested to install some of the externall packages in order to satisfy dependencies ---someone, please clean up the download area and check/write this section, what is needed for what, what are the dependencies of the nordugrid-server & nordugrid-client, what should a resource install, when does a site need to install the gacl, gsoap, etc..---

Setting up the Grid Security Infrastructure: Certificates, Authentication & Authorization

  1. The following considerations apply for both clusters and storage elements. You may find useful our certificate mini Howto.
  2. Your site needs to have certificates for the Grid services, the minimum is a host certificate but we recommend to have an LDAP (or MDS) certificate as well. In order to generate server certificate requests first install the ca_NorduGrid-certrequest-config package from the "download area -> CA certificates" block. This contains the default configuration for generating certificate requests for a nordugrid-based certificate. Generate a certificate request with grid-cert-request for host (and LDAP) certificate, send the request(s) to the CA for signature. Upon receipt the signed certificates, place them into the proper location (/etc/grid-security/), check the file permissions that is the files are owned by root and the private keys are only readable by root and that none of the files are executable.
  3. Set up your authentication policy: decide which certificates your site will accept. You do this by installing the CA packages of the trusted certificate authorities. If you want to be part of the NorduGrid TestBed (accept NorduGrid users) then you need the ca_NorduGrid package. The selected CA packages can be downloaded from the "download area -> CA certificates block". Before installing any CA package you are advised to check the credibility of the CA, verify its policy!
  4. The Certificate Authorities are responsible for maintaining a list of revoced certificates. It is the site (that is yours) responsibility to check those certificate revocation lists and deny access to Grid users presenting a revoced certificate (crl). NorduGrid provides an automatic tool for this purpose. We recommend to install the nordugrid-ca-utils from the "download area -> NorduGrid Toolkit -> latest release" which periodically keeps track of the CA revocation lists. Please note that having an outdated crl-file can close down your entire resource.
  5. Set up your authorization policy: decide which Grid users/group of Grid users (in Grid terminology which Virtual Organization) are allowed to use your resource, define the Grid mappings (Grid users -> local Unix users). The Grid mappings are listed in the so-called grid-mapfile. Within the NorduGrid there is an automatic tool which keeps the local grid-mapfiles synchronized to a central user database. If your site joins the NorduGrid you are recommended to install the nordugrid-gridmap-utils from the "download area -> NorduGrid Toolkit ->latest release". After installation you need to edit the /etc/grid-security/nordugridmap.conf and optionally create the /etc/grid-security/local-grid-mapfile (the filename is configurable in the nordugridmap.conf) with your local mappings. For further info on authorization read the VO-document. Important: you either maintain the grid mappings by hand editing the /etc/grid-security/grid-mapfile directly or use the nordugrid-gridmap-utils (nordugridmap script run through cron) to create and maintain the file for you. In the latter the utility keeps the grid-mapfile synchronized with the central NorduGrid user list. If you install the nordugrid-gridmap-utils you only have to edit the nordugridmap.conf and optionally the local-grid-mapfile, the /etc/grid-security/grid-mapfile is periodically overwritten by the nordugridmap script!

Configuring the Grid resource:

  1. Next step is the configuration of your resource, the same files have to be edited for both a computing cluster and for a Storage Element (or for a combined resource of a cluster & SE). The configuration templates serve as a configuration document with detailed description of the configuration parameters and options. The confiuration file consists of dedicated blocks for cluster and SE related services, not having a certain block means not running the corresponding service on the resource.
  2. Create your /etc/nordugrid.conf by using the configuration template nordugrid.conf.template from the /opt/nordugrid/share/doc (provided you installed the nordugrid under /opt/nordugrid/). With the nordugrid.conf among others, you configure the basic services and processes like the gridftp-server, grid-manager, the jobsubmission interface, the grid storage areas and the information providers.
  3. Create your /etc/globus.conf by using the configuration template nordugrid-globus.conf.template from the /opt/nordugrid/share/doc (provided you installed the nordugrid under /opt/nordugrid/). With globus.conf you basically set up your core information system services (configure the Openldap server and backends together with the registration processes). The core NorduGrid Grid Information Index Services (GIIS) are specified here: NorduGrid GIIS Information
  4. Optionally you can setup runtimeenvironments on your computing cluster. Setting up a runtimeenvironment involves installing a specific application software package onto the cluster (the software package is made available for the nodes as well!) and putting a runtimeenvironment initialization script named after the runtimeenvironment into the runtimeenvironment script's directory.

Startup scripts, logfiles, debug mode, test-suit:

  1. After a successfull installation & configuration on a NorduGrid resource the following services can be started:
    /etc/init.d/gridftp-server start launches the gridftp-server
    /etc/init.d/globus-mds start launches the information system (LDAP server) and the registration processes
    /etc/init.d/grid-manager start launches the grid-manager daemon
  2. For a computing resource you have to start all the three resources while a storage element requires only the gridftp-server and the information system globus-mds
  3. The logfiles can be used to check the services. The information system uses the /var/log/infoproviders and the /var/log/globus-mds files. The gridftp-server and the grid-manager logs into the /var/log/gridftp-server.log and the /var/log/grid-manager.log files respectively. The /var/log/gm-jobs.log file is used by the grid-manager for logging job information.
  4. Debug information: both in the nordugrid.conf and the globus.conf different debug levels can be set, core files can be enabled. In case of crashing services please try to produce core files. Please note that enabling debuging results in a serious performance lost (especially in the case of the MDS LDAP server), therefore use the default level of debuging in a production system. ---someone, how is the globus & nordugrid compiled, debug???---
  5. The nordugrid-client comes together with the ngtest utility. Use it to test the basic functionality of computing resource. The utility includes several tests which can be interesting to test your cluster with -- e.g. simple up- and download tests. A complete list of test-cases can be seen by running ngtest -list-cases. The default test is number 0 which tests several things like downloading from several locations, the cache, manual downloading of results and so on. We recommend at least to run the default test against a newly installed resource.
    The gsincftp gridftp client (comes with the nordugrid-globus) can be used for testing the Storage Element setup. ---someone, please! ---