diff --git a/docs/alienpy_commands.md b/docs/alienpy_commands.md index d357f13dcfe6603b130c0ac2461d5bffc191de2f..0a4309a09435119a2de4957e538a1f20a6448b9a 100644 --- a/docs/alienpy_commands.md +++ b/docs/alienpy_commands.md @@ -1,3 +1,1014 @@ -# alien.py Commands +# alien.py Command reference guide -Soon to come... +### access +``` +usage: access [options] <read|write> <lfn> [<specs>] +-s : for write requests, size of the file to be uploaded, when known +-m : for write requests, MD5 checksum of the file to be uploaded, when known +-j : for write requests, the job ID that created these files, when applicable +-f : for read requests, filter the SEs based on the given specs list +-u : for read requests, print http(s) URLs where available, and the envelopes in urlencoded format +``` + +--- + +### cat +``` +usage: cat [-options] [<filename>] +options: +-o : outputfilename +-n : number all output lines +-b : number nonblank output lines +-E : shows ends - display $ at end of each line number +-T : show tabs -display TAB characters as ^I +``` + +!!! warning "client-side implementation" + +--- + +### cd +``` +usage: cd [dir] +``` + +--- + +### cert-info +``` +Print user certificate information +``` + +!!! warning "client-side implementation" + +--- + +### cert-verify +``` +Verify the user cert against the found CA stores (file or directory) +X509_CERT_FILE/X509_CERT_DIR apply if set +``` + +!!! warning "client-side implementation" + +--- + +### certkey-match +``` +Check private key against the public key +``` + +!!! warning "client-side implementation" + +--- + +### changeDiff +``` +Show changes between the current version of the file and the previous one (same file name with a '~' suffix) +usage: changeDiff [<filename>] + +options: +``` + +--- + +### chmod +``` +Usage: chmod -R <octal mode> <path> [<path>...] + +Changes the access mode for a catalogue path +-R : do a recursive chmod starting from the given path +``` + +--- + +### chown +``` +Usage: chown -R <user>[.<group>] <file> + +Changes an owner or a group for a file +-R : do a recursive chown +``` + +--- + +### commandlist +``` +usage: commandlist +Print the server-side (as opposed to client-side) known commands +``` + +--- + +### commit +``` +usage: commit API command only, you should not need to call this directly +``` + +--- + +### cp +``` +Command format is of the form of (with the strict order of arguments): + cp <options> src dst + or + cp <options> -input input_file +where src|dst are local files if prefixed with file:// or file: or grid files otherwise +and -input argument is a file with >src dst< pairs +after each src,dst can be added comma separated specifiers in the form of: @disk:N,SE1,SE2,!SE3 +where disk selects the number of replicas and the following specifiers add (or remove) storage endpoints from the received list +options are the following : +-h : print help +-f : replace destination file (if destination is local it will be replaced only if integrity check fails) +-P : enable persist on successful close semantic +-cksum : check hash sum of the file; for downloads the central catalogue md5 will be verified; + for uploads (for xrootd client > 4.12.0) a hash type will be negociated with remote and transfer will be validated +-y <nr_sources> : use up to the number of sources specified in parallel (N.B. Ignored as it breaks download of files stored in archives) +-S <aditional TPC streams> : uses num additional parallel streams to do the transfer. (max = 15) +-chunks <nr chunks> : number of chunks that should be requested in parallel +-chunksz <bytes> : chunk size (bytes) +-T <nr_copy_jobs> : number of parralel copy jobs from a set (for recursive copy); defaults to 8 for downloads + +for the recursive copy of directories the following options (of the find command) can be used: +-glob <globbing pattern> : this is the usual AliEn globbing format; N.B. this is NOT a REGEX!!! defaults to all "*" +-select <pattern> : select only these files to be copied; N.B. this is a REGEX applied to full path!!! +-name <pattern> : select only these files to be copied; N.B. this is a REGEX applied to a directory or file name!!! +-name <verb>_string : where verb = begin|contain|ends|ext and string is the text selection criteria. +verbs are aditive : -name begin_myf_contain_run1_ends_bla_ext_root +N.B. the text to be filtered cannont have underline <_> within!!! +-parent <parent depth> : in destination use this <parent depth> to add to destination ; defaults to 0 +-a : copy also the hidden files .* (for recursive copy) +-j <queue_id> : select only the files created by the job with <queue_id> (for recursive copy) +-l <count> : copy only <count> nr of files (for recursive copy) +-o <offset> : skip first <offset> files found in the src directory (for recursive copy) +``` + +!!! warning "client-side implementation" + +--- + + +### deleteMirror +``` +Removes a replica of a file from the catalogue +Usage: + deleteMirror [-g] <lfn> <se> [<pfn>] + +Options: + -g: the lfn is a guid +``` + +--- + +### df +``` +Shows free disk space +Usage: df +``` + +--- + +### dirs +``` +It provides the mechanics of Directory Stack Builtins of bash +``` + +!!! warning "client-side implementation" + +--- + +### du + +Gives the disk space usge of one or more directories +Usage: `du [-ncs] <path> ...` +Options: +`-n`: Print raw numbers in machine readable format +`-c`: Include collections in the summary information +`-s`: Print a summary of all parameters + +--- + +### edit +``` +Command format: edit lfn +After editor termination the file will be uploaded if md5 differs +-datebck : the backup filename will be date based +N.B. EDITOR env var must be set or fallback will be mcedit (not checking if exists) +``` + +!!! warning "client-side implementation" + +--- + +### error OR `$?err` +``` +Return latest recorded error message. +``` + +!!! warning "client-side implementation" + +--- + +### exec +``` +Command format: exec lfn list_of_arguments +N.B.! The output and error streams will be captured and printed at the end of execution! +for working within application use <edit> +``` + +!!! warning "client-side implementation" + +--- + + +### exitcode OR $? +``` +Return the last known exitcode +``` + +!!! warning "client-side implementation" + +--- + +### exit +``` +Exit the shell +``` + +!!! warning "client-side implementation" + +--- + +### find2 +``` +Client-side implementation of find; it will use as default the regex option of server's find. +Command formant: find2 <options> <directory> +-select <pattern> : select only these files; N.B. this is a REGEX applied to full path!!! defaults to all ".*" +-name <pattern> : select only these files; N.B. this is a REGEX applied to a directory or file name!!! defaults to all ".*" +-name <verb>_string : where verb = begin|contain|ends|ext and string is the text selection criteria. verbs are aditive e.g.: +-name begin_myf_contain_run1_ends_bla_ext_root +N.B. the text to be filtered cannont have underline <_> within!!! +-d : return also the directories +-w[h] : long format, optionally human readable file sizes +-a : show hidden .* files +-j <queue_id> : filter files created by a certain job +-l <count> : limit the number of returned entries to at most the indicated value +-o <offset> : skip over the first <offset> results + +``` + +!!! warning "client-side implementation" + +--- + +### find +``` +usage: find [flags] <path> <pattern> +options: +-a : show hidden .* files +-s : no sorting +-x <target LFN> : create the indicated XML collection with the results of the find operation +-d : return also the directories +-w[h] : long format, optionally human readable file sizes +-j <queueid> : filter files created by a certain job ID +-l <count> : limit the number of returned entries to at most the indicated value +-o <offset> : skip over the first /offset/ results +-r : pattern is a regular expression +-f : return all LFN data as JSON fields (API flag only) +-y : (FOR THE OCDB) return only the biggest version of each file +``` + +--- + +### getSE +``` +Command format: getSE <-id | -name | -srv> identifier_string +Return the specified property for the SE specified label +``` + +!!! warning "client-side implementation" + +--- + +### grep +``` +usage: grep [-linux grep options] <pattern> [<filename>]+ +options: +``` + +--- + +### groups +``` +groups [<username>] +shows the groups current user is a member of. +``` + +--- + +### guid2lfn +``` +usage: guid2lfn <GUID> +``` + +--- + +### help +``` +usage: help [command] or ? +When command is provided, it will return the output of <command -h> +When no argument, it will provide links to documentation and a list of available commands. +``` + +!!! warning "client-side implementation" + +--- + +### kill +``` +usage: kill <jobId> [<jobId>[,<jobId>]] +``` + +--- + +### la +``` +alias of ls -a +see ls +``` + +--- + +### lfn2guid +``` +usage: lfn2guid <filename> +``` + +--- + +### lfnexpiretime +``` +usage: lfnexpiretime [-options] [<file>] +options: +-r : removes the expire time set for an LFN +-a : add a new expire time for the given LFN +-e : extends the current expire time for the given LFN +-d <number> : specifies the number of days in the expire time +-w <number> : specifies the number of weeks in the expire time +-m <number> : specifies the number of months in the expire time +-y <number> : specifies the number of years in the expire time +``` + +--- + +### listFilesFromCollection +``` +usage: listFilesFromCollection [-options] collection +options: +-z : show size and other file details +-s : silent (API only) +``` + +--- + +### listSEDistance +``` +listSEDistance: Returns the closest working SE for a particular site. Usage +options: +-site : site to base the results to, instead of using the default mapping of this client to a site +-read : use the read metrics, optionally with an LFN for which to sort the replicas. Default is to print the write metrics. +-qos : restrict the returned SEs to this particular tag +``` + +--- + +### listSEs +``` +listSEs: print all (or a subset) of the defined SEs with their details +usage: listSEs [-qos filter,by,qos] [-s] [SE name] [SE name] ... + +options: +-qos : filter the SEs by the given QoS classes. Comma separate entries for 'AND', pass multiple -qos options for an 'OR' +-s : print summary information +``` + +!!! warning "client-side implementation" + +--- + +### listTransfer +``` +listTransfer: returns all the transfers that are waiting in the system +Usage: +listTransfer [-status <status>] [-user <user>] [-id <queueId>] [-verbose] [-master] [-summary] [-all_status] [-jdl] [-destination <site>] [-list=<number(all transfers by default)>] [-desc] +``` + +--- + +### lla +``` +alias of ls -a -l +see ls +``` + +!!! warning "client-side implementation" + +--- + +### ll +``` +alias of ls -l +see ls +``` + +!!! warning "client-side implementation" + +--- + +### logout +alias of `exit` + +!!! warning "client-side implementation" + +--- + +### ls +usage: `ls [-options] [<directory>]` +options: +`-l` : long format +`-a` : show hidden .\* files +`-F` : add trailing / to directory names +`-b` : print in guid format +`-c` : print canonical paths +`-H` : human readable file sizes (1024-based); implies '-l' +the `alien.py` client will use `-F` as default + +--- + +### masterjob +``` +usage: masterjob <jobId> [-options] [merge|kill|resubmit|expunge] +options: +-status <status> : display only the subjobs with that status +-id <id> : display only the subjobs with that id +-site <id> : display only the subjobs on that site +-printid : print also the id of all the subjobs +-printsite : split the number of jobs according to the execution site + +merge : collect the output of all the subjobs that have already finished +kill : kill all the subjobs +resubmit : resubmit all the subjobs selected +expunge : delete completely the subjobs + +You can combine kill and resubmit with '-status <status>' and '-id <id>'. +For instance, if you do something like 'masterjob <jobId> -status ERROR_IB resubmit', + all the subjobs with status ERROR_IB will be resubmitted +``` + +--- + +### mcedit +``` +Command format: mcedit lfn +After editor termination the file will be uploaded if md5 differs +-datebck : the backup filename will be date based +``` + +!!! warning "client-side implementation" + +--- + +### md5sum +``` +usage: md5sum <filename1> [<or guid>] ... +``` + +--- + +### mirror +``` +mirror Copies a file into another SE +Usage: +mirror [-g] [-try <number>] [-S [se[,se2[,!se3[,qos:count]]]]] <lfn> [<SE>] + -g: Use the lfn as a guid + -S: specifies the destination SEs to be used + -try <NumOfAttempts> Specifies the number of attempts to try and mirror the file +``` + +--- + +### mkdir +``` +usage: mkdir [-options] <directory> [<directory>[,<directory>]] +options: +-p : create parents as needed +-silent : execute command silently +``` + +--- + +### motd +``` +usage: motd Message of the day +``` + +--- + +### mv +``` +usage: mv <LFN> <newLFN> +``` + +--- + +### nano +``` +Command format: nano lfn +After editor termination the file will be uploaded if md5 differs +-datebck : the backup filename will be date based +``` + +!!! warning "client-side implementation" + +--- + +### packages +``` +usage: packages +list available packages +``` + +--- + +### pfn +``` +Command format : pfn [lfn] +It will print only the list of associtated pfns (simplified form of whereis) +``` + +!!! warning "client-side implementation" + + +--- + +### pfn-status +``` +Command format: pfn_status <pfn> +It will return all flags reported by the xrootd server - this is direct access to XRootD server +``` + +!!! warning "client-side implementation" + +!!! note "No connection to central servers needed" + +--- + +### ping +``` +ping <count> +where count is integer +``` + +!!! warning "client-side implementation" + +--- + +### popd +``` +It provides the mechanics of Directory Stack Builtins of bash +``` + +!!! warning "client-side implementation" + +--- + +### prompt +``` +Shell only function +Toggle the following in the command prompt : <date> for date information and <pwd> for local directory +``` + +!!! warning "client-side implementation" + +--- + +### ps +``` +usage: ps [-options] +options: +-F l | -Fl | -L : long output format +-f <flags|status> : any number of (long or short) upper case job states, or 'a' for all, 'r' for running states, 'f' for failed, 'd' for done, 's' for queued +-u <userlist> +-s <sitelist> +-n <nodelist> +-m <masterjoblist> +-o <sortkey> +-j <jobidlist> +-l <query-limit> + +-M : show only masterjobs +-X : active jobs in extended format +-A : select all owned jobs of you +-W : select all jobs which are waiting for execution of you +-E : select all jobs which are in error state of you +-a : select jobs of all users +-b : do only black-white output +-jdl <jobid> : display the job jdl +-trace <jobid> <tag>* : display the job trace information +-id: only list the matching job IDs, for batch processing (implies -b) +``` + +--- + +### pushd +``` +It provides the mechanics of Directory Stack Builtins of bash +``` + +!!! warning "client-side implementation" + +--- + +### pwd +``` +Print current work directory +``` + +--- + +### queryML +``` +usage: queryML <ML node> +time range can be specified for a parameter: +/[starting time spec]/[ending time spec]/parameter +where the two time specs can be given in absolute epoch timestamp (in milliseconds), as positive values, +or relative timestamp to `now`, when they are negative. +For example `-60000` would be "1 minute ago" and effectively `-1` means "now". +``` + +!!! warning "client-side implementation" + +--- + +### quota +``` +Client-side implementation that make use of server's jquota and fquota (hidden by this implementation) +Command format: quota [user] +if [user] is not provided, it will be assumed the current user +``` + +!!! warning "client-side implementation" + +--- + +### registerOutput +``` +usage: registerOutput <job ID> +options: +``` + +--- + +### resubmit +``` +resubmit: resubmits a job or a group of jobs by IDs +Usage: +resubmit <jobid1> [<jobid2>....] +``` + +--- + +### rmdir +``` +usage: rmdir [<option>] <directory> +options: +--ignore-fail-on-non-empty : ignore each failure that is solely because a directory is non-empty +-p : --parents Remove DIRECTORY and its ancestors. E.g., 'rmdir -p a/b/c' is similar to 'rmdir a/b/c a/b a'. +-v : --verbose output a diagnostic for every directory processed + : --help display this help and exit + : --version output version information and exit +-silent : execute command silently +``` + +--- + +### rm +``` +usage: rm <LFN> [<LFN>[,<LFN>]] +options: +-f : ignore nonexistent files, never prompt +-r, -R : remove directories and their contents recursively +-i : prompt before every removal (for JSh clients) +``` + +--- + +### run +``` +Command format: run shell_command arguments lfn +the lfn must be the last element of the command +N.B.! The output and error streams will be captured and printed at the end of execution! +for working within application use <edit> +additiona arguments recognized independent of the shell command: +-force : will re-download the lfn even if already present +-noout : will not capture output, the actual application can be used +``` + +!!! warning "client-side implementation" + +--- + +### SEqos +``` +Command format: SEqos <SE name> +Return the QOS tags for the specified SE (ALICE:: can be ommited and capitalization does not matter) +``` + +!!! warning "(partial) client-side implementation" + +--- + +### setSite +``` +usage: setSite [site name] +options: +``` + +--- + +### showTagValue +``` +usage: showtagValue [flags] <filename> [<filename>...] +options: +-t : restrict to this (comma separated) tag list only (default is to return all available tags) +-c : restrict to these (comma separated) list of attributes +-l : list available tags only +``` + +--- + +### stat +``` +usage: stat [-v] <filename1> [<or uuid>] ... +-v : More details on the status. +``` + +--- + +### submit +``` +usage: submit <URL> +<URL> => <LFN> +<URL> => file:///<local path> +``` + +--- + +### testSE +``` +Test the functional status of Grid storage elements +Usage: testSE [options] <some SE names, numbers or @tags> +-v : verbose error messages even when the operation is expected to fail +-c : show full command line for each test +-t : time each operation +-a : test all SEs (obviously a very long operation) +``` + +--- + +### time +``` +Command format: time command arguments +Print local (client side) command execution time (it includes the network delay, tomcat frontend and database backend) +``` + +!!! warning "client-side implementation" + +--- + +### timing +``` +usage: timing [on|off] +options: +enable printing of server side command execution time +``` + +--- + +### token-destroy +``` +Delete the token{cert,key}.pem files +``` + +!!! warning "client-side implementation" + +!!! note "No connection to central servers needed" + +--- + +### token-info +``` +Print token certificate information +``` + +!!! warning "client-side implementation" + +!!! note "No connection to central servers needed" + +--- + +### token-init +``` +usage: token-init [-options] +options: +-u <username> : switch to another role of yours +-v <validity (days)> : default depends on token type +-t <tokentype> : can be one of: job, jobagent, host, user (default) +-jobid <job DN extension> : expected to be present in a job token +-hostname <FQDN> : required for a host certificate +``` + +!!! warning "client-side implementation" + +--- + +### token +``` +Print only command!!! Use >token-init< for token (re)generation, see below the arguments +usage: token [-options] + +options: +-u <username> : switch to another role of yours +-v <validity (days)> : default depends on token type +-t <tokentype> : can be one of: job, jobagent, host, user (default) +-jobid <job DN extension> : expected to be present in a job token +-hostname <FQDN> : required for a host certificate +``` + +--- + +### token-verify +``` +Verify the token cert against the found CA stores (file or directory) +``` + +!!! warning "client-side implementation" + +!!! note "No connection to central servers needed" + +--- + +### tokenkey-match +``` +Check token private key againts token public cert +``` + +!!! warning "client-side implementation" + +!!! note "No connection to central servers needed" + +--- + +### top +``` +usage: top +``` + +### touch +``` +usage: touch <LFN> [<LFN>[,<LFN>]] +``` + +### toXml +``` +usage: toXml [-i] [-x xml_file_name] [-a] [-l list_from] [lfns] +options: +-i : ignore missing entries, continue even if some paths are not/no longer available +-x : write the XML content directly in this target AliEn file +-a : (requires -x) append to the respective collection +-l : read the list of LFNs from this file, one path per line +``` + +--- + +### type +``` +usage: type <lfn> +Print the LFN type (file / directory / collection) +``` + +--- + +### uptime +``` +usage: uptime +options: +``` + +--- + +### user +``` +usage: user <user name> +Change effective role as specified. +``` + +--- + +### uuid +``` +usage: uuid <uuid|filename> [<uuid|filename> ...] +Decode v1 UUIDs and display the interesting bits +``` + +--- + +### version +``` +usage: version +Get information about the JAliEn version that answers your requests +``` + +!!! warning "client-side implementation" + +!!! note "No connection to central servers needed" + +--- + +### vi +``` +Command format: vi lfn +After editor termination the file will be uploaded if md5 differs +-datebck : the backup filename will be date based +``` + +!!! warning "client-side implementation" + +--- + +### vim +``` +Command format: vim lfn +After editor termination the file will be uploaded if md5 differs +-datebck : the backup filename will be date based +``` + +!!! warning "client-side implementation" + +--- + +### whereis +``` +usage: whereis [-options] [<filename>] +options: +-g : use the lfn as guid +-r : resolve links (do not give back pointers to zip archives) +-l : lookup the LFN of the ZIP archive (slow and expensive IO operation, only use it sparingly!) +``` + +--- + +### whoami +``` +usage: whoami +options: +-v : verbose details of the current identity +``` + +--- + +### whois +``` +usage: whois [account name] +options: +-s : search for the given string(s) in usernames +-f : also search in full names +-e : search in email addresses too +-d : search in X509 DN (subject) fields +-a : search for the given string in all the above fields +``` + +--- + +### w +``` +usage: w Show currently active users on the Grid +``` + +### xrdstat +``` +usage: xrdstat [-d [-i]] [-v] [-p PID,PID,...] [-s SE1,SE2,...] [-c] <filename1> [<or UUID>] ... +options: +-d : Check by physically downloading each replica and checking its content. Without this a stat (metadata) check is done only. +-i : When downloading each replica, ignore `stat` calls and directly try to fetch the content. +-s : Comma-separated list of SE names to restrict the checking to. Default is to check all replicas. +-c : Print the full command line in case of errors. +-v : More details on the status. +-p : Comma-separated list of job IDs to check the input data of +-o : Only show the online status (for files with tape replicas in particular) +-O : Request the file to be brought online +-4 : Force IPv4 usage on all commands +-6 : Force IPv6 usage on all commands +``` + +--- diff --git a/docs/site/ce_guide.md b/docs/site/ce_guide.md index a42a25f558eb48a009af9898780fa316076bc80b..7642b9780604a0e9476d88d885029c2fc94cc26b 100644 --- a/docs/site/ce_guide.md +++ b/docs/site/ce_guide.md @@ -22,7 +22,7 @@ This approach only requires CVMFS and a valid Grid certificate or a user/host to 3. Run the CE: ```console - ~$ jalien ComputingElement + ~$ jalien alien.site.ComputingElement ``` This will start a JAliEn CE with default configuration options. @@ -133,4 +133,4 @@ The JAliEn CE log directory is defined in LDAP, the same way as for legacy AliEn When using option 1 (CE in CVMFS), the CE can be updated by simply reloading the CE and alienv, activating the latest tag. For option 2 (CE from git), the latest version can be obtained by doing a ```git pull``` and ```compile.sh all```, before restarting. -No action is needed for the JobAgents, as these are always started using the latest tag in CVMFS. +No action is needed for the JobAgents, as these are always started using the same (or closest available) tag as the CE through CVMFS. diff --git a/docs/site/vobox_container.md b/docs/site/vobox_container.md index 57ade2691c3d153e3847d6c8cc14e82a7ff08db6..da20c151f840088b031244e912fbc9b968742ae4 100644 --- a/docs/site/vobox_container.md +++ b/docs/site/vobox_container.md @@ -71,7 +71,7 @@ Create a new MACVLAN bridge with both IPv4 and IPv6 support, named __docknet__, __--ip/ip6__ sets the desired network IPs for the container !!! info - Docker gives containers limited access to the host kernel, which __may prevent some tools from working properly__. + Docker gives containers limited access to the host kernel, which __will prevent some tools from working properly__. A possible workaround is to give the container additional privileges (see [advanced topics](#privileged-mode) for details).<br><br> As opposed to mounting CVMFS from the host, it is also possible to have it installed normally within the container. However, this will only work on a container with full access privileges. @@ -153,5 +153,5 @@ This is known as an _autofs bug_, and can be avoided by disabling autofs on the If you prefer to leave autofs enabled, you can remove the error by accessing a directory within CVMFS on the host (e.g ```/cvmfs/alice.cern.ch```), and then restarting the container. !!! warning - Be aware that this error will likely return on the next reboot. + Be aware that this error will likely return on the next reboot if autofs is left enabled. diff --git a/docs/site/xrootd_rpms.md b/docs/site/xrootd_rpms.md new file mode 100644 index 0000000000000000000000000000000000000000..030cdd959418d51f2ec6b61a74695067ff64f81c --- /dev/null +++ b/docs/site/xrootd_rpms.md @@ -0,0 +1,179 @@ +# XRootD Installation (RPMs) + +The following documentation describes how to install XRootD using RPMs. +<br>The old page (with legacy instructions) can be found [here](https://alien.web.cern.ch/content/documentation/howto/site/xrootdbyrpms) + +!!! question "" + For questions and support, please contact [Adrian Sevcenco](../../user/help/#contact-us). + +## Packages Installation + +!!! info "Prerequisites" + These instructions assume EL7/CC7 x86_64 platform that is supported and tested. + <br>__Redirector on disk server configuration is not supported nor foreseen.__ + + +!!! danger "Preparation of storage locations" + It is assumed that the server is already prepared with the necessary directories and correct rights and ownership. + + +1. Install required tools (__root user is assumed__): + + ```bash + ~# yum install https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm + ``` + +2. Install the WLCG repository, for example for CC7: + + ```bash + ~# rpm --import http://linuxsoft.cern.ch/wlcg/RPM-GPG-KEY-wlcg + + ~# yum -y install http://linuxsoft.cern.ch/wlcg/centos7/x86_64/wlcg-repo-1.0.0-1.el7.noarch.rpm + ``` + +!!! info "Updating from XRootD v4 to v5 case" + Because of configuration fixes and not only the minimum version recommended is 5.1 + See upgrading below + + +3. Install the required packages + 3.1 If fresh install, install the metapackage that will pull XRootD as a dependency (when present in epel): + + ```bash + ~# yum -y install alicexrdplugins + ``` + + 3.2 For upgrade case it's best to remove all XRootD packages + + ```bash + ~# yum remove -y xrootd* + ``` + + Then one can install either the epel version or the upstream XRootD version + <br>(see Yum repositories section from [XRootD downloads](https://xrootd.slac.stanford.edu/dload.html)) + ```bash + ~# yum install -y xrootd-client xrootd-server + ``` + + The ALICE XRootD plugins are: + - `tokenauthz` (not dependent on XRootD, to be found in WLCG repository) + - `xrootd-alicetokenacc` (dependent on XRootD and `tokenauthz`) + - `xrootd-aggregatingname2name` (dependent on XRootD) + - `alicexrdplugins` (metapackage, will pull all required dependencies) + + In special cases (when a test/preview is desired), `xrootd-alicetokenacc`, `xrootd-aggregatingname2name`, `alicexrdplugins` can be downloaded from [here](https://asevcenc.web.cern.ch/asevcenc/xrootd5alice/) + otherwise they are usually found in WLCG repository. + + +## XRootD Management Script + +!!! warning "Change of management tools!!! (and configuration template)" + The former `xrd.sh` was re-wrote and splited in `xrdctl` service management script and `xrd_mk_config` configuration generation script + <br>See the github repository [alicexrd](https://github.com/adriansev/alicexrd) + <br>When upgrading recreate `system.cnf` file (it has new content/organization) and run `xrd_mk_config` for XRootD configuration generation. + + +!!! warning "" + Tested: __Running of XRootD as non-privileged user__ + <br>Not yet tested: use of `xrd_mk_config` generated systemd service files. + +1. Deployment and setting up the environment: + <br>`xrdctl` and `xrd_mk_config` scripts and configuration files can be automatically deployed from github as follows: + + ```bash + ~$ bash <(curl -fsSLk https://raw.githubusercontent.com/adriansev/alicexrd/master/deploy_alicexrd) + ``` + The script will place the scripts within `${XRDCTLDIR:=~/alicexrd}` and configurations in `${XRDCONFDIR:=~/alicexrdconf}` + + Within `XRDCONFDIR`: + <br>```~$ cp system.cnf_CHANGEME system.cnf``` + <br> Then set `SE_NAME, LOCALROOT, OSS_SPACE_DIRS` as all former others are detected automatically. + + +2. After completing the [configuration](#script-configuration) run the script: ```$XRDCTLDIR/xrdctl -c```. + <br>(one can symlink `xrdctl` into `$HOME/bin/`) + <br>Usage: ```xrdctl arg [configuration_file]```, where argument is _one_ of : + + | Argument | Description | + |----------|-------------| + | `-l | --l | -list | --list` | list found configuration files + | `-s | --s | -status | --status` | show status of services + | `-c | --c | -check | --check` | check and restart if not running + | `-k | --k | -kill | --kill` | kill running processes + | `-f | --f | -restart | --restart` | force restart + | `-logs | --logs` | manage the logs + | `-addcron | --addcron` | add/refresh cron line + | `-removecron | --removecron` | remove the cron line + + optionaly the configuration file can be explicitly declared. + + !!! note + The operations will be applied on + - specified configuration file + __OR__ + - on *ALL* `.xrdcfg` files from `$XRDCONFDIR` + + Environment variables for `xrdctl`: + + | Argument | Description | + |----------|-------------| + | XRD_READONLY | checked by xrootd configuration; if set the export (__for ALL found configurations__) will be declared as notwritable + | XRDCTL_CMD_XRD/XRDCTL_CMD_CMSD | xrootd and cmsd executables (other than the defaults from /usr/bin) + | XRDCTL_PRELOAD | use LD_PRELOAD for xrootd/cmsd processes + | XRDRUNDIR | location of admin,core,logs,logsbackup dirs; if not set it will be XRDCTLDIR/run/ + | XRDCONFDIR | location (directory) of .xrdcfg configuration file(s); if not set it wll be XRDCTLDIR/xrootd.conf/ + | XRDCTL_DEBUG | if set (any value) it will enable various printouts of xrdctl + | XRDCTL_DEBUG_SETX | bash enable -x + | XRDCTL_DEBUG_XRD | if set enable debug flag for xrootd process + | XRDCTL_DEBUG_CMSD | if set enable debug flag for cmsd process + + +## Script Configuration + +The following table highlights environment variables that influence how ```xrd_mk_config``` works: +<br>Usage: ```xrd_mk_config [template_configuration_file]``` +<br>where the specified argument is a custom path for template configuration (otherwise the `${XRDCONFDIR}/xrootd.xrootd.cf.tmp` is used) +<br>It will parse/source `${XRDCONFDIR}/system.cnf` and generate the xrootd configuration. + +!!! warning "`${XRD_MULTISRV}` usage" + If "`${XRD_MULTISRV}`" is set, it will enable the spawning of an xrootd/cmsd process __FOR EACH `OSS_SPACE_DIRS`__ + <br>In this mode, `oss.space` declaration will be commented/omitted, + and a distinct xrootd configuration file will be generated for each directory specified by `OSS_SPACE_DIRS` + +!!! note "" + `OSS_SPACE_DIRS` will be explicitly listed, so globbing will be expanded to all possible directories. + <br>Example: `/storage*/xrddata` will be expanded by `ls` to all __existing__ directories + + +| Environment Variable | Description | +|----------------------|-------------| +| XRDRUNDIR | location of admin,core,logs,logsbackup dirs; if not set it will be XRDCTLDIR/run/ +| XRDCONFDIR | location (directory) of .xrdcfg configuration file(s); if not set it wll be XRDCTLDIR/xrootd.conf/ + + + +## XRootD MonaLisa monitoring + +The monitoring of XRootD is done by MonaLisa agent - MLSensor +This is deployed as a distinct service, found within WLCG repository. + +```bash +~# yum -y install mlsensor +``` + +There is a required configuration of service (as root): +```bash +~# cd /etc/mlsensor +~# mlsensor_config mlsensor.properties.tmp <name of ALICE storage name> +``` + +!!! warning + if multiple XRootD services are used, make sure to modify `mlsensor.properties` file + <br>and update `lia.Monitor.modules.monXrdSpace.args` variable to a space separrated list of XRootD ports + + +then the normal system enablement procedure can take place +``` +systemctl enable mlsensor +systemctl start mlsensor +``` diff --git a/mkdocs.yml b/mkdocs.yml index c732a0734f61fa6fbe6d8ea48587715e0595a663..bf48a44be66a56ecf55b52f4d3da800f08efc117 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -36,12 +36,14 @@ nav: - Migration to JAliEn: user/migration.md - Troubleshooting: user/help.md - Site: - - VOBox: site/vobox.md - - VOBox (Container): site/vobox_container.md - - VOBox (HTCondor/ARC): site/vobox_htc_arc.md - - Manage VOBox Proxy: site/vobox_proxy.md - - "Monitoring: MonALISA": site/monalisa.md - - Computing Element Guide: site/ce_guide.md + - "VOBox": site/vobox.md + - "VOBox (Container)": site/vobox_container.md + - "VOBox (HTCondor/ARC)": site/vobox_htc_arc.md + - "Manage VOBox Proxy": site/vobox_proxy.md + - "Monitoring: MonALISA": site/monalisa.md + - "Computing Element Guide": site/ce_guide.md + - "XRootD Install (RPMs)": site/xrootd_rpms.md + - "XRootD Decommissioning": http://lcg.in2p3.fr/wiki/index.php?title=ALICE_native_xrootd - Reference: - alien.py: alienpy_commands.md - JAliEn: jalien_commands.md