Create the tarball with cvmfs content directly

Create the tarball with the libraries and modulefiles from CVMFS directly,
without first copying that content in a scratch directory. This is important
to avoid generating many small files on the local file system, particularly
important if it is Lustre.
As a consequence the options and the code related to the scratch directory
have been removed.
Among the prerequisites the script checks now also that either it has been
called with sudo or with the --fakeroot flag in order to be able to build the
Singularity image.

README.md

-# Singularity contaiber for ALICE environment
+# Singularity container for ALICE environment
 
 ## Description
-A shell script to fetch the binaries from alice.cern.ch cvmfs repository for a specific environment tag and builds a singularity image for a standalone environment container
+A shell script to fetch the binaries from the ALICE CVMFS repository for a specific environment tag and recreating them inside a Singularity image for later standalone use.
 
 ## Prerequisities
-- singularity
-- cvmfs alice.cern.ch mount under /cvmfs/alice.cern.ch
-- linux OS (tested for centos7, other operating systems need to be tested)
+- **Singularity** is installed
+- The ALICE CVMFS repository is mounted under _/cvmfs/alice.cern.ch_
+- Either the script is run with sudo or with the '--fakeroot' flag for building the Singularity image
 
 ## Usage
-- git clone https://git.gsi.de/atay/alice_singularity
-- cd alice_singularity
-- sudo ./singularity_alice.sh -t <environment_tag> -o <working_directory> -y
-    - -t/--tag              
-        - specify a valid environment tag, for example "AliPhysics/vAN-20201018_ROOT6-1"
-    - -o/--output           
-        - specify a working directory, default: current directory
-    - -y/--yes              
-        - Yes to folder clean up. it will clean the content of working directory. It is not mandatory, however it will be asked by the program if not specified here
-    - -h/--help             
-        - displays help
-
-## Procedure explanation
-- The shell script checks for arguments passed by the command
-    - if no environment tag is specified, it exits
-    - if alice.cern.ch cvmfs repository is not mounted, it exists
-    - if singularity is not installed, it exits
-    - if no output is specified, it uses the current directory as the working directory
-- It checks the **<working_directory>/<environment_tag>** folder
-    - if not exists, it creates
-    - if exists, clean the content of the folder. It requires an authentication from the user and the authentication can also be given by -y/--yes tag within the command
-    - if exists and the user doesn't authenticate to clean the folder content, it exits
-- It creates "**<environment_tag>_modules.sh**" file to fetch $LD_LIBRARY_PATH from the environment
-- It executes the "<environment_tag>_modules.sh" within the singularity image "/cvmfs/alice.cern.ch/containers/fs/singularity/centos7" and fetch $LD_LIBRARY_PATH to "**<environment_tag>_LD_LIBRARY_PATH.txt**"
-- It loops over entries in "<environment_tag>_LD_LIBRARY_PATH.txt" file and detects package name and versions together with its path prefix. After it is done, it creates a copy of the binary packages under **<working_directory>/<environment_tag>/cvmfs** by preserving the symlinks. This folder represents a subresposiory of alice.cern.ch cvmfs repo for a specific environment tag
-- Fot simplicity, it copies all the modules files in the repo since they are just text files. Modules files are also needed to append binary paths to $LD_LIBRARY_PATH
-- It then compress the <working_directory>/<environment_tag>/cvmfs to be able copy into the image during build. 
-- It is now ready to build the image. It creates a "**<environment_tag>.def**" file.
-- Final stage is to build the image using "<environment_tag>.def" file.
+- `git clone https://git.gsi.de/r.grosso/alice_singularity`
+- `cd alice_singularity`
+- `sudo ./singularity_alice.sh -t <environment_tag> -o <working_directory> -c`
+-  or  `./singularity_alice.sh -t <environment_tag> -o <working_directory> -c -f`
+
+The script accepts the following options:
+- **-t**, **--tag**
+    - specify a valid AliEn environment tag, for example *AliPhysics/vAN-20201018_ROOT6-1*
+- **-f**, **--fakeroot**
+    - When building the Singularity image pass the option '--fakeroot' to build the image without being super user
+- **-h**, **--help**
+    - displays help
+
+## Procedure
+- The *alice_singularity.sh* script checks for arguments passed by the command. It exits with error:
+    - if no environment tag is specified,
+    - if alice.cern.ch cvmfs repository is not mounted,
+    - if Singularity is not installed or
+    - if neither is being run with `sudo` nor with `--fakeroot`
+- The tarball with the content extracted from CVMFS is built directly, without first copying into a scratch directory, to avoid generating many files on the local file system (probably Lustre)
+- By means of `alienv printenv` the *LD_LIBRARY_PATH* is retrieved and decomposed in its component directories
+- Each directory corresponding to a version of a given package in the *LD_LIBRARY_PATH* is added to the tar command
+- Also the modules file are added to the tar command which is evaluated producing the tarball
+- The definition file is used in the form of a here document and includes the cvmfs tarball created in the steps above and its untarring in the Singularity image
+
 ## Notes
-- The image file can be found under "**<working_directory>/<environment_tag>/<environment_tag>.sif**"
-- Image can be run with singularity "**singularity run -B /<some_path> <image.sif>**"
-    - Do not bind mount cvmfs repository with "-B /cvmfs" tag. This then will bind the host /cvmfs folder to the container /cvmfs folder. This can conflicts and prevents working. The idea is to use this container where cvmfs repository is not accessible. If you already have cvmfs mount already, you don't need container. 
-    - It will create a centos7 container with a virtual copy of alice.cern.ch cvmfs repository. However it only includes binaries for only the given environment.
-    - The environment can be initiated as usual: "**/cvmfs/alice.cern.ch/bin/alienv enter <environment_tag>**"
-        - If you have the image and are able to run it, then you don't need anything else to initiate the environment. No cvmfs mount, no ALICE account is needed. It is also possible to initiate the environment even without internet connection since you have all the binaries locally. 
-        - It is important to know that this only provides the environment, not the data or any kind of analysis code. User needs to provide it him/herself
\ No newline at end of file
+- The Singularity image file can be found under **<working_directory>/<environment_tag>/<environment_tag>.sif**
+- The image can be run with Singularity, e.g.:
+    - **singularity run -B /<some_path> <image.sif>** or
+    - **singularity exec -B /<some_path> <image.sif> /bin/bash**
+
+:warning: Do not bind mount _/cvmfs/_ with "-B /cvmfs" as this would overlap with the _/cvmfs/_ repository already generated inside the image generating conflicts. Indeed if CVMFS is mounted there is no point in using this container.
+
+- The Centos7 container only includes the binaries from the alice.cern.ch cvmfs repository for the environment tag specified to the building script.
+- The environment in the container can be initialized as usual: "**/cvmfs/alice.cern.ch/bin/alienv enter <environment_tag>**" (and for that no cvmfs nor an internet connection is needed on the host).
+- No additional data or analysis code is provided in the image.

singularity_alice.sh

 #!/bin/bash
 
-# Usage: sudo ./singularity_test.sh -t [<environment tag>] -o [<output path>]
-# Or:    ./singularity_test.sh -f -t [<environment tag>] -o [<output path>]
+# Usage: sudo ./singularity_test.sh -t [<environment tag>]
+# Or:    ./singularity_test.sh -f -t [<environment tag>]
 # Either super user privilegies or the option 'fakeroot' are needed to build a Singularity image.
 #
 # environment tag: <package_name>/<version> (mandatory field)
@@ -11,34 +11,29 @@
 #
 # Rearranged from Serhat Atay's work (atay@compeng.uni-frankfurt.de).
 
-TEMP=`getopt -o hyft:o: --long help,yes,fakeroot,tag:,output: -n singularity_alice -- "$@"`
+TEMP=`getopt -o hyft: --long help,yes,fakeroot,tag: -n singularity_alice.sh -- "$@"`
 if [ $? != 0 ] ; then echo "Terminating..." >&2 ; exit 1 ; fi
 eval set -- "$TEMP"
 
 help=false
 clean=false
 fakeroot=false
-# The default working directory is the current directory
-workdir=$(pwd)
 
 while true; do
     case "$1" in
         -h | --help ) help=true; shift ;;
-        -c | --clean ) clean=true; shift ;;
         -f | --fakeroot ) fakeroot=true; shift ;;
         -t | --tag ) tag="$2"; shift 2 ;;
-        -o | --output ) workdir="$2"; shift 2 ;;
         -- ) shift; break ;;
         * ) break ;;
     esac
 done
 
-if [ "$help" = "true"]; then
+if [[ "$help" = "true" ]]; then
     echo "options:"
     echo "-h, --help                        show brief help"
-    echo "-c, --clean                       Yes to folder clean up"
+    echo "-f, --fakeroot                    build the Singularity image with --fakeroot"
     echo "-t, --tag <tag>                   specify a valid environment tag: mandatory!"
-    echo "-o, --output <path>               specify a working directory (defaults to current directory)"
     exit
 fi
 
@@ -61,90 +56,53 @@ if ! command -v singularity &>/dev/null; then
     exit 1
 fi
 
-# remove trailing slash from workdir if present
-workdir=${workdir%/}
-
-# $tag is the environment tag in the format <package_name>/<version>. Replacing
-# '/' with '-' we get $environment which we use for naming the folders and files
-environment=${tag/\//-}
-echo "environment: $environment"
-
-# Create or clean the scratch directory under $workdir/.
-# Being environment specific, several scratch directories can coexist under the $workdir directory.
-# Exit if the scratch directory already exists, is not empty and the user does not allow to clean it.
-scratchdir=${workdir}/${environment}
-echo "scratchdir: $scratchdir"
-if ! [ -d $scratchdir ]; then
-    echo "making dir $scratchdir"
-    mkdir -p ${scratchdir}
-else
-    if ! [[ -z "$(ls -A $scratchdir)" ]]; then # if it is not empty
-        if [ "$clean" = "true" ]; then # if the permission to clean was already given as argument
-            echo "Deleting content of ${scratchdir}"
-            rm -r ${scratchdir}/*
-        else
-            echo "$scratchdir is not empty. Clean it? y/n:"
-            read cleaninput
-            for i in y Y yes YES Yes; do
-                if [[ $cleaninput == $i ]]; then
-                    clean=true
-                fi
-            done
-            if [ "$clean" = "true" ]; then
-                echo "Deleting content of ${scratchdir}"
-                rm -r ${scratchdir}/*
-            else  #dont clean the directory and exit
-                echo ${scratchdir} "Cannot continue without deleting the content of $scratchdir. Exiting..."
-                exit 1
-            fi
-        fi
+# Check that either sudo rights are available or the fakeroot flag has been given
+if [[ $EUID > 0 ]]; then
+    if ! [[ "$fakeroot" = "true" ]]; then
+        echo "Building the Singularity image requires either to run the script with sudo or with the '--fakeroot' flag. Exiting ..."
+        exit 1
     fi
 fi
 
-# Reproduce the needed libraries and modulefiles in the scratch directory:
+# $tag is the environment tag in the format <package_name>/<version>. We replace
+# '/' with '-' to get $environment which we use for naming the definition and image files
+environment=${tag/\//-}
+
+# Reproduce the needed libraries and modulefiles in the tarball (without intermediate copy)
+# 1. get LD_LIBRARY_PATH from 'alienv printenv'
 read -a allvars< <(/cvmfs/alice.cern.ch/bin/alienv printenv $tag)
 for i in ${allvars[@]}; do if [[ "$i" == *"LD_LIBRARY_PATH="* ]]; then ld_path=$i; break; fi; done;
 ld_path=${ld_path#*=}
 
-#loop to convert LD_LIBRARY_PATH entries to absolute paths of packages
+# 2. loop to add LD_LIBRARY_PATH entries to the 'tar' command
 delimiter=/Packages/
+tarcmd="tar -cvzf cvmfs.tar.gz "
 IFS=':'; for libdir in $ld_path; do
+    # The following assumes libdirs of type /cvmfs/.../Packages/SoftwareName/version/lib
     if [[ "$libdir" == *"singularity"* ]] || ! [[ "$libdir" == *"Packages"* ]]; then
         continue
     fi
-    # The following assumes libdirs of type /cvmfs/.../Packages/SoftwareName/version/lib
-    path="${libdir%$delimiter*}/Packages"
-    remaining=${libdir#*$delimiter}
-    package=${remaining%%/*}
-    version=${remaining#*/}
-    version=${version%/*}
-
-    # Synchronise the "Packages" directories from CVMFS into the image, preserving links (hoping that there is no absolute link).
-    #mkdir -p ${scratchdir}${path}/${package}
-    echo "mkdir SCRATCHDIR${path}/${package}"
-    #rsync -a ${path}/${package}/${version} ${scratchdir}${path}/${package}/
-    echo "rsync ${path}/${package}/${version} SCRATCHDIR"
-    # there must be no slash here -----------| because we want to preserve the versions directory in the new path
+    versionpath=${libdir%/*}
+    tarcmd="$tarcmd ${versionpath}"
 done
 
-# Copy modulefiles. For the time being all module files are copied.
+# 3. Add modulefiles to the 'tar' command
 for i in x86_64-2.6-gnu-4.1.2 el6-x86_64 el7-x86_64; do
     modulespath="/cvmfs/alice.cern.ch/$i/Modules"
-    mkdir -p ${scratchdir}${modulespath}
-    rsync -a $modulespath ${scratchdir}${modulespath}
+    tarcmd="$tarcmd ${modulespath}"
 done
 
-# folders 'etc' and 'bin' are needed to set the environment
-rsync -a /cvmfs/alice.cern.ch/etc ${scratchdir}/cvmfs/alice.cern.ch/
-rsync -a /cvmfs/alice.cern.ch/bin ${scratchdir}/cvmfs/alice.cern.ch/
+# 4. add 'etc' and 'bin' to the tar command
+tarcmd="$tarcmd /cvmfs/alice.cern.ch/etc /cvmfs/alice.cern.ch/bin"
 
-# Compress the copied respository. Later when extracting use '-h' to preserve symlinks
-cd $scratchdir
-tar -czvf cvmfs.tar.gz cvmfs/
+# 5. create the tarball by evaluating the tar command. Later when extracting use '-h' to preserve symlinks
+echo "Compressing the cvmfs content in a tarball ..."
+eval $(echo $tarcmd)
 
 # Create the Singularity definition file.
 # It is like /cvmfs/alice.cern.ch/containers/fs/singularity/centos7/.singularity.d/Singularity
 # with in addition the cvmfs tarball and its extraction.
+echo "Creating the Singularity definition file ..."
 cat > ${environment}.def <<EOF
 Bootstrap: docker
 From: centos:centos7
@@ -170,6 +128,7 @@ Author Serhat Atay
 EOF
 
 # Finally build the image
+echo "Building the Singularity image ..."
 if [[ "$fakeroot" = "true" ]]; then
     singularity build --fakeroot ${environment}.sif ${environment}.def
 else