Skip to content
Snippets Groups Projects

[Virgo-3] Slurm plugin needs to be recompiled

    • Open Issue created by Matteo Dessalvi @m.dessalvi 
    sbatch: error: spank: /usr/lib64/slurm/singularity-exec.so: Dlopen of plugin file failed
sbatch: error: spank: /etc/slurm/plugstack.conf.d/singularity-exec.conf:1: Failed to load plugin /usr/lib64/slurm/singularity-exec.so. Aborting.
sbatch: error: Failed to initialize plugin stack
    Feb 26 16:53:01 lxbk0724 slurmd[3826834]: slurmd: error: spank: /usr/libexec/slurm-singularity-exec.so: Incompatible plugin version
(...)
    • Status: SPANK plugin needs to be re-compiled for the current version of Slurm and re-packaged.
    Edited

    Designs

      Child items ...

      2. Activity

      • Matteo Dessalvi changed the description

        changed the description

        • Dennis Klein
          Dennis Klein @d.klein ·
          Owner
          • The plugin failed to be loaded anyway:
          Feb 26 16:53:01 lxbk0724 slurmd[3826834]: slurmd: error: spank: /usr/libexec/slurm-singularity-exec.so: Incompatible plugin version
(...)
          • Status: SPANK plugin needs to be re-compiled for the current version of Slurm and re-packaged.

          Re-compiling is actually not enough. There is also a small change needed in the source code. See https://github.com/SchedMD/slurm/commit/db697131b0350f74b90a37259a1c583e8475ea95.

          The needed change was submitted to us by Michael Mayer. I moved the plugin repo to https://github.com/GSI-HPC/slurm-singularity-exec for easier future collaboration with non-GSI folks (following the discussion in the last cluster meeting). The patch by Michael Mayer which solves the "Incompatible plugin version" error, is applied here https://github.com/GSI-HPC/slurm-singularity-exec/pull/1 and already released as version v3.0.0.

          I am updating the rpm package now.

          Edited by Dennis Klein
        • Dennis Klein
          Dennis Klein @d.klein ·
          Owner

          New packages slurm-singularity-exec-3.0.0-1 are available now (for both el9 and el8).

        • Dennis Klein
          Dennis Klein @d.klein ·
          Owner 
          ❯ rpm -qipl --changelog el9/slurm/packages/slurm-singularity-exec-3.0.0-1.el9.x86_64.rpm
Name        : slurm-singularity-exec
Epoch       : 1
Version     : 3.0.0
Release     : 1.el9
Architecture: x86_64
Install Date: (not installed)
Group       : Unspecified
Size        : 323757
License     : GPLv3
Signature   : RSA/SHA256, Tue 27 Feb 2024 05:19:20 PM CET, Key ID 6925eae660912ad4
Source RPM  : slurm-singularity-exec-3.0.0-1.el9.src.rpm
Build Date  : Tue 27 Feb 2024 05:19:06 PM CET
Build Host  : delta
Summary     : Slurm SPANK plugin to start Singularity/Apptainer containers
Description :
The Singularity SPANK plug-in provides the users with an interface to launch an
application within a Singularity container. The plug-in adds multiple
command-line options to the salloc, srun and sbatch commands. These options are
then propagated to a shell script slurm-singularity-wrapper.sh customizable by
the cluster administrator.
* Tue Feb 27 2024 Dennis Klein <d.klein@gsi.de> - 1:3.0.0-1
- Update to 3.0.0

* Tue Jan 30 2024 Dennis Klein <d.klein@gsi.de> - 1:2.0.0-3
- Install config to `/usr/share/slurm` because `/etc/slurm` may be an NFS
  share

* Wed Jan 17 2024 Dennis Klein <d.klein@gsi.de> - 1:2.0.0-2
- Rebuilt for rocky+epel-9

* Tue Jan 16 2024 Dennis Klein <d.klein@gsi.de> - 1:2.0.0-1
- Add epoch to fix sorting order with old versioning scheme

* Mon Jan 15 2024 Dennis Klein <d.klein@gsi.de> - 2.0.0-1
- Update to 2.0.0

* Fri Feb 24 2023 Victor Penso <v.penso@gsi.de> 21.08-1
- Replace Singularity with Apptainer as run-time dependency

* Fri Jul 22 2022 Victor Penso <v.penso@gsi.de> 21.08
- Build package against Slurm version 21.08

/usr/lib/.build-id
/usr/lib/.build-id/ef
/usr/lib/.build-id/ef/efb2966b2d74038c6877371d5ec43df98a6d8e
/usr/libexec/slurm-singularity-exec.so
/usr/libexec/slurm-singularity-wrapper.sh
/usr/share/doc/slurm-singularity-exec
/usr/share/doc/slurm-singularity-exec/README.md
/usr/share/licenses/slurm-singularity-exec
/usr/share/licenses/slurm-singularity-exec/LICENSE
/usr/share/slurm/plugstack.conf.d/singularity-exec.conf
          ❯ rpm -qipl --changelog el8/slurm/packages/slurm-singularity-exec-3.0.0-1.el8.x86_64.rpm
Name        : slurm-singularity-exec
Epoch       : 1
Version     : 3.0.0
Release     : 1.el8
Architecture: x86_64
Install Date: (not installed)
Group       : Unspecified
Size        : 311640
License     : GPLv3
Signature   : RSA/SHA256, Tue 27 Feb 2024 05:38:37 PM CET, Key ID 6925eae660912ad4
Source RPM  : slurm-singularity-exec-3.0.0-1.el8.src.rpm
Build Date  : Tue 27 Feb 2024 05:38:25 PM CET
Build Host  : delta
Summary     : Slurm SPANK plugin to start Singularity/Apptainer containers
Description :
The Singularity SPANK plug-in provides the users with an interface to launch an
application within a Singularity container. The plug-in adds multiple
command-line options to the salloc, srun and sbatch commands. These options are
then propagated to a shell script slurm-singularity-wrapper.sh customizable by
the cluster administrator.
* Tue Feb 27 2024 Dennis Klein <d.klein@gsi.de> - 1:3.0.0-1
- Update to 3.0.0

* Tue Jan 30 2024 Dennis Klein <d.klein@gsi.de> - 1:2.0.0-2
- Install config to `/usr/share/slurm` because `/etc/slurm` may be an NFS
  share

* Tue Jan 16 2024 Dennis Klein <d.klein@gsi.de> - 1:2.0.0-1
- Add epoch to fix sorting order with old versioning scheme

* Mon Jan 15 2024 Dennis Klein <d.klein@gsi.de> - 2.0.0-1
- Update to 2.0.0

* Fri Feb 24 2023 Victor Penso <v.penso@gsi.de> 21.08-1
- Replace Singularity with Apptainer as run-time dependency

* Fri Jul 22 2022 Victor Penso <v.penso@gsi.de> 21.08
- Build package against Slurm version 21.08

* Mon Jan 17 2022 Victor Penso <v.penso@gsi.de> 1.0
- First versions to be packaged

/usr/lib/.build-id
/usr/lib/.build-id/b1
/usr/lib/.build-id/b1/2a9483733236016bfbc1665467bc81f1fe01bf
/usr/libexec/slurm-singularity-exec.so
/usr/libexec/slurm-singularity-wrapper.sh
/usr/share/doc/slurm-singularity-exec
/usr/share/doc/slurm-singularity-exec/README.md
/usr/share/licenses/slurm-singularity-exec
/usr/share/licenses/slurm-singularity-exec/LICENSE
/usr/share/slurm/plugstack.conf.d/singularity-exec.conf
        • Please register or sign in to reply
        • Matteo Dessalvi
          Matteo Dessalvi @m.dessalvi ·
          Author Maintainer

          Status Update

          • New version of the plugin has been installed.
          • Login on vae24.hpc.gsi.de allows to test the new VAE.
          • Submit a test job:
          sbatch: error: plugin_load_from_file: dlopen(/usr/libexec/slurm-singularity-exec.so): /usr/libexec/slurm-singularity-exec.so: cannot open shared object file: No such file or directory
sbatch: error: spank: /usr/libexec/slurm-singularity-exec.so: Dlopen of plugin file failed
sbatch: error: spank: /etc/slurm/plugstack.conf.d/singularity-exec.conf:1: Failed to load plugin /usr/libexec/slurm-singularity-exec.so. Aborting.
sbatch: error: Failed to initialize plugin stack
          • The following options must be changed (under /etc/default/sshd_container):
          (...)
export SSHD_CONTAINER_OPTIONS="--bind /etc/slurm,/usr/libexec,/var/run/munge,/var/spool/slurm,/var/lib/sss/pipes/nss,/cvmfs"
(...)
          • /usr/libexec must be now binded directly (previous .so file was under /usr/lib64/slurm/)

          Job failed

          • A test job failed with the following error:
          ~$ cat 487_lxbk0726.err.log 
FATAL:   stat /var/spool/slurmd/job00487/slurm_script: no such file or directory
          • On lxbk0726 the directory is there but it seems that the client was not able to write into it:
          [root@lxbk0726 ~]# ls -la /var/spool/slurm/slurmd/
total 4
drwxr-xr-x. 2 root  root  24 Feb 12 14:37 .
drwxr-xr-x. 4 slurm slurm 35 Feb 12 13:49 ..
-rw-------. 1 root  root   8 Feb 12 14:37 cred_state
          • Submission script used:
          at hostname_test.sh 
#!/bin/bash
 
# Task name
#SBATCH -J hostname_test
 
# Run time limit
#SBATCH --time=00:00:45

# Working directory on shared storage
#SBATCH -D /lustre/hpc/dessalvi/Slurm_scripts/simple_tests

#SBATCH --mail-user=m.dessalvi@gsi.de                                                                                                                                       
#SBATCH --mail-type=ALL

# Standard and error output in different files
#SBATCH -o %j_%N.out.log
#SBATCH -e %j_%N.err.log

# Execute application code

hostname ; uptime ; sleep 30 ; uname -a
          • Submitted as sbatch hostname_test.sh.
          Edited by Matteo Dessalvi
        • Dennis Klein
          Dennis Klein @d.klein ·
          Owner

          /var/spool/slurm/slurmd/ is the old one, or? we use /var/spool/slurmd now?

        • Dennis Klein
          Dennis Klein @d.klein ·
          Owner

          ya, so it also needs to be adjusted in the bind mount list

        • Matteo Dessalvi
          Matteo Dessalvi @m.dessalvi ·
          Author Maintainer

          we use /var/spool/slurmd now?

          Current status (from one exe nodes) is as follows:

          [root@lxbk0726 ~]# ls -l /var/spool/
(...)
drwxr-xr-x. 4 slurm slurm   35 Feb 12 13:49 slurm
drwxr-xr-x. 4 root  root  4096 Feb 29 18:47 slurmd

          Launching jobs from bare metal works without issues, apparently only from VAEs the job switch to the other directory?

        • Dennis Klein
          Dennis Klein @d.klein ·
          Owner

          apparently only from VAEs the job switch to the other directory?

          No, we use SlurmdSpoolDir=/var/spool/slurmd always - this is a change from virgo-2 and a workaround because changing this directory produces bugs in current slurm.

          Edited by Dennis Klein
        • Dennis Klein
          Dennis Klein @d.klein ·
          Owner

          but /etc/slurm/plugstack.conf.d/singularity-exec.conf only binds /var/spool/slurm, so /var/spool/slurmd is missing.

          Edited by Dennis Klein
        • Matteo Dessalvi
          Matteo Dessalvi @m.dessalvi ·
          Author Maintainer

          @d.klein Indeed, you are absolutely right. I have changed the bind mount option in SSHD_CONTAINER_OPTIONS under /etc/default/sshd_container and now I am able to run my job from vae24.

        • Please register or sign in to reply
      Please register or sign in to reply

      https://git.gsi.de is provided by CIT→Linux&Web | GSI Helmholtzzentrum fuer Schwerionenforschung GmbH | Imprint (in German) | Privacy policy