Heterogeneous platform “HybriLIT”
OS: CERN CentOS 7
Exp. Software: CVMFS, /cvmfs/bmn.jinr.ru/centos7/
MICC EOS Storage: /eos/eos.jinr.ru/nica/bmn/
NCX EOS Storage: /eos/eosnica.jinr.ru/nica/bmn/
SLURM: 7680 Xeon cores + 6048 Xeon Phi cores + 40 GPU NVidia Tesla V
SLURM BM@N queue: 184 Xeon cores ('bmn')
Official Website
The HybriLIT heterogeneous platform is a computational component of the Multifunctional Center of the Laboratory of Information Technologies of JINR for data processing and analysis, which contains a multi-core component and computation accelerators: Intel Xeon Phi co-processors and NVIDIA graphics cards. The detailed information on the platform, rules and registration form can be found on the JINR HybriLIT Web-site.
With the HybriLIT team an agreement has been reached on using the heterogeneous cluster and Govorun supercomputer for BM@N tasks. Once you have registered in the HybriLIT complex, you can login with your account:
ssh [username]@hydra.jinr.ru
Please, carefully read the important notes after login.
The GNU Scientific Library (gsl) is missing on the Govorun supercomputer, therefore before compiling and using BmnRoot it is necessary to run every time the following command:
module add GSL/v1.16-1
The necessary packages for BmnRoot, FairSoft and FairRoot are located on the CVMFS file system. To use them, load the environment script every time you enter the NICA cluster. In other way, you can add the command to the end of ~/.bashrc file to be automatically executed every time at login.
source /cvmfs/bmn.jinr.ru/config/x86_64-centos7/cluster_config.sh
Installation of the BmnRoot software is described here at the section “Installing the BmnRoot framework”.
To run your tasks on the Govorun supercomputer of the HybriLIT platform, you can use SLURM batch system.
For an access to Govorun run the following command:
module add GVR/v1.0-1
The Govorun provides common queues, such as cascade queue with 7680 Intel Xeon processor cores , knl queue with 6048 Intel Xeon Phi coprocessor cores, and dgx queue with 40 GPU NVidia Tesla V cards. And there is a special queue for the BM@N experiment with 184 Intel Xeon physical CPU cores – bmn (please, add "-p bmn -q bmn" parameters to your job description to send tasks to the queue).
If you know how to work with SLURM (SLURM on HybriLIT), you can use sbatch command on the cluster to distribute data processing. A simple example of such user job for SLURM can be found here.
NICA-Scheduler has been developed to simplify running of user tasks without knowledge of SLLURM. You can discover how to use NICA-Scheduler here.
The storage system of the HybriLIT platform includes the following user disk spaces:
- User home directory: /lustre/home/user/[a-z]/[username], which should not be used for processing.
- User space on the EOS file system: /eos/hybrilit.jinr.ru/user/[username],
- EOS space of the MICC (LIT) cluster mounted at /eos/eos.jinr.ru/nica/bmn/users/[username],
- EOS space of the NCX cluster mounted at /eos/eosnica.jinr.ru/nica/bmn/users/[username],
- Scratch EOS space: /eos/hybrilit.jinr.ru/scratch/ – an EOS storage space for intermediate calculation data.
- Scratch local space: /run/user/$UID – a fast local storage system for temporary results.
The BM@N experimental and simulated data are located at the EOS distributed storage system (EOS documentation) of the central MICC EOS storage mounted at /eos/eos.jinr.ru/nica/bmn/ and NCX cluster mounted at /eos/eosnica.jinr.ru/nica/bmn/.
To work with EOS data, in addition to the possibility of direct using the mounted space, you can also employ XRootD and EOS protocols from any remote place, e.g. xrdcp and eos cp command to copy EOS files, or use root://ncm.jinr.ru/ (for NCX EOS), root://eos.jinr.ru/ (for MICC EOS) and root://eos-hlit.jinr.ru/ (for HybriLIT EOS) prefixes to directly open remote EOS files in the ROOT environment (like root://eos-hlit.jinr.ru//eos/hybrilit.jinr.ru/nica/…).
Attention: the HybriLIT platform and LIT MICC complex still have different Kerberos authorization. As a result, there are no permissions for direct access to the central MICC EOS storage. To have the access to eos.jinr.ru, you need to have an account on lxpub, an account on HybriLIT and the same GID (group id) on HybriLIT as on lxpub, then perform the following steps on HybriLIT:
1) Set the following variables: export KRB5_CONFIG=/etc/krb5.conf.jinr export EOS_MGM_URL=root://eos.jinr.ru 2) Run the following commands: kinit [lxpub-user]@JINR.RU eos ls /eos
In case of problems during work with the Govorun or HybriLIT platform, you can open an issue for the HybriLIT support team here.