NICA Cluster
OS: CentOS Linux 7
Exp. Software: CVMFS, /cvmfs/bmn.jinr.ru/centos7/
EOS Storage: 1 PB replicated, /eos/nica/bmn/
SLURM: 6 312 slots (Xeon log. cores)
Official Website
NICA cluster machines are available for users of the NICA collaborations. To login, use your account for ncx machines (ncx[101-106].jinr.ru), e.g.:
ssh -X [username]@ncx.jinr.ru
The detailed rules of using the NICA cluster are listed on the official NCX web site here. Please, read it carefully.
The necessary packages for BmnRoot, FairSoft and FairRoot are located on the CVMFS file system. To use them, load the environment script every time you enter the NICA cluster. In other way, you can add the command to the end of ~/.bashrc file to be automatically executed every time at login.
source /cvmfs/bmn.jinr.ru/config/x86_64-centos7/cluster_config.sh
Installation of the BmnRoot software is described here at the section “Installing the BmnRoot framework”.
The cluster storage system includes the following user disk spaces:
- User home directory: /lhep/users/$USER, which does not exceed 50 GB.
- User directory on NCX EOS file system: /eos/nica/bmn/users/$USER, where to get a necessary EOS space, you can request for the working directory by email to the software coordinator (gertsen@jinr.ru). Please, write your ncx login and a desired size of your working directory in the mail.
- Scratch spaces, /scratch1 and /scratch2, which are deployed on SAS HDDs and can be used for data processing (faster than the main EOS storage). There are no time limitations for /scratch1 and /scratch2 disks, but make sure that they are not ~100% full.
- Hot (fast) storage, /junk and /scratch – a very fast temporary storage system on SSDs, where files are automatically deleted (for /junk - after 4 days, for /scratch - after 2 weeks) if they have not been accessed.
- Additional disk storage is available at /nica/bmn1 disk (130 TB).
- Temporary space: /tmp – a fast local disk for intermediate calculation data, do not forget to delete files after the end of your task.
The BM@N experimental and simulated data are located at the NCX EOS distributed storage system (EOS documentation) mounted at /eos/nica/bmn/exp/ and /eos/nica/bmn/sim/ respectively. To work with EOS data, in addition to the possibility of direct using the mounted spaces, you can also employ XRootD and EOS protocols from any remote place, e.g. xrdcp and eos cp command to copy EOS files, or use root://ncm.jinr.ru/ prefix to directly open EOS files in the ROOT environment (like root://ncm.jinr.ru//eos/nica/bmn/exp/…).
To run your tasks on the NICA cluster, you can use SLURM batch system. If you know how to work with SLURM (SLURM on JINR CICC), you can use sbatch command on the cluster to distribute data processing. A simple example of such user job for SLURM can be found here.
NICA-Scheduler has been developed to simplify running of user tasks without knowledge of SLURM. You can find how to use NICA-Scheduler here.