Differences

This shows you the differences between two versions of the page.

--- lcrc:introduction [2018/02/22 14:22]
asc
+++ lcrc:introduction [2023/06/15 14:05]
rwang [Using interactive jobs]
@@ Line 26: / Line 26: @@
 heplogin2.lcrc.anl.gov  # login directly to hepd-0004
 </code>
-Each node has 72 CPUs and a lot of memory. After login, you will end up in a rather small "home" space which has a limit of 100 GB.
+Each node has 72 CPUs and a lot of memory. After login, you will end up in a rather small "home" space which has a limit of 500 GB.
 <code>
-/blues/gpfs/home/[USER]
+/home/[USER]
 </code>
-Apparently, LCRC cannot spend more than 50$/per users home,  since they too busy with leadership in computing. But you have other places  to store your precious source files **(we need to confirm backup policy and restoration procedure!)**. First, you can use this location:
+You can use this location to keep code etc. (but not data):
 <code>
 /lcrc/group/ATLAS/users/[USER]
 </code>
-You can go to your "home" directory  automatically during the  login using these lines:
-<code bash>
-export TRUEHOME=/lcrc/group/ATLAS/users/$USER
-echo "Changing /blues/gpfs/home/$USER to $TRUEHOME"
-cd $TRUEHOME
-export HOME=$TRUEHOME
-</code>
-in the file called ".bashrc".
@@ Line 118: / Line 108: @@
 <code bash>
-srun --pty -p  bdwall  -t 24:00:00 /bin/bash
+srun --pty -p  bdwall   -A condo -t 24:00:00 /bin/bash
 </code>
 It will allocate a new node (in bash) for 24h. It uses Xeon(R) CPU E5-2695 v4 @ 2.10GHz (36 CPUs per node).  More info about this can be found in [[http://www.lcrc.anl.gov/for-users/using-lcrc/running-jobs/running-jobs-on-bebop/|Running jobs on BeBob]]. Note that you should keep the terminal open while jobs are running.
- --- //[[Sergei&Doug]] 2018/01/04 13:36//
+When you use  bdwall partition, your jobs will accounted against default CPU allocations (100k per 4 months). Therefore, when possible, please use "hepd" partition.
+See the next section.
+===== Running Batch job on HEP resources =====
+<code bash>
+srun --pty -p hepd -t 24:00:00 /bin/bash
+module load StdEnv            # important to avoid slum bug
+</code>
+Then you can setup root etc as "source /soft/hep/setup.sh".
+SLURM is used as the batch system.  It does whole node scheduling (not "core scheduling")! If you run single core job, your allocation will be multiplied by 36 (cores!)
+Please see this page for details on how to use SLURM on LCRC http://www.lcrc.anl.gov/for-users/using-lcrc/running-jobs/running-jobs-on-bebop/
+The partion for the HEP nodes is  hepd
+To run on non HEP nodes use partition bdwall with Account - ATLAS-HEP-group
+==== Using interactive jobs ====
+First, allocate a HEP node:
+<code bash>
+salloc -N 1 -p hepd -A condo  -t 00:30:00
+</code>
+This allocates it for 30 min, but you can allocate it up to 7 days.
+You cam also allocate it on bebob:
+<code bash>
+salloc -N 1 -p bdwall --account=ATLAS-HEP-group -t 00:30:00
+</code>
+This does not login you!
+Check what node did you allocate
+<code bash>
+squeue -u user
+</code>
+Now you know the node. Then login to bebob (first!) and then ssh to this node.
+Another method is to use
+<code bash>
+srun --pty -p  bdwall  --account=ATLAS-HEP-group -t 00:30:00  /bin/bash
+</code>
+=== Running long interactive jobs ===
+See more description in: https://www.lcrc.anl.gov/for-users/using-lcrc/running-jobs/running-jobs-on-bebop/
+You should be able to do for example:
+<code>
+-ssh bebop
+-screen
+-salloc -N 1 -p hepd -A condo -t 96:00:00
+-ssh <nodename>
+-Work on interactive job for x amount of time...
+-Disconnect from screen (different than exit, see the documentation)
+-Logout
+</code>
+<code>
+-Login to the same login node screen was started on
+-screen -ls
+-Connect to screen session
+-Continue where you left off (if they allocation is still active)
+</code>
+See below for more details:
+https://www.gnu.org/software/screen/
+https://www.hamvocke.com/blog/a-quick-and-easy-guide-to-tmux/
+====== CVMFS repositories ======
+Mounted CVMFS repositories on Bebop and Swing computing node.
+<code>
+/cvmfs/atlas.cern.ch
+/cvmfs/atlas-condb.cern.ch
+/cvmfs/grid.cern.ch
+/cvmfs/oasis.opensciencegrid.org
+/cvmfs/sft.cern.ch
+/cvmfs/geant4.cern.ch
+/cvmfs/spt.opensciencegrid.org
+/cvmfs/dune.opensciencegrid.org
+/cvmfs/larsoft.opensciencegrid.org
+/cvmfs/config-osg.opensciencegrid.org
+/cvmfs/fermilab.opensciencegrid.org
+/cvmfs/icarus.opensciencegrid.org
+/cvmfs/sbn.opensciencegrid.org
+/cvmfs/sw.hsf.org
+</code>
+Note, they are not mounted on login nodes
+====== Using Singularity ======
+To run jobs on all LCRC resources using ATLAS analysis base requires Docker/Singularity.
+ Yiming (Ablet) Abulaiti created a tutorial on how to do this. {{:lcrc:analysisbaselcrc.pdf|Read this}}
+Here are the suggested steps for 21.2.51 release.
+<code>
+docker pull atlas/analysisbase:21.2.51
+</code>
+Then make singularity image:
+<code bash>
+docker run -v /var/run/docker.sock:/var/run/docker.sock -v `pwd`:/output --privileged -t --rm singularityware/docker2singularity:v2.3 atlas/analysisbase:21.2.51
+</code>
+Currently, the image for AtlasBase 2.2.51 located here:
+<code>
+/soft/hep/atlas.cern.ch/repo/containers/images/singularity/atlas_analysisbase_21.2.51-2018-11-04-01795eabe66c.img
+</code>
+You can go inside this image as:
+<code>
+singularity exec /soft/hep/atlas.cern.ch/repo/containers/images/singularity/atlas_analysisbase_21.2.51-2018-11-04-01795eabe66c.img bash -l
+</code>
+====== Using Singularity for cvmfsexec ======
+One can also setup cvmf on any LCRC nodes as this:
+<code>
+source /soft/hep/CVMFSexec/setup.sh
+</code>
+Then check:
+<code>
+ls /cvmfs/
+</code>
+You will see the mounted directory (SL7):
+<code>
+atlas-condb.cern.ch/      atlas.cern.ch/  cvmfs-config.cern.ch/  sft-nightlies.cern.ch/  sw.hsf.org/
+atlas-nightlies.cern.ch/  cms.cern.ch/    projects.cern.ch/      sft.cern.ch/            unpacked.cern.ch/
+</code>
+ --- //[[Sergei&Doug&Rui]] 2018/01/04 13:36//

ATLAS Support Center

User Tools

Site Tools

Differences

Page Tools