Differences

This shows you the differences between two versions of the page.

--- lcrc:introduction [2018/01/09 00:43] – asc
+++ lcrc:introduction [2022/01/24 16:34] – [Using interactive jobs] asc
@@ Line 4: / Line 4: @@
 https://accounts.lcrc.anl.gov/  You will need to ask to join to the "ATLAS-HEP-group" by providing your ANL user name.
 The users are usually placed into the "g-ATLAS" group.
+Please also look at the description https://collab.cels.anl.gov/display/HEPATLASLCRC
 The description here uses the bash shell. Please go to https://accounts.lcrc.anl.gov/account.php to change your shell.
-Note that this resource cannot replace the ATLAS cluster and have several problems when it comes  day-to-day work:
+At this very moment (Jan 2018), this resource cannot replace the ATLAS cluster and have several features:
-  * LCRC resources are under maintenance on Monday of each week
+  * LCRC resources are under maintenance on Monday (each week?)
   * HOME directory is tiny (100GB) and you have to use some tricks to deal with it  (not well tested)
   * logins are done using the .ssh key (often requires 1 day to change with LCRC assist)
-  * cannot be mounted on desktops
+  * Cannot be mounted on desktops
 ====== Available resources ======
@@ Line 23: / Line 26: @@
 heplogin2.lcrc.anl.gov  # login directly to hepd-0004
 </code>
-Each node has 72 CPUs and a lot of memory. After login, you will end up in a rather small "home" space which has a limit of 100 GB.
+Each node has 72 CPUs and a lot of memory. After login, you will end up in a rather small "home" space which has a limit of 500 GB.
 <code>
-/blues/gpfs/home/[USER]
+/home/[USER]
 </code>
-Apparently, LCRC cannot spend more than 50$/per users home,  since they too busy with leadership in computing. But you have other places  to store your precious source files **(we need to confirm backup policy and restoration procedure!)**. First, you can use this location:
+You can use this location to keep code etc. (but not data):
 <code>
 /lcrc/group/ATLAS/users/[USER]
 </code>
-You can go to your "home" directory  automatically during the  login using these lines:
-<code bash>
-export TRUEHOME=/lcrc/group/ATLAS/users/$USER
-echo "Changing /blues/gpfs/home/$USER to $TRUEHOME"
-cd $TRUEHOME
-export HOME=$TRUEHOME
-</code>
-in the file called ".bashrc".
@@ Line 115: / Line 108: @@
 <code bash>
-srun --pty -p  bdwall  -t 24:00:00 /bin/bash
+srun --pty -p  bdwall   -A condo -t 24:00:00 /bin/bash
 </code>
 It will allocate a new node (in bash) for 24h. It uses Xeon(R) CPU E5-2695 v4 @ 2.10GHz (36 CPUs per node).  More info about this can be found in [[http://www.lcrc.anl.gov/for-users/using-lcrc/running-jobs/running-jobs-on-bebop/|Running jobs on BeBob]]. Note that you should keep the terminal open while jobs are running.
+When you use  bdwall partition, your jobs will accounted against default CPU allocations (100k per 4 months). Therefore, when possible, please use "hepd" partition.
+See the next section.
+===== Running Batch job on HEP resources =====
+<code bash>
+srun --pty -p hepd -t 24:00:00 /bin/bash
+module load StdEnv            # important to avoid slum bug
+</code>
+Then you can setup root etc as "source /soft/hep/setup.sh".
+SLURM is used as the batch system.  It does whole node scheduling (not "core scheduling")! If you run single core job, your allocation will be multiplied by 36 (cores!)
+Please see this page for details on how to use SLURM on LCRC http://www.lcrc.anl.gov/for-users/using-lcrc/running-jobs/running-jobs-on-bebop/
+The partion for the HEP nodes is  hepd
+To run on non HEP nodes use partition bdwall with Account - ATLAS-HEP-group
+==== Using interactive jobs ====
+First, allocate a HEP node:
+<code bash>
+salloc -N 1 -p hepd -A condo  -t 00:30:00
+</code>
+This allocates it for 30 min, but you can allocate it up to 7 days.
+You cam also allocate it on bebob:
+<code bash>
+salloc -N 1 -p bdwall --account=ATLAS-HEP-group -t 00:30:00
+</code>
+This does not login you!
+Check what node did you allocate
+<code bash>
+squeue -u user
+</code>
+Now you know the node. Then login to bebob (first!) and then ssh to this node.
+Another method is to use
+<code bash>
+srun --pty -p  bdwall  --account=ATLAS-HEP-group -t 00:30:00  /bin/bash
+</code>
+=== Running long interactive jobs ===
+See more description in: https://www.lcrc.anl.gov/for-users/using-lcrc/running-jobs/running-jobs-on-bebop/
+You should be able to do for example:
+<codde>
+-ssh bebop
+-screen
+-salloc -N 1 -p hepd -A condo -t 96:00:00
+-ssh <nodename>
+-Work on interactive job for x amount of time...
+-Disconnect from screen (different than exit, see the documentation)
+-Logout
+</code>
+<code>
+-Login to the same login node screen was started on
+-screen -ls
+-Connect to screen session
+-Continue where you left off (if they allocation is still active)
+</code>
+See below for more details:
+https://www.gnu.org/software/screen/
+https://www.hamvocke.com/blog/a-quick-and-easy-guide-to-tmux/
+====== Using Singularity ======
+To run jobs on all LCRC resources using ATLAS analysis base requires Docker/Singularity.
+ Yiming (Ablet) Abulaiti created a tutorial on how to do this. {{:lcrc:analysisbaselcrc.pdf|Read this}}
+Here are the suggested steps for 21.2.51 release.
+<code>
+docker pull atlas/analysisbase:21.2.51
+</code>
+Then make singularity image:
+<code bash>
+docker run -v /var/run/docker.sock:/var/run/docker.sock -v `pwd`:/output --privileged -t --rm singularityware/docker2singularity:v2.3 atlas/analysisbase:21.2.51
+</code>
+Currently, the image for AtlasBase 2.2.51 located here:
+<code>
+/soft/hep/atlas.cern.ch/repo/containers/images/singularity/atlas_analysisbase_21.2.51-2018-11-04-01795eabe66c.img
+</code>
+You can go inside this image as:
+<code>
+singularity exec /soft/hep/atlas.cern.ch/repo/containers/images/singularity/atlas_analysisbase_21.2.51-2018-11-04-01795eabe66c.img bash -l
+</code>
  --- //[[Sergei&Doug]] 2018/01/04 13:36//