Differences

This shows you the differences between two versions of the page.

--- asc:tutorials:2014october_connect [2014/10/09 19:23] – [Lesson 6: Working on the ANL farm] asc
+++ asc:tutorials:2014october_connect [2014/10/10 15:28] (current) – [Lesson 6: Using HTCondor and Tier2] asc
@@ Line 205: / Line 205: @@
 ====== Lesson 5: Running on multiple cores ======
-This example is not needed for ATLAS connect.
+This example is not needed for ATLAS connect. If you still want to know how to run an ATLAS analysis job on several cores of your desktop,
+look at [[asc:tutorials:2014october#lesson_5running_a_job_on_multiple_cores]]
-====== Lesson 6: Working on the ANL farm ======
+====== Lesson 6: Using HTCondor and Tier2 ======
-This example is ANL-specific. If you ran analysis code for RUN I, you should be a familiar with it. We used [[http://atlaswww.hep.anl.gov/asc/arcond/ | Condor/Arcod system]] to submit jobs to the farm. Data are distributed on the farm nodes to avoid IO bottleneck.
+Lesson 5: Working on a Tier3 farm (Condor queue)
-<note important>This example works on the ANL Tier3 where Arcond/Condor is installed</note>
+In this example we will use HTCondor workload management system to send the job to be executed in a queue at a Tier3 farm.  For this example we will start from the directory lesson 4, so if you did not do the lesson 4 please do that one first and verify that your code runs locally.
+Start from the new shell and set up environment, then create this shell script that will be executed at the beginning of each job at each farm node:
-First, create a directory for this example:
+<file bash startJob.sh>
+#!/bin/bash
+export RUCIO_ACCOUNT=YOUR_CERN_USERNAME
+export ATLAS_LOCAL_ROOT_BASE=/cvmfs/atlas.cern.ch/repo/ATLASLocalRootBase
+source ${ATLAS_LOCAL_ROOT_BASE}/user/atlasLocalSetup.sh
+localSetupFAX
+source $AtlasSetup/scripts/asetup.sh 19.0.2.1,noTest
+export X509_USER_PROXY=x509up_u21183
+unzip payload.zip
+ls
+rcSetup -u; rcSetup Base,2.0.12
+rc find_packages
+rc compile
+cd MyAnalysis/util
+rm submitDir
+echo $1
+sed -n $1,$(($1+2))p inputdata.txt > Inp_$1.txt
+cp Inp_$1.txt inputdata.txt
+cat inputdata.txt
+echo "startdate $(date)"
+testRun submitDir
+echo "enddate $(date)"
+</file>
-Prepare a fresh directory:
+Make sure the RUCIO_ACCOUNT variable is properly set. Make this file executable and create the file that describes our job needs and  that we will give to condor:
-<code>
-mkdir lesson_6
-</code>
-Copy the needed directories with RootCore example:
+<file bash job.sub>
+Jobs=10
+getenv         = False
+executable     = startJob.sh
+output         = MyAnal_$(Jobs).$(Process).out
+error          = MyAnal_$(Jobs).$(Process).error
+log            = MyAnal_$(Jobs).$(Process).log
+arguments = $(Process) $(Jobs)
+environment = "IFlist=$(IFlist)"
+transfer_input_files = payload.zip,/tmp/x509up_u21183,MyAnalysis/util/inputdata.txt
+universe       = vanilla
+#Requirements   = HAS_CVMFS =?= True
+queue $(Jobs)
+</file>
-<code bash>
+To access files using FAX the jobs need a valid grid proxy. That's why we send it with each job. Proxy is the file starting with "x509up" so in both  job.sub and startJob.sh you should change "x509up_u21183" with the name of your grid proxy file. The filename you may find in the environment variable $X509_USER_PROXY.
-cp -r /users/chakanau/public/2014_tutorial_october_anl/lesson_6/*  lesson_6/
-</code>
-Then check that you can compile the analysis example using RootCore:
+You need to pack all of the working directory into a payload.zip file:
-<code bash>
+<file bash startJob.sh>
-cd lesson_6/
+startJob.sh
-source setup.sh
+rc clean
-</code>
+rm -rf RootCoreBin
+zip -r payload.zip *
+</file>
-As usual, our analysis is in "MyAnalysis/util/testRun.cxx"
+Now you may submit your task for the execution and follow its status in this way:
-Now we want to submit jobs to the data distributed on several computers of the farm. Go to the upper directory and setup the farm:
-<code bash>
-cd ../; source s_asc;
-</code>
-We will send jobs using the directory "submit". Go to this directory and check the data:
-<code bash>
-cd submit
-arc_ls -s /data2/valid2/117050/PowhegPythia_P2011C_ttbar.digit.AOD.e2657_s1933_s1964_r5534/
-arc_ls    /data2/valid2/117050/PowhegPythia_P2011C_ttbar.digit.AOD.e2657_s1933_s1964_r5534/
-</code>
-The first command shows the summary of distributed data (12 files per server), while the second lists all available data on each node.
-Now we will send the job to the farm.
-Change the line:
 <code>
-package_dir=/users/chakanau/public/2014_tutorial_october_anl/lesson_6/ANLanalysis
+chmod 755 ./startJob.sh; ./startJob.sh;
 </code>
-inside "arcond.conf" to reflect the correct path to your program. Then run "arcond" and say "y" to all questions.
-This sends jobs to the farm (2 jobs per server). Check the status as:
 <code bash>
-condor_q
+~> condor_submit job.sub
+Submitting job(s)..........
+job(s) submitted to cluster 49677.
+~> condor_q ivukotic
+-- Submitter: login.atlas.ci-connect.net : <192.170.227.199:60111> : login.atlas.ci-connect.net
+ ID      OWNER            SUBMITTED     RUN_TIME ST PRI SIZE CMD
+.0   ivukotic       10/9  10:21   0+00:00:11 R  0   0.0  startJob.sh 0
+.1   ivukotic       10/9  10:21   0+00:00:11 R  0   0.0  startJob.sh 1
+.2   ivukotic       10/9  10:21   0+00:00:11 R  0   0.0  startJob.sh 2
+.3   ivukotic       10/9  10:21   0+00:00:11 R  0   0.0  startJob.sh 3
+.4   ivukotic       10/9  10:21   0+00:00:11 R  0   0.0  startJob.sh 4
+.5   ivukotic       10/9  10:21   0+00:00:11 R  0   0.0  startJob.sh 5
+.6   ivukotic       10/9  10:21   0+00:00:11 R  0   0.0  startJob.sh 6
+.7   ivukotic       10/9  10:21   0+00:00:11 R  0   0.0  startJob.sh 7
+.8   ivukotic       10/9  10:21   0+00:00:11 R  0   0.0  startJob.sh 8
+.9   ivukotic       10/9  10:21   0+00:00:10 R  0   0.0  startJob.sh 9
+jobs; 0 completed, 0 removed, 0 idle, 10 running, 0 held, 0 suspended
 </code>
-When the jobs are done, the output files will be inside "Jobs" directory. Merge the ROOT outputs into one file as:
-<code bash>
-arc_add
-</code>
-This will create the final output file "Analysis_all.root"