ATC: ATLAS Center at ANL

User Tools

Site Tools

Trace: • Sherpa Notes
hpc:sherpa

Table of Contents

Sherpa Notes

Sherpa Homepage

wget http://www.hepforge.org/archive/sherpa/SHERPA-MC-2.1.1.tar.gz

In most cases opening Makefile and commenting out line 470/471 which have the words Manual and Examples on it will save you some time.

«< I frequently have to do 'autoreconf' before 'make' when doing a fresh install. Not sure why, just do. »>

Some Notes

  • The code generation step of Sherpa produces a makelibs compile script and many other Makefiles.in in the Process subdirectory that has full absolute path inside that points to the local Sherpa installation. Many of the configure* and automake files contain absolute paths to the directory in which the code generation step was executed. This causes problems if one tries to move the code generation to another system or folder.
  • 200900 = W+jets, 200901 = Z+jets, 200903 = Drell-Yan, 200910 = Multi-jets
  • During a run Sherpa dlopen() calls the following libraries:
    • LibCT10Sherpa.so (or whatever PDF you are using)
    • libSherpaOpenLoops.so
    • libProc2_2…
    • libopenloops.so
    • openloops/proclib/*.so
    • libSherpaHepMcOutput.so
  • Runtime Errors and Possible Meanings:
    • 'Couldn't find virtual ME for this process' → look that it has the libSherpaOpenLoops.so and the proper openloops libraries.
    • 'Process not found' → make sure Run.dat card is found, and if so, make sure Processes are defined in the card properly.

Compilation and Installation at Edison@NERSC

Local Environment

Install HepMC.

After logging into a clean shell run 

module unload altd
module unload darshan
module unload PrgEnv-intel
module load PrgEnv-gnu
module unload cray-shmem
module load scons
module load mercurial
module load cmake
module load boost
module load gsl

Compilation

Unpack Sherpa tarball and enter sherpa main directory. After configure comment out line 470 of the Makefile for Sherpa which has the word “Manual” on it. You can create a mostly-static or dynamic executable by using the '-static' and '–enable-static=yes', or '-dynamic' flags, respectively. Then run: <WRAP prewrap> 

./configure --enable-mpi --enable-gzip --with-sqlite3=/usr --enable-hepmc2=</path/to/hepmc2> --enable-openloops=</path/to/openloops> --prefix=$PWD/install CC=cc CXX=CC FC='ftn -fPIC -funderscoring'
make -j 10
make install

</WRAP>

If you go for the static flags the make will fail just before the executable is built so then do this:

<WRAP prewrap> 

./configure --enable-mpi --enable-hepmc2=/project/projectdirs/m2015/tools/HepMC-2.06.08/local/ --with-pic  LDFLAGS=-static MPICXX=CC CC=cc CXX=CC --prefix=$PWD/install --enable-openloops=/project/projectdirs/m2015/tools/OpenLoops/ --enable-gzip --enable-static=yes --enable-shared=yes
make -j 10
make -j 10 install
cd SHERPA/Run
../../libtool --tag=CXX --mode=link CC -Wl,--no-as-needed -g -O2  -rdynamic -Wl,--no-as-needed  -o Sherpa ../../*/*/*.o ../../*/*/*/*.o /project/projectdirs/m2015/tools/HepMC-2.06.08/src/*.o -lgfortran /usr/lib64/libz.a -lsqlite3 > libtool.log
# this will fail with errors then run this
sed s/"libtool: link: "//g libtool.log | sed s/"..\/..\/PDF\/CT10\/CT10_Fortran_Interface.o"//g | sed s/"..\/..\/PDF\/CT10\/CT10Pdf.o "//g | sed s/"..\/..\/PDF\/CT10\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/CT12\/CT12_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/CT12\/CT12Pdf.o "//g | sed s/"..\/..\/PDF\/CT12\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/CTEQ\/CTEQ6_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/CTEQ\/Cteq6Pdf-2008.o "//g | sed s/"..\/..\/PDF\/CTEQ\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/Electron\/PDF_Electron.o "//g | sed s/"..\/..\/PDF\/Electron\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/GRV\/GRVph_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/GRV\/grvphoton.o "//g | sed s/"..\/..\/PDF\/GRV\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst2001lo.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst2004qed.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst99.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST01LO.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST04QED.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST99.o "//g | sed s/"..\/..\/PDF\/MRST\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/MSTW\/mstwpdf.o "//g | sed s/"..\/..\/PDF\/MSTW\/PDF_MSTW.o "//g | sed s/"..\/..\/PDF\/MSTW\/SVN_Info.o "//g | sed s/"\/opt\/gcc\/4.9.2\/snos\/lib\/..\/lib64\/libquadmath.so"/"-lquadmath"/g | sed s/"\/opt\/gcc\/4.9.2\/snos\/lib\/..\/lib64\/libgfortran.so"/"-lgfortran"/g | sed s/"\/usr\/lib64\/libsqlite3.so"/"-lsqlite3"/g > compile_Sherpa.sh
sh compile_Sherpa.sh
cp Sherpa ../../install/bin/

</WRAP>

Test Run

  • Ensure proper modules are loaded as listed above.
  • Create directory in Examples/test.
  • Change to this folder and create the file Run.dat and fill it with this text: 
    (run){
 % general setting
 EVENTS 1; ERROR 0.99;
 EVENT_OUTPUT=HEPEVT[specialtest];
 
 % scales, tags for scale variations
 SP_NLOCT 1; FSF:=1.; RSF:=1.; QSF:=1.;
 SCALES METS{FSF*MU_F2}{RSF*MU_R2}{QSF*MU_Q2};
 
 % tags for process setup
 NJET:=2; LJET:=2; QCUT:=20.;
 
 % me generator settings
 ME_SIGNAL_GENERATOR Comix Amegic LOOPGEN;
 EVENT_GENERATION_MODE Weighted;
 LOOPGEN:=Internal;
 
 % exclude tau from lepton container
 MASSIVE[15] 1;
 
 % collider setup
 BEAM_1 2212; BEAM_ENERGY_1 = 4000.;
 BEAM_2 2212; BEAM_ENERGY_2 = 4000.;
}(run)
 
(processes){
 Process 93 93 -> 90 91 93{NJET};
 Order_EW 2; CKKW sqr(QCUT/E_CMS);
 NLO_QCD_Mode MC@NLO {LJET};
 ME_Generator Amegic {LJET};
 Loop_Generator LOOPGEN {LJET};
 Scales LOOSE_METS{FSF*MU_F2}{RSF*MU_R2}{QSF*MU_Q2} {7,8};
 End process;
}(processes)
 
(selector){
 Mass 11 -12 5. E_CMS
 Mass 13 -14 5. E_CMS
 Mass -11 12 5. E_CMS
 Mass -13 14 5. E_CMS
}(selector)
}(run)
  • Start an interactive session: qsub -I -q default -l mppwidth=24
  • Once this session begins you'll be back in your home directory on the worker node.
  • Change directories back: cd $PBS_O_WORKDIR.
  • Next run the initialization step of Sherpa which produces the makelibs file: aprun -n 1 -N 1 ../../install/bin/Sherpa
  • Once complete, exit the interactive session and run ./makelibs

Compilation and Installation at Cooley@ALCF

  • soft add +mvapich2
  • soft add +gcc-4.8.1
  • Unpack tarball and enter main sherpa directory.<WRAP prewrap>
    mkdir install
./configure  --enable-gzip --enable-openloops=/projects/EnergyFEC/anl/cooley/tools/OpenLoops-r1354 --enable-hepmc2=/projects/EnergyFEC/anl/cooley/tools/HepMC-2.06.09/install --with-sqlite3=/projects/EnergyFEC/anl/cooley/tools/sqlite-autoconf-3080200/install --enable-mpi --enable-static=yes --prefix=$PWD/install

    </WRAP>

  • In the Makefile remove line 472 because tukey doesn't have the 'texi2html' app installed needed to compile the manual.
    SUBDIRS = \
      ATOOLS \
      BEAM \
      PDF \
      MODEL \
      METOOLS \
      PHASIC++ \
      MCATNLO \
      EXTRA_XS \
      AMEGIC++ \
      COMIX \
      CSSHOWER++ \
      AMISIC++ \
      AHADIC++ \
      HADRONS++ \
      PHOTONS++ \
      SHRiMPS \
      SHERPA \
      AddOns \
      Manual \   # <---- this line
      Examples
  • Make & Install:
    make
make install
  • Install Scons. If you've installed OpenLoops, scons is in /path/to/OpenLoops/scons-local. Scons is used to compile the process libraries, the code for which is written when you first call Sherpa with your run card. If you don't have scons, it tries to use autoconfig which can have difficulties sometimes whereas Scons does not. 
    wget http://prdownloads.sourceforge.net/scons/scons-2.3.4.tar.gz
# Unpack tarball and enter main scons directory
mkdir install
python setup.py install --prefix=$PWD/install
export PATH=$PATH:$PWD/install/bin

Compiling on Mira/Cetus@ALCF

  • <WRAP prewrap> 
    soft add +mpiwrapper-gcc
soft add +bgqtoolchain-gcc484
soft add +python-2.7
svn export svn+ssh://atlas1.hep.anl.gov/users/svn/hpc/generators/sherpa/v220/dev/tags/01-00-00 01-00-00
cd 01-00-00
./configure --enable-mpi --with-pic --enable-gzip=/soft/libraries/alcf/current/gcc/ZLIB --enable-openloops=</path/to/openloops> --with-sqlite3=install --enable-hepmc2=</path/to/hepmc2> --host=powerpc64-bgq-linux CC=mpicc CFLAGS=-g CPPFLAGS=-g CXXFLAGS=-g MPICXX=mpicxx CXX=mpicxx FC='mpif90 -fPIC -funderscoring' FCFLAGS=-g LDFLAGS=-dynamic --prefix=$PWD/install
# build and install sqlite3
# rerun configure with: --with-sqlite3=</path/to/sqlite3>
make -j 20 install

    </WRAP>

  • Hopefully, now everything is working.
  • You may need to copy the HepMC and sqlite3 libraries to the installed sherpa library directory.
  • I have run into these problems in the past:
    • <WRAP prewrap> 
       /projects/EnergyFEC/anl/generators/sherpa/v220/rel-2-2-0_anl/SHERPA/PerturbativePhysics/.libs/libSherpaPerturbativePhysics.so: undefined reference to `std::__detail::_List_node_base::_M_hook(std::__detail::_List_node_base*)'
/projects/EnergyFEC/anl/generators/sherpa/v220/rel-2-2-0_anl/SHERPA/PerturbativePhysics/.libs/libSherpaPerturbativePhysics.so: undefined reference to `std::__detail::_List_node_base::_M_unhook()'
/projects/EnergyFEC/anl/generators/sherpa/v220/rel-2-2-0_anl/SHERPA/PerturbativePhysics/.libs/libSherpaPerturbativePhysics.so: undefined reference to `std::__detail::_List_node_base::_M_transfer(std::__detail::_List_node_base*, std::__detail::_List_node_base*)'
/projects/EnergyFEC/anl/generators/sherpa/v220/rel-2-2-0_anl/AHADIC++/Formation/.libs/libAhadicFormation.so: undefined reference to `std::__detail::_List_node_base::_M_reverse()'

      </WRAP>

Compiling on Vesta@ALCF

<WRAP prewrap> 

svn export svn+ssh://atlas1.hep.anl.gov/users/svn/hpc/generators/sherpa/v211/bgq/tags/02-01-00 sherpa-02-01-00
./configure --host=powerpc64-unknown-linux-gnu --with-sqlite3=install --enable-gzip=/soft/libraries/alcf/current/gcc/ZLIB --enable-openloops=/projects/ATLASQ/ol-mpi --enable-hepmc2=/home/projects/ATLASQ/tools/HepMC-2.06.08/install --enable-mpi --enable-shared=yes --prefix=$PWD/install --with-pic CC=mpicc CFLAGS=-pg LDFLAGS=-pg CPPFLAGS=-pg FC=mpif90 FCFLAGS=-pg MPICXX=mpicxx CXX=mpicxx CXXFLAGS=-pg

</WRAP> This step will fail because it cannot compile sqlite3 without some help. So go in the sqlite3 directory and run the following. <WRAP prewrap> 

mkdir install
./configure --enable-static=yes --prefix=$PWD/install
make
make install

</WRAP> Then move back to Sherpa root path and rerun the configure step. <WRAP prewrap> 

./configure --host=ppc64 --enable-gzip=/soft/libraries/alcf/current/gcc/ZLIB --enable-openloops=/projects/ATLASQ/ol-mpi --enable-hepmc2=/home/projects/ATLASQ/tools/HepMC-2.06.08/install --enable-mpi --enable-shared=yes --prefix=$PWD/install --with-pic CC=mpicc CFLAGS=-pg LDFLAGS=-pg CPPFLAGS=-pg FC=mpif90 FCFLAGS=-pg MPICXX=mpicxx CXX=mpicxx CXXFLAGS=-pg
make V=1 -j 10 > make.log 2>&1 &
make -j 10 install

</WRAP>

For static build: <WRAP prewrap> 

/bin/sh ../../libtool  --tag=CXX   --mode=link mpicxx -Wl,--no-as-needed -g -O2 -pg  -pg -dynamic -o Sherpa ../../*/*/*.o ../../*/*/*/*.o /home/projects/ATLASQ/tools/HepMC-2.06.08/src/*.o /home/projects/ATLASQ/generators/sherpa-02-01-00/sqlite-autoconf-3080200/install/lib/libsqlite3.a /soft/libraries/alcf/current/gcc/ZLIB/lib/libz.a -L/projects/ATLASQ/ol-mpi/lib -lopenloops -lcoli > libtool.log

</WRAP> This will fail because the PDF libraries contain overloaded functions and cannot be all loaded together. Sherpa loads these dynamically so the sed command below removes all the PDF object files. <WRAP prewrap> 

sed s/"libtool: link: "//g libtool.log | sed s/"..\/..\/PDF\/CT10\/CT10_Fortran_Interface.o"//g | sed s/"..\/..\/PDF\/CT10\/CT10Pdf.o "//g | sed s/"..\/..\/PDF\/CT10\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/CT12\/CT12_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/CT12\/CT12Pdf.o "//g | sed s/"..\/..\/PDF\/CT12\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/CTEQ\/CTEQ6_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/CTEQ\/Cteq6Pdf-2008.o "//g | sed s/"..\/..\/PDF\/CTEQ\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/Electron\/PDF_Electron.o "//g | sed s/"..\/..\/PDF\/Electron\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/GRV\/GRVph_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/GRV\/grvphoton.o "//g | sed s/"..\/..\/PDF\/GRV\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst2001lo.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst2004qed.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst99.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST01LO.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST04QED.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST99.o "//g | sed s/"..\/..\/PDF\/MRST\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/MSTW\/mstwpdf.o "//g | sed s/"..\/..\/PDF\/MSTW\/PDF_MSTW.o "//g | sed s/"..\/..\/PDF\/MSTW\/SVN_Info.o "//g > compile_Sherpa.sh
sh compile_Sherpa.sh

</WRAP>

Now to run a test. <WRAP prewrap> 

qsub -t 20 -n 1 --mode=c1 Sherpa -l run.codegen.log INIT_ONLY=1
qsub -t 20 -n 1 --mode=c4 Sherpa -l run.integration.log SQLITE_OPEN_FLAG=unix-none

</WRAP>

The SQLITE_OPEN_FLAG is explained here: https://sherpa.hepforge.org/trac/ticket/263?cversion=3&cnum_hist=2

Compiling on Ascinode

<WRAP prewrap> 

tar xzf SHERPA-MC-2.1.1.tar.gz
cd SHERPA-MC-2.1.1
./configure  --enable-gzip --enable-openloops=/users/hpcusers/tools/OpenLoops --enable-hepmc2=/users/hpcusers/svn/tools/HepMC-2.06.08/local  --enable-mpi --enable-static=yes --enable-shared=yes --prefix=$PWD/install --with-pic
make V=1 -j 10 > make.log 2>&1 &
make install

</WRAP> This will create the Sherpa executable but will have a dynamically linked executable. As seen here: 

ldd SHERPA/Run/.libs/Sherpa
	linux-vdso.so.1 =>  (0x00007ffff4bff000)
	libSherpaMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libSherpaMain.so.0 (0x00007f83841ea000)
	libSherpaInitialization.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libSherpaInitialization.so.0 (0x00007f8383fbe000)
	libSherpaPerturbativePhysics.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libSherpaPerturbativePhysics.so.0 (0x00007f8383d62000)
	libSherpaSoftPhysics.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libSherpaSoftPhysics.so.0 (0x00007f8383b36000)
	libLundTools.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libLundTools.so.0 (0x00007f8383935000)
	libSherpaTools.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libSherpaTools.so.0 (0x00007f838369b000)
	libAmisic.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAmisic.so.0 (0x00007f838348e000)
	libAmisicModel.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAmisicModel.so.0 (0x00007f8383251000)
	libAmisicTools.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAmisicTools.so.0 (0x00007f838303c000)
	libCSTools.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libCSTools.so.0 (0x00007f8382e30000)
	libCSCalculators.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libCSCalculators.so.0 (0x00007f8382c04000)
	libCSShowers.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libCSShowers.so.0 (0x00007f83829d2000)
	libCSMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libCSMain.so.0 (0x00007f83827af000)
	libAhadicMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAhadicMain.so.0 (0x00007f83825a6000)
	libAhadicTools.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAhadicTools.so.0 (0x00007f8382326000)
	libAhadicFormation.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAhadicFormation.so.0 (0x00007f8382114000)
	libAhadicDecays.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAhadicDecays.so.0 (0x00007f8381f10000)
	libShrimpsMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libShrimpsMain.so.0 (0x00007f8381d01000)
	libShrimpsEvents.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libShrimpsEvents.so.0 (0x00007f8381ac1000)
	libShrimpsBeamRemnants.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libShrimpsBeamRemnants.so.0 (0x00007f8381894000)
	libShrimpsXsecs.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libShrimpsXsecs.so.0 (0x00007f8381674000)
	libShrimpsEikonals.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libShrimpsEikonals.so.0 (0x00007f838144c000)
	libShrimpsTools.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libShrimpsTools.so.0 (0x00007f838123d000)
	libBeam.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libBeam.so.0 (0x00007f838101b000)
	libComixAmplitude.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libComixAmplitude.so.0 (0x00007f8380dd3000)
	libComixPhasespace.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libComixPhasespace.so.0 (0x00007f8380b8b000)
	libComix.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libComix.so.0 (0x00007f8380952000)
	libComixCluster.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libComixCluster.so.0 (0x00007f8380738000)
	libAmegic.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAmegic.so.0 (0x00007f83804cc000)
	libDipoleSubtraction.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libDipoleSubtraction.so.0 (0x00007f838024e000)
	libAmplitude.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAmplitude.so.0 (0x00007f837ffc6000)
	libAmegicPSGen.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAmegicPSGen.so.0 (0x00007f837fd1d000)
	libZfunctions.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libZfunctions.so.0 (0x00007f837fa76000)
	libString.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libString.so.0 (0x00007f837f83e000)
	libAmegicCluster.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAmegicCluster.so.0 (0x00007f837f61f000)
	libModelMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libModelMain.so.0 (0x00007f837f38c000)
	libModelInteractions.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libModelInteractions.so.0 (0x00007f837efc4000)
	libPDF.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPDF.so.0 (0x00007f837eda6000)
	libRemnant.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libRemnant.so.0 (0x00007f837eb8e000)
	libMCatNLOTools.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMCatNLOTools.so.0 (0x00007f837e988000)
	libMCatNLOCalculators.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMCatNLOCalculators.so.0 (0x00007f837e771000)
	libMCatNLOShowers.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMCatNLOShowers.so.0 (0x00007f837e548000)
	libMCatNLOMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMCatNLOMain.so.0 (0x00007f837e329000)
	libPhasicChannels.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhasicChannels.so.0 (0x00007f837e05e000)
	libPhasicMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhasicMain.so.0 (0x00007f837de0b000)
	libPhasicProcess.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhasicProcess.so.0 (0x00007f837db7e000)
	libPhasicSelectors.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhasicSelectors.so.0 (0x00007f837d8e3000)
	libPhasicScales.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhasicScales.so.0 (0x00007f837d694000)
	libPhasicEnhance.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhasicEnhance.so.0 (0x00007f837d48a000)
	libPhasicDecays.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhasicDecays.so.0 (0x00007f837d271000)
	libMEToolsExplicit.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMEToolsExplicit.so.0 (0x00007f837d046000)
	libMEToolsCurrents.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMEToolsCurrents.so.0 (0x00007f837ce16000)
	libMEToolsVertices.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMEToolsVertices.so.0 (0x00007f837cbea000)
	libMEToolsColors.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMEToolsColors.so.0 (0x00007f837c9db000)
	libMEToolsSpinCorrelations.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMEToolsSpinCorrelations.so.0 (0x00007f837c7ce000)
	libMEToolsLoops.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMEToolsLoops.so.0 (0x00007f837c5ab000)
	libMEToolsMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMEToolsMain.so.0 (0x00007f837c378000)
	libHadronsMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libHadronsMain.so.0 (0x00007f837c148000)
	libHadronsMEs.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libHadronsMEs.so.0 (0x00007f837bf0a000)
	libHadronsCurrents.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libHadronsCurrents.so.0 (0x00007f837bc7e000)
	libHadronsPSs.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libHadronsPSs.so.0 (0x00007f837ba68000)
	libPhotonsMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhotonsMain.so.0 (0x00007f837b85d000)
	libPhotonsTools.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhotonsTools.so.0 (0x00007f837b63b000)
	libPhotonsPhaseSpace.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhotonsPhaseSpace.so.0 (0x00007f837b430000)
	libPhotonsMEs.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhotonsMEs.so.0 (0x00007f837b1ee000)
	libExtraXS.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libExtraXS.so.0 (0x00007f837afdb000)
	libExtraXS2_2.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libExtraXS2_2.so.0 (0x00007f837ad95000)
	libExtraXS1_2.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libExtraXS1_2.so.0 (0x00007f837ab8f000)
	libExtraXS1_3.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libExtraXS1_3.so.0 (0x00007f837a984000)
	libExtraXSCluster.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libExtraXSCluster.so.0 (0x00007f837a781000)
	libExtraXSNLO.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libExtraXSNLO.so.0 (0x00007f837a55b000)
	libgfortran.so.3 => /usr/lib64/libgfortran.so.3 (0x00007f837a24e000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00000031bde00000)
	libSherpaSingleEvents.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libSherpaSingleEvents.so.0 (0x00007f8379feb000)
	libToolsPhys.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libToolsPhys.so.0 (0x00007f8379d8f000)
	libToolsMath.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libToolsMath.so.0 (0x00007f8379ada000)
	libToolsOrg.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libToolsOrg.so.0 (0x00007f837983d000)
	libz.so.1 => /lib64/libz.so.1 (0x00000031bea00000)
	libsqlite3.so.0 => /usr/lib64/libsqlite3.so.0 (0x00000031cf600000)
	libmpi_cxx.so.1 => /usr/lib64/openmpi/lib/libmpi_cxx.so.1 (0x00007f8379622000)
	libmpi.so.1 => /usr/lib64/openmpi/lib/libmpi.so.1 (0x00000031bf600000)
	libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00000031c0e00000)
	libm.so.6 => /lib64/libm.so.6 (0x00000031be200000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000031c0600000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00000031bda00000)
	libc.so.6 => /lib64/libc.so.6 (0x00000031bd600000)
	/lib64/ld-linux-x86-64.so.2 (0x00000031bd200000)
	libopen-rte.so.7 => /usr/lib64/openmpi/lib/libopen-rte.so.7 (0x00007f83793a6000)
	libopen-pal.so.6 => /usr/lib64/openmpi/lib/libopen-pal.so.6 (0x00000031bf200000)
	librt.so.1 => /lib64/librt.so.1 (0x00000031be600000)
	libnsl.so.1 => /lib64/libnsl.so.1 (0x00000031c1a00000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00000031cd600000)
	libltdl.so.7 => /usr/lib64/libltdl.so.7 (0x00000031c6200000)

To build a more statically linked executable do the following steps. <WRAP prewrap> 

cd SHERPA/Run
../../libtool --tag=CXX   --mode=link mpic++ -Wl,--no-as-needed -g -O2  -rdynamic -Wl,--no-as-needed  -o Sherpa ../../*/*/*.o ../../*/*/*/*.o /users/hpcusers/svn/tools/HepMC-2.06.08/src/*.o -lgfortran /usr/lib64/libz.a -lsqlite3 > libtool.log
sed s/"libtool: link: "//g libtool.log | sed s/"..\/..\/PDF\/CT10\/CT10_Fortran_Interface.o"//g | sed s/"..\/..\/PDF\/CT10\/CT10Pdf.o "//g | sed s/"..\/..\/PDF\/CT10\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/CT12\/CT12_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/CT12\/CT12Pdf.o "//g | sed s/"..\/..\/PDF\/CT12\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/CTEQ\/CTEQ6_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/CTEQ\/Cteq6Pdf-2008.o "//g | sed s/"..\/..\/PDF\/CTEQ\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/Electron\/PDF_Electron.o "//g | sed s/"..\/..\/PDF\/Electron\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/GRV\/GRVph_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/GRV\/grvphoton.o "//g | sed s/"..\/..\/PDF\/GRV\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst2001lo.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst2004qed.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst99.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST01LO.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST04QED.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST99.o "//g | sed s/"..\/..\/PDF\/MRST\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/MSTW\/mstwpdf.o "//g | sed s/"..\/..\/PDF\/MSTW\/PDF_MSTW.o "//g | sed s/"..\/..\/PDF\/MSTW\/SVN_Info.o "//g > compile_Sherpa.sh
sh compile_Sherpa.sh
cp Sherpa ../../install/bin/

</WRAP> Now you have: 

ldd install/bin/Sherpa
	linux-vdso.so.1 =>  (0x00007fff8e5ff000)
	libgfortran.so.3 => /usr/lib64/libgfortran.so.3 (0x00007f63da4cc000)
	libsqlite3.so.0 => /usr/lib64/libsqlite3.so.0 (0x00000031cf600000)
	libmpi_cxx.so.1 => /usr/lib64/openmpi/lib/libmpi_cxx.so.1 (0x00007f63da2b1000)
	libmpi.so.1 => /usr/lib64/openmpi/lib/libmpi.so.1 (0x00000031bf600000)
	libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00000031c0e00000)
	libm.so.6 => /lib64/libm.so.6 (0x00000031be200000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000031c0600000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00000031bda00000)
	libc.so.6 => /lib64/libc.so.6 (0x00000031bd600000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00000031bde00000)
	libopen-rte.so.7 => /usr/lib64/openmpi/lib/libopen-rte.so.7 (0x00000031bea00000)
	libopen-pal.so.6 => /usr/lib64/openmpi/lib/libopen-pal.so.6 (0x00000031bf200000)
	librt.so.1 => /lib64/librt.so.1 (0x00000031be600000)
	libnsl.so.1 => /lib64/libnsl.so.1 (0x00000031c1a00000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00000031cd600000)
	libltdl.so.7 => /usr/lib64/libltdl.so.7 (0x00000031c6200000)
	/lib64/ld-linux-x86-64.so.2 (0x00000031bd200000)

Patch Notes

Apply in this order:

  • gzip_2.patch
  • mpi_seed_mode_1.patch
  • one_lib_1.patch
  • remove_summarize_1.patch
  • resdir_1.patch

Sometimes I have trouble applying these so just use a text editor to see what changes need to be applied.

Converting Sherpa HepMC_GenEvent Format to POOL EVNT Format

Hack to Athena Algs

It was found that certain Truth containers are not created in Athena Reco transforms if the HepMC::GenEvent::signal_process_id is not greater than zero. Therefore the following hack describes a way to introduce this.

I was using the following commands on ascinode.hep.anl.gov with Athena setup with the environment asetup 17.2.12.7.10,MCProd,64,slc5, though this should work for other releases as well. Then do the following from your Athena home directory where the InstallArea will be created 

cmt show versions PhysicsAnalysis/TruthParticleID/McParticleTools

Which ever version is shown for your relase do the following: 

cmt co McParticleTools-00-19-00 PhysicsAnalysis/TruthParticleID/McParticleTools
cd McParticleTools-00-19-00 PhysicsAnalysis/TruthParticleID/McParticleTools/src/

Edit the file named HepMcReaderTool.cxx: 

StatusCode HepMcReaderTool::read( HepMC::GenEvent* evt )
{
  m_ioFrontend->fill_next_event(evt);
 
  ////////// BEGIN NEW PARTS ///////////////////
  // required to be non-zero for TruthParticleContainer creation
  evt->set_signal_process_id(1);
  // taken from Generators/Sherpa_i/trunk/src/Sherpa_i.cxx:115
  // ATLAS renormalizes the output Sherpa event weights
  if (evt->weights().size()>2) {
    evt->weights()[0]/=evt->weights()[2];
    while (evt->weights().size()>1) evt->weights().pop_back();
  }
  // Sherpa uses GeV, ATLAS uses MeV so convert all the momenta and energies.
  for (HepMC::GenEvent::particle_iterator p = evt->particles_begin(); p != evt->particles_end(); ++p) {
    // ATH_MSG_DEBUG("PDG, BAR " << (*p)->pdg_id() << " " << (*p)->barcode());
    const HepMC::FourVector fv((*p)->momentum().px() * 1000,
                               (*p)->momentum().py() * 1000,
                               (*p)->momentum().pz() * 1000,
                               (*p)->momentum().e()  * 1000);
    (*p)->set_momentum(fv);
    (*p)->set_generated_mass(1000 * (*p)->generated_mass());
  }
  ////////// END NEW PARTS ///////////////////
 
    return StatusCode::SUCCESS;
}

Then run: 

cd ../cmt
gmake
gmake install

You should now have this installed in your InstallArea directory.

Job Options for converting

Installing OpenLoops

On Edison@NERSC: 

module unload altd
module unload darshan
module unload PrgEnv-intel
module load PrgEnv-gnu
module unload cray-shmem
module load scons
module load mercurial
module load cmake
module load boost
module load gsl
./scons -j 10
./scons auto=all/ -j 15

On Mira/Cetus@ALCF: 

  • soft add +bgqtoolchain-gcc484
soft add +mpiwrapper-gcc
soft add +python2.7.9
  • Checkout from ATLAS: 
     svn co --username atlas https://openloops.hepforge.org/svn/OpenLoops/branches/public_collier atlas_ol
    • pmg4711
  • cd atlas_ol
  • Make these changes:
    Index: pyol/config/default.cfg
===================================================================
--- pyol/config/default.cfg	(revision 1831)
+++ pyol/config/default.cfg	(working copy)
@@ -8,8 +8,8 @@
 
 [OpenLoops]
 
-supported_compilers = gfortran ifort
-fortran_compiler = gfortran
+supported_compilers = gfortran ifort mpif90
+fortran_compiler = mpif90
 fortran_tool = auto
 cc = gcc
 cxx = g++
Index: lib_src/olcommon/src/kind_types.F90
===================================================================
--- lib_src/olcommon/src/kind_types.F90	(revision 1831)
+++ lib_src/olcommon/src/kind_types.F90	(working copy)
@@ -18,6 +18,7 @@
 
 
 module kind_types
+  use ISO_FORTRAN_ENV
   implicit none
 
   integer, parameter :: intkind1 = selected_int_kind(2)
@@ -35,7 +36,8 @@
   complex(ep), parameter :: cI_ep = (0._ep, 1._ep)
 #endif
 ! #ifdef USE_qp
-  integer, parameter :: qp = selected_real_kind(33)
+!  integer, parameter :: qp = selected_real_kind(33)
+  integer, parameter :: qp = REAL128
   complex(qp), parameter :: cI_qp = (0._qp, 1._qp)
 ! #endif
 
Index: lib_src/openloops/src/ol_interface.F90
===================================================================
--- lib_src/openloops/src/ol_interface.F90	(revision 1831)
+++ lib_src/openloops/src/ol_interface.F90	(working copy)
@@ -1108,7 +1108,7 @@
       if (index(trim(infofilename),"."//trim(info_file_suffix)) == 0) then
         cycle
       else
-        infofilename = trim(install_path) // "/proclib/" // trim(infofilename)
+        infofilename = trim(install_path) // "/proclib/" // infofilename(2:len(infofilename))
       end if
 
       inquire(gf_info, opened=iqopen)
  • To build: 
    ./scons -j 20
  • To download and build all processes: 
    ./scons proc=all/ -j 20

Setup:

  • On Tukey only 
    soft add +gcc-4.8.1
  • On SLC6 machine: scons needs gcc-4.6+ so use ATLAS compiler:
    •  export PATH=/cvmfs/atlas.cern.ch/repo/sw/atlas-gcc/462/x86_64/slc6/x86_64-slc6-gcc46-opt/bin:$PATH
    •  export LD_LIBRARY_PATH=/cvmfs/atlas.cern.ch/repo/sw/atlas-gcc/462/x86_64/slc6/x86_64-slc6-gcc46-opt/lib:$LD_LIBRARY_PATH
  1. Get <html><a href=“http://openloops.hepforge.org/index.html” target=“_blank”>OpenLoops</a></html>: 
    svn checkout http://openloops.hepforge.org/svn/OpenLoops/branches/public ./OpenLoops
  2. Run Scons: 
     cd OpenLoops
./scons auto=all/

    This downloads all the process libraries and takes some time. <html><a href=“http://openloops.hepforge.org/process_library.php” target=“_blank”>List of libraries.</a></html>.

  3. To compile OpenLoops as a static library, create the file openloops.cfg in the top-level directory. 
    [OpenLoops]
shared_libraries=0
hpc/sherpa.txt · Last modified: 2017/01/17 15:21 by jchilders