Sherpa Notes

Sherpa Notes

wget http://www.hepforge.org/archive/sherpa/SHERPA-MC-2.1.1.tar.gz

In most cases opening Makefile and commenting out line 470/471 which have the words Manual and Examples on it will save you some time.

«< I frequently have to do 'autoreconf' before 'make' when doing a fresh install. Not sure why, just do. »>

Some Notes

The code generation step of Sherpa produces a makelibs compile script and many other Makefiles.in in the Process subdirectory that has full absolute path inside that points to the local Sherpa installation. Many of the configure* and automake files contain absolute paths to the directory in which the code generation step was executed. This causes problems if one tries to move the code generation to another system or folder.
200900 = W+jets, 200901 = Z+jets, 200903 = Drell-Yan, 200910 = Multi-jets

During a run Sherpa dlopen() calls the following libraries:
- LibCT10Sherpa.so (or whatever PDF you are using)
- libSherpaOpenLoops.so
- libProc2_2…
- libopenloops.so
- openloops/proclib/*.so
- libSherpaHepMcOutput.so

Runtime Errors and Possible Meanings:
- 'Couldn't find virtual ME for this process' → look that it has the libSherpaOpenLoops.so and the proper openloops libraries.
- 'Process not found' → make sure Run.dat card is found, and if so, make sure Processes are defined in the card properly.

Compilation and Installation at Edison@NERSC

Local Environment

Install HepMC.

After logging into a clean shell run

module unload altd
module unload darshan
module unload PrgEnv-intel
module load PrgEnv-gnu
module unload cray-shmem
module load scons
module load mercurial
module load cmake
module load boost
module load gsl

Compilation

Unpack Sherpa tarball and enter sherpa main directory. After configure comment out line 470 of the Makefile for Sherpa which has the word “Manual” on it. You can create a mostly-static or dynamic executable by using the '-static' and '–enable-static=yes', or '-dynamic' flags, respectively. Then run: <WRAP prewrap>

./configure --enable-mpi --enable-gzip --with-sqlite3=/usr --enable-hepmc2=</path/to/hepmc2> --enable-openloops=</path/to/openloops> --prefix=$PWD/install CC=cc CXX=CC FC='ftn -fPIC -funderscoring'
make -j 10
make install

</WRAP>

If you go for the static flags the make will fail just before the executable is built so then do this:

./configure --enable-mpi --enable-hepmc2=/project/projectdirs/m2015/tools/HepMC-2.06.08/local/ --with-pic  LDFLAGS=-static MPICXX=CC CC=cc CXX=CC --prefix=$PWD/install --enable-openloops=/project/projectdirs/m2015/tools/OpenLoops/ --enable-gzip --enable-static=yes --enable-shared=yes
make -j 10
make -j 10 install
cd SHERPA/Run
../../libtool --tag=CXX --mode=link CC -Wl,--no-as-needed -g -O2  -rdynamic -Wl,--no-as-needed  -o Sherpa ../../*/*/*.o ../../*/*/*/*.o /project/projectdirs/m2015/tools/HepMC-2.06.08/src/*.o -lgfortran /usr/lib64/libz.a -lsqlite3 > libtool.log
# this will fail with errors then run this
sed s/"libtool: link: "//g libtool.log | sed s/"..\/..\/PDF\/CT10\/CT10_Fortran_Interface.o"//g | sed s/"..\/..\/PDF\/CT10\/CT10Pdf.o "//g | sed s/"..\/..\/PDF\/CT10\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/CT12\/CT12_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/CT12\/CT12Pdf.o "//g | sed s/"..\/..\/PDF\/CT12\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/CTEQ\/CTEQ6_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/CTEQ\/Cteq6Pdf-2008.o "//g | sed s/"..\/..\/PDF\/CTEQ\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/Electron\/PDF_Electron.o "//g | sed s/"..\/..\/PDF\/Electron\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/GRV\/GRVph_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/GRV\/grvphoton.o "//g | sed s/"..\/..\/PDF\/GRV\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst2001lo.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst2004qed.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst99.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST01LO.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST04QED.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST99.o "//g | sed s/"..\/..\/PDF\/MRST\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/MSTW\/mstwpdf.o "//g | sed s/"..\/..\/PDF\/MSTW\/PDF_MSTW.o "//g | sed s/"..\/..\/PDF\/MSTW\/SVN_Info.o "//g | sed s/"\/opt\/gcc\/4.9.2\/snos\/lib\/..\/lib64\/libquadmath.so"/"-lquadmath"/g | sed s/"\/opt\/gcc\/4.9.2\/snos\/lib\/..\/lib64\/libgfortran.so"/"-lgfortran"/g | sed s/"\/usr\/lib64\/libsqlite3.so"/"-lsqlite3"/g > compile_Sherpa.sh
sh compile_Sherpa.sh
cp Sherpa ../../install/bin/

</WRAP>

Test Run

Ensure proper modules are loaded as listed above.
Create directory in Examples/test.

Change to this folder and create the file Run.dat and fill it with this text:

(run){
 % general setting
 EVENTS 1; ERROR 0.99;
 EVENT_OUTPUT=HEPEVT[specialtest];
 
 % scales, tags for scale variations
 SP_NLOCT 1; FSF:=1.; RSF:=1.; QSF:=1.;
 SCALES METS{FSF*MU_F2}{RSF*MU_R2}{QSF*MU_Q2};
 
 % tags for process setup
 NJET:=2; LJET:=2; QCUT:=20.;
 
 % me generator settings
 ME_SIGNAL_GENERATOR Comix Amegic LOOPGEN;
 EVENT_GENERATION_MODE Weighted;
 LOOPGEN:=Internal;
 
 % exclude tau from lepton container
 MASSIVE[15] 1;
 
 % collider setup
 BEAM_1 2212; BEAM_ENERGY_1 = 4000.;
 BEAM_2 2212; BEAM_ENERGY_2 = 4000.;
}(run)
 
(processes){
 Process 93 93 -> 90 91 93{NJET};
 Order_EW 2; CKKW sqr(QCUT/E_CMS);
 NLO_QCD_Mode MC@NLO {LJET};
 ME_Generator Amegic {LJET};
 Loop_Generator LOOPGEN {LJET};
 Scales LOOSE_METS{FSF*MU_F2}{RSF*MU_R2}{QSF*MU_Q2} {7,8};
 End process;
}(processes)
 
(selector){
 Mass 11 -12 5. E_CMS
 Mass 13 -14 5. E_CMS
 Mass -11 12 5. E_CMS
 Mass -13 14 5. E_CMS
}(selector)
}(run)

Start an interactive session: qsub -I -q default -l mppwidth=24
Once this session begins you'll be back in your home directory on the worker node.
Change directories back: cd $PBS_O_WORKDIR.
Next run the initialization step of Sherpa which produces the makelibs file: aprun -n 1 -N 1 ../../install/bin/Sherpa
Once complete, exit the interactive session and run ./makelibs

Compilation and Installation at Cooley@ALCF

soft add +mvapich2
soft add +gcc-4.8.1

Unpack tarball and enter main sherpa directory.<WRAP prewrap>

mkdir install
./configure  --enable-gzip --enable-openloops=/projects/EnergyFEC/anl/cooley/tools/OpenLoops-r1354 --enable-hepmc2=/projects/EnergyFEC/anl/cooley/tools/HepMC-2.06.09/install --with-sqlite3=/projects/EnergyFEC/anl/cooley/tools/sqlite-autoconf-3080200/install --enable-mpi --enable-static=yes --prefix=$PWD/install

</WRAP>

In the Makefile remove line 472 because tukey doesn't have the 'texi2html' app installed needed to compile the manual.

SUBDIRS = \
      ATOOLS \
      BEAM \
      PDF \
      MODEL \
      METOOLS \
      PHASIC++ \
      MCATNLO \
      EXTRA_XS \
      AMEGIC++ \
      COMIX \
      CSSHOWER++ \
      AMISIC++ \
      AHADIC++ \
      HADRONS++ \
      PHOTONS++ \
      SHRiMPS \
      SHERPA \
      AddOns \
      Manual \   # <---- this line
      Examples

Make & Install:
```
make
make install
```
Install Scons. If you've installed OpenLoops, scons is in /path/to/OpenLoops/scons-local. Scons is used to compile the process libraries, the code for which is written when you first call Sherpa with your run card. If you don't have scons, it tries to use autoconfig which can have difficulties sometimes whereas Scons does not.
```
wget http://prdownloads.sourceforge.net/scons/scons-2.3.4.tar.gz
# Unpack tarball and enter main scons directory
mkdir install
python setup.py install --prefix=$PWD/install
export PATH=$PATH:$PWD/install/bin
```

Compiling on Mira/Cetus@ALCF

soft add +mpiwrapper-gcc
soft add +bgqtoolchain-gcc484
soft add +python-2.7
svn export svn+ssh://atlas1.hep.anl.gov/users/svn/hpc/generators/sherpa/v220/dev/tags/01-00-00 01-00-00
cd 01-00-00
./configure --enable-mpi --with-pic --enable-gzip=/soft/libraries/alcf/current/gcc/ZLIB --enable-openloops=</path/to/openloops> --with-sqlite3=install --enable-hepmc2=</path/to/hepmc2> --host=powerpc64-bgq-linux CC=mpicc CFLAGS=-g CPPFLAGS=-g CXXFLAGS=-g MPICXX=mpicxx CXX=mpicxx FC='mpif90 -fPIC -funderscoring' FCFLAGS=-g LDFLAGS=-dynamic --prefix=$PWD/install
# build and install sqlite3
# rerun configure with: --with-sqlite3=</path/to/sqlite3>
make -j 20 install

</WRAP>

Hopefully, now everything is working.
You may need to copy the HepMC and sqlite3 libraries to the installed sherpa library directory.

I have run into these problems in the past:

 /projects/EnergyFEC/anl/generators/sherpa/v220/rel-2-2-0_anl/SHERPA/PerturbativePhysics/.libs/libSherpaPerturbativePhysics.so: undefined reference to `std::__detail::_List_node_base::_M_hook(std::__detail::_List_node_base*)'
/projects/EnergyFEC/anl/generators/sherpa/v220/rel-2-2-0_anl/SHERPA/PerturbativePhysics/.libs/libSherpaPerturbativePhysics.so: undefined reference to `std::__detail::_List_node_base::_M_unhook()'
/projects/EnergyFEC/anl/generators/sherpa/v220/rel-2-2-0_anl/SHERPA/PerturbativePhysics/.libs/libSherpaPerturbativePhysics.so: undefined reference to `std::__detail::_List_node_base::_M_transfer(std::__detail::_List_node_base*, std::__detail::_List_node_base*)'
/projects/EnergyFEC/anl/generators/sherpa/v220/rel-2-2-0_anl/AHADIC++/Formation/.libs/libAhadicFormation.so: undefined reference to `std::__detail::_List_node_base::_M_reverse()'

</WRAP>

Compiling on Vesta@ALCF

svn export svn+ssh://atlas1.hep.anl.gov/users/svn/hpc/generators/sherpa/v211/bgq/tags/02-01-00 sherpa-02-01-00
./configure --host=powerpc64-unknown-linux-gnu --with-sqlite3=install --enable-gzip=/soft/libraries/alcf/current/gcc/ZLIB --enable-openloops=/projects/ATLASQ/ol-mpi --enable-hepmc2=/home/projects/ATLASQ/tools/HepMC-2.06.08/install --enable-mpi --enable-shared=yes --prefix=$PWD/install --with-pic CC=mpicc CFLAGS=-pg LDFLAGS=-pg CPPFLAGS=-pg FC=mpif90 FCFLAGS=-pg MPICXX=mpicxx CXX=mpicxx CXXFLAGS=-pg

</WRAP> This step will fail because it cannot compile sqlite3 without some help. So go in the sqlite3 directory and run the following. <WRAP prewrap>

mkdir install
./configure --enable-static=yes --prefix=$PWD/install
make
make install

</WRAP> Then move back to Sherpa root path and rerun the configure step. <WRAP prewrap>

./configure --host=ppc64 --enable-gzip=/soft/libraries/alcf/current/gcc/ZLIB --enable-openloops=/projects/ATLASQ/ol-mpi --enable-hepmc2=/home/projects/ATLASQ/tools/HepMC-2.06.08/install --enable-mpi --enable-shared=yes --prefix=$PWD/install --with-pic CC=mpicc CFLAGS=-pg LDFLAGS=-pg CPPFLAGS=-pg FC=mpif90 FCFLAGS=-pg MPICXX=mpicxx CXX=mpicxx CXXFLAGS=-pg
make V=1 -j 10 > make.log 2>&1 &
make -j 10 install

</WRAP>

For static build: <WRAP prewrap>

/bin/sh ../../libtool  --tag=CXX   --mode=link mpicxx -Wl,--no-as-needed -g -O2 -pg  -pg -dynamic -o Sherpa ../../*/*/*.o ../../*/*/*/*.o /home/projects/ATLASQ/tools/HepMC-2.06.08/src/*.o /home/projects/ATLASQ/generators/sherpa-02-01-00/sqlite-autoconf-3080200/install/lib/libsqlite3.a /soft/libraries/alcf/current/gcc/ZLIB/lib/libz.a -L/projects/ATLASQ/ol-mpi/lib -lopenloops -lcoli > libtool.log

</WRAP> This will fail because the PDF libraries contain overloaded functions and cannot be all loaded together. Sherpa loads these dynamically so the sed command below removes all the PDF object files. <WRAP prewrap>

sed s/"libtool: link: "//g libtool.log | sed s/"..\/..\/PDF\/CT10\/CT10_Fortran_Interface.o"//g | sed s/"..\/..\/PDF\/CT10\/CT10Pdf.o "//g | sed s/"..\/..\/PDF\/CT10\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/CT12\/CT12_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/CT12\/CT12Pdf.o "//g | sed s/"..\/..\/PDF\/CT12\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/CTEQ\/CTEQ6_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/CTEQ\/Cteq6Pdf-2008.o "//g | sed s/"..\/..\/PDF\/CTEQ\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/Electron\/PDF_Electron.o "//g | sed s/"..\/..\/PDF\/Electron\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/GRV\/GRVph_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/GRV\/grvphoton.o "//g | sed s/"..\/..\/PDF\/GRV\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst2001lo.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst2004qed.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst99.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST01LO.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST04QED.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST99.o "//g | sed s/"..\/..\/PDF\/MRST\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/MSTW\/mstwpdf.o "//g | sed s/"..\/..\/PDF\/MSTW\/PDF_MSTW.o "//g | sed s/"..\/..\/PDF\/MSTW\/SVN_Info.o "//g > compile_Sherpa.sh
sh compile_Sherpa.sh

</WRAP>

Now to run a test. <WRAP prewrap>

qsub -t 20 -n 1 --mode=c1 Sherpa -l run.codegen.log INIT_ONLY=1
qsub -t 20 -n 1 --mode=c4 Sherpa -l run.integration.log SQLITE_OPEN_FLAG=unix-none

</WRAP>

The SQLITE_OPEN_FLAG is explained here: https://sherpa.hepforge.org/trac/ticket/263?cversion=3&cnum_hist=2

Compiling on Ascinode

tar xzf SHERPA-MC-2.1.1.tar.gz
cd SHERPA-MC-2.1.1
./configure  --enable-gzip --enable-openloops=/users/hpcusers/tools/OpenLoops --enable-hepmc2=/users/hpcusers/svn/tools/HepMC-2.06.08/local  --enable-mpi --enable-static=yes --enable-shared=yes --prefix=$PWD/install --with-pic
make V=1 -j 10 > make.log 2>&1 &
make install

</WRAP> This will create the Sherpa executable but will have a dynamically linked executable. As seen here:

ldd SHERPA/Run/.libs/Sherpa
	linux-vdso.so.1 =>  (0x00007ffff4bff000)
	libSherpaMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libSherpaMain.so.0 (0x00007f83841ea000)
	libSherpaInitialization.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libSherpaInitialization.so.0 (0x00007f8383fbe000)
	libSherpaPerturbativePhysics.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libSherpaPerturbativePhysics.so.0 (0x00007f8383d62000)
	libSherpaSoftPhysics.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libSherpaSoftPhysics.so.0 (0x00007f8383b36000)
	libLundTools.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libLundTools.so.0 (0x00007f8383935000)
	libSherpaTools.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libSherpaTools.so.0 (0x00007f838369b000)
	libAmisic.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAmisic.so.0 (0x00007f838348e000)
	libAmisicModel.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAmisicModel.so.0 (0x00007f8383251000)
	libAmisicTools.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAmisicTools.so.0 (0x00007f838303c000)
	libCSTools.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libCSTools.so.0 (0x00007f8382e30000)
	libCSCalculators.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libCSCalculators.so.0 (0x00007f8382c04000)
	libCSShowers.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libCSShowers.so.0 (0x00007f83829d2000)
	libCSMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libCSMain.so.0 (0x00007f83827af000)
	libAhadicMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAhadicMain.so.0 (0x00007f83825a6000)
	libAhadicTools.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAhadicTools.so.0 (0x00007f8382326000)
	libAhadicFormation.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAhadicFormation.so.0 (0x00007f8382114000)
	libAhadicDecays.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAhadicDecays.so.0 (0x00007f8381f10000)
	libShrimpsMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libShrimpsMain.so.0 (0x00007f8381d01000)
	libShrimpsEvents.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libShrimpsEvents.so.0 (0x00007f8381ac1000)
	libShrimpsBeamRemnants.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libShrimpsBeamRemnants.so.0 (0x00007f8381894000)
	libShrimpsXsecs.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libShrimpsXsecs.so.0 (0x00007f8381674000)
	libShrimpsEikonals.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libShrimpsEikonals.so.0 (0x00007f838144c000)
	libShrimpsTools.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libShrimpsTools.so.0 (0x00007f838123d000)
	libBeam.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libBeam.so.0 (0x00007f838101b000)
	libComixAmplitude.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libComixAmplitude.so.0 (0x00007f8380dd3000)
	libComixPhasespace.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libComixPhasespace.so.0 (0x00007f8380b8b000)
	libComix.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libComix.so.0 (0x00007f8380952000)
	libComixCluster.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libComixCluster.so.0 (0x00007f8380738000)
	libAmegic.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAmegic.so.0 (0x00007f83804cc000)
	libDipoleSubtraction.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libDipoleSubtraction.so.0 (0x00007f838024e000)
	libAmplitude.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAmplitude.so.0 (0x00007f837ffc6000)
	libAmegicPSGen.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAmegicPSGen.so.0 (0x00007f837fd1d000)
	libZfunctions.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libZfunctions.so.0 (0x00007f837fa76000)
	libString.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libString.so.0 (0x00007f837f83e000)
	libAmegicCluster.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAmegicCluster.so.0 (0x00007f837f61f000)
	libModelMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libModelMain.so.0 (0x00007f837f38c000)
	libModelInteractions.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libModelInteractions.so.0 (0x00007f837efc4000)
	libPDF.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPDF.so.0 (0x00007f837eda6000)
	libRemnant.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libRemnant.so.0 (0x00007f837eb8e000)
	libMCatNLOTools.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMCatNLOTools.so.0 (0x00007f837e988000)
	libMCatNLOCalculators.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMCatNLOCalculators.so.0 (0x00007f837e771000)
	libMCatNLOShowers.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMCatNLOShowers.so.0 (0x00007f837e548000)
	libMCatNLOMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMCatNLOMain.so.0 (0x00007f837e329000)
	libPhasicChannels.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhasicChannels.so.0 (0x00007f837e05e000)
	libPhasicMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhasicMain.so.0 (0x00007f837de0b000)
	libPhasicProcess.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhasicProcess.so.0 (0x00007f837db7e000)
	libPhasicSelectors.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhasicSelectors.so.0 (0x00007f837d8e3000)
	libPhasicScales.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhasicScales.so.0 (0x00007f837d694000)
	libPhasicEnhance.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhasicEnhance.so.0 (0x00007f837d48a000)
	libPhasicDecays.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhasicDecays.so.0 (0x00007f837d271000)
	libMEToolsExplicit.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMEToolsExplicit.so.0 (0x00007f837d046000)
	libMEToolsCurrents.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMEToolsCurrents.so.0 (0x00007f837ce16000)
	libMEToolsVertices.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMEToolsVertices.so.0 (0x00007f837cbea000)
	libMEToolsColors.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMEToolsColors.so.0 (0x00007f837c9db000)
	libMEToolsSpinCorrelations.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMEToolsSpinCorrelations.so.0 (0x00007f837c7ce000)
	libMEToolsLoops.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMEToolsLoops.so.0 (0x00007f837c5ab000)
	libMEToolsMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMEToolsMain.so.0 (0x00007f837c378000)
	libHadronsMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libHadronsMain.so.0 (0x00007f837c148000)
	libHadronsMEs.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libHadronsMEs.so.0 (0x00007f837bf0a000)
	libHadronsCurrents.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libHadronsCurrents.so.0 (0x00007f837bc7e000)
	libHadronsPSs.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libHadronsPSs.so.0 (0x00007f837ba68000)
	libPhotonsMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhotonsMain.so.0 (0x00007f837b85d000)
	libPhotonsTools.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhotonsTools.so.0 (0x00007f837b63b000)
	libPhotonsPhaseSpace.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhotonsPhaseSpace.so.0 (0x00007f837b430000)
	libPhotonsMEs.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhotonsMEs.so.0 (0x00007f837b1ee000)
	libExtraXS.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libExtraXS.so.0 (0x00007f837afdb000)
	libExtraXS2_2.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libExtraXS2_2.so.0 (0x00007f837ad95000)
	libExtraXS1_2.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libExtraXS1_2.so.0 (0x00007f837ab8f000)
	libExtraXS1_3.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libExtraXS1_3.so.0 (0x00007f837a984000)
	libExtraXSCluster.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libExtraXSCluster.so.0 (0x00007f837a781000)
	libExtraXSNLO.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libExtraXSNLO.so.0 (0x00007f837a55b000)
	libgfortran.so.3 => /usr/lib64/libgfortran.so.3 (0x00007f837a24e000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00000031bde00000)
	libSherpaSingleEvents.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libSherpaSingleEvents.so.0 (0x00007f8379feb000)
	libToolsPhys.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libToolsPhys.so.0 (0x00007f8379d8f000)
	libToolsMath.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libToolsMath.so.0 (0x00007f8379ada000)
	libToolsOrg.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libToolsOrg.so.0 (0x00007f837983d000)
	libz.so.1 => /lib64/libz.so.1 (0x00000031bea00000)
	libsqlite3.so.0 => /usr/lib64/libsqlite3.so.0 (0x00000031cf600000)
	libmpi_cxx.so.1 => /usr/lib64/openmpi/lib/libmpi_cxx.so.1 (0x00007f8379622000)
	libmpi.so.1 => /usr/lib64/openmpi/lib/libmpi.so.1 (0x00000031bf600000)
	libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00000031c0e00000)
	libm.so.6 => /lib64/libm.so.6 (0x00000031be200000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000031c0600000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00000031bda00000)
	libc.so.6 => /lib64/libc.so.6 (0x00000031bd600000)
	/lib64/ld-linux-x86-64.so.2 (0x00000031bd200000)
	libopen-rte.so.7 => /usr/lib64/openmpi/lib/libopen-rte.so.7 (0x00007f83793a6000)
	libopen-pal.so.6 => /usr/lib64/openmpi/lib/libopen-pal.so.6 (0x00000031bf200000)
	librt.so.1 => /lib64/librt.so.1 (0x00000031be600000)
	libnsl.so.1 => /lib64/libnsl.so.1 (0x00000031c1a00000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00000031cd600000)
	libltdl.so.7 => /usr/lib64/libltdl.so.7 (0x00000031c6200000)

To build a more statically linked executable do the following steps. <WRAP prewrap>

cd SHERPA/Run
../../libtool --tag=CXX   --mode=link mpic++ -Wl,--no-as-needed -g -O2  -rdynamic -Wl,--no-as-needed  -o Sherpa ../../*/*/*.o ../../*/*/*/*.o /users/hpcusers/svn/tools/HepMC-2.06.08/src/*.o -lgfortran /usr/lib64/libz.a -lsqlite3 > libtool.log
sed s/"libtool: link: "//g libtool.log | sed s/"..\/..\/PDF\/CT10\/CT10_Fortran_Interface.o"//g | sed s/"..\/..\/PDF\/CT10\/CT10Pdf.o "//g | sed s/"..\/..\/PDF\/CT10\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/CT12\/CT12_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/CT12\/CT12Pdf.o "//g | sed s/"..\/..\/PDF\/CT12\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/CTEQ\/CTEQ6_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/CTEQ\/Cteq6Pdf-2008.o "//g | sed s/"..\/..\/PDF\/CTEQ\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/Electron\/PDF_Electron.o "//g | sed s/"..\/..\/PDF\/Electron\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/GRV\/GRVph_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/GRV\/grvphoton.o "//g | sed s/"..\/..\/PDF\/GRV\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst2001lo.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst2004qed.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst99.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST01LO.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST04QED.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST99.o "//g | sed s/"..\/..\/PDF\/MRST\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/MSTW\/mstwpdf.o "//g | sed s/"..\/..\/PDF\/MSTW\/PDF_MSTW.o "//g | sed s/"..\/..\/PDF\/MSTW\/SVN_Info.o "//g > compile_Sherpa.sh
sh compile_Sherpa.sh
cp Sherpa ../../install/bin/

</WRAP> Now you have:

ldd install/bin/Sherpa
	linux-vdso.so.1 =>  (0x00007fff8e5ff000)
	libgfortran.so.3 => /usr/lib64/libgfortran.so.3 (0x00007f63da4cc000)
	libsqlite3.so.0 => /usr/lib64/libsqlite3.so.0 (0x00000031cf600000)
	libmpi_cxx.so.1 => /usr/lib64/openmpi/lib/libmpi_cxx.so.1 (0x00007f63da2b1000)
	libmpi.so.1 => /usr/lib64/openmpi/lib/libmpi.so.1 (0x00000031bf600000)
	libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00000031c0e00000)
	libm.so.6 => /lib64/libm.so.6 (0x00000031be200000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000031c0600000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00000031bda00000)
	libc.so.6 => /lib64/libc.so.6 (0x00000031bd600000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00000031bde00000)
	libopen-rte.so.7 => /usr/lib64/openmpi/lib/libopen-rte.so.7 (0x00000031bea00000)
	libopen-pal.so.6 => /usr/lib64/openmpi/lib/libopen-pal.so.6 (0x00000031bf200000)
	librt.so.1 => /lib64/librt.so.1 (0x00000031be600000)
	libnsl.so.1 => /lib64/libnsl.so.1 (0x00000031c1a00000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00000031cd600000)
	libltdl.so.7 => /usr/lib64/libltdl.so.7 (0x00000031c6200000)
	/lib64/ld-linux-x86-64.so.2 (0x00000031bd200000)

Patch Notes

Apply in this order:

gzip_2.patch
mpi_seed_mode_1.patch
one_lib_1.patch
remove_summarize_1.patch
resdir_1.patch

Sometimes I have trouble applying these so just use a text editor to see what changes need to be applied.

Converting Sherpa HepMC_GenEvent Format to POOL EVNT Format

Hack to Athena Algs

It was found that certain Truth containers are not created in Athena Reco transforms if the HepMC::GenEvent::signal_process_id is not greater than zero. Therefore the following hack describes a way to introduce this.

I was using the following commands on ascinode.hep.anl.gov with Athena setup with the environment asetup 17.2.12.7.10,MCProd,64,slc5, though this should work for other releases as well. Then do the following from your Athena home directory where the InstallArea will be created

cmt show versions PhysicsAnalysis/TruthParticleID/McParticleTools

Which ever version is shown for your relase do the following:

cmt co McParticleTools-00-19-00 PhysicsAnalysis/TruthParticleID/McParticleTools
cd McParticleTools-00-19-00 PhysicsAnalysis/TruthParticleID/McParticleTools/src/

Edit the file named HepMcReaderTool.cxx:

StatusCode HepMcReaderTool::read( HepMC::GenEvent* evt )
{
  m_ioFrontend->fill_next_event(evt);
 
  ////////// BEGIN NEW PARTS ///////////////////
  // required to be non-zero for TruthParticleContainer creation
  evt->set_signal_process_id(1);
  // taken from Generators/Sherpa_i/trunk/src/Sherpa_i.cxx:115
  // ATLAS renormalizes the output Sherpa event weights
  if (evt->weights().size()>2) {
    evt->weights()[0]/=evt->weights()[2];
    while (evt->weights().size()>1) evt->weights().pop_back();
  }
  // Sherpa uses GeV, ATLAS uses MeV so convert all the momenta and energies.
  for (HepMC::GenEvent::particle_iterator p = evt->particles_begin(); p != evt->particles_end(); ++p) {
    // ATH_MSG_DEBUG("PDG, BAR " << (*p)->pdg_id() << " " << (*p)->barcode());
    const HepMC::FourVector fv((*p)->momentum().px() * 1000,
                               (*p)->momentum().py() * 1000,
                               (*p)->momentum().pz() * 1000,
                               (*p)->momentum().e()  * 1000);
    (*p)->set_momentum(fv);
    (*p)->set_generated_mass(1000 * (*p)->generated_mass());
  }
  ////////// END NEW PARTS ///////////////////
 
    return StatusCode::SUCCESS;
}

Then run:

cd ../cmt
gmake
gmake install

You should now have this installed in your InstallArea directory.

Job Options for converting

Installing OpenLoops

On Edison@NERSC:

module unload altd
module unload darshan
module unload PrgEnv-intel
module load PrgEnv-gnu
module unload cray-shmem
module load scons
module load mercurial
module load cmake
module load boost
module load gsl
./scons -j 10
./scons auto=all/ -j 15

On Mira/Cetus@ALCF:

soft add +bgqtoolchain-gcc484
soft add +mpiwrapper-gcc
soft add +python2.7.9

Checkout from ATLAS:

 svn co --username atlas https://openloops.hepforge.org/svn/OpenLoops/branches/public_collier atlas_ol

pmg4711

```
cd atlas_ol
```

Make these changes:

Index: pyol/config/default.cfg
===================================================================
--- pyol/config/default.cfg	(revision 1831)
+++ pyol/config/default.cfg	(working copy)
@@ -8,8 +8,8 @@
 
 [OpenLoops]
 
-supported_compilers = gfortran ifort
-fortran_compiler = gfortran
+supported_compilers = gfortran ifort mpif90
+fortran_compiler = mpif90
 fortran_tool = auto
 cc = gcc
 cxx = g++
Index: lib_src/olcommon/src/kind_types.F90
===================================================================
--- lib_src/olcommon/src/kind_types.F90	(revision 1831)
+++ lib_src/olcommon/src/kind_types.F90	(working copy)
@@ -18,6 +18,7 @@
 
 
 module kind_types
+  use ISO_FORTRAN_ENV
   implicit none
 
   integer, parameter :: intkind1 = selected_int_kind(2)
@@ -35,7 +36,8 @@
   complex(ep), parameter :: cI_ep = (0._ep, 1._ep)
 #endif
 ! #ifdef USE_qp
-  integer, parameter :: qp = selected_real_kind(33)
+!  integer, parameter :: qp = selected_real_kind(33)
+  integer, parameter :: qp = REAL128
   complex(qp), parameter :: cI_qp = (0._qp, 1._qp)
 ! #endif
 
Index: lib_src/openloops/src/ol_interface.F90
===================================================================
--- lib_src/openloops/src/ol_interface.F90	(revision 1831)
+++ lib_src/openloops/src/ol_interface.F90	(working copy)
@@ -1108,7 +1108,7 @@
       if (index(trim(infofilename),"."//trim(info_file_suffix)) == 0) then
         cycle
       else
-        infofilename = trim(install_path) // "/proclib/" // trim(infofilename)
+        infofilename = trim(install_path) // "/proclib/" // infofilename(2:len(infofilename))
       end if
 
       inquire(gf_info, opened=iqopen)

To build:
```
./scons -j 20 
```
To download and build all processes:
```
./scons proc=all/ -j 20
```

Setup:

On Tukey only
```
soft add +gcc-4.8.1
```

On SLC6 machine: scons needs gcc-4.6+ so use ATLAS compiler:

 export PATH=/cvmfs/atlas.cern.ch/repo/sw/atlas-gcc/462/x86_64/slc6/x86_64-slc6-gcc46-opt/bin:$PATH

 export LD_LIBRARY_PATH=/cvmfs/atlas.cern.ch/repo/sw/atlas-gcc/462/x86_64/slc6/x86_64-slc6-gcc46-opt/lib:$LD_LIBRARY_PATH

Get OpenLoops:

svn checkout http://openloops.hepforge.org/svn/OpenLoops/branches/public ./OpenLoops

Run Scons:
```
 cd OpenLoops
./scons auto=all/ 
```
This downloads all the process libraries and takes some time. List of libraries..
To compile OpenLoops as a static library, create the file openloops.cfg in the top-level directory.
```
[OpenLoops]
shared_libraries=0
```

Table of Contents

Sherpa Notes

Some Notes

Compilation and Installation at Edison@NERSC

Local Environment

Compilation

Test Run

Compilation and Installation at Cooley@ALCF

Compiling on Mira/Cetus@ALCF

Compiling on Vesta@ALCF

Compiling on Ascinode

Patch Notes

Converting Sherpa HepMC_GenEvent Format to POOL EVNT Format

Hack to Athena Algs

Job Options for converting

Installing OpenLoops