ATC: ATLAS Center at ANL

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Sherpa Notes ======

[[ https://sherpa.hepforge.org | Sherpa Homepage ]]

''wget http://www.hepforge.org/archive/sherpa/SHERPA-MC-2.1.1.tar.gz''

In most cases opening ''Makefile'' and commenting out line 470/471 which have the words ''Manual'' and ''Examples'' on it will save you some time.

<<< I frequently have to do 'autoreconf' before 'make' when doing a fresh install. Not sure why, just do. >>>

===== Some Notes =====
  * The code generation step of Sherpa produces a ''makelibs'' compile script and many other ''Makefiles.in'' in the ''Process'' subdirectory that has full absolute path inside that points to the local Sherpa installation. Many of the ''configure*'' and ''automake'' files contain absolute paths to the directory in which the code generation step was executed. This causes problems if one tries to move the code generation to another system or folder.
  * 200900 = W+jets, 200901 = Z+jets, 200903 = Drell-Yan, 200910 = Multi-jets

  * During a run Sherpa dlopen() calls the following libraries:
    * LibCT10Sherpa.so (or whatever PDF you are using)
    * libSherpaOpenLoops.so
    * libProc2_2...
    * libopenloops.so
    * openloops/proclib/*.so
    * libSherpaHepMcOutput.so

  * Runtime Errors and Possible Meanings:
    * 'Couldn't find virtual ME for this process' -> look that it has the libSherpaOpenLoops.so and the proper openloops libraries.
    * 'Process not found' -> make sure Run.dat card is found, and if so, make sure Processes are defined in the card properly.
===== Compilation and Installation at Edison@NERSC =====

=== Local Environment ===

Install [[hpc:hepmc | HepMC]].

After logging into a clean shell run
<code bash>
module unload altd
module unload darshan
module unload PrgEnv-intel
module load PrgEnv-gnu
module unload cray-shmem
module load scons
module load mercurial
module load cmake
module load boost
module load gsl
</code>

=== Compilation ===

Unpack Sherpa tarball and enter sherpa main directory. After configure comment out line 470 of the Makefile for Sherpa which has the word "Manual" on it. You can create a mostly-static or dynamic executable by using the '-static' and '--enable-static=yes', or '-dynamic' flags, respectively.
Then run:
<WRAP prewrap>
<code bash>
./configure --enable-mpi --enable-gzip --with-sqlite3=/usr --enable-hepmc2=</path/to/hepmc2> --enable-openloops=</path/to/openloops> --prefix=$PWD/install CC=cc CXX=CC FC='ftn -fPIC -funderscoring'
make -j 10
make install
</code>
</WRAP>

If you go for the static flags the make will fail just before the executable is built so then do this:

<WRAP prewrap>
<code bash>
./configure --enable-mpi --enable-hepmc2=/project/projectdirs/m2015/tools/HepMC-2.06.08/local/ --with-pic  LDFLAGS=-static MPICXX=CC CC=cc CXX=CC --prefix=$PWD/install --enable-openloops=/project/projectdirs/m2015/tools/OpenLoops/ --enable-gzip --enable-static=yes --enable-shared=yes
make -j 10
make -j 10 install
cd SHERPA/Run
../../libtool --tag=CXX --mode=link CC -Wl,--no-as-needed -g -O2  -rdynamic -Wl,--no-as-needed  -o Sherpa ../../*/*/*.o ../../*/*/*/*.o /project/projectdirs/m2015/tools/HepMC-2.06.08/src/*.o -lgfortran /usr/lib64/libz.a -lsqlite3 > libtool.log
# this will fail with errors then run this
sed s/"libtool: link: "//g libtool.log | sed s/"..\/..\/PDF\/CT10\/CT10_Fortran_Interface.o"//g | sed s/"..\/..\/PDF\/CT10\/CT10Pdf.o "//g | sed s/"..\/..\/PDF\/CT10\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/CT12\/CT12_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/CT12\/CT12Pdf.o "//g | sed s/"..\/..\/PDF\/CT12\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/CTEQ\/CTEQ6_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/CTEQ\/Cteq6Pdf-2008.o "//g | sed s/"..\/..\/PDF\/CTEQ\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/Electron\/PDF_Electron.o "//g | sed s/"..\/..\/PDF\/Electron\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/GRV\/GRVph_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/GRV\/grvphoton.o "//g | sed s/"..\/..\/PDF\/GRV\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst2001lo.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst2004qed.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst99.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST01LO.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST04QED.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST99.o "//g | sed s/"..\/..\/PDF\/MRST\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/MSTW\/mstwpdf.o "//g | sed s/"..\/..\/PDF\/MSTW\/PDF_MSTW.o "//g | sed s/"..\/..\/PDF\/MSTW\/SVN_Info.o "//g | sed s/"\/opt\/gcc\/4.9.2\/snos\/lib\/..\/lib64\/libquadmath.so"/"-lquadmath"/g | sed s/"\/opt\/gcc\/4.9.2\/snos\/lib\/..\/lib64\/libgfortran.so"/"-lgfortran"/g | sed s/"\/usr\/lib64\/libsqlite3.so"/"-lsqlite3"/g > compile_Sherpa.sh
sh compile_Sherpa.sh
cp Sherpa ../../install/bin/
</code>
</WRAP>

=== Test Run ===

  * Ensure proper modules are loaded as listed above.
  * Create directory in ''Examples/test''. 
  * Change to this folder and create the file ''Run.dat'' and fill it with this text: <code bash>
(run){
 % general setting
 EVENTS 1; ERROR 0.99;
 EVENT_OUTPUT=HEPEVT[specialtest];

 % scales, tags for scale variations
 SP_NLOCT 1; FSF:=1.; RSF:=1.; QSF:=1.;
 SCALES METS{FSF*MU_F2}{RSF*MU_R2}{QSF*MU_Q2};

 % tags for process setup
 NJET:=2; LJET:=2; QCUT:=20.;

 % me generator settings
 ME_SIGNAL_GENERATOR Comix Amegic LOOPGEN;
 EVENT_GENERATION_MODE Weighted;
 LOOPGEN:=Internal;

 % exclude tau from lepton container
 MASSIVE[15] 1;

 % collider setup
 BEAM_1 2212; BEAM_ENERGY_1 = 4000.;
 BEAM_2 2212; BEAM_ENERGY_2 = 4000.;
}(run)

(processes){
 Process 93 93 -> 90 91 93{NJET};
 Order_EW 2; CKKW sqr(QCUT/E_CMS);
 NLO_QCD_Mode MC@NLO {LJET};
 ME_Generator Amegic {LJET};
 Loop_Generator LOOPGEN {LJET};
 Scales LOOSE_METS{FSF*MU_F2}{RSF*MU_R2}{QSF*MU_Q2} {7,8};
 End process;
}(processes)

(selector){
 Mass 11 -12 5. E_CMS
 Mass 13 -14 5. E_CMS
 Mass -11 12 5. E_CMS
 Mass -13 14 5. E_CMS
}(selector)
}(run)
</code>

  * Start an interactive session: ''qsub -I -q default -l mppwidth=24''
  * Once this session begins you'll be back in your home directory on the worker node. 
  * Change directories back: ''cd $PBS_O_WORKDIR''. 
  * Next run the initialization step of Sherpa which produces the makelibs file: ''aprun -n 1 -N 1 ../../install/bin/Sherpa''
  * Once complete, exit the interactive session and run ''./makelibs''

===== Compilation and Installation at Cooley@ALCF =====
  * ''soft add +mvapich2''
  * ''soft add +gcc-4.8.1''
  * Unpack tarball and enter main sherpa directory.<WRAP prewrap><code bash>
mkdir install
./configure  --enable-gzip --enable-openloops=/projects/EnergyFEC/anl/cooley/tools/OpenLoops-r1354 --enable-hepmc2=/projects/EnergyFEC/anl/cooley/tools/HepMC-2.06.09/install --with-sqlite3=/projects/EnergyFEC/anl/cooley/tools/sqlite-autoconf-3080200/install --enable-mpi --enable-static=yes --prefix=$PWD/install</code></WRAP>
  * In the ''Makefile'' remove line 472 because tukey doesn't have the 'texi2html' app installed needed to compile the manual.<code make>
SUBDIRS = \
      ATOOLS \
      BEAM \
      PDF \
      MODEL \
      METOOLS \
      PHASIC++ \
      MCATNLO \
      EXTRA_XS \
      AMEGIC++ \
      COMIX \
      CSSHOWER++ \
      AMISIC++ \
      AHADIC++ \
      HADRONS++ \
      PHOTONS++ \
      SHRiMPS \
      SHERPA \
      AddOns \
      Manual \   # <---- this line
      Examples
</code>
  * Make & Install:<code bash>
make
make install
</code>
  * Install [[http://www.scons.org/download.php|Scons]]. If you've installed OpenLoops, scons is in ''/path/to/OpenLoops/scons-local''. Scons is used to compile the process libraries, the code for which is written when you first call Sherpa with your run card. If you don't have scons, it tries to use ''autoconfig'' which can have difficulties sometimes whereas Scons does not. <code bash>
wget http://prdownloads.sourceforge.net/scons/scons-2.3.4.tar.gz
# Unpack tarball and enter main scons directory
mkdir install
python setup.py install --prefix=$PWD/install
export PATH=$PATH:$PWD/install/bin
</code>
===== Compiling on Mira/Cetus@ALCF =====

  * <WRAP prewrap> <code bash>soft add +mpiwrapper-gcc
soft add +bgqtoolchain-gcc484
soft add +python-2.7
svn export svn+ssh://atlas1.hep.anl.gov/users/svn/hpc/generators/sherpa/v220/dev/tags/01-00-00 01-00-00
cd 01-00-00
./configure --enable-mpi --with-pic --enable-gzip=/soft/libraries/alcf/current/gcc/ZLIB --enable-openloops=</path/to/openloops> --with-sqlite3=install --enable-hepmc2=</path/to/hepmc2> --host=powerpc64-bgq-linux CC=mpicc CFLAGS=-g CPPFLAGS=-g CXXFLAGS=-g MPICXX=mpicxx CXX=mpicxx FC='mpif90 -fPIC -funderscoring' FCFLAGS=-g LDFLAGS=-dynamic --prefix=$PWD/install
# build and install sqlite3
# rerun configure with: --with-sqlite3=</path/to/sqlite3>
make -j 20 install</code></WRAP>
  * Hopefully, now everything is working.
  * You may need to copy the HepMC and sqlite3 libraries to the installed sherpa library directory.
  * I have run into these problems in the past:
    * <WRAP prewrap> <code make> /projects/EnergyFEC/anl/generators/sherpa/v220/rel-2-2-0_anl/SHERPA/PerturbativePhysics/.libs/libSherpaPerturbativePhysics.so: undefined reference to `std::__detail::_List_node_base::_M_hook(std::__detail::_List_node_base*)'
/projects/EnergyFEC/anl/generators/sherpa/v220/rel-2-2-0_anl/SHERPA/PerturbativePhysics/.libs/libSherpaPerturbativePhysics.so: undefined reference to `std::__detail::_List_node_base::_M_unhook()'
/projects/EnergyFEC/anl/generators/sherpa/v220/rel-2-2-0_anl/SHERPA/PerturbativePhysics/.libs/libSherpaPerturbativePhysics.so: undefined reference to `std::__detail::_List_node_base::_M_transfer(std::__detail::_List_node_base*, std::__detail::_List_node_base*)'
/projects/EnergyFEC/anl/generators/sherpa/v220/rel-2-2-0_anl/AHADIC++/Formation/.libs/libAhadicFormation.so: undefined reference to `std::__detail::_List_node_base::_M_reverse()' </code> </WRAP>

===== Compiling on Vesta@ALCF =====

<WRAP prewrap>
<code bash>
svn export svn+ssh://atlas1.hep.anl.gov/users/svn/hpc/generators/sherpa/v211/bgq/tags/02-01-00 sherpa-02-01-00
./configure --host=powerpc64-unknown-linux-gnu --with-sqlite3=install --enable-gzip=/soft/libraries/alcf/current/gcc/ZLIB --enable-openloops=/projects/ATLASQ/ol-mpi --enable-hepmc2=/home/projects/ATLASQ/tools/HepMC-2.06.08/install --enable-mpi --enable-shared=yes --prefix=$PWD/install --with-pic CC=mpicc CFLAGS=-pg LDFLAGS=-pg CPPFLAGS=-pg FC=mpif90 FCFLAGS=-pg MPICXX=mpicxx CXX=mpicxx CXXFLAGS=-pg
</code></WRAP>
This step will fail because it cannot compile sqlite3 without some help. So go in the sqlite3 directory and run the following.
<WRAP prewrap>
<code bash>
mkdir install
./configure --enable-static=yes --prefix=$PWD/install
make
make install
</code></WRAP>
Then move back to Sherpa root path and rerun the configure step.
<WRAP prewrap>
<code bash>
./configure --host=ppc64 --enable-gzip=/soft/libraries/alcf/current/gcc/ZLIB --enable-openloops=/projects/ATLASQ/ol-mpi --enable-hepmc2=/home/projects/ATLASQ/tools/HepMC-2.06.08/install --enable-mpi --enable-shared=yes --prefix=$PWD/install --with-pic CC=mpicc CFLAGS=-pg LDFLAGS=-pg CPPFLAGS=-pg FC=mpif90 FCFLAGS=-pg MPICXX=mpicxx CXX=mpicxx CXXFLAGS=-pg
make V=1 -j 10 > make.log 2>&1 &
make -j 10 install
</code></WRAP>

For static build:
<WRAP prewrap><code bash>
/bin/sh ../../libtool  --tag=CXX   --mode=link mpicxx -Wl,--no-as-needed -g -O2 -pg  -pg -dynamic -o Sherpa ../../*/*/*.o ../../*/*/*/*.o /home/projects/ATLASQ/tools/HepMC-2.06.08/src/*.o /home/projects/ATLASQ/generators/sherpa-02-01-00/sqlite-autoconf-3080200/install/lib/libsqlite3.a /soft/libraries/alcf/current/gcc/ZLIB/lib/libz.a -L/projects/ATLASQ/ol-mpi/lib -lopenloops -lcoli > libtool.log
</code></WRAP>
This will fail because the PDF libraries contain overloaded functions and cannot be all loaded together. Sherpa loads these dynamically so the ''sed'' command below removes all the PDF object files.
<WRAP prewrap><code bash>
sed s/"libtool: link: "//g libtool.log | sed s/"..\/..\/PDF\/CT10\/CT10_Fortran_Interface.o"//g | sed s/"..\/..\/PDF\/CT10\/CT10Pdf.o "//g | sed s/"..\/..\/PDF\/CT10\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/CT12\/CT12_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/CT12\/CT12Pdf.o "//g | sed s/"..\/..\/PDF\/CT12\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/CTEQ\/CTEQ6_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/CTEQ\/Cteq6Pdf-2008.o "//g | sed s/"..\/..\/PDF\/CTEQ\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/Electron\/PDF_Electron.o "//g | sed s/"..\/..\/PDF\/Electron\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/GRV\/GRVph_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/GRV\/grvphoton.o "//g | sed s/"..\/..\/PDF\/GRV\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst2001lo.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst2004qed.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst99.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST01LO.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST04QED.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST99.o "//g | sed s/"..\/..\/PDF\/MRST\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/MSTW\/mstwpdf.o "//g | sed s/"..\/..\/PDF\/MSTW\/PDF_MSTW.o "//g | sed s/"..\/..\/PDF\/MSTW\/SVN_Info.o "//g > compile_Sherpa.sh
sh compile_Sherpa.sh
</code></WRAP>

Now to run a test.
<WRAP prewrap><code bash>
qsub -t 20 -n 1 --mode=c1 Sherpa -l run.codegen.log INIT_ONLY=1
qsub -t 20 -n 1 --mode=c4 Sherpa -l run.integration.log SQLITE_OPEN_FLAG=unix-none
</code></WRAP>

The SQLITE_OPEN_FLAG is explained here:
https://sherpa.hepforge.org/trac/ticket/263?cversion=3&cnum_hist=2
===== Compiling on Ascinode =====

<WRAP prewrap>
<code bash>
tar xzf SHERPA-MC-2.1.1.tar.gz
cd SHERPA-MC-2.1.1
./configure  --enable-gzip --enable-openloops=/users/hpcusers/tools/OpenLoops --enable-hepmc2=/users/hpcusers/svn/tools/HepMC-2.06.08/local  --enable-mpi --enable-static=yes --enable-shared=yes --prefix=$PWD/install --with-pic
make V=1 -j 10 > make.log 2>&1 &
make install
</code></WRAP>
This will create the Sherpa executable but will have a dynamically linked executable. As seen here:
<code bash>
ldd SHERPA/Run/.libs/Sherpa
	linux-vdso.so.1 =>  (0x00007ffff4bff000)
	libSherpaMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libSherpaMain.so.0 (0x00007f83841ea000)
	libSherpaInitialization.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libSherpaInitialization.so.0 (0x00007f8383fbe000)
	libSherpaPerturbativePhysics.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libSherpaPerturbativePhysics.so.0 (0x00007f8383d62000)
	libSherpaSoftPhysics.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libSherpaSoftPhysics.so.0 (0x00007f8383b36000)
	libLundTools.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libLundTools.so.0 (0x00007f8383935000)
	libSherpaTools.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libSherpaTools.so.0 (0x00007f838369b000)
	libAmisic.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAmisic.so.0 (0x00007f838348e000)
	libAmisicModel.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAmisicModel.so.0 (0x00007f8383251000)
	libAmisicTools.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAmisicTools.so.0 (0x00007f838303c000)
	libCSTools.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libCSTools.so.0 (0x00007f8382e30000)
	libCSCalculators.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libCSCalculators.so.0 (0x00007f8382c04000)
	libCSShowers.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libCSShowers.so.0 (0x00007f83829d2000)
	libCSMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libCSMain.so.0 (0x00007f83827af000)
	libAhadicMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAhadicMain.so.0 (0x00007f83825a6000)
	libAhadicTools.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAhadicTools.so.0 (0x00007f8382326000)
	libAhadicFormation.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAhadicFormation.so.0 (0x00007f8382114000)
	libAhadicDecays.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAhadicDecays.so.0 (0x00007f8381f10000)
	libShrimpsMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libShrimpsMain.so.0 (0x00007f8381d01000)
	libShrimpsEvents.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libShrimpsEvents.so.0 (0x00007f8381ac1000)
	libShrimpsBeamRemnants.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libShrimpsBeamRemnants.so.0 (0x00007f8381894000)
	libShrimpsXsecs.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libShrimpsXsecs.so.0 (0x00007f8381674000)
	libShrimpsEikonals.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libShrimpsEikonals.so.0 (0x00007f838144c000)
	libShrimpsTools.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libShrimpsTools.so.0 (0x00007f838123d000)
	libBeam.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libBeam.so.0 (0x00007f838101b000)
	libComixAmplitude.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libComixAmplitude.so.0 (0x00007f8380dd3000)
	libComixPhasespace.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libComixPhasespace.so.0 (0x00007f8380b8b000)
	libComix.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libComix.so.0 (0x00007f8380952000)
	libComixCluster.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libComixCluster.so.0 (0x00007f8380738000)
	libAmegic.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAmegic.so.0 (0x00007f83804cc000)
	libDipoleSubtraction.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libDipoleSubtraction.so.0 (0x00007f838024e000)
	libAmplitude.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAmplitude.so.0 (0x00007f837ffc6000)
	libAmegicPSGen.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAmegicPSGen.so.0 (0x00007f837fd1d000)
	libZfunctions.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libZfunctions.so.0 (0x00007f837fa76000)
	libString.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libString.so.0 (0x00007f837f83e000)
	libAmegicCluster.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libAmegicCluster.so.0 (0x00007f837f61f000)
	libModelMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libModelMain.so.0 (0x00007f837f38c000)
	libModelInteractions.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libModelInteractions.so.0 (0x00007f837efc4000)
	libPDF.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPDF.so.0 (0x00007f837eda6000)
	libRemnant.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libRemnant.so.0 (0x00007f837eb8e000)
	libMCatNLOTools.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMCatNLOTools.so.0 (0x00007f837e988000)
	libMCatNLOCalculators.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMCatNLOCalculators.so.0 (0x00007f837e771000)
	libMCatNLOShowers.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMCatNLOShowers.so.0 (0x00007f837e548000)
	libMCatNLOMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMCatNLOMain.so.0 (0x00007f837e329000)
	libPhasicChannels.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhasicChannels.so.0 (0x00007f837e05e000)
	libPhasicMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhasicMain.so.0 (0x00007f837de0b000)
	libPhasicProcess.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhasicProcess.so.0 (0x00007f837db7e000)
	libPhasicSelectors.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhasicSelectors.so.0 (0x00007f837d8e3000)
	libPhasicScales.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhasicScales.so.0 (0x00007f837d694000)
	libPhasicEnhance.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhasicEnhance.so.0 (0x00007f837d48a000)
	libPhasicDecays.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhasicDecays.so.0 (0x00007f837d271000)
	libMEToolsExplicit.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMEToolsExplicit.so.0 (0x00007f837d046000)
	libMEToolsCurrents.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMEToolsCurrents.so.0 (0x00007f837ce16000)
	libMEToolsVertices.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMEToolsVertices.so.0 (0x00007f837cbea000)
	libMEToolsColors.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMEToolsColors.so.0 (0x00007f837c9db000)
	libMEToolsSpinCorrelations.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMEToolsSpinCorrelations.so.0 (0x00007f837c7ce000)
	libMEToolsLoops.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMEToolsLoops.so.0 (0x00007f837c5ab000)
	libMEToolsMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libMEToolsMain.so.0 (0x00007f837c378000)
	libHadronsMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libHadronsMain.so.0 (0x00007f837c148000)
	libHadronsMEs.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libHadronsMEs.so.0 (0x00007f837bf0a000)
	libHadronsCurrents.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libHadronsCurrents.so.0 (0x00007f837bc7e000)
	libHadronsPSs.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libHadronsPSs.so.0 (0x00007f837ba68000)
	libPhotonsMain.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhotonsMain.so.0 (0x00007f837b85d000)
	libPhotonsTools.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhotonsTools.so.0 (0x00007f837b63b000)
	libPhotonsPhaseSpace.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhotonsPhaseSpace.so.0 (0x00007f837b430000)
	libPhotonsMEs.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libPhotonsMEs.so.0 (0x00007f837b1ee000)
	libExtraXS.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libExtraXS.so.0 (0x00007f837afdb000)
	libExtraXS2_2.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libExtraXS2_2.so.0 (0x00007f837ad95000)
	libExtraXS1_2.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libExtraXS1_2.so.0 (0x00007f837ab8f000)
	libExtraXS1_3.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libExtraXS1_3.so.0 (0x00007f837a984000)
	libExtraXSCluster.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libExtraXSCluster.so.0 (0x00007f837a781000)
	libExtraXSNLO.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libExtraXSNLO.so.0 (0x00007f837a55b000)
	libgfortran.so.3 => /usr/lib64/libgfortran.so.3 (0x00007f837a24e000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00000031bde00000)
	libSherpaSingleEvents.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libSherpaSingleEvents.so.0 (0x00007f8379feb000)
	libToolsPhys.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libToolsPhys.so.0 (0x00007f8379d8f000)
	libToolsMath.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libToolsMath.so.0 (0x00007f8379ada000)
	libToolsOrg.so.0 => /users/hpcusers/svn/generators/sherpa/v211/test/SHERPA-MC-2.1.1/install/lib/SHERPA-MC/libToolsOrg.so.0 (0x00007f837983d000)
	libz.so.1 => /lib64/libz.so.1 (0x00000031bea00000)
	libsqlite3.so.0 => /usr/lib64/libsqlite3.so.0 (0x00000031cf600000)
	libmpi_cxx.so.1 => /usr/lib64/openmpi/lib/libmpi_cxx.so.1 (0x00007f8379622000)
	libmpi.so.1 => /usr/lib64/openmpi/lib/libmpi.so.1 (0x00000031bf600000)
	libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00000031c0e00000)
	libm.so.6 => /lib64/libm.so.6 (0x00000031be200000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000031c0600000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00000031bda00000)
	libc.so.6 => /lib64/libc.so.6 (0x00000031bd600000)
	/lib64/ld-linux-x86-64.so.2 (0x00000031bd200000)
	libopen-rte.so.7 => /usr/lib64/openmpi/lib/libopen-rte.so.7 (0x00007f83793a6000)
	libopen-pal.so.6 => /usr/lib64/openmpi/lib/libopen-pal.so.6 (0x00000031bf200000)
	librt.so.1 => /lib64/librt.so.1 (0x00000031be600000)
	libnsl.so.1 => /lib64/libnsl.so.1 (0x00000031c1a00000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00000031cd600000)
	libltdl.so.7 => /usr/lib64/libltdl.so.7 (0x00000031c6200000)
</code>
 To build a more statically linked executable do the following steps.
<WRAP prewrap><code bash>
cd SHERPA/Run
../../libtool --tag=CXX   --mode=link mpic++ -Wl,--no-as-needed -g -O2  -rdynamic -Wl,--no-as-needed  -o Sherpa ../../*/*/*.o ../../*/*/*/*.o /users/hpcusers/svn/tools/HepMC-2.06.08/src/*.o -lgfortran /usr/lib64/libz.a -lsqlite3 > libtool.log
sed s/"libtool: link: "//g libtool.log | sed s/"..\/..\/PDF\/CT10\/CT10_Fortran_Interface.o"//g | sed s/"..\/..\/PDF\/CT10\/CT10Pdf.o "//g | sed s/"..\/..\/PDF\/CT10\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/CT12\/CT12_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/CT12\/CT12Pdf.o "//g | sed s/"..\/..\/PDF\/CT12\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/CTEQ\/CTEQ6_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/CTEQ\/Cteq6Pdf-2008.o "//g | sed s/"..\/..\/PDF\/CTEQ\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/Electron\/PDF_Electron.o "//g | sed s/"..\/..\/PDF\/Electron\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/GRV\/GRVph_Fortran_Interface.o "//g | sed s/"..\/..\/PDF\/GRV\/grvphoton.o "//g | sed s/"..\/..\/PDF\/GRV\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst2001lo.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst2004qed.o "//g | sed s/"..\/..\/PDF\/MRST\/mrst99.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST01LO.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST04QED.o "//g | sed s/"..\/..\/PDF\/MRST\/PDF_MRST99.o "//g | sed s/"..\/..\/PDF\/MRST\/SVN_Info.o "//g | sed s/"..\/..\/PDF\/MSTW\/mstwpdf.o "//g | sed s/"..\/..\/PDF\/MSTW\/PDF_MSTW.o "//g | sed s/"..\/..\/PDF\/MSTW\/SVN_Info.o "//g > compile_Sherpa.sh
sh compile_Sherpa.sh
cp Sherpa ../../install/bin/
</code>
</WRAP>
Now you have:
<code bash>
ldd install/bin/Sherpa
	linux-vdso.so.1 =>  (0x00007fff8e5ff000)
	libgfortran.so.3 => /usr/lib64/libgfortran.so.3 (0x00007f63da4cc000)
	libsqlite3.so.0 => /usr/lib64/libsqlite3.so.0 (0x00000031cf600000)
	libmpi_cxx.so.1 => /usr/lib64/openmpi/lib/libmpi_cxx.so.1 (0x00007f63da2b1000)
	libmpi.so.1 => /usr/lib64/openmpi/lib/libmpi.so.1 (0x00000031bf600000)
	libstdc++.so.6 => /usr/lib64/libstdc++.so.6 (0x00000031c0e00000)
	libm.so.6 => /lib64/libm.so.6 (0x00000031be200000)
	libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00000031c0600000)
	libpthread.so.0 => /lib64/libpthread.so.0 (0x00000031bda00000)
	libc.so.6 => /lib64/libc.so.6 (0x00000031bd600000)
	libdl.so.2 => /lib64/libdl.so.2 (0x00000031bde00000)
	libopen-rte.so.7 => /usr/lib64/openmpi/lib/libopen-rte.so.7 (0x00000031bea00000)
	libopen-pal.so.6 => /usr/lib64/openmpi/lib/libopen-pal.so.6 (0x00000031bf200000)
	librt.so.1 => /lib64/librt.so.1 (0x00000031be600000)
	libnsl.so.1 => /lib64/libnsl.so.1 (0x00000031c1a00000)
	libutil.so.1 => /lib64/libutil.so.1 (0x00000031cd600000)
	libltdl.so.7 => /usr/lib64/libltdl.so.7 (0x00000031c6200000)
	/lib64/ld-linux-x86-64.so.2 (0x00000031bd200000)
</code>

===== Patch Notes =====

Apply in this order:

  * gzip_2.patch
  * mpi_seed_mode_1.patch
  * one_lib_1.patch
  * remove_summarize_1.patch
  * resdir_1.patch

Sometimes I have trouble applying these so just use a text editor to see what changes need to be applied.

===== Converting Sherpa HepMC_GenEvent Format to POOL EVNT Format =====

=== Hack to Athena Algs ===

It was found that certain Truth containers are not created in Athena Reco transforms if the HepMC::GenEvent::signal_process_id is not greater than zero. Therefore the following hack describes a way to introduce this.

I was using the following commands on ''ascinode.hep.anl.gov'' with Athena setup with the environment ''asetup 17.2.12.7.10,MCProd,64,slc5'', though this should work for other releases as well. Then do the following from your Athena home directory where the ''InstallArea'' will be created
<code bash>
cmt show versions PhysicsAnalysis/TruthParticleID/McParticleTools
</code>
Which ever version is shown for your relase do the following:
<code bash>
cmt co McParticleTools-00-19-00 PhysicsAnalysis/TruthParticleID/McParticleTools
cd McParticleTools-00-19-00 PhysicsAnalysis/TruthParticleID/McParticleTools/src/
</code>
Edit the file named ''HepMcReaderTool.cxx'':
<code cpp>
StatusCode HepMcReaderTool::read( HepMC::GenEvent* evt )
{
  m_ioFrontend->fill_next_event(evt);
  
  ////////// BEGIN NEW PARTS ///////////////////
  // required to be non-zero for TruthParticleContainer creation
  evt->set_signal_process_id(1);
  // taken from Generators/Sherpa_i/trunk/src/Sherpa_i.cxx:115
  // ATLAS renormalizes the output Sherpa event weights
  if (evt->weights().size()>2) {
    evt->weights()[0]/=evt->weights()[2];
    while (evt->weights().size()>1) evt->weights().pop_back();
  }
  // Sherpa uses GeV, ATLAS uses MeV so convert all the momenta and energies.
  for (HepMC::GenEvent::particle_iterator p = evt->particles_begin(); p != evt->particles_end(); ++p) {
    // ATH_MSG_DEBUG("PDG, BAR " << (*p)->pdg_id() << " " << (*p)->barcode());
    const HepMC::FourVector fv((*p)->momentum().px() * 1000,
                               (*p)->momentum().py() * 1000,
                               (*p)->momentum().pz() * 1000,
                               (*p)->momentum().e()  * 1000);
    (*p)->set_momentum(fv);
    (*p)->set_generated_mass(1000 * (*p)->generated_mass());
  }
  ////////// END NEW PARTS ///////////////////

    return StatusCode::SUCCESS;
}
</code>
Then run:
<code bash>
cd ../cmt
gmake
gmake install
</code>
You should now have this installed in your ''InstallArea'' directory.

=== Job Options for converting ===
===== Installing OpenLoops =====


On Edison@NERSC:
<code bash>module unload altd
module unload darshan
module unload PrgEnv-intel
module load PrgEnv-gnu
module unload cray-shmem
module load scons
module load mercurial
module load cmake
module load boost
module load gsl
./scons -j 10
./scons auto=all/ -j 15</code>

On Mira/Cetus@ALCF:
  * <code bash>soft add +bgqtoolchain-gcc484
soft add +mpiwrapper-gcc
soft add +python2.7.9</code>
  * Checkout from ATLAS: <code bash> svn co --username atlas https://openloops.hepforge.org/svn/OpenLoops/branches/public_collier atlas_ol</code>
    * ''pmg4711''
  * <code bash>cd atlas_ol</code>
  * Make these changes:<code diff>Index: pyol/config/default.cfg
===================================================================
--- pyol/config/default.cfg	(revision 1831)
+++ pyol/config/default.cfg	(working copy)
@@ -8,8 +8,8 @@

 [OpenLoops]

-supported_compilers = gfortran ifort
-fortran_compiler = gfortran
+supported_compilers = gfortran ifort mpif90
+fortran_compiler = mpif90
 fortran_tool = auto
 cc = gcc
 cxx = g++
Index: lib_src/olcommon/src/kind_types.F90
===================================================================
--- lib_src/olcommon/src/kind_types.F90	(revision 1831)
+++ lib_src/olcommon/src/kind_types.F90	(working copy)
@@ -18,6 +18,7 @@


 module kind_types
+  use ISO_FORTRAN_ENV
   implicit none

   integer, parameter :: intkind1 = selected_int_kind(2)
@@ -35,7 +36,8 @@
   complex(ep), parameter :: cI_ep = (0._ep, 1._ep)
 #endif
 ! #ifdef USE_qp
-  integer, parameter :: qp = selected_real_kind(33)
+!  integer, parameter :: qp = selected_real_kind(33)
+  integer, parameter :: qp = REAL128
   complex(qp), parameter :: cI_qp = (0._qp, 1._qp)
 ! #endif

Index: lib_src/openloops/src/ol_interface.F90
===================================================================
--- lib_src/openloops/src/ol_interface.F90	(revision 1831)
+++ lib_src/openloops/src/ol_interface.F90	(working copy)
@@ -1108,7 +1108,7 @@
       if (index(trim(infofilename),"."//trim(info_file_suffix)) == 0) then
         cycle
       else
-        infofilename = trim(install_path) // "/proclib/" // trim(infofilename)
+        infofilename = trim(install_path) // "/proclib/" // infofilename(2:len(infofilename))
       end if

       inquire(gf_info, opened=iqopen)
</code>
  * To build: <code bash>./scons -j 20 </code>
  * To download and build all processes: <code bash>./scons proc=all/ -j 20</code>



Setup:
    * On Tukey only <code bash>soft add +gcc-4.8.1</code>
    * On SLC6 machine: scons needs gcc-4.6+ so use ATLAS compiler:
      * <code bash> export PATH=/cvmfs/atlas.cern.ch/repo/sw/atlas-gcc/462/x86_64/slc6/x86_64-slc6-gcc46-opt/bin:$PATH</code>
      * <code bash> export LD_LIBRARY_PATH=/cvmfs/atlas.cern.ch/repo/sw/atlas-gcc/462/x86_64/slc6/x86_64-slc6-gcc46-opt/lib:$LD_LIBRARY_PATH</code>
  - Get <html><a href="http://openloops.hepforge.org/index.html" target="_blank">OpenLoops</a></html>: <code bash>svn checkout http://openloops.hepforge.org/svn/OpenLoops/branches/public ./OpenLoops</code>
  - Run Scons: <code bash> cd OpenLoops
./scons auto=all/ </code> This downloads all the process libraries and takes some time. <html><a href="http://openloops.hepforge.org/process_library.php" target="_blank">List of libraries.</a></html>.
  - To compile OpenLoops as a static library, create the file openloops.cfg in the top-level directory. <code>[OpenLoops]
shared_libraries=0
</code>