ATC: ATLAS Center at ANL

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
===== Building ALPGEN =====

==== Machine dependent compile time changes ====


===For BG/Q===
----
On the command line, run \\
''soft add  +mpiwrapper-xl'' \\
and change the following in the file ''compile.mk'' \\
<code fortran>
ifeq ($(shell uname),Linux)
#  FFF = gfortran -fno-automatic
#  FF90 = gfortran -fno-automatic
  FFF = mpif77 -fno-automatic
  FF90 = mpif90 -fno-automatic
# Or if you want optimization use this
#  FFF = mpixlf77 -O5 -qarch=qp -qtune=qp -qsave -qsimd=auto -qhot=level=1 -qprefetch -qunroll=yes -qnoipa -qfloat=rsqrt:hssngl:fltint
#  FF90 = mpixlf90 -O5 -qarch=qp -qtune=qp -qsave -qsimd=auto -qhot=level=1 -qprefetch -qunroll=yes -qnoipa -qfloat=rsqrt:hssngl:fltint
#FFF = g77 -O1 -Wall -fno-automatic -Wno-globals -fno-backslash \
#        -ffast-math
endif
</code>

On line 182 of the file ''alplib/alpgen.f'' change \\
''access='append' '' \\
to \\
''position='append' '' \\

Now, Alpgen creates its own ''asinh()'' function. Alternating versions of Fortran have an asinh() function or does not have it. The version installed on the BGP does not have it, so there is no conflict. However, the BGQ version has the ''asinh()'' defined so we need to replace every mention of ''asinh()'' in Alpgen to some unique name. In this case, we chose ''argsignh()''. Use this command:
<code bash>
find . -exec grep -l "asinh" {} \; | xargs -I GGGG sh -c 'sed "s/asinh/argsinh/g" GGGG > GGGG.tmp; mv GGGG.tmp GGGG'
</code>

You will get error: \\
''ld: -f may not be used without -shared'' \\
to fix it one needs to do the following: \\
<code bash>
mpixlf77  -o wqqgen ./wqqusr.o ../wqqlib/wqq.o \
../alplib/alpgen.o ../alplib/alputi.o ../alplib/alppdf.o \
../alplib/Acp.o ../alplib/Asu3.o ../alplib/Aint.o
</code>
(or similar) to pass the linking phase.

=== Carver ===
----
Change ''compile.mk'' to have the following changes:
<code make>
module swap pgi gcc
module swap openmpi openmpi-gcc
</code>
Then everything is fine.

=== Hopper ===
----

Run \\
''module swap PrgEnv-pgi PrgEnv-gnu'' \\
That requires ''compile.mk'' to have the following changes
<code make>
ifeq ($(shell uname),Linux)
#  FFF = gfortran -fno-automatic
#  FF90 = gfortran -fno-automatic
  FFF  = ftn 
  FF90 =  ftn 
#FFF = g77 -O1 -Wall -fno-automatic -Wno-globals -fno-backslash \
#        -ffast-math
endif
</code>

=== Edison ===
----
Run \\
''module swap PrgEnv-intel PrgEnv-gnu'' \\
That requires ''compile.mk'' to have the following changes
<code make>
ifeq ($(shell uname),Linux)
#  FFF = gfortran -fno-automatic
#  FF90 = gfortran -fno-automatic
  FFF  = ftn
  FF90 =  ftn
#FFF = g77 -O1 -Wall -fno-automatic -Wno-globals -fno-backslash \
#        -ffast-math
endif
</code>

==== Introduce MPI to Alpgen ====

=== Edits to ''alplib/alpgen.f'' ===
----

In subroutine ''alsdef'' in file ''alplib/alpgen.f'' near line 209, just after \\
''c common declarations'' \\
add/merge these lines
<code fortran>
      integer mpirank
      character*5 rankstring
      common/mpi/mpirank,rankstring
      
c local variables
      double precision dummy,ranset,ranset2
      integer i,retval,chdir
c
</code>
----
Near line 230, just before \\
''c list processes'' \\
add the lines
<code fortran>
C MPI variables
      include 'mpif.h'
      integer rank, size, ierror, tag, status(MPI_STATUS_SIZE)
</code>
Following the remaining variable definitions, add this code
<code fortran>
c beginning of MPI block
c  retrieve MPI rank information
      call MPI_INIT(ierror)
      call MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierror)
      call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierror)
      mpirank = rank
      call MPI_FINALIZE(ierror)
      write(6,*) 'MPI finalized: rank ', mpirank
      write( rankstring, '(i8.8)' )  mpirank
c work in a unique directory ./work/rankstring
      retval = chdir('work/'//rankstring//CHAR(0))
      if(retval.ne.0) then
         write(6,*) 'Error changing directory to "work/rankstring" '
      endif
c end of MPI block
</code>
----
Just after the lines (near line 338)
<code fortran>
c     save parameter values to relative variables in common blocks
      call alspar
</code>
Add the lines
<code fortran>
c beginning of MPI block
      iseed(1)=iseed(1)+mod(mpirank,65536)
      iseed(2)=iseed(2)+int(mpirank/65536)
      print*, 'Seeds and mpirank', iseed(1),iseed(2),mpirank
c end of MPI block
</code>
----
In subroutine ''alstio'', just after
<code fortran>
      include 'alpgen.inc'
</code>
(near line 4584) add the lines:
<code fortran>
      integer mpirank
      character*5 rankstring
      common/mpi/mpirank,rankstring
</code>
----
Near line 4588, replace
<code fortran>
      call alustc(fname,'.wgt',tmpstr)
</code>
with 
<code fortran>
      call alustc('../../'//fname,'.'//rankstring//'.wgt',tmpstr)
</code>
immediately after
<code fortan>
      write(6,*) 'Run parameters and diagnostics written to '
     $        ,tmpstr(1:l)
</code>
This outputs the weighted event files in a common directory
----
Replace unit ''5'' with unit ''9'' for input near line 306, after \\
''C MANDATORY INPUTS add'' \\
For BlueGene
<code fortran>
      open(9,file='../../input.1',action='read') 
</code>
For Carver
<code fortran>
      open(9,file='../../input.1',access='read') 
</code>
This reads the input configuration from file, instead of from the command line input via a redirect as in the tutorial.
Then replace all \\
''READ(5,'' \\
with \\
''READ(9,''
This ensures everywhere is reading from the input file instead of ''stdin''. Only change this in the ''alplib/alpgen.f'' file.