ATC: ATLAS Center at ANL

On the command line, run
soft add +mpiwrapper-xl
and change the following in the file compile.mk

ifeq ($(shell uname),Linux)
#  FFF = gfortran -fno-automatic
#  FF90 = gfortran -fno-automatic
  FFF = mpif77 -fno-automatic
  FF90 = mpif90 -fno-automatic
# Or if you want optimization use this
#  FFF = mpixlf77 -O5 -qarch=qp -qtune=qp -qsave -qsimd=auto -qhot=level=1 -qprefetch -qunroll=yes -qnoipa -qfloat=rsqrt:hssngl:fltint
#  FF90 = mpixlf90 -O5 -qarch=qp -qtune=qp -qsave -qsimd=auto -qhot=level=1 -qprefetch -qunroll=yes -qnoipa -qfloat=rsqrt:hssngl:fltint
#FFF = g77 -O1 -Wall -fno-automatic -Wno-globals -fno-backslash \
#        -ffast-math
endif

On line 182 of the file alplib/alpgen.f change
access='append'
to
position='append'

Now, Alpgen creates its own asinh() function. Alternating versions of Fortran have an asinh() function or does not have it. The version installed on the BGP does not have it, so there is no conflict. However, the BGQ version has the asinh() defined so we need to replace every mention of asinh() in Alpgen to some unique name. In this case, we chose argsignh(). Use this command:

find . -exec grep -l "asinh" {} \; | xargs -I GGGG sh -c 'sed "s/asinh/argsinh/g" GGGG > GGGG.tmp; mv GGGG.tmp GGGG'

You will get error:
ld: -f may not be used without -shared
to fix it one needs to do the following:

mpixlf77  -o wqqgen ./wqqusr.o ../wqqlib/wqq.o \
../alplib/alpgen.o ../alplib/alputi.o ../alplib/alppdf.o \
../alplib/Acp.o ../alplib/Asu3.o ../alplib/Aint.o

(or similar) to pass the linking phase.

Change compile.mk to have the following changes:

module swap pgi gcc
module swap openmpi openmpi-gcc

Then everything is fine.

Run
module swap PrgEnv-pgi PrgEnv-gnu
That requires compile.mk to have the following changes

ifeq ($(shell uname),Linux)
#  FFF = gfortran -fno-automatic
#  FF90 = gfortran -fno-automatic
  FFF  = ftn 
  FF90 =  ftn 
#FFF = g77 -O1 -Wall -fno-automatic -Wno-globals -fno-backslash \
#        -ffast-math
endif

Run
module swap PrgEnv-intel PrgEnv-gnu
That requires compile.mk to have the following changes

ifeq ($(shell uname),Linux)
#  FFF = gfortran -fno-automatic
#  FF90 = gfortran -fno-automatic
  FFF  = ftn
  FF90 =  ftn
#FFF = g77 -O1 -Wall -fno-automatic -Wno-globals -fno-backslash \
#        -ffast-math
endif

In subroutine alsdef in file alplib/alpgen.f near line 209, just after
c common declarations
add/merge these lines

      integer mpirank
      character*5 rankstring
      common/mpi/mpirank,rankstring
 
c local variables
      double precision dummy,ranset,ranset2
      integer i,retval,chdir
c

Near line 230, just before
c list processes
add the lines

C MPI variables
      include 'mpif.h'
      integer rank, size, ierror, tag, status(MPI_STATUS_SIZE)

Following the remaining variable definitions, add this code

c beginning of MPI block
c  retrieve MPI rank information
      call MPI_INIT(ierror)
      call MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierror)
      call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierror)
      mpirank = rank
      call MPI_FINALIZE(ierror)
      write(6,*) 'MPI finalized: rank ', mpirank
      write( rankstring, '(i8.8)' )  mpirank
c work in a unique directory ./work/rankstring
      retval = chdir('work/'//rankstring//CHAR(0))
      if(retval.ne.0) then
         write(6,*) 'Error changing directory to "work/rankstring" '
      endif
c end of MPI block

Just after the lines (near line 338)

c     save parameter values to relative variables in common blocks
      call alspar

Add the lines

c beginning of MPI block
      iseed(1)=iseed(1)+mod(mpirank,65536)
      iseed(2)=iseed(2)+int(mpirank/65536)
      print*, 'Seeds and mpirank', iseed(1),iseed(2),mpirank
c end of MPI block

In subroutine alstio, just after

      include 'alpgen.inc'

(near line 4584) add the lines:

      integer mpirank
      character*5 rankstring
      common/mpi/mpirank,rankstring

Near line 4588, replace

      call alustc(fname,'.wgt',tmpstr)

with

      call alustc('../../'//fname,'.'//rankstring//'.wgt',tmpstr)

immediately after

      write(6,*) 'Run parameters and diagnostics written to '
     $        ,tmpstr(1:l)

This outputs the weighted event files in a common directory

Replace unit 5 with unit 9 for input near line 306, after
C MANDATORY INPUTS add
For BlueGene

      open(9,file='../../input.1',action='read') 

For Carver

      open(9,file='../../input.1',access='read') 

This reads the input configuration from file, instead of from the command line input via a redirect as in the tutorial. Then replace all
READ(5,
with
READ(9, This ensures everywhere is reading from the input file instead of stdin. Only change this in the alplib/alpgen.f file.