ATC: ATLAS Center at ANL

For now, use the IBM compiler. That requires compile.mk to have the following changes

ifeq ($(shell uname),Linux)
#  FFF = gfortran -fno-automatic
#  FF90 = gfortran -fno-automatic
  FFF  = mpixlf77  -fno-automatic
  FF90 =  mpixlf77  -fno-automatic
#FFF = g77 -O1 -Wall -fno-automatic -Wno-globals -fno-backslash \
#        -ffast-math
endif

Doing a make gen or make gen90 will fail at the linking stage because it must be linked with the MPI linker, mpixlf77. One needs to do

mpixlf77  -o wqqgen ./wqqusr.o ../wqqlib/wqq.o \
../alplib/alpgen.o ../alplib/alputi.o ../alplib/alppdf.o \
../alplib/Acp.o ../alplib/Asu3.o ../alplib/Aint.o

(or similar) to pass the linking phase.

On the command line, run
soft add +mpiwrapper-xl
and change the following in the file compile.mk

ifeq ($(shell uname),Linux)
#  FFF = gfortran -fno-automatic
#  FF90 = gfortran -fno-automatic
  FFF = mpif77 -fno-automatic
  FF90 = mpif90 -fno-automatic
#FFF = g77 -O1 -Wall -fno-automatic -Wno-globals -fno-backslash \
#        -ffast-math
endif

On line 182 of the file alplib/alpgen.f change
access='append'
to
position='append'

Now, Alpgen creates its own asinh() function. Alternating versions of Fortran have an asinh() function or does not have it. The version installed on the BGP does not have it, so there is no conflict. However, the BGQ version has the asinh() defined so we need to replace every mention of asinh() in Alpgen to some unique name. In this case, we chose argsignh(). Use this command:

find . -exec grep -l "asinh" {} \; | xargs -I GGGG sh -c 'sed "s/asinh/argsinh/g" GGGG > GGGG.tmp; mv GGGG.tmp GGGG'

You will get error:
ld: -f may not be used without -shared
to fix it one needs to do the following:

mpixlf77  -o wqqgen ./wqqusr.o ../wqqlib/wqq.o \
../alplib/alpgen.o ../alplib/alputi.o ../alplib/alppdf.o \
../alplib/Acp.o ../alplib/Asu3.o ../alplib/Aint.o

(or similar) to pass the linking phase.

Change compile.mk to have the following changes:

module swap pgi gcc
module swap openmpi openmpi-gcc

Then everything is fine.

module swap PrgEnv-pgi PrgEnv-gnu
That requires compile.mk to have the following changes

ifeq ($(shell uname),Linux)
#  FFF = gfortran -fno-automatic
#  FF90 = gfortran -fno-automatic
  FFF  = ftn 
  FF90 =  ftn 
#FFF = g77 -O1 -Wall -fno-automatic -Wno-globals -fno-backslash \
#        -ffast-math
endif

module swap PrgEnv-intel PrgEnv-gnu

That requires compile.mk to have the following changes

ifeq ($(shell uname),Linux) # FFF = gfortran -fno-automatic # FF90 = gfortran -fno-automatic

FFF  = ftn
FF90 =  ftn

#FFF = g77 -O1 -Wall -fno-automatic -Wno-globals -fno-backslash \ # -ffast-math endif

2) Deal with MPI

2.1) Add the lines

    integer mpirank
    character*5 rankstring
    common/mpi/mpirank,rankstring

Just after the line “c common declarations” in subroutine alsdef in file alpgen.f It will be near line 209

2.2 Add the lines

C MPI variables

    include 'mpif.h'
    integer rank, size, ierror, tag, status(MPI_STATUS_SIZE)

Just before the line “c list processes”. This will be near line 230

c beginning of MPI block

    call MPI_INIT(ierror)
    call MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierror)
    call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierror)
    mpirank = rank
    call MPI_FINALIZE(ierror)
    write(6,*) 'MPI finalized: rank ', mpirank
    write( rankstring, '(i5.5)' )  mpirank

c end of MPI block

2.3) Add the lines

c work in a unique directory ./work/rankstring

    call chdir("work")
    call chdir(rankstring)

Immediately after the end of the MPI block

2.4) Just after the lines:

c save parameter values to relative variables in common blocks

    call alspar

This will be near line 338:

Add the lines

c beginning of MPI block

    iseed(1)=iseed(1)+mod(mpirank,65536)
    iseed(2)=iseed(2)+int(mpirank/65536)
    print*, 'Seeds and mpirank', iseed(1),iseed(2),mpirank

c end of MPI block

2.3 In subroutine alstio, just after include 'alpgen.inc' (near line 4584) add the lines:

    integer mpirank
    character*5 rankstring
    common/mpi/mpirank,rankstring

2.3 In subroutine alstio, replace

call alustc(fname,'.wgt',tmpstr)

with

call alustc('../../'fname,'.'rankstring'.wgt',tmpstr) immediately after write(6,*) 'Run parameters and diagnostics written to ' $ ,tmpstr(1:l) Near line 4588. This outputs the weighted event files in a common directory 3) Replace unit 5 with unit 9 for input Near line 306, after C MANDATORY INPUTS add for BlueGene open(9,file='../../input.1',action='read') for Carver open(9,file='../../input.1',access='read') This reads the input configuration from file, instead of from the command line input via a redirect as in the tutorial Then replace all READ(5, with READ(9,