ATC: ATLAS Center at ANL

For now, use the IBM compiler. That requires

compile.mk

to have the following changes

ifeq ($(shell uname),Linux)
#  FFF = gfortran -fno-automatic
#  FF90 = gfortran -fno-automatic
  FFF  = mpixlf77  -fno-automatic
  FF90 =  mpixlf77  -fno-automatic
#FFF = g77 -O1 -Wall -fno-automatic -Wno-globals -fno-backslash \
#        -ffast-math
endif

After a

make gen

or

make gen90

fails, one needs to do the following:

mpixlf77  -o wqqgen ./wqqusr.o ../wqqlib/wqq.o \
../alplib/alpgen.o ../alplib/alputi.o ../alplib/alppdf.o \
../alplib/Acp.o ../alplib/Asu3.o ../alplib/Aint.o

(or similar) to pass the linking phase.

Run on the command line:

soft add  +mpiwrapper-xl

and change

compile.mk

to

ifeq ($(shell uname),Linux)
#  FFF = gfortran -fno-automatic
#  FF90 = gfortran -fno-automatic
  FFF = mpif77 -fno-automatic
  FF90 = mpif90 -fno-automatic
#FFF = g77 -O1 -Wall -fno-automatic -Wno-globals -fno-backslash \
#        -ffast-math
endif

On line 182 of the file alplib/alpgen.f change
access='append' to position='append'

There's no getting around the bug by switching compilers on the Q. Replace asinh with argsinh whenever it appears:

1. alpgen.f
2. zjet.f

can use this command: find . -exec grep -l “asinh” {} \; | xargs -I GGGG sh -c 'sed “s/asinh/argsinh/g” GGGG > GGGG.tmp; mv GGGG.tmp GGGG'

ran into error: ld: -f may not be used without -shared to fix it one needs to do the following:

mpixlf77 -o wqqgen ./wqqusr.o ../wqqlib/wqq.o \ ../alplib/alpgen.o ../alplib/alputi.o ../alplib/alppdf.o \ ../alplib/Acp.o ../alplib/Asu3.o ../alplib/Aint.o

(or similar) to pass the linking phase.

Carver: Change compile.mk to have the following changes:

module swap pgi gcc module swap openmpi openmpi-gcc

Then everything is fine.

Hopper:

module swap PrgEnv-pgi PrgEnv-gnu

That requires compile.mk to have the following changes

ifeq ($(shell uname),Linux) # FFF = gfortran -fno-automatic # FF90 = gfortran -fno-automatic

FFF  = ftn 
FF90 =  ftn 

#FFF = g77 -O1 -Wall -fno-automatic -Wno-globals -fno-backslash \ # -ffast-math endif

Edison

module swap PrgEnv-intel PrgEnv-gnu

That requires compile.mk to have the following changes

ifeq ($(shell uname),Linux) # FFF = gfortran -fno-automatic # FF90 = gfortran -fno-automatic

FFF  = ftn
FF90 =  ftn

#FFF = g77 -O1 -Wall -fno-automatic -Wno-globals -fno-backslash \ # -ffast-math endif

2) Deal with MPI

2.1) Add the lines

    integer mpirank
    character*5 rankstring
    common/mpi/mpirank,rankstring

Just after the line “c common declarations” in subroutine alsdef in file alpgen.f It will be near line 209

2.2 Add the lines

C MPI variables

    include 'mpif.h'
    integer rank, size, ierror, tag, status(MPI_STATUS_SIZE)

Just before the line “c list processes”. This will be near line 230

c beginning of MPI block

    call MPI_INIT(ierror)
    call MPI_COMM_SIZE(MPI_COMM_WORLD, size, ierror)
    call MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierror)
    mpirank = rank
    call MPI_FINALIZE(ierror)
    write(6,*) 'MPI finalized: rank ', mpirank
    write( rankstring, '(i5.5)' )  mpirank

c end of MPI block

2.3) Add the lines

c work in a unique directory ./work/rankstring

    call chdir("work")
    call chdir(rankstring)

Immediately after the end of the MPI block

2.4) Just after the lines:

c save parameter values to relative variables in common blocks

    call alspar

This will be near line 338:

Add the lines

c beginning of MPI block

    iseed(1)=iseed(1)+mod(mpirank,65536)
    iseed(2)=iseed(2)+int(mpirank/65536)
    print*, 'Seeds and mpirank', iseed(1),iseed(2),mpirank

c end of MPI block

2.3 In subroutine alstio, just after include 'alpgen.inc' (near line 4584) add the lines:

    integer mpirank
    character*5 rankstring
    common/mpi/mpirank,rankstring

2.3 In subroutine alstio, replace

call alustc(fname,'.wgt',tmpstr)

with

call alustc('../../'fname,'.'rankstring'.wgt',tmpstr) immediately after write(6,*) 'Run parameters and diagnostics written to ' $ ,tmpstr(1:l) Near line 4588. This outputs the weighted event files in a common directory 3) Replace unit 5 with unit 9 for input Near line 306, after C MANDATORY INPUTS add for BlueGene open(9,file='../../input.1',action='read') for Carver open(9,file='../../input.1',access='read') This reads the input configuration from file, instead of from the command line input via a redirect as in the tutorial Then replace all READ(5, with READ(9,