Building Gaussian 09 with Intel compiler and MKL

OK, my previous attempt on g03 has failed. So I try to work on g09 instead.
Building of Gaussian with Intel compiler is really a pain (at least for me). Stick to PGI if you can. The followings are what I have figured out on the building procedures of Gaussian 09.

Note: I can only build dynamically linked binary so far.

1. The machine type is determined by $g09root/gau-machine, for Linux x86-64, it should return ‘amd64’
2. Two scripts in $g09root/bsd/ directory will influence the make flags: setup-make and set-mflags
3. The CPU cachesize (and CSIZE/CSIZEW variables) is determined by $g09root/bsd/cachesize
4. Some of the environment variables are set in $g09root/bsd/bldg09 script
5. There might be other files need to be modified, but the most critical one is bsd/i386.make

If everything is in order, a single bsd/bldg09 command should build g09 binary for you. But life is not so easy. When we issued the
bsd/bldg09
command, the script will try to determine your machine type, and link appropriate make file to g09.make (in the case of amd64, i386.make is linked). That’s why we need to modify bsd/i386.make. Before that, we need to modify bsd/setup-make and bsd/set-mflags first.

For setup-make:
Search for ‘amd64’, change set fcname = “FCN=’pgf77 -Bstatic_pgi'” and subsequent lines to the following:

set fcname = “FCN=ifort”
set lfort = “ifort -fPIC -shared-intel -mkl -m64”
set blas = “BLAS=” UTIL_NAME=’libg09util.so'”
set blasp = “$blas”
set pstr = “-p”
set fcflag = “FC=’-L/opt/intel/lib/intel64 -WB -i8 -r8 -w -ftz -switch fe_new_cgoto -fp-relaxed -fp-speculation=safe -shared-intel  -mkl -m64 -parallel -openmp -fpp2 -a
uto -O3′”
set fcflagp = “FC=’-L/opt/intel/lib/intel64 -WB -i8 -r8 -w -ftz -switch fe_new_cgoto -fp-relaxed -fp-speculation=safe -shared-intel -mkl -m64 -parallel -openmp -fpp2 -a
uto -O3 -p'”
set fc2flag = “FC2=’-L/opt/intel/mkl/lib/intel64 -lmkl_intel_ilp64 -lmkl_core -lmkl_intel_thread'”
It seems these modification may not actually take effect. I modify them just to be safe (or not).

For set-mflags:
Search for ‘amd64’, change the following line:
set mty = “k8-64”
into
set mty = “”
and the lines:
set blasname = “amd64$blstr”
set mflags = “$mflags OPTOI= MMODEL=’-mcmodel=medium’ SPECFLAG=$specflag NKSEC=-DDEFKSEC=256”
into
set blasname = “”
set mflags = “$mflags OPTOI= MMODEL=-mcmodel=medium SPECFLAG=$specflag”
and another line below:
set mflags=”$mflags MACHTY=$mty GAULIBU=util.a XGEMMINI=-DXGEMM_INIT BLAS1=bsd/libf77blas-$blasname.a BLAS2=bsd/libatlas-$blasname.a”
into
set mflags=”$mflags MACHTY=$mty GAULIBU=libg09util.so XGEMMINI=-DXGEMM_INIT BLAS1= BLAS2=”

For i386.make:
Comment out BLASL/BLAS1/BLAS2/BLAS definitions.
Make the following modifications (I make this change because I want to build a formal util.so library, you don’t need to):
GAULIBU = libg09util.so
GAULIB = -lg09util # you can use GAULIB = $(GAULIBU) or GAULIB = util.so

# the two lines can be omitted, since they were already set in set-mflags

CSIZE = 12582912 # reported by bsd/cachesize, each CPU may have different cache size

CSIZEW = 1536  # can set to $(CSIZE)/8192

FPARFLAG = -openmp -parallel #-mp=nonuma

I8FLAG = -i8
R8FLAG = -r8
I8CPP1=-DI64
I8CPP2=-DP64
I8CPP3=-DPACK64
I8CPP4=-DUSE_I2
# these lines should precede I8CPP definition

XGEMMINI =-DXGEMM_INIT

# these three were adjustable, you could leave them as the original value
NISEC = -DDEFISEC=32
NJSEC = -DDEFJSEC=256
NKSEC = -DDEFKSEC=512

Here come the critical ones:
RUNCC = icc -fPIC -mkl -shared-intel
…
VECTOR = #-Mvect=assoc,recog,cachesize:$(CSIZE)$(VECTOR4)
MACHTY = #k8-64
MACH = #-tp $(MACHTY) $(TIME)
OPTOI = #-m64 -march=native -malign-double
OPTFLAGO = #$(OPTOI) -O3 -funroll-loops
…
MMODEL = -mcmodel=medium
PGISTATIC =
RUNF77 = ifort -fPIC -mkl -shared-intel $(I8FLAG) $(R8FLAG) $(MMODEL) $(DEBUGF)
RUNF77SO = ifort -fPIC -mkl -shared-intel -Bdynamic $(I8FLAG) $(R8FLAG) $(MMODEL) $(DEBUGF)
…
SYSLIBS = -L/home/frankch/app/gaussian/g09 -L/opt/intel/lib/intel64 -lifport -lifcoremt -liomp5 -lsvml -limf -lpthread -lm -lc
…
LIBS = $(NUMALIB) $(SYSLIBS) $(MKLLIB)
UNROLL         = -unroll
…
PC64 = #-pc 64 -Kieee
…
OPTFLAG = -O2 $(UNROLL) $(VECTOR)
…
FFLAGS = -WB -w -ftz -fp-relaxed -shared-intel -fp-speculation=safe $(FPARFLAG) $(PROFFLAG) $(MACH) $(OPTFLAG) $(LINK1) $(LINK2)
…
all:    libg09util.so exe
…
exe:   $(GAUEXE) $(GAULIBU) $(GAULIB) exe1 exe3 exe5 exe7 exe9 exe10 exe11 utils
…
# for g09 target
g09: …
…
$(RUNF77) $(FFLAGS) -o g09 ml0.o $(EXTOBJ) $(EXTRAS) $(LIBS) $(GAULIB)

# for util.so target, change util.so into libg09util.so

libg09util.so: $(GAULIBA) $(BLAS) $(LAPACK)

        – $(RUNAR) dv util.a bdrys.o bdam1.o
$(BLASL) $(PGISO2) $(SYSLIBS)
       $(RUNF77SO) $(FFLAGS) -shared -Wl,-soname,libg09util.so -o libg09util.so -Xlinker –whole-archive $(GAULIBA) \
        -Xlinker –no-whole-archive $(LIBS)

We don’t need to specify the mkl libraries because the compiler/linker flag -mkl have done these for us.

Another fine need to be modified:
Edit bsd/mdutil.c
change the following around line #380 or so

#define NEED_AND
#define NEED_ISHFT
#define NEED_GSR48
into
#undef NEED_AND
#undef NEED_ISHFT
#define NEED_GSR48
#define NEED_PUTENV