Building on Mac Yosemite

This system is a relatively new Mac Pro with Mac OS X 10.10 (Yosemite). The previous instructions for Mac OS X are really not helpful anymore due to changes in the OS.

Preparation

Create a suitable directory where GridPACK and related packages can be built and resulting binaries installed. This directory will subsequently be referred to as $prefix. This example uses ~/Projects/GridPACK. Before starting, type:

 set prefix = ~/Projects/GridPACK

Adjust this to your personal set up. Also, the C-shell is used in this example; some commands may need adjustment if another shell such as the Bourne-shell is used.

Compiler Set

Mac systems do not have a compiler out of the box. To get the stock Apple compiler (LLVM), XTools must be installed. Here is a good guide.

Care must be taken to use a consistent set of compilers when building GridPACK. GridPACK is a C++ code. Mac OS X provides two standard C++ libraries on modern systems: libc++ (the one Apple wants to support), and libstdc++ (that will be phased out). The stock LLVM C++ compiler will use libc++. GNU compilers, if installed, will use libstdc++. If C++ objects from GNU compilers are mixed with the system compiler, they will not link together, because of conflicting and/or missing symbols from the standard library. This gets complicated. By far the best thing to do is use the same compilers to build GridPACK and any associated libraries such as Boost and PETSc.

In addition, some other problems were encountered while getting this to work:

• Compiling Boost.MPI with the stock LLVM compiler seemed to create problems with serialization needed in the GridPACK code. This may or may not be a real issue. After the trouble was encountered, the CLang compilers were chosen and used there after. (NOTE: Boost does not report Boost.MPI unit test results for LLVM or CLang compilers on any platform).
• Global Arrays 5.5 does not appear to work with LLVM or CLang, but the SVN trunk version does. Neither version worked as shared libraries, so GA must be built static.

After a couple of aborted starts with the LLVM compiler, CLang 3.8 (installed with MacPorts) was chosen, and, after some effort, it built GridPACK. GNU C++ should be avoided.

MacPorts

The CLang compiler set and some of the prerequisite software was installed on this system using MacPorts. Several necessary packages were installed by

   sudo port install boost boost @1.59.0_2+clang38+no_single-no_static+openmpi+python27

Note that the openmpi Boost variant was installed. This will install the CLang compiler set and OpenMPI. It's a good idea to install the select packages for CLang and MPI and make sure the right versions are "selected":

   sudo port install mpi_select clang_select
   sudo port select --set mpi openmpi-clang38-fortran
   sudo port select --set clang mp-clang-3.8

The following ports were installed:

 cmake @3.7.2_0+docs+gui+python27+qt4 (active)
 boost @1.59.0_2+clang38+no_single+openmpi+python27 (active)
 OpenBLAS @0.2.19_0+clang+lapack (active)
 doxygen @1.8.13_2+qt4+wizard (active)
 graphviz @2.40.1_1+pangocairo+x11 (active)

OpenBLAS was chosen in an attempt to be compatible with [1]. Doxygen and Graphviz are optional.

Building PETSc

PETSc 3.7.4 source was downloaded and unpacked in $prefix. It was then configured and built as follows:

   setenv PETSC_DIR "$prefix/petsc-3.7.4"
   unsetenv PETSC_ARCH
   python ./configure \
       PETSC_ARCH=arch-macosx-clang-real-opt \
       COPTFLAGS="-O3" \
       CXXOPTFLAGS="-O3" \
       FOPTFLAGS="-O3" \
       --with-prefix="$prefix" \
       --with-mpi=1 \
       --with-cc=/opt/local/bin/mpicc \
       --with-fc=/opt/local/bin/mpif90 \
       --with-cxx=/opt/local/bin/mpicxx \
       --with-clanguage=c++ \
       --with-c++-support=1 \
       --with-c-support=0 \
       --with-fortran=0 \
       --with-scalar-type=real \
       --with-precision=double \
       --with-fortran-kernels=generic \
       --with-valgrind=0 \
       --with-blas-lapack-lib="/opt/local/lib/libopenblas.a" \
       --download-suitesparse=1 \
       --download-superlu_dist=1 \
       --download-parmetis=1 \
       --download-metis=1 \
       --download-f2cblaslapack=0 \
       --download-mumps=0 \
       --download-scalapack=0 \
       --with-shared-libraries=0 \
       --with-x=0 \
       --with-mpirun=mpiexec \
       --with-mpiexec=mpiexec \
       --with-debugging=0
   make PETSC_DIR="$prefix/petsc-3.7.4 PETSC_ARCH=arch-macosx-clang-real-opt all

Here, OpenBLAS, installed with MacPorts, was used for BLAS and LAPACK. Other packages (ParMETIS, SuperLU_DIST, etc.) were downloaded and built by the PETSc configuration. PETSc can be built with complex support, if desired, by changing the --with-scalar-type=real to --with-scalar-type=complex and changing the value of PETSC_ARCH.

It is possible to use the MacPorts PETSc package for GridPACK. I does not include SuperLU_DIST, however. This is left as an exercise for the reader.

Building Global Arrays

The trunk SVN version of Arrays is required here. Get the code using

 svn checkout https://svn.pnl.gov/svn/hpctools/trunk/ga $prefix/ga-svn

Configure, build, and install:

 cd $prefix/ga-svn
 ./configure \
   --enable-cxx \
   --disable-f77 \
   --enable-i4 \
   --with-mpi \
   --with-mpi-ts \
   --enable-autodetect=yes \
   --prefix="$prefix" \
   --without-blas \
   --without-lapack \
   --without-scalapack \
   --enable-shared=no \
   --enable-static=yes \
   MPICC=mpicc MPICXX=mpicxx MPIF77=mpif90 \
   MPIEXEC=mpiexec MPIRUN=mpirun NPROCS=4 \
   LDFLAGS="-L/opt/local/lib"
 make
 make install

I had trouble using shared GA libraries, so only static libraries should be built. Version 5.5 did not work (some problem with process groups).

Building and Testing GridPACK

At the time of writing, the GridPACK code needed is from a fork other than the main repository. Some minor changes in that fork will eventually make their way to the main repository. Get the GridPACK code using

 git clone https://github.com/wperkins/GridPACK.git $prefix/src/gridpack

CMake projects are designed to be built outside of the source code location. The top of the the CMake project is $prefix/src/gridpack/src. Make a subdirectory there called build or something. Configure and build GridPACK in that directory as follows:

 cd $prefix/src/gridpack/src
 mkdir build
 cd build
 setenv CC /opt/local/bin/clang
 setenv CXX /opt/local/bin/clang++
 cmake \
       -D GA_DIR:STRING="$prefix" \
       -D BOOST_ROOT:STRING="/opt/local" \
       -D PETSC_DIR:STRING="$prefix/petsc-3.7.4" \
       -D PETSC_ARCH:STRING="arch-macosx-clang-real-opt" \
       -D MPI_CXX_COMPILER:STRING='/opt/local/bin/mpicxx' \
       -D MPI_C_COMPILER:STRING='/opt/local/bin/mpicc' \
       -D MPIEXEC:STRING='/opt/local/bin/mpiexec' \
       -D MPIEXEC_MAX_NUMPROCS:STRING="4" \
       -D GRIDPACK_TEST_TIMEOUT:STRING=10 \
       -D USE_CPLEX:BOOL=OFF \
       -D USE_GLPK:BOOL=ON \
       -D GLPK_ROOT_DIR:PATH="/opt/local" \
       -D CMAKE_INSTALL_PREFIX:PATH="$prefix" \
       -D BUILD_SHARED_LIBS:BOOL=OFF \
       -D CMAKE_BUILD_TYPE:STRING=Release \
       -D CMAKE_VERBOSE_MAKEFILE:BOOL=FALSE \
       ..
 make
 make test
 make install

Note that this configuration is in src/example_configuration.sh for host WE32673.