Difference between revisions of "Software Required to Build GridPACK"

Revision as of 17:11, 17 November 2017

Building on Specific Platforms

Scripts or instructions for installing these libraries on specific platforms can be found by following the links below. We highly recommend that you find an example that resembles the platform you are intending to use. Use the scripts for that example as the basis for building GridPACK on your own system. Most scripts require only minor modifications to get them working. These usually involve modifying file names that reflect your local directory structure.

• Mac OS X (Yosemite) with MacPorts
• Red Hat Enterprise Linux 5
• Ubuntu Linux
• PNNL PIC Cluster (Linux cluster with Infiniband)
• Native Windows Port

Many clusters use modules to install software such as the compilers, MPI libraries, CMake, Git, etc. If you have such a system, you can install compilers, MPI and CMake using a set of commands such as

  module purge
  module load gcc
  module load openmpi
  module cmake

These automatically install the libraries and modify environment variables so that binaries and executables are in your path. Many systems may also have modules for Boost and PETSc, but we urge users to be more cautious about using these. These libraries are frequently not built with features that may be needed by individual applications. For example, system builds of Boost often lack MPI and PETSc is frequently missing the C++ interface. Users are probably better off using their own library builds. This also guarantees that all libraries are built with the same compiler and version of MPI.

CMake/CTest

GridPACK uses the CMake cross-platform build system. A reasonably modern version is required. Currently, version 2.8.8 or newer is required. You can check which version of CMake is on your machine by typing

 cmake -version

CMake projects are designed to be built outside of the source code location. In the top directory of a GridPACK release (GRIDPACK/src) create a subdirectory to use as the location of the build. (In this documentation, GRIDPACK stands for the location of the top level GridPACK directory.) The GRIDPACK/src directory should contain a file called CMakeList.txt. Configure and build GridPACK in the subdirectory.

MPI

A working MPI implementation is required. OpenMPI and MPICH been used successfully. We have used OpenMPI 1.8.3 and MPICH 3.2 recently. Other implementations have also been used successfully. Most MPI installations have compiler wrappers, such as mpicc and mpicxx that combine the compiler with all the directives needed to link to the MPI libraries. These can be used directly in the GridPACK configuration.

Identify the compilers and mpiexec to the configuration by including cmake options like:

   -D MPI_CXX_COMPILER:STRING='mpicxx' \
   -D MPI_C_COMPILER:STRING='mpicc' \
   -D MPIEXEC:STRING='mpiexec' \

You can check to see if these wrappers are available on your system by typing

 which mpicc

If the wrapper is available, you should see a listing pointing to the location of the wrapper. If you don't, you will need to modify your environment to include it. Note that although mpicc and mpicxx are fairly common names for the compiler wrappers, there is no standard and other implementations of MPI may use something completely different. Check with your system consultant for more information. Depending on the version of MPI you are using, you may be able to find out more information by typing

 mpicxx -v

Other options may be needed by CMake to specify the MPI environment. See the documentation here.

Global Arrays

GridPACK depends heavily on "Global Arrays". The GA libraries used with GridPACK must have the C++ interface be enabled and the Fortran interface disabled. The GridPACK configuration is not able to identify additional required libraries if the Fortran interface is enabled or independent BLAS/LAPACK libraries are used.

To configure GridPACK, specify the directory where Global Arrays is installed and any extra libraries that are required:

   -D GA_DIR:PATH=/path/to/ga/install \
   -D GA_EXTRA_LIBS:STRING="..." \
   -D USE_PROGRESS_RANKS:BOOL=FALSE \

The GA_EXTRA_LIBS variable is used to include required libraries not identified in the configuration. The USE_PROGRESS_RANKS variable depends on the runtime used to build GA.

We have used three versions of the GA build to run GridPACK. If you are using GridPACK on a Linux cluster with an Infiniband interconnect, then you can use the OpenIB runtime (--with-openib option) for building GA. This is the highest performing version of GA for clusters with Infiniband, although for large calculations you can run into problems with memory allocatin. For any system with a working version of MPI, you can also use the MPI two-sided runtime (--with-mpi-ts) or the progress ranks runtime (--with-mpi-pr). The two-sided runtime is the simplest runtime and is suitable for workstations with a limited number of cores (probably 8 or less). The two-sided runtime provides reasonable performance on a small number of cores but slows down considerably for larger numbers of cores. It is not recommended for large-scale parallel computation. The progress ranks runtime is much higher performing and approaches the performance of the OpenIB runtime. It is very reliable and runs on any platform that supports MPI. However, it has one peculiarity in that it reserves on MPI process on each SMP node to act as a communication manager. Thus, if you are running your calculation on 2 nodes with 5 processes on each node, the GridPACK application will only see 8 processes (4 on each node). To make sure that the GridPACK build is aware of this, the USE_PROGRESS_RANKS parameter should be set to TRUE when using the progress ranks build of GA.

Boost

The Boost C++ Library is used heavily throughout the GridPACK framework, and a relatively recent version is required. The configuration requires version 1.49 or later, but older versions may work. The Boost installation must include Boost::MPI which must have been built with the same MPI compiler used for GridPACK.

To configure GridPACK one need only specify where Boost is installed, like this

   -D BOOST_ROOT:STRING='/path/to/boost' \

Boost is tied quite closely to the latest features in C++ and problems can be encountered if the version of Boost that you are using was released much later than the compiler. Reverting to an earlier Boost version can sometimes eliminate problems if you are having difficulties building it. The same is true for Boost and CMake. If the CMake version was released earlier than the Boost version, CMake may have problems identifying the libraries in Boost that it needs for GridPACK. Again, going to an earlier version of Boost may fix these issues.

PETSc

GridPACK currently relies on the Portable, Extensible Toolkit for Scientific Computation (PETSc) for parallel linear algebra, and linear and nonlinear system solvers. PETSc is a complicated package with numerous options. PETSc needs to be built with MPI enabled and using the same MPI implementation used for GridPACK. It also needs to use C++ as the base language. Originally, GridPACK could only use PETSc if it was configured for complex support. The current GridPACK release can use either complex or real builds. However, most applications in GridPACK use complex matrices, so it is still preferable to configure PETSc to use complex variables. Refer to the PETSc installation documentation for additional information on how configuring PETSc.

Configuring and building PETSc is done in the top level PETSc directory. One of the configuration variables that needs to be set when configuring and building PETSc is PETSC_ARCH. In the example below, PETSC_ARCH was set to 'arch-Darwin-cxx-opt'. After the build is complete, there will be a directory beneath the top level directory with whatever name was assigned to PETSC_ARCH. This directory contains the include and lib directories for the PETSc libraries.

The GridPACK configuration must know where PETSc is installed. This is specified by two options as shown below.

   -D PETSC_DIR:STRING='/Users/d3g096/ProjectStuff/petsc-3.4.0' \
   -D PETSC_ARCH:STRING='arch-darwin-cxx-opt' \

Currently, the configuration will recognize and adjust the GridPACK build if the PETSc build includes ParMETIS and/or Superlu_DIST.

ParMETIS

GridPACK uses ParMETIS to (re)distribute an electrical network over several processors. It needs to be built with the same MPI configuration as Boost and PETSc. GridPACK configuration will find ParMETIS automatically if it has been included in the PETSc build. Otherwise, the GridPACK configuration just needs to know where ParMETIS was installed, which is specified by

   -D PARMETIS_DIR:STRING="/pic/projects/gridpack/software" \

GridPACK requires version ParMETIS version 4.0. Older versions will not work.

Doxygen

GridPACK uses Doxygen to help document code. It's use is optional. Doxygen documentation can optionally be prepared during the build process. This is enabled if Doxygen is found. Graphviz is necessary for full documentation features.