PICS Flow Code

During the last four years I have been heavily involved in a large scale scientifc computing project funded by the Department of Energy. The project is focused on the development of a sophisticated computer simulator of groundwater flow and transport (PICS groundwater). The target computer architecture is Intel Paragon supercomputer which is a distributed memory parallel machine.

My involvement in this project includes a development of parallel multiphase flow simulator in 3D. The code is designed to allow great flexibility in both utilizing complex physics and geometry that defines the model and efficient algorithms that solve the discrete model in parallel. The fractional flow model is adopted, combined with tetrahedra based mixed finite elements on logically rectangular grids in three spatial dimensions. The solution techniques are based on domain decomposition. Roughly speaking, in our approach each subdomain is attached to a processor and the code organizes the needed communications in order to solve in parallel. Efficient indefinite solvers are used to solve the discrete numerical model.

The programming environment used for code development is ANSI C with extensions such as the system for remote procedure calls (IPX) developed in the Brookhaven National Lab, Intel Paragon parallel primitives and the system for literate programming FWEB. In the earlier versions of the simulator the numerical part was written in FORTRAN. For the future, I think that writing some modules in C++ will improve considerably the structure of the code.

Computational results from a simulation with the early version of a flow and transport simulator, originally developed by Michael Celia and later further developed and parallelized by Joseph Pasciak and me, can be seen here. MPG movies (short and long) from the simulations are also available. This particular example simulates contaminant transport in porous media with uniform properites. The computational domain has fairly complex bottom topology. There are 137x129x11 grid nodes. The simulation ran on a 56-node Paragon (University of South Carolina) for 4 days.