Data Management for High-Performance Scientific Computation

Traditionally, scientists and engineers investigated natural phenomena through empirical experimentation and theoretical analysis. In recent years, increases in computing power and our understanding of how to model physical phenomena have allowed simulations to become a third investigative paradigm. Now computer simulations are a vital part of applications as diverse as weather prediction, protein folding, and aircraft design.

Because recent increases in computational power have not been matched by increases in I/O subsystem speeds, I/O is an ever-increasing fraction of run time for high-performance parallel simulation codes. To make matters worse, the degree of parallelism in I/O subsystems has actually been dropping, due to the popularity of SMP-based parallel computers and shared file systems. Another complication is the fact that scientists normally visualize the results of their simulation runs on a computer located far away from the simulation platform, and movement of the data across the internet is not infrequently the slowest part of an end-to-end simulation run.

The Panda project addresses these challenges by designing new approaches for moving data to and from local and remote disks, as well as techniques for hiding the cost of these I/O activities. The new approaches are implemented in the context of the Panda parallel I/O library and related libraries, and tested in the context of real applications. The testbed applications are usually drawn from the Center for the Simulation of Advanced Rockets, where Panda is in daily use in CSAR's production codes. We view techniques that prove their merit in the context of CSAR applications as candidates for technology transfer to widely used parallel I/O libraries, such as implementations of the MPI-IO parallel I/O standard; transfer of active buffering from Panda to the ROMIO implementation of MPI-IO is currently underway.