Scientific and technical programmers can no longer afford to treat I/O as an afterthought. The speed, memory size, and disk capacity of parallel computers continue to grow rapidly, but the rate at which disk drives can read and write data is improving far less quickly. As a result, the performance of carefully tuned parallel programs can slow dramatically when they read or write files-and the problem is likely to get far worse.
Parallel input and output techniques can help solve this problem by creating multiple data paths between memory and disks. However, simply adding disk drives to an I/O system without considering the overall software design will not significantly improve performance. To reap the full benefits of a parallel I/O system, application programmers must understand how parallel I/O systems work and where the performance pitfalls lie.
Parallel I/O for High Performance Computing directly addresses this critical need by examining parallel I/O from the bottom up. This important new book is recommended to anyone writing scientific application codes as the best single source on I/O techniques and to computer scientists as a solid up-to-date introduction to parallel I/O research.
* An overview of key I/O issues at all levels of abstraction-including hardware, through the OS and file systems, up to very high-level scientific libraries.
* Describes the important features of MPI-IO, netCDF, and HDF-5 and presents numerous examples illustrating how to use each of these I/O interfaces.
* Addresses the basic question of how to read and write data efficiently in HPC applications.
* An explanation of various layers of storage - and techniques for using disks (and sometimes tapes) effectively in HPC applications.
Suitable for the scientific researcher, computer-science student, or anyone else who might be interested in high-end computers,
Parallel I/O for High-Performance Computing is a remarkably clear guide to recent research and expertise in parallel computing, and centers on ways for computers to process very large data sets more efficiently.
Although the title makes it sound as if its focus were very narrow, this book delivers an engaging glimpse into the world of serious, high-end scientific computing. It begins with a thorough tour of the issues that surround data storage. (The author writes clearly and meticulously, and describes trends in data storage, from disk drives and RAID technology to newer developments, like holographic data-storage systems.) Standards that are used by parallel computers to connect to data storage in parallel architectures are surveyed; also, there's extensive coverage of parallel file systems that have attempted to solve the problem of accessing terabytes of information at high speeds, for simulations and forecasting applications.
This book is very good at describing and summarizing a host of standards that have been developed by both academic and corporate researchers. For the working scientific programmer, sections that look at specific APIs for fast parallel I/O perhaps will be more immediately useful. There's coverage of MPI-IO, HPF I/O, and SIO LLAPI (three possible standards for parallel computing) for optimized file I/O for parallel systems. A section on two scientific libraries (NetCDF and HDF) that simplify loading numerical data offers complete sample programs in C, which show these APIs in action.
The purpose of this book is to expose the reader to the state of the art in high-end parallel computing, and suggest that I/O often is the bottleneck to achieving the best possible performance in scientific software. This far-ranging and intelligent guide will benefit any academic or scientific programmer, as well as anyone who wants to understand the fascinating world of high-end parallel systems that are used for simulations and serious number crunching. --Richard Dragan
Topics covered:
- Overview of high-performance I/O requirements
- Trends in I/O performance
- I/O levels (RAID technology, parallel file systems)
- Storage devices (primary, secondary, and tertiary storage)
- Magnetic disk drives
- Disk performance
- Magnetic tape
- Optical disks
- Parallelism and I/O requirements
- In-depth introduction to RAID
- Interconnect standards: SCSI, High-Performance Parallel Interface (HIPPI), Gigabyte System Network (GSN), Fibre Channel, and message-passing networks
- Network-attached storage devices
- Holographic storage
- Unix file system
- Fault-tolerance and journaling
- Distributed file systems (DFS)
- Sun's Network File System (NFS)
- Shared-memory computer design and architecture
- Distributed-memory computers
- Parallel file systems: Intel Parallel File System (PFS), Parallel I/O File System (PIOFS), and General Parallel File System (GPFS)
- High-Performance File System (HPFS)
- Cluster file systems: XFS, Parallel Virtual File System (PVFS), and Berkeley River
- Review of studies on I/O access patterns for scientific software
- Collective I/O and performance
- Two-phase I/O
- Server-based collective I/O
- Disk-directed I/O (DDIO)
- Hints and automatic adaptation
- Research projects for improved parallel I/O
- Portable Parallel File Systems (PPFS)
- Panda
- Programming with low-level I/O APIs: MPI-IO, HPF I/O, and SIO Low-Level API (LLAPI)
- Scientific libraries for processing numerical data (NetCDF version 3.4 and HDF version 5)
- Sample code for reading and writing numerical data
- Fiber bundles for representing number data
- Out-of-core computations and algorithms
- Virtual memory
- Compiler-directed file layout
- Checkpointing techniques and libraries
- Data management and analysis
- Scientific databases (relational and object-oriented)
- Knowledge discovery and data mining for research applications
