This title appears in the Scientific Report :
2019
Please use the identifier:
http://hdl.handle.net/2128/23592 in citations.
Parallel I/O and Portable Data Formats
Parallel I/O and Portable Data Formats
This course will be about parallel I/O with a special focus on portable data formats. It will introduce the use of the HDF5 and NetCDF (NetCDF4 and PnetCDF) library interfaces, and hands-on exercises (in C/C++ or Fortran) will allow to immediately test and understand their usage. Performance hints,...
Saved in:
Personal Name(s): | Lührs, Sebastian (Corresponding author) |
---|---|
Contributing Institute: |
Jülich Supercomputing Center; JSC |
Imprint: |
2019
|
Conference: | VSC Training Course, Vienna (Austria), 2019-12-06 - 2019-12-06 |
Document Type: |
Lecture |
Research Program: |
Computational Science and Mathematical Methods |
Link: |
OpenAccess OpenAccess |
Publikationsportal JuSER |
This course will be about parallel I/O with a special focus on portable data formats. It will introduce the use of the HDF5 and NetCDF (NetCDF4 and PnetCDF) library interfaces, and hands-on exercises (in C/C++ or Fortran) will allow to immediately test and understand their usage. Performance hints, optimization potential, and best practices for I/O will be discussed in detail throughout the whole course.Numerical simulations conducted on current HPC systems face an ever growing need for scalability pushing the limitations on size and properties that can be accurately simulated. Therefore, ever larger data sets have to be processed, be it reading input data or writing results. Serial approaches on handling I/O in a parallel application will dominate the performance on massively parallel systems, leaving a lot of computing resources idle during those serial I/O phases.In addition to the need for parallel I/O, input and output data is often processed on different and maybe even heterogeneous platforms. Conversion processes can impose a high level of maintenance when different data representations are needed. Portable, self-describing data formats such as HDF5 and netCDF can help to solve these problems. |