This title appears in the Scientific Report :
2018
Please use the identifier:
http://hdl.handle.net/2128/17391 in citations.
Scalable I/O for parallel access to task-local files with SIONlib
Scalable I/O for parallel access to task-local files with SIONlib
Large-scale HPC applications can produce a high load on the file system. This often occurs during access to checkpoint and restart files, which have to be frequently stored to allow for an application restart after program termination or system failure. On large-sca...
Saved in:
Personal Name(s): | Lührs, Sebastian (Corresponding author) |
---|---|
Frings, Wolfgang | |
Contributing Institute: |
Jülich Supercomputing Center; JSC |
Imprint: |
2018
|
Conference: | Les séminaires de la modélisation, MdlS, Saclay (France), 2018-01-09 - 2018-01-09 |
Document Type: |
Talk (non-conference) |
Research Program: |
Computational Science and Mathematical Methods |
Link: |
OpenAccess OpenAccess |
Publikationsportal JuSER |
Large-scale HPC applications can produce a high load on the file system. This often occurs during access to checkpoint and restart files, which have to be frequently stored to allow for an application restart after program termination or system failure. On large-scale HPC systems with distributed memory, each application task will often perform such I/O individually by creating task-local file objects. This I/O can stress the metadata management components of the I/O subsystem significantly. SIONlib is a library for writing and reading binary data to/from several thousands of processors into one or a small number of physical files. The SIONlib file layout and API allow the application to take advantage of the scaling behaviour and asynchronous access of a logical task-local pattern while keeping the number of files independent of and significantly smaller than the number of processes.This talk will give an overview of the design choices for the SIONlib library as well as a summary of possible use cases. |