This title appears in the Scientific Report : 2019 
Maestro Project Introduction
Pleiter, Dirk (Corresponding author)
Jülich Supercomputing Center; JSC
2019
EuroHPC Summit Week 2019, Poznań (Poland), 2019-05-13 - 2019-05-18
Conference Presentation
Middleware for memory and data-awareness in workflows
Supercomputer Facility
OpenAccess
OpenAccess
Please use the identifier: http://hdl.handle.net/2128/23940 in citations.


In this talk, we provide an introduction to the Maestro project.This project was started in September 2018 involving the following partners:Appentra, CEA, Cray, CSCS, ECMWF, Forschungszentrum Juelich, Seagate.The goal is to develop and demonstrate a new middleware that aims at addressingtwo key shortcomings observed for today's supercomputing solutions.First, available solutions typically lack data awareness as they are optimisedfor filling processing pipelines, not for managing data.Furthermore, available solutions lack memory awareness, i.e. available middlewaredo not have information about available memory (or storage) hardware and thusare not able to make data transport decisions.Maestro applies a co-design approach for designing a solution addressing these challenges.The co-design applications, in particular, include data-intensive applications likethe weather prediction workflow from ECMWF.In this case, simulations of the IFS model produce huge amounts of data that are consumedby other applications, which generate products for ECMWF's products member states and customers.This matches well Maestro's concept of providing a pool of data objects, wheredata producers give objects to the pool and data consumers take objects from the pool.Using suitable metadata, the Maestro middleware can manage storing and transport of data objectstaking the hardware architecture into account, on which the applications are being executed.We will conclude this talk with an outlook on the work of the Maestro project.