At the same time, the proliferation of big data resulted into design of data-intensive processing paradigms like Apache big data stack. Traditionally, high performance computing (HPC) systems have been used for scientific applications involving majority of compute-intensive tasks. The dawn of exascale computing and its convergence with big data analytics has greatly spurred research interests. ![]() Together, these contributions illustrate the Pilot paradigm, its generality, and how it helps to address some challenges in distributed scientific computing. The three main contributions of this paper are: (i) an analysis of the motivations and evolution of Pilot-Job systems (ii) an outline of the Pilot abstraction, its distinguishing logical components and functionalities, its terminology, and its architecture pattern and (iii) the description of core and auxiliary properties of Pilot-Jobs systems and the analysis of seven exemplar Pilot-Job implementations. This paper offers a comprehensive analysis of Pilot-Job systems critically assessing their motivations, evolution, properties, and implementation. Ultimately, this is hindering the realization of the full impact of Pilot-Jobs by limiting their robustness, portability, and maintainability. Pilot-Job implementations have proliferated with no shared best practices or open interfaces and little interoperability. Notwithstanding the growing impact on scientific research, there is no agreement upon a definition of Pilot-Job system and no clear understanding of the underlying abstraction and paradigm. ![]() With the increasing importance of task-level parallelism in high-performance computing, Pilot-Job systems are also witnessing an adoption beyond traditional domains. They are used to consume more than 700 million CPU hours a year by the Open Science Grid communities, and by processing up to 1 million jobs a day for the ATLAS experiment on the Worldwide LHC Computing Grid. Pilot-Job systems play an important role in supporting distributed scientific computing.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |