论文标题
具有ADIOS2的WRF模型中的高性能平行I/O和原位分析
High Performance Parallel I/O and In-Situ Analysis in the WRF Model with ADIOS2
论文作者
论文摘要
随着大规模HPC群集的计算能力接近Exascale，计算能力和存储系统之间的差距越来越大。特别是，流行的高性能计算应用程序（HPC）应用程序，天气研究和预测模型（WRF）目前正在用于高分辨率预测和研究，这些预测和研究产生了非常大的数据集，尤其是在研究瞬态天气现象时。但是，已经发现WRF中当前可用的I/O选项是大规模的瓶颈。 在这项工作中，我们演示了集成下一代并行I/O框架-ADIOS2作为WRF中的新I/O后端选项的影响。首先，我们详细介绍了集成过程中遇到的实施注意事项，挫折和解决方案。接下来，我们检查I/O写入时间的结果，并将它们与当前可用的WRF I/O选项的结果进行比较。与经典的MPI-I/O解决方案相比，使用ADIOS2时，由此产生的I/O时间在数量级的速度上显示。此外，展示了节点 - 局部突发缓冲区写入功能以及ADIOS2的无线无损压缩功能，从而进一步提高了性能。最后，使用WRF预测管道证明了对天气预测的新型ADIOS2原位分析功能的使用，显示了无缝的端到端处理管道，该管道与WRF模型的执行同时发生，从而导致总时间的急剧改进。
As the computing power of large-scale HPC clusters approaches the Exascale, the gap between compute capabilities and storage systems is ever widening. In particular, the popular High Performance Computing (HPC) application, the Weather Research and Forecasting Model (WRF) is being currently being utilized for high resolution forecasting and research which generate very large datasets, especially when investigating transient weather phenomena. However, the I/O options currently available in WRF have been found to be a bottleneck at scale. In this work, we demonstrate the impact of integrating a next-generation parallel I/O framework - ADIOS2, as a new I/O backend option in WRF. First, we detail the implementation considerations, setbacks, and solutions that were encountered during the integration. Next we examine the results of I/O write times and compare them with results of currently available WRF I/O options. The resulting I/O times show over an order of magnitude speedup when using ADIOS2 compared to classic MPI-I/O based solutions. Additionally, the node-local burst buffer write capabilities as well as in-line lossless compression capabilities of ADIOS2 are showcased, further boosting performance. Finally, usage of the novel ADIOS2 in-situ analysis capabilities for weather forecasting is demonstrated using a WRF forecasting pipeline, showing a seamless end-to-end processing pipeline that occurs concurrently with the execution of the WRF model, leading to a dramatic improvement in total time to solution.
