Optimization of parallel processing of Square Kilometre Array low frequency imaging pipeline

Subjects: Computer Science >> Other Disciplines of Computer Science submitted time 2022-06-28

Wei, Jianwen Zhang, Chenfei Lao, Baoqiang Lin, James An, Tao

Abstract：

Data processing of the Square Kilometre Array (SKA) is carried out in pipeline mode, and the execution efficiency of pipeline is an important factor to be considered in SKA data processing. Continuum imaging is one of the main observation modes of SKA and is a prerequisite for many other scientific works. In this paper, we take the imaging pipeline of SKA low-frequency precursor Murchison Widefield Array (MWA), as an example and optimize the parallel processing pipeline on the China SKA Regional Centre prototype (CSRC-P). Previous optimization schemes have focused on a few performance hotspots and lacked systematic optimization of the overall pipeline, resulting in a relatively poor overall speedup ratio. In this paper, we propose a global optimization scheme that combines C++ multi-threading, Python multi-processing, and Shell multi-tasking parallelism for pipelines using multiple programming languages and image datasets that can be processed independently, and verify the accuracy of the optimization results. Experiments show taht the optimized pipeline achieves an overall speedup of 2.7 and 2.4 times on the x86 and ARM nodes of CSRC-P, respectively, and the ARM compute nodes shows good adaptability to SKA applications. The optimization strategies and methods in this paper are also applicable to other SKA applications and will be useful for the scientific operation and future operation of the SKA precursor telescope.

Peer Review Status:Awaiting Review

 Hits 5208  Downloads 444  Comment