An Autonomic Workflow Performance Manager for Weather Forecast and Research Modeling Workflows

Abstract

Parameter selection is a critical task in scientific workflows in order to maintain the accuracy of the simulation in an environment where physical conditions change dynamically such as in the case of weather research and forecast simulations. Currently, Numerical Weather Prediction (NWP) is the premier method for weather prediction, which is used by the National Oceanic and Atmospheric Administration (NOAA). It takes the current observations from observed sites as the input for numeric computer models and then produces the final prediction. Considering the large number of simulation parameters, the size of the configuration search space becomes prohibitive for rapidly evaluating and identifying the parameter configuration that leads to most accurate prediction. In this thesis, we develop an Autonomic Workflow Performance Manager (AWPM) for Hurricane Integrated Modeling System (HIMS). AWPM is implemented on top of the Apache Storm and ZooKeeper to handle multiple real-time data streams for weather forecast. AWPM can automatically manage model initialization and execution workflow and achieve better performance and efficiency. In our experiments, AWPM achieves better performance and efficiency for the model initialization and execution processes, by utilizing automatic computing, distributed computing and component-based development. We reduced the timescale of the configuration search workflow by a factor of 10 by using 20 threads with the full search method, and a factor of 20 by with the roofline method when compared to serial workflow execution as it is typically performed by domain scientists.