Load balancing of a comprehensive air quality model

Comprehensive mesoscale tropospheric chemistry-transport models (CTMs) must combine numerous physical, chemical, and other processes to simulate realistically transformations and transport of manmade and natural trace species: Starting with anthropogenic emission, meteorological transport and diffusion processes disperse the material, while radiation induced photolysis and gas phase chemistry act on the chemical composition. Aqueous phase and heterogeneous chemistry processes must also be considered in clouds and on aerosols, respectively. Chemistry-transport models are generally claimed to be well suited for massively parallel processing since the arithmetic-to-communication ratio is usually high. However this observation proofs insufficient to account for an efficient parallel performance with increasing complexity of the model. The local state of the atmosphere ensues very different branches of the modules' code and greater differences in the computational work load and consequently runtime of individual processors occur to a much larger extend during a time step than reported for meteorological models. Variable emissions, changes in actinic fluxes, and all processes associated with cloud modeling are highly variable in time and space and are identified to induce large load imbalances which severely affect the parallel efficiency. Based on a horizontal grid partitioning approach a method is proposed where the integration domain of the individual processors is locally adjusted to accomodate for load imbalances. This ensures a minimal communication volume and data exchange only with next neighbours. The interior boundary adjustments of the processors are combined with routine boundary exchange which is required each time step anyway. Two dynamic load balancing schemes were implemented and compared against a conventional equal area partition and a static load balancing scheme on an Intel Paragon with 136 compute nodes. A midsummer episode of highly elevated ozone concentrations was taken as test case. The dynamic load balancing approaches were found to perform significantly better and reduce idle times of the processors considerably. The efficiency was raised from 36% to 69% for a 128 processor configuration.