Spatial Error Characteristics of Rainstorm Forecasts by Large-Scale Numerical Models over the Northeastern Side of the Tibetan Plateau

ZHANG Junxia, KONG Xiangwei, LIU Xinwei, WANG Yong
(Lanzhou Central Meteorological Observatory, Lanzhou, Gansu, China)

Abstract

Using precipitation forecasts from the ECMWF (European Center for Medium-Range Weather Forecasts) and GRAPES (Global/Regional Assimilation and Prediction System, the China Meteorological Administration's global numerical weather prediction operational system) large-scale numerical models, along with high-density precipitation observation data, this study identifies and isolates rainstorm targets over the northeastern side of the Tibetan Plateau based on the Contiguous Rain Area (CRA) spatial verification technique. A quantitative analysis is conducted on the spatial error characteristics—including location, intensity, and pattern errors—of rainstorm forecasts from both models, summarizing the applicability of large-scale numerical models for rainstorm forecasting in this region. The results indicate that: (1) Pattern errors account for the largest proportion of total errors in both models. ECMWF exhibits the smallest intensity error proportion, followed by location errors, while GRAPES-GFS shows the smallest location error proportion, followed by intensity errors. (2) The forecasted rainstorm areas from both models are shifted westward and northward compared to observations. The ECMWF model forecasts the rainstorm center shifted west-southward, whereas GRAPES-GFS forecasts it only westward. (3) Both models significantly underestimate the rainstorm area, which can easily lead to missed rainstorm forecasts. GRAPES-GFS underestimates both maximum precipitation and average rainfall intensity by more than 40%, while ECMWF underestimates average rainfall intensity by 11.49% but overestimates maximum precipitation by 1.47%. (4) Both models demonstrate better forecasting performance for rainstorms in the southeastern Longnan region of Gansu and southwestern Shaanxi, but poorer performance in northern Shaanxi, Ningxia, and other northern regions.

Keywords: Contiguous Rain Area; spatial error; rainstorm; northeastern side of Tibetan Plateau

1. Introduction

Precipitation forecasting is one of the most important operational tasks in modern weather forecasting, with numerical weather prediction providing the most valuable reference. However, due to inherent limitations in model design and external factors such as topography, precipitation forecast products from numerical models contain certain errors in their spatiotemporal distribution. Therefore, verifying and evaluating model performance provides essential background error information that helps forecasters correct model predictions. Conventional point-to-point verification methods (such as TS scores) are sensitive to precipitation location and timing, suffer from "double penalty" issues, and can mask valuable information for forecasters. While neighborhood spatial verification methods can provide quantitative assessment of precipitation intensity, coverage, and rainband location that forecasters care about, they may produce different evaluation results due to factors like smoothing radius and filtering thresholds, potentially leading to misleading assessments.

The Contiguous Rain Area (CRA) method, a spatial verification technique based on object identification, decomposes total error into intensity, location, and pattern components, providing clear meteorological verification significance. This method first defines contiguous rain areas using a precipitation threshold within a certain region, then calculates statistical quantities such as precipitation centroid, area, and average intensity for both forecast and observation fields. The total forecast error is defined as the mean of squared differences between original forecast and observation, while the shifted error is the mean of squared differences after shifting the forecast. To analyze error sources, the forecast rain area is shifted to minimize root-mean-square error with observations, yielding a shifted forecast rain area. Consequently, total model error can be decomposed into intensity error (squared difference between shifted model mean intensity and observed mean intensity), location error (total error minus shifted error), and pattern error (shifted error minus intensity error). The CRA method has been widely applied to model precipitation forecast and radar nowcasting verification both domestically and internationally.

The northeastern side of the Tibetan Plateau lies at the northern edge of the East Asian summer monsoon transition zone. Influenced jointly by the East Asian summer monsoon system, westerly weather systems, and plateau weather systems, this region is sensitive to climate change with extremely unbalanced annual precipitation distribution. Variations in rainy season precipitation are mainly caused by changes in heavy precipitation, with rainstorms occurring from May to September accounting for a significant proportion of annual totals. Rainstorm occurrence is closely related to the region's complex topography and local microclimates. In recent years, despite continuous improvement in large-scale numerical model performance, model precipitation forecast capability still diminishes with increasing precipitation magnitude, with heavy precipitation forecasts showing significant topographic influences and large errors in both precipitation intensity and location.

This paper employs the CRA spatial verification technique to evaluate spatial biases in ECMWF and GRAPES-GFS rainstorm forecasts over the northeastern Tibetan Plateau, focusing on analyzing location errors, intensity errors, pattern errors, and model forecast tendencies for rainstorms. The aim is to provide forecasters with detailed model evaluation results for targeted forecast correction and to offer valuable bias information for model development.

1.1 Data Sources and Processing

Based on current operational applications and data availability, this study utilizes 24-hour accumulated precipitation forecasts from ECMWF and GRAPES-GFS with 0.125°×0.125° and 0.25°×0.25° resolution, respectively, for the period May-September 2019-2020 over the northeastern Tibetan Plateau (32°-40°N, 100°-111°E). Observational precipitation data from automatic weather stations (including benchmark stations, basic stations, and regional automatic stations) provided by Lanzhou Central Meteorological Observatory are used for the same period. Both model forecast data and station observations are objectively interpolated to 0.125°×0.125° grid data using variational techniques, which have been shown to produce satisfactory interpolation results meeting accuracy and smoothness requirements for objective analysis.

1.2 Research Methods

The CRA method is an object-based quantitative precipitation verification approach that examines contiguous rain areas defined by specific isohyets rather than entire precipitation fields, thereby helping forecasters better understand model error sources. The method first applies a precipitation threshold to define contiguous rain areas within a region, then calculates statistical measures including precipitation centroid, area, and average intensity for both forecast and observation fields. Total model error is defined as the mean of squared differences between original forecast and observation, while shifted error represents the mean of squared differences after shifting the forecast. To analyze error sources, the forecast rain area is shifted to minimize root-mean-square error with observations, producing a shifted forecast rain area. Total error can thus be decomposed into intensity error, pattern error, and displacement error, with specific calculation methods detailed in reference [7].

1.3 Rainstorm Processes and Effective CRA

Regional rainstorm processes over the northeastern Tibetan Plateau are relatively infrequent. A regional rainstorm event is defined when the number of automatic stations with 24-hour accumulated precipitation reaching rainstorm level (≥50 mm) exceeds a certain threshold. Based on this criterion, 18 rainstorm processes were identified from May to September 2019-2020 (Table 1). When identifying and separating contiguous rain areas for these processes, the threshold is set at ≥50 mm with a coverage requirement exceeding 0.5°×0.5° to define an effective CRA. This threshold ensures contiguous rain areas are neither too large nor too small for meaningful analysis. Among the 18 processes, ECMWF identified effective CRAs in 11 cases, while GRAPES-GFS identified 8 cases. Some processes lacked effective CRAs due to three main reasons: (1) the model failed to forecast rainstorms, (2) forecasted rainstorm area was too small and scattered, or (3) location bias was too large.

2. Results

2.1 Characteristics of Rainstorms over the Northeastern Tibetan Plateau

The 18 regional rainstorm processes occurred primarily in early and mid-August, with the highest frequency in mid-August (Figure 2). Rainstorm area varied dramatically from 1,709 km² to 61,783 km²—a nearly 36-fold difference. All observed rainstorm areas exceeding 11,000 km² were successfully identified as effective CRAs. When observed rainstorm area was smaller than 6,000 km², both models often failed to identify effective CRAs due to missed forecasts or large location biases.

2.2 Overall Errors in Rainstorm Forecasts

The distribution of location, intensity, and pattern errors for both models is shown in Figure 3 and Table 2. Pattern errors dominate in both models, accounting for 52.36% of total error in ECMWF and 52.58% in GRAPES-GFS. ECMWF shows the smallest intensity error proportion (20.73%), followed by location errors (26.93%). Conversely, GRAPES-GFS exhibits the smallest location error proportion (16.19%), followed by intensity errors (31.23%). Both location and intensity error proportions range between 10%-50% across cases. Pattern errors exceed 50% in most cases, suggesting that complex topography significantly influences forecast accuracy.

2.3 Spatial Errors in Rainstorm Forecasts

Both models forecast rainstorm centroids significantly west of observations (Figure 5). ECMWF forecasts are shifted west-southwest on average (0.36° west, 0.11° south), while GRAPES-GFS forecasts are shifted westward with minimal meridional bias (0.22° west). Displacement error analysis reveals that most ECMWF cases are shifted west-northwest (average 0.34° west, 0.08° north), while GRAPES-GFS cases are also shifted west-northwest but with smaller westward displacement (less than half of ECMWF) and similar northward displacement.

Two main factors cause displacement errors: (1) model forecasts of upper-level systems are slower than observed, and (2) observed rainstorms typically occur in southerly flow to the right of shear lines, while model forecasts concentrate near shear lines or low vortices where dynamic forcing is strongest. For example, in the August 3, 2019 case, the observed rainstorm was located in southerly flow to the right of a 700 hPa shear line, while the model forecast positioned it near the shear line itself, resulting in a west-northwest displacement error.

2.4 Rainstorm Area Errors

Both models consistently underestimate rainstorm area compared to observations (Figure 7). ECMWF forecasts are 68.33% smaller on average, while GRAPES-GFS forecasts are 16.20% smaller. The dispersion in GRAPES-GFS area forecasts is much smaller than in ECMWF, indicating GRAPES-GFS is more prone to missing rainstorm events. When observed rainstorm grid points are fewer than 70 (area ≈ 10,937.5 km²), both models tend to underestimate area, often failing to identify effective CRAs. When grid points exceed 140 (area > 21,875 km²), ECMWF forecasts show comparable probabilities of overestimation and underestimation, demonstrating better performance for larger rainstorm areas.

2.5 Intensity Errors in Rainstorm Forecasts

GRAPES-GFS average rainfall intensity is weaker than observed in all but a few cases, with an average underestimation of 43.40% (Figure 8). ECMWF average intensity is also weaker than observed, but by a smaller margin (11.49%), showing a more consistent and stable underestimation. When observed intensity exceeds 60 mm, the underestimation becomes more pronounced in both models. For maximum precipitation, GRAPES-GFS shows consistent underestimation averaging 49.33%, while ECMWF exhibits large dispersion but only slight average underestimation (1.47%), with some cases showing significant overestimation that compensates for underestimation in others.

Total precipitation (average intensity multiplied by grid points) is underestimated in 79.72% of GRAPES-GFS cases, indicating the model's internal atmospheric water vapor cycle is systematically too dry. ECMWF total precipitation shows comparable probabilities of overestimation and underestimation, suggesting its overall water vapor cycle is more realistic.

2.6 Regional Forecast Performance

Analysis of forecast performance across different regions shows that effective CRAs identified by both models are concentrated in eastern Gansu (Longnan) and most of Shaanxi (Figure 9). The highest frequencies occur in southeastern Gansu and southwestern Shaanxi, with moderate performance in central Shaanxi, Dingxi, eastern Tianshui, and Pingliang-Qingyang regions. Both models perform poorly in northern Shaanxi, Ningxia, southern Baiyin, and northern Dingxi. Overall, ECMWF and GRAPES-GFS demonstrate better forecasting capability for rainstorms in southeastern Gansu and southwestern Shaanxi but poorer performance in more northern regions.

2.7 Model Forecast Tendencies

To analyze model forecast tendencies, we define forecasts with area error ≤50% as "accurate," >50% as "overestimated," and <-50% as "underestimated." Similar classifications apply to average intensity and maximum precipitation. Both models tend to underestimate rainstorm area, average intensity, and maximum precipitation (Table 3). GRAPES-GFS shows particularly strong underestimation tendencies, with 75% of cases underestimating area and 87.5% underestimating average intensity, making it more prone to missed forecasts. ECMWF shows more balanced tendencies, with comparable frequencies of accurate, overestimated, and underestimated forecasts for average intensity and maximum precipitation.

3. Conclusions

Using 24-hour precipitation forecasts from ECMWF and GRAPES-GFS models and high-density precipitation observations from May to September 2019-2020, this study applies the CRA spatial verification technique to analyze spatial errors (displacement, intensity, and pattern errors) and forecast tendencies for rainstorms over the northeastern Tibetan Plateau. The main conclusions are:

(1) Pattern errors dominate total errors in both models, accounting for 52.36% in ECMWF and 52.58% in GRAPES-GFS. ECMWF shows the smallest intensity error proportion (20.73%), followed by location errors (26.93%), while GRAPES-GFS shows the smallest location error proportion (16.19%), followed by intensity errors (31.23%). Displacement errors are related to factors such as the speed of large-scale model forecast systems, while pattern errors may be associated with the region's complex topography.

(2) Both models forecast rainstorm areas shifted west-northwest relative to observations. ECMWF forecasts are shifted west-southwest on average (0.36° west, 0.11° south), while GRAPES-GFS forecasts are shifted westward (0.22°) with minimal meridional bias. Both models underestimate rainstorm area, but GRAPES-GFS shows greater underestimation. For rainstorm areas with fewer than 70 grid points, both models tend to underestimate, while for areas exceeding 140 grid points, ECMWF shows comparable overestimation and underestimation probabilities.

(3) ECMWF average rainfall intensity is underestimated by 11.49%, with more pronounced underestimation when observed intensity exceeds 60 mm. GRAPES-GFS average intensity is underestimated by 43.40%, showing consistent and stable underestimation. GRAPES-GFS maximum precipitation is underestimated by 49.33%, while ECMWF maximum precipitation is slightly underestimated by 1.47% due to large dispersion.

(4) Both models perform better for rainstorms in southeastern Gansu and southwestern Shaanxi, but poorer in northern Shaanxi and Ningxia. Forecast tendency analysis indicates both models significantly underestimate rainstorm area, easily leading to missed forecasts, though ECMWF demonstrates stronger forecasting capability than GRAPES-GFS.

References

[1] Cui Fene, Wang Yong, Li Huijun. Performance verification of coastal torrential rainfall forecast with several numerical products[J]. Meteorological Science and Technology, 2013, 41(4): 696-702.

[2] Davis C A, Brown B G, Bullock R, et al. The method for object based diagnostic evaluation (MODE) applied to numerical forecasts from the 2005 NSSL/SPC Spring Program[J]. Weather & Forecasting, 2009, 24(5): 1252-1267.

[3] Dai Jianhua, Mao Mao, Shao Lingling, et al. Applications of a new verification method for severe convection forecasting and nowcasting in Shanghai[J]. Advances in Meteorological Science and Technology, 2013, 3(3): 42-47.

[4] Liu Couhua, Niu Ruoyun. Object based precipitation verification method and its application[J]. Meteorological Applications, 2013, 39(6): 681-690.

[7] Davis C, Brown B, Bullock R. Object based verification of precipitation forecasts. Part I: Methodology and application to mesoscale rain areas[J]. Monthly Weather Review, 2006, 134(7): 1772-1784.

[8] Davis C, Brown B, Bullock R. Object based verification of precipitation forecasts. Part II: Application to convective rain systems[J]. Monthly Weather Review, 2006, 134(7): 1785-1795.

[9] Marzban C, Sandgathe S. Cluster analysis for verification of precipitation fields[J]. Weather & Forecasting, 2006, 21(5): 824-838.

[10] Gilleland E, Lee T C, Halley G J, et al. Computationally efficient spatial forecast verification using Baddeley's delta image metric[J]. Monthly Weather Review, 2008, 136(5): 1747-1757.

[11] Wernli H, Paulat M, Hagen M, et al. SAL: A novel quality measure for the verification of quantitative precipitation forecasts[J]. Monthly Weather Review, 2008, 136(11): 4470-4487.

[12] Wernli H, Hofmann C, Zimmer M. Spatial forecast verification methods intercomparison project: Application of the SAL technique[J]. Weather & Forecasting, 2009, 24(6): 1472-1484.

[13] Zhao Bin, Zhang Bo. Application of neighborhood spatial verification method on precipitation evaluation[J]. Torrential Rain and Disasters, 2017, 36(6): 497-504.

[14] Wang Wanjun, Yin Haitao, Zhao Jinghong, et al. Verification of numerical forecast products for Tianjin precipitation forecast in recent three years[J]. Meteorological Science and Technology, 2018, 46(4): 718-723.

[15] Sharma K, Ashrit R, Ebert E, et al. Assessment of Met Office Unified Model (UM) quantitative precipitation forecasts during the Indian summer monsoon: Contiguous Rain Area (CRA) approach[J]. Journal of Earth System Science, 2019, 128(1): 1-17.

[16] Ebert E E, Mcbride J L. Verification of precipitation in weather systems: Determination of systematic errors[J]. Journal of Hydrology, 2000, 239(1): 179-202.

[17] Das A K, Kundu P K, Roy B, et al. Performance evaluation of WRF model with different cumulus parameterizations in forecasting monsoon depressions[J]. Mausam, 2019, 70(3): 501-522.

[18] Fu Jiaolan, Dai Kan. The ECMWF model precipitation systematic error in the east of Southwest China based on the contiguous rain area method for spatial forecast verification[J]. Meteorological Applications, 2016, 42(12): 1456-1464.

[19] Wang Xinmin, Li Han. Spatial verification evaluation of Typhoon rainstorm by multiple numerical models[J]. Meteorological Applications, 2020, 46(6): 753-764.

[20] Yu Z, Chen Y J, Ebert B, et al. Benchmark rainfall verification of landfall tropical cyclone forecasts by operational ACCESS TC over China[J]. Meteorological Applications, 2020, 27: e1842.

[21] Zhuang Y, Tang X, Wang Y. Impact of track forecast error on tropical cyclone quantitative precipitation forecasts over the coastal region of China[J]. Journal of Hydrology, 2020, 589: 125347.

[22] Li Dongliang, Xie Jinnan, Wang Wen. A study of summer precipitation features and anomaly in Northwest China[J]. Chinese Journal of Atmospheric Sciences, 1997, 21(3): 331-340.

[23] Yang Zhaoming, Zhang Tiaofeng. Analysis of precipitation change and its contribution in the rainy season in the northeast Qinghai Tibet Plateau from 1961-2017[J]. Arid Zone Research, 2020, 38(1): 22-28.

[24] Zhao Qingyun, Song Songtao, Yang Guiming, et al. Spatial and temporal variations of torrential rain over Northwest China and general circulation anomalies in summer[J]. Journal of Lanzhou University (Natural Sciences), 2014, 50(4): 517-522.

[25] Li Dongliang, Shao Pengcheng, Wang Hui, et al. Advances in research of the north boundary belt of East Asia Subtropical Summer Monsoon in China[J]. Plateau Meteorology, 2013, 32(1): 305-314.

[26] Chen Jie, Huang Wei, Jin liya, et al. A climatological northern boundary index for East Asian Summer Monsoon and its interannual variability[J]. Scientia Sinica (Terrae), 2018, 48(1): 93-101.

[27] Huang Yuxia, Wang Baojian, Huang Wubin, et al. A review on rainstorm research in Northwest China[J]. Torrential Rain and Disasters, 2019, 38(5): 515-525.

[28] Chen Yuying, Chen Nan, Ren Xiaofang, et al. Analysis on forecast deviation and predictability of a rare severe rainstorm along the eastern Helan Mountain[J]. Meteorological Monthly, 2018, 44(1): 159-169.

[29] Yang Kan, Ji Xiaoling, Mao Lu, et al. Numerical simulation and comparative analysis of topographic effects on two extraordinary severe flood rainstorms in Helan Mountain[J]. Journal of Arid Meteorology, 2020, 38(4): 581-590.

[30] Liu Yuzhi, Wu Chuqiao, Jia Rui, et al. An overview of the influence of atmospheric circulation on the climate in arid and arid region of Central and East Asia[J]. Scientia Sinica(Terrae), 2018, 48(9): 1141-1152.

[31] Yin Tianyuan, Yin Shuyan, Li Fumin. Relationship between the summer extreme precipitation in the south and north of the Qinling Mountains and Western Pacific Subtropical High[J]. Arid Zone Research, 2019, 36(6): 1379-1390.

[32] Liu Couhua, Cao Yong, Fu Jiaolan. An objective analysis algorithm based on the variation method[J]. Acta Meteorologica Sinica, 2013, 71(6): 1172-1182.

[33] Zhao Haiying, Yanqing, Qiu Guiqiang, et al. Numerical simulation study of topography effects on a severe rainstorm in Shanxi Province[J]. Meteorological and Environmental Sciences, 2017, 40(2): 84-91.