Forecast verification · Bureau of Edi Gardens

Each forecast is kept and, once the hour has passed, compared with what the Bureau of Meteorology observed at Melbourne (Olympic Park). The chart shows the forecast made one, three or five days ahead against the observations; the residual below it is $\varepsilon(t) = y(t) - \hat{y}(t)$ , observed minus forecast, so a positive residual means it was warmer, windier or higher than forecast.

Quantity

Forecast made

Temperature (°C)

All quantities, by days ahead

Period

The ensemble

Each morning the 51 runs of the ECMWF ensemble for the next five days are kept for the hours 10 am to 6 pm and scored once the observations arrive. The continuous ranked probability score (CRPS) measures the whole spread of runs against what happened, in the quantity's own units: lower is better, and for a single forecast it equals the absolute error, so the ensemble's CRPS can be read directly against the single run's mean absolute error on the same hours.

Definitions

For $n$ hours with forecast $\hat{y}_i$ and observation $y_i$ , and residual $\varepsilon_i = y_i - \hat{y}_i$ :

\mathrm{bias} = \frac{1}{n}\sum_{i=1}^{n} \varepsilon_i, \qquad \mathrm{MAE} = \frac{1}{n}\sum_{i=1}^{n} \lvert \varepsilon_i \rvert, \qquad \mathrm{RMSE} = \sqrt{\frac{1}{n}\sum_{i=1}^{n} \varepsilon_i^{2}}

The CRPS of a forecast distribution $F$ for an observation $y$ (Matheson & Winkler 1976; Hersbach 2000), with $H$ the step function:

\mathrm{CRPS}(F, y) = \int_{-\infty}^{\infty} \bigl[F(x) - H(x - y)\bigr]^{2}\, dx

For an ensemble of $M$ runs $x_1, \dots, x_M$ the fair estimator (Ferro 2014) is used, the score the ensemble would earn if it had infinitely many runs drawn the same way:

\mathrm{CRPS}_{\text{fair}} = \frac{1}{M}\sum_{i=1}^{M} \lvert x_i - y \rvert \;-\; \frac{1}{2M(M-1)}\sum_{i=1}^{M}\sum_{j=1}^{M} \lvert x_i - x_j \rvert

Rank histogram (Hamill 2001): the observation's rank among the 51 runs. A flat histogram means the runs spread as widely as the outcomes do; a U shape means the outcome often falls outside the runs (too little spread); a dome means too much spread.

Forecast made N days ahead: the forecast for that hour from the model run of N days earlier (Open-Meteo Previous Runs), so lead times of N to N + 1 days. Forecast curve: hourly values joined by a monotone cubic (Fritsch & Carlson 1980), which never overshoots between hours; the dots are the hourly values that are scored. Only observations on the hour are scored; the half-hourly ones are drawn.

What to bear in mind

Different places. The station is at Olympic Park, 4 km south of the park and beside the Yarra; the forecast is for the model's grid square nearest the park, about 9 km across. Part of any steady bias is the difference between the two places, not forecast error.
Gusts are defined differently. The forecast gust is the hour's strongest; the observed gust is the strongest in the 10 minutes before each report. Observed gusts are therefore expected to read lower on average.
Feels-like. Both are the Bureau's apparent temperature for shade: observed at the station, and computed from the forecast's temperature, humidity and wind exactly as the Park Day Index computes it.
The record is short. Observations are kept from 16 October 2026; scores over a few days move a great deal from day to day. The sample size is shown with every score.

The chart as a table

Sources

Observations: Australian Government Bureau of Meteorology, Melbourne (Olympic Park), station 086338. Forecasts: ECMWF IFS and ECMWF ensemble via Open-Meteo (CC BY 4.0). Methods: section 17 of the methods. References: Ferro CAT (2014) Fair scores for ensemble forecasts, QJRMS 140: 1917–1923, doi:10.1002/qj.2270; Fritsch FN, Carlson RE (1980), SIAM J. Numer. Anal. 17: 238–246, doi:10.1137/0717021; Hamill TM (2001), Mon. Wea. Rev. 129: 550–560, doi:10.1175/1520-0493(2001)129<0550:IORHFV>2.0.CO;2; Hersbach H (2000), Wea. Forecasting 15: 559–570, doi:10.1175/1520-0434(2000)015<0559:DOTCRP>2.0.CO;2; Matheson JE, Winkler RL (1976), Management Science 22: 1087–1096, doi:10.1287/mnsc.22.10.1087.

Product IDB3D002. Back to the forecast · Methods.