[1]StephanRasp [2]Peter D.Dueben [3]SebastianScher [4]Jonathan A.Weyn
[5]SoukaynaMouatadid [1]NilsThuerey

1]Technical University of Munich, Germany 2]European Centre for Medium-range
Weather Forecasts, Reading, UK 3]Department of Meteorology and Bolin Centre
for Climate Research, Stockholm University, Sweden 4]Department of Atmospheric
Sciences, University of Washington, Seattle, USA 5]Department of Computer
Science, University of Toronto, Canada

Stephan Rasp<EMAIL_ADDRESS>
# WeatherBench: A benchmark dataset for data-driven weather forecasting

###### Abstract

Data-driven approaches, most prominently deep learning, have become powerful
tools for prediction in many domains. A natural question to ask is whether
data-driven methods could also be used to predict global weather patterns days
in advance. First studies show promise but the lack of a common dataset and
evaluation metrics make inter-comparison between studies difficult. Here we
present a benchmark dataset for data-driven medium-range weather forecasting,
a topic of high scientific interest for atmospheric and computer scientists
alike. We provide data derived from the ERA5 archive that has been processed
to facilitate the use in machine learning models. We propose simple and clear
evaluation metrics which will enable a direct comparison between different
methods. Further, we provide baseline scores from simple linear regression
techniques, deep learning models, as well as purely physical forecasting
models. The dataset is publicly available at https://github.com/pangeo-
data/WeatherBench and the companion code is reproducible with tutorials for
getting started. We hope that this dataset will accelerate research in data-
driven weather forecasting.

Deep learning, a branch of machine learning based on multi-layered artificial
neural networks, has proven to be a powerful tool for a wide range of tasks,
most notably image recognition and natural language processing (LeCun et al.,
2015). More recently, deep learning has also been used in many fields of
natural science. Much of the success of deep learning is based on the ability
of neural networks to recognize patterns in high-dimensional spaces. A natural
question to ask then is whether deep learning can also be used to predict
future weather patterns.

Currently, weather (and climate) predictions are based on purely physical
computer models, in which the governing equations, or our best approximation
thereof, of the atmosphere and ocean are solved on a discrete numerical grid
(Bauer et al., 2015). Overall, this approach has been very successful.
However, today’s numerical weather prediction (NWP) models still have
shortcoming for many important applications, for example forecasting mesoscale
convective systems over Africa (Vogel et al., 2018). Furthermore, huge amounts
of computing power are required, especially for creating probabilistic
forecasts which are usually limited to 50 ensemble members or less. For these
reasons and the growing popularity of machine learning (ML) there has been a
growing interest to improve and speed up NWP with data-driven approaches.

ML can be applied to weather prediction in many different ways. Two long-
standing applications of ML are post-processing – the correction of
statistical biases in the output of physical models – and statistical
forecasting – the prediction of variables not directly output by the physical
model. Traditionally, this has been done using simple linear techniques but
more recently modern machine learning approaches like random forests or neural
networks have been explored (Gagne et al., 2014; Taillardat et al., 2016;
McGovern et al., 2017; Lagerquist et al., 2017; Rasp and Lerch, 2018).
Typically, these approaches target very specific variables or locations
whereas the general evolution of the atmosphere is still predicted by a
physical model. Another application that has recently been explored using ML
is nowcasting, which describes the short range (up to 6 hours) prediction of
precipitation by directly extrapolating radar observation without a physical
model involved (Shi et al., 2015, 2017; Agrawal et al., 2019; Grönquist et
al., 2020).

Yet another direction for ML research is hybrid modeling, in which a physical
model is combined with data-driven components, for example replacing heuristic
cloud or radiation parameterizations (Chevallier et al., 1998; Krasnopolsky et
al., 2005; Rasp et al., 2018; Brenowitz and Bretherton, 2018; Yuval and
O’Gorman, 2020). The key idea behind these approaches is to only replace
uncertain (e.g. clouds) or computationally expensive (e.g. line-by-line
radiation) model components with machine learning emulators and leave other
model components (e.g. large-scale dynamics) untouched. However such hybrid
models also have drawbacks. First, the interaction between physical and
machine learning components are poorly understood and can lead to unexpected
instabilities and biases (Brenowitz and Bretherton, 2019). Second, they are
difficult to implement from a technical perspective because one has to
interface the machine learning components with complex climate model code,
typically written in Fortran.

Here we focus on purely-data driven prediction of the global atmospheric flow
in the medium-range. Specifically, we select lead times of 3 and 5 days, for
which the atmosphere is still reasonably deterministic but also exhibits
complex nonlinear behaviour, such as baroclinic instabilities and tropical
cyclogenesis. This forecast range is important from a societal point of view
because it delivers crucial information for disaster preparation, for example
for flooding, cold and hot spells or damaging winds (Lazo et al., 2009).
Creating a good medium-range forecast requires understanding complex
atmospheric dynamics and the interplay between several variables across a
range of scales. This sets this challenge apart from post-processing and
statistical forecasting, in which the large-scale dynamics are predicted by a
physical model, and nowcasting, in which the considered evolution is
univariate and short-term. In other words, this benchmark closely emulates the
task performed by physical NWP models.

There are several motivations for considering a purely-data driven approach.
As mentioned above current NWP is computationally expensive and, nevertheless,
has low skill for certain applications. If data-driven models were able to
learn a more efficient representation of the underlying dynamical and physical
equations, they might enable computationally cheaper forecasts. This can be
useful for many applications, for example creating very large ensembles to
better estimate the probability of extreme events. It is also possible that by
learning from a diverse set of data sources, data-driven models can outperform
physical models in areas where the latter struggle. While in this benchmark
challenge the focus is on upper-level fields of pressure and temperature – for
which physical models perform very well – the hope is that the insights gained
from this task can be leveraged for more impactful application. Further,
recent research into interpretable machine learning might provide scientists
with new analysis tools (McGovern et al., 2019; Toms et al., 2019). Finally,
there is the basic scientific question to what extent purely data-driven
models can learn the underlying dynamics of the atmosphere.

Note also that while this benchmark is framed as a data-driven prediction
challenge, the proposed framework can also be applied to post-processing using
the same metrics.

In machine learning research, the data-driven prediction of future states is
an active area of research with applications from language translation
(Sutskever et al., 2014), over audio signals (Oord et al., 2016), to numerical
simulations (Morton et al., 2018). In this context, weather forecasts are a
particularly challenging task. The behavior is highly complex and non-linear,
but also exhibits some recurring patterns, albeit only on local scales (Hamill
and Whitaker, 2006). As such the the proposed benchmark poses interesting
challenges for deep learning algorithms, e.g., to evaluate different
architectures (Ronneberger et al., 2015; He et al., 2015; Huang et al., 2016),
regularization methods (Krogh and Hertz, 1992; Srivastava et al., 2014; Xie et
al., 2017) or optimizers (Graves, 2013; Kingma and Ba, 2014).

In the last couple of years, several studies (summarized in Section 1) have
pioneered data-driven, global, medium-range weather prediction. All of them
show that there is some potential in this approach but also highlight the need
for further research. In particular, we currently lack a common benchmark
challenge to accelerate progress. Benchmark datasets can have a huge impact
because they make different algorithms quantitatively inter-comparable and
foster constructive competition, particularly in a nascent direction of
research. Famous examples are the computer vision datasets MNIST (LeCun et
al., 1998) and ImageNet (Russakovsky et al., 2015). Further, well-curated
benchmark datasets make it easier for people from different fields to work on
a problem (Ebert-Uphoff et al., 2017).

Here, we propose a benchmark problem for data-driven weather forecasting. We
provide a ready-to-use dataset for download along with specific metrics to
compare different approaches. In this paper, we start by reviewing the
previous work done on this topic (Section 1), describe the dataset (Section 2)
and the evaluation metrics (Section 3), and provide several baseline models
(Section 4). Finally, we will highlight several promising directions for
further research (Section 5) and conclude with a big picture view (Section
5.4).

Figure 1: Schematic of data-driven weather forecasting. a) Example of direct
weather prediction for 5 days lead time. The input to the neural network are
fields on a latitude-longitude grid. The fields can be several levels of the
same variable and/or different variables. The goal is to predict the same
fields some time ahead. b) Iterative forecasts are created from data-driven
models trained on a shorter lead time, for example 6 hours, which are then
iteratively called up to the required forecast lead time.

##  1 Overview of previous work

In this section, we briefly describe the four existing studies on predicting
the large-scale atmospheric state in the medium-range with a focus on the
data, methods and evaluation.

###  1.1 Dueben and Bauer (2018)

In this study, the authors trained a neural network to predict 500 hPa
geopotential (Z500; see Section 3 for details on commonly used fields), and in
some experiments 2 meter-temperature, 1 hour ahead. The training data was
taken from the ERA5 archive for the time period from 2010 to 2017 and
regridded to a 6 degree latitude-longitude grid. Two neural network variants
were used, a fully connected neural network and a spatially localized network,
similar to a convolutional neural network (CNN). After training they then
created iterative forecasts up to 120 h lead time for 10 month validation
period. They compared their data-driven forecasts to an operational NWP model
and the same model run at a spatial resolution comparable to the data-driven
method. One interesting detail is that their networks predict the difference
from one time step to the next, instead of the absolute field. To create these
iterative forecasts, they use a third-order Adams-Bashford explicit time-
stepping scheme. The CNN predicting only geopotential performed best but was
unable to beat the low-resolution physical baseline.

###  1.2 Scher (2018) and Scher and Messori (2019b)

These two studies addressed the issue of data-driven weather forecasting in a
simplified reality setting. Long runs of simplified General Circulation Models
(GMCs) were used as “reality”. Neural networks were trained to predict the
model fields several days ahead. The neural network architecture are CNNs with
an encoder-decoder setup. They take as input the instantaneous 3D model fields
at one timestep, and output the same model fields at some time later. In Scher
(2018), a separate network was trained for each lead-time up to 14 days. Scher
and Messori (2019b) trained only on 1-day forecasts, and constructed longer
forecasts iteratively. Interestingly, networks trained to directly predict a
certain forecast time, e.g. 5 days, outperformed iterative networks. The
forecasts were evaluated using the root mean squared error and the anomaly
correlation coefficient of Z500 and 800 hPa temperature. Scher (2018) used a
highly simplified GCM without hydrological cycle, and achieved very high
predictive skill. Additionally, they were able to create stable "climate" runs
(long series of consecutive forecasts) with the network. Scher and Messori
(2019b) used several more realistic and complex GCMs. The data-driven model
achieved relatively good short-term forecast skill, but was unable to generate
stable and realistic “climate” runs. In terms of neural-network architectures
they showed that architectures tuned on simplified GCMs also work on more
complex GCMs, and that the same architecture also has some prediction skill on
single-level reanalysis data.

###  1.3 Weyn et al. (2019)

In this study, reanalysis-derived Z500 and 700-300 hPa thickness at 6-hourly
time steps are predicted with deep CNNs. The data are from the Climate
Forecast System (CFS) Reanalysis from 1979–2010 with 2.5-degree horizontal
resolution and cropped to the northern hemisphere. The authors used similar
encoder-decoder convolutional networks as those used by Scher (2018) and Scher
and Messori (2019b) but also experimented with adding a convolution long
short-term memory (LSTM; Hochreiter and Schmidhuber, 1997) hidden layer. As in
Scher and Messori (2019b), forecasts are generated iteratively by feeding the
model’s outputs back in as inputs. The authors found that using two input time
steps, 6 h apart, and predicting two output time steps, performed better than
using a single step. Their best CNN forecast outperforms a climatology
benchmark at up to 120 h lead time, and appears to correctly asymptote towards
persistence forecasts at longer lead times up to 14 days.

These three approaches outline promising first steps towards data-driven
forecasting. The differences of the proposed methods already highlight the
importance of a common benchmark case to compare prediction skill.

##  2 Dataset

For the proposed benchmark, we use the ERA5 reanalysis dataset (Hersbach et
al., 2020) for training and testing. Reanalysis datasets provide the best
guess of the atmospheric state at any point in time by combining a forecast
model with the available observations. The raw data is available hourly for 40
years from 1979 to 2018 on a 0.25°latitude-longitude grid (721$\times$1440
grid points) with 37 vertical levels.

Since this raw dataset is very large (a single vertical level for the entire
time period amounts to almost 700GB of data), we regrid the data to lower
resolutions. This is also a more realistic use case, since very high
resolutions are still hard to handle for deep learning models because of GPU
memory constraints and I/O speed. In particular, we chose 5.625° (32$\times$64
grid points), 2.8125° (64$\times$128 grid points) and 1.40525° (128$\times$256
grid points) resolution for our data. The regridding was done with the xesmf
Python package (Zhuang, 2019) using a bilinear interpolation. Powers of two
for the grid are used since this is common for many deep learning
architectures where image sizes are halved in the algorithm. Further, for 3D
fields we selected 13 vertical levels: 50, 100, 150, 200, 250, 300, 400, 500,
600, 700, 850, 925, 1000 hPa. Note that it is common to use pressure in hecto-
Pascals as a vertical coordinate instead of physical height. The pressure at
sea level is approximately 1000 hPa and decreases roughly exponentially with
height. 850 hPa is at around 1.5 km height. 500 hPa is at around 5.5 km
height. If the surface pressure is smaller than a given pressure level, for
example at high altitudes, the pressure-level values are interpolated. The
selected pressure levels contain the seven pressure levels that are commonly
used for 3D output by the climate models in the Coupled Model Intercomparison
Project Phase 6 (CMIP6, Eyring et al., 2016) which could be useful for
pretraining. One regridded historical climate run is also available from the
data repository with a template workflow for downloading further CMIP data on
the Github repository.

The processed data (see Table 1) are available at
https://mediatum.ub.tum.de/1524895 (Rasp et al., 2020). The data are split
into yearly NetCDF files for each variable and resolution, packed in a zip
file. The entire dataset at 5.625° resolution has a size of 191GB. Individual
variables amount to around 25GB three-dimensional and 2GB for two-dimensional
fields. File sizes for 2.8125° and 1.40525° resolutions are a factor 4 and 16
times larger. Data processing was organized using Snakemake (Koster and
Rahmann, 2012). For further instructions on data downloading visit the Github
page111https://github.com/pangeo-data/WeatherBench. The available variables
were chosen based on meteorological consideration. Geopotential, temperature,
humidity and wind are prognostic state variables in most physical NWP and
climate models. Geopotential at a certain pressure level $p$, typically
denoted as $\Phi$ with units of m2s-2, defined as

$\Phi=\int_{0}^{z\text{ at }p}g\,dz^{\prime}$ (1)

where $z$ describes height in meters and $g=9.81$ m s-2 is the gravitational
acceleration. Horizontal relative vorticity, defined as $\partial v/\partial
x-\partial u/\partial y$, describes the rotation of air at a given point in
space. Potential vorticity (Hoskins et al., 1985; Holton, 2004) is a commonly
used quantity in synoptic meteorology which combines the rotation (vorticity)
and vertical temperature gradient of the atmosphere. It is defined as
$PV=\rho^{-1}\zeta_{a}\cdot\nabla\theta$, where $\rho$ is the density,
$\zeta_{a}$ is the absolute vorticity (relative plus the Earth’s rotation) and
$\theta$ is the potential temperature. In addition to the three-dimensional
fields, we also include several two-dimensional fields: 2 meter-temperature is
often used as an impact variable because of its relevance for human activities
and is directly affected by the diurnal solar cycle; 10 meter-wind is also an
important impact-related forecast variable, for example for wind energy;
similarly, total cloud cover is an essential variable for solar energy
forecasting. We also included precipitation but urge caution since
precipitation in reanalysis datasets often shows large deviation from
observations (e.g. Betts et al., 2019; Xu et al., 2019). Finally, we added the
top-of-atmosphere incoming solar radiation as it could be a useful input
variable to encode the diurnal cycle. Further, there are several potentially
important time-invariant fields, which are contained in the constants file.
The first three variables enclose information about the surface: the land-sea
mask is a binary field with ones for land points; the soil type consists of
seven different soil categories222Coarse = 1, Medium = 2, Medium fine = 3,
Fine = 4, Very fine = 5, Organic = 6, Tropical organic = 7, see
https://apps.ecmwf.int/codes/grib/param-db?id=43; orography is simply the
surface height. In addition, we included two-dimensional fields with the
latitude and longitude values at each point. Particularly the latitude values
could become important for the network to learn latitude-specific information
such as the grid structure or the Coriolis effect (see Section 5). The Github
code repository includes all scripts for downloading and processing of the
data. This enables users to download additional variables or regrid the data
to a different resolution.

Table 1: List of variables contained in the benchmark dataset.

Long name | Short name | Description | Unit | Levels
---|---|---|---|---
geopotential | z | Proportional to the height of a pressure level | [m2s-2] | 13 levels
temperature | t | Temperature | [K] | 13 levels
specific_humidity | q | Mixing ratio of water vapor | [kg kg-1] | 13 levels
relative_humidity | r | Humidity relative to saturation | [%] | 13 levels
u_component_of_wind | u | Wind in x/longitude-direction | [m s-1] | 13 levels
v_component_of_wind | v | Wind in y/latitude direction | [m s-1] | 13 levels
vorticity | vo | Relative horizontal vorticity | [1 s-1] | 13 levels
potential_vorticity | pv | Potential vorticity | [K m2 kg-1 s-1] | 13 levels
2m_temperature | t2m | Temperature at 2 m height above surface | [K] | Single level
10m_u_component_of_wind | u10 | Wind in x/longitude-direction at 10 m height | [m s-1] | Single level
10m_v_component_of_wind | v10 | Wind in y/latitude-direction at 10 m height | [m s-1] | Single level
total_cloud_cover | tcc | Fractional cloud cover | (0–1) | Single level
total_precipitation | tp | Hourly precipitation | [m] | Single level
toa_incident_solar_radiation | tisr | Accumulated hourly incident solar radiation | [J m-2] | Single level
constants |  | File containing time-invariant fields |  |
land_binary_mask | lsm | Land-sea binary mask | (0/1) | Single level
soil_type | slt | Soil-type categories | see text | Single level
orography | orography | Height of surface | [m] | Single level
latitude | lat2d | 2D field with latitude at every grid point | [°] | Single level
longitude | lon2d | 2D field with longitude at every grid point | [°] | Single level

##  3 Evaluation

Evaluation is done for the years 2017 and 2018. To make sure no overlap exists
between the training and test dataset, the first test date is 1 January 2017
00UTC plus forecast time (i.e. for a three day forecast the first test date
would be 4 January 2017 00UTC) while the last training target is 31 December
2016 23UTC. Further, the evaluation presented here is done on 5.625°
resolution333The evaluation of all baselines in this paper are done in this
Jupyter notebook: https://github.com/pangeo-
data/WeatherBench/blob/master/notebooks/4-evaluation.ipynb. This means that
predictions at higher resolutions have to be downscaled to the evaluation
resolution. We also evaluated some baselines at higher resolutions and found
that the scores were almost identical with differences smaller than 1%.
Therefore we are reassured that little information is lost by evaluating at a
coarser resolution.

A note on validation and testing: In machine learning it is good practice to
split the data into three parts: the training, validation and test sets. The
training dataset is used to actually fit the model. The validation dataset is
used during experimentation to check the model performance on data not seen
during training. However, there is the danger that through continued tuning of
hyperparameters one unwillingly overfits to the validation dataset. Therefore
it is advisable to keep a third, testing, dataset for final evaluations of
model performance. For this benchmark this final evaluation is done for the
years 2017 and 2018. Therefore, we strongly encourage users of this dataset to
pick a period from 1979 to 2016 for validation of their models for
hyperparameter tuning. Because meteorological fields are highly correlated in
time, it is advisable to choose a longer contiguous chunk of data for
validation instead of a completely random split. Here we chose the year 2016
for validation.

We chose 500 hPa geopotential and 850 hPa temperature as primary verification
fields. Geopotential at 500 hPa pressure, often abbreviated as Z500, is a
commonly used variable that encodes the synoptic-scale pressure distribution.
It is the standard verification variable for most medium-range NWP models.
Note that geopotential height, also commonly used, is defined as $\Phi/g$ with
units of meters. We picked 850 hPa temperature as our secondary verification
field because temperature is a more impact-related variable. 850 hPa is
usually above the planetary boundary layer and therefore not affected by
diurnal variations but provides information about broader temperature trends,
including cold spells and heat waves. In addition we also provide some
baseline scores for total 6-hourly accumulated precipitation (TP; but noting
the dubious quality mentioned above) and 2-meter temperature (T2M). However,
they will not be further discussed here.

We chose the root mean squared error (RMSE) as our primary metric because it
is easy to compute and mirrors the loss used for most ML applications. We
define the RMSE as the mean latitude-weighted RMSE over all forecasts:

$\text{RMSE}=\frac{1}{N_{\text{forecasts}}}\sum_{i}^{{N_{\text{forecasts}}}}\sqrt{\frac{1}{N_{\text{lat}}N_{\text{lon}}}\sum_{j}^{N_{\text{lat}}}\sum_{k}^{N_{\text{lon}}}L(j)(f_{i,j,k}-t_{i,j,k})^{2}}$
(2)

where $f$ is the model forecast and $t$ is the ERA5 truth. $L(j)$ is the
latitude weighting factor for the latitude at the $j$th latitude index:

$L(j)=\frac{\cos(\text{lat}(j))}{\frac{1}{N_{\text{lat}}}\sum_{j}^{N_{\text{lat}}}\cos(\text{lat}(j))}$
(3)

In addition, we also evaluate the baselines using the latitude weighted
anomaly correlation coefficient (ACC; see Section 7.6.4 of Wilks, 2006) and
the mean absolute error (MAE). The tables and figures can be found in the
Appendix A. For smooth fields like Z500 and T850 the qualitative differences
between the metrics are small. For intermittent fields like precipitation the
choice of metric matters a lot more.

##  4 Baselines

To evaluate the skill of a forecasting model it is important to have baselines
to compare to. In this section, we compute scores for several baselines. The
results are summarized in Fig. 6 and Table 2.

Figure 2: RMSE of a) 500 hPa geopotential and b) 850 hPa temperature for different baselines at 5.625° resolution. Solid lines for linear regression and CNN indicate iterative forecasts, while dots represent direct forecasts for 3 and 5 days lead time. Table 2: Baseline RMSE for 3 and 5 days forecast time at 5.625° resolution. Best machine learning baseline and physical model are highlighted. TP is 6 hourly accumulated precipitation. RMSE (3 days / 5 days) |
---|---
Baseline | Z500 [m2 s-2] | T850 [K] | T2M [K] | TP [mm]
Persistence | 936 / 1033 | 4.23 / 4.56 | 3.00 / 3.27 | 3.23 / 3.24
Climatology | 1075 | 5.51 | 6.07 | 2.36
Weekly climatology | 816 | 3.50 | 3.19 | 2.32
Linear regression (direct) | 693 / 783 | 3.19 / 3.44 | 2.39 / 2.60 | 2.37 / 2.37
Linear regression (iterative) | 718 / 810 | 3.17 / 3.48 |  |
CNN (direct) |  626 / 757 |  2.87 / 3.37 |  |
CNN (iterative) | 1114 / 1559 | 4.48 / 9.69 |  |
IFS T42 | 489 / 743 | 3.09 / 3.83 | 3.21 / 3.69 |
IFS T63 | 268 / 463 | 1.85 / 2.52 | 2.04 / 2.44 |
Operational IFS |  154 / 334 |  1.36 / 2.03 |  1.35 / 1.77 | 2.36 / 2.59

###  4.1 Persistence and Climatology

The two simplest possible forecasts are a) a persistence forecast in which the
fields at initialization time are used as forecasts ("tomorrow’s weather is
today’s weather"), and b) a climatological forecast. For the climatological
forecast, two different climatologies were computed from the training dataset
(1979–2016): first, a single mean over all times in the training dataset and,
second, a mean computed for each of the 52 calendar weeks. The weekly
climatology is significantly better, approximately matching the persistence
forecast between 1 and 2 days, since it takes into account the seasonal cycle.
This means that to be useful, a forecast system needs to beat the weekly
climatology and the persistence forecast.

###  4.2 Operational NWP model

The gold standard of medium-range NWP is the operational IFS (Integrated
Forecast System) model of the European Center for Medium-range Weather
Forecasting (ECMWF)444https://www.ecmwf.int/en/forecasts/documentation-and-
support. We downloaded the forecasts for 2017 and 2018 from the THORPEX
Interactive Grand Global Ensemble (TIGGE; Bougeault et al., 2010)
archive555The TIGGE data for total precipitation and 2m temperature was
damaged for 2017. For this reason the TIGGE evaluation for these variables is
only done using the 2018 data., which contains the operational forecasts,
initialized at 00 and 12 UTC regridded to a 0.5° by 0.5° grid, which we
further regridded to 5.625°. Note that the forecast error starts above zero
because the operational IFS is initialized from a different analysis.
Operational forecasting is computationally very expensive. The current IFS
deterministic forecast is computed on a cluster with 11,664 cores. One 10 day
forecast at 10 km resolution takes around 1 hour of real time to compute.

###  4.3 Physical NWP model run at coarser resolution

To provide physical baselines more in line with the computational resources of
a data-driven model, we ran the IFS model at two coarser horizontal
resolutions, T42 (approximately 2.8° or 310 km resolution at the equator
(NCAR, )) with 62 vertical levels and T63 (approximately 1.9° or 210 km) with
137 vertical levels. The T42 run was initialized from ERA5 whereas the T63 run
was initialized from the operational analysis. The gap in skill at $t=0$ is
caused by the conversion to spherical coordinates at coarse resolutions. For
Z500 the skill for these two runs lies in-between the operational IFS and the
machine learning baselines. For T850, the T42 run is significantly worse. The
likely reason for this is that temperature close to the ground is much more
affected by the resolution and representation of topography within the model.
Further, the model was not specifically tuned for these resolutions.
Computationally, a single forecast takes 270 seconds for the T42 model and 503
seconds for the T64 model on a single XC40 node with 36 cores. Since the
computational costs and resolutions of these runs are much closer to those of
a data-driven method, beating those baselines should be a realistic target.
note, however, that the model was not tuned to run at such coarse resolutions.

###  4.4 Linear regression

As a first purely data-driven baseline we fit a simple linear regression
model. For the direct predictions a separate model was trained for each of the
four variables. For this purpose the 2D fields were flattened from
32$\times$64 $\to$ 2048\. This was done for 3 d and 5 d forecast time. In
addition an iterative model for Z500 and T850 was trained. Here we use a
single linear regression to predict 6 hours ahead where the two fields are
concatenated (2$\times$32$\times$64 $\to$ 4096). The advantage of iterative
forecasts is that a single model is able to make predictions for any forecast
time rather than having to train several models. For iterative forecasts the
model takes its previous output as input for the next step. To create a 5 day
iterative forecast the model trained to predict 6 hour forecasts is called 20
times. For this model, the iterative forecast performs just as well as the
direct forecast due to its linear nature. At 5 days, the linear regression
forecast is about as good as the weekly climatology.

###  4.5 Simple convolutional neural network

As our deep learning baseline we chose a simple fully-convolutional neural
network. CNNs are the natural choice for spatial data since they exploit
translational invariances in images/fields. Here we train a CNN with 5 layers.
Each hidden layer has 64 channels with a convolutional kernel of size 5 and
ELU activations (Clevert et al., 2015). The input and output layers have two
channels, representing Z500 and T850. The model was trained using the Adam
optimizer (Kingma and Ba, 2014) and a mean squared error loss function. The
total number of trainable parameters is 313,858. We implemented periodic
convolutions in the longitude direction but not the latitude direction. The
implementation can be found in the Github repository. The direct CNN forecasts
beat the linear regression forecasts for 3 and 5 days forecast time. However,
at 5 days these forecasts are only marginally better than the weekly
climatology (see Table 2). This baseline, however, is to be seen simply as a
starting point for more sophisticated data driven methods. The iterative CNN
forecast, which equivalently to the linear regression iterative forecast was
created by chaining together 6 hourly predictions, performs well up to around
1.5 days but then the network’s errors grow quickly and diverge. This confirms
the findings of Scher and Messori (2019a) whose experiments showed that
training with longer lead time yields better results than chaining together
short-term forecasts. However, the poor skill of the iterative forecast could
easily be a result of using an overly simplistic network architecture. The
iterative forecasts of Weyn et al. (2019), who employ a more complex network
structure, show stable long term performance up to two weeks with realistic
statistics.

###  4.6 Example forecasts

To further illustrate the prediction task, Fig. 3 shows example geopotential
and temperature fields. The ERA5 temporal differences show several interesting
features. First, the geopotential fields and differences are much smoother
compared to the temperature fields. The differences in both fields are also
much smaller in the tropics compared to the extratropics where propagating
fronts can cause rapid temperature changes. An interesting feature is
detectable in the 6h Z500 difference field in the tropics. These alternating
patterns are signatures of atmospheric tides.

The CNN forecasts for 6h lead time are not able to capture these wave-like
patterns which hints at a failure to capture the basic physics of the
atmosphere. For 5 days forecast time the CNN model predicts unrealistically
smooth fields. This is likely a result of two factors: first, the two input
fields used in this baseline CNN contain insufficient information to create a
skillful 5 day forecast; and second, at 5 days the atmosphere already shows
some chaotic behavior which causes a model trained with a simple RMSE loss to
predict smooth fields (see Section 5). The IFS operational forecast has much
smaller errors than the CNN forecast. It is able to capture the propagation of
tropical waves. Its main errors appear at 5 days in the mid-latitudes where
extratropical cycles are in slightly wrong positions.

Figure 3: Example fields for 2017-01-01 00UTC initialization time. The top two
rows show the ERA5 "truth" fields for geopotential (Z500) and temperature
(T850) at initialization time (t=0h) and for 6h and 5d forecast time. In
addition, the difference between the forecast times and the initialization
time is shown. The third and fourth rows show the forecasts from the CNN
model. Rows five and six show the IFS operational model. For the CNN forecasts
the first column is identical to the ERA5 truth. We selected the 6h iterative
CNN model for the 6h forecast but the 5d direct CNN model for the 5 day
forecast. For the IFS the initial states (t=0h) differ slightly albeit not
visibly. In addition to the forecast fields the error relative to the ERA5
"truth" is shown in the third and fifth columns. Please note that the
colorbars for the difference fields change.

##  5 Discussion

###  5.1 Weather-specific challenges

From a ML perspective, state-to-state weather prediction is similar to image-
to-image translation. For this sort of problem many deep learning techniques
have been developed in recent years (Kaji and Kida, 2019). However,
forecasting weather differs in some important ways from typical image-to-image
applications and raises several open questions.

First, the atmosphere is three-dimensional. So far, this aspect has not been
taken into account. In the networks of Scher and Messori (2019a), for example,
the different levels have been treated as separate channels of the CNN.
However, simply using a three-dimensional CNN might not work either because
atmospheric dynamics and grid spacings change in the vertical, thereby
violating the assumption of translation invariance which underlies the
effectiveness of CNNs. This directly leads to the next challenge: On a regular
latitude-longitude grid, the dynamics also change with latitude because
towards the poles the grid cells become increasingly stretched. This is in
addition to the Coriolis effect, the deflection of wind caused by the rotation
of Earth, which also depends on latitude. A possible solution in the
horizontal could be to use spherical convolutions (Cohen et al., 2018;
Perraudin et al., 2019; Jiang et al., 2019) or to feed in latitude information
to the network.

Another potential issue is the limited amount of training data available. 40
years of hourly data amounts to around 350,000 samples. However, the samples
are correlated in time. If one assumes that a new weather situation occurs
every day, then the number of samples is reduced to around 15,000. Without
empirical evidence it is hard to estimate whether this number is sufficient to
train complex networks without overfitting. Should overfitting be a problem,
one could try transfer learning. In transfer learning, the network is
pretrained on a similar task or dataset, for example, climate model
simulations, and then finetuned on the actual data. This is common practice in
computer vision and has been successfully applied to seasonal ENSO forecasting
(Ham et al., 2019). Another common method to prevent overfitting is data
augmentation, which in traditional computer vision is done by e.g. randomly
rotating or flipping the image. However, many of the traditional data
augmentation techniques are questionable for physical fields. Random
rotations, for example, will likely not work for this dataset since the x and
y directions are physically distinct. Thus, finding good data augmentation
techniques for physical fields is an outstanding problem. Using ensemble
analyses and forecasts could provide more diversity in the training dataset.

Finally, there are technical challenges. Data for a single variable with ten
levels at 5.625° resolution take up around 30 GB of data. For a network with
several variables or even at higher resolution, the data might not fit into
CPU RAM any more and data loading could become a bottleneck. For image files,
efficient data loaders have been created666See e.g.
https://keras.io/preprocessing/image/ or
https://pytorch.org/tutorials/beginner/data_loading_tutorial.html. One
promising but so far unexplored option is to use Tensorflow’s TFRecords
(https://www.tensorflow.org/tutorials/load_data/tfrecord). For netCDF files,
however, so far no efficient solution exists to our knowledge. Further, one
can assume that to create a competitive data-driven NWP model, high
resolutions have to be used, for which GPU RAM quickly becomes a limitation.
This suggests that multi-GPU training might be necessary to scale up this
approach (potentially similar to the technical achievement of Kurth et al.
(2018)).

###  5.2 Probabilistic forecasts and extremes

One important aspect that is not currently addressed by this benchmark is
probabilistic forecasting. Because of the chaotic evolution of the atmosphere,
it is very important to also have an estimate of the uncertainty of a
forecast. In physical NWP this is done by running several forecasts, called an
ensemble, from slightly different initial conditions and potentially with
different or stochastic model physics (Palmer, 2019). From this Monte Carlo
forecast one can then estimate a probability distribution. A different
approach, which is often taken in statistical post-processing of NWP
forecasts, is to directly estimate a parametric distribution (e.g. Gneiting et
al., 2005; Rasp and Lerch, 2018). For a probabilistic forecast to be reliable
the forecast uncertainty has to be an accurate indicator of the error. A good
first order approximation for this is the spread (ensemble standard deviation)
to error (RMSE) ratio which should be one (Leutbecher and Palmer, 2008). A
more stringent test is to use a proper probabilistic scoring rule, for example
the continuous ranked probability score (CRPS) (Gneiting and Raftery, 2007;
Gneiting and Katzfuss, 2014). For deterministic forecast the CRPS reduces to
the mean absolute error. Extending this benchmark to probabilistic forecasting
simply requires computing a probabilistic score. How to produce probabilistic
data-driven forecasts is a very interesting research question in its own
right. We encourage users of this benchmark to explore this dimension.

A related issue is the question of extreme weather situations, for example
heat waves. These events are, by definition, rare, which means that they will
contribute little to regular verification metrics like the RMSE. However, for
society these events are highly important. For this reason, it would make
sense to evaluate extreme situations separately. But defining extremes is
ambiguous which is why there is no standard metric for evaluating extremes.
The goal of this benchmark is to provide a simple, clear problem. Therefore,
we decided to omit extremes for now but users are encouraged to chose their
own verification of extremes.

###  5.3 Climate simulations

Another aspect that is untouched by the benchmark challenge proposed here is
climate prediction. Even though weather and climate deal with the same
underlying physical system, they pose different forecasting challenges. In
weather forecasting the goal is to predict the state of the atmosphere at a
specific time into the future. This is only possible up to the prediction
horizon of the atmosphere, which is thought to be at roughly two weeks.
Climate models, on the other hand, are evaluated by comparing long-term
statistics to observations, for example the mean surface temperature (Stocker
et al., 2013). Scher and Messori (2019a) created iterative climate time scale
runs with their data-driven models and compared first and second-order
statistics. They found that the model sometimes produced a stable climate but
with significant biases and a poor seasonal cycle. This indicates that, so
far, iterative data-driven models have been unable to produce physically
reasonable long-term predictions. This remains a key challenge for future
research. While not specifically included in this benchmark, a good test for
climate simulations is to look at long term mean statistics and the seasonal
cycle as done in Figs. 6 and 7 of Scher and Messori (2019a).

Climate change simulations represent another step up in complexity. To start
with, external greenhouse gas forcing would have to be included. Further,
future climates will produce atmospheric states that lie outside of the
historical manifold of states. Plain neural networks are very bad at
extrapolating to climates beyond what they have seen in the training dataset
(Rasp et al., 2018). For this reason, climate change simulations with current
data-driven methods are likely not a good idea. However, research into
physical machine learning is ongoing and might offer new opportunities in the
near future (e.g. Bar-Sinai et al., 2019; Beucler et al., 2019).

###  5.4 Promising research directions

There is a wide variety of promising research directions for data-driven
weather forecasting. The most obvious direction is to increase the amount of
data used for training and the complexity of the network architecture. This
dataset provides a, so far, unexploited volume and diversity of data for
training. It is up to future research to find out exactly which combination of
variables will turn out to be useful. Further, this dataset offers a four
times higher horizontal resolution than all previous studies. The hope is that
this data will enable researcher to train more complex models than have
previously been used.

With regards to model architecture, there is a huge variety of network
architectures that can be explored. U-Nets (Ronneberger et al., 2015) have
been used extensively for image segmentation tasks that require computations
across several spatial scales. Resnets (He et al., 2015) are currently the
state of the art for image classification and their residual nature could be a
good fit for state-to-state forecasting tasks. For synthesis tasks, generative
adverserial networks (GANs) (Goodfellow et al., 2014) were shown to be
particularly powerful for creating realistic natural images and fluid flows
(Xie et al., 2018). This might be attractive since minimizing a mean loss,
such as the MSE, for random or stochastic data leads to unrealistically smooth
predictions as seen in Fig. 3. Conditional GANs (Mirza and Osindero, 2014;
Isola et al., 2016) could potentially alleviate this issue but it is still
unclear to what extent GAN predictions are able to recover the multi-variate
distribution of the training samples.

The dataset is available at https://mediatum.ub.tum.de/1524895 (Rasp et al.,
2020). Code, instructions for dowloading the data and evaluating forecasts can
be found at https://github.com/pangeo-data/WeatherBench.

In this paper a benchmark dataset for data-driven weather forecasting is
presented. It focuses on global medium-range (roughly 2 days to 2 weeks)
prediction. With the rise of deep learning in physics, weather prediction is a
challenging and interesting target because of the large overlap with
traditional deep learning tasks (Reichstein et al., 2019). While first
attempts have been made in this direction, as discussed in Section 1, the
field currently lacks a common dataset which enables the inter-comparison of
different methods. We hope that this benchmark can provide a foundation for
accelerated research in this area. Loosely following Ebert-Uphoff et al.
(2017), the key features of this benchmark are:

  * •

Scientific impact: Numerical weather forecasting impacts many aspects of
society. Currently, NWP model run on massive super-computers at very high
computational cost. Building a capable data-driven model would be beneficial
in many ways (see Section WeatherBench: A benchmark dataset for data-driven
weather forecasting). In addition, there is the open, and highly debated,
question whether fully data-driven methods are able to learn a good
representation of atmospheric physics.

  * •

Challenge for data science: While global weather prediction is conceptually
similar an image-to-image task, and therefore allows for the application of
many state-of-the-art deep learning techniques, there are some unique
challenges to this problem: the three-dimensional, anisotropic nature of the
atmosphere; non-uniform-grids; potentially limited amounts of training data
and the technical challenge of handling large data volumes.

  * •

Clear metric for success: We defined a single metric (RMSE) for two fields
(500 hPa geopotential and 850 hPa temperature). These scores provide a simple
measure of success for data-driven, medium-range forecast model.

  * •

Quick start: The code repository contains a quick-start Jupyter notebook for
reading the data, training a neural network and evaluating the predictions
against the target data. In addition, the repository contains many functions
which are likely to be used frequently, for example an implementation of
periodic convolutions in Keras.

  * •

Reproducibility and citability: All baselines and results form this paper are
fully reproducible from the code repository. Further, the baseline predictions
are all saved in the data repository. The data has been assigned a permanent
DOI.

  * •

Communication platform: We will use the Github code repository as an evolving
hub for this project. We encourage users of this dataset to start by forking
the repository and eventually merge code that might be useful for others back
into the main branch. The main platform for communication, e.g. asking
questions, about this project will be Github issues.

We hope that this benchmark will foster collaboration between atmospheric and
data scientists in the ways we imagined and beyond.

##  Appendix A Additional metrics

The anomaly correlation coefficient (ACC) is defined as

$\text{ACC}=\frac{\sum_{i,j,k}L(j)f^{\prime}_{i,j,k}t^{\prime}_{i,j,k}}{\sqrt{\sum_{i,j,k}L(j)f^{\prime
2}_{i,j,k}\sum_{i,j,k}L(j)t^{\prime 2}_{i,j,k}}}$ (4)

where the prime ′ denotes the difference to the climatology. Here the
climatology is defined as $\mathrm{climatology}_{j,k}=\frac{1}{N_{time}}\sum
t_{j,k}$. The mean absolute error is defined just like the MSE (Eq. 2) but
with the absolute instead of the squared difference.

Figure 4: RMSE of a) 2-meter temperature and b) 6-hourly accumulated
precipitation for different baselines at 5.625° resolution.

Figure 5: ACC of a) 500 hPa geopotential, b) 850 hPa temperature, c) 2-meter
temperature and d) 6-hourly accumulated precipitation for different baselines
at 5.625° resolution.

Figure 6: MAE of a) 500 hPa geopotential, b) 850 hPa temperature, c) 2-meter
temperature and d) 6-hourly accumulated precipitation for different baselines
at 5.625° resolution.

Table 3: Baseline ACC for 3 and 5 days forecast time at 5.625° resolution. Best machine learning baseline and physical model are highlighted. TP is 6 hourly accumulated precipitation. ACC (3 days / 5 days) |
---|---
Baseline | Z500 | T850 | T2M | TP
Persistence | 0.62 / 0.53 | 0.69 / 0.65 | 0.88 / 0.85 | 0.06/0.06
Climatology | 0 | 0 | 0 | 0
Weekly climatology | 0.65 | 0.77 | 0.85 | 0.16
Linear regression (direct) | 0.76 / 0.68 | 0.81 / 0.78 | 0.92 / 0.90 | 0.15 / 0.13
Linear regression (iterative) | 0.76 / 0.67 | 0.82 / 0.78 |  |
CNN (direct) | 0.81 / 0.71 | 0.85 / 0.79 |  |
CNN (iterative) | 0.61 / 0.41 | 0.72 / 0.31 |  |
IFS T42 | 0.90 / 0.78 | 0.86 / 0.78 | 0.87 / 0.83 |
IFS T63 | 0.97 / 0.91 | 0.94 / 0.90 | 0.94 / 0.92 |
Operational IFS | 0.99 / 0.95 | 0.97 / 0.93 | 0.98 / 0.96 | 0.43 / 0.30
Table 4: Baseline MAE for 3 and 5 days forecast time at 5.625° resolution. Best machine learning baseline and physical model are highlighted. TP is 6 hourly accumulated precipitation. MAE (3 days / 5 days) |
---|---
Baseline | Z500 [m2 s-2] | T850 [K] | T2M [K] | TP [mm]
Persistence | 572 / 634 | 2.91 / 3.12 | 1.81 / 1.95 | 1.07 / 1.08
Climatology | 708 | 3.84 | 3.77 | 0.92
Weekly climatology | 525 | 2.48 | 2.00 | 0.88
Linear regression (direct) | 431 / 489 | 2.23 / 2.41 | 1.48 / 1.60 | 0.96 / 0.97
Linear regression (iterative) | 447 / 508 | 2.20 / 2.42 |  |
CNN (direct) | 403 / 476 | 2.02 / 2.38 |  |
CNN (iterative) | 892 / 1263 | 3.49 / 7.49 |  |
IFS T42 | 295 / 449 | 1.99 / 2.53 | 1.83 / 2.12 |
IFS T63 | 188 / 307 | 1.30 / 1.73 | 1.28 / 1.53 |
Operational IFS | 97 / 198 | 0.93 / 1.35 | 0.84 / 1.08 | 0.69 / 0.81

SR, PD, SS and JW conceived the idea. SR prepared the data and baselines and
led the writing. All authors contributed to the manuscript.

The authors declare no competing interests.

###### Acknowledgements.

Stephan Rasp acknowledges funding from the German Research Foundation (DFG).
We thank the Copernicus Climate Change Service (C3S) for allowing us to
redistribute the data. Peter D. Dueben gratefully acknowledges funding from
the Royal Society for his University Research Fellowship and the ESIWACE2
project. The ESIWACE2 project have received funding from the European Union’s
Horizon 2020 research and innovation programme under grant agreement No
823988.

## References

  * Agrawal et al. (2019) Agrawal, S., Barrington, L., Bromberg, C., Burge, J., Gazen, C., and Hickey, J.: Machine Learning for Precipitation Nowcasting from Radar Images, URL https://arxiv.org/abs/1912.12132, 2019.
  * Bar-Sinai et al. (2019) Bar-Sinai, Y., Hoyer, S., Hickey, J., and Brenner, M. P.: Learning data-driven discretizations for partial differential equations, Proceedings of the National Academy of Sciences, 116, 15 344–15 349, 10.1073/PNAS.1814058116, URL https://www.pnas.org/content/116/31/15344.short, 2019.
  * Bauer et al. (2015) Bauer, P., Thorpe, A., and Brunet, G.: The quiet revolution of numerical weather prediction, Nature, 525, 47–55, 10.1038/nature14956, URL http://www.nature.com/doifinder/10.1038/nature14956, 2015.
  * Betts et al. (2019) Betts, A. K., Chan, D. Z., and Desjardins, R. L.: Near-Surface Biases in ERA5 Over the Canadian Prairies, Frontiers in Environmental Science, 7, 10.3389/fenvs.2019.00129, URL https://www.frontiersin.org/article/10.3389/fenvs.2019.00129/full, 2019\.
  * Beucler et al. (2019) Beucler, T., Rasp, S., Pritchard, M., and Gentine, P.: Achieving Conservation of Energy in Neural Network Emulators for Climate Modeling, URL http://arxiv.org/abs/1906.06622, 2019.
  * Bougeault et al. (2010) Bougeault, P., Toth, Z., Bishop, C., Brown, B., Burridge, D., Chen, D. H., Ebert, B., Fuentes, M., Hamill, T. M., Mylne, K., Nicolau, J., Paccagnella, T., Park, Y.-Y., Parsons, D., Raoult, B., Schuster, D., Dias, P. S., Swinbank, R., Takeuchi, Y., Tennant, W., Wilson, L., and Worley, S.: The THORPEX Interactive Grand Global Ensemble, Bulletin of the American Meteorological Society, 91, 1059–1072, 10.1175/2010BAMS2853.1, URL http://journals.ametsoc.org/doi/10.1175/2010BAMS2853.1, 2010.
  * Brenowitz and Bretherton (2018) Brenowitz, N. D. and Bretherton, C. S.: Prognostic Validation of a Neural Network Unified Physics Parameterization, Geophysical Research Letters, 45, 6289–6298, 10.1029/2018GL078510, URL http://doi.wiley.com/10.1029/2018GL078510, 2018.
  * Brenowitz and Bretherton (2019) Brenowitz, N. D. and Bretherton, C. S.: Spatially Extended Tests of a Neural Network Parametrization Trained by Coarse-graining, Journal of Advances in Modeling Earth Systems, p. 2019MS001711, 10.1029/2019MS001711, URL https://onlinelibrary.wiley.com/doi/abs/10.1029/2019MS001711, 2019\.
  * Chevallier et al. (1998) Chevallier, F., Chéruy, F., Scott, N. A., and Chédin, A.: A Neural Network Approach for a Fast and Accurate Computation of a Longwave Radiative Budget, Journal of Applied Meteorology, 37, 1385–1397, 10.1175/1520-0450(1998)037<1385:ANNAFA>2.0.CO;2, URL http://journals.ametsoc.org/doi/abs/10.1175/1520-0450%281998%29037%3C1385%3AANNAFA%3E2.0.CO%3B2, 1998\.
  * Clevert et al. (2015) Clevert, D.-A., Unterthiner, T., and Hochreiter, S.: Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs), URL http://arxiv.org/abs/1511.07289, 2015.
  * Cohen et al. (2018) Cohen, T. S., Geiger, M., Köhler, J., and Welling, M.: Spherical CNNs, in: 6th International Conference on Learning Representations, ICLR 2018 - Conference Track Proceedings, International Conference on Learning Representations, ICLR, 2018.
  * Dueben and Bauer (2018) Dueben, P. D. and Bauer, P.: Challenges and design choices for global weather and climate models based on machine learning, Geosci. Model Dev., 10.5194/gmd-2018-148, URL https://www.geosci-model-dev-discuss.net/gmd-2018-148/gmd-2018-148.pdf, 2018\.
  * Ebert-Uphoff et al. (2017) Ebert-Uphoff, I., Thompson, D. R., Demir, I., Gel, Y. R., Hill, M. C., Karpatne, A., Guereque, M., Kumar, V., Cabral-Cano, E., and Smyth, P.: A VISION FOR THE DEVELOPMENT OF BENCHMARKS TO BRIDGE GEOSCIENCE AND DATA SCIENCE, in: 7th International Workshop on Climate Informatics, URL https://is-geo.org/, 2017.
  * Eyring et al. (2016) Eyring, V., Bony, S., Meehl, G. A., Senior, C. A., Stevens, B., Stouffer, R. J., and Taylor, K. E.: Overview of the Coupled Model Intercomparison Project Phase 6 (CMIP6) experimental design and organization, Geoscientific Model Development, 9, 1937–1958, 10.5194/gmd-9-1937-2016, URL https://www.geosci-model-dev.net/9/1937/2016/, 2016.
  * Gagne et al. (2014) Gagne, D. J., McGovern, A., Xue, M., II, D. J. G., McGovern, A., and Xue, M.: Machine Learning Enhancement of Storm-Scale Ensemble Probabilistic Quantitative Precipitation Forecasts, Weather and Forecasting, 29, 1024–1043, 10.1175/WAF-D-13-00108.1, URL http://journals.ametsoc.org/doi/abs/10.1175/WAF-D-13-00108.1, 2014\.
  * Gneiting and Katzfuss (2014) Gneiting, T. and Katzfuss, M.: Probabilistic Forecasting, Annual Review of Statistics and Its Application, 1, 125–151, 10.1146/annurev-statistics-062713-085831, URL http://www.annualreviews.org/doi/10.1146/annurev-statistics-062713-085831, 2014\.
  * Gneiting and Raftery (2007) Gneiting, T. and Raftery, A. E.: Strictly Proper Scoring Rules, Prediction, and Estimation, Journal of the American Statistical Association, 102, 359–378, 10.1198/016214506000001437, URL http://amstat.tandfonline.com/doi/abs/10.1198/016214506000001437, 2007\.
  * Gneiting et al. (2005) Gneiting, T., Raftery, A. E., Westveld, A. H., and Goldman, T.: Calibrated Probabilistic Forecasting Using Ensemble Model Output Statistics and Minimum CRPS Estimation, Monthly Weather Review, 133, 1098–1118, 10.1175/MWR2904.1, 2005.
  * Goodfellow et al. (2014) Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., and Bengio, Y.: Generative adversarial nets, in: Advances in neural information processing systems, pp. 2672–2680, 2014.
  * Graves (2013) Graves, A.: Generating Sequences With Recurrent Neural Networks, URL http://arxiv.org/abs/1308.0850, 2013.
  * Grönquist et al. (2020) Grönquist, P., Yao, C., Ben-Nun, T., Dryden, N., Dueben, P., Li, S., and Hoefler, T.: Deep Learning for Post-Processing Ensemble Weather Forecasts, URL http://arxiv.org/abs/2005.08748, 2020.
  * Ham et al. (2019) Ham, Y. G., Kim, J. H., and Luo, J. J.: Deep learning for multi-year ENSO forecasts, 10.1038/s41586-019-1559-7, 2019.
  * Hamill and Whitaker (2006) Hamill, T. M. and Whitaker, J. S.: Probabilistic Quantitative Precipitation Forecasts Based on Reforecast Analogs: Theory and Application, Monthly Weather Review, 134, 3209–3229, 10.1175/MWR3237.1, URL http://journals.ametsoc.org/doi/10.1175/MWR3237.1, 2006.
  * He et al. (2015) He, K., Zhang, X., Ren, S., and Sun, J.: Deep Residual Learning for Image Recognition, URL http://arxiv.org/abs/1512.03385, 2015.
  * Hersbach et al. (2020) Hersbach, H., Bell, B., Berrisford, P., Hirahara, S., Horányi, A., Muñoz-Sabater, J., Nicolas, J., Peubey, C., Radu, R., Schepers, D., Simmons, A., Soci, C., Abdalla, S., Abellan, X., Balsamo, G., Bechtold, P., Biavati, G., Bidlot, J., Bonavita, M., Chiara, G., Dahlgren, P., Dee, D., Diamantakis, M., Dragani, R., Flemming, J., Forbes, R., Fuentes, M., Geer, A., Haimberger, L., Healy, S., Hogan, R. J., Hólm, E., Janisková, M., Keeley, S., Laloyaux, P., Lopez, P., Lupu, C., Radnoti, G., Rosnay, P., Rozum, I., Vamborg, F., Villaume, S., and Thépaut, J.: The ERA5 Global Reanalysis, Quarterly Journal of the Royal Meteorological Society, p. qj.3803, 10.1002/qj.3803, URL https://onlinelibrary.wiley.com/doi/abs/10.1002/qj.3803, 2020\.
  * Hochreiter and Schmidhuber (1997) Hochreiter, S. and Schmidhuber, J.: Long short-term memory., Neural computation, 9, 1735–80, URL http://www.ncbi.nlm.nih.gov/pubmed/9377276, 1997.
  * Holton (2004) Holton, J. R.: An Introduction to Dynamic Meteorology, vol. 88, 10.1119/1.1987371, 2004.
  * Hoskins et al. (1985) Hoskins, B. J., McIntyre, M. E., and Robertson, A. W.: On the use and significance of isentropic potential vorticity maps, Quarterly Journal of the Royal Meteorological Society, 111, 877–946, 10.1002/qj.49711147002, URL http://doi.wiley.com/10.1002/qj.49711147002, 1985.
  * Huang et al. (2016) Huang, G., Liu, Z., van der Maaten, L., and Weinberger, K. Q.: Densely Connected Convolutional Networks, URL http://arxiv.org/abs/1608.06993, 2016.
  * Isola et al. (2016) Isola, P., Zhu, J.-Y., Zhou, T., and Efros, A. A.: Image-to-Image Translation with Conditional Adversarial Networks, URL https://arxiv.org/pdf/1611.07004v1.pdf, 2016.
  * Jiang et al. (2019) Jiang, C. M., Huang, J., Kashinath, K., Prabhat, Marcus, P., and Niessner, M.: Spherical CNNs on Unstructured Grids, URL http://arxiv.org/abs/1901.02039, 2019.
  * Kaji and Kida (2019) Kaji, S. and Kida, S.: Overview of image-to-image translation by use of deep neural networks: denoising, super-resolution, modality conversion, and reconstruction in medical imaging, URL http://arxiv.org/abs/1905.08603, 2019.
  * Kingma and Ba (2014) Kingma, D. P. and Ba, J.: Adam: A Method for Stochastic Optimization, arXiv, 1412.6980, URL http://arxiv.org/abs/1412.6980, 2014.
  * Koster and Rahmann (2012) Koster, J. and Rahmann, S.: Snakemake–a scalable bioinformatics workflow engine, Bioinformatics, 28, 2520–2522, 10.1093/bioinformatics/bts480, URL https://academic.oup.com/bioinformatics/article-lookup/doi/10.1093/bioinformatics/bts480, 2012\.
  * Krasnopolsky et al. (2005) Krasnopolsky, V. M., Fox-Rabinovitz, M. S., and Chalikov, D. V.: New Approach to Calculation of Atmospheric Model Physics: Accurate and Fast Neural Network Emulation of Longwave Radiation in a Climate Model, Monthly Weather Review, 133, 1370–1383, 10.1175/MWR2923.1, URL http://journals.ametsoc.org/doi/abs/10.1175/MWR2923.1, 2005.
  * Krogh and Hertz (1992) Krogh, A. and Hertz, J. A.: A Simple Weight Decay Can Improve Generalization, in: Advances in Neural Information Processing Systems 4, edited by Moody, J. E., Hanson, S. J., and Lippmann, R. P., pp. 950–957, Morgan-Kaufmann, URL http://papers.nips.cc/paper/563-a-simple-weight-decay-can-improve-generalization.pdf, 1992\.
  * Kurth et al. (2018) Kurth, T., Treichler, S., Romero, J., Mudigonda, M., Luehr, N., Phillips, E., Mahesh, A., Matheson, M., Deslippe, J., Fatica, M., Prabhat, and Houston, M.: Exascale Deep Learning for Climate Analytics, URL http://arxiv.org/abs/1810.01993, 2018.
  * Lagerquist et al. (2017) Lagerquist, R., McGovern, A., and Smith, T.: Machine learning for real-time prediction of damaging straight-line convective wind, Weather and Forecasting, 32, 2175–2193, 10.1175/WAF-D-17-0038.1, 2017.
  * Lazo et al. (2009) Lazo, J. K., Morss, R. E., and Demuth, J. L.: 300 Billion Served: Sources, Perceptions, Uses, and Values of Weather Forecasts, Bulletin of the American Meteorological Society, 90, 785–798, 10.1175/2008BAMS2604.1, 2009.
  * LeCun et al. (1998) LeCun, Y., Bottou, L., Bengio, Y., and Haffner, P.: Gradient-based learning applied to document recognition, Proceedings of the IEEE, 86, 2278–2323, 10.1109/5.726791, 1998.
  * LeCun et al. (2015) LeCun, Y., Bengio, Y., and Hinton, G.: Deep learning, Nature, 521, 436–444, 10.1038/nature14539, URL http://www.nature.com/articles/nature14539, 2015.
  * Leutbecher and Palmer (2008) Leutbecher, M. and Palmer, T.: Ensemble forecasting, Journal of Computational Physics, 227, 3515–3539, 10.1016/j.jcp.2007.02.014, URL http://dl.acm.org/citation.cfm?id=1347465.1347769, 2008.
  * McGovern et al. (2017) McGovern, A., Elmore, K. L., Gagne, D. J., Haupt, S. E., Karstens, C. D., Lagerquist, R., Smith, T., Williams, J. K., McGovern, A., Elmore, K. L., II, D. J. G., Haupt, S. E., Karstens, C. D., Lagerquist, R., Smith, T., and Williams, J. K.: Using Artificial Intelligence to Improve Real-Time Decision-Making for High-Impact Weather, Bulletin of the American Meteorological Society, 98, 2073–2090, 10.1175/BAMS-D-16-0123.1, URL http://journals.ametsoc.org/doi/10.1175/BAMS-D-16-0123.1, 2017\.
  * McGovern et al. (2019) McGovern, A., Lagerquist, R., Gagne, D. J., Jergensen, G. E., Elmore, K. L., Homeyer, C. R., and Smith, T.: Making the black box more transparent: Understanding the physical implications of machine learning, Bulletin of the American Meteorological Society, 10.1175/bams-d-18-0195.1, 2019.
  * Mirza and Osindero (2014) Mirza, M. and Osindero, S.: Conditional Generative Adversarial Nets, arXiv, p. 1411.1784, URL http://arxiv.org/abs/1411.1784, 2014.
  * Morton et al. (2018) Morton, J., Witherden, F. D., Jameson, A., and Kochenderfer, M. J.: Deep Dynamical Modeling and Control of Unsteady Fluid Flows, URL http://arxiv.org/abs/1805.07472, 2018.
  * (47) NCAR: The Climate Data Guide: Common Spectral Model Grid Resolutions, URL https://climatedataguide.ucar.edu/climate-model-evaluation/common-spectral-model-grid-resolutions.
  * Oord et al. (2016) Oord, A. v. d., Dieleman, S., Zen, H., Simonyan, K., Vinyals, O., Graves, A., Kalchbrenner, N., Senior, A., and Kavukcuoglu, K.: WaveNet: A Generative Model for Raw Audio, URL http://arxiv.org/abs/1609.03499, 2016.
  * Palmer (2019) Palmer, T.: The ECMWF Ensemble prediction system: Looking back (more than) 25 years and projecting forward 25 years, Quarterly Journal of the Royal Meteorological Society, 145, 12–24, 10.1002/qj.3383, URL https://onlinelibrary.wiley.com/doi/abs/10.1002/qj.3383, 2019\.
  * Perraudin et al. (2019) Perraudin, N., Defferrard, M., Kacprzak, T., and Sgier, R.: DeepSphere: Efficient spherical convolutional neural network with HEALPix sampling for cosmological applications, Astronomy and Computing, 27, 130–146, 10.1016/j.ascom.2019.03.004, 2019.
  * Rasp and Lerch (2018) Rasp, S. and Lerch, S.: Neural Networks for Postprocessing Ensemble Weather Forecasts, Monthly Weather Review, 146, 3885–3900, 10.1175/MWR-D-18-0187.1, URL http://journals.ametsoc.org/doi/10.1175/MWR-D-18-0187.1, 2018\.
  * Rasp et al. (2018) Rasp, S., Pritchard, M. S., and Gentine, P.: Deep learning to represent subgrid processes in climate models., Proceedings of the National Academy of Sciences of the United States of America, 115, 9684–9689, 10.1073/pnas.1810286115, URL http://www.ncbi.nlm.nih.gov/pubmed/30190437http://www.pubmedcentral.nih.gov/articlerender.fcgi?artid=PMC6166853, 2018.
  * Rasp et al. (2020) Rasp, S., Dueben, P. D., Scher, S., Weyn, J. A., Mouatadid, S., and Thuerey, N.: WeatherBench: A benchmark dataset for data-driven weather forecasting, 10.14459/2019mp1524895, URL https://mediatum.ub.tum.de/1524895, 2020.
  * Reichstein et al. (2019) Reichstein, M., Camps-Valls, G., Stevens, B., Jung, M., Denzler, J., Carvalhais, N., and Prabhat: Deep learning and process understanding for data-driven Earth system science, Nature, 566, 195–204, 10.1038/s41586-019-0912-1, URL http://www.nature.com/articles/s41586-019-0912-1, 2019.
  * Ronneberger et al. (2015) Ronneberger, O., Fischer, P., and Brox, T.: U-Net: Convolutional Networks for Biomedical Image Segmentation, URL http://arxiv.org/abs/1505.04597, 2015.
  * Russakovsky et al. (2015) Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge, International Journal of Computer Vision, 115, 211–252, 10.1007/s11263-015-0816-y, 2015.
  * Scher (2018) Scher, S.: Toward Data-Driven Weather and Climate Forecasting: Approximating a Simple General Circulation Model With Deep Learning, Geophysical Research Letters, 45, 616–12, 10.1029/2018GL080704, URL https://onlinelibrary.wiley.com/doi/abs/10.1029/2018GL080704, 2018\.
  * Scher and Messori (2019a) Scher, S. and Messori, G.: Generalization properties of neural networks trained on Lorenzsystems, Nonlinear Processes in Geophysics Discussions, pp. 1–19, 10.5194/npg-2019-23, URL https://www.nonlin-processes-geophys-discuss.net/npg-2019-23/, 2019a.
  * Scher and Messori (2019b) Scher, S. and Messori, G.: Weather and climate forecasting with neural networks: using general circulation models (GCMs) with different complexity as a study ground, Geoscientific Model Development, 12, 2797–2809, 10.5194/gmd-12-2797-2019, URL https://www.geosci-model-dev.net/12/2797/2019/, 2019b.
  * Shi et al. (2015) Shi, X., Chen, Z., Wang, H., Yeung, D.-Y., Wong, W.-k., and Woo, W.-c.: Convolutional LSTM Network: A Machine Learning Approach for Precipitation Nowcasting, URL http://arxiv.org/abs/1506.04214, 2015.
  * Shi et al. (2017) Shi, X., Gao, Z., Lausen, L., Wang, H., Yeung, D.-Y., Wong, W.-k., and Woo, W.-c.: Deep Learning for Precipitation Nowcasting: A Benchmark and A New Model, URL http://arxiv.org/abs/1706.03458, 2017.
  * Srivastava et al. (2014) Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R.: Dropout: A Simple Way to Prevent Neural Networks from Overfitting, Journal of Machine Learning Research, 15, 1929–1958, URL http://www.jmlr.org/papers/volume15/srivastava14a/srivastava14a.pdf?utm_content=buffer79b43&utm_medium=social&utm_source=twitter.com&utm_campaign=buffer, 2014\.
  * Stocker et al. (2013) Stocker, T. F., Qin, D., Plattner, G.-K., Tignor, M. M., Allen, S. K., Boschung, J., Nauels, A., Xia, Y., Bex, V., Midgley, P. M., Alexander, L. V., Allen, S. K., Bindoff, N. L., Breon, F.-M., Church, J. A., Cubasch, U., Emori, S., Forster, P., Friedlingstein, P., Gillett, N., Gregory, J. M., Hartmann, D. L., Jansen, E., Kirtman, B., Knutti, R., Kumar Kanikicharla, K., Lemke, P., Marotzke, J., Masson-Delmotte, V., Meehl, G. A., Mokhov, I. I., Piao, S., Plattner, G.-K., Dahe, Q., Ramaswamy, V., Randall, D., Rhein, M., Rojas, M., Sabine, C., Shindell, D., Stocker, T. F., Talley, L. D., Vaughan, D. G., Xie, S.-P., Allen, M. R., Boucher, O., Chambers, D., Hesselbjerg Christensen, J., Ciais, P., Clark, P. U., Collins, M., Comiso, J. C., Vasconcellos de Menezes, V., Feely, R. A., Fichefet, T., Fiore, A. M., Flato, G., Fuglestvedt, J., Hegerl, G., Hezel, P. J., Johnson, G. C., Kaser, G., Kattsov, V., Kennedy, J., Klein Tank, A. M., Le Quere, C., Myhre, G., Osborn, T., Payne, A. J., Perlwitz, J., Power, S., Prather, M., Rintoul, S. R., Rogelj, J., Rusticucci, M., Schulz, M., Sedlacek, J., Stott, P. A., Sutton, R., Thorne, P. W., and Wuebbles, D.: Climate Change 2013. The Physical Science Basis. Working Group I Contribution to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change, URL https://inis.iaea.org/Search/search.aspx?orig_q=RN:45042273, 2013\.
  * Sutskever et al. (2014) Sutskever, I., Vinyals, O., and Le, Q. V.: Sequence to Sequence Learning with Neural Networks, URL http://arxiv.org/abs/1409.3215, 2014.
  * Taillardat et al. (2016) Taillardat, M., Mestre, O., Zamo, M., and Naveau, P.: Calibrated Ensemble Forecasts using Quantile Regression Forests and Ensemble Model Output Statistics., Monthly Weather Review, p. 160301131220006, 10.1175/MWR-D-15-0260.1, URL http://journals.ametsoc.org/doi/abs/10.1175/MWR-D-15-0260.1?af=R, 2016\.
  * Toms et al. (2019) Toms, B. A., Barnes, E. A., and Ebert-Uphoff, I.: Physically Interpretable Neural Networks for the Geosciences: Applications to Earth System Variability, URL http://arxiv.org/abs/1912.01752, 2019.
  * Vogel et al. (2018) Vogel, P., Knippertz, P., Fink, A. H., Schlueter, A., and Gneiting, T.: Skill of global raw and postprocessed ensemble predictions of rainfall over Northern Tropical Africa, Weather and Forecasting, 33, 369–388, 10.1175/WAF-D-17-0127.1, 2018.
  * Weyn et al. (2019) Weyn, J. A., Durran, D. R., and Caruana, R.: Can machines learn to predict weather? Using deep learning to predict gridded 500-hPa geopotential height from historical weather data, Journal of Advances in Modeling Earth Systems, p. 2019MS001705, 10.1029/2019MS001705, URL https://onlinelibrary.wiley.com/doi/abs/10.1029/2019MS001705, 2019\.
  * Wilks (2006) Wilks, D. S.: Statistical Methods in the Atmospheric Sciences, Elsevier, URL http://cds.cern.ch/record/992087, 2006.
  * Xie et al. (2017) Xie, D., Xiong, J., and Pu, S.: All You Need is Beyond a Good Init: Exploring Better Solution for Training Extremely Deep Convolutional Neural Networks with Orthonormality and Modulation, URL http://arxiv.org/abs/1703.01827, 2017.
  * Xie et al. (2018) Xie, Y., Franz, E., and Chu, M.: tempoGAN: A Temporally Coherent, Volumetric GAN for Super-resolution Fluid Flow method using a variety of complex inputs and applications in two and three dimensions, ACM Trans. Graph, 37, 15, 10.1145/3197517.3201304, URL https://doi.org/10.1145/3197517.3201304, 2018.
  * Xu et al. (2019) Xu, Z., Bi, S., Sunkavalli, K., Hadap, S., Su, H., and Ramamoorthi, R.: Deep view synthesis from sparse photometric images, ACM Transactions on Graphics, 38, 1–13, 10.1145/3306346.3323007, URL http://dl.acm.org/citation.cfm?doid=3306346.3323007, 2019.
  * Yuval and O’Gorman (2020) Yuval, J. and O’Gorman, P. A.: Use of machine learning to improve simulations of climate, URL http://arxiv.org/abs/2001.03151, 2020.
  * Zhuang (2019) Zhuang, J.: xESMF: v0.2.1, 10.5281/ZENODO.3475638, URL https://xesmf.readthedocs.io/, 2019.