EUPPBench datasets

The EUPPBench datasets are available on a small portion of Europe stored in Zarr format for an easy access allowing for slicing. The forecasts and observations datasets are already paired together, providing analysis-ready data for postprocessing benchmarking purposes.

Datasets description

There are two main datasets:

1 - Gridded Data

A forecasts and observations dataset on a regular latitude-longitude grid.

../_images/gridded_data_EUPP.jpg — In blue, the EUPPBench dataset domain inside the Base datasets over Europe’s domain.

The gridded EUPPBench postprocessing benchmark dataset contains ECMWF ensemble and deterministic forecasts over a small domain in Europe, from 45.75° to 53.5° in latitude, and from 2.5° to 10.5° in longitude, and covers the years 2017-2018.
It also contains the corresponding ERA5 reanalysis for the purpose of providing observations for the benchmark.
For some dates, it contains also reforecasts that covers 20 years of past forecasts recomputed with the most recent model version at the given date.
All the forecasts and reforecasts provided are the noon ECMWF runs.
The ensemble forecasts and reforecasts also contain by default the control run (the 0-th member).
The gridded data resolution is 0.25° x 0.25° which corresponds roughly to 25 kilometers.
Forecasts and reforecasts are 6-hourly, and include the analysis at 00Z.

There are 8 gridded sub-datasets:

1.1 - Extreme Forecast Index

All the Extreme Forecast Index (EFI) variables can be obtained for each forecast date.

It includes:

Parameter name	ECMWF key	Remarks
2 metre temperature efi	2ti
10 metre wind speed efi	10wsi
10 metre wind gust efi	10fgi
cape efi	capei
cape shear efi	capesi
Maximum temperature at 2m efi	mx2ti
Minimum temperature at 2m efi	mn2ti
Snowfall efi	sfi
Total precipitation efi	tpi

The EFI are available for the model step ranges (in hours) 0-24, 24-48, 48-72, 72-96, 96-120.

Usage: The EFI variables can be retrieved by calling

import climetlab as cml
ds = cml.load_dataset('EUPPBench-training-data-gridded-forecasts-efi')
ds.to_xarray()

Alternatively, one can use the Intake catalogue

import euppbench_datasets
cat = euppbench_datasets.open_catalog()
ds = cat.euppbench.training_data.gridded.EUPPBench_forecasts_efi.to_dask()

Example:

Note

By definition, observations are not available for Extreme Forecast Indices (EFI).

1.2 - Surface variable forecasts

The surface variables can be obtained for each forecast date, both for the ensemble (51 members) and deterministic runs.

It includes:

Parameter name	ECMWF key	Remarks
2 metre temperature	2t/t2m
10 metre U wind component	10u
10 metre V wind component	10v
Total cloud cover	tcc
100 metre U wind component	100u
100 metre V wind component	100v
Convective available potential energy	cape
Soil temperature level 1	stl1
Total column water	tcw
Total column water vapour	tcwv
Volumetric soil water layer 1	swvl1
Snow depth	sd
Convective inhibition	cin	Observations not available
Visibility	vis	Observations not available

Some missing observations will become available later.

Usage: The surface variables forecasts can be retrieved by calling

import climetlab as cml
ds = cml.load_dataset('EUPPBench-training-data-gridded-forecasts-surface', kind)
ds.to_xarray()

where the kind argument allows to select the deterministic or ensemble forecasts, by setting it to 'highres' or 'ensemble'.

Alternatively, one can use the Intake catalogue

import euppbench_datasets
cat = euppbench_datasets.open_catalog()
# Fetching the ensemble forecasts
ds_ens = cat.euppbench.training_data.gridded.EUPPBench_ensemble_forecasts_surface.to_dask()
# Fetching the deterministic (highres) forecasts
ds_hr = cat.euppbench.training_data.gridded.EUPPBench_highres_forecasts_surface.to_dask()
# Fetching the corresponding observations
ds_obs = cat.euppbench.training_data.gridded.EUPPBench_forecasts_observations_surface.to_dask()

Example:

1.3 - Pressure level variable forecasts

The variables on pressure level can be obtained for each forecast date, both for the ensemble (51 members) and deterministic runs.

It includes:

Parameter name	Level	ECMWF key
Temperature	850	t
U component of wind	700	u
V component of wind	700	v
Geopotential	500	z
Specific humidity	700	q
Relative humidity	850	r

Usage: The pressure level variables forecasts can be retrieved by calling

import climetlab as cml
ds = cml.load_dataset('eumetnet-postprocessing-benchmark-EUPP-training-data-gridded-forecasts-pressure', level, kind)
ds.to_xarray()

where the level argument is the pressure level, as a string or an integer. The kind argument allows to select the deterministic or ensemble forecasts, by setting it to 'highres' or 'ensemble'.

Alternatively, one can use the Intake catalogue, for example for the 500 hPa level:

import euppbench_datasets
cat = euppbench_datasets.open_catalog()
# Fetching the ensemble forecasts
ds_ens = cat.euppbench.training_data.gridded.EUPPBench_ensemble_forecasts_pressure_500.to_dask()
# Fetching the deterministic (highres) forecasts
ds_hr = cat.euppbench.training_data.gridded.EUPPBench_highres_forecasts_pressure_500.to_dask()
# Fetching the corresponding observations
ds_obs = cat.euppbench.training_data.gridded.EUPPBench_forecasts_observations_pressure_500.to_dask()

but the other levels can be fetched in the same way, by replacing the 500 in the calls.

Example:

1.4 - Processed surface variable forecasts

Processed surface variables can be obtained for each forecast date, both for the ensemble (51 members) and deterministic runs. A processed variable is either accumulated, averaged or filtered.

It includes:

Parameter name	ECMWF key	Remarks
Total precipitation	tp6
Surface sensible heat flux	sshf6
Surface latent heat flux	slhf6
Surface net solar radiation	ssr6
Surface net thermal radiation	str6
Convective precipitation	cp6
Maximum temperature at 2 metres	mx2t6
Minimum temperature at 2 metres	mn2t6
Surface solar radiation downwards	ssrd6
Surface thermal radiation downwards	strd6
10 metre wind gust	10fg6

All these variables are accumulated or filtered over the last 6 hours preceding a given forecast timestamp. As a consequence, a `6’ was added to the ECMWF key to denote this.

Usage: The processed surface variables forecasts can be retrieved by calling

import climetlab as cml
ds = cml.load_dataset('EUPPBench-training-data-gridded-forecasts-surface-processed', kind)
ds.to_xarray()

where the kind argument allows to select the deterministic or ensemble forecasts, by setting it to 'highres' or 'ensemble'.

Alternatively, one can use the Intake catalogue

import euppbench_datasets
cat = euppbench_datasets.open_catalog()
# Fetching the ensemble forecasts
ds_ens = cat.euppbench.training_data.gridded.EUPPBench_ensemble_forecasts_surface_processed.to_dask()
# Fetching the deterministic (highres) forecasts
ds_hr = cat.euppbench.training_data.gridded.EUPPBench_highres_forecasts_surface_processed.to_dask()
# Fetching the corresponding observations
ds_obs = cat.euppbench.training_data.gridded.EUPPBench_forecasts_observations_surface_processed.to_dask()

Example:

1.8 - Static fields

Various static fields associated to the forecast grid can be obtained, with the purpose of serving as predictors for the postprocessing.

Note

For consistency with the rest of the dataset, we use the ECMWF parameters name, terminology and units here. However, please note that - except for the Surface Geopotential - the fields provided are from other non-ECMWF data sources evaluated at grid points. Currently, the main data source being used is the Copernicus Land Monitoring Service.

It includes:

Parameter name	ECMWF key	Remarks
Land use	landu	Extracted from the CORINE 2018 dataset. Values and associated land type differ from the ECMWF one. Please look at the “legend” entry in the metadata for more details.
Model terrain height	mterh	Extracted from the EU-DEMv1.1 data elevation model dataset.
Surface Geopotential	z	The model orography can be obtained by dividing the surface geopotential by g=9.80665 ms \({}^{-2}\).

Usage: The static fields can be retrieved by calling

import climetlab as cml
ds = cml.load_dataset('EUPPBench-training-data-gridded-static-fields', parameter)
ds.to_xarray()

where the parameter argument is a string with one of the ECMWF keys described above. It is only possible to download one static field per call.

Alternatively, one can use the Intake catalogue

import euppbench_datasets
cat = euppbench_datasets.open_catalog()
# Fetching the land usage field
ds = cat.euppbench.training_data.gridded.EUPPBench_land_use.to_dask()

The other static field are also available in the same way.

Example:

4 - Explanation of the metadata

For all data, attributes specifying the sources and the license are always present. Depending on the kind of dataset, dimensions and information are embedded in the data as follow:

Gridded data

The following metadata are available in the gridded forecast, reforecast and observation data:

Metadata	Description
latitude	Latitude of the grid points.
longitude	Longitude of the grid points.
depthBelowLandLayer	Layer below the surface (valid for some variables only, here there is only the upper surface level).
number	Number of the ensemble member. The 0-th member is the control run. Also present in observation for compatibility reasons, but set to 0.
time	Forecast or reforecast date (reforecasts are only issued on Mondays and Thursdays).
year	Dimension to identify the year in the past, year=1 means a forecast valid 20 years ago at the reforecast day and month, year=20 means a forecast valid one year before the reforecast date. Only valid for reforecasts.
step	Step of the forecast (the lead time).
surface	Layer of the variable considered (here there is just one, at the surface).
isobaricInhPa	Pressure level in hectopascal (or millibar).
valid_time	Actual time and date of the corresponding forecast data.

Note

Bold metadata denotes dimensions indexing the datasets.

Stations data

For station forecast and reforecast data, the following metadata are available:

Metadata	Description
station_latitude	Latitude of the station.
station_longitude	Longitude of the station.
station_altitude	Altitude of the station (in meter).
station_id	Unique identifier of the station.
depthBelowLandLayer	Layer below the surface (valid for some variables only, here there is only the upper surface level).
number	Number of the ensemble member. The 0-th member is the control run. Also present in observation for compatibility reasons, but set to 0.
time	Forecast or reforecast date (reforecasts are only issued on Mondays and Thursdays).
year	Dimension to identify the year in the past, year=1 means a forecast valid 20 years ago at the reforecast day and month, year=20 means a forecast valid one year before the reforecast date. Only valid for reforecasts.
step	Step of the forecast (the lead time).
surface	Layer of the variable considered (here there is just one, at the surface).
isobaricInhPa	Pressure level in hectopascal (or millibar).
station_land_usage	Land usage at the station location, extracted from the CORINE 2018 dataset.
station_name	Name of the station.
model_latitude	Latitude of the model grid point.
model_longitude	Longitude of the model grid point.
model_altitude	True altitude (in meter) of the model grid point, extracted from the EU-DEMv1.1 data elevation model dataset.
model_orography	Surface height (in meter) in the model at the model grid point.
model_land_usage	Land usage at the model grid point, extracted from the CORINE 2018 dataset.
valid_time	Actual time and date of the corresponding forecast data.

Note

The metadata with `model’ in their name indicate properties of the model grid point the closest to the station location, and at which the forecasts corresponding to the station observations was extracted from the gridded dataset.

For the station observations, the following metadata are available:

Metadata	Description
altitude	Altitude of the station (in meter).
land_usage	Land usage at the station location, extracted from the CORINE 2018 dataset.
latitude	Latitude of the station.
longitude	Longitude of the station.
station_id	Unique identifier of the station.
station_name	Name of the station.
step	Step of the forecast (the lead time).
time	Forecast or reforecast date (reforecasts are only issued on Mondays and Thursdays).

5 - Major ECMWF model changes

In 2017 and 2018, there were 2 model changes of the ECMWF model on total:

Implementation date	Summary of changes	Resolution	Full IFS documentation
05-Jun-2018	Cycle 45r1	Unchanged	Cycle 45r1 full documentation
11-Jul-17	Cycle 43r3	Unchanged	Cycle 43r3 full documentation

Source: https://www.ecmwf.int/en/forecasts/documentation-and-support/changes-ecmwf-model

Tips & Tricks

Saving the data to a NetCDF file

This is particularly useful if one wants to reuse the data with another programming language. For example, if one has downloaded the observations shown in section 3 - Getting the observations corresponding to the (re)forecasts, one can save them to disk by using the xarray.Dataset.to_netcdf() functionality of the xarray Dataset:

import climetlab as cml
ds = cml.load_dataset('EUPPBench-training-data-stations-reforecasts-surface-processed', "austria")
obs = ds.get_observations_as_xarray()
obs.to_netcdf('austria_reforecasts.nc')

Finding the units of a given data

In general, we align with the units of the ECMWF data. You can find the particular units of a given data by clicking on the parameter’s name in the table above. For many variables, the units are also available in the metadata of the forecasts. For example, the following code snippet show how to retrieve the units of surface variable in the station dataset:

ds = cml.load_dataset('EUPPBench-training-data-stations-reforecasts-surface', "austria")
fcs = ds.to_xarray()
fcs.v100.units

'm s**-1'

The same remark applies equally to the data fetched via the Intake catalogue.

Data License

See the DATA_LICENSE file.

Station observations were provided by European National Meteorological Services within the framework of their open data policy, and are sourced in the metadata of the corresponding datasets.

Swiss station data are part of this dataset but are presently restricted. These station data may be obtained from IDAWEB at MeteoSwiss and we are not entitled to provide it online. Registration with IDAWEB can be initiated here. Please also read these information.

EUPPBench datasets

Datasets description

1 - Gridded Data

1.1 - Extreme Forecast Index

1.2 - Surface variable forecasts

1.3 - Pressure level variable forecasts

1.4 - Processed surface variable forecasts

1.5 - Surface variable reforecasts

1.6 - Pressure level variable reforecasts

1.7 - Processed surface variable reforecasts

1.8 - Static fields

2 - Stations Data

2.1 - Extreme Forecast Index

2.2 - Surface variable forecasts

2.3 - Pressure level variable forecasts

2.4 - Processed surface variable forecasts

2.5 - Surface variable reforecasts

2.6 - Pressure level variable reforecasts

2.7 - Processed surface variable reforecasts

3 - Getting the observations corresponding to the (re)forecasts

4 - Explanation of the metadata

Gridded data

Stations data

5 - Major ECMWF model changes

Tips & Tricks

Saving the data to a NetCDF file

Finding the units of a given data

Data License