Retrieving Data

Retrieving Data#

Not all data dependencies are shipped with the git repository, since git is not suited for handling large changing files. Instead we use separate steps in the workflow (rules executed by snakemake) to download external data using the retrieve_<dataset> rules.

Data is generally retrieved in a version-controlled manner, enabling control over input data versions, reproducibility and consistency of modelling runs. The rules download data into subfolders in the data/ directory, following the structure data/{dataset}/{source}/{version}, e.g. data/jrc_idees/primary/March-2025-V1/. Which specific data version is retrieved can be controlled in the data configuration.

For Open-TYNDP runs, most datasets can also be retrieved from a dedicated Google Cloud Storage bucket instead of their original sources. See Data Sources in the SB documentation.

Below some specific retrieve_<dataset> rules are documented. For more information on the datasets retrieved, see the data sources and Data inventory section there in the documentation.

Rule retrieve_bidding_zones#

Rule retrieve_cutout#

See Weather Data.

Rule retrieve_electricity_demand_opsd#

This rule downloads hourly electric load data for each country from the OPSD platform.

Relevant Settings

None.

Outputs

  • data/electricity_demand_opsd_raw.csv

Rule retrieve_electricity_demand_entsoe#

This rule downloads hourly electric load data for each country from the ENTSOE Transparency Platform.

Relevant Settings

None.

Outputs

  • data/electricity_demand_entsoe_raw.csv

Rule retrieve_electricity_demand_neso#

This rule downloads hourly electric load data for the United Kingdom from the NESO Data Portal.

Relevant Settings

None.

Outputs

  • data/electricity_demand_neso_raw.csv

Rule retrieve_cost_data#

This rule downloads techno-economic assumptions from the technology-data repository.

Relevant Settings

costs:
    year:

See also

Documentation of the configuration file config/config.yaml at costs

Outputs

  • data/costs/primary/{version}/costs_{year}.csv

Rule retrieve_countries_centroids#

This rule downloads country centroid geometry data by Copyright (c) 2021 Gavin Rehkemper from https://cdn.jsdelivr.net/gh/gavinr/world-countries-centroids@v1.0.0/dist/countries.geojson.

Relevant Settings

None.

Outputs

  • data/countries_centroids.geojson

Contents