---
title: "CDS functionality"
author: "Reto Stauffer, Koen Hufkens"
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{CDS functionality}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r setup, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)

# load the library
library(ncdf4)
library(terra)
library(maps)

# grab demo nc files
ncfile <- list.files(system.file(package="ecmwfr"),"*.nc", recursive = TRUE, full.names = TRUE)
ncfile <- ncfile[grepl("cds.nc", ncfile)]
```

## Downloading Data Sets from Copernicus's Climate Data Store

[Copernicus.eu](https://www.copernicus.eu/en) provides a set of interesting data sets for research, education, and applied earth sciences on their [Climate Data Store](https://cds.climate.copernicus.eu) (CDS). Among the different data sets there is the latest [ECMWF](https://www.ecmwf.int) high-resolution reanalysis data set which replaces ERA Interim (which was ERA-4; version 4). The R package `ecmwfr` provides a convenient interface to download these data sets.

## Before Downloading Data

Before you will be able to download any data you need to get a free personal
account. 

* [Register yourself for CDS services](https://cds.climate.copernicus.eu/user/register)
* [Terms and conditions](https://cds.climate.copernicus.eu/disclaimer-privacy)

To retrieve data `ecmwfr` use `wf_set_key()`to add the login details to your local keyring (requires the _R_ package [`keyring`](https://CRAN.R-project.org/package=keyring)).

Once you are in possession of your personal user (namely your user ID and a secret key) `ecmwfr` allows to send requests to CDS and/or download the data.

## The Request Syntax

CDS data retrievals are based on a `list` object which specifies the data set to be downloaded.  These definitions are called `requests` (for those who are familiar with _mars_: these are basically _mars_ requests). A `request` defined the type of the `dataset`, the `variables` to be downloaded, the time period, output `format`, `target` location, a custom `area` extent, and other details.

The request syntax is available for a range of different CDS data sets. Check the [CDS Dataset](https://cds.climate.copernicus.eu/cdsapp#!/search?type=dataset) website to see a list of available datasets and to check whether API requests are allowed or not (go to _Download Data_, select some data, show request by clicking _Show API Request_ (red button, bottom of page).

The conversion from a python based string to the list format can be automated if you use the RStudio based Addin. By selecting and using Addin -> python to list (or 'Mars to list') you dynamically convert queries copied from either ECMWF or CDS based services.

![](https://user-images.githubusercontent.com/1354258/56429618-def12080-62c3-11e9-90e4-0954e10d06f3.gif)

### An ERA-5 example

```{r demo request, echo = TRUE}
# Specify the data set
request <- list(
  "dataset_short_name" = "reanalysis-era5-pressure-levels",
  "product_type"   = "reanalysis",
  "variable"       = "temperature",
  "pressure_level" = "850",
  "year"           = "2000",
  "month"          = "04",
  "day"            = "04",
  "time"           = "00:00",
  "area"           = "70/-20/30/60",
  "format"         = "netcdf",
  "target"         = "era5-demo.nc"
  )
```

This is the "demo" request and it specifies the following:

* `dataset`: downloading ERA-5 reanalysis on pressure level(s)
* `product_type`: (deterministic) reanalysis data
* `variable`/`pressure_level`: requesting temperature on 850 hectopascal
* `year`/`month`/`day`: April 4, 2000 (one day in this example)
* `time`: valid at 00:00 UTC (date/time always in UTC),
* `area`: custom subset covering northern Europe
* `format`: output format NetCDF
* `target`: local output file `era5-demo.nc`

The data set as specified above can be downloaded calling the
`wf_request` function:

```{r spatial-request, echo = TRUE, eval = FALSE}
# Start downloading the data, the path of the file
# will be returned as a variable (ncfile)
ncfile <- wf_request(
  user = "2088",
  request = request,   
  transfer = TRUE,  
  path = "~",
  verbose = FALSE
  )
```

Depending on the request (the amount of data you are asking for) the request function may tike a while! Please note: if you try to download larger amounts of data it is suggested to split the data sets, e.g., download year-by-year, or even month-by-month, if you are trying to download several varaiables/fields.

Once the retrieval has finished you should now be the owner of a NetCDF containing the requested information located in the current working directory, called `era5-demo.nc`. 

Quick check:

```{r spatial-plot, echo = TRUE, figure = TRUE, fig.width = 8, fig.height = 6}
# Open NetCDF file and plot the data
# (trap read error on mac - if gdal netcdf support is missing)
r <- try(terra::rast(ncfile))

if(!inherits(r, "try-error")) {
  terra::plot(r, main = "ERA-5 Reanalysis Demo (2m Temperature 850 hPa)")
  maps::map("world", add = TRUE)
}
```
