Author: Callum Waite & Shandiya Balasubramaniam
Date: 2023-10-13

# Temporal filtering

#### Callum Waite & Shandiya Balasubramaniam

#### 2023-10-13

Queries to the ALA will almost always require some form of temporal filtering. It is important to know how these types of data are stored in the ALA and how we can query them to obtain desired filters.

The ALA database possesses numerous date and time fields that relate to each observation. Here we provide descriptions of each of these fields and how they are best used to obtain specific queries. Ultimately, there are two ways users can filter temporal queries:

- filter using pre-existing/defined parameters, such as specific years or months

- filter within a bespoke date and/or time range

All temporal filtering is conducted using `galah_filter()`. All temporal fields described below can be queried for exact matches (`==`), greater/less than (`>`, `<`) or greater/less than or equal to (`<=`, `>=`). Queries for multiple fields or multiple queries of the same field can be combined in a single `galah_filter()` call to filter a time window.

### Year, Month and Day

The ALA contains in-built `year`, `month`, and `day` fields for every record. These are queried as numeric fields (i.e. July = `7`) and can be used for quick data exploration and filtering. These fields are most useful when the date limits of a query can be easily defined by year, month and/or day.

For instance, we can get monthly counts of amphibians from 2021 using the `year` and `month` fields.

    library(galah)
    library(tidyverse)
    library(gt)
    library(lubridate)

    galah_config(email = "your_email_here", verbose = FALSE)

    galah_call() |>
      filter(class == "Amphibia", year == 2021) |>
      group_by(month) |>
      count() |>
      collect()

    ## # A tibble: 12 × 2
    ##    month count
    ##    <chr> <int>
    ##  1 11    83810
    ##  2 10    38208
    ##  3 12    36578
    ##  4 9     27616
    ##  5 1     22352
    ##  6 8     18758
    ##  7 3     17964
    ##  8 2     16550
    ##  9 7      8945
    ## 10 4      7769
    ## 11 6      6961
    ## 12 5      5996

It is also important to observe that the outputted `month` column is of type `character` even though the values are numeric. This is the case for each of the `year`, `month` and `day` fields. However, they can be queried as either numeric or character values within `filter()`.

One limitation of using these fields for queries with `filter()` is their independence; they cannot be used to query within windows bounded by two dates because the day and month filters are applied universally. For instance, consider the native perennial Australian wildflower Chamaescilla corymbosa, whose known growth and flowering times are from August–October. We might be interested in the number of records for this species in the first week of spring (i.e. September) in each of the last 10 years. The following query does not provide all results between 1/9/2013 and 7/9/2023. Rather, it will only return results that fall within all 3 windows at once.

    galah_call() |>
      filter(species == "Chamaescilla corymbosa",
             year >= 2013,
             year <= 2023,
             month == 9,
             day >= 1,
             day <= 7) |>
      group_by(year) |>
      count() |>
      collect() |>
      arrange(year)

    ## # A tibble: 11 × 2
    ##    year  count
    ##    <chr> <int>
    ##  1 2013      8
    ##  2 2014     13
    ##  3 2015      8
    ##  4 2016      7
    ##  5 2017      1
    ##  6 2018      9
    ##  7 2019      6
    ##  8 2020     20
    ##  9 2021     36
    ## 10 2022     26
    ## 11 2023     54

### Occurrence dates

For a more bespoke way to query exact dates of records, users can use the `eventDate` field. This field contains the exact date and time information of records and enables specific time windows to be queried easily. The only caveat is that the time/date must be provided in a specific format to `filter()` for the query to work.

The required format of dates in `eventDate` is the ISO 8601 International Date Standard format. This requires dates and times to be of the form “YYYY-MM-DDTHH:MM:SSZ”. Note that the `T` in the middle should be the actual letter “T” to delimit the date and time components, while the “Z” officially denotes that the time should be queried as UTC (Greenwich Meridian) time. Timezones can be confusing at the best of times, however it is easiest to remember that all ALA records are recorded at the local time of their location, and all times are then treated as effectively being UTC times.

The upshot of this specific formatting is that, for instance, the time I am writing this paragraph, 4:26pm on the 2nd of August 2023, would be represented as `"2023-08-02T16:26:44Z"` in the ALA, even though officially my timezone is `"+0930"`.

Because `eventDate` specifies the time to seconds, it is recommended that greater or less than queries are used rather than exact matches. When used with `filter()`, we can easily identify how many records of the humpback whale (Megaptera novaeangliae) have occurred since the species was removed from the Australian threatened species list on 26/02/2022.

    galah_call() |>
      filter(species == "Megaptera novaeangliae",
             eventDate >= "2022-02-26T00:00:00Z") |>
      count() |>
      collect()

    ## # A tibble: 1 × 1
    ##   count
    ##   <int>
    ## 1  1240

It can be unintuitive to provide dates in this format. Luckily, it is very simple to convert standard R dates or {lubridate} dates into this format because they are already in the required “YYYY-MM-DD” form. If we took the above date (26/02/2022), it could be converted to this form using base R or lubridate as follows:

    humpback_date <- "26/02/2022"
    # Base R
    paste0(as.Date(humpback_date, format = "%d/%m/%Y"), "T00:00:00Z")

    ## [1] "2022-02-26T00:00:00Z"

    # lubridate
    paste0(dmy(humpback_date), "T00:00:00Z")

    ## [1] "2022-02-26T00:00:00Z"

After sending a query, any outputted `eventDate` values returned by a `galah` query will be of date class `"POSIXct"`.

### Upload dates

The other important date field present in the ALA pertains to the date that the record was provided to the ALA. This field is called `firstLoadedDate` and is formatted in exactly the same manner as `eventDate`.

Different data providers provide batches of records to the ALA at different intervals. iNaturalist Australia provide weekly uploads of data, while eBird provides yearly uploads. `firstLoadedDate` can be especially useful for finding new records to the ALA that have been provided since you last checked. For instance, we can use it to see how many observations of Sulphur-Crested Cockatoos recorded in the first week of 2023 were actually loaded into the ALA by the following week:

    # Total records of Cactua galerita in Jan 1-7
    galah_call() |>
      filter(species == "Cacatua galerita",
             eventDate >= "2023-01-07T00:00:00Z",
             eventDate < "2023-01-08T00:00:00Z") |>
      count() |>
      collect()

    ## # A tibble: 1 × 1
    ##   count
    ##   <int>
    ## 1   407

    # Records of Cactua galerita uploaded in Jan 1-14
    galah_call() |>
      filter(species == "Cacatua galerita",
             eventDate >= "2023-01-07T00:00:00Z",
             eventDate < "2023-01-08T00:00:00Z",
             firstLoadedDate < "2023-01-15T00:00:00Z") |>
      count() |>
      collect()

    ## # A tibble: 1 × 1
    ##   count
    ##   <int>
    ## 1     5

Note that no lower bound is required for `firstLoadedDate` because `eventDate` imposes that by proxy (records can’t be uploaded before they’ve occurred).