# connector

## Introduction

The `connector` package provides a set of functions to connect to different data sources (such as databases and file systems) and read and write data from them using a consistent interface.

It is designed to be a generic and extensible package, so that new data sources can be added easily.

This vignette demonstrates how to use the `connector` package to connect to either a file system or a database to access different types of data.

## Connector configuration

The main function in this package is `connect()`. This function, based on a configuration file or a list, creates a `connectors` object with a `connector` for each of the specified data sources. The configuration file can be in list format, JSON, or YAML format.

The input list (or configuration file) must have the following structure:

- Only `metadata`, `env`, and `datasources` fields are allowed.
- All elements must be named.
- `datasources` is mandatory.
- `metadata` and `env` must each be a list of named character vectors of length 1.
- `datasources` must be a list of unnamed lists.
- Each datasource must have the named character element `name` and the named list element `backend`.
- For each backend, `type` must be provided.

## Working example

Here is an example anyone can run to see how the `connector` package works. We will use the configuration file provided below, which uses the file system as the connection type for ADaM and TFL data.

`_connector.yml:`

    metadata:
      adam_path: !expr file.path(getwd(), "adam")
      tfl_path: !expr file.path(getwd(), "tfl")

    datasources:
      - name: "adam"
        backend:
          type: "connector::connector_fs"
          path: "{metadata.adam_path}"
      - name: "tfl"
        backend:
          type: "connector::connector_fs"
          path: "{metadata.tfl_path}"

As you can see, the configuration file contains metadata about the paths to the directories where the data will be stored, and two data sources: `adam` and `tfl`, both using the `connector_fs` backend to connect to file system folders. Note that the paths to the directories are defined using metadata variables (e.g., `{metadata.adam_path}`), which allows you to easily change the paths in one place.

Now, let’s run the example:

The first step is to create the connections to the data sources.

    # Load data connections
    db <- connect()
    #> ────────────────────────────────────────────────────────────────────────────────
    #> Connection to:
    #> → adam
    #> • connector::connector_fs
    #> • /private/var/folders/kv/q2rqqp3s0s5f9rxn_854l2lm0000gp/T/RtmpZyw2fH/filefe513b5ee525/adam
    #> ────────────────────────────────────────────────────────────────────────────────
    #> Connection to:
    #> → tfl
    #> • connector::connector_fs
    #> • /private/var/folders/kv/q2rqqp3s0s5f9rxn_854l2lm0000gp/T/RtmpZyw2fH/filefe513b5ee525/tfl

Next, we manipulate the iris dataset and store it in the `adam` connector. This means we will create a subset of the iris dataset and save it as an RDS file in the `adam` directory.

    ## Iris data
    setosa <- iris |>
      filter(Species == "setosa")
    ## Store data
    db$adam |>
      write_cnt(setosa, "setosa.rds")

We can also create more complex summaries and store them in the same connector.

    mean_for_all_iris <- iris |>
      group_by(Species) |>
      summarise_all(list(mean, median, sd, min, max))

    db$adam |>
      write_cnt(mean_for_all_iris, "mean_iris.rds")

    ## List and load data
    db$adam |>
      list_content_cnt()
    #> [1] "mean_iris.rds" "setosa.rds"

We can also read back the data we just created and filter it further using the `read_cnt()` function.

    # Read and filter data
    setosa_filtered <- db$adam |>
      read_cnt("setosa") |>
      filter(Sepal.Length > 5)
    #> → Found one file: '/private/var/folders/kv/q2rqqp3s0s5f9rxn_854l2lm0000gp/T/RtmpZyw2fH/filefe513b5ee525/adam/setosa.rds'

Finally, we can create a plot with the `ggplot2` package and store it in the `tfl` connector.

    # Create a plot
    plot_setosa <- ggplot(setosa_filtered) +
      aes(x = Sepal.Length, y = Sepal.Width) +
      geom_point()

    ## Store data and plot objects
    db$tfl |>
      write_cnt(plot_setosa$data, "setosa_data.csv")
    db$tfl |>
      write_cnt(plot_setosa, "setosa_plot.rds")

    ## Store plot image
    tmp_file <- tempfile(fileext = ".png")
    ggsave(tmp_file, plot_setosa)
    #> Saving 3 x 3 in image
    db$tfl |>
      upload_cnt(tmp_file, "setosa_plot.png")

    # List all files in the TFL directory
    db$tfl |>
      list_content_cnt()
    #> [1] "setosa_data.csv" "setosa_plot.png" "setosa_plot.rds"