Author: Sebastian Krantz
Date: 2024-05-20

# collapse for tidyverse Users

#### Sebastian Krantz

#### 2024-05-20

collapse is a C/C++ based package for data transformation and statistical computing in R that aims to enable greater performance and statistical complexity in data manipulation tasks and offers a stable, class-agnostic, and lightweight API. It is part of the core fastverse, a suite of lightweight packages with similar objectives.

The tidyverse set of packages provides a rich, expressive, and consistent syntax for data manipulation in R centering on the tibble object and tidy data principles (each observation is a row, each variable is a column).

collapse fully supports the tibble object and provides many tidyverse-like functions for data manipulation. It can thus be used to write tidyverse-like data manipulation code that, thanks to low-level vectorization of many statistical operations and optimized R code, typically runs much faster than native tidyverse code (in addition to being much more lightweight in dependencies).

Its aim is not to create a faster tidyverse, i.e., it does not implements all aspects of the rich tidyverse grammar or changes to it1, and also takes inspiration from other leading data manipulation libraries to serve broad aims of performance, parsimony, complexity, and robustness in data manipulation for R.

## Namespace and Global Options

collapse data manipulation functions familiar to tidyverse users include `fselect`, `fgroup_by`, `fsummarise`, `fmutate`, `across`, `frename`, and `fcount`. Other functions like `fsubset`, `ftransform`, and `get_vars` are inspired by base R, while again other functions like `join`, `pivot`, `roworder`, `colorder`, `rowbind`, etc. are inspired by other data manipulation libraries such das data.table and polars.

By virtue of the f- prefixes, the collapse namespace has no conflicts with the tidyverse, and these functions can easily be substituted in a tidyverse workflow.

R users willing to replace the tidyverse have the additional option to mask functions and eliminate the prefixes with `set_collapse`. For example

    library(collapse)
    set_collapse(mask = "manip") # version >= 2.0.0

makes available functions `select`, `group_by`, `summarise`, `mutate`, `rename`, `count`, `subset`, and `transform` in the collapse namespace and detaches and re-attaches the package, such that the following code is executed by collapse:

    mtcars |>
      subset(mpg > 11) |>
      group_by(cyl, vs, am) |>
      summarise(across(c(mpg, carb, hp), mean),
                qsec_wt = weighted.mean(qsec, wt))
    #   cyl vs am      mpg     carb        hp  qsec_wt
    # 1   4  0  1 26.00000 2.000000  91.00000 16.70000
    # 2   4  1  0 22.90000 1.666667  84.66667 21.04028
    # 3   4  1  1 28.37143 1.428571  80.57143 18.75509
    # 4   6  0  1 20.56667 4.666667 131.66667 16.33306
    # 5   6  1  0 19.12500 2.500000 115.25000 19.21275
    # 6   8  0  0 15.98000 2.900000 191.00000 17.01239
    # 7   8  0  1 15.40000 6.000000 299.50000 14.55297

Note that the correct documentation still needs to be called with prefixes, i.e., `?fsubset`. See `?set_collapse` for further options to the package, which also includes optimization options such as `nthreads`, `na.rm`, `sort`, and `stable.algo`. Note also that if you use collapse’s namespace masking, you can use `fastverse::fastverse_conflicts()` to check for namespace conflicts with other packages.

## Using the Fast Statistical Functions

A key feature of collapse is that it not only provides functions for data manipulation, but also a full set of statistical functions and algorithms to speed up statistical calculations and perform more complex statistical operations (e.g. involving weights or time series data).

Notably among these, the Fast Statistical Functions is a consistent set of S3-generic statistical functions providing fully vectorized statistical operations in R.

Specifically, operations such as calculating the mean via the S3 generic `fmean()` function are vectorized across columns and groups and may also involve weights or transformations of the original data:

    fmean(mtcars$mpg)     # Vector
    # [1] 20.09062
    fmean(EuStockMarkets) # Matrix
    #      DAX      SMI      CAC     FTSE
    # 2530.657 3376.224 2227.828 3565.643
    fmean(mtcars)         # Data Frame
    #        mpg        cyl       disp         hp       drat         wt       qsec         vs         am
    #  20.090625   6.187500 230.721875 146.687500   3.596563   3.217250  17.848750   0.437500   0.406250
    #       gear       carb
    #   3.687500   2.812500

    fmean(mtcars$mpg, w = mtcars$wt)  # Weighted mean
    # [1] 18.54993
    fmean(mtcars$mpg, g = mtcars$cyl) # Grouped mean
    #        4        6        8
    # 26.66364 19.74286 15.10000
    fmean(mtcars$mpg, g = mtcars$cyl, w = mtcars$wt)   # Weighted group mean
    #        4        6        8
    # 25.93504 19.64578 14.80643
    fmean(mtcars[5:10], g = mtcars$cyl, w = mtcars$wt) # Of data frame
    #       drat       wt     qsec        vs        am     gear
    # 4 4.031264 2.414750 19.38044 0.9148868 0.6498031 4.047250
    # 6 3.569170 3.152060 18.12198 0.6212191 0.3787809 3.821036
    # 8 3.205658 4.133116 16.88529 0.0000000 0.1203808 3.240762
    fmean(mtcars$mpg, g = mtcars$cyl, w = mtcars$wt, TRA = "fill") # Replace data by weighted group mean
    #  [1] 19.64578 19.64578 25.93504 19.64578 14.80643 19.64578 14.80643 25.93504 25.93504 19.64578
    # [11] 19.64578 14.80643 14.80643 14.80643 14.80643 14.80643 14.80643 25.93504 25.93504 25.93504
    # [21] 25.93504 14.80643 14.80643 14.80643 14.80643 25.93504 25.93504 25.93504 14.80643 19.64578
    # [31] 14.80643 25.93504
    # etc...

The data manipulation functions of collapse are integrated with these Fast Statistical Functions to enable vectorized statistical operations. For example, the following code

    mtcars |>
      subset(mpg > 11) |>
      group_by(cyl, vs, am) |>
      summarise(across(c(mpg, carb, hp), fmean),
                qsec_wt = fmean(qsec, wt))
    #   cyl vs am      mpg     carb        hp  qsec_wt
    # 1   4  0  1 26.00000 2.000000  91.00000 16.70000
    # 2   4  1  0 22.90000 1.666667  84.66667 21.04028
    # 3   4  1  1 28.37143 1.428571  80.57143 18.75509
    # 4   6  0  1 20.56667 4.666667 131.66667 16.33306
    # 5   6  1  0 19.12500 2.500000 115.25000 19.21275
    # 6   8  0  0 15.98000 2.900000 191.00000 17.01239
    # 7   8  0  1 15.40000 6.000000 299.50000 14.55297

gives exactly the same result as above, but the execution is much faster (especially on larger data), because with Fast Statistical Functions, the data does not need to be split by groups, and there is no need to call `lapply()` inside the `across()` statement: `fmean.data.frame()` is simply applied to a subset of the data containing columns `mpg`, `carb` and `hp`.

The Fast Statistical Functions also have a method for grouped data, so if we did not want to calculate the weighted mean of `qsec`, the code would simplify as follows:

    mtcars |>
      subset(mpg > 11) |>
      group_by(cyl, vs, am) |>
      select(mpg, carb, hp) |>
      fmean()
    #   cyl vs am      mpg     carb        hp
    # 1   4  0  1 26.00000 2.000000  91.00000
    # 2   4  1  0 22.90000 1.666667  84.66667
    # 3   4  1  1 28.37143 1.428571  80.57143
    # 4   6  0  1 20.56667 4.666667 131.66667
    # 5   6  1  0 19.12500 2.500000 115.25000
    # 6   8  0  0 15.98000 2.900000 191.00000
    # 7   8  0  1 15.40000 6.000000 299.50000

Note that all functions in collapse, including the Fast Statistical Functions, have the default `na.rm = TRUE`, i.e., missing values are skipped in calculations. This can be changed using `set_collapse(na.rm = FALSE)` to give behavior more consistent with base R.

Another thing to be aware of when using Fast Statistical Functions inside data manipulation functions is that they toggle vectorized execution wherever they are used. E.g.

    mtcars |> group_by(cyl) |> summarise(mpg = fmean(mpg) + min(qsec)) # Vectorized
    #   cyl      mpg
    # 1   4 41.16364
    # 2   6 34.24286
    # 3   8 29.60000

calculates a grouped mean of `mpg` but adds the overall minimum of `qsec` to the result, whereas

    mtcars |> group_by(cyl) |> summarise(mpg = fmean(mpg) + fmin(qsec)) # Vectorized
    #   cyl      mpg
    # 1   4 43.36364
    # 2   6 35.24286
    # 3   8 29.60000
    mtcars |> group_by(cyl) |> summarise(mpg = mean(mpg) + min(qsec))   # Not vectorized
    #   cyl      mpg
    # 1   4 43.36364
    # 2   6 35.24286
    # 3   8 29.60000

both give the mean + the minimum within each group, but calculated in different ways: the former is equivalent to `fmean(mpg, g = cyl) / fmin(qsec, g = cyl)`, whereas the latter is equal to `sapply(gsplit(mpg, cyl), function(x) mean(x) + min(x))`.

See `?fsummarise` and `?fmutate` for more detailed examples. This eager vectorization approach is intentional as it allows users to vectorize complex expressions and fall back to base R if this is not desired.

To take full advantage of collapse, it is highly recommended to use the Fast Statistical Functions as much as possible. You can also set `set_collapse(mask = "all")` to replace statistical functions in base R like `sum` and `mean` with the collapse versions (toggling vectorized execution in all cases), but this may affect other parts of your code2.

## Writing Efficient Code

It is also performance-critical to correctly sequence operations and limit excess computations. tidyverse code is often inefficient simply because the tidyverse allows you to do everything. For example, `mtcars |> group_by(cyl) |> filter(mpg > 13) |> arrange(mpg)` is permissible but inefficient code as it filters and reorders grouped data, requiring modifications to both the data frame and the attached grouping object. collapse does not allow calls to `fsubset()` on grouped data, and messages about it in `roworder()`, encouraging you to write more efficient code.

The above example can also be optimized because we are subsetting the whole frame and then doing computations on a subset of columns. It would be more efficient to select all required columns during the subset operation:

    mtcars |>
      subset(mpg > 11, cyl, vs, am, mpg, carb, hp, qsec, wt) |>
      group_by(cyl, vs, am) |>
      summarise(across(c(mpg, carb, hp), fmean),
                qsec_wt = fmean(qsec, wt))
    #   cyl vs am      mpg     carb        hp  qsec_wt
    # 1   4  0  1 26.00000 2.000000  91.00000 16.70000
    # 2   4  1  0 22.90000 1.666667  84.66667 21.04028
    # 3   4  1  1 28.37143 1.428571  80.57143 18.75509
    # 4   6  0  1 20.56667 4.666667 131.66667 16.33306
    # 5   6  1  0 19.12500 2.500000 115.25000 19.21275
    # 6   8  0  0 15.98000 2.900000 191.00000 17.01239
    # 7   8  0  1 15.40000 6.000000 299.50000 14.55297

Without the weighted mean of `qsec`, this would simplify to

    mtcars |>
      subset(mpg > 11, cyl, vs, am, mpg, carb, hp) |>
      group_by(cyl, vs, am) |>
      fmean()
    #   cyl vs am      mpg     carb        hp
    # 1   4  0  1 26.00000 2.000000  91.00000
    # 2   4  1  0 22.90000 1.666667  84.66667
    # 3   4  1  1 28.37143 1.428571  80.57143
    # 4   6  0  1 20.56667 4.666667 131.66667
    # 5   6  1  0 19.12500 2.500000 115.25000
    # 6   8  0  0 15.98000 2.900000 191.00000
    # 7   8  0  1 15.40000 6.000000 299.50000

Finally, we could set the following options to toggle unsorted grouping, no missing value skipping, and multithreading across the three columns for more efficient execution.

    mtcars |>
      subset(mpg > 11, cyl, vs, am, mpg, carb, hp) |>
      group_by(cyl, vs, am, sort = FALSE) |>
      fmean(nthreads = 3, na.rm = FALSE)
    #   cyl vs am      mpg     carb        hp
    # 1   6  0  1 20.56667 4.666667 131.66667
    # 2   4  1  1 28.37143 1.428571  80.57143
    # 3   6  1  0 19.12500 2.500000 115.25000
    # 4   8  0  0 15.98000 2.900000 191.00000
    # 5   4  1  0 22.90000 1.666667  84.66667
    # 6   4  0  1 26.00000 2.000000  91.00000
    # 7   8  0  1 15.40000 6.000000 299.50000

Setting these options globally using `set_collapse(sort = FALSE, nthreads = 3, na.rm = FALSE)` avoids the need to set them repeatedly.

## Conclusion

collapse enhances R both statistically and computationally and is a good option for tidyverse users searching for more efficient and lightweight solutions to data manipulation and statistical computing problems in R. For more information, I recommend starting with the short vignette on Documentation Resources.

R users willing to write efficient/lightweight code and completely replace the tidyverse in their workflow are also encouraged to closely examine the fastverse suite of packages. collapse alone may not always suffice, but 99% of tidyverse code can be replaced with an efficient and lightweight fastverse solution.

1. Notably, tidyselect, lambda expressions, and many of the smaller helper functions are left out.↩︎

2. When doing this, make sure to refer to base R functions explicitly using `::` e.g. `base::mean`.↩︎