# Drift Detection with datadriftR

# Introduction

In production, the data generating process rarely stays still. Seasonality, product changes, user behavior, and policy updates can shift the distribution of your features—and sometimes even change the relationship between features and the target.

Example (concept drift): You train a model to predict whether electricity prices will go up based on demand and price signals. After a market reform, the same demand level can imply a different price movement. In practice this often shows up as a rising error rate or shifting model outputs.

datadriftR helps you detect these shifts early in streaming settings, so you can trigger investigation, retraining, or alerts.

# Drift Taxonomy

Many “drift” problems are forms of dataset shift:

\[P_{train}(X, Y) \neq P_{prod}(X, Y)\]

What changed?

- Covariate shift: \(P(X)\) changes while \(P(Y|X)\) stays roughly the same.
- Concept drift: \(P(Y|X)\) changes (the mapping from inputs to outputs changes).

# Drift Patterns

Drift can also differ by how it unfolds over time:

- Abrupt: change happens at once (deployments, outages, policy changes)
- Gradual: mixture of old/new concepts (slow adoption, rollout)
- Incremental: continuously moving distribution (wear-and-tear, trend)
- Recurring: seasonal or cyclical patterns (day/night, weekdays, holidays)

Common drift patterns

# Available Methods

Method | What you feed
--- | ---
DDM, EDDM, HDDM-A/W | Binary (0/1) error stream
KSWIN, ADWIN, Page-Hinkley, KL Divergence | Numeric stream (or rolling window/batches)
ProfileDifference | Two profiles (e.g., PDPs)

# Quick Start

Quick start with `detect_drift()`:

    set.seed(1)
    x <- c(rnorm(300, 0, 1), rnorm(200, 3, 1))

    detect_drift(x, method = "page_hinkley", delta = 0.05, threshold = 50)
    #>   index    value  type
    #> 1   315 2.964078 drift

# Part 1: Error-Rate Detectors (Binary Stream)

These methods monitor a stream of prediction errors:

    error_stream <- as.integer(y_pred != y_true)

They work well when you can observe labels reasonably quickly (or with a known delay).

## Simulating Model Errors

    set.seed(123)

    n_good <- 500
    n_bad <- 500
    error_stream <- c(
      rbinom(n_good, 1, prob = 0.05),
      rbinom(n_bad, 1, prob = 0.30)
    )
    true_drift_error <- n_good + 1

## Comparing All Error-Rate Methods

    error_methods <- c("ddm", "eddm", "hddm_a", "hddm_w")

    first_index <- function(res, type) {
      idx <- res$index[res$type == type]
      if (length(idx) == 0) NA_integer_ else idx[1]
    }

    error_results <- do.call(rbind, lapply(error_methods, function(m) {
      res <- detect_drift(error_stream, method = m, include_warnings = TRUE)
      warning_idx <- first_index(res, "warning")
      drift_idx <- first_index(res, "drift")
      data.frame(
        Method = gsub("_", "-", toupper(m)),
        Warning = warning_idx,
        Drift = drift_idx,
        DriftDelay = if (!is.na(drift_idx)) drift_idx - true_drift_error else NA,
        stringsAsFactors = FALSE
      )
    }))

    error_results
    #>   Method Warning Drift DriftDelay
    #> 1    DDM     401   538         37
    #> 2   EDDM      NA   461        -40
    #> 3 HDDM-A     548   570         69
    #> 4 HDDM-W     546   549         48

## Online Processing (Streaming)

    ddm <- DDM$new()
    drifts <- c()

    for (i in seq_along(error_stream)) {
      ddm$add_element(error_stream[i])
      if (ddm$change_detected) {
        drifts <- c(drifts, i)
        ddm$reset()
      }
    }

    data.frame(Method = "DDM", True = true_drift_error, Detected = drifts)
    #>   Method True Detected
    #> 1    DDM  501      538

The loop above is the “low-level” way to run a detector. For convenience, `detect_drift()` provides the same idea as a single function call:

    ddm_res <- detect_drift(error_stream, method = "ddm", include_warnings = FALSE)
    ddm_res
    #>   index value  type
    #> 1   538     1 drift

# Part 2: Continuous-Stream Detectors (Numeric Data)

These methods work with any numeric stream—sensor readings, feature values, model predictions.

## Simulating Sensor Data

    set.seed(456)

    n_normal <- 300
    n_faulty <- 200
    sensor_stream <- c(
      rnorm(n_normal, mean = 20, sd = 1),
      rnorm(n_faulty, mean = 28, sd = 2)
    )
    true_drift_sensor <- n_normal + 1

## Comparing All Distribution Methods

    dist_methods <- c("kswin", "adwin", "page_hinkley")

    dist_results <- do.call(rbind, lapply(dist_methods, function(m) {
      res <- detect_drift(sensor_stream, method = m)
      data.frame(
        Method = gsub("_", "-", toupper(m)),
        Detected = if (nrow(res) > 0) res$index[1] else NA,
        Delay = if (nrow(res) > 0) res$index[1] - true_drift_sensor else NA,
        stringsAsFactors = FALSE
      )
    }))

    dist_results
    #>         Method Detected Delay
    #> 1        KSWIN      313    12
    #> 2        ADWIN      320    19
    #> 3 PAGE-HINKLEY      306     5

# Part 3: KL Divergence

Sometimes you want to compare recent values to a reference window (latency, transaction amount, sensor readings, model scores).

`KLDivergence` is a simple histogram-based implementation of Kullback–Leibler divergence. When the divergence crosses a threshold, it flags drift.

    set.seed(789)

    n_ref <- 400
    n_shift <- 400
    latency_ms <- c(
      rlnorm(n_ref, meanlog = log(100), sdlog = 0.25),
      rlnorm(n_shift, meanlog = log(180), sdlog = 0.30)
    )
    true_drift_kld <- n_ref + 1

    window <- 200
    kld <- KLDivergence$new(bins = 30, drift_level = 0.15)
    kld$set_initial_distribution(latency_ms[1:window])

    kl <- rep(NA_real_, length(latency_ms))
    for (t in (window + 1):length(latency_ms)) {
      current <- latency_ms[(t - window + 1):t]
      kld$add_distribution(current)
      kl[t] <- kld$get_kl_result()
    }

    detected_kld <- which(kl > kld$drift_level)[1]
    data.frame(True = true_drift_kld, Detected = detected_kld, Threshold = kld$drift_level)
    #>   True Detected Threshold
    #> 1  401      295      0.15

# Part 4: ProfileDifference (Model Behavior)

Partial Dependence Profiles (PDPs) show how a model’s prediction changes with a feature. When concept drift occurs, the relationship between features and target changes—PDPs capture this.

## Available Methods

Method | Description
--- | ---
pdi | Profile Disparity Index - compares derivative signs
L2 | L2 norm between profiles
L2_derivative | L2 norm of profile derivatives

## Example: Electricity Prices (elec2)

The `elec2` dataset from the `dynaTree` package is a classic benchmark for concept drift—Australian electricity prices where market dynamics changed over time. This section requires the optional packages `dynaTree` and `ranger`. For readability we rename `x1..x4`/`y`, and for speed we train on a small subset from each time period.

    library(dynaTree)
    library(ranger)

    elec2_env <- new.env(parent = emptyenv())
    data("elec2", package = "dynaTree", envir = elec2_env)
    elec2_df <- get("elec2", envir = elec2_env)
    stopifnot(is.data.frame(elec2_df))

    names(elec2_df) <- c("nswprice", "nswdemand", "vicprice", "vicdemand", "class_raw")
    elec2_df$class <- factor(elec2_df$class_raw, levels = c(1, 2), labels = c("DOWN", "UP"))
    elec2_df$class_raw <- NULL

    split_idx <- floor(nrow(elec2_df) / 2)
    period1_data <- elec2_df[1:split_idx, ]
    period2_data <- elec2_df[(split_idx + 1):nrow(elec2_df), ]

    n_train <- min(2000, nrow(period1_data), nrow(period2_data))
    period1_train <- period1_data[1:n_train, ]
    period2_train <- period2_data[1:n_train, ]

    rf1 <- ranger(class ~ nswprice + nswdemand + vicprice + vicdemand,
                     data = period1_train, probability = TRUE, num.trees = 200, seed = 1)
    rf2 <- ranger(class ~ nswprice + nswdemand + vicprice + vicdemand,
                     data = period2_train, probability = TRUE, num.trees = 200, seed = 1)

    compute_pdp_rf <- function(model, data, var, grid) {
      preds <- sapply(grid, function(val) {
        newdata <- data
        newdata[[var]] <- val
        mean(predict(model, newdata)$predictions[, "UP"])
      })
      list(x = grid, y = preds)
    }

    demand_grid <- seq(min(elec2_df$nswdemand), max(elec2_df$nswdemand), length.out = 50)
    pdp1 <- compute_pdp_rf(rf1, period1_train, "nswdemand", demand_grid)
    pdp2 <- compute_pdp_rf(rf2, period2_train, "nswdemand", demand_grid)

## Comparing All Methods

    # PDI (Profile Disparity Index)
    pd_pdi <- ProfileDifference$new(method = "pdi", deriv = "gold")
    pd_pdi$set_profiles(pdp1, pdp2)
    res_pdi <- pd_pdi$calculate_difference()

    # L2 norm
    pd_l2 <- ProfileDifference$new(method = "L2")
    pd_l2$set_profiles(pdp1, pdp2)
    res_l2 <- pd_l2$calculate_difference()

    # L2 derivative
    pd_l2d <- ProfileDifference$new(method = "L2_derivative")
    pd_l2d$set_profiles(pdp1, pdp2)
    res_l2d <- pd_l2d$calculate_difference()

    data.frame(
      Method = c("PDI", "L2", "L2_derivative"),
      Distance = round(c(res_pdi$distance, res_l2$distance, res_l2d$distance), 4)
    )
    #>          Method Distance
    #> 1           PDI   0.0284
    #> 2            L2  84.4088
    #> 3 L2_derivative   0.0005

Higher distance indicates the model learned different demand-price relationships in each period—concept drift detected.

# Quick Reference

What you monitor | Typical signal | Recommended
--- | --- | ---
Binary error stream | Sudden or gradual accuracy drop | DDM, EDDM, HDDM-A/W
Numeric stream | Feature/sensor/score distribution shift | KSWIN, ADWIN, Page-Hinkley
Reference vs. current batch | Periodic samples (e.g., daily latency) | KLDivergence
Model behavior profiles | PDP/feature effect changes | ProfileDifference

# References

The drift detection methods implemented in datadriftR are based on established algorithms from the streaming machine learning literature. For Python implementations, see:

- River (riverml.xyz): Online machine learning library with drift detectors
- scikit-multiflow (scikit-multiflow.github.io): Stream learning framework

Source code: github.com/ugurdar/datadriftR