# Introduction to smriti: Structural Variance Preservation

## The Imputation Uncertainty Principle

Modern machine learning imputation algorithms (like `missForest`) excel at minimizing point-wise prediction error (RMSE). However, this point-wise optimization inherently shrinks the variance of the imputed values, causing structural variance collapse. In longitudinal Growth Curve Models (GCM), this crushes the latent slope variance (\(\sigma^2_S\)), destroying the statistical power needed to track patient trajectories over time.

The `smriti` package resolves this by decoupling prediction from structural geometry. It utilizes a two-stage architecture: 1. Initialization: Non-parametric imputation bridges the missingness to establish a dense matrix. 2. Lagrangian Projection: A C++ gradient descent layer projects the hallucinated data toward a target covariance manifold while preserving fidelity to the initial imputed values. The augmented loss function is

\[L(X) = \frac{1}{2}\|X - X_{\text{imp}}\|_F^2 + \frac{\lambda}{2}\|\operatorname{cov}(X) - \Sigma_{\text{target}}\|_F^2\]

where the first term anchors the solution near the initial imputation and the second (governed by \(\lambda\)) enforces the covariance structure.

## The Robustness-Efficiency Tradeoff

Real-world clinical data often contains heavy-tailed skew or corrupted sensor artifacts. The `smriti_impute()` function handles this via the `robust` routing toggle:

- `robust = FALSE`: Uses pairwise-complete Pearson covariance, projected to the nearest positive-semidefinite matrix to correct any non-PSD artefacts from pairwise deletion. Best for well-behaved, approximately-Normal data.
- `robust = TRUE`: Constructs the target from pairwise Spearman correlations (rank-based, outlier-resistant) and column-wise MAD scale estimates. The resulting matrix is projected to the nearest PSD manifold, producing a target that is structurally robust to severe outliers (e.g., broken EHR sensors).

## Fidelity-Constraint Balance

The penalty weight `lambda` controls the trade-off between preserving the original imputation values and matching the target covariance. At `lambda = 1.0` (the default) both objectives are weighted equally. Increasing `lambda` enforces the covariance constraint more strictly but allows greater deviation from the initial imputation. The `learning_rate` (default `0.001`) governs gradient step size; `max_iter` (default `2000`) bounds the optimisation.

## Example: Shielding Against Corrupted EHR Data

    library(smriti)
    library(missForest)

    # Load clinical data with structural missingness and sensor artifacts
    data <- read.csv("clinical_proxy.csv")

    # Execute robust refinement to isolate the structural manifold
    clean_data <- smriti_impute(
      data       = data,
      time_cols  = c("T1", "T2", "T3", "T4"),
      robust     = TRUE,
      lambda     = 1.0
    )