# AutoMLR

Automated **survival-analysis**, **binary-classification**,
**continuous-outcome**, and **ordinal-outcome** machine-learning combination
models in R. Train and compare model variants on multi-cohort data with a
single call, select models by outcome-appropriate out-of-fold metrics, and
report integrated diagnostics.

> Status: **1.0.0 release candidate**. Survival, binary, continuous, and
> ordinal workflows are available with cohort-aware preparation, 18-variant
> default candidate pools, single-/two-model combination search, HTML reports,
> CSV audit tables, publication-style figures, and automated tests.

---

## Why AutoMLR?

AutoMLR is a multi-outcome biomedical modeling framework. The survival
workflow is designed to answer one question end-to-end:

> *Given multiple cohorts with survival outcomes, which single model (or
> combination) gives the most robust prognostic signature, and how confident
> am I?*

## Survival Algorithm Families

| # | Family              | Algorithm                      | Package          |
|---|---------------------|--------------------------------|------------------|
| 1 | Penalised Cox       | Lasso Cox                      | `glmnet`         |
| 2 | Penalised Cox       | Ridge Cox                      | `glmnet`         |
| 3 | Penalised Cox       | Elastic Net Cox                | `glmnet`         |
| 4 | Classic Cox         | Stepwise Cox                   | `survival`       |
| 5 | Boosted Cox         | CoxBoost                       | `CoxBoost`       |
| 6 | Dimension-reduced   | plsRcox (partial least squares)| `plsRcox`        |
| 7 | Dimension-reduced   | SuperPC                        | `superpc`        |
| 8 | Tree-boosted        | GBM-Cox                        | `gbm`            |
| 9 | Kernel              | survival-SVM                   | `survivalsvm`    |
| 10| Tree-ensemble       | Random Survival Forest (RSF)   | `randomForestSRC`|

## Installation

```r
# install from a local source checkout
remotes::install_local("AutoMLR", dependencies = c("Depends", "Imports"))

# or load directly from a local checkout:
# pkgload::load_all("AutoMLR")
```

AutoMLR keeps heavyweight model engines as optional suggested dependencies so
the package can be installed even on machines where some modelling packages are
hard to compile. Missing backends skip only the affected model variants while
the remaining algorithms and reports continue to run.

```r
deps <- check_automlr_dependencies()
deps

# Install only the optional engines you need.
install.packages(c("glmnet", "gbm", "randomForestSRC", "timeROC"))
```

Progress messages are mirrored to a session log when logging is enabled:

```r
initialize_auto_logging("automlr_logs")
# Run AutoMLR analysis here.
disable_auto_logging()
```

Model-evaluation and ensemble-fitting messages such as `Evaluating ...` and
`Fitting ...` are then written to the log file as well as the console.

## Minimal Example

```r
library(AutoMLR)

# Use the canonical CRAN survival::lung dataset.
data("cancer", package = "survival")
lung <- na.omit(lung)
lung$cohort <- rep(c("A", "B"), length.out = nrow(lung))

prep <- prepare_cohort_input(
  data    = lung,
  cohort  = "cohort",
  time    = "time",
  status  = "status"
)
report_cohort_intersection(prep)

params <- automlr_parameters()
list_surv_algorithms()

# `prep` can be passed directly; cohort labels are then used automatically for
# stability diagnostics in the combination table.
ens <- fit_surv_ensemble(
  prep,
  params = automlr_parameters(algorithms = "lasso_cox", verbose = FALSE),
  min_models = 1,
  max_models = 1
)

export_surv_results(
  ens,
  output_dir = "results/lung_example",
  formats = c("pdf", "png")
)
```

This verifies: data contract, cohort intersection, parameter defaults, and the
10-algorithm registry load. `list_model_variants()` shows the concrete
algorithm + hyperparameter candidates; `count_surv_combinations()` reports how
many combinations will be searched. Single-model fitting, all-subsets
combination ranking, and weighted ensemble fitting. By default, combinations
are searched across distinct base algorithm families rather than pairing two
thresholds from the same algorithm.
Combination tables keep C-index as the primary rank by default while reporting
queue / subsample stability, member failure rate, risk spread, and optional
time-dependent AUC diagnostics. When `fit_surv_ensemble()` receives the
`automlr_input` object returned by `prepare_cohort_input()`, the original cohort
labels are wired into `stability_groups` automatically.
The `surv_svm` candidate is evaluated with k-fold resampling by default because
the optional `survivalsvm` backend can fail on single-row predictions during
leave-one-out cross-validation. Users can restore the old behavior with
`automlr_parameters(surv_svm_resampling = "loocv")` or exclude the model via
the `algorithms` argument.
`export_surv_results()` writes a batch result bundle with an HTML report,
publication figures, diagnostic figures, CSV tables, fitted RDS objects, and
session metadata. It also exports per-sample risk scores plus time-dependent
AUC / ROC tables when `timeROC` is installed, and writes bilingual English /
Chinese Markdown summaries for data preparation, base-model screening, and
ensemble selection under `summaries/`. The current regular survival report has
four step-specific interpretation files: `data_summary.md`,
`base_model_summary.md`, `ensemble_summary.md`, and
`explainability_summary.md`. The last one explains feature importance,
SHAP-style approximations, nomogram, calibration, DCA, and model forest-plot
outputs. The default publication bundle is
deduplicated to a final figure set: an all-single-model cohort C-index heatmap,
an IRLS-style combination-by-cohort benchmark matrix with mean C-index side
bars, multi-cohort Kaplan-Meier panels when risk groups are estimable, and
multi-cohort timeROC panels when `timeROC` is available.
Publication heatmaps omit rows with no finite metric values, keep partial
missing cells as explicit `NA` tiles, and adapt color scales to the finite
metric range so small differences remain visible.

For a figure-rich walkthrough with standalone code blocks, see
`inst/tutorials/AutoMLR_tutorial.md` after installing or unpacking the package.

## Lower-level evaluation helpers

Prepared cohort objects can be converted to modeling matrices without using
internal `:::` calls:

```r
xy <- automlr_input_to_surv_xy(prep)

single <- evaluate_algorithm_loocv(
  "lasso_cox",
  X = xy$X,
  y = xy$y,
  params = automlr_parameters(algorithms = "lasso_cox", verbose = FALSE)
)
```

The same convenience pattern is available for the other workflows:
`automlr_input_to_binary_xy()`, `automlr_input_to_continuous_xy()`, and
`automlr_input_to_ordinal_xy()`.

For threshold-style ensemble selection, users may keep fixed cutoffs or ask
AutoMLR to recommend a data-adaptive cutoff from the candidate metric
distribution:

```r
params_auto <- automlr_parameters(
  algorithms = c("lasso_cox", "ridge_cox"),
  auto_min_cindex = TRUE,
  auto_quantile = 0.50,
  verbose = FALSE
)

ens_auto <- fit_surv_ensemble(
  prep,
  params = params_auto,
  strategy = "threshold"
)

ens_auto$selection_threshold
```

`auto_quantile = 0.50` uses the median candidate C-index; larger values such
as `0.75` are more selective. Analogous controls are available as
`auto_min_auc`, `auto_min_r2`, and `auto_min_qwk` for binary, continuous, and
ordinal workflows.

## Binary classification workflow

The binary workflow mirrors the survival workflow but uses positive-class
probabilities and ROC AUC:

```r
data(iris)
iris2 <- iris[iris$Species != "setosa", ]
iris2$cohort <- rep(c("A", "B"), length.out = nrow(iris2))
iris2$sample_id <- paste0("iris_", seq_len(nrow(iris2)))

prep_bin <- prepare_binary_cohort_input(
  iris2,
  cohort = "cohort",
  outcome = "Species",
  id = "sample_id",
  positive_class = "versicolor",
  negative_class = "virginica"
)

params_bin <- binarymlr_parameters(
  resampling = "loocv",
  verbose = FALSE
)
count_binary_combinations(params_bin, min_size = 1, max_size = 2)

bin_fit <- fit_binary_ensemble(
  prep_bin,
  params = params_bin,
  min_models = 1,
  max_models = 2,
  rank_by = "auc"
)

export_binary_results(
  bin_fit,
  output_dir = "results/iris_binary",
  formats = c("png", "pdf"),
  summary_language = "bilingual"
)
```

The default binary pool contains 18 model variants across penalized logistic,
standard / stepwise logistic, GBM, random forest, PCA-logistic, and Gaussian
naive Bayes families. The default search includes single models and two-model
probability combinations while excluding same-algorithm variant pairs.
`export_binary_results()` writes ROC/PR curves, cohort AUC heatmaps,
combination benchmark matrices, calibration, DCA, confusion matrix, feature
importance, a model-performance forest plot, predicted probabilities,
threshold metrics, preprocessing audit tables, resampling manifests, and the
same four step-specific bilingual interpretation documents used by the
survival report. Multi-class outcomes are rejected by default to avoid
accidental "positive vs all other" analyses; set `collapse_other = TRUE` only
when that collapse is intentional. Exported prediction tables distinguish
apparent probabilities from out-of-fold probabilities, and the default report
uses the out-of-fold probabilities for model diagnostics.

## Continuous outcome workflow

The continuous workflow targets numeric outcomes and ranks models by
out-of-fold RMSE by default:

```r
d <- mtcars
d$sample_id <- rownames(mtcars)
d$cohort <- rep(c("A", "B"), length.out = nrow(d))

prep_cont <- prepare_continuous_cohort_input(
  d,
  cohort = "cohort",
  outcome = "mpg",
  id = "sample_id"
)

params_cont <- continuousmlr_parameters(
  resampling = "kfold",
  k_folds = 5,
  verbose = FALSE
)

cont_fit <- fit_continuous_ensemble(
  prep_cont,
  params = params_cont,
  min_models = 1,
  max_models = 2,
  rank_by = "rmse"
)

export_continuous_results(
  cont_fit,
  output_dir = "results/mtcars_continuous",
  formats = c("png", "pdf")
)
```

The default pool contains 18 regression variants. Exported outputs include
single-model and combination rankings, predictions, cohort-level RMSE / MAE /
R-squared diagnostics, observed-vs-predicted plots, residual distributions,
feature importance, preprocessing audits, fitted objects, session metadata,
and an HTML report.

## Ordinal outcome workflow

The ordinal workflow targets ordered categorical outcomes and ranks models by
out-of-fold quadratic weighted kappa:

```r
data(iris)
iris$cohort <- rep(c("A", "B", "C"), length.out = nrow(iris))
iris$sample_id <- paste0("iris_", seq_len(nrow(iris)))

prep_ord <- prepare_ordinal_cohort_input(
  iris,
  cohort = "cohort",
  outcome = "Species",
  id = "sample_id",
  ordered_levels = c("setosa", "versicolor", "virginica")
)

params_ord <- ordinalmlr_parameters(
  resampling = "kfold",
  k_folds = 5,
  verbose = FALSE
)

ord_fit <- fit_ordinal_ensemble(
  prep_ord,
  params = params_ord,
  min_models = 1,
  max_models = 2,
  rank_by = "qwk"
)

export_ordinal_results(
  ord_fit,
  output_dir = "results/iris_ordinal",
  formats = c("png", "pdf")
)
```

Ordinal exports include QWK rankings, predicted ordered classes, cohort
performance, confusion matrices, observed-score plots, feature importance,
preprocessing audits, fitted objects, session metadata, and an HTML report.
Users should supply `ordered_levels` whenever the natural order is not already
encoded in an ordered factor.

## Extreme screening

For exploratory upper-bound analysis, `extreme_surv_screen()` first fits all
candidate variants on the full dataset and evaluates them on the same samples
to rank apparent single- or two-model combinations. It then carries only the
top N combinations into a real stratified train/validation seed search.

```r
extreme <- extreme_surv_screen(
  prep,
  top_n = 10,
  seeds = 1:500,
  train_fraction = 0.70,
  min_models = 1,
  max_models = 2,
  rank_by = "apparent_cindex"
)

extreme$combination_summary
extreme$top_combinations
extreme$best

export_extreme_screen_results(
  extreme,
  output_dir = "results/lung_extreme_screen",
  formats = c("png", "pdf"),
  summary_language = "bilingual"
)
```

The first stage is intentionally optimistic and is meant to estimate the
dataset's apparent ceiling, not external validation performance. The second
stage reports the best random seed, model combination, train C-index,
validation C-index, and cohort-level validation diagnostics when cohorts are
available. `export_extreme_screen_results()` writes complete audit tables plus
Morandi-toned PNG/PDF figures for apparent ranking, seed-search distributions,
seed-by-combination heatmaps, best-by-seed trajectories, apparent-vs-validation
gaps, and cohort-level top-row diagnostics. It also writes a bilingual English
/ Chinese `summary_report.md` by default, or a single-language report with
`summary_language = "zh"` or `"en"`, summarizing the current best apparent model,
best validation seed,
train / validation C-index, top seed-search rows, combination-level stability,
and any failed seed-search fits.

## Release Scope

AutoMLR 1.0.0 includes survival, binary, continuous, and ordinal workflows;
cohort-aware input preparation; 18-variant default candidate pools; single-
and two-model combination search; publication figures; HTML reports; fitted
objects; CSV audit tables; bilingual summaries for survival and binary
workflows; and automated tests covering the main exported interfaces.

## Data contract

Single long-format data frame:

- One row per sample.
- A `cohort` column (character / factor) identifying dataset membership.
- `time` column (numeric, > 0).
- `status` column (0/1, where 1 = event).
- All remaining columns treated as candidate features.

`prepare_cohort_input()` splits by cohort and returns the **feature
intersection** across cohorts, along with a diagnostic report.

## License

MIT (see `LICENSE`).
