# AutoMLR Survival Modeling Tutorial

This tutorial shows a complete AutoMLR survival-analysis workflow from
multi-cohort data preparation to regular ensemble modeling, publication-style
figures, explainability diagnostics, bilingual result summaries, and optional
extreme screening.

The layout follows a structured workflow: each section explains the purpose,
shows a standalone code block, displays representative output figures, and
then gives a short interpretation guide.

![AutoMLR workflow](figures/automlr_workflow.svg)

## 1. What AutoMLR Does

AutoMLR is designed for multi-cohort biomedical machine learning. The survival
workflow remains the most figure-rich path, and the package now also provides
parallel binary, continuous, and ordinal outcome workflows. It answers five
practical questions:

- Which base survival model is strongest?
- Does a two-model combination improve over the best single model?
- Are results stable across cohorts?
- Which variables drive the final risk score?
- What is the apparent upper bound under a controlled extreme-screening mode?

The package keeps C-index as the default selection criterion. Other metrics,
including cohort stability, time-dependent AUC, failure rate, model complexity,
Kaplan-Meier separation, risk-score diagnostics, SHAP-style approximations,
calibration, decision-curve analysis, and model-level forest summaries, are
exported for interpretation rather than replacing C-index as the main ranking
metric.

For non-survival outcomes, AutoMLR keeps the same engineering pattern:
prepare the cohort-aware input, define conservative defaults, evaluate 18
model variants, search single- and two-model combinations, and export tables,
publication-style figures, fitted objects, and an HTML report.

## 2. Installation

Install AutoMLR from a local source package or source directory:

```r
remotes::install_local("AutoMLR_1.0.0.tar.gz", dependencies = TRUE)
```

For local source checkouts:

```r
pkgload::load_all("AutoMLR")
```

Load the package:

```r
library(AutoMLR)
```

## 3. Input Data Contract

AutoMLR expects one long-format data frame:

| Required column | Meaning |
| --- | --- |
| `cohort` | Cohort or dataset label |
| `time` | Survival time |
| `status` | Event indicator, where `1` means event and `0` means censored |
| feature columns | Numeric predictors shared across cohorts |

The helper `prepare_cohort_input()` computes the feature intersection across
cohorts and returns an `automlr_input` object.

For other outcome types, the same long-format convention is used:

| Outcome type | Preparation helper | Primary ranking metric |
| --- | --- | --- |
| Binary | `prepare_binary_cohort_input()` | OOF ROC AUC |
| Continuous | `prepare_continuous_cohort_input()` | OOF RMSE |
| Ordinal | `prepare_ordinal_cohort_input()` | OOF quadratic weighted kappa |

### Example Dataset

The following example uses the standard `survival::lung` dataset as a small
example dataset. The tutorial figures were regenerated from an 80-sample
lung run split into two artificial cohorts, so the code below mirrors that
example. Real projects should replace this block with the user's multi-cohort
clinical or molecular matrix.

```r
data("cancer", package = "survival")

lung <- stats::na.omit(lung)
lung$status <- as.integer(lung$status == 2)
lung$sample_id <- paste0("lung_", seq_len(nrow(lung)))
lung$cohort <- rep(c("Lung_A", "Lung_B"), length.out = nrow(lung))

lung <- lung[seq_len(80), c(
  "sample_id", "cohort", "time", "status",
  "age", "sex", "ph.ecog", "ph.karno", "pat.karno",
  "meal.cal", "wt.loss"
)]
```

Prepare the AutoMLR input:

```r
prep <- prepare_cohort_input(
  data = lung,
  cohort = "cohort",
  time = "time",
  status = "status",
  id = "sample_id"
)

report_cohort_intersection(prep)
```

### Interpretation

Check three things before fitting models:

- Each cohort should have enough samples and enough events.
- The shared feature count should be large enough for the intended model class.
- Dropped features should be reviewed if different cohorts were measured on
  different platforms.

## 4. Set User-Facing Defaults

The current recommended defaults are intentionally conservative:

```r
params <- automlr_parameters(
  seed = 123L,
  eval_times = c(365, 1095, 1825),
  time_unit = "auto",
  n_cores = 1L,
  stability_resamples = 0L,
  verbose = FALSE
)
```

`time_unit` controls the labels in time-dependent ROC, nomogram, calibration,
and KM panels. Use `"days"`, `"months"`, or `"years"` when the source dataset
has a known unit. The default `"auto"` only infers the display unit from the
follow-up scale; it does not transform the data.

For the built-in `lung` dataset, survival time is measured in days:

```r
params <- automlr_parameters(
  seed = 101L,
  eval_times = c(180, 365, 730),
  time_unit = "days",
  stability_resamples = 2L,
  n_cores = 1L,
  verbose = FALSE
)
```

For a fast tutorial run, restrict algorithms to two families:

```r
params <- automlr_parameters(
  algorithms = c("lasso_cox", "ridge_cox"),
  eval_times = c(180, 365, 730),
  time_unit = "days",
  stability_resamples = 0L,
  verbose = FALSE
)
```

For a full run on the lung example, use the default algorithm pool while
keeping the known day-scale horizons:

```r
params <- automlr_parameters(
  eval_times = c(180, 365, 730),
  time_unit = "days",
  verbose = FALSE
)
```

The full default pool expands to about 18 model variants across the 10 survival
algorithm families. The final package default searches single models and
two-model combinations:

```r
list_model_variants(params)
count_surv_combinations(params, min_size = 1, max_size = 2)
```

## 5. Fit Regular Survival Ensembles

Run the regular modeling workflow:

```r
ens <- fit_surv_ensemble(
  prep,
  params = params,
  algorithms = params$algorithms,
  strategy = "best_subset",
  min_models = 1,
  max_models = 2,
  weight_method = "cindex",
  allow_same_algorithm = FALSE,
  rank_by = "cindex",
  diagnostic_times = params$eval_times,
  verbose = FALSE
)
```

The object contains:

```r
names(ens)
ens$selected_candidates
ens$weights
head(ens$evaluation$summary)
head(ens$combination_evaluation$summary)
```

### Base-Model Table

`ens$evaluation$summary` reports each model variant:

- `candidate_key`: model variant identifier
- `algo_key`: base algorithm
- `cindex`: LOOCV C-index
- `failure_rate`: failed LOOCV fold fraction
- `risk_sd`: risk-score spread
- `stability_score`: cohort/resampling stability diagnostic

### Combination Table

`ens$combination_evaluation$summary` reports each candidate combination:

- `combination_key`: single model or two-model combination
- `cindex`: combined LOOCV C-index
- `selection_score`: ranking score
- `weights`: member model weights
- `stability_score`: diagnostic stability score
- `mean_failure_rate`: member failure diagnostic

## 6. Export a Complete Regular Report

Export all regular-analysis outputs:

```r
regular_export <- export_surv_results(
  ens,
  output_dir = "results/final_regular",
  formats = c("png", "pdf"),
  dpi = 300,
  top_n = 20,
  publication = TRUE,
  summary_language = "bilingual"
)
```

The export directory contains:

```r
regular_export$report
regular_export$tables
regular_export$summaries
regular_export$publication_figures
```

Important files:

| Folder | File | Purpose |
| --- | --- | --- |
| `report/` | `index.html` | Human-readable HTML report |
| `summaries/` | `summary_report.md` | Bilingual English/Chinese executive summary |
| `summaries/` | `data_summary.md` | Data-preparation summary |
| `summaries/` | `base_model_summary.md` | Base-model screening summary |
| `summaries/` | `ensemble_summary.md` | Ensemble-selection summary |
| `summaries/` | `explainability_summary.md` | SHAP, nomogram, calibration, DCA, and forest-plot interpretation |
| `tables/` | `single_models.csv` | Full base-model table |
| `tables/` | `combinations.csv` | Full combination table |
| `tables/` | `risk_scores.csv` | Per-sample risk scores |
| `tables/` | `time_auc.csv`, `time_roc_curve.csv` | Time-dependent AUC and ROC coordinates |
| `tables/` | `cohort_km_stats.csv` | KM panel log-rank and HR statistics |
| `tables/` | `feature_importance.csv` | Final-model feature importance |
| `tables/` | `shap_approx_contributions.csv` | Approximate per-sample feature contributions |
| `tables/` | `risk_score_nomogram.csv` | Nomogram-like risk-score calibration grid |
| `tables/` | `calibration_curve.csv` | Observed versus predicted risk summaries |
| `tables/` | `dca_curve.csv` | Decision-curve net benefit table |
| `tables/` | `model_cindex_forest.csv` | C-index estimates for the model forest plot |
| `figures/publication/` | `*.png`, `*.pdf` | Final publication-style figures |

## 7. Read the Publication Figures

### 7.1 All-Model Cohort Heatmap

![All-model cohort heatmap](figures/regular_all_model_cohort_heatmap.png)

This heatmap compares single-model C-index values across cohorts and the
overall dataset. Use it to identify models that perform well overall but may
be unstable in a specific cohort.

Interpretation checklist:

- Strong models should have high overall C-index.
- A robust model should not collapse in one cohort.
- If the best overall model is cohort-specific, treat it as hypothesis
  generating rather than definitive.

### 7.2 Combination Benchmark Matrix

![Combination benchmark matrix](figures/regular_combination_benchmark_matrix.png)

This benchmark matrix ranks candidate single models and two-model combinations.
The side bar summarizes mean C-index, while the grid shows cohort-level
behavior. The current layout gives a narrower heatmap and a wider mean C-index
bar panel so that top-row ranking remains readable in small reports.

Interpretation checklist:

- Prefer a combination only if it improves C-index or stability meaningfully.
- If the best combination and best single model are tied, the simpler model may
  be preferable.
- Check whether the combination is driven by one cohort or stable across all
  cohorts.

### 7.3 Multi-Cohort Kaplan-Meier Curves

![Multi-cohort KM panel](figures/regular_multi_cohort_km_panel.png)

The Kaplan-Meier panel stratifies samples into high- and low-risk groups using
the exported risk score. It is a visual diagnostic of clinical separation. Each
panel now includes a number-at-risk table below the survival curve.

Interpretation checklist:

- Clear separation supports clinical usefulness.
- Small cohort panels can be noisy; inspect sample counts and events.
- The number-at-risk rows show whether late curve separation is based on enough
  remaining samples.
- KM curves are diagnostic; they do not replace cross-validated C-index.

### 7.4 Multi-Cohort Time-Dependent ROC

![Multi-cohort timeROC panel](figures/regular_multi_cohort_time_roc_panel.png)

When `timeROC` is installed, AutoMLR exports time-dependent ROC curves at
`eval_times`. The time labels use `time_unit`: for example, the built-in lung
run displays `180 days`, `365 days`, and `730 days`, while a year-scale dataset
can display `1 year`, `3 years`, and `5 years`.

Interpretation checklist:

- AUC values provide time-specific discrimination diagnostics.
- Compare time points; a model may work better early or late.
- Confirm that `time_unit` matches the original survival-time unit before
  interpreting horizons.
- Keep C-index as the selection metric unless a project explicitly prioritizes
  a time-specific endpoint.

### 7.5 Feature Importance Barplot

![Feature importance barplot](figures/regular_feature_importance_barplot.png)

The feature-importance barplot summarizes which predictors contribute most to
the final ensemble risk score. It is designed as a compact first-pass
interpretability view.

Interpretation checklist:

- Treat feature importance as a model diagnostic, not as causal evidence.
- Check whether top features are clinically plausible and consistently
  measured across cohorts.
- If several correlated predictors appear, interpret them as a feature group.

### 7.6 SHAP-Style Summary Plot

![SHAP summary plot](figures/regular_shap_summary_plot.png)

The SHAP-style summary plot approximates each feature's contribution to the
final risk score. Points show sample-level contribution patterns, and feature
value is encoded by color when available.

Interpretation checklist:

- Wide horizontal spread indicates stronger risk-score influence.
- Color gradients can reveal whether high or low feature values increase risk.
- Approximate SHAP diagnostics are most useful for explanation and audit, not
  for selecting the final model.

### 7.7 SHAP-Style Dependence Plot

![SHAP dependence plot](figures/regular_shap_dependence_plot.png)

The dependence plot focuses on the strongest feature and shows how its value
relates to its estimated contribution.

Interpretation checklist:

- Smooth monotone trends are easier to explain clinically.
- Nonlinear or threshold-like patterns should be checked against data quality.
- Outlying points can identify samples worth reviewing.

### 7.8 Risk-Score Nomogram

![Risk-score nomogram](figures/regular_risk_score_nomogram.png)

The nomogram-style plot converts the final risk score into horizon-specific
event-risk scales. It is intentionally risk-score based so it can be generated
for many model classes.

Interpretation checklist:

- Confirm that horizons use the correct `time_unit`.
- Use this as a communication aid; the underlying calibration table should be
  checked before clinical interpretation.
- Avoid over-interpreting precise tick positions in small datasets.

### 7.9 Calibration Curve

![Calibration curve](figures/regular_calibration_curve.png)

The calibration curve compares predicted event risk with observed event
frequency at selected horizons.

Interpretation checklist:

- Curves close to the diagonal suggest better calibration.
- Large deviations can indicate overfitting, cohort mismatch, or insufficient
  event counts.
- Calibration is horizon-specific; a model can be calibrated at one time point
  and less calibrated at another.

### 7.10 Decision-Curve Analysis

![Decision-curve analysis](figures/regular_decision_curve_analysis.png)

Decision-curve analysis estimates net benefit across risk thresholds. It helps
evaluate whether a risk score may be clinically useful under different action
thresholds.

Interpretation checklist:

- Compare the model curve with treat-all and treat-none references.
- Focus on clinically plausible threshold ranges.
- DCA is a decision-support diagnostic; it should be interpreted with external
  validation whenever possible.

### 7.11 Model C-Index Forest Plot

![Model C-index forest plot](figures/regular_model_cindex_forest.png)

The model forest plot summarizes C-index estimates for top models or
combinations. It is useful for comparing the final model against close
alternatives.

Interpretation checklist:

- Prefer models that are both high-performing and stable.
- If confidence intervals overlap heavily, emphasize simplicity and
  reproducibility.
- Use the forest plot together with the cohort heatmap to avoid choosing a
  cohort-specific result.

## 8. Understand the Bilingual Summary

The regular summary is generated automatically:

```r
summarize_surv_analysis_results(
  ens,
  top_n = 5,
  language = "bilingual"
)
```

The report is English first and Chinese second. The regular survival workflow
has four key interpretation checkpoints:

| Checkpoint | Function | Exported file | What it explains |
| --- | --- | --- | --- |
| Data preparation | `summarize_data_preparation()` | `summaries/data_summary.md` | Cohort count, sample/event balance, shared features, and data-risk notes |
| Base-model screening | `summarize_base_models()` | `summaries/base_model_summary.md` | Top single-model variants, LOOCV C-index, failures, stability, and risk-score spread |
| Ensemble selection | `summarize_ensemble_results()` | `summaries/ensemble_summary.md` | Best single/two-model combination, weights, C-index delta, and stability diagnostics |
| Explainability and utility | `summarize_explainability_results()` | `summaries/explainability_summary.md` | Feature importance, SHAP-style contribution diagnostics, nomogram, calibration, DCA, and model forest plot |

The integrated `summaries/summary_report.md` combines all four checkpoints and
adds key warnings about sample size, event count, stability, model complexity,
and interpretation limits.

Single sections can also be generated:

```r
summarize_data_preparation(prep, language = "bilingual")
summarize_base_models(ens, language = "bilingual")
summarize_ensemble_results(ens, language = "bilingual")
summarize_explainability_results(ens, language = "bilingual")
```

## 9. Binary Classification Workflow

AutoMLR also includes a binary-classification workflow that mirrors
the survival workflow. The main difference is that ROC AUC becomes the primary
selection metric and the model output is a positive-class probability.

```r
data(iris)

iris2 <- iris[iris$Species != "setosa", ]
iris2$cohort <- rep(c("A", "B"), length.out = nrow(iris2))
iris2$sample_id <- paste0("iris_", seq_len(nrow(iris2)))

prep_bin <- prepare_binary_cohort_input(
  iris2,
  cohort = "cohort",
  outcome = "Species",
  id = "sample_id",
  positive_class = "versicolor",
  negative_class = "virginica"
)

params_bin <- binarymlr_parameters(
  resampling = "loocv",
  verbose = FALSE
)
list_binary_model_variants(params_bin)
count_binary_combinations(params_bin, min_size = 1, max_size = 2)
```

Fit the binary ensemble:

```r
bin_fit <- fit_binary_ensemble(
  prep_bin,
  params = params_bin,
  min_models = 1,
  max_models = 2,
  weight_method = "auc",
  allow_same_algorithm = FALSE,
  rank_by = "auc",
  verbose = FALSE
)
```

Export the binary report:

```r
binary_export <- export_binary_results(
  bin_fit,
  output_dir = "results/final_binary",
  formats = c("png", "pdf"),
  top_n = 20,
  publication = TRUE,
  summary_language = "bilingual"
)
```

The binary report writes:

| Output | Purpose |
| --- | --- |
| `single_models.csv` | Full binary base-model LOOCV table |
| `combinations.csv` | Single-model and two-model probability-combination ranking |
| `predicted_probabilities.csv` | Per-sample final probabilities and classes |
| `threshold_metrics.csv` | Youden and fixed 0.5 threshold metrics |
| `roc_curve.csv`, `pr_curve.csv` | ROC and precision-recall coordinates |
| `calibration_curve.csv`, `dca_curve.csv` | Calibration and decision-curve diagnostics |
| `feature_importance.csv` | Median-ablation feature importance |
| `model_performance_forest.csv` | OOF ROC AUC with approximate 95% CI |
| `preprocessing_report.csv` | Missingness / variance filtering and imputation audit |
| `resampling_manifest.csv` | Fold membership used for OOF probability estimation |

Interpretation checklist:

- ROC AUC is the default selection metric.
- Exported diagnostics use out-of-fold probabilities by default; apparent
  probabilities are still exported for audit.
- Multi-class outcomes are rejected unless `collapse_other = TRUE` is set
  intentionally.
- PR-AUC, balanced accuracy, calibration, DCA, and confusion matrix are
  diagnostics for interpretation and threshold review.
- Threshold-dependent metrics should be interpreted with the selected threshold
  and should not replace AUC-based model screening by default.

## 10. Continuous and Ordinal Outcome Workflows

Continuous and ordinal outcomes use the same model-combination grammar as the
binary workflow, but change the target, primary metric, and report figures.

### Continuous Outcome

Use this when the endpoint is numeric, such as a lab value, score, expression
signature, imaging measurement, or continuous clinical phenotype.

```r
d <- mtcars
d$sample_id <- rownames(mtcars)
d$cohort <- rep(c("A", "B"), length.out = nrow(d))

prep_cont <- prepare_continuous_cohort_input(
  d,
  cohort = "cohort",
  outcome = "mpg",
  id = "sample_id"
)

params_cont <- continuousmlr_parameters(
  resampling = "kfold",
  k_folds = 5,
  verbose = FALSE
)

count_continuous_combinations(params_cont, min_size = 1, max_size = 2)

cont_fit <- fit_continuous_ensemble(
  prep_cont,
  params = params_cont,
  min_models = 1,
  max_models = 2,
  rank_by = "rmse"
)

cont_export <- export_continuous_results(
  cont_fit,
  output_dir = "results/final_continuous",
  formats = c("png", "pdf"),
  publication = TRUE
)
```

The continuous report writes single-model rankings, combination rankings,
per-sample predictions, cohort-level RMSE / MAE / R-squared diagnostics,
observed-vs-predicted plots, residual distributions, feature importance,
preprocessing audits, and an HTML report.

### Ordinal Outcome

Use this when the endpoint has a natural order, such as grade I/II/III,
low/intermediate/high response, or ordered toxicity categories. Always provide
`ordered_levels` unless the outcome is already an ordered factor.

```r
data(iris)
iris$cohort <- rep(c("A", "B", "C"), length.out = nrow(iris))
iris$sample_id <- paste0("iris_", seq_len(nrow(iris)))

prep_ord <- prepare_ordinal_cohort_input(
  iris,
  cohort = "cohort",
  outcome = "Species",
  id = "sample_id",
  ordered_levels = c("setosa", "versicolor", "virginica")
)

params_ord <- ordinalmlr_parameters(
  resampling = "kfold",
  k_folds = 5,
  verbose = FALSE
)

count_ordinal_combinations(params_ord, min_size = 1, max_size = 2)

ord_fit <- fit_ordinal_ensemble(
  prep_ord,
  params = params_ord,
  min_models = 1,
  max_models = 2,
  rank_by = "qwk"
)

ord_export <- export_ordinal_results(
  ord_fit,
  output_dir = "results/final_ordinal",
  formats = c("png", "pdf"),
  publication = TRUE
)
```

The ordinal report writes QWK rankings, predicted ordered classes, cohort
performance, confusion matrices, observed-score plots, feature importance,
preprocessing audits, and an HTML report. QWK is the primary screening metric;
accuracy, balanced accuracy, class MAE, score RMSE, and Spearman correlation
are diagnostics.

## 11. Optional Extreme Screening

Extreme screening estimates an optimistic apparent upper bound and then tests
only the apparent top combinations across many train/validation seeds.

This mode is useful for exploring the best achievable split-specific result,
but the apparent stage should not be interpreted as external validation.

```r
extreme <- extreme_surv_screen(
  prep,
  params = automlr_parameters(verbose = FALSE),
  top_n = 5,
  seeds = 1:500,
  train_fraction = 0.70,
  min_models = 1,
  max_models = 2,
  weight_method = "cindex",
  allow_same_algorithm = FALSE,
  min_risk_sd = 1e-8,
  rank_by = "apparent_cindex",
  verbose = FALSE
)
```

Export the extreme-screening report:

```r
extreme_export <- export_extreme_screen_results(
  extreme,
  output_dir = "results/final_extreme",
  formats = c("png", "pdf"),
  top_n = 10,
  top_seed_rows = 30,
  dpi = 300,
  summary_language = "bilingual"
)
```

## 12. Read the Extreme-Screening Figures

### 11.1 Apparent Top C-Index

![Extreme apparent top C-index](figures/extreme_apparent_top_cindex.png)

This figure shows the top apparent full-data C-index results. It answers:
"What is the optimistic apparent ceiling on this dataset?"

Interpretation checklist:

- Treat this as an upper-bound exploration.
- Do not report apparent C-index as external validation.
- Use the apparent top-N only to decide which candidates enter seed search.

### 11.2 Seed Validation Distribution

![Seed validation distribution](figures/extreme_seed_validation_distribution.png)

This figure shows how each apparent top combination behaves across many
train/validation splits.

Interpretation checklist:

- Prefer models with both a high maximum and a reasonable distribution.
- A single outstanding seed may reflect a favorable split.
- Compare medians and spread, not only the best seed.

### 11.3 Seed-by-Combination Heatmap

![Seed-combination heatmap](figures/extreme_seed_combination_heatmap.png)

The heatmap shows validation C-index across seeds and top combinations.

Interpretation checklist:

- Stable horizontal bands suggest repeatable performance.
- Sparse high values suggest seed sensitivity.
- Missing or failed cells should be checked in `failed_seed_search_rows.csv`.

### 11.4 Best Validation C-Index by Seed

![Best validation by seed](figures/extreme_best_validation_by_seed.png)

This line plot shows the best validation result per seed. The top seed is
useful for audit and reproducibility, but it should be interpreted together
with the full seed distribution.

## 13. Current Default Parameters

The user-facing default table is installed with the package:

```r
system.file("parameter_defaults.md", package = "AutoMLR")
```

The key defaults are:

| Step | Parameter | Default |
| --- | --- | --- |
| Regular ensemble | `min_models` | `1` |
| Regular ensemble | `max_models` | `2` |
| Time labels | `time_unit` | `"auto"` |
| Regular export | `top_n` | `20` |
| Regular export | `summary_language` | `"bilingual"` |
| Extreme screening | `top_n` | `5` |
| Extreme screening | `seeds` | `1:500` |
| Extreme screening | `train_fraction` | `0.70` |
| Extreme export | `top_seed_rows` | `30` |

## 14. Recommended Reporting Language

For manuscripts or internal reports, state the workflow clearly:

```text
Candidate survival models were evaluated by leave-one-out cross-validated
Harrell C-index. Single-model and two-model combinations were ranked primarily
by C-index. Cohort-level C-index, time-dependent AUC, Kaplan-Meier risk
stratification with number-at-risk tables, model failure rate, risk-score
spread, feature importance, SHAP-style contribution diagnostics, calibration,
decision-curve analysis, and model C-index forest summaries were exported as
diagnostic outputs. Time-to-event horizons were labelled according to the
specified survival-time unit. Extreme-screening apparent performance was used
only for upper-bound exploration and was not interpreted as external
validation.
```

## 15. Troubleshooting

| Symptom | Likely cause | Suggested action |
| --- | --- | --- |
| No shared features | Cohorts have incompatible feature names | Harmonize feature IDs before `prepare_cohort_input()` |
| Many failed folds | Model is unstable for small sample size or feature scale | Reduce algorithms, screen features, or inspect failure messages |
| Best seed looks unusually high | Favorable train/validation split | Inspect seed distribution and `best_by_seed.csv` |
| timeROC files are empty | `timeROC` unavailable or too few events | Install `timeROC` and check event counts |
| ROC or nomogram horizon labels look wrong | Survival time unit was not specified | Set `time_unit = "days"`, `"months"`, or `"years"` in `automlr_parameters()` |
| Combination does not beat single model | Single model is already strongest | Prefer simpler model unless stability improves |

## 16. Tutorial Style References

This tutorial structure was designed after reviewing several strong R
biomedical visualization and documentation resources:

- IOBR book: <https://iobr.github.io/book/>
- ggtree book: <https://guangchuangyu.github.io/ggtree-book/chapter-ggtree.html>
- ComplexHeatmap reference book: <https://jokergoo.github.io/ComplexHeatmap-reference/book/>
- Biomedical Knowledge Mining book: <https://yulab-smu.top/biomedical-knowledge-mining-book/>

The main lessons adopted here are: a pipeline-first layout, standalone code
blocks, figure-centered interpretation, explicit parameter defaults, and a
short "how to read this result" guide after each major output.
