# Persistent Indexes and Lifecycle

`bigANNOY` v3 adds explicit index lifecycle support around persisted Annoy files. That makes it possible to:

- build an index once and reopen it later
- choose whether a reopened index should load eagerly or lazily
- check whether a native handle is currently live
- close loaded handles explicitly in long sessions
- validate the Annoy file against its recorded metadata before reuse

This vignette focuses on those operational workflows rather than on search quality or benchmark tuning.

## Why Lifecycle Management Matters

Annoy indexes are stored on disk. In practice, that means the useful object is not just the result of a single build call, but a persisted pair:

- the `.ann` index file
- the `.meta` sidecar metadata file

The `bigannoy_index` object returned by `bigANNOY` is a session-level wrapper around those files. It remembers the key metadata and can optionally hold a live native handle for faster repeated searches within the same R session.

## Load the Packages

    library(bigANNOY)
    library(bigmemory)

## Build an Index in Lazy Mode

We will create a small reference matrix, write the Annoy index into a temporary directory, and keep the returned object in lazy mode so the first search is what loads the live handle.

    artifact_dir <- file.path(tempdir(), "bigannoy-lifecycle")
    dir.create(artifact_dir, recursive = TRUE, showWarnings = FALSE)

    ref_dense <- matrix(
      c(
        0.0, 0.1, 0.2,
        0.1, 0.0, 0.1,
        0.2, 0.1, 0.0,
        1.0, 1.1, 1.2,
        1.1, 1.0, 1.1,
        1.2, 1.1, 1.0
      ),
      ncol = 3,
      byrow = TRUE
    )

    ref_big <- as.big.matrix(ref_dense)
    index_path <- file.path(artifact_dir, "ref.ann")
    metadata_path <- paste0(index_path, ".meta")

    index <- annoy_build_bigmatrix(
      ref_big,
      path = index_path,
      n_trees = 25L,
      metric = "euclidean",
      seed = 123L,
      load_mode = "lazy"
    )

    index

    #> <bigannoy_index>
    #>   path: /var/folders/h9/npmqbtmx4wlblg4wks47yj5c0000gn/T//RtmpBEyDSE/bigannoy-lifecycle/ref.ann
    #>   metadata: /var/folders/h9/npmqbtmx4wlblg4wks47yj5c0000gn/T//RtmpBEyDSE/bigannoy-lifecycle/ref.ann.meta
    #>   index_id: annoy-20260327203934-2a8b6582c143
    #>   metric: euclidean
    #>   trees: 25
    #>   items: 6
    #>   dimension: 3
    #>   build_seed: 123
    #>   build_threads: -1
    #>   build_backend: cpp
    #>   load_mode: lazy
    #>   loaded: FALSE
    #>   file_size: 2968
    #>   file_md5: 2a8b6582c143e941abc77e79789e227e
    #>   prefault: FALSE

The returned object points to the persisted files, but the native handle is not loaded yet.

    annoy_is_loaded(index)

    #> [1] FALSE

    file.exists(index$path)

    #> [1] TRUE

    file.exists(index$metadata_path)

    #> [1] TRUE

## Inspect the Sidecar Metadata

The sidecar metadata file is meant to support safe reopen and validation workflows. It records the metric, dimension, item count, build settings, and a small file signature for the persisted Annoy file.

    metadata <- read.dcf(index$metadata_path)
    metadata[, c(
      "index_id",
      "metric",
      "n_dim",
      "n_ref",
      "n_trees",
      "build_seed",
      "build_backend",
      "file_size",
      "file_md5"
    )]

index_id | metric | n_dim | n_ref | n_trees | build_seed | build_backend | file_size | file_md5
--- | --- | --- | --- | --- | --- | --- | --- | ---
annoy-20260327203934-2a8b6582c143 | euclidean | 3 | 6 | 25 | 123 | cpp | 2968 | 2a8b6582c143e941abc77e79789e227e
annoy-20260327203934-2a8b6582c143 | euclidean | 3 | 6 | 25 | 123 | cpp | 2968 | 2a8b6582c143e941abc77e79789e227e
annoy-20260327203934-2a8b6582c143 | euclidean | 3 | 6 | 25 | 123 | cpp | 2968 | 2a8b6582c143e941abc77e79789e227e

The important point is not the exact formatting of the metadata file, but that the persisted index is now self-describing enough to be reopened and checked in later sessions.

## Lazy Loading Versus Eager Loading

There are two lifecycle modes:

- `"lazy"` keeps only metadata in memory until the first search
- `"eager"` loads a native handle immediately when the index object is created or reopened

The index we just built is lazy.

    annoy_is_loaded(index)

    #> [1] FALSE

The first search loads the handle automatically.

    first_result <- annoy_search_bigmatrix(index, k = 2L, search_k = 100L)

    annoy_is_loaded(index)

    #> [1] TRUE

    first_result$index

2 | 3
1 | 3
2 | 1
5 | 6
4 | 6
5 | 4

    round(first_result$distance, 3)

0.173 | 0.283
0.173 | 0.173
0.173 | 0.283
0.173 | 0.283
0.173 | 0.173
0.173 | 0.283

Once the handle is loaded, repeated searches in the same session can reuse it.

    second_result <- annoy_search_bigmatrix(index, k = 2L, search_k = 100L)

    identical(first_result$index, second_result$index)

    #> [1] TRUE

    all.equal(first_result$distance, second_result$distance)

    #> [1] TRUE

## Validate Without Loading

Validation and loading are related, but they are not the same thing. Sometimes you want to confirm that the metadata and file signature still look right without paying the cost of loading the native handle yet.

    annoy_close_index(index)
    annoy_is_loaded(index)

    #> [1] FALSE

    validation_no_load <- annoy_validate_index(
      index,
      strict = TRUE,
      load = FALSE
    )

    validation_no_load$valid

    #> [1] TRUE

    validation_no_load$checks[, c("check", "passed", "severity")]

check | passed | severity
--- | --- | ---
index_file | TRUE | error
metric | TRUE | error
dimensions | TRUE | error
items | TRUE | error
file_size | TRUE | error
file_md5 | TRUE | error
file_mtime | TRUE | warning

    annoy_is_loaded(index)

    #> [1] FALSE

Because `load = FALSE`, the validation report checks the recorded metadata against the current file without changing the loaded state of the object.

## Validate and Load Explicitly

If you do want validation to also confirm that the Annoy index can be opened successfully, set `load = TRUE`.

    validation_with_load <- annoy_validate_index(
      index,
      strict = TRUE,
      load = TRUE
    )

    validation_with_load$valid

    #> [1] TRUE

    tail(validation_with_load$checks[, c("check", "passed", "severity")], 2L)

check | passed | severity
--- | --- | ---
file_mtime | TRUE | warning
load | TRUE | error

    annoy_is_loaded(index)

    #> [1] TRUE

This is a useful pattern before long-running queries or before handing a reopened index to downstream analysis code.

## Close a Loaded Handle Explicitly

Explicit close support is helpful in long R sessions, in tests, and in code that wants deterministic control over when handles are released.

    annoy_close_index(index)
    annoy_is_loaded(index)

    #> [1] FALSE

The persisted `.ann` file is still there, so the next search can load it again.

    reload_result <- annoy_search_bigmatrix(index, k = 2L, search_k = 100L)

    annoy_is_loaded(index)

    #> [1] TRUE

    reload_result$index

2 | 3
1 | 3
2 | 1
5 | 6
4 | 6
5 | 4

## Reopen the Same Index in a New Object

The more important persistence workflow is reopening the same files into a new `bigannoy_index` object. This is what a later R session would typically do.

`annoy_open_index()` and `annoy_load_bigmatrix()` both support this pattern. The main distinction is semantic: `annoy_load_bigmatrix()` is a friendlier name when you are thinking in terms of `bigmemory` workflows, while `annoy_open_index()` makes the persisted-index lifecycle more explicit.

    reopened_lazy <- annoy_open_index(
      path = index$path,
      load_mode = "lazy"
    )

    reopened_eager <- annoy_load_bigmatrix(
      path = index$path,
      load_mode = "eager"
    )

    annoy_is_loaded(reopened_lazy)

    #> [1] FALSE

    annoy_is_loaded(reopened_eager)

    #> [1] TRUE

The eager reopen path loads immediately. The lazy reopen path waits until first use.

    reopened_result <- annoy_search_bigmatrix(
      reopened_lazy,
      k = 2L,
      search_k = 100L
    )

    annoy_is_loaded(reopened_lazy)

    #> [1] TRUE

    reopened_result$index

2 | 3
1 | 3
2 | 1
5 | 6
4 | 6
5 | 4

## Lifecycle State Lives in the Session Object

The persisted files are shared, but loaded-state tracking is per-object and per-session. Closing one in-memory object does not invalidate another object that already opened the same index.

    annoy_close_index(reopened_lazy)
    c(
      original = annoy_is_loaded(index),
      reopened_lazy = annoy_is_loaded(reopened_lazy),
      reopened_eager = annoy_is_loaded(reopened_eager)
    )

    #>       original  reopened_lazy reopened_eager
    #>           TRUE          FALSE           TRUE

This is a useful mental model:

- the `.ann` file is the durable asset
- the `bigannoy_index` object is the session-level controller
- the loaded handle is cached inside that controller only for the current session

## What Happens If Validation Fails?

In normal workflows, `annoy_validate_index(..., strict = TRUE)` is the safest default because it stops immediately when critical checks fail. If you want a diagnostic report instead of an error, use `strict = FALSE`.

    report <- annoy_validate_index(
      reopened_eager,
      strict = FALSE,
      load = FALSE
    )

    report$valid

    #> [1] TRUE

    report$checks[, c("check", "passed", "severity")]

check | passed | severity
--- | --- | ---
index_file | TRUE | error
metric | TRUE | error
dimensions | TRUE | error
items | TRUE | error
file_size | TRUE | error
file_md5 | TRUE | error
file_mtime | TRUE | warning

That pattern is especially helpful when you are writing higher-level code that wants to display a validation report before deciding whether to rebuild or reload an index.

## Recommended Workflow

For most projects, a sensible lifecycle pattern looks like this:

1. build the index once with `annoy_build_bigmatrix()`
2. keep the `.ann` file and the `.meta` file together
3. reopen with `annoy_open_index()` or `annoy_load_bigmatrix()` in later sessions
4. run `annoy_validate_index()` before important downstream work
5. use lazy loading for lighter startup or eager loading for repeated search sessions
6. call `annoy_close_index()` when you want explicit control over loaded handles

## Recap

`bigANNOY` v3 turns persisted Annoy files into a more explicit lifecycle:

- build once, reopen later
- choose eager or lazy loading
- test loaded state with `annoy_is_loaded()`
- close handles with `annoy_close_index()`
- validate persisted files with `annoy_validate_index()`

The next vignette to read after this one is usually File-Backed bigmemory Workflows, which focuses on descriptor files, file-backed matrices, and streamed output destinations.