# User defined validation plan

### R package `niaidMI`

# 1 Purpose of this document

This document describes the “user defined validation plan” as described in “Develop, Validate, and Execute Code”. Thus, it documents the steps to be taken to reasonably ensure this package functions according to specifications. The function documentation serves as the specifications, and details of the formulas and methods may be found in the KHAA SAP.

# 2 Rationale

Since this is a package with limited scope that will be installed and/or used by other applications, many of the supporting deliverables that are typical for computer system validation are not applicable. The Security plan is not required, because any security functionality to control use of the package is the responsibility of the application using the package. Access to the code is controlled by GitHub. Furthermore, the package will rely on GitHub Backup/Restoration and Disaster Recovery of the source code. Applications that incorporate the package are responsible for their own Disaster Recovery and Backup/Restoration processes. The Business Continuity Plan is the responsibility of the application that incorporates the package. This package may be placed on CRAN.

# 3 Roles

The following roles will be involved in validation:

- Code Author: Nathan Morris
- Second Progammer: Chaoran Hu

# 4 Methods overview and important internal functions

The detailed demonstration of multiple imputation with hidden Markov model (MI-HMM) can be found in the KHAA SAP. We briefly review the major steps of MI-HMM as the following algorithm.

1. Generate \(M\) bootstrap data from original data;

2. Fit (`niaidMI:::.baum_welsh`) HMM to each bootstrap data and get estimate \(\theta_m\), \(m=1,...,M\);

3. Impute (`niaidMI:::.impute_one`) original data with \(P(missing|obsvered, \theta_m)\) to provide \(M\) imputations.

The start value for estimation (Baun-Welch algorithm) is evaluating as empirical transition matrix and initial probability from original data.

# 5 Validation plan

Two critical working blocks in R package`niaidMI` are `niaidMI:::.baum_welsh` that estimates the unknown parameters in HMM with Baun-Welch algorithm and `niaidMI:::.impute_one` that impute the missing values with estimated HMM. So, our validation plan will also focus on these two working blocks. Additionally, functions’ capability of handling edge cases will also be tested. The validation process is automated by `testthat`.

## 5.1 Testing of edge cases

Many cases of clearly incorrect inputs are automatically tested in this package to ensure the package correctly throws errors and warnings. These include edge cases for the following 2 user-friendly functions:

- bootstrap_param_est()
- impute()

R code is available in “/niaidMI/tests/testthat/test-edge-cases.R”.

## 5.2 Testing of `niaidMI:::.baum_welsh`

Data used in testing is simulated by `niaidMI::sim_data()`. Double programming was used to demonstrate that results from the model fit including both parameter estimates and the log-likelihood calculations yield the same results. Considering the nature of floating point number, equal is defined as two numbers difference is less than `sqrt(.Machine$double.eps)=1.490116e-08`, that is the default value in `base::all.equal`.

R code is available in “/niaidMI/tests/testthat/test-est.R”.

## 5.3 Testing of `niaidMI:::.impute_one`

`niaidMI:::.impute_one` is tested by double programming in the sense that independent implementations from code author and second programmer provide the identical results when using the same transition matrix and initial probability of HMM. In our testing, a data with 200 patients and 28 days observations is generated by `niaidMI::sim_data()` with dropout rate 0.02 and sporatic rate 0.2, which yield around 1825 missing values. 100 imputations are generated by code author and second programmer’s implementations separately. In this testing, equal is defined as identical because Niaid-OS is a 8-levels ordered scale.

R code is available in “/niaidMI/tests/testthat/test-impute.R”.

## 5.4 Limitation of this validation plan

Validation will not be processed for high-level functions in `niaidMI` but just two most important working blocks for two reasons. First, similar to other imputation methods, our method includes significant number of random number generators. The common approach to keep result reproducible is using random number seed `set.seed`. However, one disadvantage of random number seed in R is its sensitivity to the order of random number generators. We provide a toy example here.

    foo1 <- function()
    {
        x <- rnorm(1)
        x + runif(1)
    }

    foo2 <- function()
    {
        x <- runif(1)
        x + rnorm(1)
    }

    set.seed(123)
    foo1()

    ## [1] -0.1514987

    set.seed(123)
    foo2()

    ## [1] 1.088132

Second, in the MI-HMM method, the `niaidMI:::.impute_one` function will take the estimate result from `niaidMI:::.baum_welsh` as input. However, as previously mentioned, the code author and second programmer’s estimation function can provide the same results only up to some numerical floating point accuracy. However, this tiny numerical difference in estimate will be sufficent to cause a few imputations to differ between the two programs. In practice, we are confident this will not have a meaningful impact. According to our simulation result, for the same simulated data used in previous section using our chosen random seed, only 0.1% missing values were imputed differently. Reproducible code is provided in “niaidMI/inst/validation_code_arxiv/test-imp-explorer.R” that served as support of this statement (second reason).

Finally, we note that the primary implementation of the package was used in a rigorous simulation study demonstrating that its statistical properties including bias and coverage are appropriate.

# 6 Revisions

## 6.1 Version control

The source code for this package is stored in a secured GitHub repository. Semantic versioning (https://semver.org/) will be utilized for version numbering starting with version 0.1.0.

## 6.2 Revise code

When changes are made to the code, the SOPs should be followed. Since the code is in a version controlled repository, all code changes and the rationale for the modifications are traceable with GitHub. The code will be reviewed for accuracy when a new revision of the package is released, per the automated processes described above.

## 6.3 Periodic review strategy

Since this is an R package and not a full application, it will have a limited periodic review. The validation documentation (including training information) will be reviewed for accuracy when a new revision of the package is released. Automated testing will be performed when the updated package is built. Given the limited scope of this package, any incidents, problems, deviations, action items, and changes will be reviewed when they are occur and addressed as applicable. No additional monitoring or trending will be performed.