---
title: "How to perform extra (to MCMC fitting) calculus using runMCMCbtadjust: with a focus on the possibilities offered by Nimble"
author: "Frédéric Gosselin"
email: "frederic.gosselin@inrae.fr"
date: "`r Sys.Date()`"
output: rmarkdown::html_vignette
bibliography: references.bib
csl: ecology.csl 
vignette: >
  %\VignetteEngine{knitr::knitr}
  %\VignetteIndexEntry{How to perform extra (to MCMC fitting) calculus using runMCMCbtadjust: with a focus on the possibilities offered by Nimble}
  %\usepackage[UTF-8]{inputenc}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
  collapse = TRUE,
  comment = "#>"
)
knitr::opts_chunk$set(message = FALSE)
knitr::opts_chunk$set(warnings = FALSE)

### first calculates conditional (logical) variables to activate - or not - child documents whether the environment is OK to build this vignette
condition_rstan<-TRUE
if (nchar(system.file(package='rstan'))==0) {condition_rstan<-FALSE}

condition_nimble<-TRUE   
	  if (nchar(system.file(package='nimble'))==0) {condition_nimble<-FALSE}

condition_parallel<-TRUE
if (nchar(system.file(package='parallel'))==0) {condition_parallel<-FALSE}


```

# Introduction

This file is meant to present the possibilities of the function `runMCMC_btadjust` in the `runMCMCbtadjust` package to perform calculus at the end of MCMC fitting a priori only under Windows at present (cf. <https://groups.google.com/g/nimble-users/c/MPpY4Y5NIgk>). This is offered by the component `extraCalculations` of `control.MCMC`. We here propose to calculate mode-type DIC as well as a sampled posterior GOF p-value at the end of the MCMC fit to give examples of the use of this component.

We do it by using the possibilities of `NIMBLE` to do so. At present, `runMCMC_btadjust` also allows to run MCMC with `JAGS` or `greta`. Extra (i.e. post MCMC) calculations for `greta` are as easier to be performed after running `runMCMC_btadjust` than inside it, while I am not aware of such possibilities with `JAGS`: I therefore here present this possibility only for `NIMBLE`.

To do these extra calculations, we will **write R commands within an expression that take profit of objects already built within `runMCMC_btadjust`**. In particular, the **current R Nimble model** - obtained by the command `nimble::nimbleModel` command within `runMCMC_btadjust` - for the `i`-th chain can be referred to by **`Model[[i]]`** and its C counterpart - obtained by the command `nimble::compileNimble` command applied on **`Model[[i]]`** - by **`CModel[[i]]`** while the configured model - obtained by the command `nimble::configureMCMC` command within `runMCMC_btadjust` - for the `i`-th chain can be referred to by `ConfModel[[i]]` and the model ready to be run in terms of MCMC iterations is `CModelMCMC[[i]]` for the `i`-th chain. **Note that the treatment of `i` in the above references will differ according to whether MCMC has been parallelized or not**. We will therefore provide two examples: one without parallelization and one with.

**One very important extra object built within `runMCMC_btadjust` is the mcmc.list object containing the samples after convergence and adequate thinning, called `samplesList.temp`.**



```{r, child=if (condition_nimble & condition_rstan) "../man/fragments/Nimble_extraCalculations_DIC_fragment.Rmd"}
```


```{r, child=if (condition_nimble & condition_rstan & condition_parallel) "../man/fragments/Nimble_extraCalculations_DIC_parallel_fragment.Rmd"}
```


# Goodness-of-fit p-value

We will now give another illustration of use of the `extraCalculations` argument in `runMCMC_btadjust` to **calculate goodness-of-fit (GOF) p-values**. More precisely we will propose a first series of adaptable scripts to calculate **sampled posterior GOF p-values** (@Gosselin2011) to diagnose the adequacy of the fitted model relative to the data at hand. Indeed, based on `NIMBLE` possibilities, the `extraCalculations` argument constitutes a natural way of doing these calculations without having to rewrite codes of the model within R. The very principle of sampled posterior GOF p-values is to base its calculation on a single set of parameters sampled from the posterior distribution - thus not having to repeatedly feed the model with parameter values as above.

There are many discrepancy measures that can be used to calculate these p-values (see @Herpigny201518 or @Godeau2020). We will here concentrate ourselves on diagnosing the distribution of data, using in part the notion of randomized quantile residuals (@Dunn1996236) and its extension to replicated data.

The **principle of sampled posterior GOF p-values** is very simple: first, sample randomly a *unique* value of model parameters from the posterior distribution; second, simulate many values of replicated data based on the model with this unique set of parameter values; second, compare the discrepancy function evaluated on these replicated data values with the discrepancy function evaluated on the observed data - still with this unique set of parameter values if the discrepancy function also depends on statistical parameters; third, construct an empirical p-value synthesizing these comparisons. @Gosselin2011 showed that, whatever the discrepancy function, this p-value is expected to follow a uniform distribution between 0 and 1 when the statistical model is the model that was used to generate the data - i.e. is the "true" model - which is not the case with the more classical- posterior predictive p-value (@Robins20001143) - which is sometimes referred to as the "Bayesian p-value".



```{r, child=if (condition_nimble & condition_rstan) "../man/fragments/Nimble_extraCalculations_GOF_fragment.Rmd"}
```


```{r, child=if (condition_nimble & condition_rstan & condition_parallel) "../man/fragments/Nimble_extraCalculations_GOF_parallel_fragment.Rmd"}
```


# Discussion

An important property we have not stated regarding the `extraCalculations` argument in `runMCMC_btadjust` is that if the expression it contains produces an error, `runMCMC_btadjust` will not be stopped and the MCMC result will still be given. The only impact is that the `extra` component in the attributes of the resulting object will contain an error, and not what was intended to be calculated. In such a case, if the MCMC has been long to obtain, if the output contains all the required parameters, and if some debugging is necessary, it will be possible to keep the first MCMC result and run a second call to `runMCMC_btadjust` with a very low `niter.max` parameter and the corrected code in `extraCalculations`, now not with a reference to `samplesList.temp` which will now be nearly void and non-informative but to the output of the first call to `runMCMC_btadjust`.

As we have stated above, the two examples we have shown above - calculations of information criteria and calculation of goodness-of-fit p-values - come with different levels of generality of the code we proposed: while the IC code should adapt to nearly every situation where mode-type DIC and classical DIC are wished, the GOF p-values calculations will have to be adapted to every model, first by specifying the objects (data or parameters) that have to be replicated and second the discrepancy functions used in the GOF p-values. IC calculations could in the next versions be put in hard code of `runMCMC_btadjust` (as WAIC was done in `NIMBLE`). At present, we keep it separate and propose to calculate it through additional code in `extraCalculations`. It also remains to be seen if use of - `NIMBLE` - online WAIC calculation - which at present requires extra MCMC sampling on one chain and thus can be numerically demanding - could be replaced by calculations with `extraCalculations` on the already fitted samples - thus removing the requirement of extra MCMC sampling for WAIC calculation.

Of course, the calculations we propose to do through `extraCalculations` could also be done out of `runMCMC_btadjust`: the user would then need to rebuild the environment necessary to do it - especially the result of `nimbleModel` - which is not so difficult. Actually, I used to do these calculations both outside `runMCMC_btadjust` but also with non-`NIMBLE` functions - e.g. having to rewrite the log probabilities in R or C a second time. A first advantage of what I propose above is to use the `NIMBLE` objects to do so, first sparing some time to the analyst and second ensuring that the models/codes are well the ones that are wished and tha    t were used for model fitting. A second advantage of doing this inside `runMCMC_btadjust`, is in the case we use the `includeAllStochNodes=TRUE` option without saving all these nodes in the output - e.g. when these nodes are numerous:  then the approach proposed allows to spare some bytes in the ensuing RData files.

# Conclusion

We have here shown the capacities provided by the `extraCalculations` argument in `runMCMC_btadjust` which allows the analyst to use `runMCMC_btadjust` not only to fit a MCMC rigorously - which is its original aim - but also to provide other important information related to the model, such as information criteria or goodness-of-fit p-values. This enables us to provide a more rigorous use of MCMCs, first both to ensure convergence is reached and we have a sufficient number of quasi-independent samples from the posterior distribution but also to provide tools to compare the model with other models fit on the same data - information criteria - and to diagnose the coherence of the model with data - with GOF p-values.

We insist that these possibilities have been developed and tested under Windows and that in some Linux environments these procedures might encounter problems that we hope we will solve soon (cf. <https://groups.google.com/g/nimble-users/c/MPpY4Y5NIgk>).

# Acknowledgements
I wished to thank the Nimble users mailing list.

The initial development of this package (up to version 1.1.1) was performed within the GAMBAS project funded by the French Agence Nationale pour la Recherche (ANR-18-CE02-0025) (cf. <https://gambas.cirad.fr/>).

<img src="../man/figures/logo_gambas.jpg">

# References
