Author: Sebastian Weber

# Simulation based calibration for RBesT

#### Sebastian Weber

#### Fri Mar 13 09:17:05 2026

This report documents the results of a simulation based calibration (SBC) run for `RBesT`. The calibration data will be generated whenever relevant changes to the `gMAP` function were made. The calibration runs are performed for typical use cases of `gMAP`. These include the three likelihoods (binomial, gaussian & Poisson), a sparse (\(2\) trials) and dense (\(10\) trials) data situation and finally a run with a very/less conservative prior choice for between-trial heterogeniety parameter.

The calibration data presented here has been generated at and with the `RBesT` git version as:

    ## Created:  2026-01-30 09:26:46 UTC
    ## git hash: 359c1b3e61d46456784ec84787fae4777a16159b
    ## MD5:      8b3061114af13b65704d8ed12a4087ae

The MD5 hash of the calibration data file presented here must match the above listed MD5:

    ## /home/runner/work/RBesT/RBesT/inst/sbc/calibration.rds
    ##                     "8b3061114af13b65704d8ed12a4087ae"

# Introduction

Simulation based calibration (SBC) is a necessary condition which must be met for any Bayesian analysis with proper priors. The details are presented in Talts, et. al (see https://arxiv.org/abs/1804.06788).

Self-consistency of any Bayesian analysis with a proper prior:

\[ p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta|\tilde{y}) \, p(\tilde{y}|\tilde{\theta}) \, p(\tilde{\theta}) \] \[ \Leftrightarrow p(\theta) = \iint \mbox{d}\tilde{y} \, \mbox{d}\tilde{\theta} \, p(\theta,\tilde{y},\tilde{\theta}) \]

SBC procedure:

Repeat \(s=1, ..., S\) times:

1. Sample from the prior \[\tilde{\theta} \sim p(\theta)\]

2. Sample fake data \[\tilde{y} \sim p(y|\tilde{\theta})\]

3. Obtain \(L\) posterior samples \[\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\]

4. Calculate the rank \(r_s\) of the prior draw \(\tilde{\theta}\) wrt to the posterior sample \(\{\theta_1, ..., \theta_L\} \sim p(\tilde{\theta}|\tilde{y})\) which falls into the range \([0,L]\) out of the possible \(L+1\) ranks. The rank is calculated as \[r_s = \sum_{l=1}^L \mathbb{I}[ \theta_l < \tilde{\theta}]\]

The \(S\) ranks then form a uniform \(0-1\) density and the count in each bin has a binomial distribution with probability of \[p(r \in \mbox{Any Bin}) =\frac{(L+1)}{S}.\]

## Hierarchical intercept only (random-effects intercept) model for binomial, gaussian and Poisson likelihood

Likelihood:

- Binary \[y_i|\theta_{j} \sim \mbox{Bernoulli}(\theta_j), \] \[g(\theta) = \mbox{logit}(\theta)\]
- Normal \[y_i|\theta_{j} \sim \mbox{Normal}(\theta_j, \sigma^2), \] \[g(\theta) = \theta\]
- Poisson \[y_i|\theta_{j} \sim \mbox{Poisson}(\theta_j), \] \[g(\theta) = \log(\theta)\]

Hierarchical prior:

\[ g(\theta_j)|\mu,\tau \sim \mbox{Normal}(\mu, \tau^2)\]

\[\mu \sim \mbox{Normal}(m_\mu, s^2_\mu)\] \[\tau \sim \mbox{Normal}^+(0, s^2_\tau)\]

The fake data simulation function returns for binomial and Poisson data the sum of the responses while for normal the mean summary is used. Please refer to the `sbc_tools.R` and `make_reference_rankhist.R` R programs for the implementation details.

The reference runs are created with \(L=1023\) posterior draws for each replication and a total of \(S=10^4\) replications are run per case. For the evaluation here the results are reduced to \(B=L'+1=64\) bins to ensure a sufficiently large sample size per bin.

# SBC results

## Sampler Diagnostics Overview

family | data_scenario | sd_tau | N | total_divergent | total_divergent_sim_fraction | min_ess | max_Rhat | total_large_Rhat | min_lp_ess_bulk | min_lp_ess_tail
--- | --- | --- | --- | --- | --- | --- | --- | --- | --- | ---
binomial | dense | 0.5 | 10000 | 267 | 0.022 | 86 | 1.029 | 0 | 234 | 215
binomial | dense | 1.0 | 10000 | 411 | 0.029 | 37 | 1.042 | 0 | 145 | 172
binomial | sparse | 0.5 | 10000 | 1218 | 0.056 | 9 | 1.217 | 1 | 19 | 9
binomial | sparse | 1.0 | 10000 | 2438 | 0.109 | 7 | 1.473 | 1 | 13 | 43
gaussian | dense | 0.5 | 10000 | 409 | 0.028 | 34 | 1.063 | 0 | 211 | 111
gaussian | dense | 1.0 | 10000 | 432 | 0.022 | 23 | 1.103 | 0 | 106 | 72
gaussian | sparse | 0.5 | 10000 | 22175 | 0.466 | 3 | 1.840 | 1 | 11 | 21
gaussian | sparse | 1.0 | 10000 | 41402 | 0.399 | 2 | 3.895 | 1 | 5 | 17
poisson | dense | 0.5 | 10000 | 781 | 0.059 | 101 | 1.027 | 0 | 155 | 168
poisson | dense | 1.0 | 10000 | 1839 | 0.112 | 12 | 1.233 | 1 | 100 | 67
poisson | sparse | 0.5 | 10000 | 23669 | 0.399 | 2 | 2.702 | 1 | 6 | 16
poisson | sparse | 1.0 | 10000 | 35266 | 0.408 | 2 | 11.114 | 1 | 4 | 16

Note: Large Rhat is defined as exceeding 1.2.

# Summary Statistics

## \(\chi^2\) Statistic, \(\mu\)

| data_scenario | likelihood | sd_tau | parameter | statistic | df | p.value
--- | --- | --- | --- | --- | --- | --- | ---
1 | dense | binomial | 0.5 | mu | 68.083 | 63 | 0.308
13 | dense | binomial | 1 | mu | 61.619 | 63 | 0.526
25 | dense | gaussian | 0.5 | mu | 73.613 | 63 | 0.170
37 | dense | gaussian | 1 | mu | 59.789 | 63 | 0.591
49 | dense | poisson | 0.5 | mu | 38.310 | 63 | 0.994
61 | dense | poisson | 1 | mu | 57.946 | 63 | 0.657
73 | sparse | binomial | 0.5 | mu | 62.106 | 63 | 0.508
78 | sparse | binomial | 1 | mu | 68.762 | 63 | 0.289
83 | sparse | gaussian | 0.5 | mu | 44.467 | 63 | 0.963
88 | sparse | gaussian | 1 | mu | 63.411 | 63 | 0.462
93 | sparse | poisson | 0.5 | mu | 60.608 | 63 | 0.562
98 | sparse | poisson | 1 | mu | 58.496 | 63 | 0.637

## \(\chi^2\) Statistic, \(\tau\)

| data_scenario | likelihood | sd_tau | parameter | statistic | df | p.value
--- | --- | --- | --- | --- | --- | --- | ---
2 | dense | binomial | 0.5 | tau | 85.171 | 63 | 0.033
14 | dense | binomial | 1 | tau | 67.802 | 63 | 0.317
26 | dense | gaussian | 0.5 | tau | 50.214 | 63 | 0.878
38 | dense | gaussian | 1 | tau | 76.902 | 63 | 0.112
50 | dense | poisson | 0.5 | tau | 75.891 | 63 | 0.128
62 | dense | poisson | 1 | tau | 71.949 | 63 | 0.206
74 | sparse | binomial | 0.5 | tau | 66.355 | 63 | 0.362
79 | sparse | binomial | 1 | tau | 74.138 | 63 | 0.159
84 | sparse | gaussian | 0.5 | tau | 68.915 | 63 | 0.284
89 | sparse | gaussian | 1 | tau | 71.296 | 63 | 0.221
94 | sparse | poisson | 0.5 | tau | 50.688 | 63 | 0.868
99 | sparse | poisson | 1 | tau | 55.130 | 63 | 0.749

## \(\chi^2\) Statistic, group estimates \(\theta\)

| data_scenario | likelihood | sd_tau | parameter | statistic | df | p.value
--- | --- | --- | --- | --- | --- | --- | ---
3 | dense | binomial | 0.5 | theta[10] | 42.445 | 63 | 0.978
4 | dense | binomial | 0.5 | theta[1] | 57.523 | 63 | 0.671
5 | dense | binomial | 0.5 | theta[2] | 64.397 | 63 | 0.427
6 | dense | binomial | 0.5 | theta[3] | 49.984 | 63 | 0.883
7 | dense | binomial | 0.5 | theta[4] | 61.875 | 63 | 0.516
8 | dense | binomial | 0.5 | theta[5] | 51.302 | 63 | 0.854
9 | dense | binomial | 0.5 | theta[6] | 69.696 | 63 | 0.263
10 | dense | binomial | 0.5 | theta[7] | 72.550 | 63 | 0.192
11 | dense | binomial | 0.5 | theta[8] | 74.035 | 63 | 0.161
12 | dense | binomial | 0.5 | theta[9] | 60.864 | 63 | 0.553
15 | dense | binomial | 1 | theta[10] | 60.531 | 63 | 0.565
16 | dense | binomial | 1 | theta[1] | 65.690 | 63 | 0.384
17 | dense | binomial | 1 | theta[2] | 65.318 | 63 | 0.396
18 | dense | binomial | 1 | theta[3] | 54.170 | 63 | 0.778
19 | dense | binomial | 1 | theta[4] | 69.581 | 63 | 0.266
20 | dense | binomial | 1 | theta[5] | 83.200 | 63 | 0.045
21 | dense | binomial | 1 | theta[6] | 58.253 | 63 | 0.646
22 | dense | binomial | 1 | theta[7] | 40.448 | 63 | 0.988
23 | dense | binomial | 1 | theta[8] | 56.768 | 63 | 0.697
24 | dense | binomial | 1 | theta[9] | 53.389 | 63 | 0.801
27 | dense | gaussian | 0.5 | theta[10] | 47.949 | 63 | 0.920
28 | dense | gaussian | 0.5 | theta[1] | 72.435 | 63 | 0.195
29 | dense | gaussian | 0.5 | theta[2] | 56.435 | 63 | 0.708
30 | dense | gaussian | 0.5 | theta[3] | 72.000 | 63 | 0.205
31 | dense | gaussian | 0.5 | theta[4] | 59.110 | 63 | 0.616
32 | dense | gaussian | 0.5 | theta[5] | 71.283 | 63 | 0.222
33 | dense | gaussian | 0.5 | theta[6] | 82.266 | 63 | 0.052
34 | dense | gaussian | 0.5 | theta[7] | 70.195 | 63 | 0.249
35 | dense | gaussian | 0.5 | theta[8] | 66.099 | 63 | 0.370
36 | dense | gaussian | 0.5 | theta[9] | 54.950 | 63 | 0.755
39 | dense | gaussian | 1 | theta[10] | 80.026 | 63 | 0.073
40 | dense | gaussian | 1 | theta[1] | 57.421 | 63 | 0.675
41 | dense | gaussian | 1 | theta[2] | 53.184 | 63 | 0.806
42 | dense | gaussian | 1 | theta[3] | 38.810 | 63 | 0.993
43 | dense | gaussian | 1 | theta[4] | 57.190 | 63 | 0.682
44 | dense | gaussian | 1 | theta[5] | 61.069 | 63 | 0.545
45 | dense | gaussian | 1 | theta[6] | 46.310 | 63 | 0.943
46 | dense | gaussian | 1 | theta[7] | 52.122 | 63 | 0.834
47 | dense | gaussian | 1 | theta[8] | 66.381 | 63 | 0.361
48 | dense | gaussian | 1 | theta[9] | 70.566 | 63 | 0.240
51 | dense | poisson | 0.5 | theta[10] | 62.707 | 63 | 0.487
52 | dense | poisson | 0.5 | theta[1] | 89.216 | 63 | 0.017
53 | dense | poisson | 0.5 | theta[2] | 56.806 | 63 | 0.695
54 | dense | poisson | 0.5 | theta[3] | 63.142 | 63 | 0.471
55 | dense | poisson | 0.5 | theta[4] | 60.442 | 63 | 0.568
56 | dense | poisson | 0.5 | theta[5] | 63.066 | 63 | 0.474
57 | dense | poisson | 0.5 | theta[6] | 67.763 | 63 | 0.318
58 | dense | poisson | 0.5 | theta[7] | 50.342 | 63 | 0.876
59 | dense | poisson | 0.5 | theta[8] | 67.661 | 63 | 0.321
60 | dense | poisson | 0.5 | theta[9] | 55.565 | 63 | 0.736
63 | dense | poisson | 1 | theta[10] | 78.630 | 63 | 0.089
64 | dense | poisson | 1 | theta[1] | 52.211 | 63 | 0.832
65 | dense | poisson | 1 | theta[2] | 57.728 | 63 | 0.664
66 | dense | poisson | 1 | theta[3] | 65.344 | 63 | 0.395
67 | dense | poisson | 1 | theta[4] | 59.686 | 63 | 0.595
68 | dense | poisson | 1 | theta[5] | 70.669 | 63 | 0.237
69 | dense | poisson | 1 | theta[6] | 53.184 | 63 | 0.806
70 | dense | poisson | 1 | theta[7] | 84.493 | 63 | 0.037
71 | dense | poisson | 1 | theta[8] | 46.413 | 63 | 0.942
72 | dense | poisson | 1 | theta[9] | 101.338 | 63 | 0.002
75 | sparse | binomial | 0.5 | theta[1] | 62.106 | 63 | 0.508
76 | sparse | binomial | 0.5 | theta[2] | 63.014 | 63 | 0.476
77 | sparse | binomial | 0.5 | theta[3] | 52.403 | 63 | 0.827
80 | sparse | binomial | 1 | theta[1] | 64.051 | 63 | 0.439
81 | sparse | binomial | 1 | theta[2] | 50.688 | 63 | 0.868
82 | sparse | binomial | 1 | theta[3] | 71.949 | 63 | 0.206
85 | sparse | gaussian | 0.5 | theta[1] | 59.290 | 63 | 0.609
86 | sparse | gaussian | 0.5 | theta[2] | 59.981 | 63 | 0.585
87 | sparse | gaussian | 0.5 | theta[3] | 72.794 | 63 | 0.187
90 | sparse | gaussian | 1 | theta[1] | 76.506 | 63 | 0.118
91 | sparse | gaussian | 1 | theta[2] | 90.880 | 63 | 0.012
92 | sparse | gaussian | 1 | theta[3] | 61.235 | 63 | 0.539
95 | sparse | poisson | 0.5 | theta[1] | 72.794 | 63 | 0.187
96 | sparse | poisson | 0.5 | theta[2] | 62.016 | 63 | 0.511
97 | sparse | poisson | 0.5 | theta[3] | 73.830 | 63 | 0.165
100 | sparse | poisson | 1 | theta[1] | 61.158 | 63 | 0.542
101 | sparse | poisson | 1 | theta[2] | 64.589 | 63 | 0.421
102 | sparse | poisson | 1 | theta[3] | 47.718 | 63 | 0.924

## Session Info

    ## R version 4.5.3 (2026-03-11)
    ## Platform: x86_64-pc-linux-gnu
    ## Running under: Ubuntu 24.04.3 LTS
    ##
    ## Matrix products: default
    ## BLAS:   /usr/lib/x86_64-linux-gnu/openblas-pthread/libblas.so.3
    ## LAPACK: /usr/lib/x86_64-linux-gnu/openblas-pthread/libopenblasp-r0.3.26.so;  LAPACK version 3.12.0
    ##
    ## locale:
    ##  [1] LC_CTYPE=C.UTF-8       LC_NUMERIC=C           LC_TIME=C.UTF-8
    ##  [4] LC_COLLATE=C.UTF-8     LC_MONETARY=C.UTF-8    LC_MESSAGES=C.UTF-8
    ##  [7] LC_PAPER=C.UTF-8       LC_NAME=C              LC_ADDRESS=C
    ## [10] LC_TELEPHONE=C         LC_MEASUREMENT=C.UTF-8 LC_IDENTIFICATION=C
    ##
    ## time zone: UTC
    ## tzcode source: system (glibc)
    ##
    ## attached base packages:
    ## [1] tools     stats     graphics  grDevices utils     datasets  methods
    ## [8] base
    ##
    ## other attached packages:
    ##  [1] purrr_1.2.1         rstan_2.32.7        StanHeaders_2.32.10
    ##  [4] here_1.0.2          ggplot2_4.0.2       broom_1.0.12
    ##  [7] tidyr_1.3.2         dplyr_1.2.0         assertthat_0.2.1
    ## [10] knitr_1.51
    ##
    ## loaded via a namespace (and not attached):
    ##  [1] sass_0.4.10           generics_0.1.4        digest_0.6.39
    ##  [4] magrittr_2.0.4        evaluate_1.0.5        grid_4.5.3
    ##  [7] RColorBrewer_1.1-3    fastmap_1.2.0         rprojroot_2.1.1
    ## [10] jsonlite_2.0.0        pkgbuild_1.4.8        backports_1.5.0
    ## [13] gridExtra_2.3         QuickJSR_1.9.0        scales_1.4.0
    ## [16] codetools_0.2-20      jquerylib_0.1.4       cli_3.6.5
    ## [19] rlang_1.1.7           withr_3.0.2           cachem_1.1.0
    ## [22] yaml_2.3.12           otel_0.2.0            parallel_4.5.3
    ## [25] inline_0.3.21         vctrs_0.7.1           R6_2.6.1
    ## [28] matrixStats_1.5.0     stats4_4.5.3          lifecycle_1.0.5
    ## [31] pkgconfig_2.0.3       RcppParallel_5.1.11-2 pillar_1.11.1
    ## [34] bslib_0.10.0          gtable_0.3.6          loo_2.9.0
    ## [37] glue_1.8.0            Rcpp_1.1.1            xfun_0.56
    ## [40] tibble_3.3.1          tidyselect_1.2.1      farver_2.1.2
    ## [43] htmltools_0.5.9       rmarkdown_2.30        compiler_4.5.3
    ## [46] S7_0.2.1