
<!-- README.md is generated from README.Rmd. Please edit that file -->

# Revisiting Whittaker-Henderson Smoothing

<!-- badges: start -->

[![R-CMD-check](https://github.com/GuillaumeBiessy/WH/actions/workflows/R-CMD-check.yaml/badge.svg)](https://github.com/GuillaumeBiessy/WH/actions/workflows/R-CMD-check.yaml)
<!-- badges: end -->

## What is Whittaker-Henderson smoothing ?

### Origin

Whittaker-Henderson (WH) smoothing is a graduation method designed to
mitigate the effects of sampling fluctuations in a vector of evenly
spaced discrete observations. Although this method was originally
proposed by Bohlmann (1899), it is named after Whittaker (1923), who
applied it to graduate mortality tables, and Henderson (1924), who
popularized it among actuaries in the United States. The method was
later extended to two dimensions by Knorr (1984). WH smoothing may be
used to build experience tables for a broad spectrum of life insurance
risks, such as mortality, disability, long-term care, lapse, mortgage
default and unemployment.

### The one-dimensional case

Let $\mathbf{y}$ be a vector of observations and $\mathbf{w}$ a vector
of positive weights, both of size $n$. The estimator associated with
Whittaker-Henderson smoothing is given by:

$$
\hat{\mathbf{y}} = \underset{\boldsymbol{\theta}}{\text{argmin}}\{F(\mathbf{y},\mathbf{w},\boldsymbol{\theta}) + R_{\lambda,q}(\boldsymbol{\theta})\}
$$

where:

- $F(\mathbf{y},\mathbf{w},\boldsymbol{\theta}) = \underset{i = 1}{\overset{n}{\sum}} w_i(y_i - \theta_i)^2$
  represents a fidelity criterion with respect to the observations,

- $R_{\lambda,q}(\boldsymbol{\theta}) = \lambda \underset{i = 1}{\overset{n - q}{\sum}} (\Delta^q\boldsymbol{\theta})_i^2$
  represents a smoothness criterion.

In the latter expression, $\lambda \ge 0$ is a smoothing parameter and
$\Delta^q$ denotes the forward difference operator of order $q$, such
that for any $i\in\{1,\dots,n - q\}$:

$$
(\Delta^q\boldsymbol{\theta})_i = \underset{k = 0}{\overset{q}{\sum}} \begin{pmatrix}q \\ k\end{pmatrix}(- 1)^{q - k} \theta_{i + k}.
$$

Define $W = \text{Diag}(\mathbf{w})$, the diagonal matrix of weights,
and $D_{n,q}$ as the order $q$ difference matrix of dimensions
$(n-q) \times n$, such that
$(D_{n,q}\boldsymbol{\theta})_i = (\Delta^q\boldsymbol{\theta})_i$ for
all $i \in [1, n-q]$. The first- and second-order difference matrices
are given by:

$$
D_{n,1} = \begin{bmatrix}
-1 & 1 & 0 & \ldots & 0 \\
0 & -1 & 1 & \ldots & 0 \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
0 & \ldots & 0 & -1 & 1 \\
\end{bmatrix}
\quad\text{and}\quad
D_{n,2} = \begin{bmatrix}
1 & -2 & 1 & 0 & \ldots & 0 \\
0 & 1 & -2 & 1 & \ldots & 0 \\
\vdots & \vdots & \vdots & \vdots & \ddots & \vdots \\
0 & \ldots & 0 & 1 & -2 & 1 \\
\end{bmatrix}.
$$

while higher-order difference matrices follow the recursive formula
$D_{n,q} = D_{n - 1,q - 1}D_{n,1}$. The fidelity and smoothness criteria
can be rewritten with matrix notations as:

$$
F(\mathbf{y},\mathbf{w},\boldsymbol{\theta}) = (\mathbf{y} - \boldsymbol{\theta})^TW(\mathbf{y} - \boldsymbol{\theta}) \quad \text{and} \quad R_{\lambda,q}(\boldsymbol{\theta}) = \lambda\boldsymbol{\theta}^TD_{n,q}^TD_{n,q}\boldsymbol{\theta}
$$

and the WH smoothing estimator thus becomes:

$$
\hat{\mathbf{y}} = \underset{\boldsymbol{\theta}}{\text{argmin}} \left\lbrace(\mathbf{y} - \boldsymbol{\theta})^TW(\mathbf{y} - \boldsymbol{\theta}) + \boldsymbol{\theta}^TP_{\lambda}\boldsymbol{\theta}\right\rbrace
$$

where $P_{\lambda} = \lambda D_{n,q}^TD_{n,q}$.

### The two-dimensional case

In the two-dimensional case, consider a matrix $Y$ of observations and a
matrix $\Omega$ of non-negative weights, both of dimensions
$n_x \times n_z$. The WH smoothing estimator solves:

$$
\widehat{Y} = \underset{\Theta}{\text{argmin}}\{F(Y,\Omega, \Theta) + R_{\lambda,q}(\Theta)\}
$$

where:

- $F(Y,\Omega, \Theta) = \sum_{i = 1}^{n_x}\sum_{j = 1}^{n_z} \Omega_{i,j}(Y_{i,j} - \Theta_{i,j})^2$
  represents a fidelity criterion with respect to the observations,

- $R_{\lambda,q}(\Theta) = \lambda_x \sum_{j = 1}^{n_z}\sum_{i = 1}^{n_x - q_x} (\Delta^{q_x}\Theta_{\bullet,j})_i^2 + \lambda_z \sum_{i = 1}^{n_x}\sum_{j = 1}^{n_z - q_z} (\Delta^{q_z}\Theta_{i,\bullet})_j^2$
  is a smoothness criterion with $\lambda = (\lambda_x,\lambda_z)$.

This latter criterion adds row-wise and column regularization criteria
to $\Theta$, with respective orders $q_x$ and $q_z$, weighted by
non-negative smoothing parameters $\lambda_x$ and $\lambda_z$. In matrix
notation, let $\mathbf{y} = \textbf{vec}(Y)$,
$\mathbf{w} = \textbf{vec}(\Omega)$, and
$\boldsymbol{\theta} = \textbf{vec}(\Theta)$ as the vectors obtained by
stacking the columns of the matrices $Y$, $\Omega$, and $\Theta$,
respectively. Additionally, denote $W = \text{Diag}(\mathbf{w})$ and
$n = n_x \times n_z$. The fidelity and smoothness criteria become:

$$
\begin{aligned}
F(\mathbf{y},\mathbf{w}, \boldsymbol{\theta}) &= (\mathbf{y} - \boldsymbol{\theta})^TW(\mathbf{y} - \boldsymbol{\theta}) \\
R_{\lambda,q}(\boldsymbol{\theta}) &= \boldsymbol{\theta}^{T}(\lambda_x I_{n_z} \otimes D_{n_x,q_x}^{T}D_{n_x,q_x} + \lambda_z D_{n_z,q_z}^{T}D_{n_z,q_z} \otimes I_{n_x}) \boldsymbol{\theta}.
\end{aligned}
$$

and the associated estimator takes the same form as in the
one-dimensional case except

$$P_{\lambda} = \lambda_x I_{n_z} \otimes D_{n_x,q_x}^{T}D_{n_x,q_x} + \lambda_z D_{n_z,q_z}^{T}D_{n_z,q_z} \otimes I_{n_x}.$$

### An explicit solution

If $W + P_{\lambda}$ is invertible, the WH smoothing equation admits the
closed-form solution:

$$\hat{\mathbf{y}} = (W + P_{\lambda})^{-1}W\mathbf{y}.$$

Indeed, as a minimum, $\hat{\mathbf{y}}$ satisfies:

$$0 = \left.\frac{\partial}{\partial \boldsymbol{\theta}}\right|_{\hat{\mathbf{y}}}\left\lbrace(\mathbf{y} - \boldsymbol{\theta})^{T}W(\mathbf{y} - \boldsymbol{\theta}) + \boldsymbol{\theta}^{T}P_{\lambda}\boldsymbol{\theta}\right\rbrace = - 2 W(y - \hat{\mathbf{y}}) +2P_{\lambda} \hat{\mathbf{y}}.$$

It follows that $(W + P_{\lambda})\hat{\mathbf{y}} = W\mathbf{y}$,
proving the above result. If $\lambda \neq 0$, $W + P_{\lambda}$ is
invertible as long as $\mathbf{w}$ has $q$ non-zero elements in the
one-dimensional case, and $\Omega$ has at least $q_x \times q_z$
non-zero elements spread across $q_x$ different rows and $q_z$ different
columns in the two-dimensional case. These conditions are always met in
real datasets.

## How to use the package?

The `WH` package features a unique main function `WH`. Two arguments are
mandatory for this function:

- The vector (or matrix in the two-dimension case) `d` corresponding to
  the number of observed events of interest by age (or by age and
  duration in the two-dimension case). `d` should have named elements
  (or rows and columns) for the model results to be extrapolated.

- The vector (or matrix in the two-dimension case) `ec` corresponding to
  the portfolio central exposure by age (or by age and duration in the
  two-dimension case) whose dimensions should match those of `d`. The
  contribution of each individual to the portfolio central exposure
  corresponds to the time the individual was actually observed with
  corresponding age (and duration in the two-dimension case). It always
  ranges from 0 to 1 and is affected by individuals leaving the
  portfolio, no matter the cause, as well as censoring and truncating
  phenomena.

Additional arguments may be supplied, whose description is given in the
documentation of the functions.

The package also embed two fictive agregated datasets to illustrate how
to use it:

- `portfolio_mortality` contains the agregated number of deaths and
  associated central exposure by age for an annuity portfolio.

- `portfolio_LTC` contains the agregated number of deaths and associated
  central exposure by age and duration (in years) since the onset of LTC
  for the annuitant database of a long-term care portfolio.

``` r
# One-dimensional case
WH_1d_fit <- WH(portfolio_mort$d, portfolio_mort$ec)
Outer procedure completed in 14 iterations, smoothing parameters: 9327, final LAML: 32.1
```

``` r
# Two-dimensional case
WH_2d_fit <- WH(portfolio_LTC$d, portfolio_LTC$ec)
Outer procedure completed in 63 iterations, smoothing parameters: 1211.41,    1.09, final LAML: 276
```

Function `WH` outputs objects of class `"WH_1d"` and `"WH_2d"` to which
additional functions (including generic S3 methods) may be applied:

- The `print` function provides a glimpse of the fitted results

``` r
WH_1d_fit
An object fitted using the WH function
Initial data contains 45 data points:
  Observation positions:  50  to  94 
Smoothing parameter selected: 9327 
Associated degrees of freedom: 6.8 
WH_2d_fit
An object fitted using the WH function
Initial data contains 450 data points:
  First dimension:  70  to  99 
  Second dimension:  0  to  14 
Smoothing parameters selected: 1211.4    1.1 
Associated degrees of freedom: 47 
```

- The `plot` function generates rough plots of the model fit, the
  associated standard deviation, the model residuals or the associated
  degrees of freedom. See the `plot.WH_1d` and `plot.WH_2d` functions
  help for more details.

``` r
plot(WH_1d_fit)
```

<img src="man/figures/README-plot-1.png" width="100%" style="display: block; margin: auto;" />

``` r
plot(WH_1d_fit, "res")
```

<img src="man/figures/README-plot-2.png" width="100%" style="display: block; margin: auto;" />

``` r
plot(WH_1d_fit, "edf")
```

<img src="man/figures/README-plot-3.png" width="100%" style="display: block; margin: auto;" />

``` r

plot(WH_2d_fit)
```

<img src="man/figures/README-plot-4.png" width="100%" style="display: block; margin: auto;" />

``` r
plot(WH_2d_fit, "std_y_hat")
```

<img src="man/figures/README-plot-5.png" width="100%" style="display: block; margin: auto;" />

- The `predict` function generates an extrapolation of the model. It
  requires a `newdata` argument, a named list with one or two elements
  corresponding to the positions of the new observations. In the
  two-dimension case constraints are used so that the predicted values
  matches the fitted values for the initial observations (see Carballo,
  Durban, and Lee 2021 to understand why this is required).

``` r
WH_1d_fit |> predict(newdata = 40:99) |> plot()
```

<img src="man/figures/README-predict-1.png" width="100%" style="display: block; margin: auto;" />

``` r
WH_2d_fit |> predict(newdata = list(age = 60:109, duration = 0:19)) |> plot()
```

<img src="man/figures/README-predict-2.png" width="100%" style="display: block; margin: auto;" />

- The `vcov` may be used to retrieve the variance-covariance matrix of
  the model if necessary.

- Finally the `output_to_df` function converts an `"WH_1d"` or `"WH_2d"`
  object into a `data.frame`. Information about the fit is discarded in
  the process. This function may be useful to produce better
  visualizations from the data, for example using the ggplot2 package.

``` r
WH_1d_df <- WH_1d_fit |> output_to_df()
WH_2d_df <- WH_2d_fit |> output_to_df()
```

## Further WH smoothing theory

See the upcoming paper available
[here](https://hal.science/hal-04124043)

## References

<div id="refs" class="references csl-bib-body hanging-indent"
entry-spacing="0">

<div id="ref-bohlmann1899ausgleichungsproblem" class="csl-entry">

Bohlmann, Georg. 1899. “Ein Ausgleichungsproblem.” *Nachrichten von Der
Gesellschaft Der Wissenschaften Zu Göttingen, Mathematisch-Physikalische
Klasse* 1899: 260–71.

</div>

<div id="ref-carballo2021prediction" class="csl-entry">

Carballo, Alba, Maria Durban, and Dae-Jin Lee. 2021. “Out-of-Sample
Prediction in Multidimensional p-Spline Models.” *Mathematics* 9 (15):
1761.

</div>

<div id="ref-henderson1924new" class="csl-entry">

Henderson, Robert. 1924. “A New Method of Graduation.” *Transactions of
the Actuarial Society of America* 25: 29–40.

</div>

<div id="ref-knorr1984multidimensional" class="csl-entry">

Knorr, Frank E. 1984. “Multidimensional Whittaker-Henderson Graduation.”
*Transactions of Society of Actuaries* 36: 213–55.

</div>

<div id="ref-whittaker1923new" class="csl-entry">

Whittaker, Edmund Taylor. 1923. “On a New Method of Graduation.”
*Proceedings of the Edinburgh Mathematical Society* 41: 63–75.

</div>

</div>
