Author: Ioannis Kosmidis
Date: 2017-07-01

# Multinomial logistic regression using brglm2

#### Ioannis Kosmidis

#### 01 July 2017

# brmultinom

The brglm2 R package provides `brmultinom()` which is a wrapper of `brglmFit` for fitting multinomial logistic regression models (a.k.a. baseline category logit models) using either maximum likelihood or maximum penalized likelihood or any of the various bias reduction methods described in `brglmFit()`. `brmultinom()` uses the equivalent Poisson log-linear model, by appropriately re-scaling the Poisson means to match the multinomial totals (a.k.a. the “Poisson trick”). The mathematical details and algorithm on using the Poisson trick for mean-bias reduction are given in Kosmidis and Firth (2011).

This vignettes illustrates the use of `brmultinom()` and of the associated methods, using the alligator food choice example in Agresti (2002, sec. 7.1)

# Alligator data

The alligator data set ships with brglm2. Agresti (2002, sec. 7.1) provides a detailed description of the variables recorded in the data set.

    library("brglm2")
    data("alligators", package = "brglm2")

## Maximum likelihood estimation

The following chunk of code reproduces Agresti (2002, Table 7.4). Note that in order to get the estimates and standard errors reported in the latter table, we have to explicitly specify the contrasts that Agresti (2002) uses.

    agresti_contrasts <- list(lake = contr.treatment(levels(alligators$lake), base = 4),
                              size = contr.treatment(levels(alligators$size), base = 2))
    all_ml <- brmultinom(foodchoice ~ size + lake , weights = freq,
                         data = alligators,
                         contrasts = agresti_contrasts,
                         ref = 1,
                         type = "ML")
    all_ml_summary <- summary(all_ml)
    ## Estimated regression parameters
    round(all_ml_summary$coefficients, 2)
    #>              (Intercept) size<=2.3 lakeHancock lakeOklawaha lakeTrafford
    #> Invertebrate       -1.55      1.46       -1.66         0.94         1.12
    #> Reptile            -3.31     -0.35        1.24         2.46         2.94
    #> Bird               -2.09     -0.63        0.70        -0.65         1.09
    #> Other              -1.90      0.33        0.83         0.01         1.52
    ## Estimated standard errors
    round(all_ml_summary$standard.errors, 2)
    #>              (Intercept) size<=2.3 lakeHancock lakeOklawaha lakeTrafford
    #> Invertebrate        0.42      0.40        0.61         0.47         0.49
    #> Reptile             1.05      0.58        1.19         1.12         1.12
    #> Bird                0.66      0.64        0.78         1.20         0.84
    #> Other               0.53      0.45        0.56         0.78         0.62

## Mean and median bias reduction

Fitting the model using mean-bias reducing adjusted score equations gives

    all_mean <- update(all_ml, type = "AS_mean")
    summary(all_mean)
    #> Call:
    #> brmultinom(formula = foodchoice ~ size + lake, data = alligators,
    #>     weights = freq, contrasts = agresti_contrasts, ref = 1, type = "AS_mean")
    #>
    #> Coefficients:
    #>              (Intercept)  size<=2.3 lakeHancock lakeOklawaha lakeTrafford
    #> Invertebrate   -1.490818  1.4019624  -1.5600445   0.90109748     1.081322
    #> Reptile        -2.908819 -0.3209487   0.9800182   2.10269205     2.559008
    #> Bird           -1.940190 -0.5828768   0.6213169  -0.41459453     1.026221
    #> Other          -1.815225  0.3147753   0.7792789   0.06001729     1.450717
    #>
    #> Std. Errors:
    #>              (Intercept) size<=2.3 lakeHancock lakeOklawaha lakeTrafford
    #> Invertebrate   0.4250711 0.3962267   0.6003569    0.4742043    0.4924905
    #> Reptile        0.8864071 0.5565974   1.0218118    0.9619417    0.9604963
    #> Bird           0.6276892 0.6084642   0.7436454    1.0458118    0.8031649
    #> Other          0.5177641 0.4443760   0.5521476    0.7489610    0.6154619
    #>
    #> Residual Deviance: 540.8142
    #> Log-likelihood: -270.4071
    #> AIC: 580.8142
    #>
    #> Type of estimator: AS_mean (mean bias-reducing adjusted score equations)
    #> Number of Fisher Scoring iterations: 20

The corresponding fit using median-bias reducing adjusted score equations is

    all_median <- update(all_ml, type = "AS_median")
    summary(all_median)
    #> Call:
    #> brmultinom(formula = foodchoice ~ size + lake, data = alligators,
    #>     weights = freq, contrasts = agresti_contrasts, ref = 1, type = "AS_median")
    #>
    #> Coefficients:
    #>              (Intercept)  size<=2.3 lakeHancock lakeOklawaha lakeTrafford
    #> Invertebrate   -1.508934  1.4134512  -1.6080461   0.90952486     1.090687
    #> Reptile        -3.128931 -0.3379387   1.1208787   2.28462590     2.743766
    #> Bird           -2.018328 -0.6036652   0.6554327  -0.55207076     1.044909
    #> Other          -1.854740  0.3201676   0.8002874   0.02330188     1.469440
    #>
    #> Std. Errors:
    #>              (Intercept) size<=2.3 lakeHancock lakeOklawaha lakeTrafford
    #> Invertebrate   0.4238148 0.3949248   0.6066348    0.4720116    0.4901870
    #> Reptile        0.9729317 0.5699878   1.1068113    1.0427073    1.0411046
    #> Bird           0.6449317 0.6254237   0.7621085    1.1250548    0.8221610
    #> Other          0.5208087 0.4456384   0.5538846    0.7614315    0.6170972
    #>
    #> Residual Deviance: 540.2445
    #> Log-likelihood: -270.1223
    #> AIC: 580.2445
    #>
    #> Type of estimator: AS_median (median bias-reducing adjusted score equations)
    #> Number of Fisher Scoring iterations: 9

The estimates and the estimated standard errors from bias reduction are close to those for maximum likelihood. As a result, it is unlikely that either mean or median bias is of any real consequence for this particular model and data combination.

# Infinite estimates and multinomial logistic regression

Let’s scale the frequencies in `alligators` by 3 in order to get a sparser data set. The differences between maximum likelihood and mean and median bias reduction should be more apparent on the resulting data set. Here we have to “slow-down” the Fisher scoring iteration (by scaling the step-size), because otherwise the Fisher information matrix quickly gets numerically rank-deficient. The reason is data separation (Albert and Anderson 1984).

    all_ml_sparse <- update(all_ml, weights = round(freq/3), slowit = 0.1)
    #> Warning: brglmFit: algorithm did not converge. Try changing the optimization
    #> algorithm defaults, e.g. the defaults for one or more of `maxit`, `epsilon`,
    #> `slowit`, and `response_adjustment`; see `?brglm_control` for default values
    #> and available options
    summary(all_ml_sparse)
    #> Call:
    #> brmultinom(formula = foodchoice ~ size + lake, data = alligators,
    #>     weights = round(freq/3), contrasts = agresti_contrasts, ref = 1,
    #>     type = "ML", slowit = 0.1)
    #>
    #> Coefficients:
    #>              (Intercept)   size<=2.3 lakeHancock lakeOklawaha lakeTrafford
    #> Invertebrate   -1.892117  1.74080751  -1.7936496   0.96047783    1.0747116
    #> Reptile       -20.147831 -1.05673340  18.5968919  19.28937520   19.4244818
    #> Bird           -2.225759 -0.33405987   0.9538709 -17.82640619    0.6957762
    #> Other          -1.730365  0.04554623   1.1091441  -0.07652817    1.2070845
    #>
    #> Std. Errors:
    #>               (Intercept) size<=2.3  lakeHancock lakeOklawaha lakeTrafford
    #> Invertebrate    0.8042604 0.7409450    1.1883685    0.8431835    0.8917234
    #> Reptile      8961.4328424 1.2808647 8961.4329000 8961.4328731 8961.4328758
    #> Bird            1.1849748 1.1598538    1.3243541 9691.1869944    1.5455222
    #> Other           0.8869382 0.7815002    0.9589614    1.3379641    1.0844370
    #>
    #> Residual Deviance: 161.913
    #> Log-likelihood: -80.95651
    #> AIC: 201.913
    #>
    #> Type of estimator: ML (maximum likelihood)
    #> Number of Fisher Scoring iterations: 100

Specifically, judging from the estimated standard errors, the estimates for `(Intercept)`, `lakeHancock`, `lakeOklawaha` and `lakeTrafford` for `Reptile` and `lakeHancock` for `Bird` seem to be infinite.

To quickly check if that’s indeed the case we can use the `check_infinite_estimates()` method of the detectseparation R package.

    library("detectseparation")
    se_ratios <- check_infinite_estimates(all_ml_sparse)
    plot(se_ratios)

Some of the estimated standard errors diverge as the number of Fisher scoring iterations increases, which is evidence of complete or quasi-complete separation (Lesaffre and Albert 1989).

In contrast, both mean and median bias reduction result in finite estimates

    all_mean_sparse <- update(all_ml_sparse, type = "AS_mean")
    #> Warning: brglmFit: algorithm did not converge. Try changing the optimization
    #> algorithm defaults, e.g. the defaults for one or more of `maxit`, `epsilon`,
    #> `slowit`, and `response_adjustment`; see `?brglm_control` for default values
    #> and available options
    summary(all_mean_sparse)
    #> Call:
    #> brmultinom(formula = foodchoice ~ size + lake, data = alligators,
    #>     weights = round(freq/3), contrasts = agresti_contrasts, ref = 1,
    #>     type = "AS_mean", slowit = 0.1)
    #>
    #> Coefficients:
    #>              (Intercept)   size<=2.3 lakeHancock lakeOklawaha lakeTrafford
    #> Invertebrate   -1.679166  1.53865356  -1.4524459    0.8578547    0.9583248
    #> Reptile        -2.690142 -0.76139583   1.4034218    1.9746355    2.0975417
    #> Bird           -1.820489 -0.25358805   0.7182241   -0.5960054    0.6373495
    #> Other          -1.517520  0.04996741   0.9501188    0.0650087    1.0685650
    #>
    #> Std. Errors:
    #>              (Intercept) size<=2.3 lakeHancock lakeOklawaha lakeTrafford
    #> Invertebrate   0.7986891 0.7384924   1.0933475    0.8526695    0.9002981
    #> Reptile        1.5495030 1.0465660   1.7628376    1.7032872    1.7165409
    #> Bird           1.0387375 1.0051346   1.1808965    1.8023116    1.3550386
    #> Other          0.8606603 0.7662525   0.9397836    1.2305932    1.0633025
    #>
    #> Residual Deviance: 164.8781
    #> Log-likelihood: -82.43904
    #> AIC: 204.8781
    #>
    #> Type of estimator: AS_mean (mean bias-reducing adjusted score equations)
    #> Number of Fisher Scoring iterations: 100

    all_median_sparse <- update(all_ml_sparse, type = "AS_median")
    #> Warning: brglmFit: algorithm did not converge. Try changing the optimization
    #> algorithm defaults, e.g. the defaults for one or more of `maxit`, `epsilon`,
    #> `slowit`, and `response_adjustment`; see `?brglm_control` for default values
    #> and available options
    summary(all_median_sparse)
    #> Call:
    #> brmultinom(formula = foodchoice ~ size + lake, data = alligators,
    #>     weights = round(freq/3), contrasts = agresti_contrasts, ref = 1,
    #>     type = "AS_median", slowit = 0.1)
    #>
    #> Coefficients:
    #>              (Intercept)   size<=2.3 lakeHancock lakeOklawaha lakeTrafford
    #> Invertebrate   -1.738082  1.57656100   -1.609922   0.87918464    0.9772042
    #> Reptile        -3.665205 -0.89723096    2.189606   2.82422096    2.9353768
    #> Bird           -1.991964 -0.28576535    0.809448  -1.37215072    0.6272446
    #> Other          -1.613287  0.04721269    1.008715  -0.01732907    1.1038820
    #>
    #> Std. Errors:
    #>              (Intercept) size<=2.3 lakeHancock lakeOklawaha lakeTrafford
    #> Invertebrate   0.7922387 0.7310932   1.1339003    0.8411236    0.8888226
    #> Reptile        2.4233320 1.1696060   2.5992946    2.5306117    2.5410958
    #> Bird           1.0941201 1.0683064   1.2317597    2.5546837    1.4277810
    #> Other          0.8695136 0.7700381   0.9449315    1.2711053    1.0676085
    #>
    #> Residual Deviance: 162.9524
    #> Log-likelihood: -81.4762
    #> AIC: 202.9524
    #>
    #> Type of estimator: AS_median (median bias-reducing adjusted score equations)
    #> Number of Fisher Scoring iterations: 100

# Relevant resources

`?brglmFit` and `?brglm_control` contain quick descriptions of the various bias reduction methods supported in brglm2. The `iteration` vignette describes the iteration and gives the mathematical details for the bias-reducing adjustments to the score functions for generalized linear models.

# Citation

If you found this vignette or brglm2, in general, useful, please consider citing brglm2 and the associated paper. You can find information on how to do this by typing `citation("brglm2")`.

# References

Agresti, A. 2002. Categorical Data Analysis. Wiley.
Albert, A., and J. A. Anderson. 1984. “On the Existence of Maximum Likelihood Estimates in Logistic Regression Models.” Biometrika 71 (1): 1–10.
Kosmidis, I., and D. Firth. 2011. “Multinomial Logit Bias Reduction via the Poisson Log-Linear Model.” Biometrika 98 (3): 755–59.
Lesaffre, E., and A. Albert. 1989. “Partial Separation in Logistic Discrimination.” Journal of the Royal Statistical Society. Series B (Methodological) 51 (1): 109–16. https://www.jstor.org/stable/2345845.