Author: Jorge Cabral

# Optimization methods

#### Jorge Cabral

## Optimization

In the “Generalized Maximum Entropy framework” vignette a solution for the constrained optimization problem was proposed to compute \(\widehat{\boldsymbol{\beta}}^{GME}\). However, a more efficient way is to solve the unconstrained dual formulation of the problem, which is a function of \(\hat {\boldsymbol\lambda}\). The resulting function, \(\mathcal{M}(\boldsymbol{\lambda})\), is referred to as the minimal value function [1].
Let
\[\begin{equation} \Omega_k \left( \widehat{\boldsymbol\lambda}\right) = \sum_{m=1}^M exp(-z_{km}\sum_{n=1}^N \hat\lambda_n x_{nk}) \end{equation}\] and \[\begin{equation} \Psi_n \left( \widehat{\boldsymbol\lambda}\right) = \sum_{j=1}^J exp(-\hat\lambda_n v_{n}). \end{equation}\]

The primal–dual relationship is

\[\begin{equation} \underset{\mathbf{p},\mathbf{w}}{\operatorname{argmax}} \left\{-\mathbf{p}' \ln \mathbf{p} - \mathbf{w}' \ln \mathbf{w} \right\} = \underset{\boldsymbol{\lambda}}{\operatorname{argmin}} \left\{ \mathbf{y}'\boldsymbol{\lambda} - \sum_{k=1}^{K+1}log\left( \Omega_k(\boldsymbol{\lambda})\right) - \sum_{n=1}^{N}log\left( \Psi_n(\boldsymbol{\lambda})\right)\right\} \equiv \mathcal{M}(\boldsymbol{\lambda}). \end{equation}\]

\(\mathcal{M}(\boldsymbol{\lambda})\), may be interpreted as a constrained expected log-likelihood function. Estimation is considerably simplified using this dual version. The analytical gradient of the dual problem is

\[\begin{equation} \nabla_{\boldsymbol{\lambda}}\mathcal{M} = \mathbf{y}-\mathbf{XZp} - \mathbf{Vw} \end{equation}\]

\(\boldsymbol{\hat\lambda}\) is then used to solve

\[\begin{equation} \hat p_{km} = \frac{exp(-z_{km}\sum_{n=1}^N \hat\lambda_n x_{nk})}{\Omega_k(\boldsymbol{\lambda})} \end{equation}\] and \[\begin{equation} \hat w_{nj} = \frac{exp(-\hat\lambda_n v_{n})}{\Psi_n(\boldsymbol{\lambda})}. \end{equation}\]

\(\widehat{\boldsymbol{\beta}}^{GME}\) and \(\boldsymbol{\hat \epsilon}\) are given by

\[\begin{equation} \widehat{\boldsymbol{\beta}}^{GME}=\mathbf{Z\hat p} \end{equation}\] and \[\begin{equation} \boldsymbol{\hat \epsilon}=\mathbf{V\hat w}. \end{equation}\]

## Examples

Different numerical optimization options will be explained using `dataThesis` (see “Generalized Maximum Entropy framework”).

    library(GCEstim)

Assume that \(z_k^{upper}=50\), \(k\in\left\lbrace 0,\dots,4\right\rbrace\) is a “wide upper bound” for the signal support space and set \(M=5\) and \(J=3\).
The primal version of the optimization problem previously defined can be solved by the optimization methods:

- Rsolnp package [3]

  - `primal.solnp`, using Sequential Quadratic Programming (SQP);

- NlcOptim package [4]

  - `primal.solnl`, using the augmented Lagrange multiplier method with an SQP interior algorithm.

    res.lmgce.50.primal.solnp <-
      GCEstim::lmgce(
        y ~ .,
        data = dataThesis,
        support.signal = c(-50, 50),
        twosteps.n = 0,
        method = "primal.solnp"
      )

    res.lmgce.50.primal.solnl <-
      GCEstim::lmgce(
        y ~ .,
        data = dataThesis,
        support.signal = c(-50, 50),
        twosteps.n = 0,
        method = "primal.solnl"
      )

Both methods converged which can be confirmed by the \(0\) returned by

    res.lmgce.50.primal.solnp$convergence
    #> [1] 0
    res.lmgce.50.primal.solnl$convergence
    #> [1] 0

The dual version of the optimization problem can be solved by the optimization methods:

- optim package
  - `dual.BFGS`, default, using Broyden-Fletcher-Goldfarb-Shanno quasi-Newton method [5–8];
  - `dual.CG`, a conjugate gradients method [9];
  - `dual.L-BFGS-B`, using a box-constrained optimization with limited-memory modification of the BFGS quasi-Newton method [10];
- optimx package [11]
  - `dual.Rcgmin`, using conjugate gradient minimization of nonlinear functions with box constraints using Dai/Yuan update [12] (see Rcgmin);
  - `dual.bobyqa`, using a derivative-free optimization by quadratic approximation [13] (see minqa [14]);
  - `dual.newuoa`, using a derivative-free optimization by quadratic approximation [13] (see minqa [14]);
  - `dual.nlminb`, unconstrained and box-constrained optimization using PORT routines [15] (see nlminb);
  - `dual.nlm`, using a Newton-type algorithm [16,17] (see nlm);
- lbfgs package [18]
  - `dual.lbfgs`, using the Limited-memory Broyden-Fletcher-Goldfarb-Shanno;
- lbfgsb3c package [19]
  - `dual.lbfgsb3c`, using L-BFSC-B implemented in Fortran code and with an Rcpp interface;

    res.lmgce.50.dual.BFGS <-
      GCEstim::lmgce(
        y ~ .,
        data = dataThesis,
        support.signal = c(-50, 50),
        twosteps.n = 0,
        method = "dual.BFGS"
      )

    res.lmgce.50.dual.CG <-
      GCEstim::lmgce(
        y ~ .,
        data = dataThesis,
        support.signal = c(-50, 50),
        twosteps.n = 0,
        method = "dual.CG"
      )

Any error or warning produced by the optimization procedure results in outputting \(1\). Note that `dual.BFGS` converged while `dual.CG` did not. Note that the default optimization setting were kept. Conjugate gradient methods will generally be more fragile but may be successful in much larger optimization problems.

    res.lmgce.50.dual.BFGS$convergence
    #> [1] 0
    res.lmgce.50.dual.CG$convergence
    #> [1] 1

The values of \(\boldsymbol{\hat\lambda}\), \(\hat {\mathbf{p}}\) and \(\hat {\mathbf{w}}\) for `dual.BFGS` are, respectively,

    res.lmgce.50.dual.BFGS$lambda
    #>  [1]  0.0172214766  0.0105465378  0.0147487344 -0.0072518353  0.0026476807
    #>  [6] -0.0071868443  0.0165004568 -0.0118064272 -0.0004856784 -0.0102305511
    #> [11] -0.0187843424  0.0094976353 -0.0192101106  0.0062020781  0.0087906122
    #> [16] -0.0086329951  0.0004843064  0.0057980041  0.0074915815 -0.0168005375
    #> [21] -0.0019353538  0.0055851315 -0.0226289864  0.0073436134 -0.0219178696
    #> [26]  0.0251168796 -0.0263107086 -0.0016885513 -0.0148184680  0.0024364861
    #> [31]  0.0083324562  0.0036587737 -0.0168960874 -0.0004604358  0.0064889427
    #> [36]  0.0106249791  0.0041605924 -0.0148020194 -0.0402658466 -0.0017502044
    #> [41]  0.0008957668 -0.0053531038 -0.0103267408  0.0318543198  0.0045891476
    #> [46]  0.0130277951  0.0156347080  0.0207359884 -0.0032553092 -0.0007164471
    #> [51]  0.0206885720  0.0019557500 -0.0187385161 -0.0018445038 -0.0366124685
    #> [56] -0.0326915878  0.0025368175  0.0018611674  0.0039667774 -0.0083271771
    #> [61]  0.0012261629 -0.0027121047 -0.0111571634  0.0100160316  0.0074757423
    #> [66]  0.0296219562  0.0164761205 -0.0204664514  0.0173164705  0.0081156260
    #> [71]  0.0228910041  0.0049246012 -0.0047486693  0.0102932891 -0.0005895098

    res.lmgce.50.dual.BFGS$p
    #>                   p_1       p_2       p_3       p_4        p_5
    #> (Intercept) 0.2336971 0.2154835 0.1986894 0.1832042 0.16892579
    #> X001        0.2011858 0.2005911 0.1999982 0.1994071 0.19881772
    #> X002        0.3410137 0.2487293 0.1814188 0.1323236 0.09651452
    #> X003        0.2567399 0.2245897 0.1964655 0.1718632 0.15034169
    #> X004        0.2680295 0.2286385 0.1950367 0.1663731 0.14192214

    res.lmgce.50.dual.BFGS$w
    #>          p_1       p_2       p_3
    #> 1  0.2560944 0.3268222 0.4170835
    #> 2  0.2849673 0.3308688 0.3841640
    #> 3  0.2666138 0.3285390 0.4048472
    #> 4  0.3680883 0.3321647 0.2997471
    #> 5  0.3209166 0.3331772 0.3459063
    #> 6  0.3677727 0.3321855 0.3000418
    #> 7  0.2591394 0.3273488 0.4135118
    #> 8  0.3903469 0.3302491 0.2794040
    #> 9  0.3356285 0.3333281 0.3310434
    #> 10 0.3826161 0.3310135 0.2863704
    #> 11 0.4248332 0.3256083 0.2495585
    #> 12 0.2896369 0.3313325 0.3790305
    #> 13 0.4269460 0.3252605 0.2477935
    #> 14 0.3045231 0.3324779 0.3629990
    #> 15 0.2928036 0.3316182 0.3755782
    #> 16 0.3748094 0.3316789 0.2935116
    #> 17 0.3310499 0.3333281 0.3356220
    #> 18 0.3063697 0.3325856 0.3610447
    #> 19 0.2986607 0.3320863 0.3692530
    #> 20 0.4149980 0.3271322 0.2578698
    #> 21 0.3425093 0.3332499 0.3242408
    #> 22 0.3073444 0.3326394 0.3600162
    #> 23 0.4439217 0.3222099 0.2338684
    #> 24 0.2993310 0.3321350 0.3685340
    #> 25 0.4403903 0.3228818 0.2367279
    #> 26 0.2240201 0.3197090 0.4562709
    #> 27 0.4621903 0.3184277 0.2193820
    #> 28 0.3413348 0.3332698 0.3253954
    #> 29 0.4051917 0.3284940 0.2663143
    #> 30 0.3219009 0.3332011 0.3448981
    #> 31 0.2948637 0.3317917 0.3733446
    #> 32 0.3162197 0.3330353 0.3507451
    #> 33 0.4154713 0.3270624 0.2574663
    #> 34 0.3355091 0.3333286 0.3311623
    #> 35 0.3032149 0.3323971 0.3643880
    #> 36 0.2846194 0.3308322 0.3845484
    #> 37 0.3138982 0.3329480 0.3531538
    #> 38 0.4051104 0.3285047 0.2663849
    #> 39 0.5304739 0.2999373 0.1695887
    #> 40 0.3416281 0.3332651 0.3251069
    #> 41 0.3291141 0.3333155 0.3375704
    #> 42 0.3588961 0.3326958 0.3084082
    #> 43 0.3830871 0.3309699 0.2859429
    #> 44 0.1986117 0.3118228 0.4895655
    #> 45 0.3119209 0.3328646 0.3552145
    #> 46 0.2740597 0.3295837 0.3963566
    #> 47 0.2628201 0.3279529 0.4092271
    #> 48 0.2415236 0.3239554 0.4345210
    #> 49 0.3488118 0.3330974 0.3180908
    #> 50 0.3367208 0.3333219 0.3299573
    #> 51 0.2417171 0.3239973 0.4342856
    #> 52 0.3241454 0.3332481 0.3426064
    #> 53 0.4246059 0.3256453 0.2497489
    #> 54 0.3420768 0.3332575 0.3246657
    #> 55 0.5128261 0.3053548 0.1818190
    #> 56 0.4936776 0.3107358 0.1955866
    #> 57 0.3214331 0.3331900 0.3453769
    #> 58 0.3245877 0.3332562 0.3421561
    #> 59 0.3147941 0.3329830 0.3522229
    #> 60 0.3733189 0.3317937 0.2948874
    #> 61 0.3275626 0.3332998 0.3391376
    #> 62 0.3462140 0.3331695 0.3206165
    #> 63 0.3871584 0.3305770 0.2822646
    #> 64 0.2873248 0.3311093 0.3815659
    #> 65 0.2987324 0.3320916 0.3691760
    #> 66 0.2068220 0.3146087 0.4785692
    #> 67 0.2592425 0.3273662 0.4133913
    #> 68 0.4331826 0.3241926 0.2426248
    #> 69 0.2556946 0.3267512 0.4175542
    #> 70 0.2958408 0.3318707 0.3722885
    #> 71 0.2328197 0.3219575 0.4452228
    #> 72 0.3103765 0.3327936 0.3568298
    #> 73 0.3559824 0.3328315 0.3111861
    #> 74 0.2860916 0.3309851 0.3829233
    #> 75 0.3361198 0.3333256 0.3305546

The following table shows a comparison of efficiency between all the optimization methods available. Note that in the current version, the default optimization parameters of each dependency package are used.

    method.opt <-
      c(
        "primal.solnl",
        "primal.solnp",
        "dual.BFGS",
        "dual.CG",
        "dual.L-BFGS-B",
        "dual.Rcgmin",
        "dual.bobyqa",
        "dual.newuoa",
        "dual.nlminb",
        "dual.nlm",
        "dual.lbfgs",
        "dual.lbfgsb3c"
      )

    compare_methods <- data.frame(
      method = method.opt,
      time = NA,
      r.squared = NA,
      error.measure = NA,
      error.measure.cv.mean = NA,
      beta.error = NA,
      nep = NA,
      nep.cv.mean = NA,
      convergence = NA)

    for (i in 1:length(method.opt)) {
    start.time <- Sys.time()
    res.method <-
      lmgce(
        y ~ .,
        data = dataThesis,
        support.signal = c(-50,50),
        twosteps.n = 0,
        method = method.opt[i]
        )

    compare_methods$time[i] <- difftime(Sys.time(),
                                        start.time,
                                        units = "secs")
    compare_methods$r.squared[i] <- summary(res.method)$r.squared
    compare_methods$error.measure[i] <- res.method$error.measure
    compare_methods$error.measure.cv.mean[i] <- res.method$error.measure.cv.mean
    compare_methods$beta.error[i] <- accmeasure(coef(res.method), coef.dataThesis)
    compare_methods$nep[i] <- res.method$nep
    compare_methods$nep.cv.mean [i] <- res.method$nep.cv.mean
    compare_methods$convergence[i] <- res.method$convergence
    }

    compare_methods_ordered <- compare_methods[order(compare_methods$time),]

    compare_methods_ordered$convergence <- factor(compare_methods_ordered$convergence,
                                                  levels = c(0,1),
                                                  labels = c(TRUE, FALSE))

optimization method | time (s) | \(RMSE_{\widehat{y}}\) | \(CV \text{-} RMSE_{\widehat{y}}\) | \(RMSE_{\widehat{\beta}}\) | Convergence
--- | --- | --- | --- | --- | ---
dual.lbfgsb3c | 0.130 | 0.951 | 0.984 | 2.747 | TRUE
dual.L-BFGS-B | 0.283 | 0.951 | 0.984 | 2.747 | TRUE
dual.BFGS | 0.294 | 0.951 | 0.984 | 2.746 | TRUE
dual.lbfgs | 0.299 | 0.951 | 0.984 | 2.746 | TRUE
dual.CG | 0.551 | 1.165 | 1.055 | 2.824 | FALSE
dual.nlminb | 2.453 | 0.951 | 0.984 | 2.746 | TRUE
primal.solnp | 2.557 | 0.951 | 0.984 | 2.746 | TRUE
dual.Rcgmin | 2.609 | 0.951 | 0.984 | 2.746 | TRUE
dual.nlm | 2.708 | 0.951 | 0.984 | 2.746 | TRUE
primal.solnl | 2.837 | 0.951 | 0.984 | 2.746 | TRUE
dual.newuoa | 33.727 | 0.951 | 0.984 | 2.746 | TRUE
dual.bobyqa | 63.507 | 0.951 | 0.984 | 2.747 | TRUE

## References

1.
Golan A, Judge GG, Miller D. Maximum Entropy Econometrics : Robust Estimation with Limited Data. Wiley; 1996.
2.
Ye Y. Interior Algorithms for Linear, Quadratic, and Linearly Constrained Non-Linear Programming. PhD thesis. Department of ESS, Stanford University; 1987.
3.
Ghalanos A, Theussl S. Rsolnp: General Non-Linear Optimization Using Augmented Lagrange Multiplier Method.; 2015.
4.
Chen X, Yin X. NlcOptim: Solve Nonlinear Optimization with Nonlinear Constraints.; 2019. https://CRAN.R-project.org/package=NlcOptim
5.
Broyden CG. The convergence of a class of double-rank minimization algorithms 1. General considerations. IMA Journal of Applied Mathematics. 1970;6:76-90. doi:10.1093/imamat/6.1.76
6.
Fletcher R. A new approach to variable metric algorithms. The Computer Journal. 1970;13:317-322. doi:10.1093/comjnl/13.3.317
7.
Goldfarb D. A family of variable-metric methods derived by variational means. Mathematics of computation. 1970;24:23-26. doi:10.1090/s0025-5718-1970-0258249-6
8.
Shanno DF. Conditioning of quasi-newton methods for function minimization. Mathematics of computation. 1970;24:647-656. doi:10.1090/s0025-5718-1970-0274029-x
9.
Fletcher R, Reeves CM. Function minimization by conjugate gradients. The Computer Journal. 1964;7:149-154. doi:10.1093/comjnl/7.2.149
10.
Byrd RH, Lu P, Nocedal J, Zhu C. A limited memory algorithm for bound constrained optimization. SIAM Journal on Scientific Computing. 1995;16:1190-1208. doi:10.1137/0916069
11.
Nash JC, Varadhan R. Unifying optimization algorithms to aid software system users: Optimx for r. Journal of Statistical Software. 2011;43:1-14. doi:10.18637/jss.v043.i09
12.
Dai YH, Yuan Y. An efficient hybrid conjugate gradient method for unconstrained optimization. Annals of Operations Research. 2001;103:33-47. doi:10.1023/A:1012930416777
13.
Powell M. The BOBYQA algorithm for bound constrained optimization without derivatives. Technical Report, Department of Applied Mathematics and Theoretical Physics. Published online January 2009.
14.
Bates D, Mullen KM, Nash JC, Varadhan R. Minqa: Derivative-Free Optimization Algorithms by Quadratic Approximation.; 2024. https://CRAN.R-project.org/package=minqa
15.
Gay DM. Usage summary for selected optimization routines. Computing Science Technical Report. 1990;153.
16.
Dennis JE, Schnabel RB. Numerical Methods for Unconstrained Optimization and Nonlinear Equations. Prentice-Hall; 1983.
17.
Schnabel RB, Koonatz JE, Weiss BE. A modular system of algorithms for unconstrained minimization. ACM transactions on mathematical software Association for Computing Machinery. 1985;11:419-440. doi:10.1145/6187.6192
18.
Coppola A, Stewart B, Okazaki N. Lbfgs: Limited-Memory BFGS Optimization.; 2022. https://CRAN.R-project.org/package=lbfgs
19.
Fidler ML, Nash JC, Zhu C, Byrd R, Nocedal J, Morales JL. Lbfgsb3c: Limited Memory BFGS Minimizer with Bounds on Parameters with Optim() ’c’ Interface.; 2024. https://CRAN.R-project.org/package=lbfgsb3c

## Acknowledgements

This work was supported by Fundação para a Ciência e Tecnologia (FCT) through CIDMA and projects https://doi.org/10.54499/UIDB/04106/2020 and https://doi.org/10.54499/UIDP/04106/2020.