MNL estimates incorrect
Posted: 26 Jun 2022, 20:07
Hello,
I'm very new to choice modelling and I've been trying to analyse my data with Apollo, first starting with a simple MNL and then with a MIXL. However, the results that I get from a MNL with Apollo are completely different than those that I got with R package gmnl and in Stata (those also make sense, while Apollo ones don't). I've tried using different estimation routines, but they all end up with a similar result. I'm wondering what could be causing the difference and how to fix it, as I think I've specified the utilities correctly?
My data comes from a choice experiment where each participant made 8 choices among three unlabelled alternatives, one of which was an opt-out. Each alternative had four continuous attributes, one of which was payment that they would receive.
You can find the code and outputs for both models below. Thank you so much for your help!
################################MNL
apollo_initialise()
apollo_control = list(
modelName = "MNL full",
modelDescr = "poskus",
indivID = "ID",
nCores = 4,
outputDirectory = "output",
panelData=T
)
database=DCE
apollo_beta=c(asc_1=0,
asc_2=0,
asc_nochoice=0,
b_fallow=0,
b_landsc=0,
b_meadow=0,
b_payment=0)
apollo_fixed=c("asc_1")
apollo_inputs=apollo_validateInputs()
apollo_probabilities=function(apollo_beta, apollo_inputs, functionality="estimate"){
apollo_attach(apollo_beta, apollo_inputs)
on.exit(apollo_detach(apollo_beta,apollo_inputs))
P=list()
V=list()
V[["Nochoice"]]=asc_nochoice
V[["Alt1"]]=asc_1+b_fallow*alt1.fallow+b_landsc*alt1.landsc+b_meadow*alt1.meadow+b_payment*alt1.payment
V[["Alt2"]]=asc_2+b_fallow*alt2.fallow+b_landsc*alt2.landsc+b_meadow*alt2.meadow+b_payment*alt2.payment
mnl_settings = list(
alternatives = c(Nochoice=1, Alt1=2, Alt2=3),
choiceVar = Choice,
utilities = V
)
P[["model"]] = apollo_mnl(mnl_settings, functionality)
P = apollo_panelProd(P, apollo_inputs, functionality)
P = apollo_prepareProb(P, apollo_inputs, functionality)
return(P)
}
estimate_settings=list(estimationRoutine="BHHH")
full.m = apollo_estimate(apollo_beta, apollo_fixed, apollo_probabilities, apollo_inputs, estimate_settings)
modelOutput_settings=list(printCovar=T,printPVal=2)
modelsumFE<-apollo_modelOutput(full.m, modelOutput_settings)
########## Model output:
Model run by Ziva using Apollo 0.2.7 on R 4.1.3 for Windows.
www.ApolloChoiceModelling.com
Model name : MNL full
Model description : poskus
Model run at : 2022-06-26 16:59:24
Estimation method : bhhh
Model diagnosis : successive function values within relative tolerance limit (reltol)
Number of individuals : 426
Number of rows in database : 3568
Number of modelled outcomes : 3568
Number of cores used : 4
Model without mixing
LL(start) : -3919.85
LL(0) : -3919.85
LL(C) : -3848.66
LL(final) : -3730.4
Rho-square (0) : 0.0483
Adj.Rho-square (0) : 0.0468
Rho-square (C) : 0.0307
Adj.Rho-square (C) : 0.0292
AIC : 7472.8
BIC : 7509.87
Estimated parameters : 6
Time taken (hh:mm:ss) : 00:00:22.56
pre-estimation : 00:00:20.42
estimation : 00:00:1.3
post-estimation : 00:00:0.85
Iterations : 36 (successive function values within relative tolerance limit (reltol))
Min abs eigenvalue of Hessian : 49.02786
Unconstrained optimisation.
Estimates:
Estimate s.e. t.rat.(0) p(2-sided) Rob.s.e. Rob.t.rat.(0) p(2-sided)
asc_1 0.000000 NA NA NA NA NA NA
asc_2 0.482161 0.043160 11.1715 0.00000 0.078739 6.124 9.154e-10
asc_nochoice -0.635585 0.142185 -4.4701 7.817e-06 0.134150 -4.738 2.160e-06
b_fallow -0.019060 0.009968 -1.9121 0.05586 0.009866 -1.932 0.05337
b_landsc 0.011238 0.009655 1.1640 0.24443 0.009428 1.192 0.23328
b_meadow -0.008461 0.008854 -0.9556 0.33925 0.008059 -1.050 0.29377
b_payment -0.004235 2.9279e-04 -14.4650 0.00000 2.4936e-04 -16.984 0.00000
Classical covariance matrix:
asc_2 asc_nochoice b_fallow b_landsc b_meadow b_payment
asc_2 0.001863 1.617e-05 -7.834e-05 -5.397e-05 -5.091e-05 -1.616e-06
asc_nochoice 1.617e-05 0.020216 0.001215 0.001096 9.7733e-04 2.202e-05
b_fallow -7.834e-05 0.001215 9.936e-05 7.358e-05 6.082e-05 1.023e-06
b_landsc -5.397e-05 0.001096 7.358e-05 9.322e-05 5.569e-05 5.146e-07
b_meadow -5.091e-05 9.7733e-04 6.082e-05 5.569e-05 7.839e-05 5.944e-07
b_payment -1.616e-06 2.202e-05 1.023e-06 5.146e-07 5.944e-07 8.573e-08
Robust covariance matrix:
asc_2 asc_nochoice b_fallow b_landsc b_meadow b_payment
asc_2 0.006200 0.001683 -1.031e-05 -1.136e-05 2.353e-05 4.935e-06
asc_nochoice 0.001683 0.017996 0.001133 0.001080 9.2037e-04 1.612e-05
b_fallow -1.031e-05 0.001133 9.733e-05 7.342e-05 5.667e-05 6.818e-07
b_landsc -1.136e-05 0.001080 7.342e-05 8.889e-05 5.608e-05 4.451e-07
b_meadow 2.353e-05 9.2037e-04 5.667e-05 5.608e-05 6.495e-05 5.695e-07
b_payment 4.935e-06 1.612e-05 6.818e-07 4.451e-07 5.695e-07 6.218e-08
Here you can find the results from Stata for comparison:
CONDITIONAL LOGIT model
. clogit choice payment ASC $cond, group(idchoice) nolog
Conditional (fixed-effects) logistic regression
Number of obs = 10,224
LR chi2(5) = 874.66
Prob > chi2 = 0.0000
Log likelihood = -3306.7411 Pseudo R2 = 0.1168
------------------------------------------------------------------------------
choice | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
payment | .0064578 .0003533 18.28 0.000 .0057654 .0071503
ASC | .1289545 .1550375 0.83 0.406 -.1749134 .4328225
fallow | -.110634 .0101162 -10.94 0.000 -.1304613 -.0908066
landsc | -.1268549 .0099619 -12.73 0.000 -.1463798 -.10733
meadow | -.0787455 .0087474 -9.00 0.000 -.0958901 -.0616009
------------------------------------------------------------------------------
. estat ic
Akaike's information criterion and Bayesian information criterion
-----------------------------------------------------------------------------
Model | N ll(null) ll(model) df AIC BIC
-------------+---------------------------------------------------------------
. | 10,224 -3744.071 -3306.741 5 6623.482 6659.645
-----------------------------------------------------------------------------
Note: BIC uses N = number of observations. See [R] BIC note.
. estat vce, c
Correlation matrix of coefficients of clogit model
| choice
e(V) | payment ASC fallow landsc meadow
-------------+--------------------------------------------------
choice |
payment | 1.0000
ASC | 0.6492 1.0000
fallow | 0.2884 0.8152 1.0000
landsc | 0.1791 0.7513 0.7332 1.0000
meadow | 0.2112 0.7375 0.6751 0.6281 1.0000
. estat sum
Estimation sample clogit Number of obs = 10,224
-------------------------------------------------------------------
Variable | Mean Std. Dev. Min Max
-------------+-----------------------------------------------------
choice | .3333333 .4714276 0 1
payment | 111.7743 104.5241 0 270
ASC | .3333333 .4714276 0 1
fallow | 3.264867 4.094869 0 10
landsc | 3.058294 3.851992 0 10
meadow | 3.055947 3.850655 0 10
-------------------------------------------------------------------
. summarize choice payment ASC $cond
Variable | Obs Mean Std. Dev. Min Max
-------------+---------------------------------------------------------
choice | 10,224 .3333333 .4714276 0 1
payment | 10,224 111.7743 104.5241 0 270
ASC | 10,224 .3333333 .4714276 0 1
fallow | 10,224 3.264867 4.094869 0 10
landsc | 10,224 3.058294 3.851992 0 10
-------------+---------------------------------------------------------
meadow | 10,224 3.055947 3.850655 0 10
I'm very new to choice modelling and I've been trying to analyse my data with Apollo, first starting with a simple MNL and then with a MIXL. However, the results that I get from a MNL with Apollo are completely different than those that I got with R package gmnl and in Stata (those also make sense, while Apollo ones don't). I've tried using different estimation routines, but they all end up with a similar result. I'm wondering what could be causing the difference and how to fix it, as I think I've specified the utilities correctly?
My data comes from a choice experiment where each participant made 8 choices among three unlabelled alternatives, one of which was an opt-out. Each alternative had four continuous attributes, one of which was payment that they would receive.
You can find the code and outputs for both models below. Thank you so much for your help!
################################MNL
apollo_initialise()
apollo_control = list(
modelName = "MNL full",
modelDescr = "poskus",
indivID = "ID",
nCores = 4,
outputDirectory = "output",
panelData=T
)
database=DCE
apollo_beta=c(asc_1=0,
asc_2=0,
asc_nochoice=0,
b_fallow=0,
b_landsc=0,
b_meadow=0,
b_payment=0)
apollo_fixed=c("asc_1")
apollo_inputs=apollo_validateInputs()
apollo_probabilities=function(apollo_beta, apollo_inputs, functionality="estimate"){
apollo_attach(apollo_beta, apollo_inputs)
on.exit(apollo_detach(apollo_beta,apollo_inputs))
P=list()
V=list()
V[["Nochoice"]]=asc_nochoice
V[["Alt1"]]=asc_1+b_fallow*alt1.fallow+b_landsc*alt1.landsc+b_meadow*alt1.meadow+b_payment*alt1.payment
V[["Alt2"]]=asc_2+b_fallow*alt2.fallow+b_landsc*alt2.landsc+b_meadow*alt2.meadow+b_payment*alt2.payment
mnl_settings = list(
alternatives = c(Nochoice=1, Alt1=2, Alt2=3),
choiceVar = Choice,
utilities = V
)
P[["model"]] = apollo_mnl(mnl_settings, functionality)
P = apollo_panelProd(P, apollo_inputs, functionality)
P = apollo_prepareProb(P, apollo_inputs, functionality)
return(P)
}
estimate_settings=list(estimationRoutine="BHHH")
full.m = apollo_estimate(apollo_beta, apollo_fixed, apollo_probabilities, apollo_inputs, estimate_settings)
modelOutput_settings=list(printCovar=T,printPVal=2)
modelsumFE<-apollo_modelOutput(full.m, modelOutput_settings)
########## Model output:
Model run by Ziva using Apollo 0.2.7 on R 4.1.3 for Windows.
www.ApolloChoiceModelling.com
Model name : MNL full
Model description : poskus
Model run at : 2022-06-26 16:59:24
Estimation method : bhhh
Model diagnosis : successive function values within relative tolerance limit (reltol)
Number of individuals : 426
Number of rows in database : 3568
Number of modelled outcomes : 3568
Number of cores used : 4
Model without mixing
LL(start) : -3919.85
LL(0) : -3919.85
LL(C) : -3848.66
LL(final) : -3730.4
Rho-square (0) : 0.0483
Adj.Rho-square (0) : 0.0468
Rho-square (C) : 0.0307
Adj.Rho-square (C) : 0.0292
AIC : 7472.8
BIC : 7509.87
Estimated parameters : 6
Time taken (hh:mm:ss) : 00:00:22.56
pre-estimation : 00:00:20.42
estimation : 00:00:1.3
post-estimation : 00:00:0.85
Iterations : 36 (successive function values within relative tolerance limit (reltol))
Min abs eigenvalue of Hessian : 49.02786
Unconstrained optimisation.
Estimates:
Estimate s.e. t.rat.(0) p(2-sided) Rob.s.e. Rob.t.rat.(0) p(2-sided)
asc_1 0.000000 NA NA NA NA NA NA
asc_2 0.482161 0.043160 11.1715 0.00000 0.078739 6.124 9.154e-10
asc_nochoice -0.635585 0.142185 -4.4701 7.817e-06 0.134150 -4.738 2.160e-06
b_fallow -0.019060 0.009968 -1.9121 0.05586 0.009866 -1.932 0.05337
b_landsc 0.011238 0.009655 1.1640 0.24443 0.009428 1.192 0.23328
b_meadow -0.008461 0.008854 -0.9556 0.33925 0.008059 -1.050 0.29377
b_payment -0.004235 2.9279e-04 -14.4650 0.00000 2.4936e-04 -16.984 0.00000
Classical covariance matrix:
asc_2 asc_nochoice b_fallow b_landsc b_meadow b_payment
asc_2 0.001863 1.617e-05 -7.834e-05 -5.397e-05 -5.091e-05 -1.616e-06
asc_nochoice 1.617e-05 0.020216 0.001215 0.001096 9.7733e-04 2.202e-05
b_fallow -7.834e-05 0.001215 9.936e-05 7.358e-05 6.082e-05 1.023e-06
b_landsc -5.397e-05 0.001096 7.358e-05 9.322e-05 5.569e-05 5.146e-07
b_meadow -5.091e-05 9.7733e-04 6.082e-05 5.569e-05 7.839e-05 5.944e-07
b_payment -1.616e-06 2.202e-05 1.023e-06 5.146e-07 5.944e-07 8.573e-08
Robust covariance matrix:
asc_2 asc_nochoice b_fallow b_landsc b_meadow b_payment
asc_2 0.006200 0.001683 -1.031e-05 -1.136e-05 2.353e-05 4.935e-06
asc_nochoice 0.001683 0.017996 0.001133 0.001080 9.2037e-04 1.612e-05
b_fallow -1.031e-05 0.001133 9.733e-05 7.342e-05 5.667e-05 6.818e-07
b_landsc -1.136e-05 0.001080 7.342e-05 8.889e-05 5.608e-05 4.451e-07
b_meadow 2.353e-05 9.2037e-04 5.667e-05 5.608e-05 6.495e-05 5.695e-07
b_payment 4.935e-06 1.612e-05 6.818e-07 4.451e-07 5.695e-07 6.218e-08
Here you can find the results from Stata for comparison:
CONDITIONAL LOGIT model
. clogit choice payment ASC $cond, group(idchoice) nolog
Conditional (fixed-effects) logistic regression
Number of obs = 10,224
LR chi2(5) = 874.66
Prob > chi2 = 0.0000
Log likelihood = -3306.7411 Pseudo R2 = 0.1168
------------------------------------------------------------------------------
choice | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
payment | .0064578 .0003533 18.28 0.000 .0057654 .0071503
ASC | .1289545 .1550375 0.83 0.406 -.1749134 .4328225
fallow | -.110634 .0101162 -10.94 0.000 -.1304613 -.0908066
landsc | -.1268549 .0099619 -12.73 0.000 -.1463798 -.10733
meadow | -.0787455 .0087474 -9.00 0.000 -.0958901 -.0616009
------------------------------------------------------------------------------
. estat ic
Akaike's information criterion and Bayesian information criterion
-----------------------------------------------------------------------------
Model | N ll(null) ll(model) df AIC BIC
-------------+---------------------------------------------------------------
. | 10,224 -3744.071 -3306.741 5 6623.482 6659.645
-----------------------------------------------------------------------------
Note: BIC uses N = number of observations. See [R] BIC note.
. estat vce, c
Correlation matrix of coefficients of clogit model
| choice
e(V) | payment ASC fallow landsc meadow
-------------+--------------------------------------------------
choice |
payment | 1.0000
ASC | 0.6492 1.0000
fallow | 0.2884 0.8152 1.0000
landsc | 0.1791 0.7513 0.7332 1.0000
meadow | 0.2112 0.7375 0.6751 0.6281 1.0000
. estat sum
Estimation sample clogit Number of obs = 10,224
-------------------------------------------------------------------
Variable | Mean Std. Dev. Min Max
-------------+-----------------------------------------------------
choice | .3333333 .4714276 0 1
payment | 111.7743 104.5241 0 270
ASC | .3333333 .4714276 0 1
fallow | 3.264867 4.094869 0 10
landsc | 3.058294 3.851992 0 10
meadow | 3.055947 3.850655 0 10
-------------------------------------------------------------------
. summarize choice payment ASC $cond
Variable | Obs Mean Std. Dev. Min Max
-------------+---------------------------------------------------------
choice | 10,224 .3333333 .4714276 0 1
payment | 10,224 111.7743 104.5241 0 270
ASC | 10,224 .3333333 .4714276 0 1
fallow | 10,224 3.264867 4.094869 0 10
landsc | 10,224 3.058294 3.851992 0 10
-------------+---------------------------------------------------------
meadow | 10,224 3.055947 3.850655 0 10