Important: Read this before posting to this forum

  1. This forum is for questions related to the use of Apollo. We will answer some general choice modelling questions too, where appropriate, and time permitting. We cannot answer questions about how to estimate choice models with other software packages.
  2. There is a very detailed manual for Apollo available at http://www.ApolloChoiceModelling.com/manual.html. This contains detailed descriptions of the various Apollo functions, and numerous examples are available at http://www.ApolloChoiceModelling.com/examples.html. In addition, help files are available for all functions, using e.g. ?apollo_mnl
  3. Before asking a question on the forum, users are kindly requested to follow these steps:
    1. Check that the same issue has not already been addressed in the forum - there is a search tool.
    2. Ensure that the correct syntax has been used. For any function, detailed instructions are available directly in Apollo, e.g. by using ?apollo_mnl for apollo_mnl
    3. Check the frequently asked questions section on the Apollo website, which discusses some common issues/failures. Please see http://www.apollochoicemodelling.com/faq.html
    4. Make sure that R is using the latest official release of Apollo.
  4. If the above steps do not resolve the issue, then users should follow these steps when posting a question:
    1. provide full details on the issue, including the entire code and output, including any error messages
    2. posts will not immediately appear on the forum, but will be checked by a moderator first. This may take a day or two at busy times. There is no need to submit the post multiple times.

MNL estimates incorrect

Ask questions about the results reported after estimation. If the output includes errors, please include your model code if possible.
Post Reply
zivaal
Posts: 3
Joined: 20 Jun 2022, 15:37

MNL estimates incorrect

Post by zivaal »

Hello,

I'm very new to choice modelling and I've been trying to analyse my data with Apollo, first starting with a simple MNL and then with a MIXL. However, the results that I get from a MNL with Apollo are completely different than those that I got with R package gmnl and in Stata (those also make sense, while Apollo ones don't). I've tried using different estimation routines, but they all end up with a similar result. I'm wondering what could be causing the difference and how to fix it, as I think I've specified the utilities correctly?

My data comes from a choice experiment where each participant made 8 choices among three unlabelled alternatives, one of which was an opt-out. Each alternative had four continuous attributes, one of which was payment that they would receive.

You can find the code and outputs for both models below. Thank you so much for your help!
################################MNL
apollo_initialise()
apollo_control = list(
modelName = "MNL full",
modelDescr = "poskus",
indivID = "ID",
nCores = 4,
outputDirectory = "output",
panelData=T
)
database=DCE
apollo_beta=c(asc_1=0,
asc_2=0,
asc_nochoice=0,
b_fallow=0,
b_landsc=0,
b_meadow=0,
b_payment=0)
apollo_fixed=c("asc_1")
apollo_inputs=apollo_validateInputs()

apollo_probabilities=function(apollo_beta, apollo_inputs, functionality="estimate"){
apollo_attach(apollo_beta, apollo_inputs)
on.exit(apollo_detach(apollo_beta,apollo_inputs))
P=list()

V=list()
V[["Nochoice"]]=asc_nochoice
V[["Alt1"]]=asc_1+b_fallow*alt1.fallow+b_landsc*alt1.landsc+b_meadow*alt1.meadow+b_payment*alt1.payment
V[["Alt2"]]=asc_2+b_fallow*alt2.fallow+b_landsc*alt2.landsc+b_meadow*alt2.meadow+b_payment*alt2.payment

mnl_settings = list(
alternatives = c(Nochoice=1, Alt1=2, Alt2=3),
choiceVar = Choice,
utilities = V
)

P[["model"]] = apollo_mnl(mnl_settings, functionality)
P = apollo_panelProd(P, apollo_inputs, functionality)
P = apollo_prepareProb(P, apollo_inputs, functionality)

return(P)

}
estimate_settings=list(estimationRoutine="BHHH")
full.m = apollo_estimate(apollo_beta, apollo_fixed, apollo_probabilities, apollo_inputs, estimate_settings)
modelOutput_settings=list(printCovar=T,printPVal=2)
modelsumFE<-apollo_modelOutput(full.m, modelOutput_settings)

########## Model output:
Model run by Ziva using Apollo 0.2.7 on R 4.1.3 for Windows.
www.ApolloChoiceModelling.com

Model name : MNL full
Model description : poskus
Model run at : 2022-06-26 16:59:24
Estimation method : bhhh
Model diagnosis : successive function values within relative tolerance limit (reltol)
Number of individuals : 426
Number of rows in database : 3568
Number of modelled outcomes : 3568

Number of cores used : 4
Model without mixing

LL(start) : -3919.85
LL(0) : -3919.85
LL(C) : -3848.66
LL(final) : -3730.4
Rho-square (0) : 0.0483
Adj.Rho-square (0) : 0.0468
Rho-square (C) : 0.0307
Adj.Rho-square (C) : 0.0292
AIC : 7472.8
BIC : 7509.87

Estimated parameters : 6
Time taken (hh:mm:ss) : 00:00:22.56
pre-estimation : 00:00:20.42
estimation : 00:00:1.3
post-estimation : 00:00:0.85
Iterations : 36 (successive function values within relative tolerance limit (reltol))
Min abs eigenvalue of Hessian : 49.02786

Unconstrained optimisation.

Estimates:
Estimate s.e. t.rat.(0) p(2-sided) Rob.s.e. Rob.t.rat.(0) p(2-sided)
asc_1 0.000000 NA NA NA NA NA NA
asc_2 0.482161 0.043160 11.1715 0.00000 0.078739 6.124 9.154e-10
asc_nochoice -0.635585 0.142185 -4.4701 7.817e-06 0.134150 -4.738 2.160e-06
b_fallow -0.019060 0.009968 -1.9121 0.05586 0.009866 -1.932 0.05337
b_landsc 0.011238 0.009655 1.1640 0.24443 0.009428 1.192 0.23328
b_meadow -0.008461 0.008854 -0.9556 0.33925 0.008059 -1.050 0.29377
b_payment -0.004235 2.9279e-04 -14.4650 0.00000 2.4936e-04 -16.984 0.00000


Classical covariance matrix:
asc_2 asc_nochoice b_fallow b_landsc b_meadow b_payment
asc_2 0.001863 1.617e-05 -7.834e-05 -5.397e-05 -5.091e-05 -1.616e-06
asc_nochoice 1.617e-05 0.020216 0.001215 0.001096 9.7733e-04 2.202e-05
b_fallow -7.834e-05 0.001215 9.936e-05 7.358e-05 6.082e-05 1.023e-06
b_landsc -5.397e-05 0.001096 7.358e-05 9.322e-05 5.569e-05 5.146e-07
b_meadow -5.091e-05 9.7733e-04 6.082e-05 5.569e-05 7.839e-05 5.944e-07
b_payment -1.616e-06 2.202e-05 1.023e-06 5.146e-07 5.944e-07 8.573e-08

Robust covariance matrix:
asc_2 asc_nochoice b_fallow b_landsc b_meadow b_payment
asc_2 0.006200 0.001683 -1.031e-05 -1.136e-05 2.353e-05 4.935e-06
asc_nochoice 0.001683 0.017996 0.001133 0.001080 9.2037e-04 1.612e-05
b_fallow -1.031e-05 0.001133 9.733e-05 7.342e-05 5.667e-05 6.818e-07
b_landsc -1.136e-05 0.001080 7.342e-05 8.889e-05 5.608e-05 4.451e-07
b_meadow 2.353e-05 9.2037e-04 5.667e-05 5.608e-05 6.495e-05 5.695e-07
b_payment 4.935e-06 1.612e-05 6.818e-07 4.451e-07 5.695e-07 6.218e-08


Here you can find the results from Stata for comparison:
CONDITIONAL LOGIT model
. clogit choice payment ASC $cond, group(idchoice) nolog

Conditional (fixed-effects) logistic regression

Number of obs = 10,224
LR chi2(5) = 874.66
Prob > chi2 = 0.0000
Log likelihood = -3306.7411 Pseudo R2 = 0.1168

------------------------------------------------------------------------------
choice | Coef. Std. Err. z P>|z| [95% Conf. Interval]
-------------+----------------------------------------------------------------
payment | .0064578 .0003533 18.28 0.000 .0057654 .0071503
ASC | .1289545 .1550375 0.83 0.406 -.1749134 .4328225
fallow | -.110634 .0101162 -10.94 0.000 -.1304613 -.0908066
landsc | -.1268549 .0099619 -12.73 0.000 -.1463798 -.10733
meadow | -.0787455 .0087474 -9.00 0.000 -.0958901 -.0616009
------------------------------------------------------------------------------

. estat ic

Akaike's information criterion and Bayesian information criterion

-----------------------------------------------------------------------------
Model | N ll(null) ll(model) df AIC BIC
-------------+---------------------------------------------------------------
. | 10,224 -3744.071 -3306.741 5 6623.482 6659.645
-----------------------------------------------------------------------------
Note: BIC uses N = number of observations. See [R] BIC note.

. estat vce, c

Correlation matrix of coefficients of clogit model

| choice
e(V) | payment ASC fallow landsc meadow
-------------+--------------------------------------------------
choice |
payment | 1.0000
ASC | 0.6492 1.0000
fallow | 0.2884 0.8152 1.0000
landsc | 0.1791 0.7513 0.7332 1.0000
meadow | 0.2112 0.7375 0.6751 0.6281 1.0000

. estat sum

Estimation sample clogit Number of obs = 10,224

-------------------------------------------------------------------
Variable | Mean Std. Dev. Min Max
-------------+-----------------------------------------------------
choice | .3333333 .4714276 0 1
payment | 111.7743 104.5241 0 270
ASC | .3333333 .4714276 0 1
fallow | 3.264867 4.094869 0 10
landsc | 3.058294 3.851992 0 10
meadow | 3.055947 3.850655 0 10
-------------------------------------------------------------------

. summarize choice payment ASC $cond

Variable | Obs Mean Std. Dev. Min Max
-------------+---------------------------------------------------------
choice | 10,224 .3333333 .4714276 0 1
payment | 10,224 111.7743 104.5241 0 270
ASC | 10,224 .3333333 .4714276 0 1
fallow | 10,224 3.264867 4.094869 0 10
landsc | 10,224 3.058294 3.851992 0 10
-------------+---------------------------------------------------------
meadow | 10,224 3.055947 3.850655 0 10
stephanehess
Site Admin
Posts: 974
Joined: 24 Apr 2020, 16:29

Re: MNL estimates incorrect

Post by stephanehess »

Hi

it looks like your model specifications are different. You only have one ASC in the Stata code, while in the Apollo model, the asc_nochoice seems to capture a lot of the disutility of the no-payment option. Also, the sample size is different.

Stephane
--------------------------------
Stephane Hess
www.stephanehess.me.uk
zivaal
Posts: 3
Joined: 20 Jun 2022, 15:37

Re: MNL estimates incorrect

Post by zivaal »

Hi,

Thank you for your fast reply. Unfortunaly, fixing none of this helps with the model - even when I remove ASC1 (in Stata, No choice is coded as ASC, so I left that one) from model specification and remove a couple of duplicates that I didn't notice earlier the model results remain different from Stata. Please find the updated code and output below.

Best wishes,
Ziva

########################## Code ############################
apollo_initialise()
apollo_control = list(
modelName = "MNL full",
modelDescr = "poskus",
indivID = "ID",
nCores = 4,
outputDirectory = "output",
panelData=T
)
database=DCE
apollo_beta=c(asc_nochoice=0,
b_fallow=0,
b_landsc=0,
b_meadow=0,
b_payment=0)
apollo_fixed=c()
apollo_inputs=apollo_validateInputs()

apollo_probabilities=function(apollo_beta, apollo_inputs, functionality="estimate"){
apollo_attach(apollo_beta, apollo_inputs)
on.exit(apollo_detach(apollo_beta,apollo_inputs))
P=list()

V=list()
V[["Nochoice"]]=asc_nochoice
V[["Alt1"]]=b_fallow*alt1.fallow+b_landsc*alt1.landsc+b_meadow*alt1.meadow+b_payment*alt1.payment
V[["Alt2"]]=b_fallow*alt2.fallow+b_landsc*alt2.landsc+b_meadow*alt2.meadow+b_payment*alt2.payment

mnl_settings = list(
alternatives = c(Nochoice=1, Alt1=2, Alt2=3),
choiceVar = Choice,
utilities = V
)

P[["model"]] = apollo_mnl(mnl_settings, functionality)
P = apollo_panelProd(P, apollo_inputs, functionality)
P = apollo_prepareProb(P, apollo_inputs, functionality)

return(P)

}
estimate_settings=list(estimationRoutine="BHHH")
full.m = apollo_estimate(apollo_beta, apollo_fixed, apollo_probabilities, apollo_inputs, estimate_settings)
modelOutput_settings=list(printCovar=T,printPVal=2)
modelsumFE<-apollo_modelOutput(full.m, modelOutput_settings)

########################### model output ##############################
Model run by Ziva using Apollo 0.2.7 on R 4.1.3 for Windows.
www.ApolloChoiceModelling.com

Model name : MNL full
Model description : poskus
Model run at : 2022-06-28 17:33:55
Estimation method : bhhh
Model diagnosis : successive function values within relative tolerance limit (reltol)
Number of individuals : 426
Number of rows in database : 3408
Number of modelled outcomes : 3408

Number of cores used : 4
Model without mixing

LL(start) : -3744.07
LL(0) : -3744.07
LL(C) : -3676.94
LL(final) : -3627.89
Rho-square (0) : 0.031
Adj.Rho-square (0) : 0.0297
Rho-square (C) : 0.0133
Adj.Rho-square (C) : 0.012
AIC : 7265.78
BIC : 7296.45

Estimated parameters : 5
Time taken (hh:mm:ss) : 00:00:13.47
pre-estimation : 00:00:12.26
estimation : 00:00:0.49
post-estimation : 00:00:0.73
Iterations : 15 (successive function values within relative tolerance limit (reltol))
Min abs eigenvalue of Hessian : 50.62134

Unconstrained optimisation.

Estimates:
Estimate s.e. t.rat.(0) p(2-sided) Rob.s.e. Rob.t.rat.(0) p(2-sided)
asc_nochoice -0.622570 0.139928 -4.44923 8.618e-06 0.124387 -5.00509 5.583e-07
b_fallow 8.8477e-04 0.009735 0.09089 0.927581 0.009932 0.08908 0.929019
b_landsc 0.025523 0.009415 2.71083 0.006712 0.009019 2.82995 0.004656
b_meadow 0.005303 0.008800 0.60264 0.546746 0.008074 0.65683 0.511292
b_payment -0.003911 2.9005e-04 -13.48557 0.000000 2.6152e-04 -14.95639 0.000000


Classical covariance matrix:
asc_nochoice b_fallow b_landsc b_meadow b_payment
asc_nochoice 0.019580 0.001182 0.001047 9.6048e-04 2.090e-05
b_fallow 0.001182 9.476e-05 6.835e-05 5.881e-05 8.809e-07
b_landsc 0.001047 6.835e-05 8.865e-05 5.286e-05 3.892e-07
b_meadow 9.6048e-04 5.881e-05 5.286e-05 7.744e-05 4.773e-07
b_payment 2.090e-05 8.809e-07 3.892e-07 4.773e-07 8.413e-08

Robust covariance matrix:
asc_nochoice b_fallow b_landsc b_meadow b_payment
asc_nochoice 0.015472 0.001062 9.7411e-04 8.5323e-04 1.355e-05
b_fallow 0.001062 9.865e-05 7.057e-05 6.194e-05 8.820e-07
b_landsc 9.7411e-04 7.057e-05 8.134e-05 5.571e-05 5.594e-07
b_meadow 8.5323e-04 6.194e-05 5.571e-05 6.519e-05 6.347e-07
b_payment 1.355e-05 8.820e-07 5.594e-07 6.347e-07 6.839e-08
stephanehess
Site Admin
Posts: 974
Joined: 24 Apr 2020, 16:29

Re: MNL estimates incorrect

Post by stephanehess »

Hi

I can only assume then that you made a mistake in preparing the data for Apollo. Your model fit is a lot worse in Apollo, and if the model specification is correct (which it looks to be), then it must be the data

Stephane
--------------------------------
Stephane Hess
www.stephanehess.me.uk
zivaal
Posts: 3
Joined: 20 Jun 2022, 15:37

Re: MNL estimates incorrect

Post by zivaal »

Hi,

I'm sorry for my late reply and thank you for your response. I've manually checked the data that I used in Apollo and in Stata and I haven't found any differences between them, besides the different format (long vs wide). You can see the head of both datasets below:

Apollo format:
Q10 task choice alt1.fallow alt1.landsc alt1.meadow alt1.payment alt2.fallow alt2.landsc alt2.meadow alt2.payment alt3.fallow
<dbl> <dbl> <dbl> <int> <int> <int> <int> <int> <int> <int> <int> <int>
1 100218307 1 1 0 0 10 270 0 10 5 200 0
2 100218307 2 1 10 0 0 270 0 5 10 200 0
3 100218307 3 1 5 3 3 50 0 10 0 270 0
4 100218307 4 1 10 10 0 200 10 3 3 50 0
5 100218307 5 2 5 5 3 50 0 0 10 200 0
6 100218307 6 2 0 3 10 120 10 0 0 270 0


Stata format:

Q10 task Alt fallow landsc meadow payment choice optout
<int> <dbl> <int> <int> <int> <int> <int> <dbl> <dbl>
1 100218307 1 1 0 0 10 270 1 0 1
2 100218307 1 2 0 10 5 200 0 0
3 100218307 1 3 0 0 0 0 0 1
4 100218307 2 1 10 0 0 270 1 0
5 100218307 2 2 0 5 10 200 0 0
6 100218307 2 3 0 0 0 0 0 1
dpalma
Posts: 190
Joined: 24 Apr 2020, 17:54

Re: MNL estimates incorrect

Post by dpalma »

Hi Ziva,

Stephane and I looked at your data, and there seems to be a few issues with it.

In the Apollo dataset, the variable "Choice" takes values from 1 to 6, but there are only three alternatives.

In the stata data, the "Alt" variable takes the same value in multiple rows for the same "task".

I would recommend you look into it again.

Best wishes
David
Post Reply