Important: Read this before posting to this forum

  1. This forum is for questions related to the use of Apollo. We will answer some general choice modelling questions too, where appropriate, and time permitting. We cannot answer questions about how to estimate choice models with other software packages.
  2. There is a very detailed manual for Apollo available at http://www.ApolloChoiceModelling.com/manual.html. This contains detailed descriptions of the various Apollo functions, and numerous examples are available at http://www.ApolloChoiceModelling.com/examples.html. In addition, help files are available for all functions, using e.g. ?apollo_mnl
  3. Before asking a question on the forum, users are kindly requested to follow these steps:
    1. Check that the same issue has not already been addressed in the forum - there is a search tool.
    2. Ensure that the correct syntax has been used. For any function, detailed instructions are available directly in Apollo, e.g. by using ?apollo_mnl for apollo_mnl
    3. Check the frequently asked questions section on the Apollo website, which discusses some common issues/failures. Please see http://www.apollochoicemodelling.com/faq.html
    4. Make sure that R is using the latest official release of Apollo.
  4. If the above steps do not resolve the issue, then users should follow these steps when posting a question:
    1. provide full details on the issue, including the entire code and output, including any error messages
    2. posts will not immediately appear on the forum, but will be checked by a moderator first. This may take a day or two at busy times. There is no need to submit the post multiple times.

version 0.2.7 and 0.2.1 generate different results

Ask questions about how to estimate models and how to change your settings for estimation.
Post Reply
wulalala
Posts: 6
Joined: 22 Jan 2022, 00:34

version 0.2.7 and 0.2.1 generate different results

Post by wulalala »

Hi there,

I ran a nested logit model on same dataset using same codes in version 0.2.1 and version 0.2.7.
V0.2.1 generated meaningful results (i.e., significant estimates, logsum coefficient between 0 and 1, standard error with actual numbers as values),
but V0.2.7 generated crazy results (i.e., standard errors and t statistics are all NaN, logsum coefficient bigger than 1, got error of Hessian could not be calculated).
which results do you suggest to trust and use?

Thank you
dpalma
Posts: 190
Joined: 24 Apr 2020, 17:54

Re: version 0.2.7 and 0.2.1 generate different results

Post by dpalma »

Hi,

Could you attach your script and the output in both versions? Than can help us figure out what is driving this issue.

Best wishes
David
wulalala
Posts: 6
Joined: 22 Jan 2022, 00:34

Re: version 0.2.7 and 0.2.1 generate different results

Post by wulalala »

Thank you for the response.

the code I used is:

### Clear memory
rm(list = ls())

### Load Apollo library
library(apollo)
library(reshape2)
### Initialise code
apollo_initialise()

### Set core controls
apollo_control = list(
modelName ="Apollo_try_1",
modelDescr ="NL_two_level on mode choice",
indivID ="caseID",
panelData=FALSE,
weights="WTTRDFIN"
)
database = read.csv("C:/Users/data.csv",header=TRUE)
apollo_beta=c(
asc_4=0,
asc_0=0,
asc_30=0,
asc_2=0,
asc_1=0,
asc_31=0,
asc_32=0,

b_tt_30=0,
b_tt_0=0,
b_tt_2=0,
b_tt_4=0,
b_tt_1=0,
b_tt_31=0,

b_c=0,

b_r_2=0,
b_r_30=0,
b_r_31=0,

b_acc_t_o=0,
b_acc_t_d=0,

lambda_300=0.5

)
apollo_fixed = c("asc_2")
apollo_inputs = apollo_validateInputs()
apollo_probabilities=function(apollo_beta, apollo_inputs, functionality="estimate"){

### Attach inputs and detach after function exit
apollo_attach(apollo_beta, apollo_inputs)
on.exit(apollo_detach(apollo_beta, apollo_inputs))

### Create list of probabilities P
P = list()

### List of utilities: these must use the same names as in mnl_settings, order is irrelevant
v = list()

#
v[['0']]=asc_0 + b_tt_0*ivtt_0/0.8 + b_c*cost_0 + b_acc_t_o * acc_t_o_0 + b_acc_t_d * acc_t_d_0
v[['1']]=asc_1 + b_tt_1 * ivtt_1/0.8 + b_c*cost_1 + b_acc_t_o * acc_t_o_1 + b_acc_t_d * acc_t_d_1
v[['2']]=asc_2 + b_tt_2 * ivtt_2/0.8 + b_r_2*(tp>18)
v[['30']]=asc_30 + b_tt_30 * ivtt_30/0.8
v[['4']]=asc_4 + b_tt_4* ivtt_4/0.8 + b_c*cost_4 + b_acc_t_o * acc_t_o_4
v[['31']]=asc_31 + b_tt_31 * ivtt_31/0.8 + b_c*cost_31 + b_acc_t_o * acc_t_o_31 + b_r_31*(tp>18)
v[['32']]=asc_32 + b_tt_30 * ivtt_32/0.8 + b_c*cost_32 + b_acc_t_o * acc_t_o_32 + b_r_30*(tp>18)

### Specify nests for NL model
nlNests = list(root=1, '300'=lambda_300)

### Specify tree structure for NL model
nlStructure= list()
nlStructure[["root"]] = c("300","0","1","4","2","30")
nlStructure[["300"]] = c("31","32")
# nlStructure[['auto']] =c("0","1","4","2","30")



### Define settings for NL model
nl_settings <- list(
alternatives = c("0"=0,"1"=1,"2"=2,"30"=30,"4"=4,"31"=31,"32"=32),
avail = list("0"=ava_0, "1"=ava_1,"2"=ava_2,"30"=ava_30,"4"=ava_4,"31"=ava_31,"32"=ava_32),

choiceVar = choice,
V=v,
nlNests = nlNests,
nlStructure = nlStructure
)


### Compute probabilities using NL model
P[["model"]] = apollo_nl(nl_settings, functionality)


### Take product across observation for same individual
#P = apollo_panelProd(P, apollo_inputs, functionality)

P=apollo_weighting(P,apollo_inputs,functionality)

### Prepare and return outputs of function
P = apollo_prepareProb(P, apollo_inputs, functionality)
return(P)
}
model = apollo_estimate(apollo_beta, apollo_fixed, apollo_probabilities, apollo_inputs)
apollo_modelOutput(model,list(printPVal = TRUE))



Then the result I got from v0.2.1 are:

Model name : Apollo_try_1
Model description : NL_two_level on mode choice
Model run at : 2022-07-24 19:41:49
Estimation method : bfgs
Model diagnosis : successful convergence
Number of individuals : 1202
Number of observations : 1202

Number of cores used : 1
Model without mixing

LL(start) : -2252.931
LL(0) : -135874.8
LL(final) : -75010.96
Rho-square (0) : 0.4479
Adj.Rho-square (0) : 0.4478
AIC : 150059.9
BIC : 150156.7


Estimated parameters : 19
Time taken (hh:mm:ss) : 00:00:33.45
pre-estimation : 00:00:0.42
estimation : 00:00:15.35
post-estimation : 00:00:17.67
Iterations : 51
Min abs eigenvalue of Hessian : 13.79981

Estimates:
Estimate s.e. t.rat.(0) p(1-sided) Rob.s.e. Rob.t.rat.(0) p(1-sided)
asc_4 0.32405 0.108356 2.991 0.001392 NA NA NA
asc_0 -1.41540 0.096069 -14.733 0.000000 NA NA NA
asc_30 -0.73030 0.044914 -16.260 0.000000 NA NA NA
asc_2 0.00000 NA NA NA NA NA NA
asc_1 4.50184 0.218546 20.599 0.000000 NA NA NA
asc_31 2.43985 0.051411 47.458 0.000000 NA NA NA
asc_32 0.86737 0.070991 12.218 0.000000 NA NA NA
b_tt_1 -0.02561 4.3579e-04 -58.775 0.000000 NA NA NA
b_tt_0 -0.01158 2.7496e-04 -42.123 0.000000 NA NA NA
b_tt_30 -0.02711 2.9747e-04 -91.123 0.000000 NA NA NA
b_tt_4 -0.01045 6.3879e-04 -16.358 0.000000 NA NA NA
b_tt_2 -0.01163 1.0923e-04 -106.450 0.000000 NA NA NA
b_tt_31 -0.03438 2.8219e-04 -121.841 0.000000 NA NA NA
b_c -0.04411 0.001712 -25.765 0.000000 NA NA NA
b_r_2 -0.73457 0.037518 -19.579 0.000000 NA NA NA
b_r_30 -0.68862 0.036589 -18.820 0.000000 NA NA NA
b_r_31 -0.47024 0.029363 -16.015 0.000000 NA NA NA
b_acc_t_o -0.01365 4.1957e-04 -32.543 0.000000 NA NA NA
b_acc_t_d -0.01152 3.9993e-04 -28.802 0.000000 NA NA NA
lambda_300 0.35345 0.011997 29.461 0.000000 NA NA NA

Nesting structure for NL model component :
Nest: root (1)
|-Nest: micro (0.3535)
| |-Alternative: 31
| '-Alternative: 32
|----Alternative: 0
|----Alternative: 1
|----Alternative: 4
|----Alternative: 2
'----Alternative: 30

the result using v0.2.7
Model name : Apollo_try_1
Model description : NL_two_level on mode choice
Model run at : 2022-07-24 19:46:50
Estimation method : bfgs
Model diagnosis : successful convergence
Number of individuals : 1202
Number of rows in database : 1202
Number of modelled outcomes : 1202

Number of cores used : 1
Model without mixing

LL(start) : -2252.93
LL(0) : -135874.8
LL(C) : Not applicable
LL(final) : -70139.01
Rho-square (0) : 0.4838
Adj.Rho-square (0) : 0.4837
AIC : 140316
BIC : 140412.8

Estimated parameters : 19
Time taken (hh:mm:ss) : 00:00:19.97
pre-estimation : 00:00:0.38
estimation : 00:00:13.56
post-estimation : 00:00:6.03
Iterations : 81

Unconstrained optimisation.

Estimates:
Estimate s.e. t.rat.(0) p(1-sided) Rob.s.e. Rob.t.rat.(0) p(1-sided)
asc_4 -2.185482 NA NA NA NA NA NA
asc_0 -2.375304 NA NA NA NA NA NA
asc_30 -1.854660 NA NA NA NA NA NA
asc_2 0.000000 NA NA NA NA NA NA
asc_1 3.062217 NA NA NA NA NA NA
asc_31 2.751639 NA NA NA NA NA NA
asc_32 -1.309109 NA NA NA NA NA NA
b_tt_1 -0.024928 NA NA NA NA NA NA
b_tt_0 -0.010182 NA NA NA NA NA NA
b_tt_30 -0.026641 NA NA NA NA NA NA
b_tt_4 -0.006377 NA NA NA NA NA NA
b_tt_2 -0.013657 NA NA NA NA NA NA
b_tt_31 -0.045183 NA NA NA NA NA NA
b_c -0.037324 NA NA NA NA NA NA
b_r_2 -1.352313 NA NA NA NA NA NA
b_r_30 -2.909122 NA NA NA NA NA NA
b_r_31 -0.887248 NA NA NA NA NA NA
b_acc_t_o -0.017844 NA NA NA NA NA NA
b_acc_t_d -0.013958 NA NA NA NA NA NA
lambda_300 1.178495 NA NA NA NA NA NA

Nesting structure for NL model component :
Nest: root (1)
|-Nest: micro (1.1785)
| |-Alternative: 31
| '-Alternative: 32
|----Alternative: 0
|----Alternative: 1
|----Alternative: 4
|----Alternative: 2
'----Alternative: 30

WARNING: The nesting parameter for nest "300" should be between 0 and 1 (the nesting parameter for nest "root"), yet its value is 1.1785.
stephanehess
Site Admin
Posts: 974
Joined: 24 Apr 2020, 16:29

Re: version 0.2.7 and 0.2.1 generate different results

Post by stephanehess »

Hi

neither output makes sense as the starting LL is better than the final LL. Can you confirm that the two alternatives nested together are available at the same time? Also, what happens in an MNL model in terms of fit?

Stephane
--------------------------------
Stephane Hess
www.stephanehess.me.uk
wulalala
Posts: 6
Joined: 22 Jan 2022, 00:34

Re: version 0.2.7 and 0.2.1 generate different results

Post by wulalala »

Hi, is it because I used sample weights so the starting LL is better than final LL? Is starting LL accounting weights?
stephanehess
Site Admin
Posts: 974
Joined: 24 Apr 2020, 16:29

Re: version 0.2.7 and 0.2.1 generate different results

Post by stephanehess »

do your weights average to 1?

and what happens with MNL?
--------------------------------
Stephane Hess
www.stephanehess.me.uk
wulalala
Posts: 6
Joined: 22 Jan 2022, 00:34

Re: version 0.2.7 and 0.2.1 generate different results

Post by wulalala »

what do you mean by "weights average to 1"?
Just want to make sure I'm including weights properly before I try the MNL.
Each observation has a weight. The value of weights ranges between 30 to 400. I specified the column of weights in apollo_control, and also have another line of P=apollo_weighting(P,apollo_inputs,functionality) in apollo_probabilities. Am I applying weights correctly?

Thank you!
wulalala
Posts: 6
Joined: 22 Jan 2022, 00:34

Re: version 0.2.7 and 0.2.1 generate different results

Post by wulalala »

with currently how I applied weights, running a simple MNL model with only subset of the variables with v0.2.1 and 0.2.7 are shown below.
The MNL and nested logit model are based on the same dataset, but the nested logit model break the alternative 3 into three sub alternatives:3, 30 and 31.

running version 0.2.1 gave:
Model name : Apollo_try_1
Model description : Simple MNL model on mode choice
Model run at : 2022-08-29 17:36:09
Estimation method : bfgs
Model diagnosis : successful convergence
Number of individuals : 1222
Number of observations : 1222

Number of cores used : 1
Model without mixing

LL(start) : -1779.875
LL(0) : -113609.3
LL(final) : -59454.34
Rho-square (0) : 0.4767
Adj.Rho-square (0) : 0.4766
AIC : 118932.7
BIC : 118994


Estimated parameters : 12
Time taken (hh:mm:ss) : 00:00:12.39
pre-estimation : 00:00:0.48
estimation : 00:00:5.68
post-estimation : 00:00:6.24
Iterations : 29
Min abs eigenvalue of Hessian : 14.26168

Estimates:
Estimate s.e. t.rat.(0) p(1-sided) Rob.s.e. Rob.t.rat.(0) p(1-sided)
asc_4 -3.30300 0.119217 -27.706 0.000 NaN NaN NaN
asc_0 -4.36605 0.119418 -36.561 0.000 NaN NaN NaN
asc_3&30&31 -0.80870 0.138610 -5.834 2.700e-09 NaN NaN NaN
asc_2 -3.10955 0.156354 -19.888 0.000 NaN NaN NaN
asc_1 0.00000 NA NA NA NA NA NA
b_tt_1 -0.27468 0.004899 -56.067 0.000 NaN NaN NaN
b_tt_0 -0.10238 0.002996 -34.177 0.000 NaN NaN NaN
b_tt_3&30&31 -0.34850 0.003397 -102.577 0.000 NaN NaN NaN
b_tt_4 -0.17065 0.009093 -18.766 0.000 NaN NaN NaN
b_tt_2 -0.12850 0.001374 -93.536 0.000 NaN NaN NaN
b_c -0.25201 0.014738 -17.099 0.000 NaN NaN NaN
b_acc_t_o -0.12103 0.003042 -39.781 0.000 NaN NaN NaN
b_acc_t_d -0.06632 0.003518 -18.850 0.000 NaN NaN NaN

running Version 0.2.7 gave me:
Model name : Apollo_try_1
Model description : Simple MNL model on mode choice
Model run at : 2022-08-29 17:22:47
Estimation method : bfgs
Model diagnosis : successful convergence
Number of individuals : 1222
Number of rows in database : 1222
Number of modelled outcomes : 1222

Number of cores used : 1
Model without mixing

LL(start) : -1779.88
LL(0) : -113609.3
LL(C) : Not applicable
LL(final) : -59454.13
Rho-square (0) : 0.4767
Adj.Rho-square (0) : 0.4766
AIC : 118932.3
BIC : 118993.6

Estimated parameters : 12
Time taken (hh:mm:ss) : 00:00:4.11
pre-estimation : 00:00:0.45
estimation : 00:00:2.29
post-estimation : 00:00:1.37
Iterations : 35

Unconstrained optimisation.

Estimates:
Estimate s.e. t.rat.(0) p(1-sided) Rob.s.e. Rob.t.rat.(0) p(1-sided)
asc_4 -3.30304 NA NA NA NA NA NA
asc_0 -4.36604 NA NA NA NA NA NA
asc_3&30&31 -0.80870 NA NA NA NA NA NA
asc_2 -3.10954 NA NA NA NA NA NA
asc_1 0.00000 NA NA NA NA NA NA
b_tt_1 -0.27468 NA NA NA NA NA NA
b_tt_0 -0.10238 NA NA NA NA NA NA
b_tt_3&30&31 -0.34850 NA NA NA NA NA NA
b_tt_4 -0.17065 NA NA NA NA NA NA
b_tt_2 -0.12850 NA NA NA NA NA NA
b_c -0.25201 NA NA NA NA NA NA
b_acc_t_o -0.12103 NA NA NA NA NA NA
b_acc_t_d -0.06632 NA NA NA NA NA NA

what I am hoping to do:
I asked a question about applying weights in prediction (http://www.apollochoicemodelling.com/fo ... f=16&t=386). 0.2.7 was recommended to solve the problem I asked, but v0.2.7 gave me estimation result with NA s.e., t statistics and p values ( as shown above).

Thank you for your help!
stephanehess
Site Admin
Posts: 974
Joined: 24 Apr 2020, 16:29

Re: version 0.2.7 and 0.2.1 generate different results

Post by stephanehess »

Hi, in relation to the below, there is a bug in that the starting LL reported by Apollo does not incorporate weights, so you can ignore that for now. What I mean with whether the weights average to 1 is that across your observations, the sum of weights should ideally be the same as the number of observations.
wulalala wrote: 29 Aug 2022, 23:48 what do you mean by "weights average to 1"?
Just want to make sure I'm including weights properly before I try the MNL.
Each observation has a weight. The value of weights ranges between 30 to 400. I specified the column of weights in apollo_control, and also have another line of P=apollo_weighting(P,apollo_inputs,functionality) in apollo_probabilities. Am I applying weights correctly?

Thank you!
--------------------------------
Stephane Hess
www.stephanehess.me.uk
stephanehess
Site Admin
Posts: 974
Joined: 24 Apr 2020, 16:29

Re: version 0.2.7 and 0.2.1 generate different results

Post by stephanehess »

Hi

if you look at the MNL results, you can clearly see that in both 0.2.1 and 0.2.7 your model has an identification issue (no robust standard errors). So that's an issue you need to solve before you start looking at nested logit

Stephane
--------------------------------
Stephane Hess
www.stephanehess.me.uk
Post Reply