ApolloChoiceModelling forum

Hi there,

I ran a nested logit model on same dataset using same codes in version 0.2.1 and version 0.2.7.
V0.2.1 generated meaningful results (i.e., significant estimates, logsum coefficient between 0 and 1, standard error with actual numbers as values),
but V0.2.7 generated crazy results (i.e., standard errors and t statistics are all NaN, logsum coefficient bigger than 1, got error of Hessian could not be calculated).
which results do you suggest to trust and use?

Thank you

Hi,

Could you attach your script and the output in both versions? Than can help us figure out what is driving this issue.

Best wishes
David

Thank you for the response.

the code I used is:

### Clear memory
rm(list = ls())

### Load Apollo library
library(apollo)
library(reshape2)
### Initialise code
apollo_initialise()

### Set core controls
apollo_control = list(
modelName ="Apollo_try_1",
modelDescr ="NL_two_level on mode choice",
indivID ="caseID",
panelData=FALSE,
weights="WTTRDFIN"
)
database = read.csv("C:/Users/data.csv",header=TRUE)
apollo_beta=c(
asc_4=0,
asc_0=0,
asc_30=0,
asc_2=0,
asc_1=0,
asc_31=0,
asc_32=0,

b_tt_30=0,
b_tt_0=0,
b_tt_2=0,
b_tt_4=0,
b_tt_1=0,
b_tt_31=0,

b_c=0,

b_r_2=0,
b_r_30=0,
b_r_31=0,

b_acc_t_o=0,
b_acc_t_d=0,

lambda_300=0.5

)
apollo_fixed = c("asc_2")
apollo_inputs = apollo_validateInputs()
apollo_probabilities=function(apollo_beta, apollo_inputs, functionality="estimate"){

### Attach inputs and detach after function exit
apollo_attach(apollo_beta, apollo_inputs)
on.exit(apollo_detach(apollo_beta, apollo_inputs))

### Create list of probabilities P
P = list()

### List of utilities: these must use the same names as in mnl_settings, order is irrelevant
v = list()

#
v[['0']]=asc_0 + b_tt_0*ivtt_0/0.8 + b_c*cost_0 + b_acc_t_o * acc_t_o_0 + b_acc_t_d * acc_t_d_0
v[['1']]=asc_1 + b_tt_1 * ivtt_1/0.8 + b_c*cost_1 + b_acc_t_o * acc_t_o_1 + b_acc_t_d * acc_t_d_1
v[['2']]=asc_2 + b_tt_2 * ivtt_2/0.8 + b_r_2*(tp>18)
v[['30']]=asc_30 + b_tt_30 * ivtt_30/0.8
v[['4']]=asc_4 + b_tt_4* ivtt_4/0.8 + b_c*cost_4 + b_acc_t_o * acc_t_o_4
v[['31']]=asc_31 + b_tt_31 * ivtt_31/0.8 + b_c*cost_31 + b_acc_t_o * acc_t_o_31 + b_r_31*(tp>18)
v[['32']]=asc_32 + b_tt_30 * ivtt_32/0.8 + b_c*cost_32 + b_acc_t_o * acc_t_o_32 + b_r_30*(tp>18)

### Specify nests for NL model
nlNests = list(root=1, '300'=lambda_300)

### Specify tree structure for NL model
nlStructure= list()
nlStructure[["root"]] = c("300","0","1","4","2","30")
nlStructure[["300"]] = c("31","32")
# nlStructure[['auto']] =c("0","1","4","2","30")



### Define settings for NL model
nl_settings <- list(
alternatives = c("0"=0,"1"=1,"2"=2,"30"=30,"4"=4,"31"=31,"32"=32),
avail = list("0"=ava_0, "1"=ava_1,"2"=ava_2,"30"=ava_30,"4"=ava_4,"31"=ava_31,"32"=ava_32),

choiceVar = choice,
V=v,
nlNests = nlNests,
nlStructure = nlStructure
)


### Compute probabilities using NL model
P[["model"]] = apollo_nl(nl_settings, functionality)


### Take product across observation for same individual
#P = apollo_panelProd(P, apollo_inputs, functionality)

P=apollo_weighting(P,apollo_inputs,functionality)

### Prepare and return outputs of function
P = apollo_prepareProb(P, apollo_inputs, functionality)
return(P)
}
model = apollo_estimate(apollo_beta, apollo_fixed, apollo_probabilities, apollo_inputs)
apollo_modelOutput(model,list(printPVal = TRUE))



Then the result I got from v0.2.1 are:

Model name : Apollo_try_1
Model description : NL_two_level on mode choice
Model run at : 2022-07-24 19:41:49
Estimation method : bfgs
Model diagnosis : successful convergence
Number of individuals : 1202
Number of observations : 1202

Number of cores used : 1
Model without mixing

LL(start) : -2252.931
LL(0) : -135874.8
LL(final) : -75010.96
Rho-square (0) : 0.4479
Adj.Rho-square (0) : 0.4478
AIC : 150059.9
BIC : 150156.7


Estimated parameters : 19
Time taken (hh:mm:ss) : 00:00:33.45
pre-estimation : 00:00:0.42
estimation : 00:00:15.35
post-estimation : 00:00:17.67
Iterations : 51
Min abs eigenvalue of Hessian : 13.79981

Estimates:
Estimate s.e. t.rat.(0) p(1-sided) Rob.s.e. Rob.t.rat.(0) p(1-sided)
asc_4 0.32405 0.108356 2.991 0.001392 NA NA NA
asc_0 -1.41540 0.096069 -14.733 0.000000 NA NA NA
asc_30 -0.73030 0.044914 -16.260 0.000000 NA NA NA
asc_2 0.00000 NA NA NA NA NA NA
asc_1 4.50184 0.218546 20.599 0.000000 NA NA NA
asc_31 2.43985 0.051411 47.458 0.000000 NA NA NA
asc_32 0.86737 0.070991 12.218 0.000000 NA NA NA
b_tt_1 -0.02561 4.3579e-04 -58.775 0.000000 NA NA NA
b_tt_0 -0.01158 2.7496e-04 -42.123 0.000000 NA NA NA
b_tt_30 -0.02711 2.9747e-04 -91.123 0.000000 NA NA NA
b_tt_4 -0.01045 6.3879e-04 -16.358 0.000000 NA NA NA
b_tt_2 -0.01163 1.0923e-04 -106.450 0.000000 NA NA NA
b_tt_31 -0.03438 2.8219e-04 -121.841 0.000000 NA NA NA
b_c -0.04411 0.001712 -25.765 0.000000 NA NA NA
b_r_2 -0.73457 0.037518 -19.579 0.000000 NA NA NA
b_r_30 -0.68862 0.036589 -18.820 0.000000 NA NA NA
b_r_31 -0.47024 0.029363 -16.015 0.000000 NA NA NA
b_acc_t_o -0.01365 4.1957e-04 -32.543 0.000000 NA NA NA
b_acc_t_d -0.01152 3.9993e-04 -28.802 0.000000 NA NA NA
lambda_300 0.35345 0.011997 29.461 0.000000 NA NA NA

Nesting structure for NL model component :
Nest: root (1)
|-Nest: micro (0.3535)
| |-Alternative: 31
| '-Alternative: 32
|----Alternative: 0
|----Alternative: 1
|----Alternative: 4
|----Alternative: 2
'----Alternative: 30

the result using v0.2.7
Model name : Apollo_try_1
Model description : NL_two_level on mode choice
Model run at : 2022-07-24 19:46:50
Estimation method : bfgs
Model diagnosis : successful convergence
Number of individuals : 1202
Number of rows in database : 1202
Number of modelled outcomes : 1202

Number of cores used : 1
Model without mixing

LL(start) : -2252.93
LL(0) : -135874.8
LL(C) : Not applicable
LL(final) : -70139.01
Rho-square (0) : 0.4838
Adj.Rho-square (0) : 0.4837
AIC : 140316
BIC : 140412.8

Estimated parameters : 19
Time taken (hh:mm:ss) : 00:00:19.97
pre-estimation : 00:00:0.38
estimation : 00:00:13.56
post-estimation : 00:00:6.03
Iterations : 81

Unconstrained optimisation.

Estimates:
Estimate s.e. t.rat.(0) p(1-sided) Rob.s.e. Rob.t.rat.(0) p(1-sided)
asc_4 -2.185482 NA NA NA NA NA NA
asc_0 -2.375304 NA NA NA NA NA NA
asc_30 -1.854660 NA NA NA NA NA NA
asc_2 0.000000 NA NA NA NA NA NA
asc_1 3.062217 NA NA NA NA NA NA
asc_31 2.751639 NA NA NA NA NA NA
asc_32 -1.309109 NA NA NA NA NA NA
b_tt_1 -0.024928 NA NA NA NA NA NA
b_tt_0 -0.010182 NA NA NA NA NA NA
b_tt_30 -0.026641 NA NA NA NA NA NA
b_tt_4 -0.006377 NA NA NA NA NA NA
b_tt_2 -0.013657 NA NA NA NA NA NA
b_tt_31 -0.045183 NA NA NA NA NA NA
b_c -0.037324 NA NA NA NA NA NA
b_r_2 -1.352313 NA NA NA NA NA NA
b_r_30 -2.909122 NA NA NA NA NA NA
b_r_31 -0.887248 NA NA NA NA NA NA
b_acc_t_o -0.017844 NA NA NA NA NA NA
b_acc_t_d -0.013958 NA NA NA NA NA NA
lambda_300 1.178495 NA NA NA NA NA NA

Nesting structure for NL model component :
Nest: root (1)
|-Nest: micro (1.1785)
| |-Alternative: 31
| '-Alternative: 32
|----Alternative: 0
|----Alternative: 1
|----Alternative: 4
|----Alternative: 2
'----Alternative: 30

WARNING: The nesting parameter for nest "300" should be between 0 and 1 (the nesting parameter for nest "root"), yet its value is 1.1785.

Hi

neither output makes sense as the starting LL is better than the final LL. Can you confirm that the two alternatives nested together are available at the same time? Also, what happens in an MNL model in terms of fit?

Stephane

Hi, is it because I used sample weights so the starting LL is better than final LL? Is starting LL accounting weights?

do your weights average to 1?

and what happens with MNL?

what do you mean by "weights average to 1"?
Just want to make sure I'm including weights properly before I try the MNL.
Each observation has a weight. The value of weights ranges between 30 to 400. I specified the column of weights in apollo_control, and also have another line of P=apollo_weighting(P,apollo_inputs,functionality) in apollo_probabilities. Am I applying weights correctly?

Thank you!

with currently how I applied weights, running a simple MNL model with only subset of the variables with v0.2.1 and 0.2.7 are shown below.
The MNL and nested logit model are based on the same dataset, but the nested logit model break the alternative 3 into three sub alternatives:3, 30 and 31.

running version 0.2.1 gave:
Model name : Apollo_try_1
Model description : Simple MNL model on mode choice
Model run at : 2022-08-29 17:36:09
Estimation method : bfgs
Model diagnosis : successful convergence
Number of individuals : 1222
Number of observations : 1222

Number of cores used : 1
Model without mixing

LL(start) : -1779.875
LL(0) : -113609.3
LL(final) : -59454.34
Rho-square (0) : 0.4767
Adj.Rho-square (0) : 0.4766
AIC : 118932.7
BIC : 118994


Estimated parameters : 12
Time taken (hh:mm:ss) : 00:00:12.39
pre-estimation : 00:00:0.48
estimation : 00:00:5.68
post-estimation : 00:00:6.24
Iterations : 29
Min abs eigenvalue of Hessian : 14.26168

Estimates:
Estimate s.e. t.rat.(0) p(1-sided) Rob.s.e. Rob.t.rat.(0) p(1-sided)
asc_4 -3.30300 0.119217 -27.706 0.000 NaN NaN NaN
asc_0 -4.36605 0.119418 -36.561 0.000 NaN NaN NaN
asc_3&30&31 -0.80870 0.138610 -5.834 2.700e-09 NaN NaN NaN
asc_2 -3.10955 0.156354 -19.888 0.000 NaN NaN NaN
asc_1 0.00000 NA NA NA NA NA NA
b_tt_1 -0.27468 0.004899 -56.067 0.000 NaN NaN NaN
b_tt_0 -0.10238 0.002996 -34.177 0.000 NaN NaN NaN
b_tt_3&30&31 -0.34850 0.003397 -102.577 0.000 NaN NaN NaN
b_tt_4 -0.17065 0.009093 -18.766 0.000 NaN NaN NaN
b_tt_2 -0.12850 0.001374 -93.536 0.000 NaN NaN NaN
b_c -0.25201 0.014738 -17.099 0.000 NaN NaN NaN
b_acc_t_o -0.12103 0.003042 -39.781 0.000 NaN NaN NaN
b_acc_t_d -0.06632 0.003518 -18.850 0.000 NaN NaN NaN

running Version 0.2.7 gave me:
Model name : Apollo_try_1
Model description : Simple MNL model on mode choice
Model run at : 2022-08-29 17:22:47
Estimation method : bfgs
Model diagnosis : successful convergence
Number of individuals : 1222
Number of rows in database : 1222
Number of modelled outcomes : 1222

Number of cores used : 1
Model without mixing

LL(start) : -1779.88
LL(0) : -113609.3
LL(C) : Not applicable
LL(final) : -59454.13
Rho-square (0) : 0.4767
Adj.Rho-square (0) : 0.4766
AIC : 118932.3
BIC : 118993.6

Estimated parameters : 12
Time taken (hh:mm:ss) : 00:00:4.11
pre-estimation : 00:00:0.45
estimation : 00:00:2.29
post-estimation : 00:00:1.37
Iterations : 35

Unconstrained optimisation.

Estimates:
Estimate s.e. t.rat.(0) p(1-sided) Rob.s.e. Rob.t.rat.(0) p(1-sided)
asc_4 -3.30304 NA NA NA NA NA NA
asc_0 -4.36604 NA NA NA NA NA NA
asc_3&30&31 -0.80870 NA NA NA NA NA NA
asc_2 -3.10954 NA NA NA NA NA NA
asc_1 0.00000 NA NA NA NA NA NA
b_tt_1 -0.27468 NA NA NA NA NA NA
b_tt_0 -0.10238 NA NA NA NA NA NA
b_tt_3&30&31 -0.34850 NA NA NA NA NA NA
b_tt_4 -0.17065 NA NA NA NA NA NA
b_tt_2 -0.12850 NA NA NA NA NA NA
b_c -0.25201 NA NA NA NA NA NA
b_acc_t_o -0.12103 NA NA NA NA NA NA
b_acc_t_d -0.06632 NA NA NA NA NA NA

what I am hoping to do:
I asked a question about applying weights in prediction (http://www.apollochoicemodelling.com/fo ... f=16&t=386). 0.2.7 was recommended to solve the problem I asked, but v0.2.7 gave me estimation result with NA s.e., t statistics and p values ( as shown above).

Thank you for your help!

Hi, in relation to the below, there is a bug in that the starting LL reported by Apollo does not incorporate weights, so you can ignore that for now. What I mean with whether the weights average to 1 is that across your observations, the sum of weights should ideally be the same as the number of observations.

wulalala wrote: ↑29 Aug 2022, 23:48 what do you mean by "weights average to 1"?
Just want to make sure I'm including weights properly before I try the MNL.
Each observation has a weight. The value of weights ranges between 30 to 400. I specified the column of weights in apollo_control, and also have another line of P=apollo_weighting(P,apollo_inputs,functionality) in apollo_probabilities. Am I applying weights correctly?

Thank you!

Hi

if you look at the MNL results, you can clearly see that in both 0.2.1 and 0.2.7 your model has an identification issue (no robust standard errors). So that's an issue you need to solve before you start looking at nested logit

Stephane