Important: Read this before posting to this forum

  1. This forum is for questions related to the use of Apollo. We will answer some general choice modelling questions too, where appropriate, and time permitting. We cannot answer questions about how to estimate choice models with other software packages.
  2. There is a very detailed manual for Apollo available at http://www.ApolloChoiceModelling.com/manual.html. This contains detailed descriptions of the various Apollo functions, and numerous examples are available at http://www.ApolloChoiceModelling.com/examples.html. In addition, help files are available for all functions, using e.g. ?apollo_mnl
  3. Before asking a question on the forum, users are kindly requested to follow these steps:
    1. Check that the same issue has not already been addressed in the forum - there is a search tool.
    2. Ensure that the correct syntax has been used. For any function, detailed instructions are available directly in Apollo, e.g. by using ?apollo_mnl for apollo_mnl
    3. Check the frequently asked questions section on the Apollo website, which discusses some common issues/failures. Please see http://www.apollochoicemodelling.com/faq.html
    4. Make sure that R is using the latest official release of Apollo.
  4. If the above steps do not resolve the issue, then users should follow these steps when posting a question:
    1. provide full details on the issue, including the entire code and output, including any error messages
    2. posts will not immediately appear on the forum, but will be checked by a moderator first. This may take a day or two at busy times. There is no need to submit the post multiple times.

Worse LL for general model in MIXL

Ask questions about post-estimation functions (e.g. prediction, conditionals, etc) or other processing of results.
Post Reply
anhntl
Posts: 13
Joined: 02 Feb 2021, 09:40

Worse LL for general model in MIXL

Post by anhntl »

Dear Professor Stephane Hess,


I am testing the difference of preference of an attribute between two groups of respondents by LL ratio test for the pooled sample:

My restricted model (attributes homogenous between 2 groups):
V = beta_x * x + ...
My general model (testing if there is a difference of preference for an attribute between two groups):
V= (beta_xvisitor * visitor +beta_xhousehold *household) * x +....

- I use 500 MLHS draws
- I use different draws for visitor and household, starting value of general model from restricted model.
- I want to control for differences in both mean sensitivity and sigmas.

But then I encountered the worse LL for the general model. I read your comment on a post in forum with same questions as mine: viewtopic.php?f=15&t=57

The reason of worsening LL is that I use different draws for two parameters. But this case is a little different than mine, and I dont know what is the reason behind:
- In his case: the two treatments are not specific to individual people but to individual choices, and the estimation is losing the correlation for coefficient between choice tasks by applying separate draws. And you suggest estimating a correlated random parameters
- In my case: two parameters are specific to different respondents. And it might not be necessary to control for the correlation between visitors and household parameters?

- When I estimate model with the correlated parameter of tourists and visitors:
randcoeff[["b.width.h"]] = mu.b.width.h + sigma.b.width.h * draws.width.h
randcoeff[["b.width.v"]] = mu.b.width.v + sigma.b.width.h.v * draws.width.h + sigma.b.width.v * draws.width.v
the LL is still worse than the restricted model

- When I estimate with same draws,
randcoeff[["b.width.h"]] = mu.b.width.h + sigma.b.width.h * draws.width.h
randcoeff[["b.width.v"]] = mu.b.width.v + sigma.b.width.v * draws.width.h
the LL difference is very small (0.22 over LL -3162).


LL reduced model: -3162.41
LL general model with same draws: -3162.19
LL general model with correlated parameter: -3166.217


I am attaching my code here for your consideration. Thank you very much.
Best regards,
Anh

My restricted model
################ MIXL POOL #################

apollo_initialise()
apollo_control = list(
modelName ="MIXL_Pool500",
modelDescr ="Pool Residents-Tourists",
indivID ="id",
mixing=TRUE,panelData=TRUE,
nCores=3
)
database<- read.csv("D:\\Research\\Theses\\Second Chapter\\Working\\Data\\Final\\apollo_pool1.csv", header=T)
### Vector of parameters, including any that are kept fixed in estimation
apollo_beta=c( mu.b.width=1, sigma.b.width=0.1,# sigma.hsk.h=0.1,
sigma.hsk=0.1,
mu.b.asc = 1, mu.b.access=1,mu.b.tax=-1,
mu.b.facR=1, mu.b.facRT=1, mu.b.facT=1,
mu.b.strucG=1, mu.b.strucSB=1, mu.b.strucRC=1, mu.b.strucRStairs=1,
sigma.b.asc = 0.1, sigma.b.access=0.1,sigma.b.tax=0.1,
sigma.b.facR=0.1, sigma.b.facRT=0.1, sigma.b.facT=0.1,
sigma.b.strucG=0.1, sigma.b.strucSB=0.1, sigma.b.strucRC=0.1, sigma.b.strucRStairs=0.1)

### Vector with names (in quotes) of parameters to be kept fixed at their starting value in apollo_beta, use apollo_beta_fixed = c() if none
apollo_fixed = c()

### Read in starting values for at least some parameters from existing model output file
apollo_beta=apollo_readBeta(apollo_beta,apollo_fixed,"MNL_Pool",overwriteFixed = FALSE)

# ################################################################# #
#### DEFINE RANDOM COMPONENTS ####
# ################################################################# #

### Set parameters for generating draws
apollo_draws = list(
interDrawsType = "mlhs",
interNDraws = 500,
interUnifDraws = c(),
interNormDraws = c("draws.width",
"draws.asc","draws.access","draws.facR",
"draws.facRT","draws.facT", "draws.strucSB","draws.strucG",
"draws.strucRC","draws.strucRStairs","draws.tax", #"draws.hsk.h",
"draws.hsk1","draws.hsk2","draws.hsk3"),
intraDrawsType = "halton",
intraNDraws = 0,
intraUnifDraws = c(),
intraNormDraws = c()
)

### Create random parameters
apollo_randCoeff = function(apollo_beta, apollo_inputs){
randcoeff = list()
randcoeff[["b.width"]] = mu.b.width + sigma.b.width * draws.width
#randcoeff[["ec.h"]] = sigma.hsk.h * draws.hsk.h
randcoeff[["ec1"]] = sigma.hsk * draws.hsk1
randcoeff[["ec2"]] = sigma.hsk * draws.hsk2
randcoeff[["ec3"]] = sigma.hsk * draws.hsk3

randcoeff[["b.asc"]] = mu.b.asc + sigma.b.asc * draws.asc
randcoeff[["b.access"]] = mu.b.access + sigma.b.access * draws.access
randcoeff[["b.tax"]] = -exp( mu.b.tax + sigma.b.tax * draws.tax)
randcoeff[["b.facR"]] = mu.b.facR + sigma.b.facR * draws.facR
randcoeff[["b.facRT"]] = mu.b.facRT + sigma.b.facRT * draws.facRT
randcoeff[["b.facT"]] = mu.b.facT + sigma.b.facT * draws.facT
randcoeff[["b.strucSB"]] = mu.b.strucSB + sigma.b.strucSB * draws.strucSB
randcoeff[["b.strucRC"]] = mu.b.strucRC + sigma.b.strucRC * draws.strucRC
randcoeff[["b.strucG"]] = mu.b.strucG + sigma.b.strucG * draws.strucG
randcoeff[["b.strucRStairs"]] = mu.b.strucRStairs + sigma.b.strucRStairs * draws.strucRStairs

return(randcoeff)
}

# ################################################################# #
#### GROUP AND VALIDATE INPUTS ####
# ################################################################# #

apollo_inputs = apollo_validateInputs()


# ################################################################# #
#### DEFINE MODEL AND LIKELIHOOD FUNCTION ####
# ################################################################# #

apollo_probabilities=function(apollo_beta, apollo_inputs, functionality="estimate"){

### Function initialisation: do not change the following three commands
### Attach inputs and detach after function exit
apollo_attach(apollo_beta, apollo_inputs)
on.exit(apollo_detach(apollo_beta, apollo_inputs))

### Create list of probabilities P
P = list()

### List of utilities: these must use the same names as in mnl_settings, order is irrelevant
V = list()
V[['alt.1']] = b.asc+ b.access* access.1+b.facR* facR.1+b.width*width.1+
b.facRT* facRT.1+b.facT* facT.1+b.strucRC* strucRC.1+
b.strucSB* strucSB.1+#ec.h*hh+
ec1*hh

V[['alt.2']] = b.access* access.2+b.facR* facR.2+b.width*width.2+
b.facRT* facRT.2+b.facT* facT.2+b.strucG* strucG.2+b.strucRC* strucRC.2+
b.strucRStairs* strucRStairs.2+b.strucSB* strucSB.2+b.tax* tax.2+ec2*hh

V[['alt.3']] = b.width*width.3+
b.access* access.3+b.facR* facR.3+
b.facRT* facRT.3+b.facT* facT.3+b.strucG* strucG.3+b.strucRC* strucRC.3+
b.strucRStairs* strucRStairs.3+b.strucSB* strucSB.3+b.tax* tax.3+ec3*hh

### Define settings for MNL model component
mnl_settings = list(
alternatives = c(alt.1=1, alt.2=2,alt.3=3),
avail = list(alt.1=1, alt.2=1,alt.3=1),
choiceVar = choice,
V = V
)

### Compute probabilities using MNL model
P[['model']] = apollo_mnl(mnl_settings, functionality)

### Take product across observation for same individual
P = apollo_panelProd(P, apollo_inputs, functionality)

### Average across inter-individual draws
P = apollo_avgInterDraws(P, apollo_inputs, functionality)

### Prepare and return outputs of function
P = apollo_prepareProb(P, apollo_inputs, functionality)
return(P)
}

# ################################################################# #
#### MODEL ESTIMATION ####
# ################################################################# #

model = apollo_estimate(apollo_beta, apollo_fixed,
apollo_probabilities, apollo_inputs)#, estimate_settings=list(hessianRoutine="maxLik"))

apollo_modelOutput(model)
apollo_saveOutput(model)
# ################################################################# #


My general model

################ Width Test #################

apollo_initialise()
apollo_control = list(
modelName ="MIXL_Width_Test",
modelDescr ="Pool Residents-Tourists",
indivID ="id",
mixing=TRUE,panelData=TRUE,
nCores=3
)
database<- read.csv("D:\\Research\\Theses\\Second Chapter\\Working\\Data\\Final\\apollo_pool1.csv", header=T)
### Vector of parameters, including any that are kept fixed in estimation
apollo_beta=c( mu.b.width.h=0.006442, sigma.b.width.h=0.004580, sigma.hsk=0.1,
mu.b.width.v=0.006442, sigma.b.width.v=0.004580,
mu.b.asc = 1, mu.b.access=1,mu.b.tax=-1,
mu.b.facR=1, mu.b.facRT=1, mu.b.facT=1,
mu.b.strucG=1, mu.b.strucSB=1, mu.b.strucRC=1, mu.b.strucRStairs=1,
sigma.b.asc = 0.1, sigma.b.access=0.1,sigma.b.tax=0.1,
sigma.b.facR=0.1, sigma.b.facRT=0.1, sigma.b.facT=0.1,
sigma.b.strucG=0.1, sigma.b.strucSB=0.1, sigma.b.strucRC=0.1, sigma.b.strucRStairs=0.1)

### Vector with names (in quotes) of parameters to be kept fixed at their starting value in apollo_beta, use apollo_beta_fixed = c() if none
apollo_fixed = c()

### Read in starting values for at least some parameters from existing model output file
apollo_beta=apollo_readBeta(apollo_beta,apollo_fixed,"MIXL_Pool500",overwriteFixed = FALSE)

# ################################################################# #
#### DEFINE RANDOM COMPONENTS ####
# ################################################################# #

### Set parameters for generating draws
apollo_draws = list(
interDrawsType = "mlhs",
interNDraws = 500,
interUnifDraws = c(),
interNormDraws = c("draws.width.h","draws.width.v",
"draws.asc","draws.access","draws.facR",
"draws.facRT","draws.facT", "draws.strucSB","draws.strucG",
"draws.strucRC","draws.strucRStairs","draws.tax", #"draws.hsk.h",
"draws.hsk1","draws.hsk2","draws.hsk3"),
intraDrawsType = "halton",
intraNDraws = 0,
intraUnifDraws = c(),
intraNormDraws = c()
)

### Create random parameters
apollo_randCoeff = function(apollo_beta, apollo_inputs){
randcoeff = list()
randcoeff[["b.width.h"]] = mu.b.width.h + sigma.b.width.h * draws.width.h
randcoeff[["b.width.v"]] = mu.b.width.v + sigma.b.width.v * draws.width.v
randcoeff[["ec1"]] = sigma.hsk * draws.hsk1
randcoeff[["ec2"]] = sigma.hsk * draws.hsk2
randcoeff[["ec3"]] = sigma.hsk * draws.hsk3

randcoeff[["b.asc"]] = mu.b.asc + sigma.b.asc * draws.asc
randcoeff[["b.access"]] = mu.b.access + sigma.b.access * draws.access
randcoeff[["b.tax"]] = -exp( mu.b.tax + sigma.b.tax * draws.tax)
randcoeff[["b.facR"]] = mu.b.facR + sigma.b.facR * draws.facR
randcoeff[["b.facRT"]] = mu.b.facRT + sigma.b.facRT * draws.facRT
randcoeff[["b.facT"]] = mu.b.facT + sigma.b.facT * draws.facT
randcoeff[["b.strucSB"]] = mu.b.strucSB + sigma.b.strucSB * draws.strucSB
randcoeff[["b.strucRC"]] = mu.b.strucRC + sigma.b.strucRC * draws.strucRC
randcoeff[["b.strucG"]] = mu.b.strucG + sigma.b.strucG * draws.strucG
randcoeff[["b.strucRStairs"]] = mu.b.strucRStairs + sigma.b.strucRStairs * draws.strucRStairs

return(randcoeff)
}

# ################################################################# #
#### GROUP AND VALIDATE INPUTS ####
# ################################################################# #

apollo_inputs = apollo_validateInputs()


# ################################################################# #
#### DEFINE MODEL AND LIKELIHOOD FUNCTION ####
# ################################################################# #

apollo_probabilities=function(apollo_beta, apollo_inputs, functionality="estimate"){

### Function initialisation: do not change the following three commands
### Attach inputs and detach after function exit
apollo_attach(apollo_beta, apollo_inputs)
on.exit(apollo_detach(apollo_beta, apollo_inputs))

### Create list of probabilities P
P = list()

### List of utilities: these must use the same names as in mnl_settings, order is irrelevant
V = list()
V[['alt.1']] = b.asc+ b.access* access.1+b.facR* facR.1+
b.facRT* facRT.1+b.facT* facT.1+b.strucRC* strucRC.1+
b.strucSB* strucSB.1+b.width.h*width.1*hh+b.width.v*width.1*vit+
ec1*hh

V[['alt.2']] = b.access* access.2+b.facR* facR.2+b.width.h*width.2*hh+b.width.v*width.2*vit+
b.facRT* facRT.2+b.facT* facT.2+b.strucG* strucG.2+b.strucRC* strucRC.2+
b.strucRStairs* strucRStairs.2+b.strucSB* strucSB.2+b.tax* tax.2+ec2*hh

V[['alt.3']] = b.width.h*width.3*hh+b.width.v*width.3*vit+
b.access* access.3+b.facR* facR.3+
b.facRT* facRT.3+b.facT* facT.3+b.strucG* strucG.3+b.strucRC* strucRC.3+
b.strucRStairs* strucRStairs.3+b.strucSB* strucSB.3+b.tax* tax.3+ec3*hh

### Define settings for MNL model component
mnl_settings = list(
alternatives = c(alt.1=1, alt.2=2,alt.3=3),
avail = list(alt.1=1, alt.2=1,alt.3=1),
choiceVar = choice,
V = V
)

### Compute probabilities using MNL model
P[['model']] = apollo_mnl(mnl_settings, functionality)

### Take product across observation for same individual
P = apollo_panelProd(P, apollo_inputs, functionality)

### Average across inter-individual draws
P = apollo_avgInterDraws(P, apollo_inputs, functionality)

### Prepare and return outputs of function
P = apollo_prepareProb(P, apollo_inputs, functionality)
return(P)
}

# ################################################################# #
#### MODEL ESTIMATION ####
# ################################################################# #

model = apollo_estimate(apollo_beta, apollo_fixed,
apollo_probabilities, apollo_inputs)#, estimate_settings=list(hessianRoutine="maxLik"))

apollo_modelOutput(model)
apollo_saveOutput(model)
apollo_lrTest("MIXL_Pool500","MIXL_Width_Test")
stephanehess
Site Admin
Posts: 974
Joined: 24 Apr 2020, 16:29

Re: Worse LL for general model in MIXL

Post by stephanehess »

Hi,

can you please also show us the estimates

Thanks

Stephane
--------------------------------
Stephane Hess
www.stephanehess.me.uk
anhntl
Posts: 13
Joined: 02 Feb 2021, 09:40

Re: Worse LL for general model in MIXL

Post by anhntl »

Dear Professor Stephane,
Thank you for your reply. Below image is my estimates:

- Restricted Model:
LL(start) : -7016.876
LL(0) : -3948.413
LL(final) : -3162.413
Estimated parameters : 23
Iterations : 54
Min abs eigenvalue of Hessian : 10.22751
Estimation method : bfgs
Model diagnosis : successful convergence
Number of individuals : 599
Number of rows in database : 3594
Number of modelled outcomes : 3594
Overview of choices for MNL model component :
alt.1 alt.2 alt.3
Times available 3594.00 3594.00 3594.0
Times chosen 1329.00 1201.00 1064.0
Percentage chosen overall 36.98 33.42 29.6
Percentage chosen when available 36.98 33.42 29.6

- General Model correlated parameters:

LL(start) : -3176.349
LL(0) : -3948.413
LL(final) : -3166.217
Estimated parameters : 26
Iterations : 35
Min abs eigenvalue of Hessian : 9.295534


- General model with same draws:
LL(start) : -3162.413
LL(0) : -3948.413
LL(final) : -3162.188
Estimated parameters : 25
Iterations : 27
Min abs eigenvalue of Hessian : 10.12413

cor.png
cor.png (129.08 KiB) Viewed 5731 times
stephanehess
Site Admin
Posts: 974
Joined: 24 Apr 2020, 16:29

Re: Worse LL for general model in MIXL

Post by stephanehess »

Hi

your results make sense. The "general model with same draws" has a slightly better LL than the "restricted model" but the difference is of course not significant at all. But this is not surprising. The parameters mu.b.width.h and mu.b.width.v are not significantly different from each other, while the same applies for the two sigmas, which have very low significance anyway. So the fit wouldn't be better

Stephane
--------------------------------
Stephane Hess
www.stephanehess.me.uk
Post Reply