Important: Read this before posting to this forum

  1. This forum is for questions related to the use of Apollo. We will answer some general choice modelling questions too, where appropriate, and time permitting. We cannot answer questions about how to estimate choice models with other software packages.
  2. There is a very detailed manual for Apollo available at http://www.ApolloChoiceModelling.com/manual.html. This contains detailed descriptions of the various Apollo functions, and numerous examples are available at http://www.ApolloChoiceModelling.com/examples.html. In addition, help files are available for all functions, using e.g. ?apollo_mnl
  3. Before asking a question on the forum, users are kindly requested to follow these steps:
    1. Check that the same issue has not already been addressed in the forum - there is a search tool.
    2. Ensure that the correct syntax has been used. For any function, detailed instructions are available directly in Apollo, e.g. by using ?apollo_mnl for apollo_mnl
    3. Check the frequently asked questions section on the Apollo website, which discusses some common issues/failures. Please see http://www.apollochoicemodelling.com/faq.html
    4. Make sure that R is using the latest official release of Apollo.
  4. If the above steps do not resolve the issue, then users should follow these steps when posting a question:
    1. provide full details on the issue, including the entire code and output, including any error messages
    2. posts will not immediately appear on the forum, but will be checked by a moderator first. This may take a day or two at busy times. There is no need to submit the post multiple times.

Choosing the best model

Ask general questions about model specification and estimation that are not Apollo specific but relevant to Apollo users.
Post Reply
DavidKL
Posts: 15
Joined: 27 Mar 2021, 16:30

Choosing the best model

Post by DavidKL »

Hi David and Stephane!

I would like to ask how to choose the best fit model.

1. Which one is the most reliable information criterion?

I would like to compare two MNL model. The second one is include several sociodemographic interactions, so have more parameters. I get lower log-likelihood, AIC and higher Adj. Pszeudo R^2 for the sociodemographic model. The result of likelihood-ratio test show significant improvement. However, I get higher BIC value.
I know that, BIC takes into account the number of observations.
Is this an important aspect?

It is worth to deciding and choosing between models by take into account the value of all the information criteria?

2. Can the above-mentioned information criteria only be applied when comparing with non-nested logit-type models? Can they be used to compare other models (e.g.: MNL-RPL, MNL-LC, RPL-LC) without exception?

Thank you!

David
stephanehess
Site Admin
Posts: 974
Joined: 24 Apr 2020, 16:29

Re: Choosing the best model

Post by stephanehess »

David

model fitting is in many ways an art, and it's not a good idea to stick rigorously to model fit statistics alone. So for example, a model that fits better might not be performing so well in terms of realism of the results, such as WTP.

My general approach is that I build up model complexity gradually, and at step compare model fit as well as other criteria, such as WTP and elasticities.

With large datasets, a likelihood ratio test will often reject the base model, but you need to ask yourself whether the improvements are really substantial enough. An improvement in LL by a few units when the base LL is -10K is not really changing your prediction performance. But conversely, you need to think that sometimes, additional parameters have a lot of behavioural meaning even though they don't change model fit very much.

On your other question, you can also use BIC etc to compare nested models.

Stephane
--------------------------------
Stephane Hess
www.stephanehess.me.uk
DavidKL
Posts: 15
Joined: 27 Mar 2021, 16:30

Re: Choosing the best model

Post by DavidKL »

Thank you Stephane!

It helped a lot! This is a very interesting topic for me.
Post Reply