ApolloChoiceModelling forum

Hi David and Stephane!

I would like to ask how to choose the best fit model.

1. Which one is the most reliable information criterion?

I would like to compare two MNL model. The second one is include several sociodemographic interactions, so have more parameters. I get lower log-likelihood, AIC and higher Adj. Pszeudo R^2 for the sociodemographic model. The result of likelihood-ratio test show significant improvement. However, I get higher BIC value.
I know that, BIC takes into account the number of observations.
Is this an important aspect?

It is worth to deciding and choosing between models by take into account the value of all the information criteria?

2. Can the above-mentioned information criteria only be applied when comparing with non-nested logit-type models? Can they be used to compare other models (e.g.: MNL-RPL, MNL-LC, RPL-LC) without exception?

Thank you!

David

David

model fitting is in many ways an art, and it's not a good idea to stick rigorously to model fit statistics alone. So for example, a model that fits better might not be performing so well in terms of realism of the results, such as WTP.

My general approach is that I build up model complexity gradually, and at step compare model fit as well as other criteria, such as WTP and elasticities.

With large datasets, a likelihood ratio test will often reject the base model, but you need to ask yourself whether the improvements are really substantial enough. An improvement in LL by a few units when the base LL is -10K is not really changing your prediction performance. But conversely, you need to think that sometimes, additional parameters have a lot of behavioural meaning even though they don't change model fit very much.

On your other question, you can also use BIC etc to compare nested models.

Stephane

Thank you Stephane!

It helped a lot! This is a very interesting topic for me.