Dear Stephane,
I´ve conducted the Out of Sample test function in Apollo resulting in the following outputs:
LL per obs in estimation sample LL per obs in validation sample % difference
1 -1.0348 -1.0144 1.97
2 -1.0338 -1.0256 0.79
3 -1.0236 -1.1150 -8.94
4 -1.0296 -1.0592 -2.88
5 -1.0289 -1.0652 -3.53
6 -1.0386 -0.9809 5.56
7 -1.0367 -0.9970 3.83
8 -1.0339 -1.0222 1.13
9 -1.0378 -0.9923 4.38
10 -1.0312 -1.0448 -1.32
Average -1.0329 -1.0317 0.10
Unfortunately, I did not find any specific quantifiable interpretation guideline for the out of sample outputs in the manual.
Can you tell how the results can be interpreted? And is it okay to stay with the default of 10% validation sample for a total sample size of N=330?
Thanks for your support in advance.
Best,
J.
Important: Read this before posting to this forum
- This forum is for questions related to the use of Apollo. We will answer some general choice modelling questions too, where appropriate, and time permitting. We cannot answer questions about how to estimate choice models with other software packages.
- There is a very detailed manual for Apollo available at http://www.ApolloChoiceModelling.com/manual.html. This contains detailed descriptions of the various Apollo functions, and numerous examples are available at http://www.ApolloChoiceModelling.com/examples.html. In addition, help files are available for all functions, using e.g. ?apollo_mnl
- Before asking a question on the forum, users are kindly requested to follow these steps:
- Check that the same issue has not already been addressed in the forum - there is a search tool.
- Ensure that the correct syntax has been used. For any function, detailed instructions are available directly in Apollo, e.g. by using ?apollo_mnl for apollo_mnl
- Check the frequently asked questions section on the Apollo website, which discusses some common issues/failures. Please see http://www.apollochoicemodelling.com/faq.html
- Make sure that R is using the latest official release of Apollo.
- Users can check which version they are running by entering packageVersion("apollo").
- Then check what is the latest full release (not development version) at http://www.ApolloChoiceModelling.com/code.html.
- To update to the latest official version, just enter install.packages("apollo"). To update to a development version, download the appropriate binary file from http://www.ApolloChoiceModelling.com/code.html, and install the package from file
- If the above steps do not resolve the issue, then users should follow these steps when posting a question:
- provide full details on the issue, including the entire code and output, including any error messages
- posts will not immediately appear on the forum, but will be checked by a moderator first. This may take a day or two at busy times. There is no need to submit the post multiple times.
Out of sample results
-
stephanehess
- Site Admin
- Posts: 1040
- Joined: 24 Apr 2020, 16:29
Re: Out of sample results
Hi
there is no specific hard rule in terms of what size of difference is acceptable, but your differences look pretty small, suggesting no specific risk of overfitting
Stephane
there is no specific hard rule in terms of what size of difference is acceptable, but your differences look pretty small, suggesting no specific risk of overfitting
Stephane
Re: Out of sample results
Dear Prof. Stephane,
I hope this email finds you well.
I have some follow-up questions regarding the same topic. How should we interpret the average difference in log-likelihood (LL) between the estimated sample and validation sample? Does a smaller value for percentage difference imply that the estimated model fits the validation sample better compared to situations with a higher difference?
In my case, I'm obtaining an average difference value of -2.7 % for MNL and -4.7% fro MMNL model. Is this acceptable?
Also, Is there any way to calculate the "% correct predicted" metric for validation in Apollo?
Thank you for your assistance.
Best regards,
K.
I hope this email finds you well.
I have some follow-up questions regarding the same topic. How should we interpret the average difference in log-likelihood (LL) between the estimated sample and validation sample? Does a smaller value for percentage difference imply that the estimated model fits the validation sample better compared to situations with a higher difference?
In my case, I'm obtaining an average difference value of -2.7 % for MNL and -4.7% fro MMNL model. Is this acceptable?
Also, Is there any way to calculate the "% correct predicted" metric for validation in Apollo?
Thank you for your assistance.
Best regards,
K.
Last edited by kkavta on 09 Feb 2024, 18:14, edited 2 times in total.
-
stephanehess
- Site Admin
- Posts: 1040
- Joined: 24 Apr 2020, 16:29
Re: Out of sample results
Hi
apologies for the slow reply
So what you're finding is that you MMNL overfits the estimation data a bit more than MNL, but these differences are small.
No, Apollo does not compute the % correct predicted metric as this is a misleading metric that is contrary to probabilistic choice models. It should never be used. See Kenneth Train's book
Stephane
apologies for the slow reply
So what you're finding is that you MMNL overfits the estimation data a bit more than MNL, but these differences are small.
No, Apollo does not compute the % correct predicted metric as this is a misleading metric that is contrary to probabilistic choice models. It should never be used. See Kenneth Train's book
Stephane