Important: Read this before posting to this forum

  1. This forum is for questions related to the use of Apollo. We will answer some general choice modelling questions too, where appropriate, and time permitting. We cannot answer questions about how to estimate choice models with other software packages.
  2. There is a very detailed manual for Apollo available at http://www.ApolloChoiceModelling.com/manual.html. This contains detailed descriptions of the various Apollo functions, and numerous examples are available at http://www.ApolloChoiceModelling.com/examples.html. In addition, help files are available for all functions, using e.g. ?apollo_mnl
  3. Before asking a question on the forum, users are kindly requested to follow these steps:
    1. Check that the same issue has not already been addressed in the forum - there is a search tool.
    2. Ensure that the correct syntax has been used. For any function, detailed instructions are available directly in Apollo, e.g. by using ?apollo_mnl for apollo_mnl
    3. Check the frequently asked questions section on the Apollo website, which discusses some common issues/failures. Please see http://www.apollochoicemodelling.com/faq.html
    4. Make sure that R is using the latest official release of Apollo.
  4. If the above steps do not resolve the issue, then users should follow these steps when posting a question:
    1. provide full details on the issue, including the entire code and output, including any error messages
    2. posts will not immediately appear on the forum, but will be checked by a moderator first. This may take a day or two at busy times. There is no need to submit the post multiple times.

Predictions for post estimation

Ask questions about post-estimation functions (e.g. prediction, conditionals, etc) or other processing of results.
Post Reply
JuliavB
Posts: 41
Joined: 18 Aug 2021, 13:36

Predictions for post estimation

Post by JuliavB »

Dear all,

for my post-estimation analysis I´ve used the following code:

predictions_base = apollo_prediction(model, apollo_probabilities, apollo_inputs,
prediction_settings=list(runs=30))

forecast = apollo_prediction(model, apollo_probabilities, apollo_inputs)

On the basis of the resulting list I´ve analysed in how many cases the option with the highest probability was chosen.

prediction_overview <- predictions_base[[1]]
table(apply(prediction_overview[,3:5],1,max) == prediction_overview$chosen)
table(apply(prediction_overview[,3:5],1,max) == prediction_overview$chosen) /
sum(table(apply(prediction_overview[,3:5],1,max) == prediction_overview$chosen))

As I have 3 options to choose (status quo + 2 alternatives) in my experiment a random draw would have 33,3% probability.
From the above analysis I´ve found out that in 47% of observations the option with the highest probability is chosen. So the model improves the forecast by ~ 14%.

1. Are my above thoughts and conclusions right to answer the question how many choices can be predicted correctly by the model?
2. Or do I have to take a random sample from my total sample to do a out ouf sample test for post estimation ?

Any advice is highly appreciated.
Thanks and best,
J.
stephanehess
Site Admin
Posts: 974
Joined: 24 Apr 2020, 16:29

Re: Predictions for post estimation

Post by stephanehess »

Hi

sorry for the slow reply.

I would never look at whether the chosen option has the highest probability of being chosen. That metric is not in line with probabilistic choice models (see e.g. the discussion in Train).

Instead, you should look at the average probability of correct prediction, which Apollo gives you in prediction too, and then compare that to 1/3

Or of course just look at rho2

Stephane
--------------------------------
Stephane Hess
www.stephanehess.me.uk
JuliavB
Posts: 41
Joined: 18 Aug 2021, 13:36

Re: Predictions for post estimation

Post by JuliavB »

Hi Stephane,

thank you for your response.

My results in predictions for one of my models are as follows:

Aggregated prediction
at MLE Sampled mean Sampled std.dev. Quantile 0.025 Quantile 0.975
SQ 495.4 496.8 38.21 434.2 565.8
RefA 930.3 931.5 23.76 885.5 965.6
RefB 856.3 853.8 18.08 823.7 884.1

Average prediction
at MLE Sampled mean Sampled std.dev. Quantile 0.025 Quantile 0.975
SQ 0.2171 0.2177 0.016743 0.1903 0.2479
RefA 0.4077 0.4082 0.010412 0.3880 0.4231
RefB 0.3752 0.3741 0.007924 0.3609 0.3874

The output from apollo_prediction is a list with two elements: a data.frame containing the predictions at
the estimated values, and an array with predictions for different values of the parameters drawn from
their asymptotic distribution.
>
> forecast = apollo_prediction(model, apollo_probabilities, apollo_inputs)
Your model contains continuous random parameters. apollo_prediction will perform averaging across draws for
these. For predictions at the level of individual draws, please call the apollo_probabilities function
using model$estimate as the parameters, and with functionality="raw".
Running predictions from model using parameter estimates...
Prediction at model estimates
SQ RefA RefB
Aggregate 495.40 930.34 856.26
Average 0.22 0.41 0.38

The output from apollo_prediction is a matrix containing the predictions at the estimated values.
__________
Rho-square (0) : 0.2023
__________

But where can I see the average probability of correct prediction?
Can you give any help on the interpretation of these results?

Thank you very much in advance!
JuliavB
Posts: 41
Joined: 18 Aug 2021, 13:36

Re: Predictions for post estimation

Post by JuliavB »

Oh, and the parameter estimates are all highly significant.
stephanehess
Site Admin
Posts: 974
Joined: 24 Apr 2020, 16:29

Re: Predictions for post estimation

Post by stephanehess »

try

Code: Select all

mean(prediction_overview$chosen)
--------------------------------
Stephane Hess
www.stephanehess.me.uk
JuliavB
Posts: 41
Joined: 18 Aug 2021, 13:36

Re: Predictions for post estimation

Post by JuliavB »

I´ve estimated it with the following code and results:

predictions_base = apollo_prediction(model, apollo_probabilities, apollo_inputs,
+ prediction_settings=list(runs=30))
Your model contains continuous random parameters. apollo_prediction will perform averaging across draws for
these. For predictions at the level of individual draws, please call the apollo_probabilities function
using model$estimate as the parameters, and with functionality="raw".
Running predictions from model using parameter estimates...
Running predictions across draws from the asymptotic distribution for maximum likelihood estimates.
Predicting for set of draws 1/30...
Predicting for set of draws 2/30...
Predicting for set of draws 3/30...
Predicting for set of draws 4/30...
Predicting for set of draws 5/30...
Predicting for set of draws 6/30...
Predicting for set of draws 7/30...
Predicting for set of draws 8/30...
Predicting for set of draws 9/30...
Predicting for set of draws 10/30...
Predicting for set of draws 11/30...
Predicting for set of draws 12/30...
Predicting for set of draws 13/30...
Predicting for set of draws 14/30...
Predicting for set of draws 15/30...
Predicting for set of draws 16/30...
Predicting for set of draws 17/30...
Predicting for set of draws 18/30...
Predicting for set of draws 19/30...
Predicting for set of draws 20/30...
Predicting for set of draws 21/30...
Predicting for set of draws 22/30...
Predicting for set of draws 23/30...
Predicting for set of draws 24/30...
Predicting for set of draws 25/30...
Predicting for set of draws 26/30...
Predicting for set of draws 27/30...
Predicting for set of draws 28/30...
Predicting for set of draws 29/30...
Predicting for set of draws 30/30...

Aggregated prediction
at MLE Sampled mean Sampled std.dev. Quantile 0.025 Quantile 0.975
SQ 495.4 496.8 38.21 434.2 565.8
RefA 930.3 931.5 23.76 885.5 965.6
RefB 856.3 853.8 18.08 823.7 884.1

Average prediction
at MLE Sampled mean Sampled std.dev. Quantile 0.025 Quantile 0.975
SQ 0.2171 0.2177 0.016743 0.1903 0.2479
RefA 0.4077 0.4082 0.010412 0.3880 0.4231
RefB 0.3752 0.3741 0.007924 0.3609 0.3874

The output from apollo_prediction is a list with two elements: a data.frame containing the predictions at
the estimated values, and an array with predictions for different values of the parameters drawn from
their asymptotic distribution.
>
> forecast = apollo_prediction(model, apollo_probabilities, apollo_inputs)
Your model contains continuous random parameters. apollo_prediction will perform averaging across draws for
these. For predictions at the level of individual draws, please call the apollo_probabilities function
using model$estimate as the parameters, and with functionality="raw".
Running predictions from model using parameter estimates...
Prediction at model estimates
SQ RefA RefB
Aggregate 495.40 930.34 856.26
Average 0.22 0.41 0.38

The output from apollo_prediction is a matrix containing the predictions at the estimated values.
>
>
> prediction_overview <- predictions_base[[1]]
>
> mean(prediction_overview$chosen)
[1] 0.3804237
_________________
But is there a specific manner how this number 0.38 and also the Rho2 0.2023 can be interpreted?
I did not find any hard benchmarks for that (only for R2 for linear models). But I suppose that for explaining human behaviour Rho2 of 0.20 is not too bad, meaning that 20% of variance can be explained by the model?
Your advice is highly appreciated.
stephanehess
Site Admin
Posts: 974
Joined: 24 Apr 2020, 16:29

Re: Predictions for post estimation

Post by stephanehess »

Hi

rho2 depends on a lot of characteristics, so we can't have simple benchmarks as with R2. But a rho2 around 0.2 is pretty common

Stephane
--------------------------------
Stephane Hess
www.stephanehess.me.uk
JuliavB
Posts: 41
Joined: 18 Aug 2021, 13:36

Re: Predictions for post estimation

Post by JuliavB »

Thank you very much for your support.
Anyhow, is it the right interpretation of rho2 that 20% of variance can be explained by the model?
And is there any recommendable literature on rho squared?
stephanehess
Site Admin
Posts: 974
Joined: 24 Apr 2020, 16:29

Re: Predictions for post estimation

Post by stephanehess »

--------------------------------
Stephane Hess
www.stephanehess.me.uk
Post Reply