ApolloChoiceModelling forum

Dear all,

for my post-estimation analysis I´ve used the following code:

predictions_base = apollo_prediction(model, apollo_probabilities, apollo_inputs,
prediction_settings=list(runs=30))

forecast = apollo_prediction(model, apollo_probabilities, apollo_inputs)

On the basis of the resulting list I´ve analysed in how many cases the option with the highest probability was chosen.

prediction_overview <- predictions_base[[1]]
table(apply(prediction_overview[,3:5],1,max) == prediction_overview$chosen)
table(apply(prediction_overview[,3:5],1,max) == prediction_overview$chosen) /
sum(table(apply(prediction_overview[,3:5],1,max) == prediction_overview$chosen))

As I have 3 options to choose (status quo + 2 alternatives) in my experiment a random draw would have 33,3% probability.
From the above analysis I´ve found out that in 47% of observations the option with the highest probability is chosen. So the model improves the forecast by ~ 14%.

1. Are my above thoughts and conclusions right to answer the question how many choices can be predicted correctly by the model?
2. Or do I have to take a random sample from my total sample to do a out ouf sample test for post estimation ?

Any advice is highly appreciated.
Thanks and best,
J.

Hi

sorry for the slow reply.

I would never look at whether the chosen option has the highest probability of being chosen. That metric is not in line with probabilistic choice models (see e.g. the discussion in Train).

Instead, you should look at the average probability of correct prediction, which Apollo gives you in prediction too, and then compare that to 1/3

Or of course just look at rho2

Stephane

Hi Stephane,

thank you for your response.

My results in predictions for one of my models are as follows:

Aggregated prediction
at MLE Sampled mean Sampled std.dev. Quantile 0.025 Quantile 0.975
SQ 495.4 496.8 38.21 434.2 565.8
RefA 930.3 931.5 23.76 885.5 965.6
RefB 856.3 853.8 18.08 823.7 884.1

Average prediction
at MLE Sampled mean Sampled std.dev. Quantile 0.025 Quantile 0.975
SQ 0.2171 0.2177 0.016743 0.1903 0.2479
RefA 0.4077 0.4082 0.010412 0.3880 0.4231
RefB 0.3752 0.3741 0.007924 0.3609 0.3874

The output from apollo_prediction is a list with two elements: a data.frame containing the predictions at
the estimated values, and an array with predictions for different values of the parameters drawn from
their asymptotic distribution.
>
> forecast = apollo_prediction(model, apollo_probabilities, apollo_inputs)
Your model contains continuous random parameters. apollo_prediction will perform averaging across draws for
these. For predictions at the level of individual draws, please call the apollo_probabilities function
using model$estimate as the parameters, and with functionality="raw".
Running predictions from model using parameter estimates...
Prediction at model estimates
SQ RefA RefB
Aggregate 495.40 930.34 856.26
Average 0.22 0.41 0.38

The output from apollo_prediction is a matrix containing the predictions at the estimated values.
__________
Rho-square (0) : 0.2023
__________

But where can I see the average probability of correct prediction?
Can you give any help on the interpretation of these results?

Thank you very much in advance!

Oh, and the parameter estimates are all highly significant.

try

I´ve estimated it with the following code and results:

predictions_base = apollo_prediction(model, apollo_probabilities, apollo_inputs,
+ prediction_settings=list(runs=30))
Your model contains continuous random parameters. apollo_prediction will perform averaging across draws for
these. For predictions at the level of individual draws, please call the apollo_probabilities function
using model$estimate as the parameters, and with functionality="raw".
Running predictions from model using parameter estimates...
Running predictions across draws from the asymptotic distribution for maximum likelihood estimates.
Predicting for set of draws 1/30...
Predicting for set of draws 2/30...
Predicting for set of draws 3/30...
Predicting for set of draws 4/30...
Predicting for set of draws 5/30...
Predicting for set of draws 6/30...
Predicting for set of draws 7/30...
Predicting for set of draws 8/30...
Predicting for set of draws 9/30...
Predicting for set of draws 10/30...
Predicting for set of draws 11/30...
Predicting for set of draws 12/30...
Predicting for set of draws 13/30...
Predicting for set of draws 14/30...
Predicting for set of draws 15/30...
Predicting for set of draws 16/30...
Predicting for set of draws 17/30...
Predicting for set of draws 18/30...
Predicting for set of draws 19/30...
Predicting for set of draws 20/30...
Predicting for set of draws 21/30...
Predicting for set of draws 22/30...
Predicting for set of draws 23/30...
Predicting for set of draws 24/30...
Predicting for set of draws 25/30...
Predicting for set of draws 26/30...
Predicting for set of draws 27/30...
Predicting for set of draws 28/30...
Predicting for set of draws 29/30...
Predicting for set of draws 30/30...

Aggregated prediction
at MLE Sampled mean Sampled std.dev. Quantile 0.025 Quantile 0.975
SQ 495.4 496.8 38.21 434.2 565.8
RefA 930.3 931.5 23.76 885.5 965.6
RefB 856.3 853.8 18.08 823.7 884.1

Average prediction
at MLE Sampled mean Sampled std.dev. Quantile 0.025 Quantile 0.975
SQ 0.2171 0.2177 0.016743 0.1903 0.2479
RefA 0.4077 0.4082 0.010412 0.3880 0.4231
RefB 0.3752 0.3741 0.007924 0.3609 0.3874

The output from apollo_prediction is a list with two elements: a data.frame containing the predictions at
the estimated values, and an array with predictions for different values of the parameters drawn from
their asymptotic distribution.
>
> forecast = apollo_prediction(model, apollo_probabilities, apollo_inputs)
Your model contains continuous random parameters. apollo_prediction will perform averaging across draws for
these. For predictions at the level of individual draws, please call the apollo_probabilities function
using model$estimate as the parameters, and with functionality="raw".
Running predictions from model using parameter estimates...
Prediction at model estimates
SQ RefA RefB
Aggregate 495.40 930.34 856.26
Average 0.22 0.41 0.38

The output from apollo_prediction is a matrix containing the predictions at the estimated values.
>
>
> prediction_overview <- predictions_base[[1]]
>
> mean(prediction_overview$chosen)
[1] 0.3804237
_________________
But is there a specific manner how this number 0.38 and also the Rho2 0.2023 can be interpreted?
I did not find any hard benchmarks for that (only for R2 for linear models). But I suppose that for explaining human behaviour Rho2 of 0.20 is not too bad, meaning that 20% of variance can be explained by the model?
Your advice is highly appreciated.

Hi

rho2 depends on a lot of characteristics, so we can't have simple benchmarks as with R2. But a rho2 around 0.2 is pretty common

Stephane

Thank you very much for your support.
Anyhow, is it the right interpretation of rho2 that 20% of variance can be explained by the model?
And is there any recommendable literature on rho squared?

I recommend this paper: https://doi.org/10.1016/j.jocm.2016.02.001