Important: Read this before posting to this forum

  1. This forum is for questions related to the use of Apollo. We will answer some general choice modelling questions too, where appropriate, and time permitting. We cannot answer questions about how to estimate choice models with other software packages.
  2. There is a very detailed manual for Apollo available at http://www.ApolloChoiceModelling.com/manual.html. This contains detailed descriptions of the various Apollo functions, and numerous examples are available at http://www.ApolloChoiceModelling.com/examples.html. In addition, help files are available for all functions, using e.g. ?apollo_mnl
  3. Before asking a question on the forum, users are kindly requested to follow these steps:
    1. Check that the same issue has not already been addressed in the forum - there is a search tool.
    2. Ensure that the correct syntax has been used. For any function, detailed instructions are available directly in Apollo, e.g. by using ?apollo_mnl for apollo_mnl
    3. Check the frequently asked questions section on the Apollo website, which discusses some common issues/failures. Please see http://www.apollochoicemodelling.com/faq.html
    4. Make sure that R is using the latest official release of Apollo.
  4. If the above steps do not resolve the issue, then users should follow these steps when posting a question:
    1. provide full details on the issue, including the entire code and output, including any error messages
    2. posts will not immediately appear on the forum, but will be checked by a moderator first. This may take a day or two at busy times. There is no need to submit the post multiple times.

estimation in WTP-space

Ask questions about errors you encouunter. Please make sure to include full details about your model specifications, and ideally your model file.
maa033
Posts: 35
Joined: 23 Jul 2020, 14:00

estimation in WTP-space

Post by maa033 »

I have a problem With NANs and NAs for std.error and rob.std.error in the output when I run the model in WTP-Space. This typically appears when I work With random cost parameter and specify the distribution of the cost-parameter to be negative lognormal:

randcoeff[["cost_B"]] = -exp(cost_B_mu + cost_B_sig * draws_cost)

If I use a normal distribution for the cost parameter, like the one below, the results are fine With no NANs or NAs.
randcoeff[["cost_T"]] = cost_B_mu + cost_B_sig * draws_cost

Is this a problem in my data or a misspecification of the distribution?
stephanehess
Site Admin
Posts: 974
Joined: 24 Apr 2020, 16:29

Re: estimation in WTP-space

Post by stephanehess »

Hi

there could be many reasons for this, and it's difficult to diagnose without seeing your outputs, and also how they compare to the model with Normals. A Normal distribution for cost of course makes no sense, so you're right to try negative Lognormals. One thing to consider is whether your starting values for the negative Lognormal are appropriate. I would advise starting cost_B_mu at something like -3 or even -5

Best wishes

Stephane
--------------------------------
Stephane Hess
www.stephanehess.me.uk
maa033
Posts: 35
Joined: 23 Jul 2020, 14:00

Re: estimation in WTP-space

Post by maa033 »

Thanks. It worked With higher starring value for the mean (-3). However, I kept the starting value for the std.dev at 0, and this resulted in a very large and significant std.deviation, but insignificant mean cost estimate. Should I use a higher (absolute) starting value for the std.dev. of the cost parameter as well?
stephanehess
Site Admin
Posts: 974
Joined: 24 Apr 2020, 16:29

Re: estimation in WTP-space

Post by stephanehess »

Hi

again, it's not possible for us to really help without seeing output. Also, when you say an insignificant mean, do you mean the estimate for cost_B_mu - this is not the estimate of the mean of cost_B, but the estimate of the mean of log(-cost_B). The fact that cost_B_mu is not significantly different from zero does not imply that the mean of cost_B is not significantly different from zero. See Table 2 in http://www.stephanehess.me.uk/papers/jo ... B_2012.pdf

Stephane
--------------------------------
Stephane Hess
www.stephanehess.me.uk
maa033
Posts: 35
Joined: 23 Jul 2020, 14:00

Re: estimation in WTP-space

Post by maa033 »

This is the output when the cost parameter is specified as negative lognormal Distributed With starting value -3. What I do is estimating a merged dataset, consisting of data collected in two different situations (baseline and treatment) to test for possible differences in stated preferences across the situations. Thus two estimated parameters for each attribute. I am aware that I need to correct for differences in scale across the two situations. I will look into the paper you sent and find out how to transform the estimates for cost_B_mu and cost_T_mu into mean values for the cost attribute parameters.


Model name : MMNL_sim
Model description : MMNL model for simultaneous estimation of coeff
Model run at : 2020-08-18 12:25:07
Estimation method : bfgs
Model diagnosis : successful convergence
Number of individuals : 293
Number of observations : 2599

Number of cores used : 3
Number of inter-person draws : 100 (halton)

LL(start) : -2855.802
LL(0) : -2855.293
LL(final) : -1466.954
Rho-square (0) : 0.4862
Adj.Rho-square (0) : 0.4785
AIC : 2977.91
BIC : 3106.89
Estimated parameters : 22
Time taken (hh:mm:ss) : 00:14:3.27
Iterations : 93
Min abs eigenvalue of hessian : 1.196175

Estimates:
Estimate Std.err. t.ratio(0) Rob.std.err. Rob.t.ratio(0)
asc -4.8354 0.7656 -6.32 1.0061 -4.81
cost_B_mu -0.3268 0.2470 -1.32 0.2765 -1.18
cost_B_sig 3.1010 0.2563 12.10 0.2760 11.24
torsk_B_mu -0.2918 0.0411 -7.11 0.0421 -6.93
torsk_B_sig 0.2589 0.0427 6.07 0.0484 5.35
laks_B_mu -0.2584 0.0570 -4.54 0.0555 -4.66
laks_B_sig 0.3049 0.0542 5.62 0.0586 5.21
bunn_B_mu -0.4219 0.1863 -2.26 0.1639 -2.57
bunn_B_sig -0.1682 0.2753 -0.61 0.2282 -0.74
land_B_mu -0.0213 0.0104 -2.05 0.0096 -2.22
land_B_sig 0.0463 0.0150 3.09 0.0149 3.10
cost_T_mu 0.4443 0.3106 1.43 0.3417 1.30
cost_T_sig 2.4355 0.1621 15.03 0.1412 17.25
torsk_T_mu -0.4622 0.1483 -3.12 0.1850 -2.50
torsk_T_sig 0.4676 0.1244 3.76 0.1536 3.04
laks_T_mu -0.5971 0.2136 -2.79 0.2548 -2.34
laks_T_sig -0.7057 0.1955 -3.61 0.2487 -2.84
bunn_T_mu -0.7102 0.4612 -1.54 0.4438 -1.60
bunn_T_sig -1.5096 0.4573 -3.30 0.5148 -2.93
land_T_mu -0.0742 0.0310 -2.39 0.0327 -2.27
land_T_sig -0.1331 0.0396 -3.36 0.0504 -2.64
scale_base 1.0000 NA NA NA NA
scale_treat 0.6339 0.1681 3.77 0.2115 3.00

Overview of choices for model component "MNL"
alt1 alt2 alt3
Times available 2599.00 2599.00 2599.00
Times chosen 429.00 1056.00 1114.00
Percentage chosen overall 16.51 40.63 42.86
Percentage chosen when available 16.51 40.63 42.86
dpalma
Posts: 190
Joined: 24 Apr 2020, 17:54

Re: estimation in WTP-space

Post by dpalma »

Hi,

You can obtain the mean and the s.d. of your lognormal parameters using the following code snippet:

Code: Select all

deltaMethod_settings <- list(operation="lognormal", parName1="cost_B_mu", parName2="cost_B_sig")
apollo_deltaMethod(model, deltaMethod_settings)

deltaMethod_settings <- list(operation="lognormal", parName1="cost_T_mu", parName2="cost_T_sig")
apollo_deltaMethod(model, deltaMethod_settings)
This will also show the standard error of these values, so you will be able to tell if they are significant or not.

Remember that -as you mentioned- the base and treatment parameters still have a different scale, so they cannot be compared directly if you are working in preference space.

Best
David
maa033
Posts: 35
Joined: 23 Jul 2020, 14:00

Re: estimation in WTP-space

Post by maa033 »

Thanks a lot for this! Worked fine.
I tried to run the same model in WTP-Space, but got very strange results:

Model name : MMNL_sim
Model description : MMNL model for simultaneous estimation of coeff
Model run at : 2020-08-19 11:35:52
Estimation method : bfgs
Model diagnosis : successful convergence
Number of individuals : 293
Number of observations : 2599

Number of cores used : 3
Number of inter-person draws : 100 (halton)

LL(start) : -2855.802
LL(0) : -2855.293
LL(final) : -1848.092
Rho-square (0) : 0.3527
Adj.Rho-square (0) : 0.3454
AIC : 3738.18
BIC : 3861.3
Estimated parameters : 21
Time taken (hh:mm:ss) : 00:14:58.33
Iterations : 170
Min abs eigenvalue of hessian : 0.012457

Estimates:
Estimate Std.err. t.ratio(0) Rob.std.err. Rob.t.ratio(0)
asc -3.7459 0.2032 -18.43 0.2171 -17.26
cost_B_mu -13.1867 1.4974 -8.81 2.1216 -6.22
cost_B_sig -7.9258 1.3837 -5.73 2.0412 -3.88
torsk_B_mu -0.2325 3.4799 -0.07 6.0949 -0.04
torsk_B_sig -10.0244 3.1396 -3.19 5.5856 -1.79
laks_B_mu -8.5628 NaN NaN 0.8000 -10.70
laks_B_sig 9.6588 3.0225 3.20 5.6484 1.71
bunn_B_mu -10.1113 3.9581 -2.55 7.7880 -1.30
bunn_B_sig -18.5782 2.8551 -6.51 3.8263 -4.86
land_B_mu -9.0525 NaN NaN 0.5508 -16.43
land_B_sig 6.9587 0.4527 15.37 0.3451 20.16
cost_T_mu -9.8385 0.6132 -16.04 0.3676 -26.77
cost_T_sig 7.9208 0.8280 9.57 0.5399 14.67
torsk_T_mu 4.7083 0.7837 6.01 1.6621 2.83
torsk_T_sig 13.0653 1.5839 8.25 2.8747 4.54
laks_T_mu 4.3214 1.9341 2.23 3.8115 1.13
laks_T_sig 0.4718 0.4190 1.13 0.2942 1.60
bunn_T_mu 0.0409 2.0169 0.02 4.2892 0.01
bunn_T_sig 11.9307 2.3452 5.09 3.1985 3.73
land_T_mu -19.8303 0.6090 -32.56 0.6813 -29.11
land_T_sig -25.0320 0.6740 -37.14 0.4242 -59.01


The model specification I used was as follows, With normal non-cost attributes and lognormal cost:
V = list()
V[['alt1']] = Certain*(asc + cost_B*(torsk_B * KT1 + Cost1 + laks_B * VL1 + bunn_B * HB1 + land_B * KL1))+
(1-Certain)*(asc + cost_T*(torsk_T * KT1 + Cost1 + laks_T * VL1 + bunn_T * HB1 + land_T * KL1))


V[['alt2']] = Certain*(cost_B*(torsk_B * KT2 + Cost2 + laks_B * VL2 + bunn_B * HB2 + land_B * KL2))+
(1-Certain)*(cost_T*(torsk_T * KT2 + Cost2 + laks_T * VL2 + bunn_T * HB2 + land_T * KL2))

V[['alt3']] = Certain*(cost_B*(torsk_B * KT3 + Cost3 + laks_B * VL3 + bunn_B * HB3 + land_B * KL3))+
(1-Certain)*(cost_T*(torsk_T * KT3 + Cost3 + laks_T * VL3 + bunn_T * HB3 + land_T * KL3))


Is this specification correct? If yes, then it might be the very low number of draws (100) that caused the WTP-estimates??







Overview of choices for model component "MNL"
alt1 alt2 alt3
Times available 2599.00 2599.00 2599.00
Times chosen 429.00 1056.00 1114.00
Percentage chosen overall 16.51 40.63 42.86
Percentage chosen when available 16.51 40.63 42.86
dpalma
Posts: 190
Joined: 24 Apr 2020, 17:54

Re: estimation in WTP-space

Post by dpalma »

Hi,

The formulation looks fine, but I would recommend using a different ASC for the base a treatment scenarios. Also, why does only one alternative have an ASC? Are the second and third alternatives unlabelled while the first one is labelled?

Keep in mind that mixed logits potentially have multiple optima, so the estimates you obtain main be dependent on the starting value of your parameters. I would recommend:
  • Increase the number of draws
  • Use the apollo_searchStart function. This function will generate random starting values for your parameters and return the parameter values that seem to lead to the best optima. You can see more details by typing ?apollo_searchStart, look at the manual (http://www.apollochoicemodelling.com/files/Apollo.pdf), or look at examples 18 and 19.
Best
David
maa033
Posts: 35
Joined: 23 Jul 2020, 14:00

Re: estimation in WTP-space

Post by maa033 »

This was helpful.
Does it matter where in the script I insert the searchStart command?
I tried to figure out from the manual where to insert it, but could not find out??
dpalma
Posts: 190
Joined: 24 Apr 2020, 17:54

Re: estimation in WTP-space

Post by dpalma »

You should call apollo_searchStart just before calling apollo_estimate, as in:

Code: Select all

apollo_beta=apollo_searchStart(apollo_beta, apollo_fixed,apollo_probabilities, apollo_inputs)
model = apollo_estimate(apollo_beta, apollo_fixed, apollo_probabilities, apollo_inputs)
Be aware that apollo_searchStart can take a long time to run, and it doesn't guarantee you getting the best solution possible (global optima of the LL), but it can help finding a better solution.
Post Reply