Important: Read this before posting to this forum

  1. This forum is for questions related to the use of Apollo. We will answer some general choice modelling questions too, where appropriate, and time permitting. We cannot answer questions about how to estimate choice models with other software packages.
  2. There is a very detailed manual for Apollo available at http://www.ApolloChoiceModelling.com/manual.html. This contains detailed descriptions of the various Apollo functions, and numerous examples are available at http://www.ApolloChoiceModelling.com/examples.html. In addition, help files are available for all functions, using e.g. ?apollo_mnl
  3. Before asking a question on the forum, users are kindly requested to follow these steps:
    1. Check that the same issue has not already been addressed in the forum - there is a search tool.
    2. Ensure that the correct syntax has been used. For any function, detailed instructions are available directly in Apollo, e.g. by using ?apollo_mnl for apollo_mnl
    3. Check the frequently asked questions section on the Apollo website, which discusses some common issues/failures. Please see http://www.apollochoicemodelling.com/faq.html
    4. Make sure that R is using the latest official release of Apollo.
  4. If the above steps do not resolve the issue, then users should follow these steps when posting a question:
    1. provide full details on the issue, including the entire code and output, including any error messages
    2. posts will not immediately appear on the forum, but will be checked by a moderator first. This may take a day or two at busy times. There is no need to submit the post multiple times.

WTP-space estimates in latent class context

Ask questions about how to estimate models and how to change your settings for estimation.
Peter_C
Posts: 17
Joined: 03 May 2020, 13:52

WTP-space estimates in latent class context

Post by Peter_C »

Hi Stephane!

I have a question about WTP-space estimates in latent class context.

I think that, in terms of MNL and LC models I have to get exactly the same results for WTP-s (regardless of using WTP-space or computing with Deltha method).

However, now I testing a stated choice database (It is not to strong database, I get a very high standard errors for some attributes) I get different WTP results.

For example, I get the following results

With the use of WTP-space approach:

Estimate s.e. t.rat.(0) p(1-sided) Rob.s.e. Rob.t.rat.(0) p(1-sided)
beta_fcontent_a 340.022370 83.92135 4.0517 2.543e-05 96.80988 3.5123 2.2215e-04
beta_fcontent_b 150.280381 44.99867 3.3397 4.1940e-04 60.54464 2.4821 0.006530


With the use of Deltha method:

Value Robust s.e. Rob t-ratio (0)
Ratio of beta_fcontent_a and beta_ar_a: 1141.328 1364.992 0.84

Running Delta method computations
Value Robust s.e. Rob t-ratio (0)
Ratio of beta_fcontent_b and beta_ar_b: 124.0429 47.7689 2.6


Could be this a consequence of the wrong database?

Thank you very much for your answer!

Peter
stephanehess
Site Admin
Posts: 974
Joined: 24 Apr 2020, 16:29

Re: WTP-space estimates in latent class context

Post by stephanehess »

Peter

difficult to know without seeing the full results.

Stephane
--------------------------------
Stephane Hess
www.stephanehess.me.uk
Peter_C
Posts: 17
Joined: 03 May 2020, 13:52

Re: WTP-space estimates in latent class context

Post by Peter_C »

- Results of two-class LC model (in preference space)

Model diagnosis : successful convergence
Number of individuals : 261
Number of observations : 2088

Number of cores used : 3
Model without mixing

LL(start) : -2293.902
LL(0) : -2293.902
LL(final, whole model) : -2043.342
LL(component_1) : -2735.635
LL(component_2) : -2267.328
Rho-square (0) : 0.1092
Adj.Rho-square (0) : 0.1027
AIC : 4116.68
BIC : 4201.34
Estimated parameters : 15
Time taken (hh:mm:ss) : 00:00:22.12
Iterations : 38

Estimates:
Estimate s.e. t.rat.(0) p(1-sided) Rob.s.e. Rob.t.rat.(0) p(1-sided)
asc_1 0.000000 NA NA NA NA NA NA
asc_2 0.178090 0.05846 3.0463 0.001159 0.061323 2.9041 0.001841
asc_3 -0.359314 0.06882 -5.2211 8.892e-08 0.059564 -6.0324 8.077e-10
beta_price_a -0.001042 9.8913e-04 -1.0535 0.146060 0.001248 -0.8351 0.201840
beta_price_b -0.003083 4.8533e-04 -6.3516 1.065e-10 7.3047e-04 -4.2201 1.221e-05
beta_lf_a 0.000000 NA NA NA NA NA NA
beta_lf_b 0.000000 NA NA NA NA NA NA
beta_mf_a -1.059268 0.23980 -4.4173 4.998e-06 0.249029 -4.2536 1.052e-05
beta_mf_b -0.089158 0.12977 -0.6871 0.246024 0.106131 -0.8401 0.200435
beta_hf_a -1.189294 0.16887 -7.0427 9.425e-13 0.199568 -5.9593 1.266e-09
beta_hf_b -0.382383 0.09096 -4.2038 1.313e-05 0.102501 -3.7305 9.554e-05
beta_ls_a 0.000000 NA NA NA NA NA NA
beta_ls_b 0.000000 NA NA NA NA NA NA
beta_ms_a -1.365950 0.23251 -5.8749 2.116e-09 0.277412 -4.9239 4.242e-07
beta_ms_b 0.151109 0.10213 1.4795 0.069503 0.112409 1.3443 0.089429
beta_hs_a -1.193212 0.19335 -6.1711 3.391e-10 0.232448 -5.1332 1.424e-07
beta_hs_b -0.228510 0.09948 -2.2970 0.010808 0.094897 -2.4080 0.008021
beta_ncs_a 0.000000 NA NA NA NA NA NA
beta_ncs_b 0.000000 NA NA NA NA NA NA
beta_cs_a -1.453355 0.18967 -7.6625 9.104e-15 0.274867 -5.2875 6.201e-08
beta_cs_b 0.645475 0.09860 6.5465 2.945e-11 0.125705 5.1349 1.412e-07
delta_a -0.722715 0.19682 -3.6720 1.2032e-04 0.236975 -3.0498 0.001145
delta_b 0.000000 NA NA NA NA NA NA



- WTP-s with use of Deltha method:

Running Delta method computations
Value Robust s.e. Rob t-ratio (0)
Ratio of beta_mf_a and beta_price_a: 1016.546 1200.944 0.85


Running Delta method computations
Value Robust s.e. Rob t-ratio (0)
Ratio of beta_hf_a and beta_price_a: 1141.328 1364.992 0.84


Running Delta method computations
Value Robust s.e. Rob t-ratio (0)
Ratio of beta_ms_a and beta_price_a: 1310.86 1496.446 0.88


Running Delta method computations
Value Robust s.e. Rob t-ratio (0)
Ratio of beta_hs_a and beta_price_a: 1145.088 1380.161 0.83


Running Delta method computations
Value Robust s.e. Rob t-ratio (0)
Ratio of beta_cs_a and beta_price_a: 1394.739 1556.314 0.9


Running Delta method computations
Value Robust s.e. Rob t-ratio (0)
Ratio of beta_mf_b and beta_price_b: 28.9222 34.6461 0.83

Running Delta method computations
Value Robust s.e. Rob t-ratio (0)
Ratio of beta_hf_b and beta_price_b: 124.0429 47.7689 2.6


Running Delta method computations
Value Robust s.e. Rob t-ratio (0)
Ratio of beta_ms_b and beta_price_b: -49.019 36.385 -1.35


Running Delta method computations
Value Robust s.e. Rob t-ratio (0)
Ratio of beta_hs_b and beta_price_b: 74.1272 33.6565 2.2


Running Delta method computations
Value Robust s.e. Rob t-ratio (0)
Ratio of beta_cs_b and beta_price_b: -209.3884 50.274 -4.16







- Results of two-class LC model (in WTP-space)

Model diagnosis : successful convergence
Number of individuals : 261
Number of observations : 2088

Number of cores used : 3
Model without mixing

LL(start) : -2293.902
LL(0, whole model) : -2293.902
LL(final, whole model) : -2046.229
Rho-square (0) : 0.108
Adj.Rho-square (0) : 0.1014
AIC : 4122.46
BIC : 4207.12

LL(0,component_1) : -2293.902
LL(final,component_1) : -2783.786
LL(0,component_2) : -2293.902
LL(final,component_2) : -2254.78

Estimated parameters : 15
Time taken (hh:mm:ss) : 00:01:46.27
pre-estimation : 00:00:3.81
estimation : 00:01:26.68
post-estimation : 00:00:15.78
Iterations : 167
Min abs eigenvalue of Hessian : 3.4e-05

Estimates:
Estimate s.e. t.rat.(0) p(1-sided) Rob.s.e. Rob.t.rat.(0) p(1-sided)
asc_1 0.000000 NA NA NA NA NA NA
asc_2 0.184758 0.05839 3.1641 7.7788e-04 0.06140 3.0091 0.001310
asc_3 -0.315623 0.06622 -4.7665 9.373e-07 0.05626 -5.6100 1.012e-08
beta_price_a -0.003473 8.2517e-04 -4.2089 1.283e-05 8.6621e-04 -4.0095 3.042e-05
beta_price_b -0.002722 4.7040e-04 -5.7862 3.600e-09 7.2077e-04 -3.7763 7.960e-05
beta_lf_a 0.000000 NA NA NA NA NA NA
beta_lf_b 0.000000 NA NA NA NA NA NA
beta_mf_a 307.263187 88.20952 3.4833 2.4761e-04 107.93435 2.8468 0.002208
beta_mf_b 32.679886 45.64531 0.7160 0.237010 40.37374 0.8094 0.209133
beta_hf_a 340.022370 83.92135 4.0517 2.543e-05 96.80988 3.5123 2.2215e-04
beta_hf_b 150.280381 44.99867 3.3397 4.1940e-04 60.54464 2.4821 0.006530
beta_ls_a 0.000000 NA NA NA NA NA NA
beta_ls_b 0.000000 NA NA NA NA NA NA
beta_ms_a 429.780001 90.09968 4.7701 9.209e-07 100.64337 4.2703 9.759e-06
beta_ms_b -44.779287 38.91653 -1.1506 0.124938 42.49402 -1.0538 0.145992
beta_hs_a 320.834936 86.59669 3.7049 1.0572e-04 89.13199 3.5995 1.5938e-04
beta_hs_b 89.182177 36.61691 2.4355 0.007435 41.80949 2.1331 0.016460
beta_ncs_a 0.000000 NA NA NA NA NA NA
beta_ncs_b 0.000000 NA NA NA NA NA NA
beta_cs_a 455.948333 92.81858 4.9123 4.502e-07 71.12041 6.4109 7.231e-11
beta_cs_b -225.421629 40.92241 -5.5085 1.809e-08 61.37646 -3.6728 1.1997e-04
delta_a -0.784601 0.21278 -3.6874 1.1327e-04 0.27494 -2.8537 0.002161
delta_b 0.000000 NA NA NA NA NA NA
stephanehess
Site Admin
Posts: 974
Joined: 24 Apr 2020, 16:29

Re: WTP-space estimates in latent class context

Post by stephanehess »

Hi, the WTP space model has clearly converged to an inferior solution if you look at the model fit. What starting values did you use?
--------------------------------
Stephane Hess
www.stephanehess.me.uk
Peter_C
Posts: 17
Joined: 03 May 2020, 13:52

Re: WTP-space estimates in latent class context

Post by Peter_C »

Hi Stephane,

I used 0 starting value for every parameters, both in preference- and WTP-space estimation.

Peter
stephanehess
Site Admin
Posts: 974
Joined: 24 Apr 2020, 16:29

Re: WTP-space estimates in latent class context

Post by stephanehess »

Peter

that's likely the problem, as in WTP space, you are then initially multiplying two parameters that are zero with each other, and the first few steps in the estimation could lead you astray. I suggest you try again with a small negative starting value for the cost coefficient.

Best wishes

Stephane
--------------------------------
Stephane Hess
www.stephanehess.me.uk
Peter_C
Posts: 17
Joined: 03 May 2020, 13:52

Re: WTP-space estimates in latent class context

Post by Peter_C »

Thank you Stephane!

The log-likelihood function reached earlier the point of convergation but the received (WTP) parameters were near identical than I get with 0 starting values.

I don't understand how I get so different results by using deltha method and WTP-space.

with 0.01 starting values for price parameters:

Model diagnosis : successful convergence
Number of individuals : 261
Number of observations : 2088

Number of cores used : 2
Model without mixing

LL(start) : -2293.902
LL(0, whole model) : -2293.902
LL(final, whole model) : -2049.201
Rho-square (0) : 0.1067
Adj.Rho-square (0) : 0.1001
AIC : 4128.4
BIC : 4213.06

LL(0,Class_1) : -2293.902
LL(final,Class_1) : -2253.414
LL(0,Class_2) : -2293.902
LL(final,Class_2) : -2793.557

Estimated parameters : 15
Time taken (hh:mm:ss) : 00:01:9.54
pre-estimation : 00:00:3.88
estimation : 00:00:46.51
post-estimation : 00:00:19.15
Iterations : 103
Min abs eigenvalue of Hessian : 7.7e-05

Estimates:
Estimate s.e. t.rat.(0) p(1-sided) Rob.s.e. Rob.t.rat.(0) p(1-sided)
asc_alt1 0.000000 NA NA NA NA NA NA
asc_alt2 0.185889 0.05850 3.1775 7.4269e-04 0.06177 3.0092 0.001310
asc_alt3 -0.292573 0.06572 -4.4518 4.259e-06 0.05725 -5.1109 1.603e-07
b_price_a -0.002335 5.3406e-04 -4.3717 6.164e-06 9.1261e-04 -2.5583 0.005259
b_price_b -0.004333 8.4999e-04 -5.0983 1.714e-07 9.0957e-04 -4.7643 9.475e-07
b_lf_a 0.000000 NA NA NA NA NA NA
b_lf_b 0.000000 NA NA NA NA NA NA
b_mf_a 25.805171 54.65237 0.4722 0.318403 49.38549 0.5225 0.300652
b_mf_b 231.377935 68.99656 3.3535 3.9902e-04 93.25640 2.4811 0.006549
b_hf_a 190.866708 62.43072 3.0573 0.001117 95.93762 1.9895 0.023324
b_hf_b 256.101888 56.37639 4.5427 2.777e-06 70.27609 3.6442 1.3410e-04
delta_a 0.779896 0.21642 3.6036 1.5693e-04 0.28863 2.7021 0.003446
delta_b 0.000000 NA NA NA NA NA NA
b_ls_a 0.000000 NA NA NA NA NA NA
b_ls_b 0.000000 NA NA NA NA NA NA
b_ms_a -83.218283 55.93253 -1.4878 0.068397 72.76930 -1.1436 0.126397
b_ms_b 346.698465 62.52782 5.5447 1.472e-08 75.21424 4.6095 2.018e-06
b_hs_a 94.823214 42.68935 2.2212 0.013167 49.99830 1.8965 0.028945
b_hs_b 233.115390 58.39092 3.9923 3.271e-05 64.56657 3.6105 1.5282e-04
b_ncs_a 0.000000 NA NA NA NA NA NA
b_ncs_b 0.000000 NA NA NA NA NA NA
b_cs_a -253.411941 58.46643 -4.3343 7.311e-06 99.96700 -2.5350 0.005623
b_cs_b 378.258663 64.80076 5.8373 2.653e-09 57.30322 6.6010 2.042e-11


with 0.5 starting values for price parameters:

Model diagnosis : successful convergence
Number of individuals : 261
Number of observations : 2088

Number of cores used : 2
Model without mixing

LL(start) : -2293.902
LL(0, whole model) : -2293.902
LL(final, whole model) : -2088.856
Rho-square (0) : 0.0894
Adj.Rho-square (0) : 0.0828
AIC : 4207.71
BIC : 4292.37

LL(0,Class_1) : -2293.902
LL(final,Class_1) : -2715.362
LL(0,Class_2) : -2293.902
LL(final,Class_2) : -2210.779

Estimated parameters : 15
Time taken (hh:mm:ss) : 00:00:52.78
pre-estimation : 00:00:3.25
estimation : 00:00:36.42
post-estimation : 00:00:13.12
Iterations : 82
Min abs eigenvalue of Hessian : 3.3e-05
Some eigenvalues of Hessian are positive, indicating potential problems!

Estimates:
Estimate s.e. t.rat.(0) p(1-sided) Rob.s.e. Rob.t.rat.(0) p(1-sided)
asc_alt1 0.000000 NA NA NA NA NA NA
asc_alt2 0.164824 0.05478 3.0090 0.001310 0.077054 2.1391 0.016215
asc_alt3 -0.301477 0.04939 -6.1044 5.158e-10 0.145377 -2.0738 0.019051
b_price_a 0.003165 7.3550e-04 4.3030 8.425e-06 0.001739 1.8200 0.034378
b_price_b -0.002973 NaN NaN NaN 0.003568 -0.8334 0.202308
b_lf_a 0.000000 NA NA NA NA NA NA
b_lf_b 0.000000 NA NA NA NA NA NA
b_mf_a 78.050605 66.57479 1.1724 0.120523 166.169997 0.4697 0.319283
b_mf_b 209.441159 NaN NaN NaN 310.215035 0.6751 0.249791
b_hf_a -91.067070 65.31790 -1.3942 0.081627 87.998735 -1.0349 0.150365
b_hf_b 233.918803 NaN NaN NaN 255.024785 0.9172 0.179509
delta_a -1.177256 0.26975 -4.3642 6.378e-06 0.351680 -3.3475 4.0768e-04
delta_b 0.000000 NA NA NA NA NA NA
b_ls_a 0.000000 NA NA NA NA NA NA
b_ls_b 0.000000 NA NA NA NA NA NA
b_ms_a 18.428679 69.46038 0.2653 0.395385 83.114660 0.2217 0.412264
b_ms_b 102.421134 27.15584 3.7716 8.110e-05 34.058661 3.0072 0.001318
b_hs_a -39.717870 64.57622 -0.6151 0.269259 60.579493 -0.6556 0.256030
b_hs_b 241.544850 NaN NaN NaN 349.484184 0.6911 0.244737
b_ncs_a 0.000000 NA NA NA NA NA NA
b_ncs_b 0.000000 NA NA NA NA NA NA
b_cs_a 364.845986 83.45261 4.3719 6.159e-06 199.818007 1.8259 0.033933
b_cs_b 132.766890 NaN NaN NaN 153.326407 0.8659 0.193270

Peter
stephanehess
Site Admin
Posts: 974
Joined: 24 Apr 2020, 16:29

Re: WTP-space estimates in latent class context

Post by stephanehess »

Peter

you still have a situation here where your model in WTP space converges to a worse solution (more negative LL) than the model in preference space. So as a result, the delta method values for preference space will be different from the wtp results for wtp space, as it is a different solution. WTP space models have a highly complex likelihood function given the non-linear utility specification, and this makes estimation harder, even more so in latent class. Your model in preference space is fine, so is there another reason why you want WTP space?

Stephane
--------------------------------
Stephane Hess
www.stephanehess.me.uk
Peter_C
Posts: 17
Joined: 03 May 2020, 13:52

Re: WTP-space estimates in latent class context

Post by Peter_C »

Hi Stephane!

Thank you for your answers!

It became completely understandable to me.

I'm just trying to test and compare results in some model specifications.

To this point, I have found that (on other databases) in case of MNL and LC models, the two approaches were completely the same.

However, it has already became clear to me that, log-likelihood finds another solution.

Best wishes,
Peter
arohamirai
Posts: 18
Joined: 28 Jun 2020, 16:26

Re: WTP-space estimates in latent class context

Post by arohamirai »

Hi Stephane,

Thank you for the solution. I actually wanted to ask a similar question regarding different results obtained from preference-space and WTP-space approach, but your solution solves my problem quite well. Actually, the above problem happens in my situation even when I use the most basic MNL model, with the WTP-space providing an inferior model fit. Following your suggestion, I changed the starting values a couple of times and finally got the results consistent.

However, I still have a question: Given you have mentioned that "WTP space models have a highly complex likelihood function", I'm wondering except keeping trying different starting values, is there anything else a DCE practitioner can do to make sure that the WTP-space model does not converge to an inferior solution? Or, maybe we should use the WTP estimates calculated from preference-space as starting values for the WTP-space model? This is a concern to me as I'm currently working on a large dataset and it takes many hours just to run a basic MXL.

Thank you very much.

Tim
Post Reply