Important: Read this before posting to this forum

  1. This forum is for questions related to the use of Apollo. We will answer some general choice modelling questions too, where appropriate, and time permitting. We cannot answer questions about how to estimate choice models with other software packages.
  2. There is a very detailed manual for Apollo available at http://www.ApolloChoiceModelling.com/manual.html. This contains detailed descriptions of the various Apollo functions, and numerous examples are available at http://www.ApolloChoiceModelling.com/examples.html. In addition, help files are available for all functions, using e.g. ?apollo_mnl
  3. Before asking a question on the forum, users are kindly requested to follow these steps:
    1. Check that the same issue has not already been addressed in the forum - there is a search tool.
    2. Ensure that the correct syntax has been used. For any function, detailed instructions are available directly in Apollo, e.g. by using ?apollo_mnl for apollo_mnl
    3. Check the frequently asked questions section on the Apollo website, which discusses some common issues/failures. Please see http://www.apollochoicemodelling.com/faq.html
    4. Make sure that R is using the latest official release of Apollo.
  4. If the above steps do not resolve the issue, then users should follow these steps when posting a question:
    1. provide full details on the issue, including the entire code and output, including any error messages
    2. posts will not immediately appear on the forum, but will be checked by a moderator first. We check the forum at least twice a week. It may thus take a couple of days for your post to appear and before we reply. There is no need to submit the post multiple times.

Convergence with different types of draws

Ask questions about errors you encouunter. Please make sure to include full details about your model specifications, and ideally your model file.
Post Reply
cybey
Posts: 60
Joined: 26 Apr 2020, 19:38

Convergence with different types of draws

Post by cybey »

Hello, everyone,

i would like to know from you if there is a rule of thumb, when to use which kind of draws (Halton, Sobol, MLHS, ...)?
During my research, severel papers recommended not to use Halton draws due to correlation with a high number of parameters. For example, the paper...

Czajkowski, Mikołaj; Budziński, Wiktor (2019): Simulation error in maximum likelihood estimation of discrete choice models. In: Journal of Choice Modelling 31, S. 73–85. DOI: 10.1016/j.jocm.2019.04.003.

... concludes that Sobol draws are usually "better" (see below) than Halton draws:

"We compare the performance of pseudo-random draws with three quasi Monte Carlo methods (Halton, Sobol and modified Latin hypercube sampling) under 27 experimental conditions that differ with respect to experimental design, number of individuals and number of choice tasks per individual. Based on a Monte Carlo simulation using 100 to 1,000,000 draws, we can compare the relative efficiency of different types of draws. We consistently find that a scrambled Sobol sequence performs the best in terms of the lowest simulation error, while being matched by scrambled Halton draws in the case of 10 attributes."

Now, in some model estimations (e.g. MIXL or ICLV), I have found that a very high number of draws increases the probability that the model will not converge or that I do get NAs for the standard errors, regardless of which draws are used. Furthermore, I have found that Sobol draws lead to NAs in the standard errors already with fewer draws than Halton draws. For example, the estimation of an ICLV with 500 Halton draws still works, whereas 500 Sobol draws result in NAs.

Do you have any advice/experience?
I look forward to your answers! :-)
Top
stephanehess
Site Admin
Posts: 1355
Joined: 24 Apr 2020, 16:29

Re: Convergence with different types of draws

Post by stephanehess »

Hi

I wouldn't go as far as objectively stating that one type of draws is better than another, but you can say the following:
- with just one random parameter, MLHS offers the most uniform coverage of the space of integration
- Haltons are very good with low numbers of dimensions, say up to 5 random parameters, but not above

In relation to your point about non-convergence, basically, more draws is ALWAYS better, in that it gets the numerical approximation closer to the model you actually wish to estimate. With low numbers of draws, you can sometimes get results for a model that is in fact not identified. So the fact that you are seeing problems with higher numbers of draws probably shows a problem with your model

Best wishes

Stephane
--------------------------------
Stephane Hess
www.stephanehess.me.uk
Top
Post Reply

Return to “Errors during model validation and/or estimation.”