General specification of eMDC (to avoid errors)
Posted: 14 Feb 2024, 10:00
Dear Stephane/David
I am working with supermarket scannerdata, and are trying to analyze this using the Apollo eMDC implementation (mainly for “practicing” the use of these models). I have purchasing data for around 700 households over the course of a year, and have picked out 15-20 specific products that I am including as my “inside goods”. I both have a dataset where household purchases are aggregated on a weekly basis, and one where it is aggregated on a monthly basis.
With around 15 products/alternatives, quite often, one of the following messages appear (I do not include my code, as I refer to several different model-specifications):
- Log-likelihood calculation fails at values close to the starting values,
- WARNING: Estimation failed. No covariance matrix to compute,
- Error in if (any(testL == 0)) cat("\nSome observations have zero probability at starting value for eMDCEV model component.").
In my case, this particularly happens when including a budget or when including explanatory variables (e.g. income) in the utility of the outside good, i.e. the estimation process is smooth when excluding these factors, but run into trouble when included. One way to seemingly aid the estimation is to reduce the number of observations per individual (to reduce the chance that the product of many individual observations come too close to zero). Using workInLogs=TRUE in apollo_control or starting values from simpler models generally doesn’t help. This lead me to three questions:
- Is it a general issue that models with a budget / outside good covariates are considerably more difficult to estimate?
- Is there a general “rule-of-thumb” as to the number of alternatives (and/or parameters) that it is feasible to include in an eMDC model? Or is the answer to this simply too case-dependent?
- Is it generally advisable to reduce the number of observations per individual (and “compensate” by increasing the number of individuals), in order to aid estimation of eMDC models with many alternatives/parameters?
I hope such "generic" questions are OK, and highly appreciate your answer!
Best regards
Tobias H. Rønn
I am working with supermarket scannerdata, and are trying to analyze this using the Apollo eMDC implementation (mainly for “practicing” the use of these models). I have purchasing data for around 700 households over the course of a year, and have picked out 15-20 specific products that I am including as my “inside goods”. I both have a dataset where household purchases are aggregated on a weekly basis, and one where it is aggregated on a monthly basis.
With around 15 products/alternatives, quite often, one of the following messages appear (I do not include my code, as I refer to several different model-specifications):
- Log-likelihood calculation fails at values close to the starting values,
- WARNING: Estimation failed. No covariance matrix to compute,
- Error in if (any(testL == 0)) cat("\nSome observations have zero probability at starting value for eMDCEV model component.").
In my case, this particularly happens when including a budget or when including explanatory variables (e.g. income) in the utility of the outside good, i.e. the estimation process is smooth when excluding these factors, but run into trouble when included. One way to seemingly aid the estimation is to reduce the number of observations per individual (to reduce the chance that the product of many individual observations come too close to zero). Using workInLogs=TRUE in apollo_control or starting values from simpler models generally doesn’t help. This lead me to three questions:
- Is it a general issue that models with a budget / outside good covariates are considerably more difficult to estimate?
- Is there a general “rule-of-thumb” as to the number of alternatives (and/or parameters) that it is feasible to include in an eMDC model? Or is the answer to this simply too case-dependent?
- Is it generally advisable to reduce the number of observations per individual (and “compensate” by increasing the number of individuals), in order to aid estimation of eMDC models with many alternatives/parameters?
I hope such "generic" questions are OK, and highly appreciate your answer!
Best regards
Tobias H. Rønn