No covariance matrix

[evahogervorst]

I am trying to estimate a MNL model with 7 app attributes that have three levels. All the attributes are effect coded with two parameters (for example, cost --> costa & costb). I have also included a 'opt out' option, which was phrased as 'prefer to download no app at all' and is utility function 'app 0', which is 0.

For some reason I get the error 'Estimation failed. No covariance matrix to compute.' I think it has to do with the singular convergence. I check my dataset and tried estimating the model without the 'opt out' option, which lead to the error: 'Singular Hessian, cannot calculate s.e.'. I could not find out what was wrong with my model based on the other topics with similar errors. Below is my code and the output.

Thank you!
----------------------------------------------------------------------------------------------------------------------------------------------------------
rm(list = ls())

### Load Apollo library
library(apollo)

### Initialise code
apollo_initialise()

### Set core controls
apollo_control = list(
modelName ="MNL_3",
modelDescr ="MNL model only attributes_noalt3",
indivID ="ID"
)


#### LOAD DATA
database = read.csv("/Users/evahogervorst/Desktop/Studie/Afstuderen/Data/Werkspace/MNLdata_onlyattributes_noalt3.csv", sep = ',', header=TRUE)
### Vector of parameters, including any that are kept fixed in estimation
apollo_beta=c(BETA_costa = 0,
BETA_costb = 0,
BETA_priva = 0,
BETA_privb = 0,
BETA_coma = 0,
BETA_comb = 0,
BETA_acca = 0,
BETA_accb = 0,
BETA_usea = 0,
BETA_useb = 0,
BETA_persa = 0,
BETA_persb = 0,
BETA_exta = 0,
BETA_extb = 0,
delta_alternative = 0)

### Vector with names (in quotes) of parameters to be kept fixed at their starting value in apollo_beta, use apollo_beta_fixed = c() if none
apollo_fixed = c()

#### GROUP AND VALIDATE INPUTS
apollo_inputs = apollo_validateInputs()


#### DEFINE MODEL AND LIKELIHOOD FUNCTION
apollo_probabilities=function(apollo_beta, apollo_inputs, functionality="estimate"){

### Attach inputs and detach after function exit
apollo_attach(apollo_beta, apollo_inputs)
on.exit(apollo_detach(apollo_beta, apollo_inputs))

### Create list of probabilities P
P = list()

### List of utilities: these must use the same names as in mnl_settings, order is irrelevant
V = list()
V[['app0']] = 0
V[['appA']] = delta_alternative + BETA_costa * cost1a + BETA_costb * cost1b + BETA_priva * priv1a + BETA_privb * priv1b + BETA_coma * com1a + BETA_comb + com1b + BETA_acca * acc1a + BETA_accb * acc1b + BETA_usea * use1a + BETA_useb * use1b + BETA_persa * pers1a + BETA_persb * pers1b + BETA_exta * ext1a + BETA_extb * ext1b
V[['appB']] = delta_alternative + BETA_costa * cost2a + BETA_costb * cost2b + BETA_priva * priv2a + BETA_privb * priv2b + BETA_coma * com2a + BETA_comb + com2b + BETA_acca * acc2a + BETA_accb * acc2b + BETA_usea * use2a + BETA_useb * use2b + BETA_persa * pers2a + BETA_persb * pers2b + BETA_exta * ext2a + BETA_extb * ext2b


### Define settings for MNL model component
mnl_settings = list(
alternatives = c(app0=0, appA=1, appB=2),
avail = list(app0=1, appA=1, appB=1),
choiceVar = choice,
V = V
)

### Compute probabilities using MNL model
P[['model']] = apollo_mnl(mnl_settings, functionality)

### Take product across observation for same individual
P = apollo_panelProd(P, apollo_inputs, functionality)

### Prepare and return outputs of function
P = apollo_prepareProb(P, apollo_inputs, functionality)
return(P)
}


#### MODEL ESTIMATION
model = apollo_estimate(apollo_beta, apollo_fixed, apollo_probabilities, apollo_inputs)

### MODEL VALIDATION

###apollo_sink()
###apollo_outOfSample(apollo_beta,
### apollo_fixed,
### apollo_probabilities,
### apollo_inputs,
### outOfSample_settings = list (validationSize = 0.33))
###apollo_sink()

#### MODEL OUTPUTS
apollo_modelOutput(model,modelOutput_settings=list(printPVal=TRUE))

apollo_saveOutput(model)
-----------------------------------------------------------------------------------------------------------------------------------------------------------------
OUTPUT
-----------------------------------------------------------------------------------------------------------------------------------------------------------------

>
> #### MODEL ESTIMATION
> model = apollo_estimate(apollo_beta, apollo_fixed, apollo_probabilities, apollo_inputs)
Preparing user-defined functions.

Testing likelihood function...

Overview of choices for MNL model component :
app0 appA appB
Times available 1512.00 1512.0 1512.00
Times chosen 838.00 369.0 305.00
Percentage chosen overall 55.42 24.4 20.17
Percentage chosen when available 55.42 24.4 20.17


Pre-processing likelihood function...

Testing influence of parameters
Starting main estimation

BGW using analytic model derivatives supplied by caller...


Iterates will be written to:
/Users/evahogervorst/MNL_3_iterations.csv it nf F RELDF PRELDF RELDX MODEL stppar
0 1 1.786329956e+03
1 4 1.461998007e+03 1.816e-01 1.658e-01 1.00e+00 G 6.64e-01
2 5 1.339591379e+03 8.373e-02 8.482e-02 3.80e-01 G 1.40e-01
3 6 1.328959500e+03 7.937e-03 6.750e-03 3.32e-01 G 1.98e-15
4 7 1.328136826e+03 6.190e-04 6.480e-04 1.25e-01 S -1.88e-15
5 8 1.328087420e+03 3.720e-05 3.532e-05 3.82e-02 S -1.88e-15
6 9 1.328077082e+03 7.784e-06 6.242e-06 1.64e-01 S 3.63e-15
7 10 1.328075634e+03 1.090e-06 9.325e-07 8.49e-02 S -3.63e-15
8 11 1.328075408e+03 1.702e-07 1.228e-07 9.67e-02 S -3.63e-15
9 12 1.328075320e+03 6.573e-08 5.380e-08 3.65e-02 S -3.63e-15
10 13 1.328075304e+03 1.255e-08 9.449e-09 9.55e-02 S -3.63e-15
11 14 1.328075299e+03 3.270e-09 2.757e-09 5.89e-02 S -3.63e-15
12 15 1.328075299e+03 3.544e-10 2.929e-10 9.23e-02 S 0.00e+00
13 16 1.328075299e+03 3.417e-11 3.069e-11 7.18e-05 S 1.66e-14

***** Singular convergence *****

Estimated parameters:
Estimate
BETA_costa -0.821935
BETA_costb -0.113798
BETA_priva 0.532970
BETA_privb -0.082607
BETA_coma 0.073161
BETA_comb -2.428525
BETA_acca 1.009997
BETA_accb -0.129587
BETA_usea 0.316059
BETA_useb -0.177152
BETA_persa -0.008474
BETA_persb 0.221608
BETA_exta 0.152582
BETA_extb -0.035154
delta_alternative 1.034001

Final LL: -1328.0753

WARNING: Estimation failed. No covariance matrix to compute.

Current process will resume in 3 seconds unless interrupted by the user...

Calculating log-likelihood at equal shares (LL(0)) for applicable models...
Calculating log-likelihood at observed shares from estimation data (LL(c)) for applicable
models...
Calculating LL of each model component...
Calculating other model fit measures

Your model was estimated using the BGW algorithm. Please acknowledge this by citing Bunch
et al. (1993) - DOI 10.1145/151271.151279
>
> ### MODEL VALIDATION
>
> ###apollo_sink()
> ###apollo_outOfSample(apollo_beta,
> ### apollo_fixed,
> ### apollo_probabilities,
> ### apollo_inputs,
> ### outOfSample_settings = list (validationSize = 0.33))
> ###apollo_sink()
>
> #### MODEL OUTPUTS
> apollo_modelOutput(model,modelOutput_settings=list(printPVal=TRUE))

[stephanehess]

Hi

can you tell us more about your data, i.e. how many levels there are for each of the attributes

[evahogervorst]

Hi Stephane,

Thank you for your quick reply. My 7 attributes all have 3 categorical levels.
However, I think the mistake was due to a typo that slipped through, I used a '+' where i should have multiplied the beta and the parameter.

Thanks anyway

[evahogervorst]

Unfortunately, even though i fixed the MNL model, I run into a similar error trying to estimate a mixed logit model. I have used the same functions and beta's as specified in the working MNL model. Additionally I have added a sigma_app for both app alternatives to capture their unobserved common factors. I keep getting the "WARNING: Some eigenvalues of the Hessian are positive, indicating convergence to a saddle point! " warning. I tried 250-1000 Halton draws and I tried to specify the problematic beta's with starting values. With 700 draws, I even get the singular hessian warning and all covariance estimates are NA.

I think my model might be overspecified, but I don't know what it could be.

I included the code and the output:
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
### Load Apollo library
library(apollo)

### Initialise code
apollo_initialise()

### Set core controls
apollo_control = list(
modelName ="MXL_Basis_EC",
modelDescr ="MXL_onlyattributes_Errorcomponent",
indivID ="ID",
mixing = TRUE,
nCores = 3
)


#### LOAD DATA
database = read.csv("/Users/evahogervorst/Desktop/Studie/Afstuderen/Data/Werkspace/MNLdata_Attr_SD_Perc_3alt_csv.csv", sep = ',', header=TRUE)

### Vector of parameters, including any that are kept fixed in estimation
apollo_beta=c(BETA_costa = 0,
BETA_costb = 0,
BETA_priva = 0,
BETA_privb = 0,
BETA_coma = 0,
BETA_comb = 0,
BETA_acca = 0,
BETA_accb = 0,
BETA_usea = 0,
BETA_useb = 0,
BETA_persa = 0,
BETA_persb = 0,
BETA_exta = 0,
BETA_extb = 0,
SIGMA_app = 1,
delta_alternative = 0)

### Vector with names (in quotes) of parameters to be kept fixed at their starting value in apollo_beta, use apollo_beta_fixed = c() if none
apollo_fixed = c()

### Set parameters for generating draws
apollo_draws = list(
interDrawsType = "halton",
interNDraws = 500,
interUnifDraws = c(),
interNormDraws = c("draws"),
intraDrawsType = "halton",
intraNDraws = 0,
intraUnifDraws = c(),
intraNormDraws = c()
)

### Create random parameters
apollo_randCoeff = function(apollo_beta, apollo_inputs){
randcoeff = list()

randcoeff[["EC_App_RND"]] = SIGMA_app * draws


return(randcoeff)
}


#### GROUP AND VALIDATE INPUTS
apollo_inputs = apollo_validateInputs()


#### DEFINE MODEL AND LIKELIHOOD FUNCTION
apollo_probabilities=function(apollo_beta, apollo_inputs, functionality="estimate"){

### Attach inputs and detach after function exit
apollo_attach(apollo_beta, apollo_inputs)
on.exit(apollo_detach(apollo_beta, apollo_inputs))

### Create list of probabilities P
P = list()

### List of utilities: these must use the same names as in mnl_settings, order is irrelevant
V = list()
V[['app0']] = 0
V[['appA']] = delta_alternative + BETA_costa * cost1a + BETA_costb * cost1b + BETA_priva * priv1a + BETA_privb * priv1b + BETA_coma * com1a + BETA_comb * com1b + BETA_acca * acc1a + BETA_accb * acc1b + BETA_usea * use1a + BETA_useb * use1b + BETA_persa * pers1a + BETA_persb * pers1b + BETA_exta * ext1a + BETA_extb * ext1b + EC_App_RND
V[['appB']] = delta_alternative + BETA_costa * cost2a + BETA_costb * cost2b + BETA_priva * priv2a + BETA_privb * priv2b + BETA_coma * com2a + BETA_comb * com2b + BETA_acca * acc2a + BETA_accb * acc2b + BETA_usea * use2a + BETA_useb * use2b + BETA_persa * pers2a + BETA_persb * pers2b + BETA_exta * ext2a + BETA_extb * ext2b + EC_App_RND


### Define settings for MNL model component
mnl_settings = list(
alternatives = c(app0=0, appA=1, appB=2),
avail = list(app0=1, appA=1, appB=1),
choiceVar = choice,
V = V
)

### Compute probabilities using MNL model
P[['model']] = apollo_mnl(mnl_settings, functionality)

### Take product across observation for same individual
P = apollo_panelProd(P, apollo_inputs, functionality)

### Average across inter-individual draws
P = apollo_avgInterDraws(P, apollo_inputs, functionality)

### Prepare and return outputs of function
P = apollo_prepareProb(P, apollo_inputs, functionality)
return(P)
}


#### MODEL ESTIMATION
model = apollo_estimate(apollo_beta, apollo_fixed,
apollo_probabilities, apollo_inputs, estimate_settings=list(hessianRoutine="maxLik"))


#### MODEL OUTPUTS
apollo_modelOutput(model,modelOutput_settings=list(printPVal=TRUE))

apollo_saveOutput(model)
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Preparing user-defined functions.

Testing likelihood function...

Overview of choices for MNL model component :
app0 appA appB
Times available 1512.00 1512.0 1512.00
Times chosen 838.00 369.0 305.00
Percentage chosen overall 55.42 24.4 20.17
Percentage chosen when available 55.42 24.4 20.17


Pre-processing likelihood function...
Creating cluster...
Preparing workers for multithreading...

Testing influence of parameters
Starting main estimation

BGW using analytic model derivatives supplied by caller...


Iterates will be written to:
/Users/evahogervorst/MXL_Basis_EC_iterations.csv it nf F RELDF PRELDF RELDX MODEL stppar
0 1 1.661101780e+03
1 2 1.538657661e+03 7.371e-02 1.146e-02 1.00e+00 G 1.90e+01
2 3 1.405520394e+03 8.653e-02 4.945e-02 1.38e-01 G 0.00e+00
3 4 1.284368699e+03 8.620e-02 5.112e-02 1.40e-01 G 0.00e+00
4 5 1.115197658e+03 1.317e-01 1.166e-01 2.59e-01 S 0.00e+00
5 7 1.075305376e+03 3.577e-02 3.258e-02 1.42e-01 S-G 0.00e+00
6 8 1.055921773e+03 1.803e-02 2.440e-02 1.38e-01 G 0.00e+00
7 10 1.049105109e+03 6.456e-03 7.294e-03 3.36e-02 S 3.01e+00
8 11 1.037521636e+03 1.104e-02 1.455e-02 1.19e-01 G 1.03e-01
9 13 1.035726124e+03 1.731e-03 2.993e-03 5.01e-02 S-G 0.00e+00
10 15 1.035106739e+03 5.980e-04 1.990e-03 5.87e-02 G-S 0.00e+00
11 16 1.034774801e+03 3.207e-04 8.135e-04 3.45e-02 G 0.00e+00
12 17 1.034379064e+03 3.824e-04 3.816e-04 1.86e-02 S 0.00e+00
13 18 1.034369060e+03 9.672e-06 1.204e-05 3.06e-03 S 0.00e+00
14 19 1.034366732e+03 2.251e-06 1.748e-06 1.32e-03 S 0.00e+00
15 20 1.034366360e+03 3.596e-07 3.511e-07 4.98e-04 G 0.00e+00
16 21 1.034366268e+03 8.921e-08 1.859e-07 5.02e-04 G 0.00e+00
17 23 1.034366154e+03 1.102e-07 8.861e-08 3.82e-04 G-S 0.00e+00
18 24 1.034366137e+03 1.564e-08 1.434e-08 1.60e-04 S 0.00e+00
19 25 1.034366135e+03 2.049e-09 1.963e-09 6.36e-05 S 0.00e+00
20 26 1.034366135e+03 3.154e-10 2.774e-10 1.63e-05 S 0.00e+00
21 27 1.034366135e+03 1.286e-11 2.282e-11 6.30e-06 S 0.00e+00

***** Relative function convergence *****
Additional convergence test using scaled estimation. Parameters will be scaled by their current estimates and additional iterations will be performed.

BGW using analytic model derivatives supplied by caller...


Iterates will be appended to:
/Users/evahogervorst/MXL_Basis_EC_iterations.csv it nf F RELDF PRELDF RELDX MODEL stppar
0 1 1.034366135e+03
1 2 1.034366135e+03 7.230e-13 9.128e-12 1.55e-04 G 0.00e+00

***** Relative function convergence *****

Estimated parameters with approximate standard errors from BHHH matrix:
Estimate BHHH se BHH t-ratio (0)
BETA_costa -0.912688 0.11982 -7.61699
BETA_costb -0.345266 0.10367 -3.33050
BETA_priva 0.566540 0.08289 6.83492
BETA_privb 0.176295 0.10422 1.69154
BETA_coma 0.335449 0.08920 3.76082
BETA_comb -0.043035 0.09964 -0.43190
BETA_acca 0.920458 0.07239 12.71521
BETA_accb 0.129110 0.10451 1.23535
BETA_usea 0.072949 0.09902 0.73671
BETA_useb -0.012626 0.11086 -0.11390
BETA_persa 0.238392 0.08756 2.72251
BETA_persb 0.002988 0.09192 0.03250
BETA_exta 0.085886 0.09150 0.93862
BETA_extb 0.129737 0.11147 1.16389
SIGMA_app 2.296600 0.20865 11.00675
delta_alternative -1.555888 0.20308 -7.66136

Final LL: -1034.3661

Calculating log-likelihood at equal shares (LL(0)) for applicable models...
Calculating log-likelihood at observed shares from estimation data (LL(c)) for applicable models...
Calculating LL of each model component...
Calculating other model fit measures
Computing covariance matrix using numerical methods (maxLik). This may take a while, no progress bar displayed.
WARNING: Some eigenvalues of the Hessian are positive, indicating convergence to a saddle point!
Computing score matrix...

Your model was estimated using the BGW algorithm. Please acknowledge this by citing Bunch et al. (1993) - DOI 10.1145/151271.151279
Warning message:
In sqrt(diag(varcov)) : NaNs produced
>
>
> #### MODEL OUTPUTS
> apollo_modelOutput(model,modelOutput_settings=list(printPVal=TRUE))
Model run by evahogervorst using Apollo 0.3.1 on R 4.3.2 for Darwin.
Please acknowledge the use of Apollo by citing Hess & Palma (2019)
DOI 10.1016/j.jocm.2019.100170
www.ApolloChoiceModelling.com

Model name : MXL_Basis_EC
Model description : MXL_onlyattributes_Errorcomponent
Model run at : 2024-04-04 20:44:51.613829
Estimation method : bgw
Model diagnosis : Relative function convergence
Optimisation diagnosis : Saddle point found
hessian properties : Some eigenvalues are positive and others negative
maximum eigenvalue : 137.716432
reciprocal of condition number : not calculated (Hessian is not negative definite)
Number of individuals : 168
Number of rows in database : 1512
Number of modelled outcomes : 1512

Number of cores used : 3
Number of inter-individual draws : 500 (halton)

LL(start) : -1661.1
LL at equal shares, LL(0) : -1661.1
LL at observed shares, LL(C) : -1503.26
LL(final) : -1034.37
Rho-squared vs equal shares : 0.3773
Adj.Rho-squared vs equal shares : 0.3677
Rho-squared vs observed shares : 0.3119
Adj.Rho-squared vs observed shares : 0.3026
AIC : 2100.73
BIC : 2185.87

Estimated parameters : 16
Time taken (hh:mm:ss) : 00:02:0.85
pre-estimation : 00:00:23.75
estimation : 00:00:21.8
initial estimation : 00:00:19.16
estimation after rescaling : 00:00:2.64
post-estimation : 00:01:15.3
Iterations : 22
initial estimation : 21
estimation after rescaling : 1

Unconstrained optimisation.

Estimates:
Estimate s.e. t.rat.(0) p(1-sided) Rob.s.e. Rob.t.rat.(0) p(1-sided)
BETA_costa -0.912688 0.11732 -7.7794 3.664e-15 0.25183 -3.62422 1.4492e-04
BETA_costb -0.345266 0.14419 -2.3945 0.008322 0.44805 -0.77060 0.22047
BETA_priva 0.566540 0.07636 7.4191 5.895e-14 0.09309 6.08614 5.783e-10
BETA_privb 0.176295 0.07777 2.2670 0.011696 0.15450 1.14105 0.12692
BETA_coma 0.335449 0.11851 2.8305 0.002324 0.36223 0.92606 0.17721
BETA_comb -0.043035 0.27814 -0.1547 0.438519 1.07180 -0.04015 0.48399
BETA_acca 0.920458 0.07691 11.9673 0.000000 0.16815 5.47409 2.199e-08
BETA_accb 0.129110 0.06677 1.9337 0.026574 0.08124 1.58927 0.05600
BETA_usea 0.072949 0.17210 0.4239 0.335826 0.61847 0.11795 0.45305
BETA_useb -0.012626 NaN NaN NaN 0.22672 -0.05569 0.47779
BETA_persa 0.238392 0.09890 2.4104 0.007967 0.29431 0.81000 0.20897
BETA_persb 0.002988 NaN NaN NaN 0.14310 0.02088 0.49167
BETA_exta 0.085886 0.14974 0.5736 0.283127 0.53257 0.16127 0.43594
BETA_extb 0.129737 0.12486 1.0391 0.149386 0.38292 0.33881 0.36738
SIGMA_app 2.296600 0.19868 11.5593 0.000000 0.20858 11.01053 0.00000
delta_alternative -1.555888 0.13001 -11.9673 0.000000 0.31911 -4.87575 5.420e-07

>
> apollo_saveOutput(model)

[evahogervorst]

another addition:

I got it to work by first running a model with only the linear components (the beta's and parameters with a 'a', for example: BETA_costa in my model) and then taking those estimated betas for defining the starting values of the linear components (BETA_costa in my model) in the model with also the non-linear components (the beta's and parameters with a 'b', for example: BETA_costb in my model). However this only works up until I change the starting values. If I use rm(list = ls()) at the start everytime I run the model, it works again. If I use that function, it works even if I change the starting values.

Is this a sufficient method if I have to use rm(list = ls()) for it to work?

[stephanehess]

Hi

it is not clear what you mean by linear and non-linear here, unless you have made non-linear transforms in the data.

Can you show the results?

In terms of rm(list = ls()), this clears the memory, which you should do before every model estimation

Stephane