Memory Leak?

[jjanmaat]

I have experienced some system problems estimating a hybrid choice mixed logit model. The code is long, so not including, and have not had the time to try and reproduce it with a code segment and simulated data.

I am running R through Rstudio on a Ubunto 22.04.5 LTS operating system, on a Dell Precision 3660 with an Intel i7-13700K processor (16 cores, 24 threads).

I have two versions of a model I am estimating. In one, I have a latent variable with a normal distribution, with nine indictors (7 point Likert, fit assuming normal) and 9 explanatory variables (dummy indicators). In the second, I add four random parameters.

I first ran the models with 100 draws. Worked. I increased the draws to 1000. Estimation stalled at the covariance matrix calculation (hours, no progress). I dropped it to 300, and it worked. I increased it to 500 and it again stalled. In this case, the stalling crashed Chrome and blanked the Rstudio window. In one attempted run, it locked up the entire machine. I am setting the number of cores to be `r trunc(0.8*availableCores()) '.

Not sure if this is an Apollo problem or an R problem.

John.

[stephanehess]

Hi

the covariance matrix calculation with hybrid choice models can indeed take a long time, especially with a large number of draws. They often take much longer than the estimation itself. It's not a memory issue generally, but simply computational cost. If you have a look at the model with 100 draws, you should see that in the run times for the different steps

Stephane

[jjanmaat]

Hello Stephane,

Thanks for the reply. The model ran successfully on a Windows machine, using 500 draws. On my Linux box it didn't. This suggests it might be an issue with the R compiled for my version of Linux.

John.

[jjanmaat]

I have narrowed the problem down somewhat. I was loading the packages 'parallel' and 'future' as well as apollo. I was also running the apollo code inside an Rmarkdown chunk. Based on a bit of internet snooping, there seems to be some problems with Rstudio on Linux with multithreading. I suspect that Rstudio is somehow interfering with cleaning up threads when the process is stopped or interrupted, such that on a subsequent execution, the program tries to initiate more threads than are available, which halts various other threads. Those other threads need not be related to Rstudio, and as such the entire system hangs.

I no longer load any parallel packages, in case there is a conflict. I can successfully run the estimations either in a separate console shell (using knitr::purl() to extract the R code from the Rmd file) or as a background job.