Input validation code works on other PC, not mine

[Hyewon]

Hello professor,
I am now utilizing "example_20" which is Latent class model.
the version of apollo is the lastest one, 0.2.2 and R version also the latest.

However, on the step of "group and validate Inputs", the code makes the error as follows;
------
Several observations per individual detected based on the value of ID. Setting panelData in apollo_control set
to TRUE.
All checks on apollo_control completed.
Error in apollo_validateData(database, apollo_control, silent = silent) :
All rows for the same individual should be next to each other in the data!
-------

I tried the code on other PCs, but some PC, it works and another it didn't.
The difference between executed one and other is, the code executed PC, the R and apollo version is older one.

I cannot find the reason why, and never seen this problem before.
The dataset and code was same between two PCs, so maybe not the problem of dataset or PC.

Here, I will also attach the code that I used.
I also made bold where it makes the error.

-----

# ################################################################# #
#### LOAD LIBRARY AND DEFINE CORE SETTINGS ####
# ################################################################# #

### Clear memory
rm(list = ls())

### Load Apollo library
library(apollo)

### Initialise code
apollo_initialise()

apollo_control = list(
modelName ="Latent_TSP",
modelDescr ="LC model with class allocation model on iwami mode choice data",
indivID ="ID",
nCores = 3
)

# ################################################################# #
#### LOAD DATA AND APPLY ANY TRANSFORMATIONS ####
# ################################################################# #

database = read.csv("iwami_final_1224.csv",header=TRUE)
colnames(database)[1] <- "ID" #change variable name error #change variable name error
# data cleaning
database <- database[database$area!=99,]
database <- database[database$gender!=99,]
database <- database[database$age!=99,]
database <- database[database$job!=99,]
database <- database[database$choice!=99,]

database$job1 <-ifelse(database$job==1,1,0) #farmer-non
database$job2 <-ifelse(database$job==6|database$job==12,1,0) #job-non


# ################################################################# #
#### DEFINE MODEL PARAMETERS ####
# ################################################################# #

### Vector of parameters, including any that are kept fixed in estimation
apollo_beta = c(asc_1 = 0,
asc_2 = 0,
beta_wt_a = 0,
beta_wt_b = 0,
beta_iv_a = 0,
beta_iv_b = 0,
beta_md_a = 0,
beta_md_b = 0,
beta_bw_a = 0,
beta_bw_b = 0,
beta_tc_a = 0,
beta_tc_b = 0,
delta_a = 0,
gamma_age_a = 0,
gamma_jobless_a = 0,
delta_b = 0,
gamma_age_b = 0,
gamma_jobless_b = 0)

### Vector with names (in quotes) of parameters to be kept fixed at their starting value in apollo_beta, use apollo_beta_fixed = c() if none
apollo_fixed = c("asc_2", "delta_b", "gamma_age_b","gamma_jobless_b")

# ################################################################# #
#### DEFINE LATENT CLASS COMPONENTS ####
# ################################################################# #

apollo_lcPars=function(apollo_beta, apollo_inputs){
lcpars = list()
lcpars[["beta_wt"]] = list(beta_wt_a, beta_wt_b)
lcpars[["beta_iv"]] = list(beta_iv_a, beta_iv_b)
lcpars[["beta_md"]] = list(beta_md_a, beta_md_b)
lcpars[["beta_bw"]] = list(beta_bw_a, beta_bw_b)
lcpars[["beta_tc"]] = list(beta_tc_a, beta_tc_b)

V=list()
V[["class_a"]] = delta_a + gamma_age_a*age + gamma_jobless_a*job2
V[["class_b"]] = delta_b + gamma_age_b*age + gamma_jobless_b*job2

mnl_settings = list(
alternatives = c(class_a=1, class_b=2),
avail = 1, # all of the alternatives are available for every choice observation in the data
choiceVar = NA,
V = V
)
lcpars[["pi_values"]] = apollo_mnl(mnl_settings, functionality="raw")

lcpars[["pi_values"]] = apollo_firstRow(lcpars[["pi_values"]], apollo_inputs)

return(lcpars)
}

# ################################################################# #
#### GROUP AND VALIDATE INPUTS ####
# ################################################################# #

apollo_inputs = apollo_validateInputs()

# ################################################################# #
#### DEFINE MODEL AND LIKELIHOOD FUNCTION ####
# ################################################################# #

apollo_probabilities=function(apollo_beta, apollo_inputs, functionality="estimate"){

### Attach inputs and detach after function exit
apollo_attach(apollo_beta, apollo_inputs)
on.exit(apollo_detach(apollo_beta, apollo_inputs))

### Create list of probabilities P
P = list()

### Define settings for MNL model component that are generic across classes
mnl_settings = list(
alternatives = c(alt1=1, alt2=2),
avail = list(alt1=1, alt2=1),
choiceVar = choice
)

### Loop over classes
s=1
while(s<=2){

### Compute class-specific utilities
V=list()
V[['alt1']] = asc_1 + beta_wt[[s]]*WT1 + beta_iv[[s]]*IV1 + beta_bw[[s]]*BW1 + beta_hw[[s]]*dt1 + beta_tc[[s]]*TC1
V[['alt2']] = asc_2 + beta_wt[[s]]*WT2 + beta_iv[[s]]*IV2 + beta_bw[[s]]*BW2 + beta_hw[[s]]*dt2 + beta_tc[[s]]*TC2

mnl_settings$V = V
mnl_settings$componentName = paste0("Class_",s)

### Compute within-class choice probabilities using MNL model
P[[paste0("Class_",s)]] = apollo_mnl(mnl_settings, functionality)

### Take product across observation for same individual
P[[paste0("Class_",s)]] = apollo_panelProd(P[[paste0("Class_",s)]], apollo_inputs ,functionality)

s=s+1}

### Compute latent class model probabilities
lc_settings = list(inClassProb = P, classProb=pi_values)
P[["model"]] = apollo_lc(lc_settings, apollo_inputs, functionality)

### Prepare and return outputs of function
P = apollo_prepareProb(P, apollo_inputs, functionality)
return(P)
}


# ################################################################# #
#### MODEL ESTIMATION ####
# ################################################################# #

#apollo_beta=apollo_searchStart(apollo_beta, apollo_fixed,apollo_probabilities, apollo_inputs)
#apollo_outOfSample(apollo_beta, apollo_fixed,apollo_probabilities, apollo_inputs)

### Estimate model
model = apollo_estimate(apollo_beta, apollo_fixed,
apollo_probabilities, apollo_inputs,
estimate_settings=list(writeIter=FALSE))

### Show output in screen
apollo_modelOutput(model)

### Save output to file(s)
apollo_saveOutput(model)


-------



Thank you so much.

[stephanehess]

Hi

older versions of Apollo used to automatically sort the data by ID, but we realised that this could cause problems with some of the post-processing functions. That's why newer versions require the user to provide data in which all observations for the same person are together in the data. The data does not need to be sorted by ID, as long as all rows for the same person are together

Stephane

[bigdreamwq]

Hello professor.
I am glad to see this reply because I have encountered the same problem. The Apollo version above 0.2 would report error.
Error in apollo_validateData(database, apollo_control, silent = silent) :
All rows for the same individual should be next to each other in the data!

My code is rather simple
apollo_initialise()

apollo_control = list(
modelName ="MNL",
modelDescr ="MNL model Exercise 1",
indivID ="ID"）

database = X1

apollo_beta=c(BETA_TC = 0,
House = 0,
Meal = 0,
Healthcare = 0,
Recreation = 0,
Price =0)

apollo_fixed = c()

apollo_inputs = apollo_validateInputs()

The confusing part is that I did provide data in which all observations for the same person are together in the data.
I provide an attachment to show how I coded my data . Is there something wrong with my data format??

Thank you very much

[dpalma]

Hi,

If there are no privacy limitations, could you share the data set and the model script so we can look at the issue in more detail?

Cheers
David

[LauraThoWal]

I am also getting the same error. I have ensured all the observations for the same individual are contiguous in the dataset, but am still getting:

Code: Select all

> apollo_choiceAnalysis(choiceAnalysis_settings, apollo_control, database)
Error in apollo_validateData(database, apollo_control, silent = TRUE) : 
  All rows for the same individual should be next to each other in the data!

Has an explanation been found? (Happy to share data/model)

[LauraThoWal]

I am having the same problem. Has a solution/explanation been found?

[theycallmemylinh]

Hi there,

I am running into the same error as the members above

Code: Select all

(i.e. Error in apollo_validateData(database, apollo_control, silent = silent) :  All rows for the same individual should be next to each other in the data!) 

I believe my data are in the correct wide format with all the rows next to each other. My data can be viewed here on the Open Science Framework: https://osf.io/jfz5h?show=view&view_only=

I tried with both an rds and csv file just to see but had the same results.

Thanks for any help you can provide.

My-Linh

Code: Select all

library(apollo)
library(here)
library(readr)

# ################################################################# #
#### LOAD LIBRARY AND DEFINE CORE SETTINGS                       ####
# ################################################################# #
rm(list = ls())

### Initialise code
apollo_initialise()

### Set core controls
apollo_control<-list(
  modelName= "dce_model1",
  modelDescr="MNL model with sociodemographics on SP data",
  indivID= "ID"
)

# ################################################################# #
#### LOAD DATA AND APPLY ANY TRANSFORMATIONS                     ####
# ################################################################# #

database<-read_rds(here("01_data","02_processed", "00_data_processed.rds"))

# ################################################################# #
#### DEFINE MODEL PARAMETERS                                     ####
# ################################################################# #

### Vector of parameters, including any that are kept fixed in estimation
apollo_beta = c(
  asc = 0,
  b_60min=0,
  b_90min=0,
  b_voucher=0,
  b_donate=0,
  b_300=0,
  b_500=0,
  b_neg=0)

apollo_fixed<-c()

# ################################################################# #
#### GROUP AND VALIDATE INPUTS                                   ####
# ################################################################# #

apollo_inputs = apollo_validateInputs()

[dpalma]

Hi Everyone,

First, I would recommend updating to Apollo v0.2.5. This version is not yet on CRAN, but you can download it from http://www.apollochoicemodelling.com/code.html . Note that you need to download the appropriate file for your operating system.

If you are using RStudio, you install this file simply by selecting Tool > Install Packages... in the menu bar, and then selecting Install from: Package Archive File, and select the file you downloaded.

If you are not using RStudio, then you can run the following code:

Code: Select all

install.packages("C:/path/to/file/apollo_0.2.5.zip", repos=NULL)

Where you need to change "C:/path/to/file/apollo_0.2.5.zip" to the appropriate path to the downloaded file.

Please let us know if that doesn't solve the issue.

Cheers
David

[LauraThoWal]

Thanks for the suggestion David. I just downloaded the v0.2.5, but unfortunately still get the same error:

Code: Select all

Error in apollo_validateData(database, apollo_control, silent = TRUE) : 
  All rows for the same individual should be next to each other in the data!

[dpalma]

TLDR: Don't store your data into a tibble, use data.frames instead.

Hi Laura,

Thanks for sending me your model script and database.

The main issue seems to be you are using tibbles instead of data.frames. tibble is a replacement for data.frame used within the tidyverse framework. Functions such as read_csv return a tibble instead of a data.frame. In time, we'll try to make Apollo compatible with the tidyverse, but for now, you should make sure that your database is a data.frame and not a tibble. To do this, after loading your data (and before making any Apollo call) you should do:

Code: Select all

database <- as.data.frame(database)

Looking at your script, I also realised that your definition of V is not appropriate. You write, for example:

Code: Select all

V[['farm']] = GameID + Country + Treatments + Round + Age + Gender + Education + Farmer + 
  CropDam + meanequity + meantrust + meanfam

As if V was an R formula (like the way you write linear regressions using the function lm). But you have to think of the definition of each element inside V like a function, so you need to explicitly write the multiplications, and there is no automatic handling of levels for non-numeric variables (factors). So you would need to create and explicitly write dummy variables for the different levels of a factor (remember you need to exclude one level to use it as a base), for example:

Code: Select all

V[['farm']] = b_GameID_2ASH*(GameID=="2ASH") + b_GameID_3ASH*(GameID=="3ASH") + ... + b_Country_Scot*(Country=="Scotland") + b_Treatments_Subsidy*(Treatments=="Subsidy") + b_Treatments_Deterrents*(Treatments=="Deterrents") + ...

Pleas let me know if this solves the issue.

Cheers
David