Hi

I'm trying to loop through around 6000 .csv files of smart meter data to run the following operations and save as .dta files.

Code:
u AMI_ID_data_no_installs-Dec-Mar.dta, clear
levelsof id, local(levels) clean
foreach X of local levels {
    capture import delimited Meter_`X'.csv, stringcols(1) clear
    gen date = daily(substr( intervalstart , 1, 11), "DMY")
    format date %td
    gen double time = clock(substr( intervalstart , 12, 12), "hms")
    format time %tcHH:MM:SS
    drop intervalstart intervalfinish
    rename kwhexport draw
    rename kwhimport feed
    order id date time draw feed, first
    keep if inrange(date, mdy(12,14,2018),mdy(3,14,2019)) | inrange(date, mdy(12,14,2019),mdy(3,14,2020))
        save Meter_`X'.dta, replace
        }
This code worked with a small subset of the files, but each time I've tried with the full set, I get this error after the loop has completed about the 70th file (the same file every time):
variable date already defined
r(110);

The .dta file used to generate the levels has more than 6000 observations (no duplicates) of the variable id (e.g. 0023fcfbc1dfa9cb5fe4), each of which has a matching .csv file named "Meter_[id]" (e.g. Meter_0023fcfbc1dfa9cb5fe4) - no duplicates. There are more .csv files than id variable observations.

I have sorted both the .dta file data by the id variable and the .csv file names in ascending order in case that makes a difference.

The .csv files do not have a date variable.

Unfortunately, there's way too much data to upload so much data so you can run the loop through and test solutions.

I'm completely flummoxed. Do you have any suggestions?

Thanks

Kevin