I have 8 folders and in every forlder I have 35042 .csv files.

I would like to import and append all files in one .dta file.

I need only a few observations by file.

My code for per folder is:

Code:
cd E:\Ercot\XYZ
clear
tempfile building
save `building', emptyok

local myfilelist : dir . files"*.csv"
foreach file of local myfilelist {
    drop _all
    import delimited `"`file'"'
    keep if settlementpointtype =="LZ"
    local outfile = subinstr("`file'",".csv","",.)
    append using "`building'"
    save `building', replace
}
Unfortunately, I get the following error:

too many filenames

So I use it:

Code:
cd E:\Ercot\XYZ
! dir *.csv /a-d /b >filelist.txt

file open myfile using filelist.txt, read
file read myfile line
import delimited `line', clear

keep if settlementpointtype =="LZ"
*save `line'.dta, replace
save master_data.dta, replace

drop _all

file read myfile line
while r(eof)==0 {
    import delimited `line'
    keep if settlementpointtype =="LZ"
*    save `line'.dta, replace
    append using master_data.dta
    save master_data.dta, replace
    drop _all
    file read myfile line
}
How I can do it faster and all folders at the same time?

Thanks,

Sebastián Kruk.