I should specify, I am new to stata, so if i misuse terminology, please forgive me.
I have several big .dta files in a folder. I cant have all of them in memory together so I'm going to loop over them. What I hope to do is avoid unnecessary i/o.
Is there any way to save (in memory only) the output from each file in a frame, append those frames and then write/save this final frame only? I an unable to append a frame that is only in memory.
Currently I have to save the output of each file, so later when I load these files again, I would have to merge them.
Code:
clear all local files : dir "path/to/the/folder" files "*.dta" cd "path/to/the/folder" display `"`files'"' foreach file of local { use "`file'", clear collapase (sum) variable1, by (variable2) save `"new_`file'.dta"', replace }
Code:
clear all local files : dir "path/to/the/folder" files "*.dta" cd "path/to/the/folder" display `"`files'"' frame create `final_frame' //everything will be appended to this foreach file of local { use "`file'", clear collapase (sum) variable1, by (variable2) frame rename default, to_append frame use final_frame frame final_frame : append using `to_append' } save "final_frame.dta", replace
0 Response to loop over files in a folder to do an operation and save the combined result as dataset
Post a Comment