Dear statalist,

I have a big file that overwhelms my memory when I try to run my script.
Therefore, I wish to break the operation down into chunks, using postfile to write the results of each loop iteration into a new .dta file.
This way, I hope to be able to check in on my progress while the loop is running, by opening the resulting file from time to time.
I mostly want to generate and modify variables according to simple decision rules, and then visualise some of the results.

However, as I tried making a minimal example using a smaller subfile, I noticed the following error: the resulting postfile only contains the results of the first loop iteration.
The loop itself works, and generates exactly what I want to have generated.
Setting -trace- I found that the loop indeed only processed the "post" command for the first iteration, and from then on only reads it without doing anything.
If I put the -post- command into the outer loop, it repeats one observation, but does surprisingly *not* overwrite each iteration until the last one, but again takes the result of the first iteration.

The results of running the code below are:
iteration, category, year, mean_difference \\
1, $Some_CategoryA$, 1930, -0.1

The results of putting the post-command into the outer loop are:
iteration, category, year, mean_difference \\
1, $Some_CategoryA$, 1930, -0.1\\
2, $Some_CategoryA$, 1930, -0.1\\
....
....
....
47 $Some_CategoryA$, 1930, -0.1\\

See my code below. For confidentiality reasons, I am sorry cannot share the actual code and data with you. Thank you in advance!!

gen x = " "
gen mean_difference = 888

capture postutil clear
tempname buffer
postfile `buffer' iteration str33 category year mean_difference using "differenceA1.dta", replace

set trace on

egen group = group(category)
su group, meanonly
forval i = 1/`r(max)' {
if group == `i'{
replace x = "yes" if $some_rule$
keep if x == "yes"
egen m_y1 = mean(y1), by (category year)
egen m_y2= mean(y2), by (category year)
replace mean_difference = m_y1 - m_y2
post `buffer' (`i') (category) (year) (mean_difference)
drop m_y1 m_y2

}

}

postclose `buffer'