[Please note: I've looked at a few other forums with similar problems, but the solutions don't seem to solve my problem]

I'm a PhD student working with very large data sets (healthcare data), ranging from 20-50GB .dta files.

I need to clean my data and as such I am attempting to remove patients in the master file (20GB) that do not appear in my user file (50KB) based on a common variable (drug ID) via the -merge- function. I have 8 different user files to merge separately with the master file (so I can produce 8 different cohort groups). I have successfully done this with 3 user files, but upon attempting with the 4th user file I receive the following error message:

op. sys. refuses to provide memory r(909);
Stata's data-storage memory manager has already allocated 90999619608 bytes and it just attempted to allocate
another 32m bytes. The operating system said no. Perhaps you are running another memory-consuming task and
the command will work later when the task completes. Perhaps you are on a multiuser system that is
especially busy and the command will work later when activity quiets down. Perhaps a system administrator
has put a limit on what you can allocate; see help memory. Or perhaps that's all the memory your computer
can allocate to Stata.
r(909);
  • I am not running any other task
  • I am on an unlimited user network
  • -query memory- and max_memory is (.) which means all the memory the operating system is willing to supply.
Other details:
  • I am using Microsoft Windows 10 Enterprise with a 64-bit operating system and 32GB RAM (31.8GB usable). My local disk has 351GB free
  • I am using STATA SE 16 (64 bit)
  • The master file (20GB is stored and opened on my laptop from an external hard drive)
  • Attempted to compress the master file (0 bytes saved)
I am aware this is an operating system problem and not STATA specifically. However, my question is has anyone else had this problem in their use of STATA and overcome it? Is there a way to get around this error and allocate more memory? I'm not an IT specialist, but it looks like I still have enough RAM left?
Any help or advice would be much appreciated.


Thank you.