Hi list, I'm buying new machines for my lab and would like to optimize the speed with which we process large datasets. At the moment one of our projects generates 5+ GB of nested JSON with each data export. A 10-core iMac Pro takes about 20 minutes to process the raw data into something useful. I'm wondering if there's any general principle with Stata about increasing speed: if I want to improve processing speed (since we need to run this processing step often) should I increase the number of cores? the clock speed of each core? does it matter if I add a high-end GPU to the machine, or does Stata not use GPUs to boost performance? any other items that I should deal with on the hardware end? (memory isn't an issue, our machines all have 64+ GB RAM).
thanks!
Related Posts with Hardware to optimize processing large text files
Help with using expand>2 while replacing values in duplicates generatedHi, I am trying to use the expand command to create duplicates and replacing one of the variables i…
Residuals in a panel data modelHi I am running a regression (using panel data) looking at the effect of income on food consumption …
Adding an interaction term into a model or stratifying data , which method is more preferable to analyse interaction terms?Hi Statlists, Hope this post finds you well. May I know why stratification seems to be less prefera…
Individual Caliper for Variables Nearest-Neighbour Matching (psmatch2)Dear Community, I aim to apply nearest neighbour matching using the mahalanobis option in the the p…
Marginal effects Tobit (mfx vs margins)Hello I'm trying to calculate the marginal effects of a Tobit model using the margins command instea…
Subscribe to:
Post Comments (Atom)
0 Response to Hardware to optimize processing large text files
Post a Comment