How to manipulate huge amounts of data

Posted by Alejandro on Stack Overflow See other posts from Stack Overflow or by Alejandro
Published on 2010-04-13T13:44:47Z Indexed on 2010/04/13 13:52 UTC
Read the original article Hit count: 300

Filed under:

memory

|

memory-usage

|

programming-paradigms

|

programming-languages

|

operating-systems

Hi there!

I'm having the following problem. I need to store huge amounts of information (~32 GB) and be able to manipulate it as fast as possible. I'm wondering what's the best way to do it (combinations of programming language + OS + whatever you think its important).

The structure of the information I'm using is a 4D array (NxNxNxN) of double-precission floats (8 bytes). Right now my solution is to slice the 4D array into 2D arrays and store them in separate files in the HDD of my computer. This is really slow and the manipulation of the data is unbearable, so this is no solution at all!

I'm thinking on moving into a Supercomputing facility in my country and store all the information in the RAM, but I'm not sure how to implement an application to take advantage of it (I'm not a professional programmer, so any book/reference will help me a lot).

An alternative solution I'm thinking on is to buy a dedicated server with lots of RAM, but I don't know for sure if that will solve the problem. So right now my ignorance doesn't let me choose the best way to proceed.

What would you do if you were in this situation? I'm open to any idea.

Thanks in advance!

© Stack Overflow or respective owner

Related posts about memory

SQL Server and Hyper-V Dynamic Memory Part 2

as seen on SQL Blog - Search for 'SQL Blog'
Part 1 of this series was an introduction and overview of Hyper-V Dynamic Memory. This part looks at SQL Server memory management and how the SQL engine responds to changing OS memory conditions. Part 2: SQL Server Memory Management As with any Windows process, sqlserver.exe has a virtual… >>> More
SQL Server and Hyper-V Dynamic Memory - Part 1

as seen on SQL Blog - Search for 'SQL Blog'
SQL and Dynamic Memory Blog Post Series Hyper-V Dynamic Memory is a new feature in Windows Server 2008 R2 SP1 that allows the memory assigned to guest virtual machines to vary according to demand. Using this feature with SQL Server is supported, but how well does it work in… >>> More
SQL Server and Hyper-V Dynamic Memory - Part 1

as seen on SQL Blog - Search for 'SQL Blog'
SQL and Dynamic Memory Blog Post Series Hyper-V Dynamic Memory is a new feature in Windows Server 2008 R2 SP1 that allows the memory assigned to guest virtual machines to vary according to demand. Using this feature with SQL Server is supported, but how well does it work in… >>> More
Python Memory leak - Solved, but still puzzled

as seen on Stack Overflow - Search for 'Stack Overflow'
Dear everyone, I have successfully debugged my own memory leak problems. However, I have noticed some very strange occurence. for fid, fv in freqDic.iteritems(): outf.write(fid+"\t") #ID for i, term in enumerate(domain): #Vector tfidf = self.tf(term… >>> More
SQL Server and Hyper-V Dynamic Memory Part 3

as seen on SQL Blog - Search for 'SQL Blog'
In parts 1 and 2 of this series we looked at the basics of Hyper-V Dynamic Memory and SQL Server memory management. In this part Serdar looks at configuration guidelines for SQL Server memory management. Part 3: Configuration Guidelines for Hyper-V Dynamic Memory and SQL Server Now that… >>> More

Related posts about memory-usage

SQL Server 2005 high memory usage and performance problems

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi there guys. I have this ASP.NET/SQLServer2005 website running on a production server (Win2003, QuadCore, 4GB). The site runs smoothly normally, but after 2-3 weeks I notice a slow performance on the site (especifically in one particular page). Also I notice that the SQL Server process is using… >>> More
Coldfusion on VPS, how much JVM heap memory?

as seen on Server Fault - Search for 'Server Fault'
Recently I got a VPS server and I'm running Coldfusion, the website was running fine until it got more and more traffic and I started to encounter 'OutOfMemory' exceptions. I thought simply to rise the memory of the VPS server, but this didn't help. After doing some Google searches I found a setting… >>> More
Design approach, string table data, variables, stl memory usage

as seen on Stack Overflow - Search for 'Stack Overflow'
I have an old structure class like this: typedef vector<vector<string>> VARTYPE_T; which works as a single variable. This variable can hold from one value over a list to data like a table. Most values are long,double, string or double [3] for coordinates (x,y,z). I just convert them as… >>> More
Normal memory usage in Rails

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm wondering how much memory usage is normal for a ruby process in a rails application? I really need something to benchmark against. In my dev environment WEBrick a single ruby process uses about 61mb to handle 10 simultaneous requests going non stop. In my prod environment Apache2+Passenger starts… >>> More
C++ memory usage

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi. I have pointer str: char * str = new char[10]; I'm using this memory for store data. How can I allocate more bytes for str and don't lost old data from str? >>> More