Database for Large number of 1kB data chunks (MySQL?)

Posted by The Unknown on Stack Overflow See other posts from Stack Overflow or by The Unknown
Published on 2009-04-29T07:55:35Z Indexed on 2010/05/04 12:48 UTC
Read the original article Hit count: 198

Filed under:
|
|

I have a very large dataset, each item in the dataset being roughly 1kB in size. The data needs to be queried rapidly by many applications distributed over a network. The dataset has more than a million items (so 500 million+ 1kB data chunks).

What would be the best method to storing this dataset (need to allow adding more items, and reading them rapidly, but never modifying already added data)? Would using a MySQL DB using the binary blob format be appropriate?

Or should each of these be stored as files on a file system?

edit: the number is 1 million items now, but needs to be able to scale to well over 500 million items easily.

© Stack Overflow or respective owner

Related posts about database

Related posts about Performance