Database for Large number of 1kB data chunks (MySQL?)
Posted
by The Unknown
on Stack Overflow
See other posts from Stack Overflow
or by The Unknown
Published on 2009-04-29T07:55:35Z
Indexed on
2010/05/04
12:48 UTC
Read the original article
Hit count: 190
I have a very large dataset, each item in the dataset being roughly 1kB in size. The data needs to be queried rapidly by many applications distributed over a network. The dataset has more than a million items (so 500 million+ 1kB data chunks).
What would be the best method to storing this dataset (need to allow adding more items, and reading them rapidly, but never modifying already added data)? Would using a MySQL DB using the binary blob format be appropriate?
Or should each of these be stored as files on a file system?
edit: the number is 1 million items now, but needs to be able to scale to well over 500 million items easily.
© Stack Overflow or respective owner