Database storage for high sample rate data in web app
- by Jim
I've got multiple sensors feeding data to my web app. Each channel is 5 samples per second and the data gets uploaded bundled together in 1 minute json messages (containing 300 samples).
The data will be graphed using flot at multiple zoom levels from 1 day to 1 minute.
I'm using Amazon SimpleDB and I'm currently storing the data in the 1 minute chunks that I receive it in. This works well for high zoom levels, but for full days there will be simply be too many rows to retrieve.
The idea I've currently got is that every hour I can crawl through the data and collect together 300 samples for the last hour and store them in an hour Domain (table if you like).
Does this sound like a reasonable solution? How have others implemented the same sort of systems?