Database storage for high sample rate data in web app

Posted by Jim on Stack Overflow See other posts from Stack Overflow or by Jim
Published on 2011-01-16T19:46:33Z Indexed on 2011/01/16 19:53 UTC
Read the original article Hit count: 270

Filed under:

database-design

|

web-applications

|

charts

|

simpledb

|

amazon-simpledb

I've got multiple sensors feeding data to my web app. Each channel is 5 samples per second and the data gets uploaded bundled together in 1 minute json messages (containing 300 samples). The data will be graphed using flot at multiple zoom levels from 1 day to 1 minute.

I'm using Amazon SimpleDB and I'm currently storing the data in the 1 minute chunks that I receive it in. This works well for high zoom levels, but for full days there will be simply be too many rows to retrieve.

The idea I've currently got is that every hour I can crawl through the data and collect together 300 samples for the last hour and store them in an hour Domain (table if you like).

Does this sound like a reasonable solution? How have others implemented the same sort of systems?

© Stack Overflow or respective owner

Related posts about database-design

(Database Design - products attributes): What is better option for product attribute database design

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I new in database design. What is better option for product attribute database design for cms?(Please suggest other options also). option 1: 1 table products{ id product_name color price attribute_name1 attribute_value1 attribute_name2 attribute_value2 attribute_name3 attribute_value3 } option… >>> More
Book Review: Pro SQL Server 2008 Relational Database Design and Implementation

as seen on SQL Blog - Search for 'SQL Blog'
Investing in proper database design is a very efficient way to cut maintenance costs. If we expect a system to last, we need to make sure it has a good solid foundation - high quality database design. Surely we can and sometimes do cut corners and save on database design to get things done faster… >>> More
Advice on database design / SQL for retrieving data with chronological order

as seen on Stack Overflow - Search for 'Stack Overflow'
I am creating a database that will help keep track of which employees have been on a certain training course. I would like to get some guidance on the best way to design the database. Specifically, each employee must attend the training course each year and my database needs to keep a history of… >>> More
Fiscal year handling strategies in database design

as seen on Stack Overflow - Search for 'Stack Overflow'
By fiscal year I mean all the data in the database (in all tables) that occurred in the particular year. Lets say that we are building an application that allows user to choose from different years. What way of implementing this would you prefer, and why: Separate fiscal year data based on multiple… >>> More
Database design for summarized data

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a new table I'm going to add to a bunch of other summarized data, basically to take some of the load off by calculating weekly avgs. My question is whether I would be better off with one model over the other. One model with days of the week as a column with an additional column for price… >>> More

Related posts about web-applications

Modularizing web applications

as seen on Stack Overflow - Search for 'Stack Overflow'
Hey all, I was wondering how big companies tend to modularize components on their page. Facebook is a good example: There's a team working on Search that has its own CSS, javascript, html, etc.. There's a team working on the news feed that has its own CSS, javascript, html, etc… >>> More
Adding AJAX Support to Windows Mobile Web Applications

as seen on Internet.com - Search for 'Internet.com'
In this article, you will learn how you can utilize AJAX scripts even in Web applications targeted for Windows Mobile applications. >>> More
Globalize your Web Applications: The Universal Character Set

as seen on Internet.com - Search for 'Internet.com'
This fourth article the Globalize your Web Applications series deals with something that's at the forefront of your web apps: the character sets. More specifically, we'll be taking a look at the Universal Character Set (UCS) and its role in creating multilingual web pages. >>> More
Globalize your Web Applications: PHP's Locale Formatting Classes

as seen on Internet.com - Search for 'Internet.com'
In part 3 of the Globalize your Web Applications series, we'll be exploring PHP's I18N_Number and I18N_Currency classes, both of which are part of the I18N Libraries. >>> More
Singleton pattern in web applications

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm using a singleton pattern for the datacontext in my web application so that I dont have to instantiate it every time, however I'm not sure how web applications work, does IIS open a thread for every user connected? if so, what would happend if my singleton is not thread safe? Also, is it OK to… >>> More