Solution for distributing MANY simple network tasks?

Posted by EmpireJones on Stack Overflow See other posts from Stack Overflow or by EmpireJones
Published on 2010-05-01T06:11:30Z Indexed on 2010/05/01 6:17 UTC
Read the original article Hit count: 217

Filed under:

clustering

|

grid

|

python

|

web-development

|

open-source

I would like to create some sort of a distributed setup for running a ton of small/simple REST web queries in a production environment. For each 5-10 related queries which are executed from a node, I will generate a very small amount of derived data, which will need to be stored in a standard relational database (such as PostgreSQL).

What platforms are built for this type of problem set? The nature, data sizes, and quantities seem to contradict the mindset of Hadoop. There are also more grid based architectures such as Condor and Sun Grid Engine, which I have seen mentioned. I'm not sure if these platforms have any recovery from errors though (checking if a job succeeds).

What I would really like is a FIFO type queue that I could add jobs to, with the end result of my database getting updated.

Any suggestions on the best tool for the job?

© Stack Overflow or respective owner

Related posts about clustering

agglomerative clustering java

as seen on Stack Overflow - Search for 'Stack Overflow'
Is there any java file that I can use to perform "agglomerative clustering" Result should provide me every level nodes id help................. >>> More
Clustering for Mere Mortals (Pt2)

as seen on SQL Team - Search for 'SQL Team'
Planning. I could stop there and let that be the entirety post #2 in this series. Planning is the single most important element in building a cluster and the Laptop Demo Cluster is no exception. One of the more awkward parts of actually creating a cluster is coordinating information between Windows… >>> More
MySQL Clustering in a Sandbox

as seen on Internet.com - Search for 'Internet.com'
MySQL's unique architecture allows for plugin storage engines. There is the MyISAM storage engine, the ARCHIVE storage engine and the InnoDB storage engine; so it makes sense then that MySQL's clustering solution involves a storage engine as well, namely the NDB (Network DataBase) storage engine. >>> More
Microsoft SQL Server High-Availability Videos and Q&A Log

as seen on SQL Blog - Search for 'SQL Blog'
You Want Videos? We Got Videos! I always enjoy getting the chance to catch up with author, consultant, and Microsoft Clustering MVP Allan Hirt . Allan and I recently presented two sessions covering an overview of high availability in Microsoft SQL Server and, the following week, a demo of how to implement… >>> More
I need advice about iscsi + zfs(or ntfs) + windows 2008 clustering

as seen on Server Fault - Search for 'Server Fault'
I want to setup a storage farm with iSCSI. I have 2 cluster node machine, 1 iscsi target machine that has 8TB installed as RAID 10. The capacity is now 8TB, but I'll upgrade the capacity in future. Let's say, I installed clusters as file server, and I connected these servers to iscsi target, then… >>> More

Related posts about grid

Metro: Introduction to CSS 3 Grid Layout

as seen on Stephen Walter - Search for 'Stephen Walter'
The purpose of this blog post is to provide you with a quick introduction to the new W3C CSS 3 Grid Layout standard. You can use CSS Grid Layout in Metro style applications written with JavaScript to lay out the content of an HTML page. CSS Grid Layout provides you with all of the benefits of using… >>> More
Metro: Introduction to CSS 3 Grid Layout

as seen on Stephen Walter - Search for 'Stephen Walter'
The purpose of this blog post is to provide you with a quick introduction to the new W3C CSS 3 Grid Layout standard. You can use CSS Grid Layout in Metro style applications written with JavaScript to lay out the content of an HTML page. CSS Grid Layout provides you with all of the benefits of using… >>> More
Scroll Viewer not visible in wpf DataGrid

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a datagrid in a grid but the scrollviewer is not visibile even though I made it auto. Below in my code. I can't figure out where's the problem. <Grid Grid.Row="0" Grid.Column="0"> <Grid.RowDefinitions> … >>> More
Am I going about this the right way?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hey Guys, I'm starting a WPF project, and just finished the base of the UI, it seems very convoluted though, so I'm not sure if I've gone around laying it out in the right way. I don't want to get to start developing the back-end and realise that I've done the front wrong, and make life harder for… >>> More
Visibility.Collapse doesnt work

as seen on Stack Overflow - Search for 'Stack Overflow'
Visibility.Collapse doesnt work in my case. below is the XAML. If i try to hide the lblCountry and cmbCountry a white space is shown between zip and practice fields. There is no option to hide an entire row of a Grid. <Grid> <Canvas Name="canDemographic" > </Canvas> … >>> More