Any frameworks or library allow me to run large amount of concurrent jobs schedully?

Posted by Yoga on Programmers See other posts from Programmers or by Yoga
Published on 2014-06-02T06:09:14Z Indexed on 2014/06/02 9:49 UTC
Read the original article Hit count: 245

Filed under:

architecture

|

concurrency

|

scalability

|

distributed-computing

|

parallel-programming

Are there any high level programming frameworks that allow me to run large amount of concurrent jobs schedully?

e.g. I have 100K of urls need to check their uptime every 5 minutes

Definitely I can write a program to handle this, but then I need to handle concurrency, queuing, error handling, system throttling, job distribution etc.

Will there be a framework that I only focus on a particular job (i.e. the ping task) and the system will take care of the scaling and error handling for me?

I am open to any language.

© Programmers or respective owner

Related posts about architecture

3-tier architecture v. 3-server architecture

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm building a traditional .NET MVC site, so I've got a natural 3-tier software architecture setup (presentation in the form of Views, business layer in the controller, and data layer in the models and data access layer). When I've deployed such sites, it usually goes either on one server (where… >>> More
"cloud architecture" concepts in a system architecture diagrams

as seen on Stack Overflow - Search for 'Stack Overflow'
If you design a distributed application for easy scale-out, or you just want to make use of any of the new “cloud computing” offerings by Amazon, Google or Microsoft, there are some typical concepts or components you usually end up using: distributed blob storage (aka S3) asynchronous, durable message… >>> More
Is the PowerPC (G4) architecture better than Intel x86 architecture for certain types of application

as seen on Super User - Search for 'Super User'
I have a PowerMac G4 from around the year 2000. It's a serviceable Unix machine but I don't do much with it since I have plenty of Pentium 4 machines around with Linux on them. I was wondering if the PowerPC based machine is capable of handling certain tasks faster or "better" than Intel based machines… >>> More
SAAS architecture and salesforce database architecture

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello all, I am architecting a software project and I want to create a SAAS (Software As a service) one. I want to model my application along the lines of Salesforce. I really like there customization features but I am not sure how they really go about it. I read that they create an ID for every… >>> More
Is there a Telecommunications Reference Architecture?

as seen on Oracle Blogs - Search for 'Oracle Blogs'
@font-face { font-family: "Arial"; }@font-face { font-family: "Courier New"; }@font-face { font-family: "Wingdings"; }@font-face { font-family: "Cambria"; }p.MsoNormal, li.MsoNormal, div.MsoNormal { margin: 0cm 0cm 0.0001pt; font-size: 12pt; font-family: "Times New Roman"; }p.MsoListParagraph… >>> More

Related posts about concurrency

SQL SERVER – Concurrency Basics – Guest Post by Vinod Kumar

as seen on SQL Authority - Search for 'SQL Authority'
This guest post is by Vinod Kumar. Vinod Kumar has worked with SQL Server extensively since joining the industry over a decade ago. Working on various versions from SQL Server 7.0, Oracle 7.3 and other database technologies – he now works with the Microsoft Technology Center (MTC) as a Technology… >>> More
Http requests / concurrency?

as seen on Stack Overflow - Search for 'Stack Overflow'
Say a website on my localhost takes about 3 seconds to do each request. This is fine, and as expected (as it is doing some fancy networking behind the scenes). However, if i open the same url in tabs (in firefox), then reload them all at the same time, it appears to load each page sequentially rather… >>> More
Mnesia: A Distributed DBMS Rooted in Concurrency

as seen on Internet.com - Search for 'Internet.com'
Find out what makes Mnesia, the Erlang-based database management system, perfect for distribution across a network of computers. >>> More
C# concurrency vs Java concurrency, which is neatly designed? which is better?

as seen on Stack Overflow - Search for 'Stack Overflow'
I am hard core fan for Doug Lee and his concurrency framework. I have also (from twitter) came across a comment from Joshua Bloch that java concurrency is one of the best framework (guess that he compared to .Net). But some of you worked in both Java Concurrency and .NET concurrency. What is your… >>> More
Queued Loadtest to remove Concurrency issues using Shared Data Service in OpenScript

as seen on Oracle Blogs - Search for 'Oracle Blogs'
Queued Processing to remove Concurrency issues in Loadtest ScriptsSome scripts act on information returned by the server, e.g. act on first item in the returned list of pending tasks/actions. This may lead to concurrency issues if the virtual users simulated in a load test scenario are not synchronized… >>> More