What is the best possible technology for pulling huge data from 4 remote servers

Posted by Habib Ullah Bahar on Stack Overflow See other posts from Stack Overflow or by Habib Ullah Bahar
Published on 2010-02-01T07:11:52Z Indexed on 2010/04/24 23:13 UTC
Read the original article Hit count: 305

Filed under:

real-time

|

server-push

|

data

|

stocks

Hello,

For one of our project, we need to pull huge real time stock data from 4 remote servers across two countries. The trivial process here, check the sources for a regular interval and save the update to database.

But as these are real time stock data of more than 1000 companies, I have to pull every second, which isn't good in case of memory, bandwidth I think.

Please give me suggestion on which technology/platform [We are flexible here. PHP, Python, Java, PERL - anyone of them will be OK for us] we should choose, it can be achieved easily and with better performance.

© Stack Overflow or respective owner

Related posts about real-time

Google Keyword Research Easy But Powerful - Real Time Results - Real Time Traffic

as seen on Ezine Articles - Search for 'Ezine Articles'
The below is not just an information section, it's a job! You have to do something here. You need to take action, don't just read it, and think "OK, this is a good idea, I should do it". You see that word "should" we need to change it to "I WILL". >>> More
Does anyone really understand how HFSC scheduling in Linux/BSD works?

as seen on Server Fault - Search for 'Server Fault'
I read the original SIGCOMM '97 PostScript paper about HFSC, it is very technically, but I understand the basic concept. Instead of giving a linear service curve (as with pretty much every other scheduling algorithm), you can specify a convex or concave service curve and thus it is possible to decouple… >>> More
Does anyone really understand how HFSC scheduling in Linux/BSD works?

as seen on Server Fault - Search for 'Server Fault'
I read the original SIGCOMM '97 PostScript paper about HFSC, it is very technically, but I understand the basic concept. Instead of giving a linear service curve (as with pretty much every other scheduling algorithm), you can specify a convex or concave service curve and thus it is possible to decouple… >>> More
Developing a Real-time Autocomplete Control in Windows Presentation Foundation (WPF)

as seen on Internet.com - Search for 'Internet.com'
Create a standalone autocomplete library that can be clubbed into any Windows Presentation Foundation (WPF) application and used with minimal efforts. >>> More
JD Edwards World Reporting Made Easy with Real Time Reporting Tools from The GL Company

as seen on Oracle Blogs - Search for 'Oracle Blogs'
Fred talks to Paul Yarwood, US Operations General Manager and Richard Crotty, North America Business Development Manager for The GL Company, an Oracle Certified Partner, and Denise Grills, Senior Director of Marketing and Product Strategy for Oracle's JD Edwards World products. They discuss how the… >>> More

Related posts about server-push

Server Push / HTTP Streaming on Windows Mobile / Windows CE

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I find that HTTP Streaming / Server Push is quite promising for my project. Does someone have any clue on how to implement this in Windows Mobile? .NET / Native / other implementations are welcomed. Preferably with permissive license. some links on HTTP Steaming / Server Push: - Push Technology -… >>> More
Server push (Comet) in Google App Engine in Python

as seen on Stack Overflow - Search for 'Stack Overflow'
How can I implement Comet / Server push in Google App Engine in Python? >>> More
Reading HTTP server push streams with Python

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm playing around trying to write a client for a site which provides data as an HTTP stream (aka HTTP server push). However, urllib2.urlopen() grabs the stream in its current state and then closes the connection. I tried skipping urllib2 and using httplib directly, but this seems to have the same… >>> More
server push or client push is better ?

as seen on Stack Overflow - Search for 'Stack Overflow'
I am developing a chat website using jsp/servlet.I will be hosting my website on gooogle appengine .Now i have some doubts regarding whether to use server push or client pull technology 1)If i use server push and if i dont close the response of servlet will it cause the server to go slow?How many… >>> More
Server push: comet vs ape?

as seen on Stack Overflow - Search for 'Stack Overflow'
I've read a little about comet and also APE. Which one is better? I want the users to see other users updated content. Like Google Wave. And in comet, there are 2 versions: iframe vs traditional ajax. what is the difference and which is better. I dont quite understand it. Thanks. >>> More