What is the fastest way to send 100,000 HTTP requests in Python?

Posted by Igor G. on Stack Overflow See other posts from Stack Overflow or by Igor G.
Published on 2010-04-13T19:19:50Z Indexed on 2010/04/13 19:23 UTC
Read the original article Hit count: 591

Filed under:
|
|
|

Hello, I am opening a file which has 100,000 url's. I need to send an http request to each url and print the status code. I am using Python 2.6, and so far looked at the many confusing ways Python implements threading/concurrency. I have even looked at the python concurrence library, but cannot figure out how to write this program correctly. Has anyone come across a similar problem? I guess generally I need to know how to perform thousands of tasks in Python as fast as possible - I suppose that means 'concurrently'.

Thank you, Igor

© Stack Overflow or respective owner

Related posts about python

Related posts about http