VB.Net HTTPWebRequest Speed is slow comparing Python URLOpen

Posted by regexhacks on Stack Overflow See other posts from Stack Overflow or by regexhacks
Published on 2011-02-12T06:21:22Z Indexed on 2011/02/12 7:25 UTC
Read the original article Hit count: 186

Filed under:

python

|

vb.net

|

http

|

comparison

Hi I am coding a web-crawler which will crawl the websites and selectively parse different sections of a web site.

I am a .Net developer so the choice was obvious that I did it in .Net but the speed was very slow which included downloading and parsing of HTMLPages

Then I tried to just download the contents first using .Net and then same domains using python but the python was very impressive in downloading data. I have achieved downloading using python but the later part is not that easy to code in python, which obviously i don't want to do.

The same batch of domain which took 100 seconds in Python was taking 20 minutes in .Net based crawler

I tried http://www.eqlit.com/ to download and in took 8 seconds in Python and same was taking 100 Seconds in .Net crawler

Does anyone anyone have any idea why this is slow in .Net but fast in python?

© Stack Overflow or respective owner

Related posts about python

unmet dependencies in Ubuntu 12.04

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I tried today to install a dvb-card on my Ubuntu 12.04 (Linux blauhai-linux 3.2.0-25-generic #40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux ). The installation failed with an error. After that, i tried to install python (it was already installed but i got this error): linux:~$… >>> More
How can I get sikuli-ide to work?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I installed sikuli-ide with sudo apt-get install sikuli-ide Everything was fine until I tried to start it from the terminal. I typed sikuli-ide But the only response I got was [info] locale: en_US The application was not started, furthermore there is no desktop file and sikuli-ide does not… >>> More
Getting PATH right for python after MacPorts install

as seen on Super User - Search for 'Super User'
I can't import some python libraries (PIL, psycopg2) that I just installed with MacPorts. I looked through these forums, and tried to adjust my PATH variable in $HOME/.bash_profile in order to fix this but it did not work. I added the location of PIL and psycopg2 to PATH. I know that Terminal is… >>> More
call python with system() in R to run a python script emulating the python console

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to pass a chunk of Python code to Python in R with something like system('python ...'), and I'm wondering if there is an easy way to emulate the python console in this case. For example, suppose the code is "print 'hello world'", how can I get the output like this in R? >>> print… >>> More
Python - Calling a non python program from python?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am currently struggling to call a non python program from a python script. I have a ~1000 files that when passed through this C++ program will generate ~1000 outputs. Each output file must have a distinct name. The command I wish to run is of the form: program_name -input -output -o1 -o2… >>> More

Related posts about vb.net

VB.NET How Do I skin a vb.net app

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to skin a vb.net app I made ive googled some stuff and I've seen skinned vb.net apps. However it seems like any time i try to find someone explaining it its a link to a pay for product. Does anyone have anything useful on this? >>> More
VB.NET IF() Coalesce and “Expression Expected” Error

as seen on ASP.net Weblogs - Search for 'ASP.net Weblogs'
I am trying to use the equivalent of the C# “??” operator in some VB.NET code that I am working in. This StackOverflow article for “Is there a VB.NET equivalent for C#'s ?? operator?” explains the VB.NET IF() statement syntax which is exactly what I am looking for... and I thought I was going to… >>> More
Would a programmer knowing C# and VB.Net ever choose VB.Net?

as seen on Programmers - Search for 'Programmers'
Now before someone tells me VB.Net isn't bad like VB was, I know it isn't. But, I've yet to speak to a programmer who is completely content that some project they work on is written in VB.Net. Basically, my question is would a programmer knowing both C# and VB.Net (and all of their team knowing both)… >>> More
Do I lose anything by coding in c# and using free online vb.net code convertors?

as seen on Programmers - Search for 'Programmers'
The company I work for uses vb.net since there are many programmers who moved up from vb6 to vb.net. Basically more vb.net resources in the company for support/maintenance vs c#. I am a c# coder and was wondering if I could just continue coding in c# and just use the many online free c# to vb.net… >>> More
What are some good/reputable/widely-used libraries written in VB.NET?

as seen on Stack Overflow - Search for 'Stack Overflow'
Generally speaking, when VB.NET and C# are compared, there is a lot of strong support for C#, accompanied by some bashing of VB.NET until a respected developer comes along and acts as The Voice Of Reason, pointing out that while VB prior to VB.NET had its fair share of issues, VB.NET is really a very… >>> More