What is the best way to download files via HTTP using .NET?

Posted by Shamika on Stack Overflow See other posts from Stack Overflow or by Shamika
Published on 2010-03-24T05:09:07Z Indexed on 2010/03/24 5:53 UTC
Read the original article Hit count: 223

Filed under:

c#

|

webclient

|

http

|

.NET

In one of my application I'm using the WebClient class to download files from a web server. Depending on the web server sometimes the application download millions of documents. It seems to be when there are lot of documents, performance vise the WebClient doesn't scale up well.

Also it seems to be the WebClient doesn't immediately close the connection it opened for the WebServer even after it successfully download the particular document.

I would like to know what other alternatives I have.

© Stack Overflow or respective owner

Related posts about c#

.NET WebRequest.PreAuthenticate not quite what it sounds like

as seen on West-Wind - Search for 'West-Wind'
I’ve run into the problem a few times now: How to pre-authenticate .NET WebRequest calls doing an HTTP call to the server – essentially send authentication credentials on the very first request instead of waiting for a server challenge first? At first glance this sound like it should be easy:… >>> More
HttpWebRequest and Ignoring SSL Certificate Errors

as seen on West-Wind - Search for 'West-Wind'
Man I can't believe this. I'm still mucking around with OFX servers and it drives me absolutely crazy how some these servers are just so unbelievably misconfigured. I've recently hit three different 3 major brokerages which fail HTTP validation with bad or corrupt certificates at least according to… >>> More
The dynamic Type in C# Simplifies COM Member Access from Visual FoxPro

as seen on West-Wind - Search for 'West-Wind'
I’ve written quite a bit about Visual FoxPro interoperating with .NET in the past both for ASP.NET interacting with Visual FoxPro COM objects as well as Visual FoxPro calling into .NET code via COM Interop. COM Interop with Visual FoxPro has a number of problems but one of them at least got a lot… >>> More
Dynamic Type to do away with Reflection

as seen on West-Wind - Search for 'West-Wind'
The dynamic type in C# 4.0 is a welcome addition to the language. One thing I’ve been doing a lot with it is to remove explicit Reflection code that’s often necessary when you ‘dynamically’ need to walk and object hierarchy. In the past I’ve had a number of ReflectionUtils that used string based expressions… >>> More
Finding a Relative Path in .NET

as seen on West-Wind - Search for 'West-Wind'
Here’s a nice and simple path utility that I’ve needed in a number of applications: I need to find a relative path based on a base path. So if I’m working in a folder called c:\temp\templates\ and I want to find a relative path for c:\temp\templates\subdir\test.txt I want to receive back subdir\test… >>> More

Related posts about webclient

Webclient (fom Mozilla) under Linux

as seen on Super User - Search for 'Super User'
I could not start Webclient under Linux ( due to factory class for GTK could not be found) Does anybody know the solution of this problem? >>> More
Webclient using download file to grab file from server - handling exceptions

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello everyone, I have a web service in which I am manipulating POST and GET methods to facilitate upload / download functionality for some files in a client/server style architecture. Basically the user is able to click a button to download a specific file, make some changes in the app, then click… >>> More
WebClient.DownloadData hangs

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi. I am trying to download file using WebClient.DownloadData. Usually the download is Ok, but for some Urls the download just hangs. I've tried to override the WebClient, and set a timeout to the WebRequest, and it didn't help. I've also tried to create WebRequest (with time out), then get the… >>> More
.NET WebClient tag property, or keeping track of downloads ?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, I am trying to implement an Asynchronous file download from a web server using private void btnDownload_Click(object sender, EventArgs e) { WebClient webClient = new WebClient(); webClient.Credentials = new NetworkCredential("test", "test"); webClient… >>> More
htmlunit eofexception

as seen on Stack Overflow - Search for 'Stack Overflow'
hi, i am using htmlunit to parse a page. but when i tried to get page from url i got an exception, lets have a look on my code: WebClient wc = new WebClient(BrowserVersion.FIREFOX_3); HtmlPage page = wc.getPage(strUrl); when compilet goes to wc.getPage(strUrl) it throws an exception i.e. java… >>> More