Improve heavy work in a loop in multithreading

Posted by xjaphx on Stack Overflow See other posts from Stack Overflow or by xjaphx
Published on 2012-08-29T08:55:51Z Indexed on 2012/08/29 9:38 UTC
Read the original article Hit count: 248

I have a little problem with my data processing.

public void ParseDetails()
{            
    for (int i = 0; i < mListAppInfo.Count; ++i)
    {
        ParseOneDetail(i);
    }
}

For 300 records, it usually takes around 13-15 minutes.

I've tried to improve by using Parallel.For() but it always stop at some point.

public void ParseDetails()
{
    Parallel.For(0, mListAppInfo.Count, i => ParseOneDetail(i));
}

In method ParseOneDetail(int index), I set an output log for tracking the record id which is under processing. Always hang at some point, I don't know why..

ParseOneDetail(): 89 ...
ParseOneDetail(): 90 ...
ParseOneDetail(): 243 ...
ParseOneDetail(): 92 ...
ParseOneDetail(): 244 ...
ParseOneDetail(): 93 ...
ParseOneDetail(): 245 ...
ParseOneDetail(): 247 ...
ParseOneDetail(): 94 ...
ParseOneDetail(): 248 ...
ParseOneDetail(): 95 ...
ParseOneDetail(): 99 ...
ParseOneDetail(): 249 ...
ParseOneDetail(): 100 ...
_ <hang at this point>

Appreciate your help and suggestions to improve this. Thank you!

Edit 1: update for method:

private void ParseOneDetail(int index)
{
    Console.WriteLine("ParseOneDetail(): " + index + " ... ");
    ApplicationInfo appInfo = mListAppInfo[index];

    var htmlWeb = new HtmlWeb();
    var document = htmlWeb.Load(appInfo.AppAnnieURL);

    // get first one only
    HtmlNode nodeStoreURL = document.DocumentNode.SelectSingleNode(Constants.XPATH_FIRST);
    appInfo.StoreURL = nodeStoreURL.Attributes[Constants.HREF].Value;
}

Edit 2: This is the error output after a while running as Enigmativity suggest,

ParseOneDetail(): 234 ...
ParseOneDetail(): 87 ...
ParseOneDetail(): 235 ...
ParseOneDetail(): 236 ...
ParseOneDetail(): 88 ...
ParseOneDetail(): 238 ...
ParseOneDetail(): 89 ...
ParseOneDetail(): 90 ...
ParseOneDetail(): 239 ...
ParseOneDetail(): 92 ...

Unhandled Exception: System.AggregateException: One or more errors occurred. ---
> System.Net.WebException: The operation has timed out
   at System.Net.HttpWebRequest.GetResponse()
   at HtmlAgilityPack.HtmlWeb.Get(Uri uri, String method, String path, HtmlDocum
ent doc, IWebProxy proxy, ICredentials creds) in D:\Source\htmlagilitypack.new\T
runk\HtmlAgilityPack\HtmlWeb.cs:line 1355
   at HtmlAgilityPack.HtmlWeb.LoadUrl(Uri uri, String method, WebProxy proxy, Ne
tworkCredential creds) in D:\Source\htmlagilitypack.new\Trunk\HtmlAgilityPack\Ht
mlWeb.cs:line 1479
   at HtmlAgilityPack.HtmlWeb.Load(String url, String method) in D:\Source\htmla
gilitypack.new\Trunk\HtmlAgilityPack\HtmlWeb.cs:line 1103
   at HtmlAgilityPack.HtmlWeb.Load(String url) in D:\Source\htmlagilitypack.new\
Trunk\HtmlAgilityPack\HtmlWeb.cs:line 1061
   at SimpleChartParser.AppAnnieParser.ParseOneDetail(ApplicationInfo appInfo) i
n c:\users\nhn60\documents\visual studio 2010\Projects\FunToolPack\SimpleChartPa
rser\AppAnnieParser.cs:line 90
   at SimpleChartParser.AppAnnieParser.<ParseDetails>b__0(ApplicationInfo ai) in
 c:\users\nhn60\documents\visual studio 2010\Projects\FunToolPack\SimpleChartPar
ser\AppAnnieParser.cs:line 80
   at System.Threading.Tasks.Parallel.<>c__DisplayClass21`2.<ForEachWorker>b__17
(Int32 i)
   at System.Threading.Tasks.Parallel.<>c__DisplayClassf`1.<ForWorker>b__c()
   at System.Threading.Tasks.Task.InnerInvoke()
   at System.Threading.Tasks.Task.InnerInvokeWithArg(Task childTask)
   at System.Threading.Tasks.Task.<>c__DisplayClass7.<ExecuteSelfReplicating>b__
6(Object )
   --- End of inner exception stack trace ---
   at System.Threading.Tasks.Task.ThrowIfExceptional(Boolean includeTaskCanceled
Exceptions)
   at System.Threading.Tasks.Task.Wait(Int32 millisecondsTimeout, CancellationTo
ken cancellationToken)
   at System.Threading.Tasks.Parallel.ForWorker[TLocal](Int32 fromInclusive, Int
32 toExclusive, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWit
hState, Func`4 bodyWithLocal, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](TSource[] ar
ray, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState, Act
ion`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithEveryt
hing, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.ForEachWorker[TSource,TLocal](IEnumerable`
1 source, ParallelOptions parallelOptions, Action`1 body, Action`2 bodyWithState
, Action`3 bodyWithStateAndIndex, Func`4 bodyWithStateAndLocal, Func`5 bodyWithE
verything, Func`1 localInit, Action`1 localFinally)
   at System.Threading.Tasks.Parallel.ForEach[TSource](IEnumerable`1 source, Act
ion`1 body)
   at SimpleChartParser.AppAnnieParser.ParseDetails() in c:\users\nhn60\document
s\visual studio 2010\Projects\FunToolPack\SimpleChartParser\AppAnnieParser.cs:li
ne 80
   at SimpleChartParser.Program.Main(String[] args) in c:\users\nhn60\documents\
visual studio 2010\Projects\FunToolPack\SimpleChartParser\Program.cs:line 15

© Stack Overflow or respective owner

Related posts about c#

Related posts about multithreading