Improve Efficiency for This Text Processing Code
- by johnv
I am writing a program that counts the number of words in a text file which is already in lowercase and separated by spaces. I want to use a dictionary and only count the word IF it's within the dictionary. The problem is the dictionary is quite large (~100,000 words) and each text document has also ~50,000 words. As such, the codes that I wrote below gets very slow (takes about 15 sec to process one document on a quad i7 machine). I'm wondering if there's something wrong with my coding and if the efficiency of the program can be improved. Thanks so much for your help. Code below:
public static string WordCount(string countInput)
{
string[] keywords = ReadDic(); /* read dictionary txt file*/
/*then reads the main text file*/
Dictionary<string, int> dict = ReadFile(countInput).Split(' ')
.Select(c => c)
.Where(c => keywords.Contains(c))
.GroupBy(c => c)
.Select(g => new { word = g.Key, count = g.Count() })
.OrderBy(g => g.word)
.ToDictionary(d => d.word, d => d.count);
int s = dict.Sum(e => e.Value);
string k = s.ToString();
return k;
}
