Running Long Process: Indexing 5GB docs with Lucene

Posted by Robert Dondo on Stack Overflow See other posts from Stack Overflow or by Robert Dondo
Published on 2010-05-14T12:14:10Z Indexed on 2010/05/14 12:24 UTC
Read the original article Hit count: 243

Filed under:

c#

|

asp

|

.NET

|

lucene

|

iis

Situation:I have an ASP .NET application that will search through docs using Lucene. I want to run the initial indexing (the index will be incremental after the initial run so there wont be need to index the whole directory again in future). Currently, I have about 5GB of docs (45000files).

Problem: My application times out before completing the process. I have altered the TimeOut like this:

HttpContext.Current.Server.ScriptTimeout = 200000;

but it still does not complete the process.

How can I run the index?

© Stack Overflow or respective owner

Related posts about c#

.NET WebRequest.PreAuthenticate not quite what it sounds like

as seen on West-Wind - Search for 'West-Wind'
I’ve run into the problem a few times now: How to pre-authenticate .NET WebRequest calls doing an HTTP call to the server – essentially send authentication credentials on the very first request instead of waiting for a server challenge first? At first glance this sound like it should be easy:… >>> More
HttpWebRequest and Ignoring SSL Certificate Errors

as seen on West-Wind - Search for 'West-Wind'
Man I can't believe this. I'm still mucking around with OFX servers and it drives me absolutely crazy how some these servers are just so unbelievably misconfigured. I've recently hit three different 3 major brokerages which fail HTTP validation with bad or corrupt certificates at least according to… >>> More
The dynamic Type in C# Simplifies COM Member Access from Visual FoxPro

as seen on West-Wind - Search for 'West-Wind'
I’ve written quite a bit about Visual FoxPro interoperating with .NET in the past both for ASP.NET interacting with Visual FoxPro COM objects as well as Visual FoxPro calling into .NET code via COM Interop. COM Interop with Visual FoxPro has a number of problems but one of them at least got a lot… >>> More
Dynamic Type to do away with Reflection

as seen on West-Wind - Search for 'West-Wind'
The dynamic type in C# 4.0 is a welcome addition to the language. One thing I’ve been doing a lot with it is to remove explicit Reflection code that’s often necessary when you ‘dynamically’ need to walk and object hierarchy. In the past I’ve had a number of ReflectionUtils that used string based expressions… >>> More
Finding a Relative Path in .NET

as seen on West-Wind - Search for 'West-Wind'
Here’s a nice and simple path utility that I’ve needed in a number of applications: I need to find a relative path based on a base path. So if I’m working in a folder called c:\temp\templates\ and I want to find a relative path for c:\temp\templates\subdir\test.txt I want to receive back subdir\test… >>> More

Related posts about asp

How to associate jquery validation when only one button if there are many??

as seen on Stack Overflow - Search for 'Stack Overflow'
Greetings, In my current project, I have gridview, search button, text box for search, text box, and submit button. -I should input string in the search box then click search button. -when click search button, it will retrieve all matches records then bind them to the view grid. -then when I click… >>> More
How to associate jquery validation with only one button if there are many?

as seen on Stack Overflow - Search for 'Stack Overflow'
Greetings, In my current project, I have gridview, search button, text box for search, text box, and submit button. -I should input string in the search box then click search button. -when click search button, it will retrieve all matches records then bind them to the view grid. -then when I click… >>> More
What's New in ASP.NET 4

as seen on Geeks with Blogs - Search for 'Geeks with Blogs'
The .NET Framework version 4 includes enhancements for ASP.NET 4 in targeted areas. Visual Studio 2010 and Microsoft Visual Web Developer Express also include enhancements and new features for improved Web development. This document provides an overview of many of the new features that are included… >>> More
April 14th Links: ASP.NET, ASP.NET MVC, ASP.NET Web API and Visual Studio

as seen on ASP.net Weblogs - Search for 'ASP.net Weblogs'
Here is the latest in my link-listing blog series: ASP.NET Easily overlooked features in VS 11 Express for Web: Good post by Scott Hanselman that highlights a bunch of easily overlooked improvements that are coming to VS 11 (and specifically the free express editions) for web development: unit… >>> More
Create Resume problem

as seen on Stack Overflow - Search for 'Stack Overflow'
hello mates, i am working on a project of online resume management system and i am encountering an exception while creating resume. [b] Exception: Data type mismatch in criteria expression. [/b] here is my code for Create Resume-1.aspx.cs using System; using System.Data; using System.Configuration; using… >>> More