Lucene.Net memory consumption and slow search when too many clauses used

Posted by Umer on Stack Overflow See other posts from Stack Overflow or by Umer
Published on 2010-06-18T13:58:29Z Indexed on 2010/06/18 14:03 UTC
Read the original article Hit count: 304

Filed under:

optimization

|

lucene.net

|

searching

I have a DB having text file attributes and text file primary key IDs and indexed around 1 million text files along with their IDs (primary keys in DB).

Now, I am searching at two levels. First is straight forward DB search, where i get primary keys as result (roughly 2 or 3 million IDs)

Then i make a Boolean query for instance as following

+Text:"test*" +(pkID:1 pkID:4 pkID:100 pkID:115 pkID:1041 .... )

and search it in my Index file.

The problem is that such query (having 2 million clauses) takes toooooo much time to give result and consumes reallly too much memory....

Is there any optimization solution for this problem ?

© Stack Overflow or respective owner

Related posts about optimization

Search Engine Optimization - The Importance of Page Optimization in Search Engine Optimization

as seen on Ezine Articles - Search for 'Ezine Articles'
In order for your website to rank well, your internal linking structure is critical to your success. This is covered some of the theory for this in various articles and blogs about Page Structure of a website, which said how you should map out the physical linking structure, but in this guide I will… >>> More
SEO Optimization - How to Master the SEO Optimization Process in Four Easy Steps

as seen on Ezine Articles - Search for 'Ezine Articles'
Search engine optimization is a critical part of any internet marketing campaign but can often be intimidating to new marketers. In this article you will learn the 4 basic components of SEO. >>> More
Keywords Optimization For Website Optimization

as seen on Ezine Articles - Search for 'Ezine Articles'
Saying that you need to do website optimization sounds like saying you need to get healthy. To get healthy we do 2 things: diet management and exercise. Lets start with diet management. Keywords are like food for your WebPages. This article explains the role of keywords in website optimization. >>> More
The Expert Secret to Search Engine Optimization - Effective Website Optimization

as seen on Ezine Articles - Search for 'Ezine Articles'
Throwing keywords into a program that shows you how popular they are and then using those keywords without doing a little bit of preliminary research and answering some very important questions can just spell disaster. There are three questions that are extremely important to ask yourself before just… >>> More
Importance of On-Page Optimization in Search Engine Optimization (SEO)

as seen on Ezine Articles - Search for 'Ezine Articles'
On-page optimization is crucial for any website. Read the importance of on-page optimization and how it would be helpful for getting high ranking in search engines. >>> More

Related posts about lucene.net

Good Lucene .NET alternative for ASP.NET website

as seen on Stack Overflow - Search for 'Stack Overflow'
Are there any good alternatives for Lucene .NET to use in a ASP.NET website? I want to index XML-, TXT-, PDF- and DOC-files. Thanks! >>> More
Where does lucene .net cache the search results?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I'm trying to figure out where Lucene stores the cached query results, and how it's configured to do so - and how long it caches for. This is for an ASP.NET 3.5 solution. I'm getting this problem: If I run a search and sort the result by a particular product field, it seems to work the very… >>> More
Lucene .Net Searching with TermVector

as seen on Stack Overflow - Search for 'Stack Overflow'
in Lucene.Net,i am creating the document for searching a word and want to display before 10 words and after 10 words.i have used TermVector. Lucene.Net.Documents.Field fldContent = new Lucene.Net.Documents.Field("content", content, Lucene.Net.Documents.Field.Store.YES, Lucene… >>> More
Lucene.Net support phrases?: What is best approach to tokenize comma-delimited data (atomically) in

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a database with a column I wish to index that has comma-delimited names, e.g., User.FullNameList = "Helen Ready, Phil Collins, Brad Paisley" I prefer to tokenize each name atomically (name as a whole searchable entity). What is the best approach for this? Did I miss a simple option to… >>> More
Lucene.NET vs SQL Server Full-text – Generating a million records and a full-text index

as seen on Dot net Slackers - Search for 'Dot net Slackers'
In this article we will take a look at how SQL Server performs with one million records in a table. We will create a quick data pumper program to fill up a table with a million dynamically created rows of data. From there we will take a look at querying the data without any special optimizations… >>> More