Get highest frequency terms from Lucene index

Posted by Julia on Stack Overflow See other posts from Stack Overflow or by Julia
Published on 2010-05-12T19:00:40Z Indexed on 2010/05/12 19:04 UTC
Read the original article Hit count: 321

Filed under:

lucene

|

index

|

full-text-search

|

java

|

frequency

Hello!

i need to extract terms with highest frequencies from several lucene indexes, to use them for some semantic analysis.

So, I want to get maybe top 30 most occuring terms(still did not decide on threshold, i will analyze results) and their per-index counts. I am aware that I might lose some precision because of potentionally dropped duplicates, but for now, lets say i am ok with that.

So for the proposed solutions, (needless to say maybe) speed is not important, since I would do static analysis, I would put accent on simplicity of implementation because im not so skilled with Lucene (not the programming guru too :/ ) and cant wrap my mind around many concepts of it..

I can not find any code samples from something similar, so all concrete advices (code, pseudocode, links to code samples...) I will apretiate very much!!!

Thank you!

© Stack Overflow or respective owner

Related posts about lucene

performance comparision between Zend Lucene and Java Lucene

as seen on Stack Overflow - Search for 'Stack Overflow'
Zend Lucene and Java Lucene are built in PHP and java repectively, and PHP language has a higher level than java. Just wondering How big the performance difference among these two, regarding to index building and data searching? Is it much more effective to let java create and rebuild index, and… >>> More
Why wasn't fast-vector-highlighter (lucene-contrib) made an official part of Lucene 3.0 core

as seen on Stack Overflow - Search for 'Stack Overflow'
I've read some Jira entries and they mentioned moving fast-vector-highlighter to core about a year ago but it never made it. Looking at the svn for contrib it seems incomplete. There are no tests for FastVectorHighlighter Documentation is lacking No samples anywhere on apache.org Anyone have… >>> More
pylucene: install error

as seen on Stack Overflow - Search for 'Stack Overflow'
I am trying to install Pylucene (pylucene-3.3-3-src.tar.gz) on my ubuntu linux 11.10. I have python 2.7.2. I was able to compile JCC (I think) because I didnt see any error when I installed it. When I tried to install Pylucene I get the following error. Can someone help? Thanks. ICU not installed /usr/bin/python… >>> More
Solr WordDelimiterFilter + Lucene Highlighter

as seen on Stack Overflow - Search for 'Stack Overflow'
I am trying to get the Highlighter class from Lucene to work properly with tokens coming from Solr's WordDelimiterFilter. It works 90% of the time, but if the matching text contains a ',' such as "1,500" the output is incorrect: Expected: 'test 1,500 this' Observed: 'test 11,500 this' I… >>> More
java AbstractMethodError

as seen on Stack Overflow - Search for 'Stack Overflow'
How to handle this error in lucene: java.lang.AbstractMethodError: org.apache.lucene.store.Directory.listAll()[Ljava/lang/String; at org.apache.lucene.index.SegmentInfos$FindSegmentsFile.run(SegmentInfos.java:568) at org.apache.lucene.index.DirectoryReader.open(DirectoryReader.java:69) … >>> More

Related posts about index

NTFS Corruption: Files created in Linux corrupted when Windows Boots

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I'm getting some file loss and corruption on my Win7/Ubuntu 12.04 dual boot setup. I have a large shared NTFS partition. I have my Windows Docs/Music/etc. directories on that file and have the comparable directors in Linux setup as a sym. link. I'm using ntfs-3g on the linux side of things to… >>> More
SQL SERVER – Force Index Scan on Table – Use No Index to Retrieve the Data – Query Hint

as seen on SQL Authority - Search for 'SQL Authority'
Recently I received the following two questions from readers and both the questions have very similar answers. Question 1: I have a unique requirement where I do not want to use any index of the table; how can I achieve this? Question 2: Currently my table uses clustered index and does seek operation;… >>> More
Covering Index versus Clustered Index (Database Index)

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I'm working on a database system and it's indexes, but I'm having a really hard time seing the clear difference between a covering index and a clustered index. I've googled my way around but hasn't got a clear cut answer on: What is the differences between the two types of indexes When do… >>> More
SQL SERVER Force Index Scan on Table Use No Index to Retrieve the Data Query Hint

as seen on Dot net Slackers - Search for 'Dot net Slackers'
Recently I received the following two questions from readers and both the questions have very similar answers.Question 1: I have a unique requirement where I do not want to use any index of the table; how can I achieve this?Question 2: Currently my table uses clustered index and does seek operation;… >>> More
Custom SNMP Cacti Data Source fails to update

as seen on Server Fault - Search for 'Server Fault'
I'm trying to create a custom SNMP datasource for Cacti but despite everything I can check being correct, it is not creating the rrd file, or updating it even when I create it. Other, standard SNMP sources are working correctly so it's not SNMP or permissions that are the problem. I've created a… >>> More