Package to compare LSA, TFIDF, Cosine metrics and Language Models

Posted by gouwsmeister on Stack Overflow See other posts from Stack Overflow or by gouwsmeister
Published on 2009-10-12T21:12:48Z Indexed on 2010/05/02 17:58 UTC
Read the original article Hit count: 301

Filed under:

document

|

similarity

|

string-similarity

Hi,

I'm looking for a package (any language, really) that I can use on a corpus of 50 documents to perform interdocument similarity testing in various metrics, like tfidf, okapi, language models, lsa, etc.

I want as a result a document similarity matrix, i.e. doc1 is x% similar to doc2, etc... This is for research purposes, not for production. I specifically want the doc similarity matrix as I want to correlate this with human ratings.

Thank you in advance!

© Stack Overflow or respective owner

Related posts about document

Document, Document, Document

as seen on SQL Blog - Search for 'SQL Blog'
A while back I blogged about using Checklists , but there's another task you want to incorporate into your workflow - documentation. Now, I'm not just talking about documenting the logic, system flow and data and structure changes, I'm also talking about documenting your daily activities (commonly… >>> More
How can I do two columns in OpenOffice.org Writer?

as seen on Super User - Search for 'Super User'
I would like to structure my document with two columns in the middle of the page, and one column in the top and in the bottom. Like my example below. How can I do two columns like this in OpenOffice.org Writer? My document My document My document My document My document My document My document My… >>> More
how to check-out document in document library programmatically in sharepoint

as seen on Stack Overflow - Search for 'Stack Overflow'
how to set a document in check-in/out mode programmatically by using webservices or objetmodel in sharepoint >>> More
How to improve performance of map that loads new overlay images

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I have inherited a website to maintain that uses a html map overlaying a real map to link specific countries to specific pages. previously it loaded the default map image, then with some javascript it would change the image src to an image with that particular country in a different colour on mouseover… >>> More
document.all vs. document.getElementById

as seen on Stack Overflow - Search for 'Stack Overflow'
When should you use document.all vs. document.getElementById What is your answer? >>> More

Related posts about similarity

Find cosine similarity in R

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm wondering if there is a built in function in R that can find the cosine similarity (or cosine distance) between two arrays? Currently, I implemented my own function, but I can't help but think that R should already come with one :) Thanks, Derek >>> More
fast similarity detection

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a large collection of objects and I need to figure out the similarities between them. To be exact: given two objects I can compute their dissimilarity as a number, a metric - higher values mean less similarity and 0 means the objects have identical contents. The cost of computing this number… >>> More
Lucene numDocs and doqFreq on custom similarity class

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi All, im doing an aplication with Lucene (im a noob with it) and im facing some problems. My aplication uses the Lucene 2.4.0 library with a custom similaraty implementation (the jar is imported) In my app im calculating doqFreq and numDocs manually (im adding the values of all indexes and then… >>> More
Document Similarity: Comparing two documents efficiently

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a loop that calculates the similarity between two documents. It collects all the tokens in a document and their scores, and places them in dictionary. It then compares the dictionaries This is what I have so far, it works, but is super slow: # Doc A cursor1.execute("SELECT token, tfidf_norm… >>> More
Advice on String Similarity Metrics (Java). Distance, sounds like or combo?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, A part of a process requires to apply String Similarity Algorithms. The results of this process will be stored and produce lets say SS_Dataset. Based on this Dataset, further decisions will have to be made. My questions are: Should i apply one or more string similarity algorithms… >>> More