Simple implementation of N-Gram, tf-idf and Cosine similarity in Python

Posted by seanieb on Stack Overflow See other posts from Stack Overflow or by seanieb
Published on 2010-03-04T15:22:30Z Indexed on 2010/05/02 17:27 UTC
Read the original article Hit count: 907

Filed under:

python

|

document

|

similarity

|

vsm

I need to compare documents stored in a DB and come up with a similarity score between 0 and 1.

The method I need to use has to be very simple. Implementing a vanilla version of n-grams (where it possible to define how many grams to use), along with a simple implementation of tf-idf and Cosine similarity.

Is there any program that can do this? Or should I start writing this from scratch?

© Stack Overflow or respective owner

Related posts about python

unmet dependencies in Ubuntu 12.04

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I tried today to install a dvb-card on my Ubuntu 12.04 (Linux blauhai-linux 3.2.0-25-generic #40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux ). The installation failed with an error. After that, i tried to install python (it was already installed but i got this error): linux:~$… >>> More
How can I get sikuli-ide to work?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I installed sikuli-ide with sudo apt-get install sikuli-ide Everything was fine until I tried to start it from the terminal. I typed sikuli-ide But the only response I got was [info] locale: en_US The application was not started, furthermore there is no desktop file and sikuli-ide does not… >>> More
Getting PATH right for python after MacPorts install

as seen on Super User - Search for 'Super User'
I can't import some python libraries (PIL, psycopg2) that I just installed with MacPorts. I looked through these forums, and tried to adjust my PATH variable in $HOME/.bash_profile in order to fix this but it did not work. I added the location of PIL and psycopg2 to PATH. I know that Terminal is… >>> More
call python with system() in R to run a python script emulating the python console

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to pass a chunk of Python code to Python in R with something like system('python ...'), and I'm wondering if there is an easy way to emulate the python console in this case. For example, suppose the code is "print 'hello world'", how can I get the output like this in R? >>> print… >>> More
Python - Calling a non python program from python?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am currently struggling to call a non python program from a python script. I have a ~1000 files that when passed through this C++ program will generate ~1000 outputs. Each output file must have a distinct name. The command I wish to run is of the form: program_name -input -output -o1 -o2… >>> More

Related posts about document

Document, Document, Document

as seen on SQL Blog - Search for 'SQL Blog'
A while back I blogged about using Checklists , but there's another task you want to incorporate into your workflow - documentation. Now, I'm not just talking about documenting the logic, system flow and data and structure changes, I'm also talking about documenting your daily activities (commonly… >>> More
How can I do two columns in OpenOffice.org Writer?

as seen on Super User - Search for 'Super User'
I would like to structure my document with two columns in the middle of the page, and one column in the top and in the bottom. Like my example below. How can I do two columns like this in OpenOffice.org Writer? My document My document My document My document My document My document My document My… >>> More
how to check-out document in document library programmatically in sharepoint

as seen on Stack Overflow - Search for 'Stack Overflow'
how to set a document in check-in/out mode programmatically by using webservices or objetmodel in sharepoint >>> More
How to improve performance of map that loads new overlay images

as seen on Pro Webmasters - Search for 'Pro Webmasters'
I have inherited a website to maintain that uses a html map overlaying a real map to link specific countries to specific pages. previously it loaded the default map image, then with some javascript it would change the image src to an image with that particular country in a different colour on mouseover… >>> More
document.all vs. document.getElementById

as seen on Stack Overflow - Search for 'Stack Overflow'
When should you use document.all vs. document.getElementById What is your answer? >>> More