KMeans clustering for more than 5 million vectors

Posted by Wajih on Stack Overflow See other posts from Stack Overflow or by Wajih
Published on 2010-08-04T08:54:12Z Indexed on 2011/01/09 12:53 UTC
Read the original article Hit count: 258

Filed under:

algorithm

|

clustering

I have hit a real problem. I need to do some Kmeans clustering for 5 million vectors, each containing about 32 cols. I tried out Mahout which requires linux and I am on windows, I am restrained from using a Linux OS and any sort of simulator.

Can anyone suggest a KMeans clustering algorithm that is scalable upto 5M vectors and can converge quickly?

I have tested a few but they wont scale. Which means they are slow and take forever to complete.

Thanks

© Stack Overflow or respective owner

Related posts about algorithm

Jpeg Algorithm vs BMP Algorithm?

as seen on Super User - Search for 'Super User'
I'm just wonder, what the differences are between creating a BMP file algorithm and JPG file algorithm ? If you know the others images' format algorithm, please post them. Thanks. >>> More
word disambiguation algorithm (Lesk algorithm)

as seen on Stack Overflow - Search for 'Stack Overflow'
Hii.. Can anybody help me to find an algorithm in Java code to find synonyms of a search word based on the context and I want to implement the algorithm with WordNet database. For example, "I am running a Java program". From the context, I want to find the synonyms for the word "running", but the… >>> More
Search algorithm (with a sort algorithm already implemented)

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, Im doing a Java application and Im facing some doubts in which concerns performance. I have a PriorityQueue which guarantees me the element removed is the one with greater priority. That PriorityQueue has instances of class Event (which implements Comparable interface). Each Event is associated… >>> More
Is there any algorithm for finding LINES by PIXEL COLORS on picture?

as seen on Stack Overflow - Search for 'Stack Overflow'
So I have Image like this I want to get something like this (I hevent drawn all lines I want but I hope you can get my idea) I need algorithm for finding all straight lines on it by just reading colors of pixels. No hard math, no Haar, no Hough. Some algorithm which would be based on points… >>> More
collsion issues with quadtree [on hold]

as seen on Game Development - Search for 'Game Development'
So i implemented a Quad tree in Java for my 2D game and everything works fine except for when i run my collision detection algorithm, which checks if a object has hit another object and which side it hit.My problem is 80% of the time the collision algorithm works but sometimes the objects just go… >>> More

Related posts about clustering

agglomerative clustering java

as seen on Stack Overflow - Search for 'Stack Overflow'
Is there any java file that I can use to perform "agglomerative clustering" Result should provide me every level nodes id help................. >>> More
Clustering for Mere Mortals (Pt2)

as seen on SQL Team - Search for 'SQL Team'
Planning. I could stop there and let that be the entirety post #2 in this series. Planning is the single most important element in building a cluster and the Laptop Demo Cluster is no exception. One of the more awkward parts of actually creating a cluster is coordinating information between Windows… >>> More
MySQL Clustering in a Sandbox

as seen on Internet.com - Search for 'Internet.com'
MySQL's unique architecture allows for plugin storage engines. There is the MyISAM storage engine, the ARCHIVE storage engine and the InnoDB storage engine; so it makes sense then that MySQL's clustering solution involves a storage engine as well, namely the NDB (Network DataBase) storage engine. >>> More
Microsoft SQL Server High-Availability Videos and Q&A Log

as seen on SQL Blog - Search for 'SQL Blog'
You Want Videos? We Got Videos! I always enjoy getting the chance to catch up with author, consultant, and Microsoft Clustering MVP Allan Hirt . Allan and I recently presented two sessions covering an overview of high availability in Microsoft SQL Server and, the following week, a demo of how to implement… >>> More
I need advice about iscsi + zfs(or ntfs) + windows 2008 clustering

as seen on Server Fault - Search for 'Server Fault'
I want to setup a storage farm with iSCSI. I have 2 cluster node machine, 1 iscsi target machine that has 8TB installed as RAID 10. The capacity is now 8TB, but I'll upgrade the capacity in future. Let's say, I installed clusters as file server, and I connected these servers to iscsi target, then… >>> More