k-means clustering in R on very large, sparse matrix?

Posted by movingabout on Stack Overflow See other posts from Stack Overflow or by movingabout
Published on 2010-06-14T18:03:36Z Indexed on 2010/06/14 18:12 UTC
Read the original article Hit count: 227

Filed under:

r

|

clustering

|

large

|

sparse-matrix

Hello,

I am trying to do some k-means clustering on a very large matrix.

The matrix is approximately 500000 rows x 4000 cols yet very sparse (only a couple of "1" values per row).

The whole thing does not fit into memory, so I converted it into a sparse ARFF file. But R obviously can't read the sparse ARFF file format. I also have the data as a plain CSV file.

Is there any package available in R for loading such sparse matrices efficiently? I'd then use the regular k-means algorithm from the cluster package to proceed.

Many thanks

Related posts about clustering

agglomerative clustering java

as seen on Stack Overflow - Search for 'Stack Overflow'
Is there any java file that I can use to perform "agglomerative clustering" Result should provide me every level nodes id help................. >>> More
Clustering for Mere Mortals (Pt2)

as seen on SQL Team - Search for 'SQL Team'
Planning. I could stop there and let that be the entirety post #2 in this series. Planning is the single most important element in building a cluster and the Laptop Demo Cluster is no exception. One of the more awkward parts of actually creating a cluster is coordinating information between Windows… >>> More
MySQL Clustering in a Sandbox

as seen on Internet.com - Search for 'Internet.com'
MySQL's unique architecture allows for plugin storage engines. There is the MyISAM storage engine, the ARCHIVE storage engine and the InnoDB storage engine; so it makes sense then that MySQL's clustering solution involves a storage engine as well, namely the NDB (Network DataBase) storage engine. >>> More
Microsoft SQL Server High-Availability Videos and Q&A Log

as seen on SQL Blog - Search for 'SQL Blog'
You Want Videos? We Got Videos! I always enjoy getting the chance to catch up with author, consultant, and Microsoft Clustering MVP Allan Hirt . Allan and I recently presented two sessions covering an overview of high availability in Microsoft SQL Server and, the following week, a demo of how to implement… >>> More
I need advice about iscsi + zfs(or ntfs) + windows 2008 clustering

as seen on Server Fault - Search for 'Server Fault'
I want to setup a storage farm with iSCSI. I have 2 cluster node machine, 1 iscsi target machine that has 8TB installed as RAID 10. The capacity is now 8TB, but I'll upgrade the capacity in future. Let's say, I installed clusters as file server, and I connected these servers to iscsi target, then… >>> More

Developer IT