Efficient alternatives to merge for larger data.frames R

Posted by Etienne Low-Décarie on Stack Overflow See other posts from Stack Overflow or by Etienne Low-Décarie
Published on 2012-06-21T21:22:58Z Indexed on 2012/06/22 9:16 UTC
Read the original article Hit count: 229

Filed under:

r

|

merge

|

data.frame

|

plyr

|

data.table

I am looking for an efficient (both computer resource wise and learning/implementation wise) method to merge two larger (size>1 million / 300 KB RData file) data frames.

"merge" in base R and "join" in plyr appear to use up all my memory effectively crashing my system.

Example
load test data frame

and try

test.merged<-merge(test, test)

or

test.merged<-join(test, test, type="all")

-

The following post provides a list of merge and alternatives:
How to join data frames in R (inner, outer, left, right)?

The following allows object size inspection:
https://heuristically.wordpress.com/2010/01/04/r-memory-usage-statistics-variable/

Data produced by anonym

Related posts about merge

Foreach PHP Error

as seen on Stack Overflow - Search for 'Stack Overflow'
I am receiving the following foreach error on my PHP file and I have no idea how to fix it. Does anyone have any ideas? When I load the page I get this: Warning: Invalid argument supplied for foreach() in /home/mysite/public_html/merge/class/global_functions.php on line 61 Warning: Invalid argument… >>> More
TFS 2010 SDK: Smart Merge - Programmatically Create your own Merge Tool

as seen on Geeks with Blogs - Search for 'Geeks with Blogs'
Technorati Tags: Team Foundation Server 2010,TFS SDK,TFS API,TFS Merge Programmatically,TFS Work Items Programmatically,TFS Administration Console,ALM The information available in the Merge window in Team Foundation Server 2010 is very important in the decision making during the merging process… >>> More
mail merge e-mail in Word 2007 with attachment

as seen on Super User - Search for 'Super User'
Is there a simple way to mail merge with Word 2007 and add an attachment? I've searched google, and all results point to pasting in VB code. I want to a small team of novice users to be able to mail merge e-mails and add attachments. Does anyone know a simple way of doing this without code? >>> More
Improving Partitioned Table Join Performance

as seen on SQL Blog - Search for 'SQL Blog'
The query optimizer does not always choose an optimal strategy when joining partitioned tables. This post looks at an example, showing how a manual rewrite of the query can almost double performance, while reducing the memory grant to almost nothing. Test Data The two tables in this example use… >>> More
Simple Branching and Merging with SVN

as seen on Dot net Slackers - Search for 'Dot net Slackers'
Its a good idea not to do too much work without checking something into source control. By too much work I mean typically on the order of a couple of hours at most, and certainly its a good practice to check in anything you have before you leave the office for the day. But what if your… >>> More

Developer IT

Efficient alternatives to merge for larger data.frames R - Developer IT

Efficient alternatives to merge for larger data.frames R

r

merge

data.frame

plyr

data.table

Related posts about r

Related posts about merge

Foreach PHP Error

TFS 2010 SDK: Smart Merge - Programmatically Create your own Merge Tool

mail merge e-mail in Word 2007 with attachment

Improving Partitioned Table Join Performance

Simple Branching and Merging with SVN

Categories cloud