Search Results

Search found 5 results on 1 pages for 'kenfar'.

Page 1/1 | 1 

• Unix sort keys cause performance problems

  - by KenFar

  My data: It's a 71 MB file with 1.5 million rows. It has 6 fields All six fields combine to form a unique key - so that's what I need to sort on. Sort statement: sort -t ',' -k1,1 -k2,2 -k3,3 -k4,4 -k5,5 -k6,6 -o output.csv input.csv The problem: If I sort without keys, it takes 30 seconds. If I sort with keys, it takes 660 seconds. I…

  Read the article

• File Sync Solution for Batch Processing (ETL)

  - by KenFar

  I'm looking for a slightly different kind of sync utility - not one designed to keep two directories identical, but rather one intended to keep files flowing from one host to another. The context is a data warehouse that currently has a custom-developed solution that moves 10,000 files a day, some of which are 1+ gbytes gzipped files, between…

  Read the article

• Unix sort 10x slower with keys specified

  - by KenFar

  My data: It's a 71 MB file with 1.5 million rows. It has 6 fields, four of which are strings of avg. 15 characters, two are integers. Three of the fields are sometimes empty. All six fields combine to form a unique key - and that's what I need to sort on. Sort statement: sort -t ',' -k1,1 -k2,2 -k3,3 -k4,4 -k5,5 -k6,6 -o a_out.csv…

  Read the article

• Best way to test instance methods without running __init__

  - by KenFar

  I've got a simple class that gets most of its arguments via init, which also runs a variety of private methods that do most of the work. Output is available either through access to object variables or public methods. Here's the problem - I'd like my unittest framework to directly call the private methods called by init with different…

  Read the article

• Best way to test class methods without running __init__

  - by KenFar

  I've got a simple class that gets most of its arguments via init, which also runs a variety of private methods that do most of the work. Output is available either through access to object variables or public methods. Here's the problem - I'd like my unittest framework to directly call the private methods called by init with different…

  Read the article

1