More efficient way to find & tar millions of files

Posted by Stu Thompson on Stack Overflow See other posts from Stack Overflow or by Stu Thompson
Published on 2010-04-23T08:40:43Z Indexed on 2010/04/23 8:43 UTC
Read the original article Hit count: 249

Filed under:

bash

|

find

|

tar

I've got a job running on my server at the command line prompt for a two days now:

find data/ -name filepattern-*2009* -exec tar uf 2008.tar {} ;

It is taking forever, and then some. Yes, there are millions of files in the target directory. But just running...

find data/ -name filepattern-*2009* -print > filesOfInterest.txt

...takes only two hours or so. At the rate my job is running, it won't be finished for a couple of weeks.. That seems unreasonable. Is there a more efficient to do this? Maybe with a more complicated bash script?

A secondary questions is "why is my current approach so slow?"

© Stack Overflow or respective owner

Related posts about bash

launching a program from bash causes bash to go to new prompt

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
When I run a program from the console, e.g. me@box:~$ firefox I expect the console to log error messages (I think this is std out or std err?) and other items from the program, firefox in this case. But today I notice that bash just opens the program and goes to a new prompt, e.g. me@box:~$… >>> More
How to debug a .bash_profile

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I was updating my .bash_profile, and unfortunetly I made a few updates and now I am getting: env: bash: No such file or directory env: bash: No such file or directory env: bash: No such file or directory env: bash: No such file or directory env: bash: No such file or directory -bash: tar: command… >>> More
Every command fails with "command not found" after changing .bash_profile?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I was updating my .bash_profile, and unfortunetly I made a few updates and now I am getting: env: bash: No such file or directory env: bash: No such file or directory env: bash: No such file or directory env: bash: No such file or directory env: bash: No such file or directory -bash: tar: command… >>> More
Is there any fundamental difference between piping in mac and linux?

as seen on Super User - Search for 'Super User'
ps -e | grep bash sample output from a linux machine: 1128 pts/14 00:00:00 bash 7491 pts/7 00:00:00 bash 12651 pts/14 00:00:00 bash 16145 pts/2 00:00:00 bash sample output from a mac machine: 58352 ttys000 0:00.09 login -pfl username /bin/bash -c exec -la bash /bin/bash 58353 ttys000… >>> More
why is $0 set to -bash?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
First login process name seems to be set to "-bash", but if I subshell then it becomes "bash". for example: root@nowere:~# echo $0 -bash root@nowere:~# bash root@nowere:~# echo $0 bash -bash is causing some scripts to fail, such as . /usr/share/debconf/confmodule exec /usr/share/debconf/frontend… >>> More

Related posts about find

Microsoft Management Console stops working when I add snap-in to it

as seen on Super User - Search for 'Super User'
I have Windows 7 Ultimate OS. I'm opening mmc.exe as administrator and trying add Certificates or any other snap-in, then while loading that snap-in MMC breaks and displays following message and after that it closes automatically once I click on close button on that message. What could be the problem… >>> More
Visual Studio Find and Replace Regular Expressions ~ find lines with quoted strings, not containing

as seen on Stack Overflow - Search for 'Stack Overflow'
Visual Studio Find and Replace Regular Expressions Find lines with quoted strings, not containing strings include or trace i am tryling to find out all lines in c++ project that contains some text as i have to use visual studio, i have to use its Find and Replace http://www.codinghorror.com/blog/2006/07/the-visual-studio-ide-and-regular-expressions… >>> More
Find, Find Next?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello everyone, I am trying to make a find, find next function for my program, which I did manage to do with this code: int findPos = 0; private void button1_Click(object sender, EventArgs e) { try { string s = textBox1.Text; richTextBox1.Focus(); … >>> More
In jQuery, using the "find' function, what does this expression mean: $(".divName").find('> div')

as seen on Stack Overflow - Search for 'Stack Overflow'
I understand the "find" function, I'm just not familiar with what ' div' means. Can anyone help? >>> More
Find which config file Apache uses to find virtual host

as seen on Super User - Search for 'Super User'
I have a rather complicated Apache setup: some config files have to be manually written and some have to be automatically generated. So far, I've had no problems. But now, I've just tried to set a site at: newsubdomain.mydomain.com and Apache directs it to the data for: www.someotherdomain.com I've… >>> More