How to remove large number of files/folders in linux

Posted by user1745713 on Super User See other posts from Super User or by user1745713
Published on 2013-10-29T15:49:35Z Indexed on 2013/10/29 15:59 UTC
Read the original article Hit count: 242

Filed under:

linux

|

delete

|

nfs

|

hadoop

We are using hadoop to split a table into smaller files to feed to mahout, but in the process, we created a huge amount of _temporary logs.

we have an nfs mount for the hadoop volume so we can use all the linux commands to delete folders files, but we just can't get them to be deleted, here's what I've tried so far:

hadoop fs -rmr /.../_temporary  : hangs for hours and does nothing

on nfs mount:

rmr -rf /.../_temporary :hangs for hours and does nothing

find . -name '*.*' -type f -delete : same as above

the folders look like this (38 of these folders inside _temporary):

drwxr-xr-x 319324 user user 319322 Oct 24 12:12 _attempt_201310221525_0404_r_000000_0

the content of these are actually folders, not files. each one of those 319322 folders has exactly one file inside. not sure why the do the logging this way.

Any help is appreciated.

© Super User or respective owner

Related posts about linux

apt-get install and update fail

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I've got a problem with apt-get update and apt-get install ... commands . every time update or installing fails and errors are : Get:1 http://dl.google.com stable Release.gpg [198B] Ign http://dl.google.com/linux/chrome/deb/ stable/main Translation-en_US Get:2 http://dl… >>> More
kernel module compiling error

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
sh@ubuntu:/home/ccpp/helloworld$ make gcc-4.6 -O2 -DMODULE -D_KERNEL_ -W -Wall -Wstrict-prototypes -Wmissing-prototypes -isystem /lib/modules/`uname -r`/build/include -c -o hello-1.o hello-1.c hello-1.c:4:0: warning: "MODULE" redefined [enabled by default] <command-line>:0:0: note: this is… >>> More
Build-Essentials installation failing

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I am having trouble accessing the several critical header files that show to be a part of the build process. The "Ubuntu Software Center" shows "Build Essentials" as installed: Next I did the following two commands, which did not improve the problem: ~$ sudo apt-get install build-essential [sudo]… >>> More
Updating Debian kernel

as seen on Super User - Search for 'Super User'
I'm trying to update my Debian machine to 2.6.32-46 (which is the new stable). However, after doing apt-get update my apt-cache search linux-image shows me: linux-headers-2.6.32-5-486 - Header files for Linux 2.6.32-5-486 linux-headers-2.6.32-5-686-bigmem - Header files for Linux 2.6.32-5-686-bigmem linux-headers-2… >>> More
Serial connection over a single USB cable (Windows to linux, or linux to linux)

as seen on Server Fault - Search for 'Server Fault'
I'm helping out with a project for an embedded device that only has USB and no serial. This device is running Linux. These days, when we need to connect to a serial port on a device we typically use a USB to serial adapter (on something like a phone system or a load balancing device, etc). I would… >>> More

Related posts about delete

?????create or replace???PL/SQL??

as seen on Oracle Blogs - Search for 'Oracle Blogs'
????T.Askmaclean.com?????10gR2??????procedure,?????????create or replace ??????????????????,????Oracle???????????????????procedure? ??Maclean ??2?10gR2???????????PL/SQL?????: ??1: ??Flashback Query ????,?????????????flashback database,??????????create or replace???SQL??source$????… >>> More
new[n] and delete every location with delete instead the whole chunk with delete[]

as seen on Stack Overflow - Search for 'Stack Overflow'
Is this valid C++ (e.g. not invoking UB) and does it achieve what I want without leaking memory? valgrinds complains about mismatching free and delete but says "no leaks are possible" in the end. int main() { int* a = new int[5]; for(int i = 0; i < 5; ++i) a[i] = i; for(int i = 0;… >>> More
flex combobox backspace or delete key does not delete highlighted text

as seen on Stack Overflow - Search for 'Stack Overflow'
Context: I am implementing a flex auto-suggest combobox - as the user types in each character: Consider the string 'Stackoverflow' and user input = 'st' 1) the data provider is filtered to show all items starting with 'st' 2) text is set to auto-suggest string such that the un-typed part is highlighted… >>> More
My delete function does not delete the targeted file

as seen on Stack Overflow - Search for 'Stack Overflow'
Basically I could upload files based on a project. Whenever I create a project, a new directory is created with the directory name as the project_name e.g. this is a test - this-is-a-test. But my problem is I couldn't delete a file in a directory. function delete_image($id) { $this->load->model(array('work_model'… >>> More
Delete a div id using ajax and jquery and delete from DB

as seen on Stack Overflow - Search for 'Stack Overflow'
I've got several div id's, each containing a different client. I want to be able to click the delete button and using ajax and jquery delete the specific div from the database. I'm getting success in AJAX but it's not deleting anything from the DB. And then obviously, upon deletion, I would like… >>> More