What is the best, python or bash for selectively concatenating lots of files?

Posted by Werner on Stack Overflow See other posts from Stack Overflow or by Werner
Published on 2010-03-12T17:47:59Z Indexed on 2010/03/12 18:07 UTC
Read the original article Hit count: 153

Filed under:

bash

|

python

Hi,

I have around 20000 files coming from the output of some program, and their names follow the format:

data1.txt
data2.txt
...
data99.txt
data100.txt
...
data999.txt
data1000.txt
...
data20000.txt

I would like to write a script that gets as input argument the number N. Then it makes blocks of N concatenated files, so if N=5, it would make the following new files:

data_new_1.txt: it would contain (concatenated) data1.txt to data5.txt (like cat data1.txt data2.txt ...> data_new_1.txt ) data_new_2.txt: it would contain (concatenated) data6.txt to data10.txt .....

I wonder what do you think would be the best approach to do this, whether bash, python or another one like awk, perl, etc.

The best approach I mean in terms of simplest code.

Thanks

© Stack Overflow or respective owner

Related posts about bash

launching a program from bash causes bash to go to new prompt

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
When I run a program from the console, e.g. me@box:~$ firefox I expect the console to log error messages (I think this is std out or std err?) and other items from the program, firefox in this case. But today I notice that bash just opens the program and goes to a new prompt, e.g. me@box:~$… >>> More
How to debug a .bash_profile

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I was updating my .bash_profile, and unfortunetly I made a few updates and now I am getting: env: bash: No such file or directory env: bash: No such file or directory env: bash: No such file or directory env: bash: No such file or directory env: bash: No such file or directory -bash: tar: command… >>> More
Every command fails with "command not found" after changing .bash_profile?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I was updating my .bash_profile, and unfortunetly I made a few updates and now I am getting: env: bash: No such file or directory env: bash: No such file or directory env: bash: No such file or directory env: bash: No such file or directory env: bash: No such file or directory -bash: tar: command… >>> More
Is there any fundamental difference between piping in mac and linux?

as seen on Super User - Search for 'Super User'
ps -e | grep bash sample output from a linux machine: 1128 pts/14 00:00:00 bash 7491 pts/7 00:00:00 bash 12651 pts/14 00:00:00 bash 16145 pts/2 00:00:00 bash sample output from a mac machine: 58352 ttys000 0:00.09 login -pfl username /bin/bash -c exec -la bash /bin/bash 58353 ttys000… >>> More
why is $0 set to -bash?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
First login process name seems to be set to "-bash", but if I subshell then it becomes "bash". for example: root@nowere:~# echo $0 -bash root@nowere:~# bash root@nowere:~# echo $0 bash -bash is causing some scripts to fail, such as . /usr/share/debconf/confmodule exec /usr/share/debconf/frontend… >>> More

Related posts about python

unmet dependencies in Ubuntu 12.04

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I tried today to install a dvb-card on my Ubuntu 12.04 (Linux blauhai-linux 3.2.0-25-generic #40-Ubuntu SMP Wed May 23 20:30:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux ). The installation failed with an error. After that, i tried to install python (it was already installed but i got this error): linux:~$… >>> More
How can I get sikuli-ide to work?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I installed sikuli-ide with sudo apt-get install sikuli-ide Everything was fine until I tried to start it from the terminal. I typed sikuli-ide But the only response I got was [info] locale: en_US The application was not started, furthermore there is no desktop file and sikuli-ide does not… >>> More
Getting PATH right for python after MacPorts install

as seen on Super User - Search for 'Super User'
I can't import some python libraries (PIL, psycopg2) that I just installed with MacPorts. I looked through these forums, and tried to adjust my PATH variable in $HOME/.bash_profile in order to fix this but it did not work. I added the location of PIL and psycopg2 to PATH. I know that Terminal is… >>> More
call python with system() in R to run a python script emulating the python console

as seen on Stack Overflow - Search for 'Stack Overflow'
I want to pass a chunk of Python code to Python in R with something like system('python ...'), and I'm wondering if there is an easy way to emulate the python console in this case. For example, suppose the code is "print 'hello world'", how can I get the output like this in R? >>> print… >>> More
Python - Calling a non python program from python?

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am currently struggling to call a non python program from a python script. I have a ~1000 files that when passed through this C++ program will generate ~1000 outputs. Each output file must have a distinct name. The command I wish to run is of the form: program_name -input -output -o1 -o2… >>> More