Best way to do text processing in linux/mac ?

Posted by euphoria83 on Stack Overflow See other posts from Stack Overflow or by euphoria83
Published on 2010-03-15T06:43:56Z Indexed on 2010/03/15 6:49 UTC
Read the original article Hit count: 263

Filed under:

text-processing

|

shell

|

perl

I generally need to do a fair amount of text processing for my research, such as removing the last token from all lines, extracting the first 2 tokens from each line, splitting each line into tokens, etc.

What is the best way to perform this ? Should I learn Perl for this? Or should I learn some kind of shell commands? The main concern is speed. If I need to write long code for such stuff, it defeats the purpose.

© Stack Overflow or respective owner

Related posts about text-processing

Performing text processing on flatpage content to include handling of custom tag

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi. I'm using flatpages app in my project to manage some html content. That content will include images, so I've made a ContentImage model allowing user to upload images using admin panel. The user should then be able to include those images in content of the flatpages. He can of course do that by… >>> More
Fuzzy Regex, Text Processing, Lexical Analysis?

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm not quite sure what terminology to search for, so my title is funky... Here is the workflow I've got: Semi-structured documents are scanned to file. The files are OCR'd to text. The text is parsed into Python objects The objects are serialized (to SQL, JSON, whatever) for use. The documents… >>> More
Getting started with character and text processing (encoding, regular expressions)

as seen on Stack Overflow - Search for 'Stack Overflow'
I'd like to learn foundations of encodings, characters and text. Understanding these is important for dealing with a large set of text whether that are log files or text source for building algorithms for collective intelligence. My current knowledge is pretty basic: something like "As long as I use… >>> More
Improve Efficiency for This Text Processing Code

as seen on Stack Overflow - Search for 'Stack Overflow'
I am writing a program that counts the number of words in a text file which is already in lowercase and separated by spaces. I want to use a dictionary and only count the word IF it's within the dictionary. The problem is the dictionary is quite large (~100,000 words) and each text document has also… >>> More
Tools for modelling data and workflows using structured text files

as seen on Programmers - Search for 'Programmers'
Consider a case when I want to try some idea of an application. But I want to avoid investing a lot of effort in coding UI/work flows/database schema etc before I see that it's going to be useful to me (as example of potential user). My idea is stay lightweight and put all the data in text files.… >>> More

Related posts about shell

How to restrict the users' shell allowing to execute shell programs

as seen on Server Fault - Search for 'Server Fault'
Is it possible to prevent any user to not use commands like ls, rm and other system commands which could harm the system. But the users should be able to execute shell programs. >>> More
Shell extension installation not recognized by Windows 7 64-bit shell

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a Copy Hook Handler shell extension that I'm trying to install on Windows 7 64-bit. The shell extension DLL is compiled in two separate versions for 32-bit and 64-bit Windows. The DLL implements DLLRegisterServer which adds the necessary registry entries. After adding the registry entries… >>> More
Running shell commands without a shell window

as seen on Stack Overflow - Search for 'Stack Overflow'
With either subprocess.call or subprocess.Popen, executing a shell command makes a shell window quicky appear and disappear. How can I run the shell command without the shell window? >>> More
Why can't I reinstall MySQL?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I've been looking all around the Internet for an answer but didn't find anything. I hope you can help me now. I have a server with MySQL. From one day to another, MySQL didn't let me enter with my root password anymore (accsess denied for user 'root'@'localhost' using password: 'YES'). So I tried… >>> More
Bash/shell script - shell output redirection inside a function

as seen on Stack Overflow - Search for 'Stack Overflow'
function grabSourceFile { cd /tmp/lmpsource wget $1 > $LOG baseName=$(basename $1) tar -xvf $baseName > $LOG cd $baseName } When I call this function The captured output is not going to the log file. The output redirection works fine until I call the… >>> More