unix tool to remove duplicate lines from a file

Posted by Nathan Fellman on Stack Overflow See other posts from Stack Overflow or by Nathan Fellman
Published on 2009-04-14T07:51:59Z Indexed on 2010/06/02 2:43 UTC
Read the original article Hit count: 405

Filed under:

unix

|

command-line

|

duplicates

I have a tool that generates tests and predicts the output. The idea is that if I have a failure I can compare the prediction to the actual output and see where they diverged. The problem is the actual output contains some lines twice, which confuses diff. I want to remove the duplicates, so that I can compare them easily. Basically, something like sort -u but without the sorting.

Is there any unix commandline tool that can do this?

© Stack Overflow or respective owner

Related posts about unix

polkit: disable all users except those in group wheel?

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
Is it possible to do the following using 1 polkit .pkla file? Disable all users except those in the wheel group from using polkit. The users in the wheel group will need to provide the root password when using polkit. /etc/polkit-1/localauthority/50-local.d/wheel-only.pkla [disable all users… >>> More
How to find other end of unix socket connection?

as seen on Server Fault - Search for 'Server Fault'
I have a process (dbus-daemon) which has many open connection over UNIX sockets. One of these connections is fd #36: =$ ps uw -p 23284 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND depesz 23284 0.0 0.0 24680 1772 ? Ss 15:25 0:00 /bin/dbus-daemon --fork… >>> More
Meaning of directories on Unix and Unix like systems

as seen on Server Fault - Search for 'Server Fault'
I've been using Linux for a couple of years now but I still haven't figured out what the origin or meaning of some the directory names are on Unix and Unix like systems. E.g. what does etc stand for or var? Where does the opt name come from? And while we're on the topic anyway. Can someone give a… >>> More
Easy understanding of UNIX and UNIX shell scripting!!

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi All, I am beginner to UNIX,UNIX Shell Scripting. Can you please guide me through any excellent sites for unix(which are easy to understand),some study materials(tutorials),video tutorials. Please help!! Thanks! >>> More
Is Mac OS X a licensed Unix or Unix-like clone that conforms to Unix specification?

as seen on Super User - Search for 'Super User'
Is Mac OS X developed on a licensed Unix or is it a Unix-like clone that, unlike Linux, conforms to Unix specification well enough to be registered as a Unix OS. Not until Leopard, Mac OS X did not gain the Unix certification. But in Leopard, Terminal still print: GNU bash, version 3.2.48(1)-release… >>> More

Related posts about command-line

WMI permissions: Select CommandLine, ProcessId FROM Win32_Process returns no data for CommandLine

as seen on Server Fault - Search for 'Server Fault'
Hi all, I am gathering performance data via WMI and would like to avoid having to use an account in the Administrators group for this purpose. The target machine is running Windows Server 2003 with the latest SP/updates. I've done what I believe to be the appropriate configuration to allow our user… >>> More
WMI permissions: Select CommandLine, ProcessId FROM Win32_Process returns no data for CommandLine

as seen on Server Fault - Search for 'Server Fault'
I am gathering performance data via WMI and would like to avoid having to use an account in the Administrators group for this purpose. The target machine is running Windows Server 2003 with the latest SP/updates. I've done what I believe to be the appropriate configuration to allow our user access… >>> More
Java 1.5 Command Line Password Masking

as seen on Stack Overflow - Search for 'Stack Overflow'
All, Our server is running Java 1.5 and I am having difficulty trying to mask user input from the command line. I am executing a jar file (java -jar my.jar) and am working through command line prompts via printlns. I cannot use Java Console. Thanks >>> More
Modifying AD Schema permissions from the command line

as seen on Geeks with Blogs - Search for 'Geeks with Blogs'
Recently while making some changes for a client, I accidently dug myself into a pretty deep hole. I was trying to explicitly deny a certain user from reading a few group policies including the Default Domain Policy. When I went in to make the change I accidently denied Authenticated Users rather… >>> More
Ask the Readers: Do You Use the Command Line?

as seen on How to geek - Search for 'How to geek'
Most people have heard of it but not everyone is familiar or comfortable with how to use this bastion of geekdom. This week we would like to know if you use the command line or not. The command line…the bastion of ultimate geekery in many peoples’ eyes. You often hear people referring to doing… >>> More