How to search for duplicate values in a huge text file having around Half Million records
- by Shibu
I have an input txt file which has data in the form of records (each row is a record and represents more or less like a DB table) and I need to find for duplicate values. For example:
Rec1: ACCOUNT_NBR_1*NAME_1*VALUE_1
Rec2: ACCOUNT_NBR_2*NAME_2*VALUE_2
Rec3: ACCOUNT_NBR_1*NAME_3*VALUE_3
In the above set, the Rec1 and Rec2 are considered to be…