split large text file without missing data record

Posted by Santosh on Super User See other posts from Super User or by Santosh
Published on 2011-12-23T10:13:07Z Indexed on 2012/10/18 11:04 UTC
Read the original article Hit count: 266

Filed under:

file

|

split

I have a 140 MB text file, which contains detail information of books in library. For each book details there is a standard format data details in text file. I need to parse it and insert the data in Database. Here, parsing text file is not an issue. I am facing problem in parsing this large file. So i decided to split the file in small file around 2 MB each file. But i can't manually split this large file in so many pieces. I got HJsplit tool, which split the file but this also doesn't helped as this split the file but 1 book details half part is in one file and rest part is in second file. so if i split this way then information will be missed.

How to split the large so that i cant miss the information ? Is there any tool which help me in this condition.

© Super User or respective owner

Related posts about file

NTFS Corruption: Files created in Linux corrupted when Windows Boots

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I'm getting some file loss and corruption on my Win7/Ubuntu 12.04 dual boot setup. I have a large shared NTFS partition. I have my Windows Docs/Music/etc. directories on that file and have the comparable directors in Linux setup as a sym. link. I'm using ntfs-3g on the linux side of things to… >>> More
Prevent malicious vulnerability scan increasing load on a server

as seen on Server Fault - Search for 'Server Fault'
Hi all, this week we have been suffering some malicious vulnerability scans to our servers, increasing the load on them, making them nearly unusable. The attack is easy to defend, just blocking the offending ip, but only after discovering it. Is there any form of prevent it? Is it normal that… >>> More
yum update failed

as seen on Server Fault - Search for 'Server Fault'
I have problem doint yum update on my OpenVZ VPS i get this error message : (56/69): glibc-devel-2.5-81.el5_8.7.x86_64.rpm | 2.4 MB 00:00 (57/69): libstdc++-devel-4.1.2-52.el5_8.1.x86_64.rpm | 2.8 MB 00:00 (58/69): binutils-2.17.50.0.6-20.el5_8.3.x86_64.rpm | 2.9 MB … >>> More
Microsoft Management Console stops working when I add snap-in to it

as seen on Super User - Search for 'Super User'
I have Windows 7 Ultimate OS. I'm opening mmc.exe as administrator and trying add Certificates or any other snap-in, then while loading that snap-in MMC breaks and displays following message and after that it closes automatically once I click on close button on that message. What could be the problem… >>> More
Opening a file opens the folder the file is in, not the file itself

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
Whenever I try to open a file (such as an .odt, or .doc) from say, the Dash or the Firefox Downloads, Ubuntu 11.10 opens Nautilus to the the folder where the file is, rather than just going to the application and loading the file straight away. In previous releases, when I clicked on a downloaded… >>> More

Related posts about split

Java .split() Method To Split XML Parameters

as seen on Stack Overflow - Search for 'Stack Overflow'
I have this line from an XML document: <?xml version="1.0" encoding="UTF-8"?> <svg xmlns:xlink="http://www.w3.org/1999/xlink" xmlns="http://www.w3.org/2000/svg" contentScriptType="text/ecmascript" width="1024" zoomAndPan="magnify" contentStyleType="text/css" viewBox="0 0 1024 768" height="768"… >>> More
Python: split files using mutliple split delimiters

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I have multiple CSV files which I need to parse in a loop to gather information. The problem is that while they are the same format, some are delimited by '\t' and others by ','. After this, I want to remove the double-quote from around the string. Can python split via multiple possible delimiters… >>> More
how to club the data split by split function

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi my data is dat1; the split statement is var splitstr = dat1.split("-"); I have splited the data from this format 2010 -02-02 to Element 0 = 2010 Element 1 = 05 Element 2 = 22 this format… >>> More
GWT: How to access/style horizontal split view's split

as seen on Stack Overflow - Search for 'Stack Overflow'
I'm trying to set the style name of the div of the split between the two widgets of a HorizontalSplitPanel in order to CSS-style it. Is there any way to do this? >>> More
OBJ model loaded in LWJGL has a black area with no texture

as seen on Game Development - Search for 'Game Development'
I have a problem with loading an .obj file in LWJGL and its textures. The object is a tree(it's a paid model from TurboSquid, so I can't post it here,but here's the link if you want to see how it should look like): http://www.turbosquid.com/FullPreview/Index.cfm/ID/701294 I wrote a custom OBJ loader… >>> More