Processing XML file with Huge data

Posted by Manish Dhanotiya on Stack Overflow See other posts from Stack Overflow or by Manish Dhanotiya
Published on 2010-03-19T21:08:03Z Indexed on 2010/03/19 21:11 UTC
Read the original article Hit count: 213

Filed under:

Xml

|

processing

|

huge

Hi,be m I am working on an application which has below requiements - 1. Download a ZIP file from a server. 2. Uncompress the ZIP file, get the content (which is in XML format) from this file into a String. 3. Pass this content into another method for parsing and further processing.

Now, my concerns here is the XML file may be of Huge size say like '100MB', and my JVM has memory of only 512 MB, so how can I get this content into Chunks and pass for Parsing and then insert the data into PL/SQL tables.

Since there can be multiple requests running at the same time and considering 512MB of memory what will be the best possible to process this. How I can get the data into Chunks and pass it as Stream for XML parsing.

I googled on this, but didnt find any implementation. :(

Thanks,

© Stack Overflow or respective owner

Related posts about Xml

Store XML,update record in XML,retrive a specific record in XML stored on BB device

as seen on Stack Overflow - Search for 'Stack Overflow'
I am writing a blackberry application where i want to store the data returned by a web service in my BB device.Earlier i was going to use SQLite for storing the data in mobile but as i googled and also did programming using SQLite and found that some BB devices dont support SQLite library and fail… >>> More
gwt+xml- can i read through incomplete XML using the GWT XML Parser

as seen on Stack Overflow - Search for 'Stack Overflow'
I have a requirement where a user is typing in XML in a text area, and I want to show the various nodes in a tree...But as the user is typing in the xml, it wont be a complete xml (since he is still typing in the XML)... How do I read an incomplete XML and correctly generate the tree? I understand… >>> More
perl xml parser get xml content within xml

as seen on Stack Overflow - Search for 'Stack Overflow'
How can I use XMLParser to get the item-@url, item-@replace and item-"value inside" for the content as a string of the node where item-@cone="one"? <cstep> <item cone="one" url="http://google.com/{ccc}/cthree" replace="{ccc}"> <itemsub conesub="conesub"> … >>> More
Reading php generated XML in flash?

as seen on Stack Overflow - Search for 'Stack Overflow'
Here is part 1 of our problem (Loading a dynamically generated XML file as PHP in Flash). Now we were able to get Flash to read the XML file, but we can only see the Flash render correctly when tested(test movie) from the actual Flash program. However, when we upload our files online to preview the… >>> More
Announcing RSS feeds of Microsoft All-In-One Code Framework code samples

as seen on Geeks with Blogs - Search for 'Geeks with Blogs'
Today, we are not only announcing Sample Browser v2 CTP, but we are also excited to announce the availability of RSS feeds of All-In-One Code Framework code samples. By using these feeds, you can easily track and download the new code samples. English RSS feeds All code samples: http://support… >>> More

Related posts about processing

configure Squid3 proxy server on Ubuntu with caching and logging

as seen on Server Fault - Search for 'Server Fault'
I have a ubuntu 11.10 machine. Installed Squid3. When i configure the squid as http_access allow all, everything works fine. my current configuration mostly default is as follows: 2012/09/10 13:19:57| Processing Configuration File: /etc/squid3/squid.conf (depth 0) 2012/09/10 13:19:57| Processing:… >>> More
apt-get fails to upgrade, install, remove etc

as seen on Ask Ubuntu - Search for 'Ask Ubuntu'
I upgraded from 11.10 to 12.04, had no issues that I noticed. Recently tried to install something via software center, but it was throwing errors. Changed to trying to sudo apt-get install instead but again no luck. I've genuinely tried as much as I know to fix this, but I can't so I figured I'd ask… >>> More
Processing a tab delimited file with shell script processing

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello, normally I would use Python/Perl for this procedure but I find myself (for political reasons) having to pull this off using a bash shell. I have a large tab delimited file that contains six columns and the second column is integers. I need to shell script a solution that would verify that… >>> More
Processing xml file VS. processing excel file from .Net

as seen on Stack Overflow - Search for 'Stack Overflow'
Hello All, I would like to ask what should be faster: reading excel file from .Net or reading xml file which contains the same data. The same is for writing. Thank you very much in advance. mayap. >>> More
configure squid3 to set up a web proxy in ubuntu12.04

as seen on Super User - Search for 'Super User'
I am in a LAN and have to use a proxy given to access the web in a very limited way. I can't even use google, github.com or SE sites. However I can use ssh to log into a server, which I have root access so basically I can do anything I want with it. So I was thinking that maybe I could use that server… >>> More