How can I store large amount of data from a database to XML (memory problem)?
- by Andrija
First, I had a problem with getting the data from the Database, it took too much memory and failed. I've set -Xmx1500M and I'm using scrolling ResultSet so that was taken care of. Now I need to make an XML from the data, but I can't put it in one file. At the moment, I'm doing it like this:
while(rs.next()){
i++;
xmlStringBuilder.append("\n\t<row>");
xmlStringBuilder.append("\n\t\t<ID>" + Util.transformToHTML(rs.getInt("id")) + "</ID>");
xmlStringBuilder.append("\n\t\t<JED_ID>" + Util.transformToHTML(rs.getInt("jed_id")) + "</JED_ID>");
xmlStringBuilder.append("\n\t\t<IME_PJ>" + Util.transformToHTML(rs.getString("ime_pj")) + "</IME_PJ>");
//etc.
xmlStringBuilder.append("\n\t</row>");
if (i%100000 == 0){
//stores the data to a file with the name i.xml
storeKBR(xmlStringBuilder.toString(),i);
xmlStringBuilder= null;
xmlStringBuilder= new StringBuilder();
}
and it works; I get 12 100 MB files. Now, what I'd like to do is to do is have all that data in one file (which I then compress) but if just remove the if part, I go out of memory. I thought about trying to write to a file, closing it, then opening, but that wouldn't get me much since I'd have to load the file to memory when I open it.
P.S. If there's a better way to release the Builder, do let me know :)
