Hi guys.

We are trying to figure out the best practice for working with very large DBs in Java. What we do is a kind of BI, i.e analyzing very large DBs, and using them to create intermediate DBs that represent intelligent knowledge of the DBs.

We are currently using JDBC, and just preforming queries using a ResultSet. As more and more data is being created, we are wondering whether more appropriate ways exist for parsing and manipulating these large DBs:

1. We need to support 'chunk' manipulation and not an entire DB at once(e.g. limit in JDBC, very poor performance)
2. We do not need to be constantly connected since we are just pulling results and creating new tables of our own.
3. We want to understand JDBC alternatives, with respect to advantages and disadvantages.
4. Whether you think JDBC is the way to go or not, what are the best practices to go by depending on context (e.g. for large DBs queried in chunks) ?

If my question is not clear, I will gladly elaborate!

THANK YOU SO MUCH!