Reading from a very large table using multiple threads (Java ) and writing them to a single file

Posted by user2534926 on Stack Overflow See other posts from Stack Overflow or by user2534926
Published on 2013-06-29T16:18:28Z Indexed on 2013/06/29 16:21 UTC
Read the original article Hit count: 124

Filed under:

java

|

multithreading

|

large-data

I am currently facing a situation where i have a table with almost 80 millions data and i have to take a dump of that table and store it into a csv file. Currently i am using a not so professional approach( with a perl script+DBI interface , printing the values to stdout and redirecting to a csv file). Now i am planning to use java threading approach. Can you suggest a way forward. Thanks in advance

© Stack Overflow or respective owner

Related posts about java

Tomcat 6: Access Control Exception?

as seen on Server Fault - Search for 'Server Fault'
I'm trying to setup a tomcat6 server, and I'm trying to match another setup someone else established. However, my deployment (default Ubuntu install) uses a policy.d/ directory structure, and the established server just uses a catalina.policy file. I've tried setting every entry in policy.d to match… >>> More
Problem in creation MDB Queue connection at Jboss StartUp

as seen on Stack Overflow - Search for 'Stack Overflow'
I am not able to create a Queue connection in JBOSS4.2.3GA Version & Java1.5, as I am using MDB as per the below details. I am putting this MDB in a jar file(named utsJar.jar) and copied it in deploy folder of JBOSS, In the test env. this MDB works well but in another env. [ env settings and… >>> More
failing to establish connection between Postgres db and gwt

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I am using Postgres and gwt 2.0 for one of my applications. I am facing problem connecting to the database. When I try to connect it gives "ClassNotFoundException". Here is what I get when I try to connect to database: java.lang.ClassNotFoundException: org.postgresql.Driver at java.net… >>> More
failing to establish connection between postgre db and gwt

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, For i am using postgre and gwt 2.0 for one of my applications. I am facing problem connecting to the database. When i try to connect it gives "ClassNotFoundException". Here is what i get when i try to connect to database: java.lang.ClassNotFoundException: org.postgresql.Driver at java.net… >>> More
Migration and deployement problems JBoss 4.2.2.GA to JBoss 6.0.0.M2

as seen on Stack Overflow - Search for 'Stack Overflow'
Hi, I'm trying to migrate an application running on JBoss 4.2.2.GA to JBoss 6.0.0.M2 I give you some log to explain my problem : boot.log : 2010-03-16 09:59:29,406 ERROR [org.jboss.system.server.profileservice.ProfileServiceBootstrap] (Thread-2) Failed to load profile: Summary of incomplete deployments… >>> More

Related posts about multithreading

C++ Multithreading on Unix

as seen on Programmers - Search for 'Programmers'
I have two related questions: 1) Are there any good books for multithreading in C++, especially now that C++11 contains multithreading in the standard library? 2) I have the Wrox Programming on Unix book (1000 pages fat red one) and within it, it uses the Unix Thread class. How does this code relate… >>> More
what are the difficulties of operating system multithreading?

as seen on Stack Overflow - Search for 'Stack Overflow'
I am reading a book that compares two ways of implementing threads, Middleware Threads and OS Threads. I have a question about these sentences: "A difficulty of operating system multithreading, however, is performance overhead. Since it is the operating system that is involved in switching threads… >>> More
TCP multicast and multithreading

as seen on Stack Overflow - Search for 'Stack Overflow'
I need to come up with clients that can multicast to other clients reliably. That implies I'll be using TCP to connect reliably between clients within a multicast group. Doesn't that come up to n^2 number of connections? That seems a little silly to me. Wouldn't/shouldn't there be a way to more easily… >>> More
Multithreading recommendation based on program description

as seen on Stack Overflow - Search for 'Stack Overflow'
I would like to describe some specifics of my program and get feedback on what the best multithreading model to use would be most applicable. I've spent a lot of time now reading on ThreadPool, Threads, Producer/Consumer, etc. and have yet to come to solid conclusions. I have a list of files (all… >>> More
How to achieve multithreading using JavaScript with IE6?

as seen on Stack Overflow - Search for 'Stack Overflow'
How to achieve multithreading using JavaScript with IE6? Is there a third party library to do this? >>> More