Global variables in hadoop.

Posted by Deepak Konidena on Stack Overflow See other posts from Stack Overflow or by Deepak Konidena
Published on 2010-05-22T16:06:40Z Indexed on 2010/05/22 16:10 UTC
Read the original article Hit count: 196

Filed under:

Hi,

My program follows a iterative map/reduce approach. And it needs to stop if certain conditions are met. Is there anyway i can set a global variable that can be distributed across all map/reduce tasks and check if the global variable reaches the condition for completion.

Something like this.

While(Condition != true){

            Configuration conf = getConf();
            Job job = new Job(conf, "Dijkstra Graph Search");

            job.setJarByClass(GraphSearch.class);
            job.setMapperClass(DijkstraMap.class);
            job.setReducerClass(DijkstraReduce.class);

            job.setOutputKeyClass(IntWritable.class);
            job.setOutputValueClass(Text.class);

}

Where condition is a global variable that is modified during/after each map/reduce execution.

© Stack Overflow or respective owner

Related posts about hadoop