Error when running a basic Hadoop code
- by Abhishek Shivkumar
I am running a hadoop code that has a partitioner class inside the job. But, when I run the command
hadoop jar Sort.jar SecondarySort inputdir outputdir
I am getting a runtime error that says
class KeyPartitioner not org.apache.hadoop.mapred.Partitioner.
I have ensured that the KeyPartitioner class has extended the Partitioner class, but why am I getting this error?
Here is the driver code:
JobConf conf = new JobConf(getConf(), SecondarySort.class);
conf.setJobName(SecondarySort.class.getName());
conf.setJarByClass(SecondarySort.class);
conf.setInputFormat(TextInputFormat.class);
conf.setOutputFormat(TextOutputFormat.class);
conf.setMapOutputKeyClass(StockKey.class);
conf.setMapOutputValueClass(Text.class);
conf.setPartitionerClass((Class<? extends Partitioner<StockKey, DoubleWritable>>) KeyPartitioner.class);
conf.setMapperClass((Class<? extends Mapper<LongWritable, Text, StockKey, DoubleWritable>>) StockMapper.class);
conf.setReducerClass((Class<? extends Reducer<StockKey, DoubleWritable, Text, Text>>) StockReducer.class);
and here is the code of the partitioner class:
public class KeyPartitioner extends Partitioner<StockKey, Text> {
@Override
public int getPartition(StockKey arg0, Text arg1, int arg2) {
int partition = arg0.name.hashCode() % arg2;
return partition;
}
}
