MR Unit

You can test Mapper and reducer using MRUnit. Its the framework build on Junit so its straight forward if you have worked with Junit.

MRUnit helps bridge the gap between MapReduce programs and JUnit by providing a set of interfaces and test harnesses, which allow MapReduce programs to be more easily tested using standard tools and practices.

To write your test you would:

1. Instantiate an instance of the MapDriver class parameterized exactly as the mapper under test.
2. Add an instance of the Mapper you are testing in the withMapper call.
3. In the withInput call pass in your key and input value, in this case a LongWritable with an arbitrary value and a Text object that contains a line from from the NCDC weather dataset contained in a String array called ‘temps’ that was set up earlier in the test (not displayed here as it would take away from the presentation).
4. Specify the expected output in the withOutput call, here we are expecting a Text object with the value of “190101” and a TemperatureAveragingPair object containing the values -61 (temperature) and a 1 (count).
5. The last call runTest feeds the specified input values into the mapper and compares the actual output against the expected output set in the ‘withOutput’ method.
   

A map/reduce pair can be tested using MRUnit’s MapReduceDriver.

 
Code Example:

package com.valassis.mrunit;


import java.io.IOException;
import java.util.ArrayList;
import java.util.List;

import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mrunit.mapreduce.MapDriver;
import org.apache.hadoop.mrunit.mapreduce.MapReduceDriver;
import org.apache.hadoop.mrunit.mapreduce.ReduceDriver;
import org.junit.Before;
import org.junit.Test;

public class MRTester {

//Specification of Mapper
MapDriver<LongWritable, Text, Text, IntWritable> mapDriver;
//Specification of Reduce
ReduceDriver<Text, IntWritable, Text, IntWritable> reduceDriver;
//Specification of MapReduce program
MapReduceDriver<LongWritable, Text, Text, IntWritable, Text, IntWritable> mapReduceDriver;

@Before
public void setUp() {
MaxTemperatureMapper mapper = new MaxTemperatureMapper();
MaxTemperatureReducer reducer = new MaxTemperatureReducer();
//Setup Mapper
mapDriver = MapDriver.newMapDriver(mapper);
//Setup Reduce
reduceDriver = ReduceDriver.newReduceDriver(reducer);
//Setup MapReduce job
mapReduceDriver = MapReduceDriver.newMapReduceDriver(mapper, reducer);
}

@Test
public void testMapper() {
//Test Mapper with this input
mapDriver.withInput(new LongWritable(), new Text("a a a a a a a"));
//Expect this output
mapDriver.withOutput(new Text("a"), new IntWritable(1));
 try {
//Run Map test with above input and ouput
mapDriver.runTest();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}

@Test
public void testReducer() {
List<IntWritable> values = new ArrayList<IntWritable>();
values.add(new IntWritable(1));
/*
values.add(new IntWritable(1));
values.add(new IntWritable(1));
values.add(new IntWritable(1));
values.add(new IntWritable(1));
values.add(new IntWritable(1));
values.add(new IntWritable(1));
*/
//Run Reduce with this input
reduceDriver.withInput(new Text("a"), values);
//Expect this output
reduceDriver.withOutput(new Text("a"), new IntWritable(1));
try {
//Run Reduce test with above input and ouput
reduceDriver.runTest();
} catch (Exception e) {
// TODO Auto-generated catch block
e.printStackTrace();
}
}
}