Understanding JUnit's Runner architecture

Update (2020): This post describes how JUnit 4 runners work and how you can create you own JUnit 4 runner. Please note that JUnit 5 is available for few years now (it has been released 2017). Maybe you should consider updating your project to JUnit 5 if you are still using JUnit 4. If you are interested in JUnit 5 you might be interested in my article about creating custom extensions for JUnit 5.

Some weeks ago I started creating a small JUnit Runner. I learned that creating custom JUnit Runners is actually quite simple. In this post I want to show you how JUnit Runners work internally and how you can use custom Runners to modify the test execution process of JUnit.

So what is a JUnit Runner?

A JUnit Runner is class that extends JUnit's abstract Runner class. Runners are used for running test classes. The Runner that should be used to run a test can be set using the @RunWith annotation.

@RunWith(MyTestRunner.class)
public class MyTestClass {

  @Test
  public void myTest() {
    ..
  }
}

JUnit tests are started using the JUnitCore class. This can either be done by running it from command line or using one of its various run() methods (this is what your IDE does for you if you press the run test button).

JUnitCore.runClasses(MyTestClass.class);

JUnitCore then uses reflection to find an appropriate Runner for the passed test classes. One step here is to look for a @RunWith annotation on the test class. If no other Runner is found the default runner (BlockJUnit4ClassRunner) will be used. The Runner will be instantiated and the test class will be passed to the Runner. Now it is Job of the Runner to instantiate and run the passed test class.

How do JUnit Runners work?

Lets look at the class hierarchy of standard JUnit Runners:

Runner is a very simple class that implements the Describable interface and has two abstract methods:

public abstract class Runner implements Describable {

  public abstract Description getDescription();

  public abstract void run(RunNotifier notifier);
}

The method getDescription() is inherited from Describable and has to return a Description. Descriptions contain the information that is later being exported and used by various tools. For example, your IDE might use this information to display the test results.
run() is a very generic method that runs something (e.g. a test class or a test suite). I think usually Runner is not the class you want to extend (it is just too generous).

In ParentRunner things get a bit more specific. ParentRunner is an abstract base class for Runners that have multiple children. It is important to understand here, that tests are structured and executed in a hierarchical order (think of a tree).
For example: You might run a test suite which contains other test suites. These test suites then might contain multiple test classes. And finally each test class can contain multiple test methods.

ParentRunner has the following three abstract methods:

public abstract class ParentRunner<T> extends Runner implements Filterable, Sortable {    

  protected abstract List<T> getChildren();

  protected abstract Description describeChild(T child);

  protected abstract void runChild(T child, RunNotifier notifier);
}

Subclasses need to return a list of the generic type T in getChildren(). ParentRunner then asks the subclass to create a Description for each child (describeChild()) and finally to run each child (runChild()).

Now let's look at two standard ParentRunners: BlockJUnit4ClassRunner and Suite.

BlockJUnit4ClassRunner is the default Runner that is used if no other Runner is provided. So this is the Runner that is typically used if you run a single test class. If you look at the source of BlockJUnit4ClassRunner you will see something like this:

public class BlockJUnit4ClassRunner extends ParentRunner<FrameworkMethod> {

  @Override
  protected List<FrameworkMethod> getChildren() {
    // scan test class for methonds annotated with @Test
  }

  @Override
  protected Description describeChild(FrameworkMethod method) {
    // create Description based on method name
  }

  @Override
  protected void runChild(final FrameworkMethod method, RunNotifier notifier) {
    if (/* method not annotated with @Ignore */) {
      // run methods annotated with @Before
      // run test method
      // run methods annotated with @After
    }
  }
}

Of course this is overly simplified, but it shows what is essentially done in BlockJUnit4ClassRunner.

The generic type parameter FrameworkMethod is basically a wrapper around java.lang.reflect.Method providing some convenience methods. In getChildren() the test class is scanned for methods annotated with @Test using reflection. The found methods are wrapped in FrameworkMethod objects and returned. describeChildren() creates a Description from the method name and runChild() finally runs the test method. BlockJUnit4ClassRunner uses a lot of protected methods internally. Depending on what you want to do exactly, it can be a good idea to check BlockJUnit4ClassRunner for methods you can override. You can have a look at the source of BlockJUnit4ClassRunner on GitHub.

The Suite Runner is used to create test suites. Suites are collections of tests (or other suites). A simple suite definition looks like this:

@RunWith(Suite.class)
@Suite.SuiteClasses({
  MyJUnitTestClass1.class,
  MyJUnitTestClass2.class,
  MyOtherTestSuite.class
})
public class MyTestSuite {}

A test suite is created by selecting the Suite Runner with the @RunWith annotation. If you look at the implementation of Suite you will see that it is actually very simple. The only thing Suite does, is to create Runner instances from the classes defined using the @SuiteClasses annotation. So getChildren() returns a list of Runners and runChild() delegates the execution to the corresponding runner.

Examples for Custom JUnit runners

With the provided information it should not be that hard to create your own JUnit Runner (at least I hope so). If you are looking for some example custom Runner implementations you can have a look at the following list:

• Fabio Strozzi created a very simple and straightforward GuiceJUnitRunner project. It gives you the option to inject Guice components in JUnit tests. Source on GitHub
• Spring's SpringJUnit4ClassRunner helps you test Spring framework applications. It allows you to use dependency injection in test classes or to create transactional test methods. Source on GitHub
• Mockito provides MockitoJUnitRunner for automatic mock initialization. Source on GitHub
• Oleaster's Java 8 Jasmine runner. Source on GitHub (shameless self promotion)

Conclusion

JUnit Runners are highly customizable and give you the option to change to complete test execution process. The cool thing is that can change the whole test process and still use all the JUnit integration points of your IDE, build server, etc.

If you only want to make minor changes it is a good idea to have a look at the protected methods of BlockJUnit4Class runner. Chances are high you find an overridable method at the right location.

In case you are interested in Olaester, you should have a look at my blog post: An alternative approach of writing JUnit tests.