Running JUnit in Parallel

Back in 2008 I had to speed up our daily build. (I should have posted about it since long, but I just didn't make it. Recently when I saw a related post on a similar topic my bad conscience overwhelmed me.) The first thing was to get a faster machine, something with four 3 GHz cores. It worked excellent! All file based operations like compile performed 3 times faster just out of the box, thanks to the included RAID 0+1 disk array. As our automated tests took half of the total build time, I dealt with them first: I applied the usual optimisations as told in my talk about practical JUnit testing, tuning section. So I managed to halve JUnit execution time.

Good, but still not fast enough. The problem was how to utilise all the shiny new cores during one build to speed it up as much as possible. So test execution needed to run in parallel. Some commercial build servers promised to be able to spread build targets over several agents. Unfortunately I had no opportunity to check them out, they cost quite beyond my budget. The only free distributed JUnit runner I found was using ComputeFarm JINI in a research project which did not look mature enough for production usage. Worth mentioning is GridGain’s JunitSuiteAdapter. It's able to distribute JUnit tests across a cluster of nodes. GridGain is a free cloud implementation, it's really hot stuff. But it's not a build solution, so integrating it into the existing build would have been difficult.

As I did not find anything useful had to come up with a minimalist home grown solution. I started with a plain JUnit target junitSequential which ran all tests in sequence:

<target name="junitSequential">

  <junit fork="yes" failureproperty="failed"
         haltonfailure="false" forkmode="perBatch">

    <classpath>
      <fileset dir="${lib.dir}" includes="*.jar" />
      <pathelement location="${classes.dir}" />
    </classpath>

    <batchtest>
      <fileset dir="${classes.dir}"
               includes="**/*Test.class" />
    </batchtest>
  </junit>

  <fail message="JUnit test FAILED" if="failed" />

</target>

I used haltonfailure="false" to execute all tests regardless if some failded or not. Otherwise <batchtest> would stop after the first broken test. With failureproperty="failed" and <fail if="failed" /> the build still failed if necessary. There is nothing special here.

Ant is able to run tasks in parallel using the <parallel> tag. (See my related post about forking several Ant calls in parallel.) A parallel running target would look like

<target name="junitParallelIdea">
  <parallel>
    <antcall target="testSomeJUnit" />
    <antcall target="testOtherJUnit" />
  </parallel>
</target>

Good, but how to split the set of tests into Some and Other? My first idea was to separate them by their names, i.e. by the first letter of the test's class name, using the inclusion pattern **/${junit.letter}*Test.class in the <batchtest>'s fileset. So I got 26 groups of tests running in parallel.

<target name="junitParallelNamedGroups">
  <parallel>

    <antcall target="-junitForLetter">
      <param name="junit.letter" value="A" />
    </antcall>
    <antcall target="-junitForLetter">
      <param name="junit.letter" value="B" />
    </antcall>
    <antcall target="-junitForLetter">
      <param name="junit.letter" value="C" />
    </antcall>
    <!-- continue with D until Z -->

  </parallel>
</target>

<target name="-junitForLetter">

  <junit fork="yes" forkmode="perBatch">

    <!-- classpath as above -->

    <batchtest>
      <fileset dir="${classes.dir}"
               includes="**/${junit.letter}*Test.class" />
    </batchtest>
  </junit>

</target>

forkmode="perBatch" created a new JVM for each group. Without forking each test class would get it's own class loader, filling up the perm space. Setting reloading="false" made things even worse. All those singletons started clashing even without considering race conditions. So I took the overhead of creating additional Java processes.

Unfortunately the grouping by letter approach had some problems. First the number of threads needed to be specified with <parallel>'s threadsperprocessor or threadcount attribute, else there would be 26 parallel processes competing for four cores. My experiments showed that two threads per processor performed best for the given set of JUnit tests. (Those JUnit tests were not "strictly unit", some tests called the database or web services, freeing the CPU during blocking. For tests with very little IO it might have looked different.)

Also my haltonfailure approach did not work because <antcall> does not return any properties set inside the called -junitForLetter target. There was no Ant command that supported that. But AntCallBack of the Antelope Ant extensions was able to do the trick: After registering the custom task with name="antcallback" I replaced the plain <antcall>s with <antcallback target="..." return="failed">.

Separating JUnit test cases by their names produced unbalanced and therefore unpredictable results regarding overall execution time. Depending on naming conventions some groups would run much longer than others. Ant's Custom Selectors are a much better way to split a fileset into a given number of parts producing a few balanced filesets with roughly the same number of test classes.

import java.io.File;

import org.apache.tools.ant.BuildException;
import org.apache.tools.ant.types.Parameter;
import org.apache.tools.ant.types.selectors.BaseExtendSelector;

public class DividingSelector extends BaseExtendSelector {

  private int counter;
  /** Number of total parts to split. */
  private int divisor;
  /** Current part to accept. */
  private int part;

  public void setParameters(Parameter[] pParameters) {
    super.setParameters(pParameters);
    for (int j = 0; j < pParameters.length; j++) {
      Parameter p = pParameters[j];
      if ("divisor".equalsIgnoreCase(p.getName())) {
        divisor = Integer.parseInt(p.getValue());
      }
      else if ("part".equalsIgnoreCase(p.getName())) {
        part = Integer.parseInt(p.getValue());
      }
      else {
        throw new BuildException("unknown " + p.getName());
      }
    }
  }

  public void verifySettings() {
    super.verifySettings();
    if (divisor <= 0 || part <= 0) {
      throw new BuildException("part or divisor not set");
    }
    if (part > divisor) {
      throw new BuildException("part must be <= divisor");
    }
  }

  public boolean isSelected(File dir, String name, File path) {
    counter = counter % divisor + 1;
    return counter == part;
  }
}

One of the four available cores was used for static code analysis, which was very CPU intensive and one was used for integration testing. The remaining two cores were dedicated to unit tests. Using 4 balanced groups of tests executing in parallel, the time spent for JUnit tests was halved again: Yippee

<target name="junitParallel4Groups">
  <parallel threadcount="4">
    <antcallback target="-junitForDivided" return="failed">
      <param name="junit.division.total" value="4" />
      <param name="junit.division.num" value="1" />
    </antcallback>
    <antcallback target="-junitForDivided" return="failed">
      <param name="junit.division.total" value="4" />
      <param name="junit.division.num" value="2" />
    </antcallback>
    <antcallback target="-junitForDivided" return="failed">
      <param name="junit.division.total" value="4" />
      <param name="junit.division.num" value="3" />
    </antcallback>
    <antcallback target="-junitForDivided" return="failed">
      <param name="junit.division.total" value="4" />
      <param name="junit.division.num" value="4" />
    </antcallback>
  </parallel>
  <fail message="JUnit test FAILED" if="failed" />
</target>

<target name="-junitForDivided">

  <junit fork="true" failureproperty="failed"
         haltonfailure="false" forkmode="perBatch">

    <!-- classpath as above -->

    <batchtest>
      <fileset dir="${classes.dir}">
        <include name="**/*Test.class" />
        <custom classname="DividingSelector" classpath="classes">
          <param name="divisor" value="${junit.division.total}" />
          <param name="part" value="${junit.division.num}" />
        </custom>
      </fileset>
    </batchtest>

  </junit>

</target>

(Download source code of DividingSelector.)

Epiloge
Using this approach I kept the option to execute the tests one after another with num=1 of total=1 providing an easy way to switch between normal and parallel execution. This was useful when debugging the build script...

<target name="junitSequential">
  <antcallback target="-junitForDivided" return="failed">
    <param name="junit.division.total" value="1" />
    <param name="junit.division.num" value="1" />
  </antcallback>
  <fail message="JUnit test FAILED" if="failed" />
</target>

Type Parameters and Reflection

Last week I had an opportunity to participate in the development of XMAdsl, an Xtext based model driven development extension of openXMA. In the new release we want to use Value Objects for all relevant values. E.g. the birth date of a person should not be an arbitrary java.util.Date, but a specific BirthDateValue instance. The Value Object hierarchy looks like that:

abstract class ValueObject<T> {
   private final T value;
   public ValueObject(T pValue) {
      value = pValue;
   }
   // shortened for brevity...
}

class ValueObjectDate extends ValueObject<Date> {
   public ValueObjectDate(Date pDate) {
      super((Date) pDate.clone()); // Date is mutable
   }
   // shortened for brevity...
}

class BirthDateValue extends ValueObjectDate {
   public BirthDateValue(Date pValue) {
      super(pValue);
   }
}

ValueObject, ValueObjectDate and all other Value Object base classes are packaged in the platform used by the generator. BirthDateValue and all other concrete subclasses are generated depending on the model. Such method signatures are more readable and passing a date of payment as a birth date by accident gets impossible. (Value Objects have other advantages, e.g. being read only, but that's not the point here.)

All Value Objects use the ValueObjectType Hibernate User Type for persistence. Obviously the User Type has to know the type of the Value Object to create new instances and the type of the value inside it to map it onto the database column. But how to get the inner type? Our first approach was to configure it. As the Hibernate configuration is generated as well, that's no big deal. Nevertheless it's a bit lame and definitely not DRY.

So the question is how to find the type parameter of a generic super class. In the given case the inner value is passed as constructor argument, so we can easily do better using reflection on the constructor parameter:

for (Constructor<?> c : pValue.getConstructors()) {
  if (c.getParameterTypes().length == 1) {
    return c.getParameterTypes()[0];
  }
}
// throw an exception

Nice. That fixed our problem but made me think further. What if there would would be no method or field with the concrete type of the type parameter, e.g. when using collections as super classes and not defining any new methods? It's also possible to find the type using reflection:

public Class<?> findType(Class<?> pValue) {
  if (pValue == Object.class) {
    // throw an exception
  }

  // is it generic itself?
  if (pValue.getTypeParameters().length > 0) {
    return (Class<?>) pValue.getTypeParameters()[0].getBounds()[0];
  }

  // is the super class typed?
  if (pValue.getSuperclass().getTypeParameters().length > 0) {
    Type superClass = pValue.getGenericSuperclass();
    ParameterizedType type = (ParameterizedType) superClass;
    return (Class<?>) type.getActualTypeArguments()[0];
  }

  // go up to super class
  return findType(pValue.getSuperclass());
}

A class knows if it's super class had type parameters using getGenericSuperclass(). If the class has a type parameter itself we don't know it due to erasure, but at least we get its lower bound. In all other cases we go up the class hierarchy. Finally a little test if it works as expected...

@Test
public void testFindType() {
  assertSame(Date.class,
             userType.findType(ValueObjectDate.class));
  assertSame(Date.class,
             userType.findType(BirthDateValue.class));
}

Holiday Blues

Holidays in Greece: Chilling out in the sun, spending time with the family, having some rest and much free time. Time to think about my job, my life and the whole universe ;-) However too much thinking makes me sad...

It has been more than half a year now that I have left Herold for good. In the end work there got quite boring and some things really sucked. But after staying there for five years the development team became my family. My bond with my colleagues kept me from leaving earlier and made it a really hard decision when the time for something new had finally come. I still remember them and from time to time I suffer moments of painful memories. I do not know how it came to be like that. There were no activities together, no common hobbies to talk about, even no geek interests to compete in. We did not meet after work or go out for a beer together, at least not more than once or twice a year.

Nevertheless I am thinking of them: Alex making his acrid remarks; Andreas refreshing me with new ideas, that always seemed to be much too revolutionary; Anton, although we hardly talked to each other; Ben making us play network games after work; Claudia my "soul-sister"; Claudio, exchanging stories about our misbehaving sons during lunch break; Dominik, always calm and relaxed; Martina having much too much energy; Petra; Richard testing a new database tool each week; Ronnie; Sylvi; Tim making me laugh about all his carpenter jokes; and Vero, the caring soul of the team, although she could really be mean sometimes.

My dear colleagues I miss you.