Oddjob in the Cloud with Amazon Web Services

It’s quite straight forward to deploy a java process to an AWS EC2 instance. I followed the EC2 Getting Started Guide to create a Linux instance, installed java by following this blog article, uploaded the Oddjob tarball and started it from Putty and voila – Oddjob running on EC2.

Unfortunately it’s all a bit manual. Could automating the deployment of Oddjob be a Job for Oddjob? I think so. Time to spend two weeks automating a ten minute manual process. The result is 239 lines of XML configuration that can be seen here.

Here’s what that XML looks like in Oddjob Explorer, once it’s run:

Oddjob Deploying Oddjob to EC2

Let’s go through the main steps.

Continue reading

Oddjob 1.5 Released

Finally after 3 years I’ve cut version 1.5 of Oddjob. Despite it being 3 years this release contains only two major new features, and both are only half implemented. Where did those years go?

I had another child. I’ve had a few already, so I know it won’t be long before I can’t complete with a Play Station. Thus I’m limiting Oddjob time to my train ride into the city. There has also been three new contracts. Each one required mastering some new technologies, and Oddjob time has been sacrificed. And I’ve been learning Scala but that’s another story…

Sorry, enough about my 3 years, what about Oddjob’s?

Far too long was spent migrating to Ivy. I didn’t want to move to Maven because Oddjob has some strange module dependecies. For instance the main project produces oddjob.jar and oj-tools.jar, and oddjob tests depend on oj-tools and oj-tools depend on oddjob, and you just can’t do that in Maven. Once I’d ‘got’ Ivy, it coped with this really well. It just took so many months to really ‘get’ Ivy…

Re-writing the Web UI was by contrast, very pleasant, despite taking over a year of commutes. First there was the ‘what to use?’ question. My previous JSF experience was painful and has put me off silver bullet server side frameworks – Wicket, Tapestry, etc. Furthermore, the client server pendulum has swung back to thick clients which IMHO is where it should be. The paradigm of server serves machine readable protocol and clients do with it what they will seems much more sensible than serving HTML. Decision made – A RESTful server. Jersey or Resteasy? Resteasy, though I can’t remember why. What to use on the client? Well Javascript obviously! But I needed a little help. JQuery seemed well thought out and easy to be up and running with compared to the more complex frameworks like Angular and React. And indeed JQuery worked out really well.

I was pretty much done with only a few bugs that were proving difficult to fix. They always turned out to be property that didn’t exist or something else that the compiler would have caught in Java. It was very frustrating so I Googled something like ‘Type Safe JavaScript’ and of course found TypeScript. So I re-wrote the entire thing in TypeScript, and puff, those pesky bugs just disappeared.

The new UI doesn’t actually provide any new functionality over the last JSF front end and it still needs lots of work but it’s at a good place for a first release. given that UI’s are not my strong point, I’m quite proud of it. Here’s a picture:

Oddjob Browser View

The other big new feature is the introduction of Tasks which are parameterised Job executions. This is well documented here in the User Guide  so I won’t say more, as I’m keen to start on Oddjob 1.6. What will be in it? Well definately some more web stuff. Probably a bit of functional funkyness, and who knows – maybe even some Scala.

A Cautionary Tale of Ant JUnit Fork Mode

My Oddjob build was taking over half an hour on my aging laptop. This is too much I thought…

Either I fix the build or buy a new laptop.

I started to look at the build and noticed most of the time was running the tests. A close look at the junit task configuration revealed I had forkyes‘ but no forkMode which was defaulting to ‘perTest‘. What if I changed this to ‘once‘?

		<junit printsummary="on" haltonfailure="no" 
				fork="yes" forkmode="once" 
				<pathelement location="${build.test.classes}" />
				<pathelement location="${java.home}/../lib/tools.jar"/>
			<classpath refid="test-classpath"/>
			<batchtest todir="${test.reports}">
				<fileset dir="${build.test.classes}">
					<include name="**/test/*TestCase.class"/>
					<include name="**/*Test.class" />
					<exclude name="**/AllTest.class" />
					<exclude name="**/*$$*Test.class" />
			<formatter type="xml" />
			<sysproperty key="basedir" value="${basedir}"/>
			<sysproperty key="oddjob.home" value="${oddjob.home}"/>
			<sysproperty key="ant.file" value="${ant.file}"/>

Everything broke.

OK, not quite everything but everything that was testing what Oddjob wrote to System.out. Oddjob changes System.out so that it can display the console of local and remote processes. This same mechanism is used by the tests to validate output. After getting no inspiration from Google, I thought…

Either I debug the Ant source code, or I buy a new laptop.

The Ant source code is quite easy to follow, and within a few hours I had picture of what was going on. The JUnit task itself replaces System.out so that it can pass the output to the formatters, and this happens on a per test case basis. So my own static System.out was being replaced by the second test that ran and not getting any further output. So…

Either I change the console capture system to initialise with each instance of Oddjob, or I buy a new laptop.

I introduced a new class to handle the initialisation of console capture with cleanup so that Oddjob and tests could be a good citizens and replace System.out when they were done. I needed to change a few dozen references from the old static field to the new class method but a few more hours later all the broken tests were now passing and the build was running in under fifteen minutes.

The build was however failing intermittently on one test that had nothing to do with System.out. It was failing cleaning up a file. I needed some logging to understand why. No Log4j output in my test? Am I going mad? I changed forkMode back to perTest and I got output. The same thing that was affecting Oddjob’s console capture was affecting Log4j. Google was some help here and I found a mailing list question and a bug report both relating to this.

The solution is to change log4j.properties to include:


I added this and once again I had Log4j output. Every test for console output now failed because it had Log4j output in it. I swore. The tests relied on Log4j being initialised before Oddjob so binding to System.out before Oddjob changed it. Oddjob would only capture output written to its System.out and missed all console logging. However with this configuration change, Log4j was following System.out and writing to Oddjob’s version once it was set.

Either I change my tests to change the console logging level before and after console capture or I buy a new laptop.

Changing log level wasn’t to hard with this helper class, but ensuring log level was restored correctly involved adding a try block to about 80 tests. Another few hours later I had Log4j logging, console capture tests passing and a build that was running in under fifteen minutes.

And my failing test? No idea – I added a try 3 times loop to the file delete. Some times you’ve just got to move on…

What a palaver just for one tiny attribute! After all that arduous work I felt I deserved a reward – so I bought a new laptop.

Java Parallel Runtime Execs Waiting for Each Other?!

Several Oddjob users have reported that, when running several execs in parallel on Windows, they all appeared to wait for each other to complete. The issue was easy to reproduce using this Oddjob configuration:

        <exec redirectStderr="true"><![CDATA[TestJob.cmd 2]]></exec>
        <exec redirectStderr="true"><![CDATA[TestJob.cmd 10]]></exec>

Where TestJob.cmd is:

ping -n %1
echo Finished pinging for %1.
exit 0

The problem can be seen here:

Parallel Exec Jobs

From the console of the first Exec Job it has clearly finished but its icon is still showing as Executing.

Java’s native process support is notoriously flaky, especially on Windows, and was the prime suspect. However, first I had to eliminate Oddjob from the enquiry. Here is some simple Java code that reproduces the problem:

public class ExecMain {

	static class Exec implements Runnable {
		private final String waitSeconds;

		Exec(String waitSeconds) {
			this.waitSeconds = waitSeconds;

		public void run() {
			long startTime = System.currentTimeMillis();

			final ByteArrayOutputStream captureOutput = new ByteArrayOutputStream();

			ProcessBuilder processBuilder = 
					new ProcessBuilder("TestJob.cmd", waitSeconds);

			try {
				final Process process = processBuilder.start();
				Thread t = new Thread(new Runnable() {
					public void run() {				
						copy(process.getInputStream(), captureOutput);
				System.out.println("Process for TestJob.cmd " + waitSeconds + 
						" finished in " + secondsFrom(startTime) + " seconds.");
				System.out.println("Output thread for TestJob.cmd " + waitSeconds + 
						" joined after " + secondsFrom(startTime) + " seconds.");
			catch (InterruptedException | IOException e) {
				throw new RuntimeException(e);

		void copy(InputStream from, OutputStream to) {
			byte[] buf = new byte[0x0400];
			try {
				while (true) {
					int r = from.read(buf);
					if (r == -1) {
					to.write(buf, 0, r);
			catch (IOException e) {
				throw new RuntimeException(e);

		int secondsFrom(long startMillis) {
			return Math.round((System.currentTimeMillis() - startMillis) / 1000);

	public static void main(String... args) {

		new Thread(new Exec("2")).start();
		new Thread(new Exec("10")).start();

And here’s the output:

Process for TestJob.cmd 2 finished in 1 seconds.
Output thread for TestJob.cmd 2 joined after 9 seconds.
Process for TestJob.cmd 10 finished in 9 seconds.
Output thread for TestJob.cmd 10 joined after 9 seconds.

We can see that the process ends as expected after a second, but joining on the stream copying thread doesn’t happen until the sibling process has finished. This can only be if the first processes output stream isn’t being closed. Is it waiting for its siblings process output stream to close too?

Hours of Googling prove fruitless. Then by happenstance, I run my sample against Java 8. It works as expected. Off to the Java bug database – nothing obvious. Oddjob is currently supported on Java 7 and above so I downloaded the latest Java 7u80 release just to see, and it works to. Here is the correct output:

Process for TestJob.cmd 2 finished in 1 seconds.
Output thread for TestJob.cmd 2 joined after 1 seconds.
Process for TestJob.cmd 10 finished in 9 seconds.
Output thread for TestJob.cmd 10 joined after 9 seconds

And now in Oddjob we can see the Exec Job completes when the process does:

Parallel Exec Fixed
So this is a story with a happy ending but a niggling loose end. What was the Java bug that caused this?

Oddjob 1.4 Released

Despite the eighteen month gap since Oddjob 1.3 there really isn’t a lot in this release. This is because I’ve been using Oddjob an awful lot, and using it mainly for automating testing. This has lead to some spin-off projects that are Oddjob plugins (Oddballs) and help Oddjob to test stuff. I won’t go into them here but I will be blogging more about them soon.

So what is in Oddjob 1.4?
Continue reading

Oddjob 1.3 Released

This release would best be described as a ‘Consolidation Release’. The biggest improvements are under the hood and result in an Oddjob that runs much better but in an almost indiscernible way. There are however a few new and noteworthy features. These include:
Continue reading

Feature Comparison of Java Job Schedulers – Plus One

Poor Oddjob, I thought as I read Craig Flichel’s Feature Comparison of Java Job Schedulers featuring Obsidian, Quartz, Cron4j and Spring. Yet again it hasn’t made the grade, it’s been passed over for the scheduling team.

Never mind I say, you’re just a little bit different and misunderstood. Let’s have a kick about in the back yard and see what you can do…
Continue reading

Jobs And Services

Life is, and always has been, about executing a sequence of jobs – hunt mammoth, skin mammoth, cook mammoth, make clothes from mammoth. In the beginning an individual would have done all these jobs, but it didn’t take long for ancient societies to realise that it was more efficient for people to specialise in one type of job and to offer the product of that job as a service. Two hundred thousand years later we have McDonald’s and Gap. Whether this is progress is for you to decide – but it is undeniable that this model has allowed vastly complex systems of millions of components (us) to function and thrive.
Continue reading

Oddjob With Spring

Spring is great for creating flexible applications by assembling loosely coupled components using XML. Here’s a simple Spring configuration file:

<beans xmlns="http://www.springframework.org/schema/beans"

  <bean id="hello" class="example.HelloWorldBean" />


Except that now you have to go back to your context assisted IDE and write some Java to launch it!
Continue reading