Tuesday 17 February 2009

Speed up Maven build

From a sprint retrospective we got a lot of 'minuses' related to the speed of Maven build executed by every developer localy on their computers many times per day. The developers were frustrated that complete build tooks from 10 to 15 minutes and estimated that they spend from 30 to 60 minutes every day just waiting for build. What in a team containing 16 developers (it was two scrum teams) gives you from 8 to 16 manhours per day. It is really a lot so we decided to try to do something about it.

At the begining I have to say that if there is such problem in the project, it usualy mean that something is wrong with modularity, or your IDE should be replaced or used in a better way to support faster code-see iterations. But it wasn't true in our case. Our problem was legacy web application famework without any support in existing IDEs.

Since I was responsible for build process I started to work on this issue. At the first I started to talk with developers about how do they build it. They simply used the most common way of application building:

mvn clean install

to rebuild this all, or

mvn install

to just repack updated sources and resources.

So I started to "profile" the build and early found surprising fact that approximately 50% of the time is taken by 'clean' goal - I found first bottleneck.

To remove it I used simple trick - move/rename operation on the folder is immediate on most of the systems in comparison with delete operation.

...
<profile>
<id>qclean</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<executions>
<execution>
<id>rename_target</id>
<phase>pre-clean</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<tasks>
<tstamp>
<format property="targetTstamp"
pattern="yyyyMMdd-HHmmss" locale="en,US" />
</tstamp>
<move todir="trash/target-${targetTstamp}"
failonerror="false">
<fileset dir="target/" />
</move>
</tasks>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
...

In the code snapshot you can see definition of new qclean (means Quick Clean) profile. The profile contains configuration of maven-antrun-plugin to be executed in a 'pre-clean' phase. The Ant script at the first generates a timestamp string into a property 'targetTimestamp' using 'tstamp' task. Then it moves and renames 'target' folder to 'trash/target-${targetTstamp}'.

Now when developers executed build using following command they simply spent just half of the time waiting for build.

mvn clean install -Pqclean

Because the 'target' folder is just moved and not deleted it is necessary to clean also 'trash' folder sometimes. But you can do it as a background operation or while you are not using the computer at all. For this purpose I created one more profile named clean.

...
<profile>
<id>clean</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<executions>
<execution>
<id>clean_trash</id>
<phase>clean</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<tasks>
<delete dir="trash/" failonerror="false" />
</tasks>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
...

It can be executed using following command.

mvn clean -Pclean

During second profiling round I found again that there is 50% of time consumed by tasks which are without any use during quick code-see development. The tasks were unit tests (please do not beat me, I can explain it ;-), integration tests, unit test coverage, checkstyle, findbugs, and other checks.

Those task of course should be performed before every commit to SCM, or the continuous integration build will fail, if there are some violations.

What I wanted, was one more profile - qbuild - which will disable those tasks if active. The reason why I didn't want to include it in already existing qclean profile was that sometimes developers want to run build with qclean profile, but also wants to have all those checks enabled.

Switching off the unit tests is simple.

...
<profile>
<id>qbuild</id>
<properties>
<maven.test.skip>true</maven.test.skip>
</properties>
<profile>
...

It will neither compile them.

To switch off other stuff I had to know ids of the executions used to configure plugins within the build process definition.

...
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>cobertura-maven-plugin</artifactId>
<executions>
<execution>
<id>verify-coverage</id>
<phase>verify</phase>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<executions>
<execution>
<id>verify-checkstyle</id>
<phase>verify</phase>
<goals>
<goal>checkstyle</goal>
</goals>
</execution>
</executions>
</plugin>
...

The snapshot shows us example of such existing configurations from our POM file.

Then I disabled them in qbuild profile simply hooking to non existing build phase - noPhase.

...
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>cobertura-maven-plugin</artifactId>
<executions>
<execution>
<id>verify-coverage</id>
<phase>noPhase</phase>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<executions>
<execution>
<id>verify-checkstyle</id>
<phase>noPhase</phase>
</execution>
</executions>
</plugin>
</plugins>
</build>
...

The developers then were able to run build four times faster and without any unused build artifacts.

mvn clean install -Pqclean,qbuild

They were all very satisfied with the result of the tuning and the "minus" never appeared again in the further retrospectives !!!

Wednesday 1 August 2007

Refactor singleton out of your code

There are a lot of posts about that the Singleton design pattern [GoF] is not pattern but anti-pattern (eg. SINGLETON - the anti-pattern!). Some posts also proposes an alternatives for Singleton design pattern or solutions for already existing Singleton design pattern problem (eg. Patterns I Hate #1: Singleton). But I didn't found any post which describes solution for big project with bunch of singleton class usage without big effort of rewriting many places by hand.

Because I was in situation that I should refactor a big project containing bunch of Singletons, I was thinking about some automatic way how to remove them. I found a solution using several refactorings. After bellow described sequence of steps you can replace Singleton design pattern by Registry design pattern [P of EAA] which solves all crucial problems of a Singleton, like testing possibility, singleton subclassing, ....

Lets have simple singleton class which we want to refactor to access it's instance not as a singleton but using some registry.

public class SingletonClass {
private static final SingletonClass INSTANCE = new SingletonClass();

public static SingletonClass getInstance() {
return INSTANCE;
}

private SingletonClass() {
super();
}

public void voidMethod(String param) {
System.out.println("Doing something in method returning void");
}

public Object objectMethod(String param) {
System.out.println("Doing something in method returning Object");
return new String("This is an String");
}
}

and also lets have very simple client class which access the singleton

public class Client {
public void clientMethod() {
SingletonClass.getInstance().voidMethod("param");
Object object = SingletonClass.getInstance().objectMethod("param");
}
}
  • At the first I create simple interface for Registry with two simple methods - getter and setter for instance of class which is currently implemented as singleton (SingletonClass in my example). There are also other variants of Registry implementation. For example you can have one generic methods getService and setService with some selector as a parameter (String ID, service class, ...). But I choosed the simplest one.

    public interface ISingletonRegistry {
    SingletonClass getSingletonClass();
    void setSingletonClass(SingletonClass singleton);
    }

  • Make SingletonClass's constructor public

  • Then implement the interface as a simplest Registry design pattern implementation - SingletonRegistry.

    public class SingletonRegistry implements ISingletonRegistry {

    private static final SingletonRegistry INSTANCE = new SingletonRegistry();

    private SingletonClass singleton;

    public static ISingletonRegistry getInstance() {
    return INSTANCE;
    }

    private SingletonRegistry() {
    //this is the reason why the constructor should be public
    singleton = new SingletonClass();
    }

    public SingletonClass getSingletonClass() {
    return singleton;
    }

    public void setSingletonClass(SingletonClass singleton) {
    this.singleton = singleton;
    }
    }

    You can see that I implemented it as singleton (of course using practise described in one of my previous posts - Singleton pattern implementation in 4 steps :-). I know that a lot of readers will have animadversions that I am creating new singleton when I want to remove other singletons. But realize that singleton is not evil every time. Also Service Locator design pattern uses singleton to access ServiceLocator instance.
    You can also see that constructor of the class initializes reference to SingletonClass instance. Of course this is just possibility. You can use some mechanism of configuration of the singleton instances. But note that it should be performed before you use SingletonRegistry by some client for the first time by. Anyway this implicit setup can stay here because you are able to set different instance of SingletonClass using setSingletonClass method.

  • Change getInstance method of SingletonClass to get instance from SingletonRegistry.

    public static SingletonClass getInstance() {
    return SingletonRegistry.getInstance().getSingletonClass();
    }

  • Over getInstance method of SingletonClass perform Inline... refactoring (Alt+Shift+I), mark 'Delete method declaration' chekbox and click OK. The method getInstance from SingletonClass disappears and all it's client classes uses SingletonRegistry to access SingletonClass.

    public class Client {
    public void clientMethod() {
    SingletonRegistry.getInstance() .getSingletonClass().voidMethod("param");
    Object object = SingletonRegistry.getInstance() .getSingletonClass().objectMethod("param");
    }
    }

  • Delete INSTANCE constant from SingletonClass class.

  • As a cherry on top of a cake you can Extract Interface (Alt+Shift+T, E) from SingletonClass. Write just new interface name, select all methods from SingletonClass which you can extract into new interface and press OK. All references to SingletonClass will be refactored to references to your newly created interface.
And that is all. Now you have SingletonClass with totally same functionality but you are able to mock it, extend or replace by different implementation setting up your instance of SingletonClass to SingletonRegistry.

References:

Monday 2 July 2007

My first real Java application

I just found very funny Java application in my archive. It was year 1999 and I was studying college in Ostrava. We had strange theme called Java Technologies. Following applet was my semestral project for the theme. It is very simple, multi threaded (!!!), simulation of cross way. A colored squares are cars and yellow squares in their corners are flashers. The cars follows traffic lights and right-hand rule. Look and enjoy :-)






Do you want sources? Download crossway.zip

What I am missing on del.icio.us

As member of a team I was looking for some solution which will enable us to share bookmarks across the team. Because I have some good experiences with del.icio.us bookmark manager and it also fulfil basic conditions, my focus was clear from the beginning.

The basic conditions were:

  1. bookmarks should be shared among all team members
  2. because the members are traveling and then they are out of corporate network we need access to bookmarks from anywhere
  3. posting and using bookmark should be as simple as possible
Then I identified additional conditions:
  1. some members uses IE, some members uses Mozilla
  2. at least some members of team (eg. me :-) has it's own personal account on del.icio.us and they want to use bookmarks from both of them at a time.
My idea was that every member will have it's own account and will be logged in using browser extension. There will be also team specific account (TA) and every member will add it into it's network. At the beginning I thought that it will enable me to see my and TA's bookmarks on one heap and I will also be able to add for:ta tag from a suggestions when posting new team bookmark. But:
  • network member's bookmarks are not merged between user's bookmarks (I mean merged just for view)
  • for:ta tag suggestion is present, but it adds bookmarks just into "links for you" section of TA (BTW at the moment it is impossible to remove proposed bookmark - see del.icio.us FAQ)
Those two problems cause that:
  • we can't use personal and TA account together with browser extension
  • somebody should be responsible for accepting proposed bookmarks of TA from "links for you"
But I found workarounds (just for Mozilla which implements Live Bookmarks). Both bookmark sets (TA's bookmarks and TA's "links for you" bookmarks) provides RSS feeds. So every member will have two Live Bookmarks through which he can access both groups of shared bookmarks. But the solution still have several imperfections:
  • you cant use browser extension to access team bookmarks
  • you can't search bookmarks by tags (but you can create more Live Bookmarks for each or some tags)
  • when somebody save bookmarks from "links for you" to main group the bookmark is present in both groups
Maybe we are trying to get pears from apple tree. I also know that del.icio.us want to stay as simple as possible and they do it very well, but still I have some proposals to del.icio.us team:
  • improve "links for you" functionality
    • user can define list of trusted users and their proposed bookmarks will be added directly (with some flag) into "my bookmarks"
    • provide possibility to delete bookmarks from "links from you" group (in progress - see del.icio.us FAQ)
  • improve "network" functionality
    • provide possiblity to configure you network member to have his bookmarks merged (just for view) between your bookmarks
  • provide RSS feeds containing users tags
Anyway, at the moment we use it with workarounds and we are happy at least with this solutions, but it can be better.

Thursday 14 June 2007

Intercepting Filter Design Pattern

Intercepting Filter is classified as an enterprise or J2EE design pattern. Intercepting Filter design pattern is typically used in cases where some messages should be processed but the target processing should be wrapped into chain of arbitrary pre/post processings as are authorization checks, message resources transformations (eg. translation, escaping), ...

Intercepting Filter design pattern is built from five basic components:

  1. FilterManager
  2. FilterChain
  3. IFilter interface
  4. Context
  5. ITarget interface

FilterManager - FilterManager class is a boundary class of Intercepting Filter design pattern. It means that a client uses it as an interface to process some target operation (represented by ITarget interface) in a specific context (represented by abstract interface Context). FilterManager is then typically responsible for FilterChain creation, it's setup (concrete filter set configuration, target propagation, ...) and finally it's execution.

FilterChain - FilterChain class is basicaly data object carrying list of filters and the target, with just very simple logic of filter queue.

IFilter - IFilter interface has just one method. The method is typically implemented as captured on following diagram. It makes some pre processing, and if everything is OK than it calls FilterChain to process to following filter.

If something is wrong (some data are wrong or missing in Context) then it can return or throw exception.

When the target is reached through chain of filters, it is processed, and the thread is returning back same way, so it can make some post processing in every filter, during which it can also make some changes to Context (eg. response translation, escaping, ...).

ITarget - ITarget interface should be implemented as message receiver or it's wrapper.

Context - Context class in this presentation of Intercepting Filter design pattern should be considered as very abstract. It means that it can be totally different type of class or it can be set of classes or it can be some Map, etc. For example it can be ServletRequest together with ServletResponse when we are talking about Intercepting Filter design pattern implementation defined by Sun - Core J2EE Patterns - Intercepting Filter.

Implementation of Intercepting Filter design pattern is pretty simple. The most complex component can be FilterManager which can implement some sophisticated way of configuration which defines mapping filters to context and their configuration. But whenever it is implemented it is very simple to extend message processing by new features or change its behavior.

Related patterns:
  • Decorator [GoF]
    • Does not have FilterChain - the order of filters is defined wrapping one filter into another.
  • Template Method [GoF]
    • Very simple version of Intercepting Filter design pattern implementing Filter Chain (template method) and IFilter implementations (methods called from template method) in one class.
  • Chain of Responsibility [GoF]
    • Does not have FilterChain - the order of "interceptors" is defined as chain of successors.
  • Interceptor [POSA2]
    • Similar also to Chain of Responsibility but the chain of successors is not "hard coded" but propagated to "interceptor" as parameter of the method call.
  • Pipes and Filters [POSA1]
    • The Pipes and Filters design pattern is oriented basically for stream related use cases. It also doesn't provide implicit possibility of post processing.
References: