Tuesday 17 February 2009

Speed up Maven build

From a sprint retrospective we got a lot of 'minuses' related to the speed of Maven build executed by every developer localy on their computers many times per day. The developers were frustrated that complete build tooks from 10 to 15 minutes and estimated that they spend from 30 to 60 minutes every day just waiting for build. What in a team containing 16 developers (it was two scrum teams) gives you from 8 to 16 manhours per day. It is really a lot so we decided to try to do something about it.

At the begining I have to say that if there is such problem in the project, it usualy mean that something is wrong with modularity, or your IDE should be replaced or used in a better way to support faster code-see iterations. But it wasn't true in our case. Our problem was legacy web application famework without any support in existing IDEs.

Since I was responsible for build process I started to work on this issue. At the first I started to talk with developers about how do they build it. They simply used the most common way of application building:

mvn clean install

to rebuild this all, or

mvn install

to just repack updated sources and resources.

So I started to "profile" the build and early found surprising fact that approximately 50% of the time is taken by 'clean' goal - I found first bottleneck.

To remove it I used simple trick - move/rename operation on the folder is immediate on most of the systems in comparison with delete operation.

...
<profile>
<id>qclean</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<executions>
<execution>
<id>rename_target</id>
<phase>pre-clean</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<tasks>
<tstamp>
<format property="targetTstamp"
pattern="yyyyMMdd-HHmmss" locale="en,US" />
</tstamp>
<move todir="trash/target-${targetTstamp}"
failonerror="false">
<fileset dir="target/" />
</move>
</tasks>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
...

In the code snapshot you can see definition of new qclean (means Quick Clean) profile. The profile contains configuration of maven-antrun-plugin to be executed in a 'pre-clean' phase. The Ant script at the first generates a timestamp string into a property 'targetTimestamp' using 'tstamp' task. Then it moves and renames 'target' folder to 'trash/target-${targetTstamp}'.

Now when developers executed build using following command they simply spent just half of the time waiting for build.

mvn clean install -Pqclean

Because the 'target' folder is just moved and not deleted it is necessary to clean also 'trash' folder sometimes. But you can do it as a background operation or while you are not using the computer at all. For this purpose I created one more profile named clean.

...
<profile>
<id>clean</id>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<executions>
<execution>
<id>clean_trash</id>
<phase>clean</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<tasks>
<delete dir="trash/" failonerror="false" />
</tasks>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</profile>
...

It can be executed using following command.

mvn clean -Pclean

During second profiling round I found again that there is 50% of time consumed by tasks which are without any use during quick code-see development. The tasks were unit tests (please do not beat me, I can explain it ;-), integration tests, unit test coverage, checkstyle, findbugs, and other checks.

Those task of course should be performed before every commit to SCM, or the continuous integration build will fail, if there are some violations.

What I wanted, was one more profile - qbuild - which will disable those tasks if active. The reason why I didn't want to include it in already existing qclean profile was that sometimes developers want to run build with qclean profile, but also wants to have all those checks enabled.

Switching off the unit tests is simple.

...
<profile>
<id>qbuild</id>
<properties>
<maven.test.skip>true</maven.test.skip>
</properties>
<profile>
...

It will neither compile them.

To switch off other stuff I had to know ids of the executions used to configure plugins within the build process definition.

...
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>cobertura-maven-plugin</artifactId>
<executions>
<execution>
<id>verify-coverage</id>
<phase>verify</phase>
<goals>
<goal>check</goal>
</goals>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<executions>
<execution>
<id>verify-checkstyle</id>
<phase>verify</phase>
<goals>
<goal>checkstyle</goal>
</goals>
</execution>
</executions>
</plugin>
...

The snapshot shows us example of such existing configurations from our POM file.

Then I disabled them in qbuild profile simply hooking to non existing build phase - noPhase.

...
<build>
<plugins>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>cobertura-maven-plugin</artifactId>
<executions>
<execution>
<id>verify-coverage</id>
<phase>noPhase</phase>
</execution>
</executions>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-checkstyle-plugin</artifactId>
<executions>
<execution>
<id>verify-checkstyle</id>
<phase>noPhase</phase>
</execution>
</executions>
</plugin>
</plugins>
</build>
...

The developers then were able to run build four times faster and without any unused build artifacts.

mvn clean install -Pqclean,qbuild

They were all very satisfied with the result of the tuning and the "minus" never appeared again in the further retrospectives !!!

AddThis Feed Button 9 comments:

Brian Fox said...

Clearly the clean plugin needs to be made more efficient, or simply supporting a move option like you've hacked up here. Another thing that dramatically speeds up builds is the inclusion of a Repository Manager. By having intelligent caching and routing optimization, you can easily see large performance gains in your builds..especially if you have several remote repositories that you use.

Anonymous said...

One other trick, but not everybody is ready to test it...

Is to use linux, there deleting is done in no time as it just remove inodes. I also use maven under linux, and 350 000 files (2/3 being generated) are removed instantly!

Anonymous said...

this is good, however if used in parent and then any of the child has profiles with antrun plugin with diff config, that child profile antrun plugin config is "included" while running this qclean

Anonymous said...

How to conditional move an directory?
For example:
if current dir = xxx then
don't move
else
move
end if

Robert said...

Can you explain how to use this with maven 2.x?

HP 60xl said...

It is great to know that you found a solution to your speed problem. Being able to accomplish things at your most preferred pace will surely bring great benefits.

Registry Booster said...

Hello

Great information in this post and I think this information will helpful for us.

Error 2908 said...

Nice post and its very useful for all the visitors of blog and also for me because I get good stuff from this post.

Anonymous said...

Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.8:run (check-invalid-patterns) on project parent: An Ant BuildException has occured:...

anyone suggest the solution for this Error...??