I’ve seen Jenkins’ father Kohsuke in Hamburg earlier this week where he presented some stats on the project activities. I don’t remember the exact numbers, but he drove home the point that Jenkins is where the action is. It seems like Jenkins moved out and took the kids (the open source community) with her. Apparently lawyers don’t code. Who knew?

It appears that Hudson’s new foster parents Sonatype are driving the recent activities trying to reunite the once happy couple in the hope to revive the lethargic Hudson. It remains to be seen if Jenkins can forgive Hudson for what he put her through. After all the hard work to build up a life of her own, Hudson’s request for forgiveness seems almost too much to ask for.

• Managers, developers and customers (or all stakeholders if you will) collaborate via a wiki and issue tracking tool. These two components tend to be heavily intertwined and are often realized within a single tool. Project management is happening in this area as well.
• Developers can access the tasks in the issue tracker from within their development environment (IDE).
• While programming, quality assurance plugins inside the IDE point out potential bugs or design flaws to the developers using the same metrics as are measured by the quality analysis run by the central CI server.
• Whenever the system is in a consistent state developers commit their changes to their local clone of the Version Control System (VCS) repository. This should happen fairly frequently; several times a day. If you don’t commit frequently you are not doing continuous integration. (side note: The VCS should contain everything that is needed to produce the system from scratch, including database scripts, operating system setup (aka infrastructure as code), etc. – everything.)
• The local Continuous Integration (CI) tool notices the changes in the VCS repository, gets the latest changes from the master repository and kicks off a build of the changed code module. The build also runs a suite of unit tests to make sure the changes haven’t broken any existing code.

• After successfully building (and testing) the code the build tool stores the created artefacts in a repository manager. Note: I believe build artefacts don’t belong in your VCS for several reasons. You shouldn’t have to store them there either, because you should be able to recreate them from any version in your VCS. But there are also reasons why it might not even be a good idea to do so. Most importantly, it makes traceability harder. For each artefact I want to know which version of my source was used to build it. If build artefacts are stored in VCS as well and therefore create new versions themselves, this becomes harder. Also, all these snapshot versions that are produced many times a day, of which some might even break the build are not worth the disk space they occupy.
• Once the local commit build is successfully run and the snapshot artefacts stored in a repository manager the local CI tool pushes the changes to the master repository.
• Similar to what happened on the developer machine the central CI tool picks up the changes and runs a commit build. The produced artefact becomes a release candidate with a new version number and is stored in the repository manager, where it can be attained by later phases. (The maven-release-plugin is a great tool to automate this.)
• The produced artefacts are then automatically deployed to a test environment. The deployment tool gets the artefacts from the repository manager and deploys them on a test environment.
• There automated acceptance tests are run by the CI tool. If these acceptance tests are successful the tested artefacts are promoted to enter the last phase in your deployment pipeline – deployment to production. It is important to note that only the commit build phase produces new artefacts. All following phases are re-using these artefacts. An example on how to configure Maven not to rebuild the artefacts in later phases can be found here. Your own deployment pipeline may include more phases than just these two. For instance a manual test phase or performance test phase. Usually these can be performed in parallel and therefore require separate environments (hint: virtual machines are of great help here, but that’s a topic for another post).
• Deployment to production either happens automatically, if you are practicing continuous deployment (good on you), or by manually deploying from your CI server. Even if done manually this should only require the click of a button and should only be possible for artefacts that have successfully completed the previous phases in your deployment pipeline.
• Periodically the central CI tool runs a code quality analysis tool, which measures code quality by analysing code coverage and other metrics. Discovered violations can be accessed and fixed directly within the IDE.
• The IDE, the VCS, the CI and code quality analysis tool all feed information back to the issue tracking tool to provide a complete picture of the status of a tracked task.
• Automated monitoring keeps an eye on the system while it is running in the target environments. This monitoring can also be utilized during automated deployments to decide if a deployment was successful or needs to be backed out. It is mostly agreed upon that the monitor should check business metrics rather than technical aspects.

You can find the entire diagram in animated form here.

Here are my favorite tools for the above process:

• Wiki: Confluence
• Issue Tracking and Project Management: Jira
• IDE: Eclipse (with Mylyn for Jira issue/task integration and Sonar IDE to see all the technical debt right in your IDE where you need it the most)
• VCS: Git/Mercurial (I am undecided on this one)
• Build: Maven
• CI: Hudson (soon to be Jenkins)
• Repository Manager: Nexus
• Quality Analysis: Sonar
• Deployment: Cargo (for deploying to an application server) + Vagrant/Puppet (for creating, provisioning and managing environments)
• Monitoring: Splunk

Tweet this

I think this is the only sensible thing to do, because as soon as Oracle’s lawyers are done with suing everyone else they will try to extract money out of Hudson. The sooner Jenkins is completely independent, the better.

Most of us (should) have got used to the following five steps when changing/adding a piece of code:

1. Save/commit your changes in your IDE
2. Run the unit tests that cover the changed area
3. Run a private local build
4. Push your changes to the master repository
5. Your CI tool automatically picks up the changes and executes your deployment pipeline

You also probably didn’t execute steps three and four every time you did step one. Mostly not because it wouldn’t make sense to do so, but because these steps are time consuming. I believe that only the first step should be manual, all others can be triggered by their previous step.

1. Save/commit your changes in your IDE
2. Your IDE automatically runs the necessary unit tests (e.g. with JUnit Max or Infinitest)
3. Your distributed CI tool automatically pulls the latest changes from the master repository and runs a local build
4. Your distributed CI tool automatically pushes the changes to the master repository
5. Your central CI tool automatically picks up the changes and executes your deployment pipeline

When practicing rigorous TDD one could even argue that not even the first step needs to be manual. The first two steps could be switched around and your IDE could run the unit tests whenever the code compiles and then automatically save your changes when all unit tests are green.

1. Your IDE automatically runs the necessary unit tests whenever the code compiles
2. Your IDE automatically commits your changes when all unit tests are green
3. Your distributed CI tool automatically pulls the latest changes from the master repository and runs a local build
4. Your distributed CI tool automatically pushes the changes to the master repository
5. Your CI tool automatically picks up the changes and executes your deployment pipeline

No more manual steps at all. Just pure coding. It will be a bright future.

I set up the distributed CI part by installing a Hudson instance on my developer machine and another one on my CI server. When my local Hudson detects a change to my local Git repository it runs jobs to pull the latest changes from the master repository and update my clone, then executes a commit build and finally pushes the changes to the master repository, where the other Hudson instance runs the full deployment pipeline, including acceptance tests. This seems to work fine, but is of course quite a bit to set up and even harder to keep in sync without native tool support by Hudson.

P.S. I made a visual representation of how I am envisaging the development process with distributed CI. You can find it here. I will probably explain the diagram in more detail in a follow up post.

On top of my most wanted list is resource awareness. A resource aware CI tool would run builds depending on which resources are available at the time. I have my builds set up to run different categories of tests and I configured my CI tool so that:

• my unit test build runs on every commit to the version control system
• integration tests, that require external systems like databases or the file system are run in a subsequent build, which gets triggered after a successful completion of the unit test build
• functional tests, that require a deployed system are run on a set schedule, e.g. nightly (if there were any changes since the last build)

This setup is by no means following any hard rules, and your own setup most likely differs from mine depending on your own needs. However, what I am fairly certain of is that you had to configure your CI tool of choice when to run your tests and that you probably don’t adjust these settings once you found one that seems to work for you. This might be fine in your eyes, but I think we can do better. What if your CI tool decided when to run tests depending on which resources are available to do so? Why not run your functional tests right now if some of your processors are idle? Why not postpone slower integration tests if there are more commit builds in the queue that are waiting to have their unit tests run? Hudson made a step in this direction by automatically building multi module Maven projects in parallel if possible, but there is certainly a lot of room for improvement.

Closely related to the above is my desire for automatic test categorization. In order to run tests in different builds with different frequencies you need to split them up into separate categories. The main reason being to run the fastest tests first in order to receive feedback as quickly as possible. What if your CI tool did this for you? If a test runs under 200 milliseconds it’s probably a unit test. If a test accesses a database it’s at least an integration test. If a test is a Selenium test it is most certainly a functional test. With a set of rules like these the CI tool could split up your tests for you. This doesn’t mean you shouldn’t organize your tests in separate folders or packages, but it means that you don’t have to if you choose not to.

In addition to this categorization, I would like CI tools to be smarter about test execution order, similar to how JUnit Max or Infinitest are doing this in your IDE. This means that previously failing tests run first; quicker tests run before slower tests; new ones before older and so forth. (Update 1 July 2011: Sometimes dreams do come true. Check out Test Load Balancer)

And last but not least I would like a visual presentation of where my builds are at, which I have described in a previous post in more detail. Go‘s deployment pipelines are so far the best solution that I have seen in this respect.

Oh yeah, one more thing. If you have been practicing CI for a while you know that broken builds have to be fixed right away. In order to achieve this everyone gets notified as soon as the build breaks. Alarms go off, lights are flashing and emails and SMSes are sent out. Everyone gets interrupted. To avoid this undesirable event it has become common practice for developers to run a full build on their local machine before committing any code. This requires discipline by each and every developer though and I would prefer to see this automated. One way to achieve this could be a distributed CI environment, which could work as follows:

• On every commit/save to your local repository/workspace your local CI client performs an update from the central repository and runs all tests.
• If no newer version is in the central repository you may configure to skip the unit tests if these are run by your IDE anyway.
• On success a push/commit to the central repository is performed where the central CI server picks up the changes and runs configured builds.
• All the way you receive unobtrusive feedback in your IDE. Only if your commit happens to break the build more interruptive measures are taken.

For this to work efficiently with TDD, where you create failing tests all the time, the above mentioned smart test execution order is essential. Otherwise you keep running your entire unit test suite over and over again even though you already know that the test you just wrote is going to fail.

‘But why all this?’ you ask. There are two main objectives for the features described above: Minimizing manual work and speeding up feedback.

Photo by Martin Kenny

I say “traditional” release management for the lack of a better categorisation. What I mean by that is having separate phases (and teams) for development, system test, user acceptance test (UAT) and finally production. The problem is that it just takes too long after the developer commits his code until it gets eventually deployed to production. In my experience, this is mostly due to delays in hand offs from one phase to another. And as we all know, thanks to Lean Thinking this is waste.

Why you (as a developer) should care

The deployments to these different environments are usually stressful, because they bare some amount of risk of taking down the system you are deploying to. And the closer you get to production, the higher the penalty for doing so becomes. Because of this risk, deployments usually take place when potential problems would have the least impact on the users of these environments, which in general happens to be out of business hours. And last but not least, deployments tend to be boring and tedious tasks. Or did you ever get excited about doing a release? Yeah, that’s what I thought.

Why you (as a manager) should care

Of course, you also care about the factors mentioned above, because you want your developers to be as happy as possible, right? But furthermore, you especially dislike the risks associated with deployments, because you have to factor them into all sorts of calculations. Additionally, all these delays between phases mean that your time to market for new features and bug fixes is way higher than it needs to be, which means you are loosing money. Constantly.

The obvious solution

It’s easy, right? The answer is continuous deployment. You fully automate your testing and deployment and you deploy straight to production after every commit. Sounds great. The problem is: Most organisations are not ready for full test automation or just not willing to give up manual testing. This may be for security or legal reasons, potential loss of revenue and/or reputation or something else. Bottom line is: Pure continuous deployment into production without any manual testing is not widely accepted. Period. I am not saying this goal is not worth aiming for. Trust me, I am all for it. I just don’t see it happening in the near future in most companies. In my opinion, the only way to practice continuous deployment into production is by having sophisticated real-time alerting combined with a system with built- in resilience. This would allow you to automatically detect and back out a failed deployment without an interruption of the system as a whole. Have a read of this blog post about how the guys at kaChing managed to achieve this.

A step in the right direction

Don’t worry, not all is lost. Even if continuous deployment is not achievable for you right now, that doesn’t mean you can’t improve your situation at all. You can take the first steps towards continuous deployment by decreasing the batch size of deployments and automating the deployment and hand-offs between the environments.

• Decreasing the batch size should be easily achieved by just allowing automatic deployments into your first test environment. Only, of course, after a successful build that runs all your automated tests.
• Automating the deployment may require some tailored scripts to suit your system. However, if you are using Hudson and your deployment artifact is a war or ear file you can use the deploy plugin, which is based on Cargo.
• Automating the hand-offs could be achieved by using a central release management dashboard that is used by developers, testers and managers.

So far I haven’t found any Hudson plugin that would provide this. What I have in mind is something like the following:

• Deployments can be configured to happen automatically or to require manual approval.
• When deployment to an environment finishes, a configured set of people gets notified
• The different environments where the application is running can be accessed directly from the dashboard
• When approving a build, and all previous builds have been approved as well (there might be dependencies), deploy to next environment.
• When rejecting a build
  1. A tester raises one or more issues for the build. The issue tracking system is tightly integrated in the dashboard and can be reached via a link, which pre-fills relevant fields about the current build and project. This automatically rejects the build.
  2. The developer that committed the change that triggered the rejected build gets notified.
  3. The developer fixes the issue (after creating a failing test to expose it, of course).
  4. The developer commits the fix with reference to the issue in the commit comment. This triggers a new automatic build, plus an automatic update of the issue and a notification to the tester who raised it (like the Trac SVN Policies Plugin).
  5. When the tester approves the new build the old build gets automatically approved as well, if this was the last outstanding issue for the old build.

In my mind it looks somewhat like this (click image to enlarge):

In order to do pure continuous deployment you could then configure the dashboard with only one environment (=production) with automatic deployment. You can, however, choose to set up as many intermediate environments as you like and also require manual approval of deployments to any of them.

Does anyone know if a Hudson plugin with this functionality exists or is in the making?

With this in place I believe deployments can be non-events, in the meaning of being effortless and stress free. This does not mean you should stop celebrating your releases, though.

Update [15.05.2010]: I finally had a chance to try out SecureCI, which is pretty much exactly what I described in this post. The guys from Coveros did a great job, so I suggest after reading this post you head over to their website, download SecureCI and give it a go. Thanks John for pointing this out to me.

In my last Java project, I set up a build server with Continuous Integration (CI) capability. I am a big fan of Test Driven Development (TDD) and I quite enjoyed Hudson telling us right away when someone checked in code that broke the build. It just gives you so much more confidence in your code and keeps it releasable at all times. In addition, we used Sonar to measure the quality of our code. I found it quite interesting to study how the different metrics changed over the course of the project. We paid particular attention to code coverage and tried to keep it as close to 100% as possible. This should happen naturally anyway if you are practicing TDD.

Setting all this up was a good experience to learn how it all fits together – But I learned my lesson and I don’t really want to do it over and over again. So, I started looking for a Virtual Appliance that had most, if not all, of these capabilities. Setting up a build server must have been done by other developers a million times before. Surely, someone along the way stuck it all on a Virtual Machine (VM) and made it open source, right? Well, that’s what I thought, but I couldn’t find any. So, I decided to create one myself. Unfortunately, I am lacking the hardware to do so at the moment. I have even already created a project for it on https://launchpad.net/java-build-server. Although, I haven’t come around to create the VM yet, I still wanted to share my idea here to maybe animate someone else to go ahead and give it a go. So here it is: my idea of a Java build server.

I planned to start with what I think of as a good enough build server: A VM based on Ubuntu Server JeOS with pre-configured installations of Subversion (Source Control), Hudson (Continuous Integration), Sonar (Quality Metrics), MySQL (to store Quality Metrics), Maven (Build) and Nexus (Enterprise Maven Repository). Once this is all working I am going to also install Trac (Wiki and issue tracking system). You might of course have your own preferences for tools to use for the tasks listed above. Feel free to swap out whatever tool you wish. You might already have an enterprise Maven repository. Fine, just use your existing one. All I am saying is, these tools will be on my build server VM; pre-configured as much as possible with standards found in the tools’ documentation, following the convention over configuration paradigm.

Creating the VM

Here is how I am planning to create my Java build server VM. I am using VMware Studio to create the VM. VMware Studio is a Virtual Appliance itself and I deployed it in VMware Server 2.0. First I created a basic VM based on Ubuntu 8.04.1. Studio automatically installs VMware tools and embeds an in-guest management component called Virtual Appliance Management Infrastructure (VAMI), which lets you configure the network settings of your VM after it has been built. This is essential, since you want your development team to be able to connect to your copy of the build server VM. Here is a list of the essential settings I used to create the base VM:

• ubuntu-8.04.1-dvd-i386
• 1 CPU
• 512 MB RAM
• 8 GB
• Network settings: DHCP (to automatically find an IP address)
• Target format: zip

After the build finished, I downloaded the zipped VM and deployed it on my VMware Server instance. I started the VM and assigned it a static IP address on the boot screen. I logged in and carried out the following steps:

• Update Ubuntu (sudo apt-get update && sudo apt-get upgrade)
• Add the universe and multiverse Ubuntu repositories
• Install Java and Tomcat (with admin packages)
• Install Subversion (SVN)
• Install Maven
• Install MySQL
• Deploy Hudson to Tomcat
• Deploy Sonar to Tomcat
• Deploy Nexus to Tomcat

Using the VM

Of course there are some project and company specific configurations that cannot be set in advance. Here is what is left to do:

• Install a VM container on a server (e.g. VMware Server)
• Download and start the VM in your container
• Set the network details on the boot screen
• Add your developers as users to Subversion (in /usr/local/svn/passwd-team)
• Create a project
• Create Hudson jobs to monitor your project

What’s next?

To make using this Java build server even easier I am also planning to create the following:

• a Maven archetype that only needs the IP address of the build server to be configured
• a script to create a new project on the build server (create SVN project, create Hudson jobs, etc.)

Image by indigoprime via Flickr