Jenkins multijob plugin

Jenkins Multijob

My assumption is that you are familiar with continuous integration concepts (http://en.wikipedia.org/wiki/Continuous_integration), and we use Jenkins (http://jenkins-ci.org/) as our build machine.

I would like to describe our use of the Jenkins multijob (https://wiki.jenkins-ci.org/display/JENKINS/Multijob+Plugin). I think this plugin simplifies a lot of hassles.

 

Development phases

In every project you usually have the following stages during development:

Phase I: Build the product and run unit tests

Phase II: Deploy the product to a testing/integration environment

Phase III: Only if all tests in previous phases passed then deploy artifacts to the nexus server.

Phase IV: Create installation.

Depending on the topology of the computers phase I and II can be done on the same computer or on different computers.

The problem is that I don’t actually want to deploy my artifacts to the nexus until the integration tests pass. The reason for this is that there might be other teams in the company that have dependencies on these artifacts (snapshot) and then they might get an invalid version if the integration tests fail.

In the case that all phases are run on the same physical machine the solution is fairly easy. We can run each phase in a different job, and since they are all on the same machine we can then use the local .m2 repository to get the artifact for phase II from phase I, and then in a separate job to deploy the artifacts from phase I after phase III passes.

In the case of multiple machines there is no simple solution except to use new artifactId’s and then the first phase will deploy to nexus so that the next phases can get the artifacts, and in the end at phase III deploy with a new artifactId.

Jenkins job: upstream/downstream

Jenkins has two mechanisms for triggering jobs. You can trigger a job after you (downstream) or you can configure yourself after another job (upstream), see https://wiki.jenkins-ci.org/display/JENKINS/Terminology.

Once you have multiple jobs attached it is very hard to visualize what is happening, though you can use the dependency graph to see all the connections.

To understand the complexity of multiple Jenkins jobs I will describe our system build that has about 20 jobs for a single version.

We are creating a web service platform.  

The first phase a team will create wsdl’s for the web services (wsdl first). The wsdl’s are created will an internal tool, so there is a job to build the wsdl’s and validate them according to internal rules.

The second team creates the tools for the design team and a platform for the web services that a third team will use to actually implement the web services.

In the end there is an installation package that is build.

Since the second team creates the tools and web service platform, we need all modules to work on both windows and linux so we want to run the jobs on both windows and linux.

Jobs:

                Core team:

§  sdk compile

§  integration test (mock web services)

§  deploy

                Design team

§  wsdl’s validation with sdk

§  deploy wsdls.

                Implementation Team

§  Import wsdl

§  Compile application

§  Integration tests (real web services)

§  Build installation

§  Deploy

We need to create a job for each built above. Then we need to chain them all. Some of the jobs can run in parallel (windows | linux , design | implementation). With the regular jobs in jenkins this is difficult, and very hard to maintain. It is hard to know where to add new jobs and what the building blocks are.

Using multijob plugin we can create blocks as written above. The structure of multijob allows us to define which jobs are parallel and which are synchronic.  Example for a multijob to build the core on windows:

I have another multijob that looks the same for linux.

A full view of our build will look like:

Once the build starts, it will create two parallel jobs, one for windows, and once for linux (each one has 3 jobs). Only when both finish successfully will the next phase of design and implementation run (also in parallel).

Other advantages of the multijob:

·         You can easily add triggers after the building blocks like emails.

·         Visualization of the process is easy and non-technical people understand the process.

How does the console of the multijob look like (I added the timestamp plugin also):

The multijob plugin is still in development stages, and the features that I would like to see added are:

·         Option to start build from job in the middle.

·         Canceling multijob should cancel all children

·         Defining which phases that fail the main build if they fail