Jenkins Multijob
My assumption is that you are familiar with continuous
integration concepts (http://en.wikipedia.org/wiki/Continuous_integration),
and we use Jenkins (http://jenkins-ci.org/)
as our build machine.
I would like to describe our use of the Jenkins multijob (https://wiki.jenkins-ci.org/display/JENKINS/Multijob+Plugin).
I think this plugin simplifies a lot of hassles.
Development phases
In every project you usually have the following stages
during development:
Phase I: Build the product and run unit tests
Phase II: Deploy the product to a testing/integration
environment
Phase III: Only if all tests in previous phases passed then
deploy artifacts to the nexus server.
Phase IV: Create installation.
Depending on the topology of the computers phase I and II
can be done on the same computer or on different computers.
The problem is that I don’t actually want to deploy my
artifacts to the nexus until the integration tests pass. The reason for this is
that there might be other teams in the company that have dependencies on these
artifacts (snapshot) and then they might get an invalid version if the
integration tests fail.
In the case that all phases are run on the same physical
machine the solution is fairly easy. We can run each phase in a different job,
and since they are all on the same machine we can then use the local .m2
repository to get the artifact for phase II from phase I, and then in a
separate job to deploy the artifacts from phase I after phase III passes.
In the case of multiple machines there is no simple solution
except to use new artifactId’s and then the first phase will deploy to nexus so
that the next phases can get the artifacts, and in the end at phase III deploy
with a new artifactId.
Jenkins job: upstream/downstream
Jenkins has two mechanisms for triggering jobs. You can
trigger a job after you (downstream) or you can configure yourself after
another job (upstream), see https://wiki.jenkins-ci.org/display/JENKINS/Terminology.
Once you have multiple jobs attached it is very hard to
visualize what is happening, though you can use the dependency graph to see all
the connections.
To understand the complexity of multiple Jenkins jobs I will
describe our system build that has about 20 jobs for a single version.
We are creating a web service platform.
The first phase a team will create wsdl’s for the web
services (wsdl first). The wsdl’s are created will an internal tool, so there
is a job to build the wsdl’s and validate them according to internal rules.
The second team creates the tools for the design team
and a platform for the web services that a third team will use to
actually implement the web services.
In the end there is an installation package that is build.
Since the second team creates the tools and web service
platform, we need all modules to work on both windows and linux so we want to
run the jobs on both windows and linux.
Jobs:
Core
team:
§
sdk compile
§
integration test (mock web
services)
§
deploy
Design
team
§
wsdl’s validation with sdk
§
deploy wsdls.
Implementation
Team
§
Import wsdl
§
Compile application
§
Integration tests (real web
services)
§
Build installation
§
Deploy
We need to create a job for each built above. Then we need
to chain them all. Some of the jobs can run in parallel (windows | linux ,
design | implementation). With the regular jobs in jenkins this is difficult,
and very hard to maintain. It is hard to know where to add new jobs and what
the building blocks are.
Using multijob plugin we can create blocks as written above.
The structure of multijob allows us to define which jobs are parallel and which
are synchronic. Example for a multijob
to build the core on windows:
I have another multijob that looks the same for linux.
A full view of our build will look like:
Once the build starts, it will create two parallel jobs, one
for windows, and once for linux (each one has 3 jobs). Only when both finish
successfully will the next phase of design and implementation run (also in parallel).
Other advantages of the multijob:
·
You can easily add triggers
after the building blocks like emails.
·
Visualization of the
process is easy and non-technical people understand the process.
How does the console of the multijob look like (I added the
timestamp plugin also):
The multijob plugin is still in development stages, and the
features that I would like to see added are:
·
Option to start build from
job in the middle.
·
Canceling multijob should
cancel all children
·
Defining which phases that
fail the main build if they fail
