Jenkins – Branches

We have an application that to fully test it with different phases we would need about 15 jobs (see the following post for simplification of jobs: http://javaandroidandrest.blogspot.com/2014/03/jenkins-multijob-plugin.html).  The problem with this is that once you branch your trunk version you do not want to duplicate all jobs.

One solution for this problem is to use the “Job Generator Plugin” (https://wiki.jenkins-ci.org/display/JENKINS/Job+Generator+Plugin). With this plugin you create a template job, and every time you run it, it will create a new job for you and based on the template values and insert the relevant parameters. This is a nice plugin to save the duplicate job and then update with all parameters, but in the end you have all jobs duplicated, and if you find a bug in the job, you need to update it for all duplications.

I decided to go in a different direction. I created my base of jobs that will work with any version of my application (trunk or branches). The only difference between jobs of different branches is the location in the SCM and the location on the disk of the build machine (we need to have separate workspaces for each branch). There are sometimes other parameters like which version of java to compile with, but as you will see all of this is not an issue.

The first plugin that we need to add is “EnvInject Plugin” (https://wiki.jenkins-ci.org/display/JENKINS/EnvInject+Plugin). This plugin will allow us to inject values to parameters in the job based on a properties file.

I create a properties file per version and per OS (I run some of the jobs on different OS’s for validation).

The value of my property file can be something like (trunk.windows.properties):

#March 26 2014

JAVA_HOME=D:\ins\java\jdk1.7.0_40

PERFORCE_MAP=//depot/Foundation/dev

MY_VERSION=trunk

APP_VERSION=current-SNAPSHOT

BASE_DIR=D:\Perforce_WS\

and for linux (trunk.linux.properties):

#March 26 2014

JAVA_HOME=/usr/java/jdk1.7.0_40/

PERFORCE_MAP=//depot/Foundation/dev

MY_VERSION=trunk

APP_VERSION=current-SNAPSHOT

BASE_DIR=/fnduser3/fnd/dev/

What is nice about this plugin is that the property files all need to sit only on the master machine, and Jenkins will copy the file to each slave and inject the properties at the beginning of the job. So in the end I have two files per branch (one for each OS) and in the file I put the different values of the parameters. You need to remember to parameterize the workspace per version, so that you can run the same job in parallel for different versions, and for easy debugging of the workspace.

So now all my jobs are version agnostic. What I need is to pass into the job is the version that I need (I have a job per OS since there might be different settings in the job per OS [the parameter file is per version]). To pass the parameter I add a parameter to the build (https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Build), my default is trunk since this is the most used value.

The only part of the job that cannot be parameterized is the SCM since we need to poll the SCM for changes. So what we need to do is create a watcher job per branch. If you have different builds based on SCM path then you will need a watcher per path.

Each watcher will poll the SCM and when it detects a change it will then trigger the generic job, and pass into the job the parameter for the version of the build. The generic job will then inject all parameters based on the property file and run the job.

To summarize: we have a watcher job per version (polls SCM), which then triggers the version agnostic jobs and injects the current version running. The job then injects all parameters relevant to the version from the properties file.

But now we have the issue of debugging. All works fine and the one of the version agnostic jobs fails. Yes I can have a look in the log and find the version according to the injected properties file:

But this is not nice. So we will need another plugin – “Build Name Setter” (https://wiki.jenkins-ci.org/display/JENKINS/Build+Name+Setter+Plugin). This plugin will allow us to change the name of the run once it starts running. So we can configure all the version agnostics jobs:

What this will do is to rename the current job and add the version to the name of the job. This way in the history of each version agnostic job you can easily see which versions have run and passed and which have failed:

Once all jobs are running fine we sometimes want to have the compiled artifacts in a link of the job. This works fine in our case, but it is not very nice, since you need to go to the history of the version agnostic job to find the right version and then get the artifact. To overcome this issue we will use the “Copy Artifact Plugin” (https://wiki.jenkins-ci.org/display/JENKINS/Copy+Artifact+Plugin). This will allow us to copy artifacts from one job to another. So on your watcher jobs, you can copy all artifacts from the other jobs, put them all in an archive folder and then share them in the watcher job.

Backing up Jenkin Jobs

There are plugins that will back you full Jenkins server (https://wiki.jenkins-ci.org/display/JENKINS/Backup+Plugin). But what if in your development team you want to backup only your jobs and add the configuration of each job to your source control.

A simple way to do this is to write a batch file that uses the Jenkins cli. To get the jenkins-cli.jar you need to go to your server at the url: http://build_server/cli/. You should get the following:

You can download the jar from the link on the top of the page.

For each command you can see how to call it.

We will use two commands. The first is to get a list of the jobs you want to backup:

java -jar jenkins-cli.jar -s http://buildevsrv list-jobs abc > jobs.list

This will save the list, filtered by the work “abc” to a file jobs.list.

The for each entry in the file we do the following command:

java -jar jenkins-cli.jar -s http://buildevsrv get-job "job_name" --username %1 --password %2 > job_name.xml

Enter your user name and password (you need permission for configuration of the job). This will save the configuration to your disk. Now you can save it in your source control.

For a full script to get all jobs by a filter:

@echo off

echo getting list of jobs

java -jar jenkins-cli.jar -s http://buildevsrv list-jobs abc > jobs.list

for /f "delims=" %%i in (jobs.list) do (

                if NOT [%1] == [] (

                                echo getting config for: %%i

                                java -jar jenkins-cli.jar -s http://buildevsrv get-job "%%i" --username %1 --password %2 > %%i.xml

                )

)

Don’t forget to pass your username and password to the command.

Create a Java CLI application with Maven

Creating a CLI application is part of our daily routine. In java it is strait forward; all you need to do is to add a class with the main function:

public static void main(String[] args) {

}

This is nice for debugging from eclipse, but if you need to release this jar with the cli working, you will face the following issues:

1.       You need to add the main class to you manifest file and compile it in the jar.

2.       You need to create a folder with all the jar’s dependencies needed to run this cli.

Main Class

maven-jar-plugin

The first issue can be solved by using the maven-jar-plugin:

<plugin>

       <groupId>org.apache.maven.plugins</groupId>

       <artifactId>maven-jar-plugin</artifactId>

       <configuration>

              <archive>

                     <manifest>

                           <addClasspath>true</addClasspath>

                           <mainClass>com.package.Main</mainClass>

                     </manifest>

              </archive>

              <finalName>jarname-cli</finalName>

       </configuration>

</plugin>

This plugin will create a jar file by the name of jarname-cli.jar, will a manifest file with the following entry:

Main-Class: com.package.Main

This will allow us to run from the command line:

java -jar jarname-cli %1 %2.

Jar Dependencies

Though if your jar has dependencies to other jars you will receive the following error when running the command line:

Exception in thread "main" java.lang.NoClassDefFoundError:

This is because we still need to package all the dependency jars into the same folder.

To solve this you need two more plugins.

maven-dependency-plugin (http://maven.apache.org/plugins/maven-dependency-plugin/)

The first plugin will calculate all the dependencies and copy them to a directory (cli):

<plugin>

       <artifactId>maven-dependency-plugin</artifactId>

       <executions>

              <execution>

                     <phase>install</phase>

                     <goals>

                           <goal>copy-dependencies</goal>

                     </goals>

                     <configuration>

       <outputDirectory>${project.build.directory}/../cli/</outputDirectory>

                     </configuration>

              </execution>

       </executions>

</plugin>

This plugin will copy all dependencies even though you don’t need all of them (it will include testing and jars not necessarily in use). You can start to manage you dependencies with the exclude, but this way you will constantly need to check that new jars are not added. The other option is to tell the plugin only the jars you want added:

                     <configuration>

                           <includeScope>runtime</includeScope>

                           <includeArtifactIds>commons-cli,spring-core,commons-logging,freemarker</includeArtifactIds>

                           <outputDirectory>${project.build.directory}/../cli/</outputDirectory>

                     </configuration>

This way any new dependencies will not be copied to the directory.

Of course if there are dependencies that you need for the execution of your cli, you will need to add them. You can use either the includeArtifactIds, or includeGroupIds whichever is easier.

This plugin is nice since it will get all the dependencies, but what it does not do is copy current jar to your cli directory. For this you need the ant plugin to copy the current jar

maven-antrun-plugin (http://maven.apache.org/plugins/maven-antrun-plugin/)

<plugin>

       <groupId>org.apache.maven.plugins</groupId>

       <artifactId>maven-antrun-plugin</artifactId>

       <executions>

              <execution>

                     <id>Copy Main Jar To Cli Directory</id>

                     <phase>install</phase>

                     <configuration>

                           <tasks>

                                  <copy todir="${project.build.directory}/../cli/">

                                         <fileset dir="${project.build.directory}/" includes="*.jar"

                                                excludes="*sources*.jar,*javadoc*.jar" />

                                  </copy>

                           </tasks>

                     </configuration>

                     <goals>

                           <goal>run</goal>

                     </goals>

              </execution>

       </executions>

</plugin>

The action is very simple, it is a copy action from ant, that will copy the jar from the target directory.

maven-clean-plugin (http://maven.apache.org/plugins/maven-clean-plugin/)

Since your maven project will create the dir and all the jars needed, you need to add the clean plugin to remove the dir when running clean in maven.

<plugin>

       <artifactId>maven-clean-plugin</artifactId>

       <configuration>

              <filesets>

                     <fileset>

                           <directory>${project.build.directory}/../cli</directory>

                     </fileset>

              </filesets>

       </configuration>

</plugin>

This solution is good for most cases. But sometimes you need more flexibility. Occasionally you might want not to have a directory with all jar’s but you might want to pack them all in one jar (thought licensing must be considered). Also sometimes you might want to add more files or resources to the jar, or maybe create a zip file instead of the jar file. For this you have the assembly plugin. The dependency plugin is actually a subset of the assembly plugin, so you can do all the above and more.

maven-assembly-plugin (http://maven.apache.org/plugins/maven-assembly-plugin/)

We will not go into the whole plugin since you can do a lot with it (for a full set of options see http://maven.apache.org/plugins/maven-assembly-plugin/assembly.html).

                     <plugin>

                           <artifactId>maven-assembly-plugin</artifactId>

                           <configuration>

                                  <finalName>wsf_${version}</finalName>

                                  <appendAssemblyId>false</appendAssemblyId>

                                  <descriptors>

                                         <descriptor>src/main/resources/assembly.xml</descriptor>

                                  </descriptors>

                                  <outputDirectory>${project.build.directory}/../assembly/</outputDirectory>

                           </configuration>

                           <executions>

                                  <execution>

                                         <id>make-assembly</id>

                                         <!-- this is used for inheritance merges -->

                                         <phase>package</phase>

                                         <!-- bind to the packaging phase -->

                                         <goals>

                                                <goal>single</goal>

                                         </goals>

                                  </execution>

                           </executions>

                     </plugin>

As you can see we can do both creating a manifest file with the mainclass, and adding other files. You can enable enhanced features by assigning an assembly.xml and do for example:

<assembly>

        <id>bundle</id>

        <formats>

               <format>jar</format>

        </formats>

        <includeBaseDirectory>false</includeBaseDirectory>

        <dependencySets>

               <dependencySet>

                       <outputDirectory>/</outputDirectory>

                       <useProjectArtifact>true</useProjectArtifact>

                       <unpack>true</unpack>

                       <scope>runtime</scope>

                       <includes>

                               <include>*:commons-cli:*</include>

                               <include>*:spring-core:*</include>

                               <include>*:commons-logging:*</include>

                               <include>*:freemarker:*</include>

                       </includes>

               </dependencySet>

        </dependencySets>

</assembly>

Here you can notice that we used the option <unpack>true</unpack>. This will unpack all jar’s and then repack them all in one jar. This way you can bundle you cli into one jar only.