Wednesday, June 4, 2014

JMX, MXBeans – random thoughts and ideas

JMX, MXBeans – random thoughts and ideas


JMX has been around for a while. A very common use for JMX is to add it as a debug interface to a web application (https://rterp.wordpress.com/tag/mxbean/). The truth is that JMX is much more than that.

The core of JMX is MBean’s that are created in your application and can then be manipulated via the JMX framework. The concept of JMX reminds me of the DCOM of Microsoft architecture and CORBA (see https://jcp.org/en/jsr/detail?id=70 for a bridge between the two). In essence you create an MBean on the server (in the MBeanServer), and you can then access this information from any other machine.

The objects in JMX are organized in an ObjectName, which is a tree hierarchy of names. This concept is actually very similar to the SNMP MIB structure.

JMX supports attributes, operations and notifications. I have seen companies build full infrastructures of application monitoring on top of JMX (including synchronizing information between machines via JMX). Since you have attributes you can save state within the MBean. With the operations you can have method actions externalized, and with the notification you can have external client register for events without the hassle of creating a message bus or structure for notification (of course spring support of all JMX functionality: http://docs.spring.io/spring-integration/docs/2.0.0.RC1/reference/html/jmx.html).

Since the concept of the JMX is very open, multiple adapters have been created around the JMX ecosystem.  Java comes with the JConsole application, so you can easily manage your MBeans even remotely. There are HTML adapters, and even SNMP adapters. So if you have a center of data that you need to share with multiple clients in multiple formats you should have a look at JMX:




If you need export you data from JMX to other platforms (like GraphIte  or Gangila) there is a nice package called JMXTrans that comes with some build in  writers and a framework to create new ones (

If all you need is to pass the information over http protocol that you can have a look at jolokia (http://www.jolokia.org/).  

Since is seems for some there is confusion between the two frameworks, both authors answered in the following question on the stackoverflow site: http://stackoverflow.com/questions/10151536/what-is-the-difference-between-jolokia-jmxtrans-when-to-choose-one-over-the


MXBean

Any class can be added to the JMX MBeanServer. If all the external actions (attributes, operations, notifications) use basic java types then you have no problem. You can use a self-written class, though the reading client must have this class in its classpath – a big constraint. To solve this issue JMX added the MXBean (http://docs.oracle.com/javase/7/docs/api/javax/management/MXBean.html) to support “open” types of java objects.

Oracle: An MXBean is a type of MBean that references only a predefined set of data types.
In other words by constraining the data types of you bean, JMX can now translate you “private classes” to a generic format using CompositeData and TabularData classes.

To use an MXBean you either implement an interface with MXBean suffix or use the @MXBean annotation. The other constraint of MXBean is that you must add the annotation @ConstructorProperties to your constructors.

When you use a concrete class in your code the MBeanServer will serialize the class into a generic class of the types:  CompositeData (http://docs.oracle.com/javase/7/docs/api/javax/management/openmbean/CompositeData.html), TabularData http://docs.oracle.com/javase/7/docs/api/javax/management/openmbean/TabularData.html.

These classes are similar to a hashmap, though they include information as to the type of each field. This way and client that supports the JMX MXBean can serialize this object and use it. If your client has your propriety class then you can easily serialize the MXBean back to your class.

The MBeanServer will validate the field types of all MXBean’s to make sure you did not use invalid types. Since you can externalize complex classes, the MXBeanServer will serialize it to one of the classes in the javax.management.openmbean package.

For most cases you do not need to add any special code, since one the MXBean is registered in the MBeanServer any access to this server via a client, the server will property serialize the class. The only case that you need to write code is the case of notification and using the user data object.

Notifications

To send a broadcast notification you create the Notification Object (http://docs.oracle.com/javase/tutorial/jmx/notifs/) The notification object has some basic field like type, source, message and other fields. If you want to add your own typed payload you need to use the User Data object. For complex classes you should be using the CompositeData or the TabularData. 

To create this class you need to specify all the fields and their types, for example:
String[] names = {"poolName", "usage", "count" };
Object[] values = { null, null, null };
OpenType[] types = { SimpleType.STRING, memoryUsageCompositeType,
            SimpleType.LONG };
CompositeType compositeType = getCompositeType(names, types);
CompositeData data = new CompositeDataSupport(compositeType, names, values);

This is obviously not very type safe or developer oriented. The class we would like to use would be like:
public class PoolUsage {
       String poolName;
       MemoryUsage usage;
       Integer count;
}

To solve this issue we will use a trick with the MBeanServer. As stated above if your class is an attribute or used in an operation the MBeanServer will do all the transformations for you. The problem is in the User Data scenario where you need to manually add the class. The trick is to have a helper class that inherits from StandardMBean. This class will have an internal field with your typed class. The trick is then when you want to serialize your class, you create the help class, assign the field with the value and the get the same field but via the getAttribute of the StandardMBean. This way the JMX framework will do the serialization for you. The other direction is the same. You create the helper class and assign the field using the JMX setAttribute (setting the composite data you got in the notification) and the retrieve the specific class using the standard getter.

Sample code:

public class PoolUsageSerializer extends StandardMBean implements PoolUsageInterface{
       private final static String ATTRIBUTE_NAME = "PoolUsage";
       protected PoolUsage poolUsage;

       public PoolUsage getPoolUsage() { return poolUsage;    }

       public void setPoolUsage(PoolUsage poolUsage) { this.poolUsage = poolUsage; }

       public PoolUsageSerializer() { super(PoolUsageInterface.class, true); }


       public static Object serialize(PoolUsage poolUsage) {
              PoolUsageSerializer me = new PoolUsageSerializer();
              me.setPoolUsage(data);
              return me.getAttribute(ATTRIBUTE_NAME);
       }

       public static PoolUsage deserialize(Object userData) {
              PoolUsageSerializer me = new PoolUsageSerializer();
              Attribute attr = new Attribute(ATTRIBUTE_NAME, userData);
              me.setAttribute(attr);
              return me.getPoolUsage();
       }
}

For more information see:

Security

JMX also supports the option to secure the server with user name and password. You can also assign roles (read/write) per user, and even secure communications via ssl. For securing your JMX on tomcat see http://tomcat.apache.org/tomcat-7.0-doc/monitoring.html.



Tuesday, April 29, 2014

Jenkins – Branches

Jenkins – Branches

We have an application that to fully test it with different phases we would need about 15 jobs (see the following post for simplification of jobs: http://javaandroidandrest.blogspot.com/2014/03/jenkins-multijob-plugin.html).  The problem with this is that once you branch your trunk version you do not want to duplicate all jobs.
One solution for this problem is to use the “Job Generator Plugin” (https://wiki.jenkins-ci.org/display/JENKINS/Job+Generator+Plugin). With this plugin you create a template job, and every time you run it, it will create a new job for you and based on the template values and insert the relevant parameters. This is a nice plugin to save the duplicate job and then update with all parameters, but in the end you have all jobs duplicated, and if you find a bug in the job, you need to update it for all duplications.
I decided to go in a different direction. I created my base of jobs that will work with any version of my application (trunk or branches). The only difference between jobs of different branches is the location in the SCM and the location on the disk of the build machine (we need to have separate workspaces for each branch). There are sometimes other parameters like which version of java to compile with, but as you will see all of this is not an issue.
The first plugin that we need to add is “EnvInject Plugin” (https://wiki.jenkins-ci.org/display/JENKINS/EnvInject+Plugin). This plugin will allow us to inject values to parameters in the job based on a properties file.

I create a properties file per version and per OS (I run some of the jobs on different OS’s for validation).
The value of my property file can be something like (trunk.windows.properties):
#March 26 2014
JAVA_HOME=D:\ins\java\jdk1.7.0_40
PERFORCE_MAP=//depot/Foundation/dev
MY_VERSION=trunk
APP_VERSION=current-SNAPSHOT
BASE_DIR=D:\Perforce_WS\
and for linux (trunk.linux.properties):
#March 26 2014
JAVA_HOME=/usr/java/jdk1.7.0_40/
PERFORCE_MAP=//depot/Foundation/dev
MY_VERSION=trunk
APP_VERSION=current-SNAPSHOT
BASE_DIR=/fnduser3/fnd/dev/
What is nice about this plugin is that the property files all need to sit only on the master machine, and Jenkins will copy the file to each slave and inject the properties at the beginning of the job. So in the end I have two files per branch (one for each OS) and in the file I put the different values of the parameters. You need to remember to parameterize the workspace per version, so that you can run the same job in parallel for different versions, and for easy debugging of the workspace.
So now all my jobs are version agnostic. What I need is to pass into the job is the version that I need (I have a job per OS since there might be different settings in the job per OS [the parameter file is per version]). To pass the parameter I add a parameter to the build (https://wiki.jenkins-ci.org/display/JENKINS/Parameterized+Build), my default is trunk since this is the most used value.



The only part of the job that cannot be parameterized is the SCM since we need to poll the SCM for changes. So what we need to do is create a watcher job per branch. If you have different builds based on SCM path then you will need a watcher per path.
Each watcher will poll the SCM and when it detects a change it will then trigger the generic job, and pass into the job the parameter for the version of the build. The generic job will then inject all parameters based on the property file and run the job.
To summarize: we have a watcher job per version (polls SCM), which then triggers the version agnostic jobs and injects the current version running. The job then injects all parameters relevant to the version from the properties file.
But now we have the issue of debugging. All works fine and the one of the version agnostic jobs fails. Yes I can have a look in the log and find the version according to the injected properties file:



But this is not nice. So we will need another plugin – “Build Name Setter” (https://wiki.jenkins-ci.org/display/JENKINS/Build+Name+Setter+Plugin). This plugin will allow us to change the name of the run once it starts running. So we can configure all the version agnostics jobs:




What this will do is to rename the current job and add the version to the name of the job. This way in the history of each version agnostic job you can easily see which versions have run and passed and which have failed:





Once all jobs are running fine we sometimes want to have the compiled artifacts in a link of the job. This works fine in our case, but it is not very nice, since you need to go to the history of the version agnostic job to find the right version and then get the artifact. To overcome this issue we will use the “Copy Artifact Plugin” (https://wiki.jenkins-ci.org/display/JENKINS/Copy+Artifact+Plugin). This will allow us to copy artifacts from one job to another. So on your watcher jobs, you can copy all artifacts from the other jobs, put them all in an archive folder and then share them in the watcher job.







SQL Injection

We all know how the classic SQL injection works (http://en.wikipedia.org/wiki/SQL_injection). A new novel way to try the technique can be found in the following picture:

Thursday, March 20, 2014

Jenkins multijob plugin

Jenkins Multijob

My assumption is that you are familiar with continuous integration concepts (http://en.wikipedia.org/wiki/Continuous_integration), and we use Jenkins (http://jenkins-ci.org/) as our build machine.
I would like to describe our use of the Jenkins multijob (https://wiki.jenkins-ci.org/display/JENKINS/Multijob+Plugin). I think this plugin simplifies a lot of hassles.

 

Development phases

In every project you usually have the following stages during development:
Phase I: Build the product and run unit tests
Phase II: Deploy the product to a testing/integration environment
Phase III: Only if all tests in previous phases passed then deploy artifacts to the nexus server.
Phase IV: Create installation.
Depending on the topology of the computers phase I and II can be done on the same computer or on different computers.
The problem is that I don’t actually want to deploy my artifacts to the nexus until the integration tests pass. The reason for this is that there might be other teams in the company that have dependencies on these artifacts (snapshot) and then they might get an invalid version if the integration tests fail.
In the case that all phases are run on the same physical machine the solution is fairly easy. We can run each phase in a different job, and since they are all on the same machine we can then use the local .m2 repository to get the artifact for phase II from phase I, and then in a separate job to deploy the artifacts from phase I after phase III passes.
In the case of multiple machines there is no simple solution except to use new artifactId’s and then the first phase will deploy to nexus so that the next phases can get the artifacts, and in the end at phase III deploy with a new artifactId.

Jenkins job: upstream/downstream

Jenkins has two mechanisms for triggering jobs. You can trigger a job after you (downstream) or you can configure yourself after another job (upstream), see https://wiki.jenkins-ci.org/display/JENKINS/Terminology.
Once you have multiple jobs attached it is very hard to visualize what is happening, though you can use the dependency graph to see all the connections.
To understand the complexity of multiple Jenkins jobs I will describe our system build that has about 20 jobs for a single version.
We are creating a web service platform.  
The first phase a team will create wsdl’s for the web services (wsdl first). The wsdl’s are created will an internal tool, so there is a job to build the wsdl’s and validate them according to internal rules.
The second team creates the tools for the design team and a platform for the web services that a third team will use to actually implement the web services.
In the end there is an installation package that is build.
Since the second team creates the tools and web service platform, we need all modules to work on both windows and linux so we want to run the jobs on both windows and linux.
Jobs:
                Core team:
§  sdk compile
§  integration test (mock web services)
§  deploy
                Design team
§  wsdl’s validation with sdk
§  deploy wsdls.
                Implementation Team
§  Import wsdl
§  Compile application
§  Integration tests (real web services)
§  Build installation
§  Deploy
We need to create a job for each built above. Then we need to chain them all. Some of the jobs can run in parallel (windows | linux , design | implementation). With the regular jobs in jenkins this is difficult, and very hard to maintain. It is hard to know where to add new jobs and what the building blocks are.
Using multijob plugin we can create blocks as written above. The structure of multijob allows us to define which jobs are parallel and which are synchronic.  Example for a multijob to build the core on windows:
I have another multijob that looks the same for linux.



A full view of our build will look like:





Once the build starts, it will create two parallel jobs, one for windows, and once for linux (each one has 3 jobs). Only when both finish successfully will the next phase of design and implementation run (also in parallel).
Other advantages of the multijob:
·         You can easily add triggers after the building blocks like emails.
·         Visualization of the process is easy and non-technical people understand the process.

How does the console of the multijob look like (I added the timestamp plugin also):



The multijob plugin is still in development stages, and the features that I would like to see added are:
·         Option to start build from job in the middle.
·         Canceling multijob should cancel all children
·         Defining which phases that fail the main build if they fail


Wednesday, February 26, 2014

Maven Profiles - per module

Maven Profiles

Activation per module

One of the hardest parts of maven, in my opinion, for someone coming from a programing background, is that the flow of maven is not a method call.
In a lot of cases we would like to create a batch process (group of plugins) that do something, and then invoke them in different modules. So what you would expect to do is to create a profile with all the plugins that you want, and then to active it on a module level.
For instance a simple example is to download all dependencies of the project to the target directory. In this case we will define the profile in the parent pom as follows:
<profile>
  <id>trace</id>
  <build>
    <plugins>
          <plugin>
               <artifactId>maven-dependency-plugin</artifactId>
               <groupId>org.apache.maven.plugins</groupId>
               <executions>
                 <execution>
                       <phase>package</phase>
                       <goals>
                         <goal>copy-dependencies</goal>
                       </goals>
                       <configuration>
                         <outputDirectory>${project.build.directory}/lib</outputDirectory>
                       </configuration>
                 </execution>
               </executions>
          </plugin>
         </plugins>
  </build>
</profile>

Now the problem is that when I invoke this profile it will be active on all modules. In this case it might be nice, but what if I have multiple modules and want to invoke the profile from only some of the modules.
This is where profile activation comes in. There is an option to activate a profile based on a system parameter like:
  <profile>
    <activation>
      <property>
        <name>debug</name>
      </property>
    </activation>
  </profile>

This will activate the profile if the system property of “debug” exists. Of course you can also add the option to check the value of the parameter:
  <profile>
    <activation>
      <property>
        <name>environment</name>
        <value>test</value>
      </property>
    </activation>
    ...
  </profile>

What is missing is that this feature only works with system parameters and not maven parameters. So you cannot set a parameter per module with a different value, and have maven evaluate it.
Another issue that you must always remember: when running maven, maven will compile the pom and create an effective pom (you can run mvn “effective-pom” to view it) and then run it. This means that you cannot do any manipulation during runtime that will affect the run of maven.

Solution – files

The solution that I found to run profiles per module is the use of file activation. Let’s say that I have some test modules. Sometimes I want to run a tomcat before running the tests and others not (just junit). In the standard way I will need to create a “tomcat” profile in each pom, and copy all the code that configures and deploys the pom.
I want to write the profile once in the parent pom, and then activate it only in those modules that need it. To do this I create the profile in the parent pom, and add the following activation:
<profile>
        <id>tomcat</id>
        <activation>
               <file>
                       <exists>profile.tomcat.active</exists>
               </file>
        </activation>                  
        ...
</profile>
 
Now for each module that I want to activate this profile, I need to have the file profile.tomcat.active in the pom directory. This will activate the profile, and the file needs to be there before you run the maven command, so you cannot copy it during the maven run.

This can also be used in a Jenkins scenario, where in the Jenkins job before running the maven, you add the files or delete them and the run the maven via Jenkins.

Monday, February 10, 2014

Sunday, February 9, 2014

Java Singleton

There are many implementations for a singleton in java.
Here is a full article of different ways and the advantages or disadvantages of each.

http://www.javaworld.com/article/2073352/core-java/simply-singleton.html

What surprised me is that the best solution is not in that blog.
The best solution seems to be using enum:

http://javarevisited.blogspot.co.il/2012/07/why-enum-singleton-are-better-in-java.html