Showing posts with label java. Show all posts
Showing posts with label java. Show all posts

Sunday, February 28, 2016

Spring Boot WebJar

Spring Boot WebJar


When creating an end to end application that includes client side code and server side code, we need a full application solution that can be easily built and deployed.
The client side may be written in different frameworks – angular, backbone. Spring Boot supports all of these solutions in multiple ways.

Static Resources

The strait forward way is to copy all resources from the client code (js, html, css...) to the static folder user the resources folder (see https://spring.io/blog/2013/12/19/serving-static-web-content-with-spring-boot).
The problem with this solution is that we are tightly couples between the client and server code. In order to compile the server code we need to copy the client compiled code to our resources folder.
To use this solution we need to create a Jenkins job that takes the client code, compiles it (using grunt or other tools) and then takes the folder that was created and copies it to the resources folder and then compile the server code.
This is definitely not the maven way. The solution that we would want is for the client code to create an artifact for us to add as a dependency in our pom file. The solution that has become a standard is what is called a webjar (www.webjar.com).

WebJar

A webjar is a zip file with the following folder structure:
META-INF\resources\webjars\artifactId\version\
When the client requests a resource from the server, Spring Boot looks in all its resource locations for the resource. The resource locations are: resource\static, resource\public, webjars.
So far the solution looks like it works nicely. The next issue is that webjar technology is not just a server side technology. Webjars come to solve another issue on the client side. When the client needs global functionality the solution for creating modules is to use webjars. What this means is that when the client needs a resource from the server, it needs to specify the artifactId and the version so that the server can retrieve it from the webjar.
For example:
<!DOCTYPE HTML>
<html xmlns:th="http://www.thymeleaf.org">
<head> 
    <title>Getting Started: Serving Web Content</title> 
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <script src="webjars/jquery/2.0.3/jquery.min.js"></script>
    <script type="text/javascript">
        $(document).ready(function() {
            $('p').animate({
                fontSize: '48px'
            }, "slow");
        });
    </script>
</head>
<body>
    <p th:text="'Hello, ' + ${name} + '!'" />
</body>
</html>

As you can see the client code references a version of the modules it wants to use: "webjars/jquery/2.0.3/jquery.min.js". This might be good in most cases, but the problem is that in a lot of cases our client side is a single page solution and does not necessarily need the webjar solution (with the full artifcatId and version). What then happens is that the client requests a resource but the server cannot find it since it needs the artifactid and version to find it within the webjar. We would like the client to request a js file or html file without the prefix of the webjar and for the server to find it and return it to the client.

ResourceHandlers

To do this we need to add our own bean that will supply the lookup for resources for spring.
We are actually going to solve here two other issues that arise with single page applications and Spring Boot.
In addition to the issue that we don’t want the client code to need to reference its own webjar, we would like to have the option to have all the client resources out of our jar, so that we can upgrade the client code without recompiling the server code.
Another issue with single page applications is that the client side navigation uses the url path to do this. So if the client side moves to an admin page, it will usually add to the url “/admin”. The problem with this is that if the use now refreshes his browser, the server will get a url request with “/admin” that the server does not have.
To solve this, we need to create another configuration bean that will inherit WebMvcConfigurerAdapter.  
The default implementation of spring is to search all webjars in the application (see https://spring.io/blog/2014/01/03/utilizing-webjars-in-spring-boot):
@Override
public void addResourceHandlers(ResourceHandlerRegistry registry) {
    if (!registry.hasMappingForPattern("/webjars/**")) {
        registry.addResourceHandler("/webjars/**").addResourceLocations(
                "classpath:/META-INF/resources/webjars/");
    }
    if (!registry.hasMappingForPattern("/**")) {
        registry.addResourceHandler("/**").addResourceLocations(
                RESOURCE_LOCATIONS);
    }
}

We will change this and override the addResourceHandler as follows:
override def addResourceHandlers(registry: ResourceHandlerRegistry): Unit = {

  if (!registry.hasMappingForPattern("/**")) {

    registry.addResourceHandler("/**").addResourceLocations(

      classpathResourceLocations: _*)

  }

}

This will now allow us to define a string array that will tell spring were to search for resources.
Our basic resource location array will be as follows:
private var classpathResourceLocations = List(

  "classpath:/META-INF/resources/", "classpath:/resources/",

  "classpath:/static/", "classpath:/public/")

This means, for spring it will first search in the resources folder of the classpath, and the in static folder, and lastly in the public folder (standard locations for resources).
But what we want is to add is the option to have a resource folder out of our jar and also to add the location of the webjar within our application without the client having to specific the full path.
classpathResourceLocations = (externalWebsite :: classpathResourceLocations) :+ s"classpath:/META-INF/resources/webjars/my.web-jar/$infoBuildVersion/"

The order that spring boot searches for resources is by the order of the string array for the locations. So the first entry will be a location to an external folder that can be specified in the application.properties in the format of a url (file:\...).
The second entry will be my web-jar with the fullpath of the resources inside the webjar. By doing this the client does not need to send the path, since we have added all the internal resources without the prefix.

Client Side Paths

 The last issue I wish to address is client side paths. The client side is written in angular so navigation in the java script is done by urls. So the client can have a url of : www.server.com\domain\method.
The “method” is not a endpoint in the server side but an endpoint in the client side. All this works fine until the user clicks on the page refresh. What then happens is that the URL is sent to the server, where the serve does not have the endpoint and returns a page not found error.
To fix this error we need to add View Controllers path in our WebMvcConfigurerAdapter as follows:
override def addViewControllers(registry: ViewControllerRegistry): Unit = {

  registry.addViewController("/").setViewName("forward:/index.html")

  registry.addViewController("/administration").setViewName("forward:/index.html")

  registry.addViewController("/license").setViewName("forward:/index.html")

  registry.addViewController("/404").setViewName("forward:/index.html")

  registry.addViewController("/download").setViewName("forward:/index.html")

}

In this method we will add all client side URL’s with a forward to the main page of the client indext.html. This will allow the server to accept the client endpoint and the redirect it back to the client.

Error endpoint

This all works fine unless the client endpoint is /error. Since this is a standard endpoint spring automatically implements this endpoint for us. Since we do not want this we must override spring’s default implementation as follows. In you configuration bean you need to add:
@Bean(name = Array("error"))

def defaultErrorView() : View  = {

  new RedirectView("index.html")

}

This implements the error bean of spring and tells spring to not use the spring implementation but in our case to redirect the error endpoint back to the client.



Saturday, January 16, 2016

Zip Streaming Log Files

Zip Streaming


The Scenario

I have a web server that supports debugging of my application. As part of the debugging I would like the option to download all log files that I have on my disk as a zip file.

Strait forward Solution

At first this looks like a very simple solution. I need to go over all the folders where I might have log files and add them to a zip file and then download the file.

Zip files

Java/Scala does not have a very convenient API for adding files to a zip file, but after searching the web it can be found (the issue is keeping the folder structure relative to a parent folder):
def createZip(folderParents: Map[File, List[File]]): ByteArrayOutputStream = {
  val bos = new ByteArrayOutputStream()
  val zipFile = new ZipOutputStream(bos)
  try {
    folderParents.foreach({ case (parentRoot, files) => createZip(zipFile, parentRoot, files) })
  }
  finally {
    zipFile.finish()
    zipFile.close()
    bos.close()
  }
  bos
}



def createZip(zipFile: ZipOutputStream, parentRoot: File, files: List[File]): Unit = {
  files.foreach(fileName => {
    val relativePath = parentRoot.toURI().relativize(fileName.toURI()).getPath()
    val zipEntry = new ZipEntry(relativePath)
    try {
      zipFile.putNextEntry(zipEntry)
      val buff = scala.io.Source.fromFile(fileName)
      val data = buff.foreach(zipFile.write(_))
      buff.close()
    }
    finally { zipFile.closeEntry() }
  }
  )
}

To then download the file from a spring boot server:
@RequestMapping(value = Array("/troubleshoot/logs"), method = Array(GET))@ResponseBodydef getLogFiles: ResponseEntity[Array[Byte]] = {
  val res = ZipUtils.createZip(folderParents)
  val headers = new HttpHeaders()
  headers.setContentType(MediaType.APPLICATION_OCTET_STREAM)
  headers.add("Content-Disposition","attachment; filename=\"log.zip\"")

  return ResponseEntity
    .ok()
    .headers(headers)
    .contentLength(res.size())
    .contentType(MediaType.parseMediaType("application/octet-stream"))
    .body(res.toByteArray);
}

Problem

The issues begin once we have very large log files. Once the log files are big (50mb and above) we start to see two issues. The first issue is that once the REST request is sent it can take a while until the file starts to download. The second issue is that we start to get out of memory exceptions.
The problem lies in the fact that to create the zip file I loaded all the files to memory and added them to the ZipOutputStream which is an in memory zip file. So what happens is that I need to load all the log files (maybe 500mb) and create the zip. Once the zip is created the download will begin. This also explains why I have an out of memory exception.


Solution

The ultimate solution to this problem is to stream the files to the client as we load them. This way we will solve both issues. We should not be loading all the files into memory.
To do this we need on one side to read the log files using a buffer and not load the full file (org.apache.commons.compress.utils.IOUtils is a good utility), and on the other side to stream the loaded data in a zip file - to do this we need to use GZIPOutputStream.
GZIPOutputStream allows streaming data to a zip file. The flow is: we load the log files, read into a buffer data, send the data to the zip stream, and the stream starts to download to the user. Then we continue this process, and we a have an end to end streaming will memory consumption of the buffer and no more.
The problem with GZIPOutputStream is that it does not support a folder structure. To solve this issue we will use the apache class TarArchiveOutputStream. The Tar Archive supports both folder structure and streaming. So we wrap the TarArchiveOutputStream with the GZIPOutputStream and then we get a folder structure that is both zipped and streamed down to the client.


def createTar(stream : OutputStream, folderParents: Map[File, List[File]]): Unit = {
  val tar = new TarArchiveOutputStream(stream)
  folderParents.foreach({ case (parentRoot, files) => createTar(tar, parentRoot, files) })
  tar.close()
}



def createTar(tOut: TarArchiveOutputStream, parentRoot: File, files: List[File]): Unit = {
  files.foreach(file => {
    val relativePath = parentRoot.toURI().relativize(file.toURI()).getPath()
    val tarEntry = new TarArchiveEntry(file)
    try {
      tarEntry.setSize(file.length())
      tOut.putArchiveEntry(tarEntry)
      IOUtils.copy(new FileInputStream(file), tOut);
    }
    finally {  tOut.closeArchiveEntry();   }
  })
}

val gzos = new GZIPOutputStream(outputStream)val buff = createTar(gzos,folderParents)
outputStream.flush()
outputStream.close()
 



Wednesday, June 4, 2014

JMX, MXBeans – random thoughts and ideas

JMX, MXBeans – random thoughts and ideas


JMX has been around for a while. A very common use for JMX is to add it as a debug interface to a web application (https://rterp.wordpress.com/tag/mxbean/). The truth is that JMX is much more than that.

The core of JMX is MBean’s that are created in your application and can then be manipulated via the JMX framework. The concept of JMX reminds me of the DCOM of Microsoft architecture and CORBA (see https://jcp.org/en/jsr/detail?id=70 for a bridge between the two). In essence you create an MBean on the server (in the MBeanServer), and you can then access this information from any other machine.

The objects in JMX are organized in an ObjectName, which is a tree hierarchy of names. This concept is actually very similar to the SNMP MIB structure.

JMX supports attributes, operations and notifications. I have seen companies build full infrastructures of application monitoring on top of JMX (including synchronizing information between machines via JMX). Since you have attributes you can save state within the MBean. With the operations you can have method actions externalized, and with the notification you can have external client register for events without the hassle of creating a message bus or structure for notification (of course spring support of all JMX functionality: http://docs.spring.io/spring-integration/docs/2.0.0.RC1/reference/html/jmx.html).

Since the concept of the JMX is very open, multiple adapters have been created around the JMX ecosystem.  Java comes with the JConsole application, so you can easily manage your MBeans even remotely. There are HTML adapters, and even SNMP adapters. So if you have a center of data that you need to share with multiple clients in multiple formats you should have a look at JMX:




If you need export you data from JMX to other platforms (like GraphIte  or Gangila) there is a nice package called JMXTrans that comes with some build in  writers and a framework to create new ones (

If all you need is to pass the information over http protocol that you can have a look at jolokia (http://www.jolokia.org/).  

Since is seems for some there is confusion between the two frameworks, both authors answered in the following question on the stackoverflow site: http://stackoverflow.com/questions/10151536/what-is-the-difference-between-jolokia-jmxtrans-when-to-choose-one-over-the


MXBean

Any class can be added to the JMX MBeanServer. If all the external actions (attributes, operations, notifications) use basic java types then you have no problem. You can use a self-written class, though the reading client must have this class in its classpath – a big constraint. To solve this issue JMX added the MXBean (http://docs.oracle.com/javase/7/docs/api/javax/management/MXBean.html) to support “open” types of java objects.

Oracle: An MXBean is a type of MBean that references only a predefined set of data types.
In other words by constraining the data types of you bean, JMX can now translate you “private classes” to a generic format using CompositeData and TabularData classes.

To use an MXBean you either implement an interface with MXBean suffix or use the @MXBean annotation. The other constraint of MXBean is that you must add the annotation @ConstructorProperties to your constructors.

When you use a concrete class in your code the MBeanServer will serialize the class into a generic class of the types:  CompositeData (http://docs.oracle.com/javase/7/docs/api/javax/management/openmbean/CompositeData.html), TabularData http://docs.oracle.com/javase/7/docs/api/javax/management/openmbean/TabularData.html.

These classes are similar to a hashmap, though they include information as to the type of each field. This way and client that supports the JMX MXBean can serialize this object and use it. If your client has your propriety class then you can easily serialize the MXBean back to your class.

The MBeanServer will validate the field types of all MXBean’s to make sure you did not use invalid types. Since you can externalize complex classes, the MXBeanServer will serialize it to one of the classes in the javax.management.openmbean package.

For most cases you do not need to add any special code, since one the MXBean is registered in the MBeanServer any access to this server via a client, the server will property serialize the class. The only case that you need to write code is the case of notification and using the user data object.

Notifications

To send a broadcast notification you create the Notification Object (http://docs.oracle.com/javase/tutorial/jmx/notifs/) The notification object has some basic field like type, source, message and other fields. If you want to add your own typed payload you need to use the User Data object. For complex classes you should be using the CompositeData or the TabularData. 

To create this class you need to specify all the fields and their types, for example:
String[] names = {"poolName", "usage", "count" };
Object[] values = { null, null, null };
OpenType[] types = { SimpleType.STRING, memoryUsageCompositeType,
            SimpleType.LONG };
CompositeType compositeType = getCompositeType(names, types);
CompositeData data = new CompositeDataSupport(compositeType, names, values);

This is obviously not very type safe or developer oriented. The class we would like to use would be like:
public class PoolUsage {
       String poolName;
       MemoryUsage usage;
       Integer count;
}

To solve this issue we will use a trick with the MBeanServer. As stated above if your class is an attribute or used in an operation the MBeanServer will do all the transformations for you. The problem is in the User Data scenario where you need to manually add the class. The trick is to have a helper class that inherits from StandardMBean. This class will have an internal field with your typed class. The trick is then when you want to serialize your class, you create the help class, assign the field with the value and the get the same field but via the getAttribute of the StandardMBean. This way the JMX framework will do the serialization for you. The other direction is the same. You create the helper class and assign the field using the JMX setAttribute (setting the composite data you got in the notification) and the retrieve the specific class using the standard getter.

Sample code:

public class PoolUsageSerializer extends StandardMBean implements PoolUsageInterface{
       private final static String ATTRIBUTE_NAME = "PoolUsage";
       protected PoolUsage poolUsage;

       public PoolUsage getPoolUsage() { return poolUsage;    }

       public void setPoolUsage(PoolUsage poolUsage) { this.poolUsage = poolUsage; }

       public PoolUsageSerializer() { super(PoolUsageInterface.class, true); }


       public static Object serialize(PoolUsage poolUsage) {
              PoolUsageSerializer me = new PoolUsageSerializer();
              me.setPoolUsage(data);
              return me.getAttribute(ATTRIBUTE_NAME);
       }

       public static PoolUsage deserialize(Object userData) {
              PoolUsageSerializer me = new PoolUsageSerializer();
              Attribute attr = new Attribute(ATTRIBUTE_NAME, userData);
              me.setAttribute(attr);
              return me.getPoolUsage();
       }
}

For more information see:

Security

JMX also supports the option to secure the server with user name and password. You can also assign roles (read/write) per user, and even secure communications via ssl. For securing your JMX on tomcat see http://tomcat.apache.org/tomcat-7.0-doc/monitoring.html.



Monday, January 6, 2014

@Configuration vs xml with prototypes

@Configuration vs xml with prototypes


There are two sides to using spring. One is the injection side, and the annotations are  @Autowire, @Inject, @Resource (see http://javaandroidandrest.blogspot.co.il/2013/05/spring-bean-overriding-between-projects.html). The other side is the definition of beans: @Component, @Repository, @Server.

The beginning of spring all injections were done via xml configuration files. With each version spring added more and more annotations so simplify matters.  When it came to injection annotations were usually used and xml files were used only when the annotation was not enough (like using maps).
For configuration files xml were the main default especially when complex configurations were including. One of the more complicated configurations is the prototype.

Prototypes

Before we go into the @Configuration, I would like to add a word on prototypes. In spring there are two scopes one is singleton and the other prototype. Singleton means that only one instance of that bean will exist in the spring context. While prototype means that multiple beans can exist. If you have a multiple singletons (from different types) that reference a prototype bean, each singleton will get its own instance of the prototype.
For example:
@Component
@Scope("prototype")
public class Chair {

}

@Component
public class Garage {

       @Autowired
       Chair chair;
}

@Component
public class House {

       @Autowired
       Chair chair;

}

Anyone that references the house of the garage will get the same instance as everyone else.  Though in the case of the chair both the garage and the house, will each have a separate instance of the chair.
Now when you need to generate during a runtime of the application another instance of the prototype then you need a lookup method.
For example let’s say that during runtime I need to create a new chair, so on my house class I will have a method of: protected Chair createChair();
We cannot create the chair with new Chair(), since then spring will not generate the class, and any injections within the chair class will not happen. So the spring solution to this is to create a lookup method. In the xml you write:
<bean id="house" class="com.domain.House"
       <lookup-method name="createChair" bean="com.domain.Chair" />
</bean>
And in the class you need to add the abstract method
@Component
abstract public class House {

       @Autowired
       Chair chair;

       abstract protected Chair createChair();
}

Spring will then implement this method and any time you call the createChair() a spring instance of the Chair will all injections will be created.

@ Configuration

With spring 3, spring add the @Configuration annotation. What this annotation allows you to do is to actually write all your xml in java code.
So for creating a bean of House we can either add @Component on the house class, we can add to the xml:
<bean id="house" class="com.domain.House"
      
</bean>
Or we can create a class with the annotation of @Configuration to create the bean:
@Configuration
public class ClassFactory {

       @Bean
       public House createHouse(){
              return new House();
       }
}

So for those people that prefer to write java code over xml files, you can now use configuration classes.
You client code still stays the same and you just inject the House using @Autowired.
Of course since the house is a singleton the method of “createHouse” will be called by spring only once and all places that inject the house will get this instance. In the case of prototypes you will add the method of:
       @Bean
       @Scope("prototype")
       public Chair createChair(){
              return new Chair();
       }

Now anytime that you inject a chair to a bean a new instance of the chair will be instantiated.
The configuration class allows you to set any properties that you want on the bean and write any code that you need. The spring framework will do its magic once the runtime finishes the method with the @Bean annotation. For example the method of “afterPropertiesSet” will be called once the @Bean method finishes.
A added advantage to the configuration bean is now you have a built in lookup method for creating prototype beans. Any class that has the @Configuration annotation is also a spring bean, so you can now inject this bean in your class and call the “createChair”. So you can now write in your house class:
@Component
public class House {

       @Autowired
       Chair chair;
      
       @Autowired
       ClassFactory factory;
      
       public Chair createChair(){
              return factory.createChair();
       }

}