Monday, November 9, 2015

Spring Boot Curator and ELK

Spring Boot Curator and ELK - JSON Message Log

Application Server

One of the things I love about spring is the simplicity of things. To create an application server that has a health and metric rest interface is very simple. For the application server I use Spring Boot (http://projects.spring.io/spring-boot/). For adding the health and metrics I use the Spring Curator (https://github.com/spring-projects/spring-boot/tree/master/spring-boot-actuator).

Now that I have my application up and running, I wan’t to monitor the application with a nice GUI with history. To do this I use ELK – Elastic Search, LogStash and Kibana. Kibana is the GUI interface. Elastic Search is the Database, and LogStash enters the data.

To implement this I could have added a mechanism to send my health and metrics directly to Elastic Search but this would be a solution specific implementation. I also do not want errors or checking of the Elastic Search Status in my application to run. This is where LogStash comes in. All I need to implement in my application is writing of the log files. Spring Boot comes with slf4j built in (http://docs.spring.io/spring-boot/docs/current/reference/html/howto-logging.html). Now all that I need is to send my health and metrics to different log files for the LogStash to use.

Log Files

First step to create the log files is to use logback as our log implementation. For that you need to add to your resources folder the logback.xml file (http://docs.spring.io/spring-boot/docs/current-SNAPSHOT/reference/htmlsingle/#boot-features-logback-extensions). Here you can define the rolling of the log file and the format.

To get the log files into Elastic Search you have two options. The first way is to configure the logstash to use parse the log files of spring boot using grok (https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html).

Since we know that we will be using LogStash it is better to have the log files formatted from the begging in json format for LogStash. To do this we will use the LogStash encoder for logback. For example:

<appender name="LogStashHealthFile" class="ch.qos.logback.core.rolling.RollingFileAppender">
    <encoder class="net.logstash.logback.encoder.LoggingEventCompositeJsonEncoder">
        <providers>
            <timestamp/>
            <version/>
            <threadName/>
            <loggerName/>
            <logLevel/>
            <logLevelValue/>
            <context/>
            <arguments/>
            <message/>
        </providers>
    </encoder>
    <File>../log/log-stash-health.log</File>
    <rollingPolicy class="ch.qos.logback.core.rolling.TimeBasedRollingPolicy">
        
        <fileNamePattern>../log/archive/log-stash-health.%d{yyyy-MM-dd}.%i.log.gz</fileNamePattern>
        <maxHistory>14</maxHistory>
        <timeBasedFileNamingAndTriggeringPolicy class="ch.qos.logback.core.rolling.SizeAndTimeBasedFNATP">
            
            <maxFileSize>5MB</maxFileSize>
        </timeBasedFileNamingAndTriggeringPolicy>
    </rollingPolicy>
</appender>

To use the encoder you need to add the following maven dependency:

<dependency>
    <groupId>net.logstash.logback</groupId>
    <artifactId>logstash-logback-encoder</artifactId>
    <version>4.5.1</version>
</dependency>

This will create our log file as a json format for LogStash.

The Message Issue

The issue that I had a hard time with, and is the main reason for this blog is the json format of the message. The logback encoder creates a json out of the log entry, where the message is a field in the json. This field is recorded as a text field in the json object and not as part of the full json object.

For example:

{

"@timestamp": "2015-11-09T10:11:19.026+02:00",

"@version": 1,

"logger_name": "com.clearforest.importer.log.health",

"level": "INFO",

"message": "{\"status\":\"UP\",\"queue\":{\"status\":\"UP\",\"Queue Name\":\"cmwell.files.ucpa\",\"Data Directory\":\"..\\\\data\\\\ActiveMQ\"},\"CMWell\":{\"status\":\"UP\",\"host\":\"vgilad:9000\"},\"nrg\":{\"status\":\"UP\",\"lastDataReceived\":\"09.11.2015 10:11:18\",\"lastWriteDate\":\"unknown\",\"cachedMessages\":1},\"diskSpace\":{\"status\":\"UP\",\"free\":24298827776,\"threshold\":10485760}}"

}

For Kibana to work we need the message of the log to be part of the full json object. To fix this when using the log in our code we need to add the following import:

import net.logstash.logback.argument.StructuredArguments._

This adds the option to pass arguments to the log file interface. The arguments are then added as a json object and not as a string field. In the code you then use the log as follows:

val res = RequestFactory.getRestContent(s"http://localhost:$serverPort/metrics")
meticLogger.info("",raw("metric",res))

and in the log file we will get

{

"@timestamp": "2015-11-09T08:59:57.480+02:00",

"@version": 1,

"level": "INFO",

"level_value": 20000,

"health": {

"status": "DOWN",

"queue": {

"status": "UP",

"Queue Name": "files.abc",

"Data Directory": "..\\data\\ActiveMQ"

"MyApp": {

"status": "DOWN",

"error": "Connection refused: connect",

"host": "abc:9000"

"diskSpace": {

"status": "UP",

"free": 24000569344,

"threshold": 10485760

}

Friday, September 18, 2015

Spring Scopes – Decoupling code

The Scenario

I have an application that has a lot of classes – more than 10. To simplify the issue, let’s say that I have a system that represents a school. So I will have a school class, a room class, teacher and all the rest, for example:

class School(){

val classes : List[Class]

def gotMoney(total: int)

}

class Class(){

val teachers: List[Teacher]

def updateTeacher(raise: int)

}

class Teacher(){

def updateTeacherSalary(raise: int)

}

So we have in place - a school that has a list of classes that each has a list of teachers. The normal flow is that the school gets money and wants to give it to each teacher. So we have a method to pass the information from the School to the Class and then to each teacher.

Let’s say that I have a web application that calls the system, and we want to add a profile parameter that is sent during run time so that we can filter which teacher’s get a raise and which don’t.

How do we pass this new parameter from the Web API to the teacher class?

The Solution

Strait forward

A classic scenario is to add a new parameter “profile” to all the methods that need the profile. The limitation of this way is that we might want to add more than one parameter. In addition this is a legacy application that has a lot of spaghetti, so this would mean adding the parameter to a lot of methods that don’t really need to know about this.

What we really need is a way for a bunch of parameters to be entered in the Web API level and then be available to other places of code.

Scope Solution

Assumptions

The current assumption is that we have spring in place and that most of the service classes are spring beans.

So the actual solution is to create a session bean with scope prototype so that we have a new bean each time that we need it. The bean is created on the Web API level and populated with the values, then in the lower level API the services that needs the new parameters will request the same prototype bean and get the values.

By using the prototype bean we also insure that are application can support multithreading and keep multiple instances of each session.

Standard web solution

If each bean will inject a standard prototype bean then each inject will get a new bean and the values will not persist. What we want is create only one bean per http request. For this spring invented a bean with scope of HttpRequest (http://www.tutorialspoint.com/spring/spring_bean_scopes.html). So what happens is that each time you request this bean spring will check if for this specific http request a bean has already been created. If yes you get the reference and if no you will get a new one.

For this to work your application must be a web-aware Spring Application Context. This is so that spring will hook into the http requests and know when to create a new instance.

But what if you want to use this same mechanism but you don’t have a web-aware context? If you did not have spring, the obvious solution would be to store the data you need on the local thread and then you can have access to it from any other class.

So spring actually supports another type of scope that is not in the standard documentation – thread.

Thread Scope

To definite your bean, you will use the standard definition of:

Spring has a class that implements the thread scope which is: SimpleThreadScope. It is a simple implementation with the limitation that it does not support a callback for the destruction of the bean, since it does not know when the thread is finished.

Since this is an unreleased class it is not supported by default and to use it you must register it with spring like any other custom spring bean:

<bean class="org.springframework.beans.factory.config.CustomScopeConfigurer">

    <property name="scopes">

        <map>

            <entry key="thread">

                <bean class="org.springframework.context.support.SimpleThreadScope"/>

            </entry>

        </map>

    </property>

</bean>

Summary

When you need to change a lot of code due to API changes, you need to think out of the box and find a solution to magically pass the parameters from one section of the code to another. The solution to this is to isolate the parameters in a spring bean using the proper scope of the bean. You need to read up on spring scopes to find the specific scope that you need. Of course you always have the option to implement your own scope

Tuesday, May 12, 2015

Securing Tomcat

Tomcat is a great web server. Though for production there is room for improvement.

There are some basic things that need to be done to make your tomcat secure. Some of them are simple like changing the default shutdown port to a different one. Others are simple but can be easily overlooked, like removing the default demo application and removing the management interface. For a list see: http://www.tomcatexpert.com/blog/2011/11/02/best-practices-securing-apache-tomcat-7.

Tomcat Users

If you want to use the tomcat manager application but what to use users, tomcat supports defining roles and users in the tomcat-users.xml file (see https://tomcat.apache.org/tomcat-7.0-doc/manager-howto.html). An example of this file is:

<tomcat-users>

</tomcat-users>

For the most part this is good enough, but what if you do not want your password in clear text? Tomcat supports the option to use MD5 instead of the clear text. To do this in the server.xml file, in the Realm node, you need to add the following:

JMX Login

Most applications need a JMX MBean for debugging and monitoring. The JVM supports user name and password in the following way.

To your application you add the following JVM args:

-Dcom.sun.management.jmxremote

-Dcom.sun.management.jmxremote.port=8007

-Dcom.sun.management.jmxremote.local.only=false

-Dcom.sun.management.jmxremote.authenticate=true

-Dcom.sun.management.jmxremote.ssl=false

-Dcom.sun.management.jmxremote.password.file={file_location}\jmxremote.password

-Dcom.sun.management.jmxremote.access.file={file_location}\jmxremote.access

You will need to change the ACL of the file for the jmxremote.password else the solution will not work (http://docs.oracle.com/javase/7/docs/technotes/guides/management/agent.html).

The problem is that the file with the password is still open and not encrypted. To fix this we need to implement our own Login Module using JASS (have a look at https://blogs.oracle.com/lmalventosa/entry/jmx_authentication_authorization).

The implement our own login module we need to change the JVM args to:

-Dcom.sun.management.jmxremote

-Dcom.sun.management.jmxremote.port=8007

-Dcom.sun.management.jmxremote.local.only=false

-Dcom.sun.management.jmxremote.ssl=false

-Djava.ext.dirs={dir_location}

-Dcom.sun.management.jmxremote.login.config=AMFConfig

-Djava.security.auth.login.config={file_location}\AMF.config

-Dcom.sun.management.jmxremote.access.file={file_location}\jmxremote.access

For the login to work, the user still needs to be in the jmxremote.access file with the proper access level (readonly, readwrite).

We have added two new entries:

java.security.auth.login.config – this entirety will tell the JVM to load the JASS configuration file, for example:

TestConfig {

plugin.com.tomcat.login.MyLoginModule Requisite passwordFile="\user\abc\jmxremote.access";

};

This will tell the JVM to use the class LoginModule for the JASS login. Since the configuration file can have more than one definition, you need to specific which module to use with: com.sun.management.jmxremote.login.config.

There are many tutorials for this (http://docs.oracle.com/javase/7/docs/technotes/guides/management/agent.html, https://blogs.oracle.com/lmalventosa/entry/jmx_authentication_authorization).

What they don’t write is that you need to create a separate jar that includes your login module (plugin.com.tomcat.login.LoginModule). Once you have this jar you need to put it in the java external folder. Adding it to the classpath is not enough. No we usually don’t have access to the java external folder, so since java 6 you can add another path for the external dir: java.ext.dirs. This will load all default external jars and your login.

An example code for the login module is:

public class MyLoginModule implements LoginModule {

private Subject subject;

private CallbackHandler callbackHandler;

private Map<String, ?> options;

private JMXPrincipal user;

private boolean succeeded = false;

public MyLoginModule() {

}

@Override

public void initialize(Subject subject, CallbackHandler callbackHandler, Map<String, ?> sharedState, Map<String, ?> options) {

this.subject = subject;

this.callbackHandler = callbackHandler;

this.options = options;

succeeded = false;

}

//function for map

private Map<String,String> getUserPassword(String passwordFile){

Map<String,String> map = new HashMap<String, String>();

File file = new File(passwordFile);

if (file.exists()) {

try (BufferedReader buffer = new BufferedReader(new InputStreamReader(new FileInputStream(file)))) {

while (true) {

String line = buffer.readLine();

if (line == null)

break;

String[] fields = line.split("\\s+");

String fileUser = fields[0];

String filePassword = fields[1];

map.put(fileUser, filePassword);

}

} catch (IOException e) {

return map;

}

else{

System.out.println("password file not found: " + passwordFile);

}

return map;

}

public boolean validateUser(String passwordFile, String userName, String password) {

String md5 = EncryptionHelper.md5(password);

Map<String, String> userPasswords = getUserPassword(passwordFile);

String filePassword = userPasswords.get(userName);

if (md5.equals(filePassword)) {

return true;

}

return false;

}

@Override

public boolean login() throws LoginException {

String passwordFile = (String) options.get("passwordFile");

if (callbackHandler == null) {

throw new LoginException("Oops, callbackHandler is null");

}

Callback[] callbacks = new Callback[2];

callbacks[0] = new NameCallback("name:");

callbacks[1] = new PasswordCallback("password:", false);

try {

callbackHandler.handle(callbacks);

} catch (IOException e) {

throw new LoginException("Oops, IOException calling handle on callbackHandler");

} catch (UnsupportedCallbackException e) {

throw new LoginException("Oops, UnsupportedCallbackException calling handle on callbackHandler");

}

NameCallback nameCallback = (NameCallback) callbacks[0];

PasswordCallback passwordCallback = (PasswordCallback) callbacks[1];

String name = nameCallback.getName();

String password = new String(passwordCallback.getPassword());

user = new JMXPrincipal(name);

if (validateUser(passwordFile, name, password)) {

succeeded = true;

return succeeded;

} else {

succeeded = false;

throw new FailedLoginException("Wrong Login! Incorrect User or Password");

}

@Override

public boolean commit() throws LoginException {

subject.getPrincipals().add(user);

return succeeded;

}

@Override

public boolean logout() throws LoginException {

subject.getPrincipals().remove(user);

return false;

}

@Override

public boolean abort() throws LoginException {

return false;

}

Debugging

To debug the login you have a few options. The regular attach process will work nicely and you can debug your login. In addition on the server side you can add the following system parameter: -Djava.security.debug=all.

Also on the client side you can run the jconsole from the command line with the following parameter:

Jconsole –debug. This will open a debug window and print the errors from the server side.