java

22 posts

Java - Sanitize file and folder name

A quick way to sanitize file name in Java:

name.replaceAll("\\\\|/|\\||:|\\?|\\*|\"|<|>|\\p{Cntrl}", "_")  

And also for folder name:

name.replaceAll("\\.|\\\\|/|\\||:|\\?|\\*|\"|<|>|\\p{Cntrl}", "_")  

Taken from here.

Spring Async Task Executor with Event Bus

I have a operation which talks to database, so it may be slow. So I was looking for a way to make it asynchronous, then I discovered Spring 4 has async task executor and ListenableFuture and it works well with current Google Guava EventBus.

We created a new AsyncListenableTaskExecutor first. SimpleAsyncTaskExecutor does not reuse any threads, rather it starts up a new thread for each invocation. But it's good enough.

private final AsyncListenableTaskExecutor taskExecutor = new SimpleAsyncTaskExecutor("my task");  

Then we submit a task to the executor.

this.taskExecutor.submitListenable(() -> {  
    saveToDB();
    return null;
}).addCallback(
    (result) -> this.eventBus.post(new SaveOKEvent()),
    (ex)     -> this.eventBus.post(new SaveFailedEvent(ex))
);

Then we adds listeners to both SaveOKEvent and SaveFailedEvent to handle the success and failure case. Done!

Elasticsearch - Delete documents by type

If you want to delete documents in Elasticsearch by type using Java API, below are some options:

  • For Elasticsearch 1.x, use the deprecated prepareDeleteByQuery method of Client. 2.x has removed this method.
  • For Elasticsearch 2.x, use delete-by-query plugin.

Or use scroll/scan API as below.

SearchResponse scrollResponse = this.client.prepareSearch(INDEX_NAME)  
        .setTypes(type)
        .setSearchType(SearchType.SCAN)
        .setScroll(new TimeValue(60000))
        .setQuery(QueryBuilders.matchAllQuery())
        .setSize(100)
        .get();
final BulkRequestBuilder bulkRequestBuilder = this.client.prepareBulk().setRefresh(true);  
while (true) {  
    if (scrollResponse.getHits().getHits().length == 0) {
        break;
    }

    scrollResponse.getHits().forEach(hit -> bulkRequestBuilder.add(
        this.client.prepareDelete(INDEX_NAME, type, hit.getId()))
    );
    scrollResponse = this.client.prepareSearchScroll(scrollResponse.getScrollId())
            .setScroll(new TimeValue(60000))
            .get();
}
if (bulkRequestBuilder.numberOfActions() > 0) {  
    bulkRequestBuilder.get();
}

Properties ordering of Groovy JsonSlurper parsing

Groovy JsonSlurper is a useful tool to parse JSON strings. For a JSON object, the parsing result is a Map object. In certain cases, we want to keep the iteration order of Map properties same as encounter order in original JSON string.

By default, JsonSlurper uses a TreeMap, so the properties are actually sorted. Given following program, the result will be obj => {a=0, x=2, z=1}.

public class Test {  
    public static void main(String[] args) {
        String jsonString = "{\"obj\": {\"a\": 0, \"z\": 1, \"x\": 2}}";
        JsonSlurper jsonSlurper = new JsonSlurper();
        Map map = (Map) jsonSlurper.parseText(jsonString);
        map.forEach((k, v) -> System.out.println(String.format("%s => %s", k, v)));
    }
}

To keep the original properties ordering, you can add -Djdk.map.althashing.threshold=512 as the JVM argument, then the output will be obj => {a=0, z=1, x=2}.

The reason is in the source code of groovy.json.internal.LazyMap used by JsonSlurper. See in GitHub. So If jdk.map.althashing.threshold system property is set, LazyMap will use a LinkedHashMap implementation instead of TreeMap, then it will keep the properties ordering.

private static final String JDK_MAP_ALTHASHING_SYSPROP = System.getProperty("jdk.map.althashing.threshold");

private void buildIfNeeded() {  
   if (map == null) {
        /** added to avoid hash collision attack. */
        if (Sys.is1_7OrLater() && JDK_MAP_ALTHASHING_SYSPROP != null) {
            map = new LinkedHashMap<String, Object>(size, 0.01f);
        } else {
            map = new TreeMap<String, Object>();
        }

        for (int index = 0; index < size; index++) {
            map.put(keys[index], values[index]);
        }
        this.keys = null;
        this.values = null;
    }
}

Please note this solution should be used as a hack as it depends on Groovy's internal implementation details. So this behavior may change in future version of Groovy.

Note for Java 8

jdk.map.althashing.threshold system property is removed in Java SE 8, but this hack still works in Java 8 as the implementation only checks the existence of this system property, but not actually uses it.

New Book - A Practical Guide for Java 8 Lambdas and Streams

This book is not the first book about Java 8 lambda expressions and streams, and it's definitely not the last book about lambda expressions and streams. Java 8 is a Java platform upgrade which the community looking forward to for a long time. Lambda expressions and streams quickly gain popularity in Java developers. There are already a lot of books and online tutorials about lambda expressions and streams. This book is trying to explain lambda expressions and streams from a different perspective.

  • For lambda expressions, this book explains in details based on JSR 335.
  • For streams, this book covers fundamental concepts of Java core library.
  • This book provides how-to examples for lambda expressions and streams.
  • This book also covers the important utility class Optional.

Lambda expressions and streams are easy to understand and use. This book tries to provide some insights about how to use them efficiently.

Buy this book

Tips for using ProGuard with Spring framework

ProGuard is a is a free Java class file shrinker, optimizer, obfuscator, and preverifier. You may want to use ProGuard to obfuscate your Java binary code first before you release it to customers, especially for Android apps, on-premise enterprise apps or libraries. The whole obfuscation process is very painful and you need to run a lot of tests to make sure your code still works properly after obfuscation.

Here are some tips to use ProGuard, especially when Spring framework is used.

Use the Maven plugin

If you use Maven to manage your project, then you should use the Maven plugin for ProGuard. It's easy to set up and use.

<plugin>  
    <groupId>com.github.wvengen</groupId>
    <artifactId>proguard-maven-plugin</artifactId>
    <version>2.0.10</version>
    <executions>
        <execution>
            <id>proguard</id>
            <phase>package</phase>
            <goals>
                <goal>proguard</goal>
            </goals>
        </execution>
    </executions>
    <configuration>
        <obfuscate>true</obfuscate>
        <injar>${shaded-jar.name}</injar>
        <outjar>${shaded-jar.name}</outjar>
        <libs>
            <lib>${java.bootstrap.classes}</lib>
            <lib>${java.cryptographic.extension.classes}</lib>
            <lib>${java.secure.socket.extension.classes}</lib>
        </libs>
        <injarNotExistsSkip>true</injarNotExistsSkip>
        <options>
        </options>
    </configuration>
    <dependencies>
        <dependency>
            <groupId>net.sf.proguard</groupId>
            <artifactId>proguard-base</artifactId>
            <version>5.2.1</version>
            <scope>runtime</scope>
        </dependency>
    </dependencies>
</plugin>  

In <options> of <configuration>, there should be a list of <option> to configure ProGuard.

Multi-modules project

If your Maven projects have multiple modules, then you should use Maven shade plugin to create a shaded jar for all your modules, then run ProGuard against this single jar. This can make sure ProGuard has the correct mappings for all your application's classes.

<plugin>  
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-shade-plugin</artifactId>
    <version>2.4</version>
    <configuration>
        <outputFile>${shaded-jar.name}</outputFile>
        <artifactSet>
            <includes>
                <include>com.myapp:*</include>
            </includes>
        </artifactSet>
        <transformers>
            <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                <resource>META-INF/spring.factories</resource>
            </transformer>
            <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                <resource>META-INF/spring.handlers</resource>
            </transformer>
            <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                <resource>META-INF/spring.schemas</resource>
            </transformer>
            <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer">
                <resource>META-INF/spring.provides</resource>
            </transformer>
        </transformers>
    </configuration>
    <executions>
        <execution>
            <phase>package</phase>
            <goals>
                <goal>shade</goal>
            </goals>
        </execution>
    </executions>
</plugin>  

If you use Spring, make sure transformers are added to process Spring's various files.

Disable optimization class/marking/final

By default ProGuard marks classes as final when possible even when classes are not declared as final. This causes problems for Spring as Spring doesn't allow classes with @Configuration annotation to be final.Use following <option> to disable it.

<option>-optimizations !class/marking/final</option>  

Adapt Spring resources

If you use configuration files like spring.factories to customise Spring, make sure these configuration files are adapted by ProGuard, otherwise the class names in those files will be wrong. META-INF/spring.* in following code specifies Spring configuration files.

<option>-adaptresourcefilecontents **.properties,META-INF/MANIFEST.MF,META-INF/spring.*</option>  

Keep annotations

Spring uses annotations extensively, so annotations should be kept in the runtime to make sure Spring still works properly. *Annotation* in code below is used to keep annotations.

<option>-keepattributes Exceptions,InnerClasses,Signature,Deprecated,SourceFile,LineNumberTable,*Annotation*,EnclosingMethod</option>  

Keep application launch class

If you use Spring Boot, the Application class should be kept to launch the app. The option in code below keeps any class with main method.

<option>-keepclasseswithmembers public class * { public static void main(java.lang.String[]);}</option>  

Keep your REST resources classes

If your app exposes a REST API, make sure those resources classes are kept. Most likely you rely on Jackson or other libraries to convert your resources objects to JSON or XML. These libraries use reflection to find out the properties in your resources classes, so these resources classes should be kept to make sure the JSON/XML representations are correct.

For example, given a resource class User,

public class User {  
    private String firstName;
    private String lastName;

    public String getFirstName() {
        return this.firstName;
    }

    public String getLastName() {
        return this.lastName;
    }
}

After ProGuard processed this class file, the methods getFirstName and getLastName may be renamed to something like a or b. Then Jackson cannot use reflection to find JavaBean properties in this class file. The output will be just an empty JSON object.

<option>-keep public class com.myapp.**.model.** { *; }</option>  

Process bean classes

You can also following examples in ProGuard website to process bean classes by keeping setter and getter methods.

<option>  
-keep class com.myapp.**.model.** {
    void set*(***);
    boolean is*();
    *** get*();
}
</option>  

Add name to Spring beans

If Spring annotations @Service, @Component and @Configuration are used to declare beans, make sure a name is assigned to each bean, e.g. @Component("userHelper") or @Service("userService"). This is because when no name is assigned, Spring uses the class's name as the bean name, but ProGuard will change class names to something like a, b, or c. This will have name conflicts across different packages. For example, package com.myapp.a.a has a class a, package com.myapp.a.b also has a class a, these two class use the same bean name a, but the type is different. So beans should be explicitly named to avoid name conflicts.

Keep members with Spring annotations

If you use Spring annotations like @Value to inject values into your classes like below:

@Value("${myval}")
private String myVal;  

ProGuard is smart enough to infer that the value of myVal is null as this variable has not been assigned any value, so it replaces all occurrences of myVal with null in the binary code, then a lot of NullPointerExceptions will be thrown at runtime. To avoid this, use following options:

<option>-keepclassmembers class * {  
    @org.springframework.beans.factory.annotation.Autowired *;
    @org.springframework.beans.factory.annotation.Value *;
}
</option>  

Build Apache Camel Custom Component

If you create a custom Apache Camel component, you can build it using Maven to generate necessary metadata, then this component can be auto-discovered by Camel.

Create a custom component following the guide. Add file META-INF/services/org/apache/camel/component/FOO to src/main/resources folder with content like below:

class=com.example.CustomComponent  

Then add following code to Maven's pom.xml. Maven plugin camel-package-maven-plugin is used to generate component.properties file.

<build>  
    <plugins>
        <plugin>
            <groupId>org.apache.camel</groupId>
            <artifactId>camel-package-maven-plugin</artifactId>
            <version>${camel.version}</version>
            <executions>
                <execution>
                    <goals>
                        <goal>prepare-components</goal>
                    </goals>
                    <phase>generate-resources</phase>
                </execution>
            </executions>
        </plugin>
    </plugins>
</build>  

Then add Maven dependency of org.apache.camel:apt. This apt module processes Camel annotations and generate component JSON schema and HTML documentation. See Camel 2.15 release note.

<dependency>  
    <groupId>org.apache.camel</groupId>
    <artifactId>apt</artifactId>
    <version>${camel.version}</version>
    <scope>provided</scope>
</dependency>  

After this, you should be able to list your component and its JSON schema from JMX.

FileNotFoundException when using Jersey with Spring Boot

When using Jersey with Spring Boot, you may see an FileNotFoundException error like below when Spring Boot starts:

org.springframework.beans.BeanInstantiationException: Failed to instantiate [myapp.JerseyConfig$$EnhancerBySpringCGLIB$$380920c5]:  
Constructor threw exception; nested exception is org.glassfish.jersey.server.internal.scanning.ResourceFinderException:  
java.io.FileNotFoundException: /Users/myapp.jar!/lib/myapp-ws.jar (No such file or directory)  

This error only happens using java -jar to run executable Spring Boot jar file. This is because Spring Boot packages library jars in the lib directory of the executable stand-alone jar. Jersey's class loader cannot scan those jars embedded in the executable jar. In the above error message, Jersey was trying to find myapp-ws.jar on the local file system, but cannot find it because it's embedded in the lib directory of myapp.jar.

To solve this issue, library jars which contain Jersey related classes need to be unpacked first before Spring Boot runs. This is supported by Spring Boot' Maven plugin using requiresUnpack, see doc.

Below is an example of the Maven pom.xml file.

<build>  
    <plugins>
        <plugin>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-maven-plugin</artifactId>
            <configuration>
                <requiresUnpack>
                    <dependency>
                        <groupId>com.myapp</groupId>
                        <artifactId>rest-api</artifactId>
                    </dependency>
                </requiresUnpack>
            </configuration>
        </plugin>
    </plugins>
</build>  

c3p0 for Java 6

If you are using c3p0 for db connection pooling and also using Java 6, be sure to use version c3p0-0.9.5-pre8, no the latest version c3p0-0.9.5-pre9. Starting from version c3p0-0.9.5-pre9, c3p0 interface com.mchange.v2.c3p0.PooledDataSource extends from Java 7's java.lang.AutoClosable, so c3p0 c3p0-0.9.5-pre9 cannot run on Java 6. See source code 0.9.5-pre9 and 0.9.5-pre8 for the difference.

Common JPA/Hibernate Issues - javax.persistence.EntityExistsException and org.hibernate.LazyInitializationException

When using JPA/Hibernate, there are some common errors.

javax.persistence.EntityExistsException: A different object with the same identifier value was already associated with the session

This means when Hibernate is trying to save entities when committing the transaction, two new entities have been added but they are related to the same database table row. You should check the Cascading settings of your entities. You may declare CascadeType.PERSIST or CascadeType.ALL on one of your entities, but still trying to persist dependent entities explictly using save. This will cause two entities to be persisted: One from cascading and another one from explict persisting operation. You can remove the explict persisting operation and let Hibernate to manage dependent entities.

Below is a simple example.

class User {  
  @OneToMany(cascade=CascadeType.ALL, mappedBy="user")
  private Set<Role> roles; 
}

user.addRole(role); //Role entity will be saved by cascading  
roleDao.save(role); //This causes javax.persistence.EntityExistsException  

org.hibernate.LazyInitializationException: failed to lazily initialize a collection of role

This is a very common issue when using Hibernate managed objects in web layer. If you google it, you can find hundreds of pages about this issue. Hibernate official document is a good start point to understand this issue.

When an entity object has a lazily-loaded collection and this object is passed to the web layer. When web layer code trying to access entity's collection object, objects in the collection need to be loaded. But the Hibernate session has already been closed, then Hibernate throws org.hibernate.LazyInitializationException. After understanding how this issue may happen, it's easy to find out how to solve it.

Load eagerly

Instead of loading collections lazily, you can load them eagerly. This means collections of this entity object have already been populated before used. To do this, use FetchType.EAGER.

class User {  
  @OneToMany(fetch=FetchType.EAGER, cascade=CascadeType.ALL, mappedBy="user")
  private Set<Role> roles; 
}

Loading eagerly may have performance issues because it may load unnecessary objects into memory. So check usage of the entity object first before applying this. In the above example, it's common that Role objects are also accessed when using User objects, so it makes sense to load Role entities eagerly.

Keep Hibernate session open

Another solution is to keep the Hibernate session open in web layer code, then when web layer code trying to load collection objects, the session is still open to do the loading. This is usually referred as Open Session In View pattern. This pattern has been widely used for a long time, Spring framework has support for this pattern. But this pattern is not considered as a good practice. It pollutes the web layer code with Hibernate exceptions and may have performance issues.

Load on-demand

If keeping Hibernate session open is not a good option and you want to fully populate an entity object, you can initialize it manually. Hibernate provides Hibernate.initialize method to initialize lazy-loading collections.

For the code below, User entity's roles collection is lazily loaded. Use Hibernate.initialize(user) to load roles collection.

class User {  
  @OneToMany(cascade=CascadeType.ALL, mappedBy="user")
  private Set<Role> roles; 
}

Sometimes an entity object has many lazy-loading collections and entities in those collections have other lazy-loading collections. To fully populate an object graph, you may need to use Hibernate.initialize multiple times. Below is a utility class to fully populate an object graph. Use HibernateUtils.recursiveInitialize to fully populate an object graph. com.myapp.model is the package name of model classes, which is used to avoid checking irrelevant classes.

import java.beans.PropertyDescriptor;  
import java.util.Collection;  
import java.util.HashSet;

import org.apache.commons.beanutils.PropertyUtils;  
import org.hibernate.Hibernate;  
import org.slf4j.Logger;  
import org.slf4j.LoggerFactory;

import com.google.common.collect.Sets;

public class HibernateUtils {  
  private HibernateUtils() {

  }

  private static final Logger logger = LoggerFactory
      .getLogger(HibernateUtils.class);

  public static void recursiveInitialize(Object obj) {
    new RecursiveInitializer().initialize(obj);
  }

  private static class RecursiveInitializer {
    private HashSet<Object> checkedObjects = Sets.newHashSet();

    public void initialize(Object obj) {
      if (obj == null || obj.getClass().getCanonicalName() == null) {
        return;
      }
      if (checkedObjects.contains(obj)
          || !obj.getClass().getCanonicalName()
              .contains("com.myapp.model")) {
        return;
      }
      checkedObjects.add(obj);
      if (!Hibernate.isInitialized(obj)) {
        Hibernate.initialize(obj);
      }
      PropertyDescriptor[] properties = PropertyUtils
          .getPropertyDescriptors(obj);
      for (PropertyDescriptor propertyDescriptor : properties) {
        try {
          Object propertyObj = PropertyUtils.getProperty(obj,
              propertyDescriptor.getName());
          if (propertyObj != null) {
            initialize(propertyObj);
          }
          if (propertyObj instanceof Collection<?>) {
            for (Object item : (Collection<?>) propertyObj) {
              initialize(item);
            }
          }
        } catch (Exception e) {
          logger.warn("Ignore property : " + propertyDescriptor, e);
        }
      }
    }
  }
}

Use DTO

Use DTO - Data transfer object is another solution. DTOs are simple objects with no relations to Hibernate managed entities. DTOs are easy to use in web layer.

Code below is an example of using DTOs. UserDto is the DTO object of User entity object. It doesn't have any Hibernate annotations and refer to another DTO class RoleDto.

class UserDto {  
  private Set<RoleDto> roles;
}