Triaging High CPU Utilization in Liferay

XTIVIA has been working with Liferay platform for over 10 years; over this period, we have had ample opportunity to build our understanding of how Liferay interacts with the Java Virtual Machine (JVM), and how to triage performance issues in a Liferay JVM. This post details the systematic approach we took to triage a recent performance issue in a client environment where the Liferay application was showing high CPU utilization.

The initial steps

Whenever there is a performance issue with Liferay, or another Java application, the first step would be to determine if there is an obvious configuration error that is causing these issues in the first place. The various items to verify in a Liferay application are:

1. Liferay logs. In many cases, if there are issues in an application, the first step to check would be the logs printed out by the application. Logs provide a very deep insight into application issues.
2. The portal properties file.
3. Check for server configuration parameters to verify if the VM allows enough processes, and open files to be opened by the user the Java process is running as.
4. The other areas would be to check if the environment has been sized properly to accommodate user requests.

JVM sizing

Normally a JVM that has not been sized properly demonstrates severe performance problems. To verify if a JVM has been sized properly, it is a good practice to enable verbose garbage collection logging. A JVM garbage collection analysis would be a good place to start to see if the JVM is displaying any patterns of high pause times, memory leak etc.

Memory leak

If an application shows memory leaks, then a good place to start would be by collecting heap dumps for the application. A heap dump is a snapshot of the memory of a Java process.The snapshot contains information about the Java objects and classes in the heap at the moment the snapshot is triggered. Because there are different formats for persisting this data, there might be some differences in the information provided. Typically, a full garbage collection is triggered before the heap dump is written, so the dump contains information about the remaining objects in the heap. Heap dump analysis provides insight into the objects, methods or classes that could be causing a memory leak.

The snapshot contains information about the Java objects and classes in the heap at the moment the snapshot is triggered. Because there are different formats for persisting this data, there might be some differences in the information provided. Typically, a full garbage collection is triggered before the heap dump is written, so the dump contains information about the remaining objects in the heap. Heap dump analysis provides insight into the objects, methods or classes that could be causing a memory leak.

Thread dumps

If a memory leak is not the reason the environment is using high CPU, then the next step would be to determine what threads are causing the high CPU utilization in the servers.

Thread IDs

The first step would be to determine the threads that are using the most CPU int the server. The following threads are the ones causing high CPU in the servers. We obtained this information from the ‘top -H’ command in linux.

63508 tcserver 20 0 9825016 7.071g 11260 R 79.6 66.8 215:21.84 java
63599 tcserver 20 0 9825016 7.071g 11260 R 71.4 66.8 215:22.97 java
42639 tcserver 20 0 9825016 7.071g 11260 R 66.8 66.8 35:08.04 java
63448 tcserver 20 0 9825016 7.071g 11260 R 64.8 66.8 215:15.86 java

Taking thread dumps

It is recommended that jstack application (bundled with HotSpot JDK) be used to take thread dumps. Please note that it is recommended that the thread dumps be taken as the user that the process is running as, and that the dumps redirected to a separate file to reduce the noise in the logs. Also, if the CPU is pegged high for a very long time, 10 thread dumps taken at 30-second intervals should have enough information.

Please note that the thread ids are in hexadecimal in the thread dumps. For example, the thread IDs listed in human-readable format are displayed as their equivalent hex codes in the thread dump file.

F814

F86F

A68F

F7D8

Of these, the following three threads are the ones suspected of causing high CPU because their state doesn’t change through the dumps. These looping threads are the main targets for investigation:

RuntimePageImpl-24

RuntimePageImpl-33

RuntimePageImpl-38

Also, from the thread dump analysis the following code in particular was the cause of the problem:

java.util.HashMap.containsValue(HashMap.java:753)

And the code displayed above was being called in:

org.apache.jsp.applicationdisplay.html.view_jsp._jspService(view_jsp.java:331)

Digging further, we determined that this method was being called in a custom jsp in the version control system. A member variable was directly defined in the following jsp, and is not a considered a best practice:

/src/main/webapp/applicationdisplay/html/init.jsp

LinkedHashMap<String, String> displayMap = new LinkedHashMap<String, String>();
for(int i = 0; i < itemSize; i++){
if("#".equals(urls[i].trim()))

This member variable was getting reconstructed every time it was called. Due to the number of pages this method is called in, the processing time is very resource intensive. The high CPU utilization was because of this code, and because of the large amount of data that was being stored in the hash map.

Armed with this information, the development team was able to implement code changes, improving the performance of the application significantly. We have linked to the thread dumps here for your reference.

We hope that this blog post helped you in understanding how to troubleshoot performance issues in Liferay. If you need assistance with your Liferay application performance tuning, do reach out to us at https://www.xtivia.com/contact/ or [email protected].

You can also continue to explore DevOps features for Liferay DXP by checking out The Top 5 DevOps features in Liferay DXP, and Tuning Basic JVM Performance for DXP.