Top 10 reasons why your Enterprise Java Application is slowby KARUN SUBRAMANIAN on JULY 27, 2014Let\u2019s face it. Nobody likes a slow application. It is our job to make sure our applications\u00a0 fulfill the functionality need completely, but at the same time delivers\u00a0good user experience.In\u00a016 years of my experience in technical support primarily in the world of JEE, while I cannot honestly say \u2018I have seen it all\u2019, I can say \u201cI have seen a\u00a0lot\u201c. Here are the top\u00a010 reasons why your Java application may slow.Note that this list is NOT ordered in anyway, meaning reason 1 is not always the most popular reason for the slowness. These are all equally critical issues that can cause your application to crawl.<\/p><\/blockquote>\n
Source: Top 10 reasons why your Enterprise Java Application is slow<\/a><\/em> This is a gem of a blog post. Very instructive and useful. Most of what is says can be considered common knowledge but by being concisely written and with relevant examples it turns something superfluous to useful. Top 10 reasons why your Enterprise Java Application is slowby KARUN SUBRAMANIAN on JULY 27, 2014Let\u2019s face it. … <\/p>\n
\n
\n
\n
\n1. There is a sudden spike in the\u00a0\u2018Load\u2019 that you did not notice<\/strong><\/span>
\nThis is especially true when your application processes requests from external entities (for example, a financial institution processing batch updates from a data provider). Since you cannot really control (for the most part) what others might send, you can suffer a surge in incoming requests and hence your application can slow down.<\/span>
\nHow can\u00a0you monitor?<\/strong><\/span>
\nYou must monitor the \u2018throughput\u2019<\/strong> of the application (specifically, number of requests\/minute). You will most probably need a commercial APM (Application Performance Management)<\/strong> tool to monitor this reliably. If you don\u2019t have an APM solution implemented (why ?), try to come up with some home grown script\/program that can monitor this. For one or my clients<\/strong>, I had to write a program that performed a \u2018row count\u2019 of a Database table that stored the \u2018processed orders\u2019. I had to run this program through a scheduler (cron) periodically and send an alert email if the number of processed records crossed a threshold.<\/span>
\n2. There is a poorly responding backend system<\/strong><\/span>
\nYour application may talk to one or\u00a0more backend systems other than a Database. Perhaps it talks to an external system using a Web Service call<\/strong>. Perhaps it sends a WebSphere MQ message to aremote Queue Manager<\/strong> (that in turn routes the message to an external system to get a response). If any of these external systems respond poorly (or even stop completely), your application will suffer.<\/span>
\nHow can\u00a0you monitor?<\/strong><\/span>
\nYou must monitor the response <\/strong>time of the external systems experienced by your application.<\/strong> A periodic synthetic transaction might help<\/strong> (if feasible). If you log the response times in a log file, you could monitor the particular entry in the log file. But the best bet would be to use an APM solution and instrument the specific method that makes the external call. Tools such as AppDynamics and CA Wily Introscope<\/strong> (among many others) can automatically detect the backends and reveal the response times.<\/span>
\n3. Your database calls are taking for ever<\/strong><\/span>
\nThis is a big one. It is possible that the sql queries (or Stored procedures) that originate from your application run extremely slow at the Database Server. This could be\u00a0due to\u00a0the following reasons:<\/span>
\na. Database Server is running our System resources<\/strong> such as CPU, Memory etc,<\/span>
\nb.\u00a0A Block<\/strong> in the Database<\/span>
\nc.\u00a0Missing index in the database,\u00a0<\/strong><\/span>
\nd. Your query really runs slow (perhaps you are missing a \u2018where\u2019 clause<\/strong>). In this case, work with a DBA to tune the query\/SP.<\/span>
\nHow can you monitor ?<\/strong><\/span>
\nTo some extent, monitoring from the DB side should help i.e the DBAs should have monitoring for things like long running queries, blocks, deadlocks etc. From application\u2019s perspective, you must monitor the JDBC response times experienced by your application<\/strong>. You can either log this information in log file (expensive and not recommended) or use an APM tool. New generation APM tools like Dynatrace<\/strong> (among many others) can reveal slow running JDBC calls including the actual SQL query or stored procedure<\/strong>.<\/span>
\n4. You are running out Database connections<\/strong><\/span>
\nThis is another big one. Typically 50 JDBC connections per JVM should be more than enough. But it greatly depends on the Application. They key is to make sure you are not maxing out of DB connections. When you max out the DB Connection Pool,<\/strong> your requests are going to wait<\/strong> for an available connection which results in painful slow response time. The reason why you are maxing out of the DB connections should be found out. It could be that you are NOT closing the DB connections properly.<\/span>
\nHow\u00a0can you\u00a0monitor?<\/strong><\/span>
\nIf your application server has JMX console<\/strong>, Connection pools can be monitored via the MBeans, but it can be a painful process. You can also write your own Java program (JMX client<\/strong>) to connect and retrieve values from the Mbeans. Easy way is to install a commercial APM tool<\/span>
\n![\"jmx\"](\"https:\/\/i0.wp.com\/karunsubramanian.com\/wp-content\/uploads\/2014\/07\/jmx-300x120.jpg?resize=300%2C120\")
<\/a><\/span>
\nImage from wiki<\/em><\/span>
\n5. You are running out of Threads<\/strong><\/span>
\nWork enters to your application via a Thread. You only have limited number of Threads to process incoming requests (limited by the Thread Pool<\/strong>). When you exhaust all the available threads in the Thread pool, subsequent requests go into \u2018wait\u2019 state, waiting for an available thread \u2013 painful wait time<\/strong>. The reason why you are running of out thread should be investigated. It could be because of a poorly performing backend or a slow DB response time.<\/span>
\nHow can\u00a0you monitor?<\/strong><\/span>
\nIf your application server has JMX console<\/strong>,\u00a0Thread pools can be monitored via the MBeans, but it can be a painful process. You can also write your own Java program (JMX client) to connect and retrieve values from the Mbeans. Easy way is to install a commercial APM tool. Some Application Server consoles (such as IBM Websphere application server<\/strong>) have built in monitoring dashboards as part of the Admin console. But it can be extremely slow and not very flexible.<\/span>
\n6. The Server hosting your application is running slow or running out of resources<\/strong><\/span>
\nYes, even with Cloud and dynamically expandable hardware, the fact is hardware is not unlimited. When the CPU hits 100% in your host Server, your application is going to suffer. With shared hardware in most enterprises, it is possible that some other application is chewing up all the CPU and your application just becomes a victim.<\/span>
\nHow can\u00a0you Monitor?<\/strong><\/span>
\nOperating system level monitoring tools will help. Tools like Nagios<\/strong> can help monitor the infrastructure effectively.<\/span>
\n7. Super excessive logging in your Application<\/strong><\/span>
\nLogging is expensive, in terms of resource usage. While debug logging will help in certain situations, in production environments, keep the logging to a minimum level (perhaps \u2018INFO\u2019 or \u2018WARN\u2019 log4j levels). An additional side effect of excessive logging is Disk drives filling up (which can have its own consequences). One of my clients was logging every single sql statement generated by the application (Hibernate verbose logging) and the application was generating 500 MB log file every day. While the performance impact was not too bad in this case, it created disk space issue until I disabled the verbose logging.<\/span>
\nHow can you monitor ?<\/strong><\/span>
\nPeriodically check your Application log files<\/strong> (JVM system out and system error). If you notice unnecessary noise, get rid of those logging entries. Checkout logstash<\/strong>.<\/span>
\n8. Garbage collection overhead<\/strong><\/span>
\nThis is a critical performance area for any Java application. If you are running of Java Heap, the JVM will initiate GC to keep up with the memory demand. GC tuning is a separate and exhaustive topic by itself. Short story is GC will consume resources and excessive GC overhead will slow down your application. Pay attention to the Minimum (-Xms) and Maximum (-Xmx) Heap size values in the Java command line options.<\/span>
\nHow can you monitor?<\/strong><\/span>
\nEnable verbose GC logging and watch the log file. You enable verbose GC logging with the options -verbose:gc\u00a0-XX:+PrintGCDetails\u00a0XX:+PrintGCTimeStamps.<\/span>
\nOnce the logs are collected, you can either eyeball it or use a tool like \u201cIBM Pattern Modeling and Analysis Tool for Java Garbage Collector\u201d<\/strong><\/span>
\n![\"gc\"](\"https:\/\/i0.wp.com\/karunsubramanian.com\/wp-content\/uploads\/2014\/07\/gc-255x300.png?resize=255%2C300\")
<\/a><\/span>
\nImage from ibm.com<\/em><\/span>
\n9. Third party jar files that you have included in your application is buggy<\/strong><\/span>
\nIt is not uncommon to use third party jar files to use certain functionality for your application. However care must be taken to ensure there are no performance implications. Even if the jar is reliable and proven (such as spring.jar), there may be issues when you upgrade the spring version without analyzing the release notes etc. Perhaps you may be using the API in a certain way that changed with the newer release of spring. I have seen third party jars causing severe memory leak. I have also seen a rules engine (that was integrated into my client\u2019s JEE Application) that ran into \u2018infinite loop\u2019 issue causing the entire application to slow down.<\/span>
\nHow\u00a0can you\u00a0monitor?<\/strong><\/span>
\nThis can be trick to monitor. There is no bullet proof way to monitor just one jar file\/plugin unless your application specifically uses the thrid part jar for certain specific transactions, in\u00a0which case you can use an APM tool to instrument montioring for\u00a0just that particular method. Even then, you won\u2019t know if it is your code or the third part jar that is creating the issue. Through analysis is required to troubleshoot such issue.<\/span>
\n10. Poor Application Architecture<\/strong><\/span>
\nNo matter how much tuning you put in your code, application server and database servers, if the Architecture is flawed, your application will be slow. For example,<\/span>
\na. Even when not required, using\u00a0\u2018persistent\u2019 messages with JMS.<\/span>
\nb.\u00a0Storing more than necessary\u00a0data in a HTTP \u00a0\u2018Session\u2019 increasing its size<\/span>
\nc. Not using (or using misconfigured) caching solution<\/span>
\nApplication architecture is a vast subject and scores of books have been written on this subject. Study carefully.<\/span>
\nHow\u00a0can you\u00a0monitor?<\/strong><\/span>
\nThere is no specific monitoring to be done to monitor design flaws. Design issues will be revealed when you thoroughly analyze the problem and the available monitoring data.<\/span>
\nThere you have it. Application performance tuning starts from the design stage. There are various reasons your application could be slow. You must be diligent to monitor as much relevant metrics you can about your application and act promptly to avoid customer impact. Investing in a good quality APM solution adds heaps<\/em> (no pun intended) of ammunition to your weapons in the battle of performance monitoring.<\/span>
\nGood luck.<\/span><\/p>\n","protected":false},"excerpt":{"rendered":"