Persisting JMH Results for Regression Detection



JMH is a tool to measure Java code that is executed within  micro-second time scales. You may ask the question why do I care about the performance of code that by definition is executed in a very short period of time? It is important to only optimise code that will have a significant effect on the overall application. Sometimes a small portion of the your application's code is executed many times or your latency requirements of your application is in the micro-second time scale.

Once your application is performing as required it can be difficult to maintain that performance whilst making code changes. An automated collection of micro-benchmarks running on key components of your application is an invaluable tool in maintaining the required performance. As a starting point a history of your JMH results needs to be stored in order to analyse trends in benchmark results.

Influx JMH reporter is an open source tool we have written that parses the JMH JSON results file from a benchmark and publishes it to an InfluxDB host. We are then able to graph and analyse this data over time in order to detect regressions.

At the end of each JMH run execute:

java -jar influx-jmh-reporter-all.jar -u "http://influxhost:8086" -r results.json

This will persist the results in InfluxDB :
> show measurements
com.mycompany.mybenchmarks.PerformanceTestSuite.myBenchmark
> select * from "com.mycompany.mybenchmarks.PerformanceTestSuite.myBenchmark"
githash hostname       jvm     jvmArgs mode    score   scoreError  scoreUnit
------- --------      ---     ------- ----    -----   ----------  ---------
 7hf3k   myserver /bin/java  thrpt 28.4726       2       ops/us 
...


Stored data is monitored to detect a degradation in performance. Automated detection can be challenging as it requires isolating environmental factors in order to improve test measurement consistency.  JMH goes a long way however additional techniques such as reducing scheduling jitter (see Reducing System Jitter by Mark Price) goes much further.







Comments

Post a Comment

Popular posts from this blog

Obtaining the time in Java with nano-second precision

Image
At TransFICC for MiFID compliance (as explained in my previous blog) we need to obtain the time of an event within 100 microsecond accuracy with respect to UTC. To achieve this in Java we first need to be able to obtain the time from the OS with better than millisecond precision. We can then make use of Precision Time Protocol (PTP) to ensure the system clock is synchronised to UTC with sufficient accuracy. Take a look at the excellent posts from Luke Bigum on how to achieve this . As of Java version 8 it is not possible to directly obtain the time with higher than millisecond precision. Java 8 System.currentTimeMillis() will return the time since the Unix epoch in milliseconds. Higher precision time is possible on some underlying systems such as Linux. On Linux the code that the OpenJDK JVM executes is: The full source can be found  here . The code makes the function call gettimeofday(). This returns the time since the Linux epoch in micro seconds however it is rounded to th
Read more

Time, Where does it come from?

Image
This is part of a two part series on time measurement. Part 2 will explain how to obtain accurate time measurement in Java. This first part explains the general concept of time measurement. How do we measure time? With a clock of course. So how do we build a clock? The first thing we need is to observe something that occurs at a constant rate. Humans have built clocks based on observations of changes in their environment that appeared to occur at a constant rate. Sundial Some of these earliest clocks were Sundials. The shadow cast by the Sundial moves based on the rate of rotation of the earth on its axis. This observation of the Earth's rotation is what Greenwich Mean Time was based on. At the Royal Observatory in Greenwich astronomers would observe the sun and other stars in order to calibrate their clocks. In principal on average the sun would appear directly above in the sky at 12h00.00. This is why its called Mean time. The current
Read more