Saturday, December 30, 2017

Gradle Recipe: Building a Fat JAR

So far, I haven't seen a Build Tool that is easy to use. Doesn't matter if it was called Make, Maven or SBT. They're all very complex and far away from easy and intuitive.

Since I'm struggling a lot with Gradle lately, this is the first post that shows how I solved my (common) problem: Building a Fat JAR.

I don't know if I see things differently, but my first expectation on a Build Tool is, that it builds my software with all the configured dependencies. Therefore, when using Gradle, you can choose between three dependency types:

  • compile
  • runtime
  • testCompile
As soon as you add a new dependency, you would probably declare it as a compile dependency, because you need it to be able to compile your code. In case you are adding TestNG or JUnit, you declare it as testCompile, because you only need it for your tests. The runtime dependency extends from the compile dependency. My understanding of runtime would mean, that I need this dependency in order to be able to execute my code (for example on a production environment). The good thing is, that it means exactly this. The bad thing is, that it does not automatically pack all your dependencies together, so that you can ship your executable.

Lets say you use Apache Log4j2:

dependencies {
   runtime 'org.apache.logging.log4j:log4j-core:2.10.0'
}

From a user perspective, I would assume that this is everything I need to do - but its not. If you try to execute the JAR that is being build, you'll get a ClassNotFoundException.

The solution is the following build.gradle file:

apply plugin: 'java'

test {
   useTestNG()
}

repositories {
   mavenCentral()
}

dependencies {
   compile 'org.apache.logging.log4j:log4j-api:2.10.0'
   compile 'org.apache.logging.log4j:log4j-core:2.10.0'
   testCompile group: 'org.testng', name: 'testng', version: '6.13.1'
}

task fatJar(type: Jar) {
   baseName = project.name
   from { configurations.compile.collect { it.isDirectory() ? it : zipTree(it) } }
   with jar
   manifest {
      attributes(
         'Main-Class': 'org.tobster.foo.GradleExample'
      )
   }
}

assemble.finalizedBy fatJar

Since this is a very common problem, you'll find a lot of solutions, but this was the only one that worked for me. The most interesting part for me is, that you have to write a custom Gradle Task that assembles a Fat JAR. Why do I've to do that? Wouldn't it be possible to have some kind of flag, e.g. fatJAR=true. Maybe that would be to easy.

Wednesday, September 13, 2017

Data Visualization with Grafana and Elasticsearch

If you want to store and visualize data, you have a lot of technologies to choose from. Two such technologies are Elasticsearch (ES) and Grafana. Something that I really like about ES, is that its very easy to use and that it needs no complicated setup or configuration. It just works out-of-the-box (production environments may require more sophisticated configuration). Even though Logstash and Kibana are probably the most well known companions for ES, especially when it comes to visualization of logging data, Grafana is also quite good for time series based data and it keeps your technology stack simple. In the following, I will describe how to store data in Elasticsearch 5.6.0 through its HTTP API and visualize it with Grafana 4.4.3.

Elasticsearch

  1. Download ES 5.6.0 from the official site and unpack
  2. bin/elasticsearch
  3. Web-Frontend can be found at http://localhost:9300
Elasticsearch is a document database that stores documents as JSON. Documents can be stored and retrieved by its HTTP API (there are also libraries for the most popular programming languages). Furthermore, ES stores documents into what is called indices and types. An index could be compared to a database in a relational DBMS and a type could be compared with a table.
For example: you could have an index called customers with a type invoices and another type addresses. You can think of types about something that logically belongs to an index (a customer has an address and one or more invoices).

Grafana

  1. Download Grafana from the official site and unpack
  2. bin/grafana-server
  3. Web-Frontend can be found at http://localhost:3000
Configure a Data Source:
  1. Go to http://localhost:3000/datasources and click on "add data source"
  2. Enter a name for the data source and select "Elasticsearch" from the Type drop-down
  3. As long as you didn't changed the default ES configuration, enter "http://localhost:9300" in the Url field.
  4. Access must be set to "Proxy"
  5. Enter the name of your index (for this example, we need no pattern)
  6. Every JSON document you insert into ES must have an ISO date field, e.g. 2017-07-20T14:00:00. This is important, because as mentioned before, Grafana is designed to work with time series based data. Therefore, enter the name of the field that contains the ISO date into the field "time field name" (without a leading @). I called this field "insert_time" (as shown in the screenshot below).
  7. Since we are using ES 5.6.0, select version 5.x from the Version drop-down
Configure a Dashboard:
  1. Go to http://localhost:3000/?orgId=1 and click on "Create your first dashboard"
  2. Click on "Graph" to create a new graph dashboard
  3. Click on "Panel Title", then click on "Edit" (after that, the Graph configuration appears down below)
  4. In the "General" tab, you can give the dashboard a name
  5. In the top-right corner, select a time range (e.g. last 90 days), where you know that there is some data
  6. If you wonder why you don't see anything, go to the "Display" tab and
    • Draw Modes: select "Lines"
    • Stacking & Null value: select "connected" for Null value
  7. In case your JSON documents have a field that contains a numeric value that you want to visualize, you can also use the existing aggregate functions. Therefore, go to the "Metrics" tab and for example select "Average" as Metric and your timestamp/date field for "Group by" and select "Date Histogram" from the drop-down.
That's it! Thank you for reading.