Data Visualization with Grafana and Elasticsearch

If you want to store and visualize data, you have a lot of technologies to choose from. Two such technologies are Elasticsearch (ES) and Grafana. Something that I really like about ES, is that its very easy to use and that it needs no complicated setup or configuration. It just works out-of-the-box (production environments may require more sophisticated configuration). Even though Logstash and Kibana are probably the most well known companions for ES, especially when it comes to visualization of logging data, Grafana is also quite good for time series based data and it keeps your technology stack simple. In the following, I will describe how to store data in Elasticsearch 5.6.0 through its HTTP API and visualize it with Grafana 4.4.3.

Elasticsearch

1. Download ES 5.6.0 from the official site and unpack
2. bin/elasticsearch
3. Web-Frontend can be found at http://localhost:9300

Elasticsearch is a document database that stores documents as JSON. Documents can be stored and retrieved by its HTTP API (there are also libraries for the most popular programming languages). Furthermore, ES stores documents into what is called indices and types. An index could be compared to a database in a relational DBMS and a type could be compared with a table.
For example: you could have an index called customers with a type invoices and another type addresses. You can think of types about something that logically belongs to an index (a customer has an address and one or more invoices).

Grafana

1. Download Grafana from the official site and unpack
2. bin/grafana-server
3. Web-Frontend can be found at http://localhost:3000

Configure a Data Source:

1. Go to http://localhost:3000/datasources and click on "add data source"
2. Enter a name for the data source and select "Elasticsearch" from the Type drop-down
3. As long as you didn't changed the default ES configuration, enter "http://localhost:9300" in the Url field.
4. Access must be set to "Proxy"
5. Enter the name of your index (for this example, we need no pattern)
6. Every JSON document you insert into ES must have an ISO date field, e.g. 2017-07-20T14:00:00. This is important, because as mentioned before, Grafana is designed to work with time series based data. Therefore, enter the name of the field that contains the ISO date into the field "time field name" (without a leading @). I called this field "insert_time" (as shown in the screenshot below).
7. Since we are using ES 5.6.0, select version 5.x from the Version drop-down

Configure a Dashboard:

1. Go to http://localhost:3000/?orgId=1 and click on "Create your first dashboard"
2. Click on "Graph" to create a new graph dashboard
3. Click on "Panel Title", then click on "Edit" (after that, the Graph configuration appears down below)
4. In the "General" tab, you can give the dashboard a name
5. In the top-right corner, select a time range (e.g. last 90 days), where you know that there is some data
6. If you wonder why you don't see anything, go to the "Display" tab and
• Draw Modes: select "Lines"
• Stacking & Null value: select "connected" for Null value
7. In case your JSON documents have a field that contains a numeric value that you want to visualize, you can also use the existing aggregate functions. Therefore, go to the "Metrics" tab and for example select "Average" as Metric and your timestamp/date field for "Group by" and select "Date Histogram" from the drop-down.

That's it! Thank you for reading.