OpenIDM 4.5 includes support for integration with ElasticSearch. For those not familiar, ElasticSearch is part of what is commonly referred to as the ELK stack:
E – ElasticSearch: Processes incoming data and allows for it to be quickly and RESTfully analysed and searched.
L – LogStash: A logging processor that takes input data, puts it through a pipeline of operations and outputs it somewhere ( usually into ElasticSearch ).
K – Kibana: Visualisation engine that utilises the data provided by ElasticSearch to create visual charts of all different types that can be embedded into web pages.
Together they enable you to create slick visualisations of data.
In the latest 4.5 release of OpenIDM two new features have been introduced:
- An ElasticSearch audit handler, that can be used to push audit events into ElasticSearch.
- Configurable dashboard widgets in the admin interface. There are a number of widgets available OOTB, one of which enables the embedding of Kibana visualisations.
This is incredibly powerful and relatively easy to set up, so I wanted to do a quick blog to explain how to it.
In this blog we do not actually need LogStash as we will use the OpenIDM audit handler to push data to ElasticSearch. However ordinarily may well want to use LogStash as well to process and transform additional logging data ( as well as just audit data ).
Installing ElasticSearch & Kibana
The first step is to download and install ElasticSearch from https://www.elastic.co/downloads/elasticsearch
Simply unzip ElasticSearch somewhere and start it up:
mv elasticsearch-2.4.1 elasticsearch
You can check that it is running by navigating to http://localhost.localdomain.com:9200, and you should see something like this:
Next, download and install Kibana from https://www.elastic.co/products/kibana
Again, just unzip it somewhere:
tar -xvf kibana-4.6.1-darwin-x86_64.tar.gz
But before starting it up check the following file:
And ensure it is pointing at your elasticsearch instance:
Then start it up:
If you can see this kibana splash page, then everyone is on track.
Configuring OpenIDM 4.5 for ElasticSearch Audit
Log into OpenIDM as the administrator and navigate to Configure, System Preferences
Then select the Audit tab
And add an ElasticSearch event handler
Give your handler a name, configure all the events you are interested in, and make sure Enabled is selected
Note: In earlier builds there was a bug where to enable the handler you had to manually edit conf/audit.json. If you follow all the steps and something is not working, double check that the handler is set to “enabled” : true
Scroll down and configure the handler parameters as below:
host: your host
port: 9200 ( unless you changed it)
Press Submit then on the next page you must remember to scroll down and press Save
Now, put your elasticsearch process window somewhere you can see it, then log out and back in of OpenIDM and take a look. You should see some activity:
OpenIDM is now successfully pushing audit data to elasticsearch!
Processing Audit Data in Kibana
Now we need to configure Kibana to process and do things with all that juicy audit data from OpenIDM. Navigate to Kibana again and replace “logstash-*” with “audit, then hit Create:
We now have an Audit indices, and you should see the OpenIDM audit attributes have already been catalogued:
And if you click Discover you should be able to browse the data!
Creating a Visualisation
Now with our data, we can quickly create visualisations. There are many blogs on this subject so I won’t go into detail here but I will create a quick visualisation to show failed authentications into OpenIDM over time.
The easiest way to do this is to generate such an event ( try logging in with the wrong password for example ) then take a look at the data.
We should see something like this:
And if we drill down further:
eventName:”authentication” AND result:”FAILED”
Now we have just failed authentication events, ( not successful ones) :
Hit Save and give the search it a name:
Now go to Visualize and select Vertical bar chart:
And use our saved search:
As you can guess this isn’t quite right, but if we configure the x-axis and press Apply ( play button )
We get something that looks a bit more useful. Now if you try generating some failed authentications and refreshing the page you should see this update.
Save the visualisation.
Adding the Visualisation to OpenIDM
Now it is time to bring it all together. Select Share Visualisation
Examine the first URL ( the Embed URL ).
Copy the URL somewhere, just the http://localhost… discard the iframe tag e.g.
Navigate to OpenIDM and log in as administrator, then Dashboards and New Dashboard:
Call it whatever you want and press Create:
Add a widget to your new dashboard:
Select Embed Webpage:
Then close the widgets menu (x):
And enter the following values:
URL: The URL from the Kibana visualisation
Height: You may have to play with this a bit, but 200px is a good start.
Title: Whatever you like.
And press Save.
You should now see your visualisation in OpenIDM!
In around an hour, we have configured the ELK stack, integrated it with OpenIDM and created a (very) basic visualisation. This is really the tip of the ice berg of what you could do with OpenIDM and ELK and I plan to explore this in much more detail in future blogs and share any useful visualisations I come up with.
This blog post was first published @ http://identity-implementation.blogspot.no/, included here with permission from the author.