24 December 2019

Dave’s Holiday Project 2019

My holiday project for 2019/2020 is a log management system inspired by SPLUNK! using open source tools. The tools I will use are:

Process

The process of finding and installing the software above was straightforward and simple. I downloaded and installed Docker and Kinematic, the GUI management overlay for docker. I downloaded docker images for Solr and NiFi from a terminal command line.

docker pull solr
docker create solr solr
docker pull nifi
docker create nifi nifi
docker start solr
docker start nifi

Log Management

Log management is a deceptively simple term. The logs from information technology sources can be much more than an exciting text to read when your big plans for Saturday night fall through. With the right software these logs can drive interactive dashboards that supply real time information about business processes. It is said they can be used to drive artificial intelligence powered software to make useful predictions. That is the ultimate intent of this holiday project.

Docker

Docker allows me to containerize machines for cross environment compatibility.

I installed the latest Docker and Kitematic on my Macbook Air and used the terminal command docker pull solr to obtain a pre-built solr instance in a docker container.

Next I used the terminal command `docker create solr solr to turn the image into a container. I started Kitematic and the container appear there with a randomly generated name. I used Kitematic to rename it to solr.
I then used Kitematic to assign a port on the local machine to the docker container. I used the same port on both. Port 8983.

My web browser could now access the solr dashboard on http://localhost:8983/nifi

SOLR

Solr is an Apache project. It is mainly used as a distributed indexing and search engine, but it is also a nosql database.

Once installed it presents a web interface at port 8983 in standalone mode, and in Cloud Mode it uses ZooKeeper to manage distributed storage and search across multiple instances, perhaps on multiple machines. In cloud mode it presents a Zookeeper instance at port 9983.

solr_home
This is where solr keeps some important configuration files. Its location varies depending on how you install solr. Using the web interface is the easiest way of finding where it is. Click on Java Properties and scroll down until you see it. Mine was solr.​solr.​home located at /var/solr/data.

NiFi

Apache NiFi is a data flow engine that allows the design of processes using a graphical user interface. In this project I will use NiFi to ingest, process and send log data to Solr.

I obtained a NiFi docker image and created a container for it the same way I did for Solr above.

The canned demonstrations of NiFi on the web are deceptively simple. Ingesting a file and sending it to a local file is straightforward. Tailing a file, splitting it into lines, parsing the lines to obtain named attributes and sending the attributes to Solr is another matter. If it was easy people selling solutions that do it all for you would go broke! It isn’t easy.

Once set up in Docker Nifi is available on http://localhost:32770/nifi