Storm-cluster

Storm docker cluster using docker-compose. Including a Prometheus/Graphite container for Storms Metrics Reporting API v2 and Grafana for metrics visualization. Docker-compose/stack also includes Kafka service as example testdata source.

Based on offical Apache Storm docker image , Graphite docker image / Prometheus docker image, Grafana docker image, wurstmeister Apache Kafka image and cAdvisor.

Configuration:

• For swarm deploy set up docker swarm and set labels / edit placement labels accordingly in docker-stack.yml
• Adding configs to storm.yaml file overwrites the default values (for version 2.2.0).
• configure the used metric exporter and potential metric filter regex expressions according to Prometheus/Graphite usage in storm.yaml
• to collect host metrics with the included Prometheus service install and start Prometheus Node exporter on hosts
• If graphite is used set cAdvisor enviroment variable to use statsd storage driver of the graphite service:

environment: 
- storage_driver=statsd
- storage_driver_host=graphite:8125

Usage:

(Re)build and deploy:

$ docker-compose -f docker-stack.yml build

$ docker stack deploy --compose-file docker-stack.yml storm-cluster

or utilize scripts in /buildscripts

Get a cmdline on nimbus / supervisor (on the corresponding machine):

$ ./nimbus.sh

Run topology on nimbus (with shell on nimbus):

$ storm jar /topology.jar MainTopology Mytopology

Alternatively:

• configure enviroment variables (TOPOLOGY_JAR, TOPOLOGY_MAIN, TOPOLOGY_Name) in Dockerfile

• on nimbus shell:

  /./deploy_topology.sh

Write in Kafka example topic

• get shell on kafka:

  ./kafka.sh

• execute kafka generator jar

  java -jar kafka-generator-1.0-SNAPSHOT-jar-with-dependencies.jar

• the example messages generated by this kafka-generator are compatible to the provided example topology

Accessing via Browser on host:

When using swarm deployment, services are available via all host IPs. (see: ingress)

Storm UI: http://host:8080

Prometheus Dashboard: http://host:9090

Grafana Dashboard: http://host:3000 (userdetails: admin/admin)