EFK (Elasticsearch-Fluentd-Kibana) in kubeadm-dind-cluster

When you wish to have your logs sent to Elasticsearch and browse them with Kibana you should add to your k8s cluster EFK stack.

My first attempt to install EFK was with https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch

However it was fluentd was failing to send logs to Elasticsearch.

Fortunately on digitalocean website there is a blog post describing on how to do it and it happens to be working on a dind cluster. https://www.digitalocean.com/community/tutorials/how-to-set-up-an-elasticsearch-fluentd-and-kibana-efk-logging-stack-on-kubernetes

All the config posted over there is valid except for few caveats related to how dind stores logs.  To make fluentd pods have access to logs in fluentd.yaml add in volumeMounts:

And in volumes:

Also in elasticsearch-statefulset.yaml decrease required storage space and match storageClassName to the one that exists in the cluster in volumeClaimTemplates:

Here local-path provisioner has been used and storage is set to a low value like 1Gi.

Local-path provisioner has been added to the cluster (https://github.com/rancher/local-path-provisioner)

Now EFK can be onboarded onto a cluster as follows:

Check if you can access elasticsearch and kibana:

After port-forward provide container name relevant to your cluster. Check access on a host running kubectl port-forward

Now your system logs (not related to user pods, like kube-apiserver) should be fed by fluentd to elasticsearch and kibana can read them for es backend. You can have workload pods logs logs flowing into elasticsearch or build a dedicated instance of EFK just for that purpose to segregate system vs workload logging.

 

Instance not started due to Neutron misconfiguration on Ubuntu 14.04

While attempting to start instance the following problem appears:

/var/log/nova/nova-compute.log

 /var/log/neutron/server.log

Solution:

Add in /etc/nova/nova.conf in [DEFAULT] section:

Problem described here:

https://ask.openstack.org/en/question/26938/virtualinterfacecreateexception-virtual-interface-creation-failed/