EFK (Elasticsearch-Fluentd-Kibana) in kubeadm-dind-cluster

When you wish to have your logs sent to Elasticsearch and browse them with Kibana you should add to your k8s cluster EFK stack.

My first attempt to install EFK was with https://github.com/kubernetes/kubernetes/tree/master/cluster/addons/fluentd-elasticsearch

However it was fluentd was failing to send logs to Elasticsearch.

Fortunately on digitalocean website there is a blog post describing on how to do it and it happens to be working on a dind cluster. https://www.digitalocean.com/community/tutorials/how-to-set-up-an-elasticsearch-fluentd-and-kibana-efk-logging-stack-on-kubernetes

All the config posted over there is valid except for few caveats related to how dind stores logs.  To make fluentd pods have access to logs in fluentd.yaml add in volumeMounts:

And in volumes:

Also in elasticsearch-statefulset.yaml decrease required storage space and match storageClassName to the one that exists in the cluster in volumeClaimTemplates:

Here local-path provisioner has been used and storage is set to a low value like 1Gi.

Local-path provisioner has been added to the cluster (https://github.com/rancher/local-path-provisioner)

Now EFK can be onboarded onto a cluster as follows:

Check if you can access elasticsearch and kibana:

After port-forward provide container name relevant to your cluster. Check access on a host running kubectl port-forward

Now your system logs (not related to user pods, like kube-apiserver) should be fed by fluentd to elasticsearch and kibana can read them for es backend. You can have workload pods logs logs flowing into elasticsearch or build a dedicated instance of EFK just for that purpose to segregate system vs workload logging.

 

kubeadm-dind-cluster or kind

Even though it was recently retired in favour of kind (https://kind.sigs.k8s.io/kubeadm-dind (https://github.com/kubernetes-retired/kubeadm-dind-cluster) is still a great way to run a kubernetes cluster locally on laptop (in my case it is MacBook). It’s main limitation is that it supports v1.14.1, while kind supports v1.15.3 (though yesterday v1.16 version got released). Both rely on the same principle that nodes are not baremetal nodes nor VM, they are plain docker containers.  Then kubeadm is used to deploy k8s clusters on docker-based k8s cluster. System pods as well as user pods run as embedded containers inside docker container hosts (hence dind = docker-in-docker)

Yet with kubeadm-dind-cluster you can easily switch between CNI plugins, and it works between Docker for Desktop restarts and OS reboots.

Switching between CNI plugins is as easy as modifying inside an install script the following variable:

Main issue with kind is that OS reboot (not necessarily docker restart) causes originally assigned IPs to docker containers acting as k8s nodes to be changed. Hence after creating a cluster, making some configuration changes, or deploying pods etc we are forced to redo the job. Root cause of the problem is that docker doesn’t maintain originally assigned IPs to containers across docker restarts/OS reboots – at least it is not forced to do it (https://github.com/moby/moby/issues/2801). It is possible to assign static IPs to containers when starting containers with docker run. Unfortunately kind does not offer such option (for now). Issue is tracked at: https://github.com/kubernetes-sigs/kind/issues/148

All in all, when you want to practise with k8s (multi-node) on your laptop kubeadm-dind-cluster. kind is a very near future when it is made stable across docker restarts. 

ConfD – making your network elements programmable

Tail-F, originally a Swedish company bought over by Cisco Systems, created a management agent software framework for network elements and applications – ConfD. If employed as a part of software – ConfD allows to be programmability an inherent part of it. This post will us a very tangible example of its power – while Linux natively doesn’t offer NETCONF to manage interfaces – with use of ConfD and special program that makes use of it – programmability is made possible.

ConfD uses, what most of up-to-speed network engineers should know, YANG – a data modeling language for NETCONF (if we treat NETCONF as a new SNMP then YANG is like SMI – “special version” of ASN.1 was for SNMP) – better description can be found here: http://www.tail-f.com/what-is-yang/

The way to depict what ConfD does is – let’s say if you create your own program that acts as a router and you need a CLI for e.g. show ip route and additionally you would like to have a possibility to check routes over the NETCONF then it’s going to be done automatically for you. ConfD will run as a process next to your routing process and will manage it’s configuration, offer CLI and NETCONF (also RESTCONF/SNMP/WebAPI if Premium version is used). Of course that means that a if you want to implement ConfD shall be a part of implementing your software, whole configuration, CLIs etc.

There are 2 versions of confd available – free (basic) and more capable (paid) premium – comparison can be found here: http://www.tail-f.com/confd-basic

More on how ConfD works and its composition (for example there is a database – CDB – that keeps the config) can be found on Tail-F site.

Let’s get to business, demonstration of how it can be used is based on:

  • confd basic 6.3
  • ydk 0.7.1 (there will be a separate post on ydk, installation, usage and what is it)

The very reason for using confd 6.3 is that newer versions i.e. 6.4 and 6.6 use YDK 1.1 for modelling and example program I wanted to test with YDK on newer confd versions uses YANG 1.1 while YDK has just partial support for it (full support is for YDK 1.0 – RFC 6020).

In order to install confd:

  1. Download the 6.3 and 6.4 versions appropriate for you os from https://developer.cisco.com/site/confD/downloads/
  2. Install it: 

  3. Source confdrc file (so as to have access to confd cli tools:

  4. Run some example intro example:

  5. access confd CLI by running confd_cli command unless you started using make cli that takes you directly to CLI

 

Run the example:

Example I would like to use is called linuxcfg but id doesn’t work out of the box. When both versions 6.3.and 6.4 are installed the Makefile in ipmibs directory must be overwritten in 6.3 examples folder:

When this is done you can compile the example.

ip a from linux level:

show ip ipAddressEntry from confd level:

Ok so maybe output is not ideal like… IP addressing in HEX.

You can also use pipes and regex to filter what you only need, you can save output, export to csv, json, curly-braces (like Juniper CLI) etc.

From now on you are on your own….just kiddn’ in the next post I will show how to setup YDK and query linucfg over NETCONF with YANG models provided with this example. Now we have a running linuxcfg with confd that exposes NETCONF. More to follow…

Automatically provisioning baremetal/VM server

During my endavours I had a situation where I had to provision 10 servers (install all of them manually and configure same things on all of them, same files etc).

There is a nice alternative to it called Stacki from StackIQ (bought by Teradata last year). What it offers is specialized PXE server that is used to boot baremetal/VM servers (CentOS/Redhat/Ubuntu). 

Its architecture look as follows (Stacki server == Frontend, server to be provisioned == backend):

Firstly in CSV file you prepare a list of hosts with their MACs and as a next step you add puppet that will be used to provision the servers after booting.

More can be found here: 

https://github.com/Teradata/stacki

Frontend machine can be a VM – actually it worked pretty nice – tested with provisioning other VMs.

Connecting over SSH and running a command over NETCONF

When HW/virtualized/containerized network element offers NETCONF interface to manage, it is extremely beneficial to use it for repetitive tasks (upgrades, sanity checking, route table checking etc).

NETCONF can be used over different transports as below:

In case SSH is used then it must be made sure that SSH subsystem is enabled in SSH config on a device. 

If a YANG model is available then TailF offers a client and Java class generator under:

https://github.com/tail-f-systems/JNC

Alternatively, as in the example below, Python can be used to manually connect over SSH and based on YANG model (if used) instruct the device to perform a specific task.

There is a library in Python for that purpose called ncclient (NETCONF client): 

https://github.com/ncclient/ncclient

Firstly we need to do the proper import in our client script after installing the library:

Let’s define method used for connecting to the device:

Create an object class that inherits from RPC class of ncclient library and define a method that will be compose an XML NETCONF message based on YANG model:

Connect to the device and perform requested action (probably not the safest way to use clear text password to connect):

The request in NETCONF formatted XML would look as follows:

And a corresponding YANG model:

Lastly in XML: