vSphere ESXi access without vSphere client

Today I stumbled on quite irritating problem. If you are OSX or Linux user and your ESXi setup doesn’t have vCenter (which comes with a price) then you are:

  1. Doomed to run a VM with Windows and install a vSphere client there (it requires Windows and .NET)
  2. Lose your time over trying to run it using Wine (there are instances where where with specific wine version and vSphere client it could work – I didn’t manage) or Crossover (if on Mac)
  3. Install VMware Fusion (if on Mac) – but it is a paid solution, on the other hand you might already have it – it is capable of connecting to vCenter and ESXi hosts.
  4. Install ESXI host client UI https://labs.vmware.com/flings/esxi-embedded-host-client

Following assumes there is internet connectivity from your ESXi host (but that’s not a big deal if there is not, just SCP to upload vib package), so connect to your ESXi host over SSH and then:

Now you can access your ESXi host over UI: https://<esxi_host_ip/ui/

ConfD – making your network elements programmable

Tail-F, originally a Swedish company bought over by Cisco Systems, created a management agent software framework for network elements and applications – ConfD. If employed as a part of software – ConfD allows to be programmability an inherent part of it. This post will us a very tangible example of its power – while Linux natively doesn’t offer NETCONF to manage interfaces – with use of ConfD and special program that makes use of it – programmability is made possible.

ConfD uses, what most of up-to-speed network engineers should know, YANG – a data modeling language for NETCONF (if we treat NETCONF as a new SNMP then YANG is like SMI – “special version” of ASN.1 was for SNMP) – better description can be found here: http://www.tail-f.com/what-is-yang/

The way to depict what ConfD does is – let’s say if you create your own program that acts as a router and you need a CLI for e.g. show ip route and additionally you would like to have a possibility to check routes over the NETCONF then it’s going to be done automatically for you. ConfD will run as a process next to your routing process and will manage it’s configuration, offer CLI and NETCONF (also RESTCONF/SNMP/WebAPI if Premium version is used). Of course that means that a if you want to implement ConfD shall be a part of implementing your software, whole configuration, CLIs etc.

There are 2 versions of confd available – free (basic) and more capable (paid) premium – comparison can be found here: http://www.tail-f.com/confd-basic

More on how ConfD works and its composition (for example there is a database – CDB – that keeps the config) can be found on Tail-F site.

Let’s get to business, demonstration of how it can be used is based on:

  • confd basic 6.3
  • ydk 0.7.1 (there will be a separate post on ydk, installation, usage and what is it)

The very reason for using confd 6.3 is that newer versions i.e. 6.4 and 6.6 use YDK 1.1 for modelling and example program I wanted to test with YDK on newer confd versions uses YANG 1.1 while YDK has just partial support for it (full support is for YDK 1.0 – RFC 6020).

In order to install confd:

  1. Download the 6.3 and 6.4 versions appropriate for you os from https://developer.cisco.com/site/confD/downloads/
  2. Install it: 

  3. Source confdrc file (so as to have access to confd cli tools:

  4. Run some example intro example:

  5. access confd CLI by running confd_cli command unless you started using make cli that takes you directly to CLI

 

Run the example:

Example I would like to use is called linuxcfg but id doesn’t work out of the box. When both versions 6.3.and 6.4 are installed the Makefile in ipmibs directory must be overwritten in 6.3 examples folder:

When this is done you can compile the example.

ip a from linux level:

show ip ipAddressEntry from confd level:

Ok so maybe output is not ideal like… IP addressing in HEX.

You can also use pipes and regex to filter what you only need, you can save output, export to csv, json, curly-braces (like Juniper CLI) etc.

From now on you are on your own….just kiddn’ in the next post I will show how to setup YDK and query linucfg over NETCONF with YANG models provided with this example. Now we have a running linuxcfg with confd that exposes NETCONF. More to follow…

Modifying virtual machine images

What to do when you have a virtual machine image and for a example you need to some files contents like ssh config or so? Modified images can be uploaded to glance – repeating same step after running several VMs of the same type can be easily avoided in this way.

There are few tools that can be used for that purpose and are extremely powerful (most importantly, these are usually run in place :

Example – install wireshark & nmap in an RHEL minimal install image (we assume there is Internet access from a machine running virt-customize command) – below is basically running locally a VM, modifying it and saving the changes to the image.

Example – modify ssh server config. Following edits a file in a filesystems and saves the changes to the image.

Guestfish gives access to the filesystem – it is more powerful the virt-edit in a sense that it allows browsing through the filesystems rather than modifying a file that you know a path for. You can also create new files and add contents to them. Example sequence of step to perform is as follows:

1) OPTIONAL: export LIBGUESTFS_BACKEND=direct

2) guestfish -a <qcow2/img>

3) run

4) list-filesystems

5) mount <root_filesystem> /

6) modify files

7) umount /

8) exit

Increase LV size of a qcow2 image

When you get a qcow2 image with a given size – it can’t be simply changed on-the-fly while running a VM or by giving just more space to a VM flavor in OpenStack. Situation gets even more complex when image has LVs inside but fortunately by using guestfish and virt-resize image can be suited to one’s needs. Below are the steps that I used to perform such modifications:

Default image: image-name-250G.qcow2
Resized: image-name-750G.qcow2

1. Check which device to resize (this image has LVM created PV on /dev/sda2):

2. Resize image (from 250 to 750GB) – resizing is NOT performed in place:

3. Resize disk and specific device (in this case it is /dev/sda2) and LVM PV:

4. Go to guestfish and use free space on VGS to create additional LV, create additional filesystem and mount point (DISABLE 64bit flag on EXT4, required by this image as it uses outdated e2fsck
that doesn’t support 64bit option):

5. Upload image to glance.

Automatically provisioning baremetal/VM server

During my endavours I had a situation where I had to provision 10 servers (install all of them manually and configure same things on all of them, same files etc).

There is a nice alternative to it called Stacki from StackIQ (bought by Teradata last year). What it offers is specialized PXE server that is used to boot baremetal/VM servers (CentOS/Redhat/Ubuntu). 

Its architecture look as follows (Stacki server == Frontend, server to be provisioned == backend):

Firstly in CSV file you prepare a list of hosts with their MACs and as a next step you add puppet that will be used to provision the servers after booting.

More can be found here: 

https://github.com/Teradata/stacki

Frontend machine can be a VM – actually it worked pretty nice – tested with provisioning other VMs.

Kill a hanging machine in VMware ESXi

It might not be possible to kill a “running” (in reality hanging) VM from Vcenter GUI or using vSphere Client. 

In such situation a login via SSH to ESXi host is needed, list all VMs and kill the appropriate one.

Connecting over SSH and running a command over NETCONF

When HW/virtualized/containerized network element offers NETCONF interface to manage, it is extremely beneficial to use it for repetitive tasks (upgrades, sanity checking, route table checking etc).

NETCONF can be used over different transports as below:

In case SSH is used then it must be made sure that SSH subsystem is enabled in SSH config on a device. 

If a YANG model is available then TailF offers a client and Java class generator under:

https://github.com/tail-f-systems/JNC

Alternatively, as in the example below, Python can be used to manually connect over SSH and based on YANG model (if used) instruct the device to perform a specific task.

There is a library in Python for that purpose called ncclient (NETCONF client): 

https://github.com/ncclient/ncclient

Firstly we need to do the proper import in our client script after installing the library:

Let’s define method used for connecting to the device:

Create an object class that inherits from RPC class of ncclient library and define a method that will be compose an XML NETCONF message based on YANG model:

Connect to the device and perform requested action (probably not the safest way to use clear text password to connect):

The request in NETCONF formatted XML would look as follows:

And a corresponding YANG model:

Lastly in XML:

Juniper Contrail – how to check Next-Hop

One might need to check which MPLSoGRE/UDP/VXLAN tunnel is used to reach a specific prefix on a vrouter.

There are few steps needed to find it out after logging in to a compute node running a VM/contrainer with an interface of interest (its prefix or prefix reachable via it) :

    1. List all interfaces and see which vrouter VRF a specific interface maps to:

    2. Use the VRF id to dump routes in a routing table for this VRF and check a prefix in question and its next-hop id:

    3. Check next-hop id parameters with:

Inter-AS Option B between Juniper routers

Recently I stumbled on a topic related to interconnecting 2 Juniper routers with Inter-AS Option B. 

With this kind of connectivity it is not enough to have control plane working properly i.e. prefixes exchanged and visible in appropriate routing-instances on both ends – for data plane to work you need to have next-hop resolved – and it is implicit in case E-BGP peering is sourced with interface address. In case loopback is used as a source you need to have it resolved by:

  • either a LSP between the two ASBR (actually no label required, but it must appear in inet.3, it can even be a static dummy LSP)
  • or configure routing-options resolution rib bgp.l3vpn.0 resolution-ribs inet.0 which will allow L3VPN routes to be resolved in inet.0 instead of inet.3

More details can be found on the Juniper forum where I found the solution:

https://forums.juniper.net/t5/Routing/interprovider-l3vpn-option-B/td-p/255305

 

There are also 2 useful commands that can be used for troubleshooting next-hop connectivity