OpenStack – stackblog.net

Juniper vMX – MPLS over lt interface

mazur Juniper, OpenStack May 12, 2019 0 Comment

I was wondering if I can run MPLS over OVS, VXLAN tenant networks in Openstack (for example VLAN tagging won’t work in such case, not by default at least, didn’t have an option to test with vlan_transparent setting on network level). Wanted to quickly check it out between a pair of vMXes with static LSPs remembering I can run logical-systems or routing-instances interconnected between themselves with tunnel interface (lt). That works just fine but only with logical-systems.

LS-vmx01 lt-0/0/0.1<–>lt-0/0/0.2 vmx01 ge-0/0/2.0<–>ge-0/0/2.0 vmx02 lt-0/0/0.2<—lt-0/0/0.1 LS-vmx02

vmx01 cfg:

 set logical-systems LS interfaces lt-0/0/0 unit 1
 set logical-systems LS routing-options static route 1.1.1.0/30 next-hop 2.2.2.2
 set chassis fpc 0 pic 0 tunnel-services bandwidth 10g
 set chassis fpc 0 lite-mode
 set chassis network-services enhanced-ip
 set interfaces lt-0/0/0 unit 1 encapsulation ethernet
 set interfaces lt-0/0/0 unit 1 peer-unit 2
 set interfaces lt-0/0/0 unit 1 family inet address 2.2.2.1/30
 set interfaces lt-0/0/0 unit 1 family mpls
 set interfaces lt-0/0/0 unit 2 encapsulation ethernet
 set interfaces lt-0/0/0 unit 2 peer-unit 1
 set interfaces lt-0/0/0 unit 2 family inet address 2.2.2.2/30
 set interfaces lt-0/0/0 unit 2 family mpls
 set interfaces ge-0/0/2 mtu 1600
 set interfaces ge-0/0/2 unit 0 family inet address 192.168.168.22/30
 set interfaces ge-0/0/2 unit 0 family mpls
 set routing-options static route 1.1.1.0/30 static-lsp-next-hop path-to-lo-vmx01
 set protocols mpls static-label-switched-path path-to-lo-vmx01 ingress next-hop 192.168.168.21
 set protocols mpls static-label-switched-path path-to-lo-vmx01 ingress to 1.1.1.1
 set protocols mpls static-label-switched-path path-to-lo-vmx01 ingress push 1000123
 set protocols mpls static-label-switched-path path-from-lo-vmx01 transit 1000124 next-hop 2.2.2.1
 set protocols mpls static-label-switched-path path-from-lo-vmx01 transit 1000124 pop
 set protocols mpls interface ge-0/0/2.0
 set protocols mpls interface lt-0/0/0.2

set logical-systems LS interfaces lt-0/0/0 unit 1

set logical-systems LS routing-options static route 1.1.1.0/30 next-hop 2.2.2.2

set chassis fpc 0 pic 0 tunnel-services bandwidth 10g

set chassis fpc 0 lite-mode

set chassis network-services enhanced-ip

set interfaces lt-0/0/0 unit 1 encapsulation ethernet

set interfaces lt-0/0/0 unit 1 peer-unit 2

set interfaces lt-0/0/0 unit 1 family inet address 2.2.2.1/30

set interfaces lt-0/0/0 unit 1 family mpls

set interfaces lt-0/0/0 unit 2 encapsulation ethernet

set interfaces lt-0/0/0 unit 2 peer-unit 1

set interfaces lt-0/0/0 unit 2 family inet address 2.2.2.2/30

set interfaces lt-0/0/0 unit 2 family mpls

set interfaces ge-0/0/2 mtu 1600

set interfaces ge-0/0/2 unit 0 family inet address 192.168.168.22/30

set interfaces ge-0/0/2 unit 0 family mpls

set routing-options static route 1.1.1.0/30 static-lsp-next-hop path-to-lo-vmx01

set protocols mpls static-label-switched-path path-to-lo-vmx01 ingress next-hop 192.168.168.21

set protocols mpls static-label-switched-path path-to-lo-vmx01 ingress to 1.1.1.1

set protocols mpls static-label-switched-path path-to-lo-vmx01 ingress push 1000123

set protocols mpls static-label-switched-path path-from-lo-vmx01 transit 1000124 next-hop 2.2.2.1

set protocols mpls static-label-switched-path path-from-lo-vmx01 transit 1000124 pop

set protocols mpls interface ge-0/0/2.0

set protocols mpls interface lt-0/0/0.2

vmx02 cfg:

 set logical-systems LS interfaces lt-0/0/0 unit 1
 set logical-systems LS routing-options static route 2.2.2.0/30 next-hop 1.1.1.2
 set chassis fpc 0 pic 0 tunnel-services bandwidth 10g
 set chassis fpc 0 lite-mode
 set chassis network-services enhanced-ip
 set interfaces lt-0/0/0 unit 1 encapsulation ethernet
 set interfaces lt-0/0/0 unit 1 peer-unit 2
 set interfaces lt-0/0/0 unit 1 family inet address 1.1.1.1/30
 set interfaces lt-0/0/0 unit 1 family mpls
 set interfaces lt-0/0/0 unit 2 encapsulation ethernet
 set interfaces lt-0/0/0 unit 2 peer-unit 1
 set interfaces lt-0/0/0 unit 2 family inet address 1.1.1.2/30
 set interfaces lt-0/0/0 unit 2 family mpls
 set interfaces ge-0/0/2 mtu 1600
 set interfaces ge-0/0/2 unit 0 family inet address 192.168.168.21/30
 set interfaces ge-0/0/2 unit 0 family mpls
 set routing-options static route 2.2.2.0/30 static-lsp-next-hop path-to-lo-vmx02
 set routing-options static route 1.1.1.0/30 static-lsp-next-hop path-to-lo-vmx01
 set protocols mpls static-label-switched-path path-to-lo-vmx02 ingress next-hop 192.168.168.22
 set protocols mpls static-label-switched-path path-to-lo-vmx02 ingress to 2.2.2.1
 set protocols mpls static-label-switched-path path-to-lo-vmx02 ingress push 1000124
 set protocols mpls static-label-switched-path path-from-lo-vmx02 transit 1000123 next-hop 1.1.1.1
 set protocols mpls static-label-switched-path path-from-lo-vmx02 transit 1000123 pop
 set protocols mpls interface ge-0/0/2.0
 set protocols mpls interface lt-0/0/0.2

set logical-systems LS interfaces lt-0/0/0 unit 1

set logical-systems LS routing-options static route 2.2.2.0/30 next-hop 1.1.1.2

set chassis fpc 0 pic 0 tunnel-services bandwidth 10g

set chassis fpc 0 lite-mode

set chassis network-services enhanced-ip

set interfaces lt-0/0/0 unit 1 encapsulation ethernet

set interfaces lt-0/0/0 unit 1 peer-unit 2

set interfaces lt-0/0/0 unit 1 family inet address 1.1.1.1/30

set interfaces lt-0/0/0 unit 1 family mpls

set interfaces lt-0/0/0 unit 2 encapsulation ethernet

set interfaces lt-0/0/0 unit 2 peer-unit 1

set interfaces lt-0/0/0 unit 2 family inet address 1.1.1.2/30

set interfaces lt-0/0/0 unit 2 family mpls

set interfaces ge-0/0/2 mtu 1600

set interfaces ge-0/0/2 unit 0 family inet address 192.168.168.21/30

set interfaces ge-0/0/2 unit 0 family mpls

set routing-options static route 2.2.2.0/30 static-lsp-next-hop path-to-lo-vmx02

set routing-options static route 1.1.1.0/30 static-lsp-next-hop path-to-lo-vmx01

set protocols mpls static-label-switched-path path-to-lo-vmx02 ingress next-hop 192.168.168.22

set protocols mpls static-label-switched-path path-to-lo-vmx02 ingress to 2.2.2.1

set protocols mpls static-label-switched-path path-to-lo-vmx02 ingress push 1000124

set protocols mpls static-label-switched-path path-from-lo-vmx02 transit 1000123 next-hop 1.1.1.1

set protocols mpls static-label-switched-path path-from-lo-vmx02 transit 1000123 pop

set protocols mpls interface ge-0/0/2.0

set protocols mpls interface lt-0/0/0.2

Main thing here was to set family mpls on lt-0/0/0 units and add lt-0/0/0.2 interface to protocols mpls.

Only after that POP entries appeared in routing and forwarding tables:

inet.0: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)
 = Active Route, - = Last Active, * = Both 
 1.1.1.0/30         *[Direct/0] 00:32:27
                     &gt;  via lt-0/0/0.2
 1.1.1.2/32         *[Local/0] 00:32:27
                        Local via lt-0/0/0.2
 2.2.2.0/30         *[Static/5] 00:35:42
                       to 192.168.168.22 via ge-0/0/2.0, Push 1000124
 inet.3: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)
 = Active Route, - = Last Active, * = Both 
 2.2.2.1/32         *[MPLS/6/1] 00:35:42, metric 0
                       to 192.168.168.22 via ge-0/0/2.0, Push 1000124
 mpls.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)
 = Active Route, - = Last Active, * = Both 
 1000123            *[MPLS/6] 00:32:27, metric 1
                       to 1.1.1.1 via lt-0/0/0.2, Pop
 1000123(S=0)       *[MPLS/6] 00:32:27, metric 1
                       to 1.1.1.1 via lt-0/0/0.2, Pop

inet.0: 7 destinations, 7 routes (7 active, 0 holddown, 0 hidden)

= Active Route, - = Last Active, * = Both

1.1.1.0/30 *[Direct/0] 00:32:27

> via lt-0/0/0.2

1.1.1.2/32 *[Local/0] 00:32:27

Local via lt-0/0/0.2

2.2.2.0/30 *[Static/5] 00:35:42

to 192.168.168.22 via ge-0/0/2.0, Push 1000124

inet.3: 1 destinations, 1 routes (1 active, 0 holddown, 0 hidden)

= Active Route, - = Last Active, * = Both

2.2.2.1/32 *[MPLS/6/1] 00:35:42, metric 0

to 192.168.168.22 via ge-0/0/2.0, Push 1000124

mpls.0: 8 destinations, 8 routes (8 active, 0 holddown, 0 hidden)

= Active Route, - = Last Active, * = Both

1000123 *[MPLS/6] 00:32:27, metric 1

to 1.1.1.1 via lt-0/0/0.2, Pop

1000123(S=0) *[MPLS/6] 00:32:27, metric 1

to 1.1.1.1 via lt-0/0/0.2, Pop

 2.2.2.0/30         user     0 10.135.7.22       Push 1000124      613     2 ge-0/0/2.0
 
 1000123            user     0 1.1.1.1           Pop        647     2 lt-0/0/0.2
 1000123(S=0)       user     0 1.1.1.1           Pop        648     2 lt-0/0/0.2

2.2.2.0/30 user 0 10.135.7.22 Push 1000124 613 2 ge-0/0/2.0

1000123 user 0 1.1.1.1 Pop 647 2 lt-0/0/0.2

1000123(S=0) user 0 1.1.1.1 Pop 648 2 lt-0/0/0.2

Modifying virtual machine images

mazur OpenStack June 3, 2018glance, guestfish, openstack, virt-customize, virt-edit 0 Comment

What to do when you have a virtual machine image and for a example you need to some files contents like ssh config or so? Modified images can be uploaded to glance – repeating same step after running several VMs of the same type can be easily avoided in this way.

There are few tools that can be used for that purpose and are extremely powerful (most importantly, these are usually run in place :

virt-customize (http://libguestfs.org/virt-customize.1.html)

Example – install wireshark & nmap in an RHEL minimal install image (we assume there is Internet access from a machine running virt-customize command) – below is basically running locally a VM, modifying it and saving the changes to the image.

virt-customize -v -x -a rhel-guest-image-version.qcow2 --run-command "yum install <a href="https://dl.fedoraproject.org/pub/epel/epel-release-latest-$(rpm">https://dl.fedoraproject.org/pub/epel/epel-release-latest-$(rpm</a> -E '%{rhel}').noarch.rpm -y" --run-command 'yum install tcpdump -y' --run-command 'yum install wireshark -y' --run-command 'yum install nmap -y'

virt-customize -v -x -a rhel-guest-image-version.qcow2 --run-command "yum install <a href="https://dl.fedoraproject.org/pub/epel/epel-release-latest-$(rpm">https://dl.fedoraproject.org/pub/epel/epel-release-latest-$(rpm</a> -E '%{rhel}').noarch.rpm -y" --run-command 'yum install tcpdump -y' --run-command 'yum install wireshark -y' --run-command 'yum install nmap -y'

virt-edit (http://libguestfs.org/virt-edit.1.html)

Example – modify ssh server config. Following edits a file in a filesystems and saves the changes to the image.

virt-edit -a image-name.qcow2 /etc/ssh/sshd_config

1	virt-edit -a image-name.qcow2 /etc/ssh/sshd_config

guestfish (http://libguestfs.org/guestfish.1.html)

Guestfish gives access to the filesystem – it is more powerful the virt-edit in a sense that it allows browsing through the filesystems rather than modifying a file that you know a path for. You can also create new files and add contents to them. Example sequence of step to perform is as follows:

1) OPTIONAL: export LIBGUESTFS_BACKEND=direct

2) guestfish -a <qcow2/img>

3) run

4) list-filesystems

5) mount <root_filesystem> /

6) modify files

7) umount /

8) exit

Increase LV size of a qcow2 image

mazur OpenStack June 3, 2018June 3, 2018glance, guestfish, openstack, virt-resize 0 Comment

When you get a qcow2 image with a given size – it can’t be simply changed on-the-fly while running a VM or by giving just more space to a VM flavor in OpenStack. Situation gets even more complex when image has LVs inside but fortunately by using guestfish and virt-resize image can be suited to one’s needs. Below are the steps that I used to perform such modifications:

Default image: image-name-250G.qcow2
Resized: image-name-750G.qcow2

1. Check which device to resize (this image has LVM created PV on /dev/sda2):

guestfish -a image-name.qcow
run
pvs-full -- SHOWS WHICH DEVICE TO RESIZE

guestfish -a image-name.qcow

run

pvs-full -- SHOWS WHICH DEVICE TO RESIZE

2. Resize image (from 250 to 750GB) – resizing is NOT performed in place:

cp image-name-250G.qcow2 image-name-750G.qcow2
qemu-img resize image-name-750G.qcow2 +500GB

1 2	cp image-name-250G.qcow2 image-name-750G.qcow2 qemu-img resize image-name-750G.qcow2 +500GB

3. Resize disk and specific device (in this case it is /dev/sda2) and LVM PV:

virt-resize --expand /dev/sda2 image-name-250G.qcow2 image-name-750G.qcow2

1	virt-resize --expand /dev/sda2 image-name-250G.qcow2 image-name-750G.qcow2

4. Go to guestfish and use free space on VGS to create additional LV, create additional filesystem and mount point (DISABLE 64bit flag on EXT4, required by this image as it uses outdated e2fsck
that doesn’t support 64bit option):

guestfish -a image-name-750G.qcow
run
vgs-full

lvcreate lv_additional_lv  VolGroup00 512000 (500GB)
lvs-full
mkfs-opts ext4 /dev/VolGroup00/lv_additional_lv features:^64bit
list-filesystems

mount mkfs-opts ext4 /dev/VolGroup00/lv_root / 
mkdir /additional_directory -- directory for new LV
vi /etc/fstab -- add entry
/dev/mapper/VolGroup00-lv_additional_lv /dbdata ext4 defaults,nodev 1 2

umount /
Exit

guestfish -a image-name-750G.qcow

run

vgs-full

lvcreate lv_additional_lv VolGroup00 512000 (500GB)

lvs-full

mkfs-opts ext4 /dev/VolGroup00/lv_additional_lv features:^64bit

list-filesystems

mount mkfs-opts ext4 /dev/VolGroup00/lv_root /

mkdir /additional_directory -- directory for new LV

vi /etc/fstab -- add entry

/dev/mapper/VolGroup00-lv_additional_lv /dbdata ext4 defaults,nodev 1 2

umount /

Exit

5. Upload image to glance.

Heat templates with cinder volumes

mazur OpenStack May 13, 2018May 13, 2018cinder 0 Comment

While trying to launch several VMs using heat templates on RHEL OpenStack Liberty only 2 out of 8 launched. Rest failed because cinder volumes could not be created.

heat stack status was CREATE_FAILED and cinder volume status was error

/var/log/cinder/volume.log

HTTPInternalServerError: HTTPInternalServerError (HTTP 500)
(...)
Unable to get internal tenant context: Missing required config parameters.
(...)
Unable to get Cinder internal context, will not use image-volume cache

HTTPInternalServerError: HTTPInternalServerError (HTTP 500)

(...)

Unable to get internal tenant context: Missing required config parameters.

(...)

Unable to get Cinder internal context, will not use image-volume cache

For a brief moment there was some log on a console related to haproxy failure

While checking the services I noticed that both haproxy and swift-proxy are not running so just decided to restart them

systemctl restart haproxy.service

systemctl start openstack-swift-proxy.service

After that I could again create cinder volumes via heat templates. So be careful – create VMs from templates with cinder one by one.

IOS-XRv running on OpenStack

mazur Cisco, OpenStack March 14, 2015cisco, openstack, spice 0 Comment

In order to access IOS-XRv running on OpenStack compute node Spice console is required instead of VNC.

Instructions can be found under the following links:

http://joshrestivo.com/?p=32

https://ask.openstack.org/en/question/9199/having-error-in-spice-console/

NTP synchronization

mazur OpenStack March 14, 2015ntp, openstack 0 Comment

Date must be correctly synchronized on all nodes belonging to OpenStack cluster.

If date is not in sync between nodes then for example compute nodes may be seen as down from controller node.

Accessing cloud images

mazur OpenStack March 14, 2015March 14, 2015juno, nova, openstack 0 Comment

For both Fedora and Ubuntu cloud images we can access them using ssh public key generated during bootup.

Key can be obtained from a console log:

Horizon -> Instances -> <fedora_ubuntu_instance> -> Console -> View Full Console Log

sudo vim /var/lib/nova/instances/<instance_id>/console.log

ssh -i <ssh_rsa_dsa_key> <fedora|ubuntu>@<ip_address>

1	ssh -i <ssh_rsa_dsa_key> <fedora\|ubuntu>@<ip_address>

SSH can be done from from a proper ip namespace on compute nodes:

ip netns exec <netns_uiid> ssh....

1	ip netns exec <netns_uiid> ssh....

Or from any different host if Public IP is associated with the instance.

Nova installation on OpenSUSE 13.1

mazur OpenStack March 14, 2015juno, nova, openstack, opensuse 0 Comment

Firewalld

Firewall rules can be modified from GUI:

yast2 firewall

1

yast2 firewall
Assign internal interfaces to “Internal zone”

Add SSH service to “External zone”

RabbitMQ

In order to allow to access rabbitmq from a remote (and local) host to rabbitmq server before starting rabbitmq-server add in /etc/rabbitmq/rabbitmq.config:

[{rabbit, [{loopback_users, []}]}].

1	[{rabbit, [{loopback_users, []}]}].

Instance creation fails with “Cannot find suitable CPU” on Debian 7

mazur OpenStack March 14, 2015March 14, 2015debian, juno, nova, openstack 0 Comment

KVM by default does not work (even though vmx/smx is enabled for CPUs) – instance creation fails with “Cannot find suitable CPU”.

Solution:

In /etc/libvirtd/qemu.conf:

/etc/libvirtd/qemu.conf

security_driver = "none"

1

security_driver = "none"
remove file /var/cache/libvirt/qemu/capabilities and restart host (libvirtd restart does not suffice)
After restart check capabilities with:

virsh capabilities

1

virsh capabilities

VNC and firewalld on Centos 7 compute node

mazur OpenStack March 14, 2015March 14, 2015centos, juno, nova, openstack 0 Comment

By default firewalld service is enabled and may cause troubles in accessing via VNC to instances running on Centos compute node.

Solution:

By default zone=public is used, so it is enough just to add 5900/5901 ports or/and vnc-server service to this zone:

firewall-cmd --zone=public --add-port=5900/tcp
firewall-cmd --zone=public --add-port=5901/tcp

1 2	firewall-cmd --zone=public --add-port=5900/tcp firewall-cmd --zone=public --add-port=5901/tcp

Result:

public (default, active)
  interfaces: br-int br-tun eno16777736 eno33554984 eno50332208
  sources:
  services: dhcpv6-client ssh vnc-server
  ports: 5900/tcp 5901/tcp
  masquerade: no
  forward-ports:
  icmp-blocks:
  rich rules:

public (default, active)

interfaces: br-int br-tun eno16777736 eno33554984 eno50332208

sources:

services: dhcpv6-client ssh vnc-server

ports: 5900/tcp 5901/tcp

masquerade: no

forward-ports:

icmp-blocks:

rich rules:

← Previous