Posts

Securing CloudStack 4.11 with HTTPS/TLS

,

Apache CloudStack is generally considered secure out of the box, however it does have the capability of protecting both system VM traffic as well as management traffic with TLS certificates. Prior to version 4.11 CloudStack used Tomcat as the web server and servlet container. With 4.11 this has been changed to embedded Jetty web server and servlet container that makes CloudStack more secure and independent of distribution provided Tomcat version which could be prone to security issues. This changes the management server’s TLS configuration.

In this blog post we’ll go through how to implement native TLS to protect both the system VMs – CPVM and SSVM – as well as the management server with public TLS certificates.

Please note since SSL is now being deprecated we now refer to the replacement – TLS – throughout this post.

Overview

The TLS configuration in CloudStack provides different functionality depending on the system role:

  • TLS protecting the management server is a Jetty configuration which protects the main web GUI and API endpoint only. The configuration of this is handled in the underlying embedded Jetty configuration files.
  • System VMs:
    • The Console Proxy VM TLS configuration provides secure HTTPS connection between the main console screen (your browser) and the CPVM itself.
    • The Secondary Storage VM SSL configuration provides a secure HTTPS connection for uploads and downloads of templates / ISOs / volumes to secondary storage as well as between zones.
    • System VM SSL configuration is carried out through global settings as well as the CloudStack GUI.

System VM HTTPS configuration

Global settings

The following global setting are required configured to allow system VM TLS configuration:

Global settingValue
consoleproxy.url.domaindomain used for CPVM (see below)
consoleproxy.sslEnabledSwitches SSL configuration of the CPVM on / off
Values true | false
secstorage.ssl.cert.domaindomain used for SSVM (see below)
secstorage.encrypt.copySwitches SSL configuration of the SSVM on / off
Values true | false

The URL configurations can take three formats – and these also determine what kind of TLS certificate is required.

  • Blank: if left blank / unconfigured the URLs used for CPVM and SSVM will simply be passed as the actual public IP addresses of the system VMs.
  • Static URL: e.g. console.mydomain.com or ssvm.mydomain.com. In these cases CloudStack rely on external URL load balancing / redirection and/or DNS resolution of the URL to the IP address of the CPVM or SSVM. This can be achieved in a number of different ways through load balancing appliances or scripted DNS updates.
    This configuration relies on:

    • The same URL used for both CPVM and SSVM, or
    • a multi-domain certificate provided to cover both URLs if different ones are used for CPVM and SSVM.
  • Dynamic URL: e.g. *.mydomain.com. In this case CloudStack will redirect the connections to the CPVM / SSVM to the URL “a-b-c-d.mydomain.com” where a/b/c/d represent the IP address, i.e. a real world URL would be 192-168-34-145.mydomain.com.
    This relies on two things:

    • DNS name resolution configured for the full public system VM IP range, such that all combinations of “a-b-c-d.mydomain.com” can be resolved. Please note in CloudStack version 4.11 the public IP range used purely by system VMs can be limited by reserving a subrange of public IP addresses just for system use.
    • An TLS wildcard certificate covering the full “mydomain.com” domain name.

Configuration process – GUI

The first step is to configure the four global settings above, then restart the CloudStack management service to make these settings live.

Next upload the TLS root certificate chain, the actual TLS certificate as well as the PKCS8 formatted private key using the “TLS certificate” button in the zone configuration. In this example we use a wildcard certificate.

Configuration process – API / CloudMonkey

The uploadCustomCertificate API call can be used to upload the TLS certificates. Please note the upload process does require at least two API calls – more depending on how many intermediary certificates are used. If you use CloudMonkey the certificates can be uploaded in cleartext – otherwise they have to be URLencoded when passed as part of a normal HTTP GET API call.

  • In the first API call the combined root and intermediary certificates are uploaded. In this API call the following parameters are passed – note we don’t pass the private key:
    • id=1
    • name: Give the certificate a name.
    • certificate: the root / intermediary certificate in cleartext, with all formatting / line breaks in place. In the example below we pass a chain of root and intermediary certificates.
    • domainsuffix: provide the suffix used, e.g. *.mydomain.com.
  • The second API call in our example uploads the site certificate. In this case we do not give the certificate a name (CloudStack automatically names this):
    • id=2
    • certificate: the issued site certificate, again in cleartext.
    • privatekey: the private key, same cleartext format as certificate.
    • domainsuffix: as above.
cloudmonkey upload customcertificate id=1 name=RootCertificate certificate='-----BEGIN CERTIFICATE-----
MIIE0DCCsdf8HqjeIHgkqhkiG9w0BAQsFADCBgzELMAk
...8V3Idv7kaWKK3245lsoynJuh87/BKONvPi8BDAB
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIEfTCCA2WgAwIBAgIDG+cVMA8djwj1ldggKd9d9s
.....mw4TRfZHcYQFHfjDCmrw==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIEADCCAuigAwIBAgIBADANBgkqhkiG9w0BAQUFA
.....csbkqWletNw+vHX/bvZ8=
-----END CERTIFICATE-----'
domainsuffix='*.mydomain.com'

cloudmonkey upload customcertificate id=2 certificate='-----BEGIN CERTIFICATE-----
MIIGrjCCBZagAwIBAgIJAJ....
...xKjPTkOLfwMVWXc8Ul25t7lkyi0+a9jZxFAuDXFRgkQnbw==
-----END CERTIFICATE-----'
privatekey='-----BEGIN PRIVATE KEY-----
MIIEvEbidvik1gkqhkiG9w0BAQEFAASCBK
.....rEF5Qyuyserre87d234jj/Uddf
-----END PRIVATE KEY-----'
domainsuffix='*.mydomain.com'

 

System VM restart

Once uploaded the CPVM and SSVM will automatically restart to pick up the new certificates. If the system VMs do not restart cleanly they can be destroyed and will come back online with the TLS configuration in place.

Testing the TLS protected system VMs

To test the Console Proxy VM simply open the console to any user VM or system VM. The popup browser window will at this point show up as a non-secure website. The reason for this is the actual console session is provided in an inline frame – and the popup page itself is presented from the unsecured management servers. If you however look at the page source for the page the HTTPS link which presents the inline frame shows up in the expected format:

Once the management server is also TLS protected the CPVM console popup window will also show as secured:

The SSVM will also now utilise the HTTPS links for all browser based uploads as well as downloads from secondary storage:

Securing the CloudStack management server GUI with HTTPS

Protecting the management servers requires creating a PCKS12 certificate and updating the underlying Jetty configuration to take this into account. This is described in http://wiki.eclipse.org/Jetty/Howto/Configure_SSL#Configuring_Jetty

First of all combine the TLS private key file, certificate and root certificate chain into one file (in the specified order), then convert this into PKCS12 format and write it to a new CloudStack keystore. Enter a new keystore password when prompted:

# cat myprivatekey.key mycertificate.crt gd_bundle-g2-g1.crt > mycombinedcert.crt

# openssl pkcs12 -in mycombinedcert.crt -export -out mycombinedcert.pkcs12
Enter Export Password: ************
Verifying - Enter Export Password:************

# keytool -importkeystore -srckeystore mycombinedcert.pkcs12 -srcstoretype PKCS12 -destkeystore /etc/cloudstack/management/keystore.pkcs12 -deststoretype pkcs12
Importing keystore mycombinedcert.pkcs12 to /etc/cloudstack/management/keystore.pkcs12...
Enter destination keystore password:************
Re-enter new password:************
Enter source keystore password:************
Entry for alias 1 successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled

 

Next edit /etc/cloudstack/management/server.properties and update the following:

https.enable=true
https.keystore=/etc/cloudstack/management/keystore.pkcs12
https.keystore.password=<enter the same password as used for conversion>

 

In addition automatic redirect from HTTP/port 8080 to HTTPS/port 8443 can also be configured in /usr/share/cloudstack-management/webapp/WEB-INF/web.xml. Add the following section before the section around line 22:

  <security-constraint>
    <web-resource-collection>
      <web-resource-name>Everything</web-resource-name>;
      <url-pattern>*</url-pattern>
    </web-resource-collection>
    <user-data-constraint>
      <transport-guarantee>CONFIDENTIAL></transport-guarantee>
    </user-data-constraint>
  </security-constraint>

 

Lastly restart the management service:

# systemctl restart cloudstack-management

Conclusion

The TLS capabilities of CloudStack has been extended over the last few releases – and we hope this article helps to explain the new global settings and configuration procedure.

About The Author

Dag Sonstebo is  a Cloud Architect at ShapeBlue, The Cloud Specialists. Dag spends most of his time designing, implementing and automating IaaS solutions based on Apache CloudStack.


Networking KVM for CloudStack – a 2018 revisit for CentOS7 and Ubuntu 18.04

, ,

Introduction

We published the original blog post on KVM networking in 2016 – but in the meantime we have moved on a generation in CentOS and Ubuntu operating systems, and some of the original information is therefore out of date. In this revisit of the original blog post we cover new configuration options for CentOS 7.x as well as Ubuntu 18.04, both of which are now supported hypervisor operating systems in CloudStack 4.11. Ubuntu 18.04 has replaced the legacy networking model with the new Netplan implementation, and this does mean different configuration both for linux bridge setups as well as OpenvSwitch.

KVM hypervisor networking for CloudStack can sometimes be a challenge, considering KVM doesn’t quite have the same mature guest networking model found in the likes of VMware vSphere and Citrix XenServer. In this blog post we’re looking at the options for networking KVM hosts using bridges and VLANs, and dive a bit deeper into the configuration for these options. Installation of the hypervisor and CloudStack agent is pretty well covered in the CloudStack installation guide, so we’ll not spend too much time on this.

Network bridges

On a linux KVM host guest networking is accomplished using network bridges. These are similar to vSwitches on a VMware ESXi host or networks on a XenServer host (in fact networking on a XenServer host is also accomplished using bridges).

A KVM network bridge is a Layer-2 software device which allows traffic to be forwarded between ports internally on the bridge and the physical network uplinks. The traffic flow is controlled by MAC address tables maintained by the bridge itself, which determine which hosts are connected to which bridge port. The bridges allow for traffic segregation using traditional Layer-2 VLANs as well as SDN Layer-3 overlay networks.

KVMnetworking41

Linux bridges vs OpenVswitch

The bridging on a KVM host can be accomplished using traditional linux bridge networking or by adopting the OpenVswitch back end. Traditional linux bridges have been implemented in the linux kernel since version 2.2, and have been maintained through the 2.x and 3.x kernels. Linux bridges provide all the basic Layer-2 networking required for a KVM hypervisor back end, but it lacks some automation options and is configured on a per host basis.

OpenVswitch was developed to address this, and provides additional automation in addition to new networking capabilities like Software Defined Networking (SDN). OpenVswitch allows for centralised control and distribution across physical hypervisor hosts, similar to distributed vSwitches in VMware vSphere. Distributed switch control does require additional controller infrastructure like OpenDaylight, Nicira, VMware NSX, etc. – which we won’t cover in this article as it’s not a requirement for CloudStack.

It is also worth noting Citrix started using the OpenVswitch backend in XenServer 6.0.

Network configuration overview

For this example we will configure the following networking model, assuming a linux host with four network interfaces which are bonded for resilience. We also assume all switch ports are trunk ports:

  • Network interfaces eth0 + eth1 are bonded as bond0.
  • Network interfaces eth2 + eth3 are bonded as bond1.
  • Bond0 provides the physical uplink for the bridge “cloudbr0”. This bridge carries the untagged host network interface / IP address, and will also be used for the VLAN tagged guest networks.
  • Bond1 provides the physical uplink for the bridge “cloudbr1”. This bridge handles the VLAN tagged public traffic.

The CloudStack zone networks will then be configured as follows:

  • Management and guest traffic is configured to use KVM traffic label “cloudbr0”.
  • Public traffic is configured to use KVM traffic label “cloudbr1”.

In addition to the above it’s important to remember CloudStack itself requires internal connectivity from the hypervisor host to system VMs (Virtual Routers, SSVM and CPVM) over the link local 169.254.0.0/16 subnet. This is done over a host-only bridge “cloud0”, which is created by CloudStack when the host is added to a CloudStack zone.

 

KVMnetworking42

Linux bridge configuration – CentOS

In the following CentOS example we have changed the NIC naming convention back to the legacy “eth0” format rather than the new “eno16777728” format. This is a personal preference – and is generally done to make automation of configuration settings easier. The configuration suggested throughout this blog post can also be implemented using the new NIC naming format.

Across all CentOS versions the “NetworkManager” service is also generally disabled, since this has been found to complicate KVM network configuration and cause unwanted behaviour:

 
# systemctl stop NetworkManager
# systemctl disable NetworkManager

To enable bonding and bridging CentOS 7.x requires the modules installed / loaded:

 
# modprobe --first-time bonding
# yum -y install bridge-utils

If IPv6 isn’t required we also add the following lines to /etc/sysctl.conf:

net.ipv6.conf.all.disable_ipv6 = 1 
net.ipv6.conf.default.disable_ipv6 = 1
net.ipv6.conf.lo.disable_ipv6 = 1

In CentOS the linux bridge configuration is done with configuration files in /etc/sysconfig/network-scripts/. Each of the four individual NIC interfaces are configured as follows (eth0 / eth1 / eth2 / eth3 are all configured the same way). Note there is no IP configuration against the NICs themselves – these purely point to the respective bonds:

# vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
NAME=eth0
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
HWADDR=00:0C:12:xx:xx:xx
NM_CONTROLLED=no

The bond configurations are specified in the equivalent ifcfg-bond scripts and specify bonding options as well as the upstream bridge name. In this case we’re just setting a basic active-passive bond (mode=1) with up/down delays of zero and status monitoring every 100ms (miimon=100). Note there are a multitude of bonding options – please refer to the CentOS / RedHat official documentation to tune these to your specific use case.

# vi /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
NAME=bond0
TYPE=Bond
BRIDGE=cloudbr0
ONBOOT=yes
NM_CONTROLLED=no
BONDING_OPTS="mode=active-backup miimon=100 updelay=0 downdelay=0"

The same goes for bond1:

# vi /etc/sysconfig/network-scripts/ifcfg-bond1
DEVICE=bond1
NAME=bond1
TYPE=Bond
BRIDGE=cloudbr1
ONBOOT=yes
NM_CONTROLLED=no
BONDING_OPTS="mode=active-backup miimon=100 updelay=0 downdelay=0"

Cloudbr0 is configured in the ifcfg-cloudbr0 script. In addition to the bridge configuration we also specify the host IP address, which is tied directly to the bridge since it is on an untagged VLAN:

# vi /etc/sysconfig/network-scripts/ifcfg-cloudbr0
DEVICE=cloudbr0
ONBOOT=yes
TYPE=Bridge
IPADDR=192.168.100.20
NETMASK=255.255.255.0
GATEWAY=192.168.100.1
NM_CONTROLLED=no
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=no
DELAY=0

Cloudbr1 does not have an IP address configured hence the configuration is simpler:

# vi /etc/sysconfig/network-scripts/ifcfg-cloudbr1
DEVICE=cloudbr1
ONBOOT=yes
TYPE=Bridge
BOOTPROTO=none
NM_CONTROLLED=no
DELAY=0
DEFROUTE=no
IPV4_FAILURE_FATAL=no
IPV6INIT=no

Optional tagged interface for storage traffic

If a dedicated VLAN tagged IP interface is required for e.g. storage traffic this can be accomplished by created a VLAN on top of the bond and tying this to a dedicated bridge. In this case we create a new bridge on bond0 using VLAN 100:

# vi /etc/sysconfig/network-scripts/ifcfg-bond.100
DEVICE=bond0.100
VLAN=yes
BOOTPROTO=none
ONBOOT=yes
TYPE=Unknown
BRIDGE=cloudbr100

The bridge can now be configured with the desired IP address for storage connectivity:

# vi /etc/sysconfig/network-scripts/ifcfg-cloudbr100
DEVICE=cloudbr100
ONBOOT=yes
TYPE=Bridge
VLAN=yes
IPADDR=10.0.100.20
NETMASK=255.255.255.0
NM_CONTROLLED=no
DELAY=0

Internal bridge cloud0

When using linux bridge networking there is no requirement to configure the internal “cloud0” bridge, this is all handled by CloudStack.

Network startup

Note – once all network startup scripts are in place and the network service is restarted you may lose connectivity to the host if there are any configuration errors in the files, hence make sure you have console access to rectify any issues.

To make the configuration live restart the network service:

# systemctl restart network

To check the bridges use the brctl command:

# brctl show
bridge name bridge id STP enabled interfaces
cloudbr0 8000.000c29b55932 no bond0
cloudbr1 8000.000c29b45956 no bond1

The bonds can be checked with:

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0c:xx:xx:xx:xx
Slave queue ID: 0

Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0c:xx:xx:xx:xx
Slave queue ID: 0

Linux bridge configuration – Ubuntu

With the 18.04 “Bionic Beaver” release Ubuntu have retired the legacy way of configuring networking through /etc/network/interfaces in favour of Netplan – https://netplan.io/reference. This changes how networking is configured – although the principles around bridge configuration are the same as in previous Ubuntu versions.

First of all ensure correct hostname and FQDN are set in /etc/hostname and /etc/hosts respectively.

To stop network bridge traffic from traversing IPtables / ARPtables also add the following lines to /etc/sysctl.conf, this prevents bridge traffic from traversing IPtables / ARPtables on the host.

# vi /etc/sysctl.conf
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

Ubuntu 18.04 installs the “bridge-utils” and bridge/bonding kernel options by default, and the corresponding modules are also loaded by default, hence there are no requirements to add anything to /etc/modules.

In Ubuntu 18.04 all interface, bond and bridge configuration are configured using cloud-init and the Netplan configuration in /etc/netplan/XX-cloud-init.yaml. Same as for CentOS we are configuring basic active-passive bonds (mode=1) with status monitoring every 100ms (miimon=100), and configuring bridges on top of these. As before the host IP address is tied to cloudbr0:

# vi /etc/netplan/50-cloud-init.yaml
network:
    ethernets:
        eth0:
            dhcp4: no
        eth1:
            dhcp4: no
        eth2:
            dhcp4: no
        eth3:
            dhcp4: no
    bonds:
        bond0:
            dhcp4: no
            interfaces:
                - eth0
                - eth1
            parameters:
                mode: active-backup
                primary: eth0
        bond1:
            dhcp4: no
            interfaces:
                - eth2
                - eth3
            parameters:
                mode: active-backup
                primary: eth2
    bridges:
        cloudbr0:
            addresses:
                - 192.168.100.20/24
            gateway4: 192.168.100.1
            nameservers:
                search: [mycloud.local]
                addresses: [192.168.100.5,192.168.100.6]
            interfaces:
                - bond0
        cloudbr1:
            dhcp4: no
            interfaces:
                - bond1
    version: 2

Optional tagged interface for storage traffic

To add an options VLAN tagged interface for storage traffic add a VLAN and a new bridge to the above configuration:

# vi /etc/netplan/50-cloud-init.yaml
    vlans:
        bond100:
            id: 100
            link: bond0
            dhcp4: no
    bridges:
        cloudbr100:
            addresses:
               - 10.0.100.20/24
            interfaces:
               - bond100

Internal bridge cloud0

When using linux bridge networking the internal “cloud0” bridge is again handled by CloudStack, i.e. there’s no need for specific configuration to be specified for this.

Network startup

Note – once all network startup scripts are in place and the network service is restarted you may lose connectivity to the host if there are any configuration errors in the files, hence make sure you have console access to rectify any issues.

To make the configuration reload Netplan with

# netplan apply

To check the bridges use the brctl command:

# brctl show
bridge name	bridge id		STP enabled	interfaces
cloud0		8000.000000000000	no
cloudbr0	8000.52664b74c6a7	no		bond0
cloudbr1	8000.2e13dfd92f96	no		bond1
cloudbr100	8000.02684d6541db	no		bond100

To check the VLANs and bonds:

# cat /proc/net/vlan/config
VLAN Dev name | VLAN ID
Name-Type: VLAN_NAME_TYPE_RAW_PLUS_VID_NO_PAD
bond100 | 100 | bond0
# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth1
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 10
Permanent HW addr: 00:0c:xx:xx:xx:xx
Slave queue ID: 0

Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 10
Permanent HW addr: 00:0c:xx:xx:xx:xx
Slave queue ID: 0

 

OpenVswitch bridge configuration – CentOS

The OpenVswitch version in the standard CentOS repositories is relatively old (version 2.0). To install a newer version either locate and install this from a third party CentOS/Fedora/RedHat repository, alternatively download and compile the packages from the OVS website http://www.openvswitch.org/download/ (notes on how to compile the packages can be found in http://docs.openvswitch.org/en/latest/intro/install/fedora/).

Once packages are available install and enable OVS with

# yum localinstall openvswitch-<version>.rpm
# systemctl start openvswitch
# systemctl enable openvswitch

In addition to this the bridge module should be blacklisted. Experience has shown that even blacklisting this module does not prevent it from being loaded. To force this set the module install to /bin/false. Please note the CloudStack agent install depends on the bridge module being in place, hence this step should be carried out after agent install.

echo "install bridge /bin/false" > /etc/modprobe.d/bridge-blacklist.conf

As with linux bridging above the following examples assumes IPv6 has been disabled and legacy ethX network interface names are used. In addition the hostname has been set in /etc/sysconfig/network and /etc/hosts.

Add the initial OVS bridges using the ovs-vsctl toolset:

# ovs-vsctl add-br cloudbr0
# ovs-vsctl add-br cloudbr1
# ovs-vsctl add-bond cloudbr0 bond0 eth0 eth1
# ovs-vsctl add-bond cloudbr1 bond1 eth2 eth3

This will configure the bridges in the OVS database, but the settings will not be persistent. To make the settings persistent we need to configure the network configuration scripts in /etc/sysconfig/network-scripts/, similar to when using linux bridges.

Each individual network interface has a generic configuration – note there is no reference to bonds at this stage. The following ifcfg-eth script applies to all interfaces:

# vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
TYPE=Ethernet
BOOTPROTO=none
NAME=eth0
ONBOOT=yes
NM_CONTROLLED=no
HOTPLUG=no
HWADDR=00:0C:xx:xx:xx:xx

The bonds reference the interfaces as well as the upstream bridge. In addition the bond configuration specifies the OVS specific settings for the bond (active-backup, no LACP, 100ms status monitoring):

# vi /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBond
OVS_BRIDGE=cloudbr0
BOOTPROTO=none
BOND_IFACES="eth0 eth1"
OVS_OPTIONS="bond_mode=active-backup lacp=off other_config:bond-detect-mode=miimon other_config:bond-miimon-interval=100"
HOTPLUG=no
# vi /etc/sysconfig/network-scripts/ifcfg-bond1
DEVICE=bond1
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBond
OVS_BRIDGE=cloudbr1
BOOTPROTO=none
BOND_IFACES="eth2 eth3"
OVS_OPTIONS="bond_mode=active-backup lacp=off other_config:bond-detect-mode=miimon other_config:bond-miimon-interval=100"
HOTPLUG=no

The bridges are now configured as follows. The host IP address is specified on the untagged cloudbr0 bridge:

# vi /etc/sysconfig/network-scripts/ifcfg-cloudbr0
DEVICE=cloudbr0
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
IPADDR=192.168.100.20
NETMASK=255.255.255.0
GATEWAY=192.168.100.1
HOTPLUG=no

Cloudbr1 is configured without an IP address:

# vi /etc/sysconfig/network-scripts/ifcfg-cloudbr1
DEVICE=cloudbr1
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=none
HOTPLUG=no

Internal bridge cloud0

Under CentOS7.x and CloudStack 4.11 the cloud0 bridge is automatically configured, hence no additional configuration steps required.

Optional tagged interface for storage traffic

If a dedicated VLAN tagged IP interface is required for e.g. storage traffic this is accomplished by creating a VLAN tagged fake bridge on top of one of the cloud bridges. In this case we add it to cloudbr0 with VLAN 100:

# ovs-vsctl add-br cloudbr100 cloudbr0 100
# vi /etc/sysconfig/network-scripts/ifcfg-cloudbr100
DEVICE=cloudbr100
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
IPADDR=10.0.100.20
NETMASK=255.255.255.0
OVS_OPTIONS="cloudbr0 100"
HOTPLUG=no

Additional OVS network settings

To finish off the OVS network configuration specify the hostname, gateway and IPv6 settings:

vim /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=kvmhost1.mylab.local
GATEWAY=192.168.100.1
NETWORKING_IPV6=no
IPV6INIT=no
IPV6_AUTOCONF=no

VLAN problems when using OVS

Kernel versions older than 3.3 had some issues with VLAN traffic propagating between KVM hosts. This has not been observed in CentOS 7.5 (kernel version 3.10) – however if this issue is encountered look up the OVS VLAN splinter workaround.

Network startup

Note – as mentioned for linux bridge networking – once all network startup scripts are in place and the network service is restarted you may lose connectivity to the host if there are any configuration errors in the files, hence make sure you have console access to rectify any issues.

To make the configuration live restart the network service:

# systemctl restart network

To check the bridges use the ovs-vsctl command. The following shows the optional cloudbr100 on VLAN 100:

# ovs-vsctl show
49cba0db-a529-48e3-9f23-4999e27a7f72
    Bridge "cloudbr0";
        Port "cloudbr0";
            Interface "cloudbr0"
                type: internal
        Port "cloudbr100"
            tag: 100
            Interface "cloudbr100"
                type: internal
        Port "bond0"
            Interface "veth0";
            Interface "eth0"
    Bridge "cloudbr1"
        Port "bond1"
            Interface "eth1"
            Interface "veth1"
        Port "cloudbr1"
            Interface "cloudbr1"
                type: internal
    Bridge "cloud0"
        Port "cloud0"
            Interface "cloud0"
                type: internal
    ovs_version: "2.9.2"

The bond status can be checked with the ovs-appctl command:

ovs-appctl bond/show bond0
---- bond0 ----
bond_mode: active-backup
bond may use recirculation: no, Recirc-ID : -1
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
lacp_status: off
active slave mac: 00:0c:xx:xx:xx:xx(eth0)

slave eth0: enabled
active slave
may_enable: true

slave eth1: enabled
may_enable: true

To ensure that only OVS bridges are used also check that linux bridge control returns no bridges:

# brctl show
bridge name	bridge id		STP enabled	interfaces

As a final note – the CloudStack agent also requires the following two lines added to /etc/cloudstack/agent/agent.properties after install:

network.bridge.type=openvswitch
libvirt.vif.driver=com.cloud.hypervisor.kvm.resource.OvsVifDriver

OpenVswitch bridge configuration – Ubuntu

As discussed earlier in this blog post Ubuntu 18.04 introduced Netplan as a replacement to the legacy “/etc/network/interfaces” network configuration. Unfortunately Netplan does not support OVS, hence the first challenge is to revert Ubuntu to the legacy configuration method.

To disable Netplan first of all add “netcfg/do_not_use_netplan=true” to the GRUB_CMDLINE_LINUX option in /etc/default/grub. The following example also shows the use of legacy interface names as well as IPv6 being disabled:

GRUB_CMDLINE_LINUX="net.ifnames=0 biosdevname=0 ipv6.disable=1 netcfg/do_not_use_netplan=true"

Then rebuild GRUB and reboot the server:

grub-mkconfig -o /boot/grub/grub.cfg

To set the hostname first of all edit “/etc/cloud/cloud.cfg” and set this to preserve the system hostname:

preserve_hostname: true

Thereafter set the hostname with hostnamectl:

hostnamectl set-hostname --static --transient --pretty <hostname>

Now remove Netplan, install OVS from the Ubuntu repositories as well the “ifupdown” package to get standard network functionality back:

apt-get purge nplan netplan.io
apt-get install openvswitch-switch
apt-get install ifupdown

As for CentOS we need to blacklist the bridge module to prevent standard bridges being created. Please note the CloudStack agent install depends on the bridge module being in place, hence this step should be carried out after agent install.

echo "install bridge /bin/false" > /etc/modprobe.d/bridge-blacklist.conf

To stop network bridge traffic from traversing IPtables / ARPtables also add the following lines to /etc/sysctl.conf:

# vi /etc/sysctl.conf
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

Same as for CentOS we first of all add the OVS bridges and bonds from command line using the ovs-vsctl command line tools. In this case we also add the additional tagged fake bridge cloudbr100 on VLAN 100:

# ovs-vsctl add-br cloudbr0
# ovs-vsctl add-br cloudbr1
# ovs-vsctl add-bond cloudbr0 bond0 eth0 eth1 bond_mode=active-backup other_config:bond-detect-mode=miimon other_config:bond-miimon-interval=100
# ovs-vsctl add-bond cloudbr1 bond1 eth2 eth3 bond_mode=active-backup other_config:bond-detect-mode=miimon other_config:bond-miimon-interval=100
# ovs-vsctl add-br cloudbr100 cloudbr0 100

As for linux bridge all network configuration is applied in “/etc/network/interfaces”:

# vi /etc/network/interfaces
# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
iface eth0 inet manual
iface eth1 inet manual
iface eth2 inet manual
iface eth3 inet manual

auto cloudbr0
allow-ovs cloudbr0
iface cloudbr0 inet static
  address 192.168.100.20
  netmask 255.255.155.0
  gateway 192.168.100.100
  dns-nameserver 192.168.100.5
  ovs_type OVSBridge
  ovs_ports bond0

allow-cloudbr0 bond0 
iface bond0 inet manual 
  ovs_bridge cloudbr0 
  ovs_type OVSBond 
  ovs_bonds eth0 eth1 
  ovs_option bond_mode=active-backup other_config:miimon=100

auto cloudbr1
allow-ovs cloudbr1
iface cloudbr1 inet manual

allow-cloudbr1 bond1 
iface bond1 inet manual 
  ovs_bridge cloudbr1 
  ovs_type OVSBond 
  ovs_bonds eth2 eth3 
  ovs_option bond_mode=active-backup other_config:miimon=100

Network startup

Since Ubuntu 14.04 the bridges have started automatically without any requirement for additional startup scripts. Since OVS uses the same toolset across both CentOS and Ubuntu the same processes as described earlier in this blog post can be utilised.

# ovs-appctl bond/show bond0
# ovs-vsctl show

To ensure that only OVS bridges are used also check that linux bridge control returns no bridges:

# brctl show
bridge name	bridge id		STP enabled	interfaces

As mentioned earlier the following also needs added to the /etc/cloudstack/agent/agent.properties file:

network.bridge.type=openvswitch
libvirt.vif.driver=com.cloud.hypervisor.kvm.resource.OvsVifDriver

Internal bridge cloud0

In Ubuntu there is no requirement to add additional configuration for the internal cloud0 bridge, CloudStack manages this.

Optional tagged interface for storage traffic

Additional VLAN tagged interfaces are again accomplished by creating a VLAN tagged fake bridge on top of one of the cloud bridges. In this case we add it to cloudbr0 with VLAN 100 at the end of the interfaces file:

# ovs-vsctl add-br cloudbr100 cloudbr0 100
# vi /etc/network/interfaces
auto cloudbr100
allow-cloudbr0 cloudbr100
iface cloudbr100 inet static
  address 10.0.100.20
  netmask 255.255.255.0
  ovs_type OVSIntPort
  ovs_bridge cloudbr0
  ovs_options tag=100

Conclusion

As KVM is becoming more stable and mature, more people are going to start looking at using it rather that the more traditional XenServer or vSphere solutions, and we hope this article will assist in configuring host networking. As always we’re happy to receive feedback , so please get in touch with any comments, questions or suggestions.

About The Author

Dag Sonstebo is  a Cloud Architect at ShapeBlue, The Cloud Specialists. Dag spends most of his time designing, implementing and automating IaaS solutions based on Apache CloudStack.

Working towards CloudStack zero downtime upgrades

,

As most people know, Apache CloudStack has gained a reputation as a solid, low maintenance dependable cloud orchestration platform. That’s why in last year’s Gartner Magic Quadrant so many leaders and challengers were organisations underpinning their services with Apache CloudStack. However, version upgrades – whilst being much simpler than many competing technologies – have always been the pain point for CloudStack operators. The irony is that upgrading CloudStack itself is usually relatively painless, but upgrading its distributed networking Virtual Routers often results in network downtime for users for a number of minutes, requiring user maintenance windows.

At ShapeBlue we have a vision that CloudStack based clouds – whatever their size and complexity – should be able to be upgraded with zero downtime. No maintenance windows, no service interruptions: zero downtime. Achieving this will allow all CloudStack users/operators to benefit from the vast array of new functionality being added by the CloudStack community in every release.

We set out on the journey towards zero downtime a number of months ago and have been working hard with the CloudStack community on the first steps (it is important to note that “we” includes many people in the CloudStack community who have contributed to this work). Below, I set out the detail of what we’ve achieved so far and what we hope to be achieving in the future, but if readers just want the headline: CloudStack 4.11.1 has up to an 80%+ reduction in network downtime during upgrades compared to CloudStack 4.9.3, and downtime is near eliminated when using redundant VRs.

What’s the problem when upgrading?

During upgrades, CloudStack’s virtual routers (VRs) have to be restarted and usually destroyed and recreated (this also sometimes has to be done during day-to-day operations, but is most apparent during upgrades). These restarts usually lead to downtime for users – in some cases up to several minutes. Whilst Redundant Virtual Routers can mitigate against this they do have some limitations with regards to backward compatibility and are therefore not always a solution to the problem.

Downtime reductions in CloudStack 4.11.1

With the changes made in 4.11.1 (described below) we have managed to achieve significant reductions in network downtime during VR restarts. Please note these improvements will vary from one environment to another, and will be dependent on hypervisor type, hypervisor version, storage backend as well as network bandwidth, so we suggest testing in your own environment to determine the benefits. We’ve tested with a typical VR configuration in our virtualised lab environment.

The testing setup used is as follows:

  • CloudStack 4.9.3 and 4.11.1 environments built in parallel. To maintain the same hypervisor versions across both tests the following hypervisor versions were used:
    • VMware vSphere 5.5
    • KVM on CentOS7
    • XenServer 7.0
  • Environment configuration: In each test we build a simple isolated network with:
    • 10 VMs
    • 10 IP addresses
    • Firewall rules configured on all IP addresses

Downtime was measured as follows:

  • For egress traffic we measured the total amount of time an outbound ping would fail from a hosted VM during the restart process.
  • For ingress traffic we assumed a hosted service on a CloudStack VM and measured the amount of time SSH would be unavailable for during the restart process.
  • In all tests we carried out a “restart network with cleanup” operation and measured the above times. Note – with the new parallel VR restart process (see below) we no longer care how long the overall process takes – we are only interested in how long the network is impacted for. As a result we’ve simply measured the sum of time services were unavailable for (note this time may in some cases be a sum of multiple downtime periods).
  • Tests were repeated multiple times and average number of seconds calculated for ingress and egress downtime across tests for each hypervisor. To illustrate our best case scenarios we’ve also included the shortest measured downtime figure.

Results are as follows:

EnvironmentACS 4.9.3 avgACS 4.11.1 avg (lowest)Reduction avg (highest)
VMware 5.5119s21s (12s)
82% (90%)
KVM / CentOS744s26s (9s)40% (80%)
XenServer 7.0181s33s (15s)82% (92%)

 

How these results were achieved

Existing improvements made in CloudStack 4.11

A number of changes were made in CloudStack 4.11 designed to improve VR restart performance:

  • The system VM has been upgraded from Debian 7 (init based) to Debian 9 (systemd based)
  • The patching process and boot times have been improved, and we have also eliminated reboots after patching
  • The system VM disk size has been reduced, leading to faster deployment time.
  • The VPN backend in the VR has been upgraded to Strongswan, which provides improved VPN performance
  • The redundant VR (RVR) mechanisms have been improved.
  • Code base has been refactored, and it is now easier to build and maintain VR code
  • A number of stability improvements made

 Changes  in CloudStack 4.11.1 – Parallel VR restarts

CloudStack 4.11.1 will ship with a new feature: Parallel VR Restarts, which  changes the behaviour of the “restart network with cleanup” option. In previous CloudStack versions this method would be a serial action where the original VR would be stopped and destroyed and then a new VR started. In CloudStack 4.11.1 this has now been changed to a parallel process where a “restart with cleanup” means:

  • A new VR is started in the background while the old one is still running and providing networking services.
  • Once the new VR is up and has checked in to CloudStack management the old VR is simply stopped and destroyed.
  • This is followed by a last configuration step where ARP caches at neighbours are updated.

With this method there is no negotiation between old and new VR, CloudStack simply orchestrates the parallel startup of the new VR. As a result this method does not have any pre-requisites around the version of the original VR – meaning it can be used for VR restarts after upgrade from considerable older CloudStack versions to 4.11.1.

It is worth noting that this 4.11.1 feature does not make large reductions in the actual  VR processing time itself – however with the parallel startup this doesn’t affect network downtime, and the downtime itself is more connected to the final handover of network processing from old to new VR.

In addition to the considerable reduction in normal VR restart downtime, this feature also introduces a much improved redundant VR restart – this comes close to eliminating network downtime when redundant VR networks are restarted, but does obviously mean the old and new VRs need to be version compatible. In our own testing we have seen downtime for redundant VR networks near eliminated.

Coming in future versions

Advanced  parallel restarts

The next step on the journey is to add further handshaking between old and new VR:

  • New VR will be started in parallel to old, but with some network services and / or network interfaces disabled.
  • Once new VR is up CloudStack management will do an external handover from old VR to new, i.e. handle VR connectivity via the hypervisor.

Fully negotiated redundant VR restarts

The last step on the journey will be aiming towards a fully redundant handover from old to new VR:

  • In this final step the end goal is to make all VRs redundant capable, which will reduce same version restart times as well as future upgrade restart times.
  • New VR will again be started in parallel to old, but will be configured with the redundancy options currently used in the RVR.
  • Once new VR is up the old and new VRs will internally negotiate the handover of all networking connectivity and services, before the old VR is shut down.

– – –

During this journey there are a number of tasks needing carried out – both to make the VR internal processing more efficient as well as improving the backend network restart mechanisms:

  • General speedup of IPtables rules application
  • Fix and improvement of the DNS / DHCP configuration to eliminate repetition of processing steps and cut down on processing time
  • Further improvements of redundant Virtual Router: VRRP2 configuration, and/ or move to a different VR HA solution
  • A move to make all VRs redundant capable by default
  • Move from python2 to python3
  • Consider a move from IPtables to NFtables
  • Converge and make network topologies flexible, refactor and merge VPC and non-VPC code base

Conclusion

With the changes implemented in 4.11.1 we have already made a huge step forward in reducing network downtime as part of VR restarts – whether this is during day to day operation or as part of a CloudStack version upgrade. With downtime reduced by up to 80% and average figures of less than 30 seconds this is a considerable improvement – and this is only the first step on the journey.

We have not yet achieved our goal of “zero downtime upgrades” but it is worth considering that the network interruptions that CloudStack can now achieve during an upgrade will be less than the timeouts for many applications and protocols.

In the coming CloudStack versions we hope to continue this development and further reduce the figures, working towards the ultimate goal of “zero downtime upgrades”. 

About The Author

Dag Sonstebo is a Cloud Architect at ShapeBlue, The Cloud Specialists. Dag spends his time designing, implementing and automating IaaS solutions based around Apache CloudStack.

 

Layer 2 Networks in CloudStack

,

Layer 2 (also known as the Data Link Layer) is the second in the seven-layer OSI reference model for network protocol design, and is equivalent to the link layer (the lowest layer) in the TCP/IP network model. Layer 2 is the network layer used to transfer data between adjacent network nodes in a wide area network, or between nodes on the same local area network.

We have recently introduced a new Layer 2 network offering in CloudStack. This enables all network services to be provided from network equipment external to CloudStack. As a result, the Virtual Router is not required, and VMs deployed to the network will get their IP addresses and other network services from these external appliances. This new network type is acting like a shared network, but it leaves all network services to be managed outside of CloudStack.

This article describes how to set up a Layer 2 network in the upcoming 4.11 CloudStack release.

The network offering

Following CloudStack’s architecture, there must first be a specific network offering enabled in order to deploy a L2 network. Such an offering can be created by selecting ‘Guest Type: L2’ (it also supports the ability to specify VLAN). When selected, the user will need to specify a VLAN id when creating the network, and the network will be implemented using a dynamically assigned VLAN. Otherwise, the administrator will have to manage this externally from CloudStack, by appliance or other VM. Once created, the network offering needs to be enabled in order to implement the L2 network.

 

The network

CloudStack Administrators are able to create Layer 2 networks simply by clicking ‘Network’, where a new button is introduced: ‘Add L2 Guest Network’. By clicking this button, a CloudStack administrator will have the ability to fill in the network specifics to implement it. Please note, that where the network offering requires a VLAN, CloudStack administrators must specify a particular VLAN id. When all details are complete the network can be saved by clicking ‘OK’. At this stage our Layer 2 network is ready to use and we can start deploying VMs in it.

Add network button:

Add network pop-up:

How it looks

As soon as we get our VMs into ‘Running’ state, we can see on the NICs section that there are no network details available (such as IP Address, gateway, etc.), but the VM is reachable and available through the Console Proxy VM. Since we don’t have any of these services, our network is running without a Virtual Router, which makes the deployment fast.

On the physical network side, the cloud administrators needs to setup all the resources managing the network services, since they are now external to CloudStack. They need to setup routers/switches, DHCP, additional load balancers and etc. Please note that once setup, all services are managed externally, and therefore CloudStack cannot handle IP assignments, port forwarding or other networking services that the Virtual Router normally does. Layer 2 type networks can only be managed by cloud administrators.

About the author

Boris Stoyanov is Software Engineer in testing at ShapeBlue, The Cloud Specialists. Bobby spends his time testing features for the Apache CloudStack Community and for our ShapeBlue clients.

A Beginners Guide To CloudStack Networking

, , , , ,

Many people find it challenging to get started with CloudStack’s networking. There are some basic concepts, which although not overly complicated, are not especially obvious either. This blog will try to explain these underlying concepts, in order to make getting started with CloudStack networking models much easier.

Networking KVM for CloudStack

, , ,

Introduction

KVM hypervisor networking for CloudStack can sometimes be a challenge, considering KVM doesn’t quite have the matured guest networking model found in the likes of VMware vSphere and Citrix XenServer. In this blog post we’re looking at the options for networking KVM hosts using bridges and VLANs, and dive a bit deeper into the configuration for these options. Installation of the hypervisor and CloudStack agent is pretty well covered in the CloudStack installation guide, so we’ll not spend too much time on this.

Network bridges

On a linux KVM host guest networking is accomplished using network bridges. These are similar to vSwitches on a VMware ESXi host or networks on a XenServer host (in fact networking on a XenServer host is also accomplished using bridges).

A linux network bridge is a Layer-2 software device which allows traffic to be forwarded between ports internally on the bridge and the physical network uplinks. The traffic flow is controlled by MAC address tables maintained by the bridge itself, which determine which hosts are connected to which bridge port. The bridges allows for traffic segregation using traditional Layer-2 VLANs as well as SDN Layer-3 overlay networks.

KVMnetworking41

Linux bridges vs OpenVswitch

The bridging on a KVM host can be accomplished using traditional linux bridge networking or by adopting an OpenVswitch back end. Traditional linux bridges have been implemented in the linux kernel since version 2.2, and have been maintained through the 2.4 and 2.6 kernels. Linux bridges provide all the basic Layer-2 networking required for a KVM hypervisor back end, but it lacks some automation options and is configured on a per host basis.

OpenVswitch was developed to address this, and provides additional automation in addition to new networking capabilities like Software Defined Networking (SDN). OpenVswitch allows for centralised control and distribution across physical hypervisor hosts, similar to distributed vSwitches in VMware vSphere. Distributed switch control does require additional controller infrastructure like OpenDaylight, Nicira, VMware NSX, etc. – which we won’t cover in this article as it’s not a requirement for CloudStack.

It is also worth noting Citrix started using the OpenVswitch backend in XenServer 6.0.

Network configuration overview

For this example we will configure the following networking model, assuming a linux host with four network interfaces which are bonded for resilience. We also assume all switch ports are trunk ports:

  • Network interfaces eth0 + eth1 are bonded as bond0.
  • Network interfaces eth1 + eth2 are bonded as bond1.
  • Bond0 provides the physical uplink for the bridge “cloudbr0”. This bridge carries the untagged host network interface / IP address, and will also be used for the VLAN tagged guest networks.
  • Bond1 provides the physical uplink for the bridge “cloudbr1”. This bridge handles the VLAN untagged public traffic.

 

The CloudStack zone networks will then be configured as follows:

  • Management and guest traffic is configured to use KVM traffic label “cloudbr0”.
  • Public traffic is configured to use KVM traffic label “cloudbr1”.

 

In addition to the above it’s important to remember CloudStack itself require internal connectivity from the hypervisor host to system VMs (Virtual Routers, SSVM and CPVM) over the link local 169.254.0.0/16 subnet. This is done over a host-only bridge “cloud0”, which is created by CloudStack when the host is added to a CloudStack zone.

 

KVMnetworking42

Linux bridge configuration

CentOS

In CentOS the linux bridge configuration is done with configuration files in /etc/sysconfig/network-scripts.

Each of the four individual NIC interfaces are configured as follows (eth0 / eth1 / eth2 / eth3 are all configured the same way):

# vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
TYPE=Ethernet
USERCTL=no
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
HWADDR=00:0C:12:xx:xx:xx
NM_CONTROLLED=no
ONBOOT=yes

The bond configurations are specificied in the equivalent ifcfg-bond scripts and specify bonding options as well as the upstream bridge name. In this case we’re just setting a basic active-passive bond (mode=1) with status monitoring every 100ms (miimon=100):

# vi /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
ONBOOT=yes
BONDING_OPTS='mode=1 miimon=100'
BRIDGE=cloudbr0
NM_CONTROLLED=no

The same goes for bond1:

# vi /etc/sysconfig/network-scripts/ifcfg-bond1
DEVICE=bond1
ONBOOT=yes
BONDING_OPTS='mode=1 miimon=100'
BRIDGE=cloudbr1
NM_CONTROLLED=no

Cloudbr0 is configured in the ifcfg-cloudbr0 script. In addition to the bridge configuration we also specify the host IP address, which is tied directly to the bridge since it is on an untagged VLAN:

# vi /etc/sysconfig/network-scripts/ifcfg-cloudbr0
DEVICE=cloudbr0
ONBOOT=yes
TYPE=Bridge
IPADDR=192.168.100.20
NETMASK=255.255.255.0
GATEWAY=192.168.100.1
NM_CONTROLLED=no
DELAY=0

Cloudbr1 does not have an IP address configured hence the the configuration is simpler:

# vi /etc/sysconfig/network-scripts/ifcfg-cloudbr1
DEVICE=cloudbr1
ONBOOT=yes
TYPE=Bridge
NM_CONTROLLED=no
DELAY=0

Optional tagged interface for storage traffic

If a dedicated VLAN tagged IP interface is required for e.g. storage traffic this can be accomplished by created a VLAN tagged bond and tying this to a dedicated bridge. In this case we create a new bridge on bond0 using VLAN 100:

# vi /etc/sysconfig/network-scripts/ifcfg-bond.100
DEVICE=bond0.100
VLAN=yes
BOOTPROTO=static
ONBOOT=yes
TYPE=Unknown
BRIDGE=cloudbr100

The bridge can now be configured with the desired IP address for storage connectivity:

# vi /etc/sysconfig/network-scripts/ifcfg-cloudbr100
DEVICE=cloudbr100
ONBOOT=yes
TYPE=Bridge
VLAN=yes
IPADDR=10.0.100.20
NETMASK=255.255.255.0
NM_CONTROLLED=no
DELAY=0

Internal bridge cloud0

When using linux bridge networking there is no requirement to configure the internal “cloud0” bridge, this is all handled by CloudStack.

Network startup

Note – once all network startup scripts are in place and the network service is restarted you may lose connectivity to the host if there are any configuration errors in the files, hence make sure you have console access to rectify any issues.

To make the configuration live restart the network service:

# service network restart

To check the bridges use the brctl command:

# brctl show
bridge name bridge id STP enabled interfaces
cloudbr0 8000.000c29b55932 no bond0
cloudbr1 8000.000c29b45956 no bond1

The bonds can be checked with:

# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0c:xx:xx:xx:xx
Slave queue ID: 0

Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0c:xx:xx:xx:xx
Slave queue ID: 0

Ubuntu

To use bonding and linux bridge networking in Ubuntu first install the following:

# apt-get install ifenslave-2.6 bridge-utils

Also add the bonding and bridge modules to the kernel modules to be loaded at boot time:

# vi /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
loop
lp
rtc
bonding
bridge

Before continuing also make sure the correct hostname and FQDN are set in /etc/hostname and /etc/hosts respectively. Also add the following lines to /etc/sysctl.conf:

# vi /etc/sysctl.conf
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

All interface, bond and bridge configuration are configured in /etc/network/interfaces. Same as for CentOS we are configuring basic active-passive bonds (mode=1) with status monitoring every 100ms (miimon=100), and configuring bridges on top of these. As before the host IP address is tied to cloudbr0:

# vi /etc/network/interfaces
# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet manual
bond-master bond0

auto eth1
iface eth1 inet manual
bond-master bond0

auto eth2
iface eth2 inet manual
bond-master bond1

auto eth3
iface eth3 inet manual
bond-master bond1

auto bond0
iface bond0 inet manual
bond-mode active-backup
bond-miimon 100
bond-slaves none

auto bond1
iface bond1 inet manual
bond-mode active-backup
bond-miimon 100
bond-slaves none

auto cloudbr0
iface cloudbr0 inet static
address 192.168.100.20
gateway 192.168.100.1
netmask 255.255.255.0
dns-nameservers 8.8.8.8 8.8.4.4
dns-domain mynet.local
bridge_ports bond0
bridge_fd 5
bridge_stp off
bridge_maxwait 1

auto cloudbr1
iface cloudbr1 inet manual
bridge_ports bond1
bridge_fd 5
bridge_stp off
bridge_maxwait 1

Optional tagged interface for storage traffic

Dedicated VLAN tagged IP interface for e.g. storage traffic is again accomplished by creating a VLAN tagged bond and tying this to a dedicated bridge. As above we add the following to /etc/network/interfaces  to create a new bridge on bond0 using VLAN 100:

auto bond0.100
iface bond0.100 inet manual
vlan-raw-device bond0

auto cloudbr100
iface cloudbr100 inet static
address 10.0.100.20
netmask 255.255.255.0
bridge_ports bond0.100
bridge_fd 5
bridge_stp off
bridge_maxwait 1

Internal bridge cloud0

When using linux bridge networking the internal “cloud0” bridge is again handled by CloudStack, i.e. there’s no need for specific configuration to be specified for this.

Network startup

Note – once all network startup scripts are in place and the network service is restarted you may lose connectivity to the host if there are any configuration errors in the files, hence make sure you have console access to rectify any issues.

To make the configuration live restart the network service:

# service networking restart

To check the bridges use the brctl command:

# brctl show
bridge name bridge id STP enabled interfaces
cloudbr0 8000.000c29b43c4d no bond0
cloudbr1 8000.000c29b43c61 no bond1
cloudbr100 8000.000c29b43c4d no bond0.100

To check the VLANs and bonds:

# cat /proc/net/vlan/config
VLAN Dev name | VLAN ID
Name-Type: VLAN_NAME_TYPE_RAW_PLUS_VID_NO_PAD
bond0.100 | 100 | bond0
# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth1
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 10
Permanent HW addr: 00:0c:xx:xx:xx:xx
Slave queue ID: 0

Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 10
Permanent HW addr: 00:0c:xx:xx:xx:xx
Slave queue ID: 0

 

OpenVswitch bridge configuration

CentOS

To configure OVS bridges in CentOS we first of all add the bridges using the ovs-vctl command, then add bonds to these:

# ovs-vsctl add-br cloudbr0
# ovs-vsctl add-br cloudbr1
# ovs-vsctl add-bond cloudbr0 bond0 eth0 eth1
# ovs-vsctl add-bond cloudbr1 bond1 eth2 eth3

This will configure the bridges in the OVS database, but the settings will not be persistent. To make the settings persistent we need to configure the network configuration scripts in /etc/sysconfig/network-scripts/, similar to when using linux bridges.

Each individual network interface has a generic configuration – note there is no reference to bonds at this stage. The following ifcfg-eth script applies to all interfaces:

# vi /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
HWADDR=00:0C:xx:xx:xx:xx
HOTPLUG=no

The bonds reference the interfaces as well as the upstream bridge. In addition the bond configuration specifies the OVS specific settings for the bond (active-backup, no LACP, 100ms status monitoring):

# vi /etc/sysconfig/network-scripts/ifcfg-bond0
DEVICE=bond0
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBond
OVS_BRIDGE=cloudbr0
BOOTPROTO=none
BOND_IFACES="eth0 eth1"
OVS_OPTIONS="bond_mode=active-backup lacp=off other_config:bond-detect-mode=miimon other_config:bond-miimon-interval=100"
HOTPLUG=no
# vi /etc/sysconfig/network-scripts/ifcfg-bond1
DEVICE=bond1
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBond
OVS_BRIDGE=cloudbr1
BOOTPROTO=none
BOND_IFACES="eth2 eth3"
OVS_OPTIONS="bond_mode=active-backup lacp=off other_config:bond-detect-mode=miimon other_config:bond-miimon-interval=100"
HOTPLUG=no

The bridges are now configured as follows. The host IP address is specified on the untagged cloudbr0 bridge:

# vi /etc/sysconfig/network-scripts/ifcfg-cloudbr0
DEVICE=cloudbr0
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
IPADDR=192.168.100.20
NETMASK=255.255.255.0
HOTPLUG=no

Cloudbr1 is configured without an IP address:

# vi /etc/sysconfig/network-scripts/ifcfg-cloudbr1
DEVICE=cloudbr1
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=none
HOTPLUG=no

Internal bridge cloud0

In addition to the above we also need to configure the internal only cloud0 bridge. This is required when using OVS bridging only, i.e. when the network bridge kernel module has been disabled. If the module is enabled CloudStack will configure the internal bridge using linux bridge, whilst allowing all other bridges to be configured using OVS. Note the CloudStack agent will create this bridge, hence there is no need to configure it using the ovs-vsctl command.
Since there is no routing involved for the internal bridge we simply configure this with IP address 169.254.0.1/16:

# vi /etc/sysconfig/network-scripts/ifcfg-cloud0
DEVICE=cloud0
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
HOTPLUG=no
IPADDR=169.254.0.1
NETMASK=255.255.0.0

Optional tagged interface for storage traffic

If a dedicated VLAN tagged IP interface is required for e.g. storage traffic this is accomplished by creating a VLAN tagged fake bridge on top of one of the cloud bridges. In this case we add it to cloudbr0 with VLAN 100:

# ovs-vsctl add-br cloudbr100 cloudbr0 100
# vi /etc/sysconfig/network-scripts/ifcfg-cloudbr100
DEVICE=cloudbr100
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
IPADDR=10.0.100.20
NETMASK=255.255.255.0
HOTPLUG=no

Additional OVS network settings

To finish off the OVS network configuration specify the hostname, gateway and IPv6 settings:

vim /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=kvmhost1.mylab.local
GATEWAY=192.168.100.1
NETWORKING_IPV6=no
IPV6INIT=no
IPV6_AUTOCONF=no

VLAN problems when using OVS

Due to bugs in legacy network interface drivers there are in certain circumstances issues with getting VLAN traffic to propagate between KVM hosts. This is a known issue, and the OpenVswitch VLAN FAQ is a useful place to start any troubleshooting.

One workaround for this issue is to configure the “VLAN splinters” setting on the network interfaces. This is accomplished with the following command:

ovs-vsctl set interface eth0 other-config:enable-vlan-splinters=true

The problem arises when trying to make this setting persistent across reboots, as the command can not be ran as part of the normal ifcfg-eth script.

One way to make this persistent is to add the following lines to the end of the ifup-ovs script:

# vi /etc/sysconfig/network-scripts/ifup-ovs
if [ -x /sbin/ifup-localovs ]; then
/sbin/ifup-localovs
fi

Then add the commands to a new file ifup-localovs:

# vi /sbin/ifup-localovs
#!/bin/sh
ovs-vsctl set interface eth0 other-config:enable-vlan-splinters=true
ovs-vsctl set interface eth1 other-config:enable-vlan-splinters=true
ovs-vsctl set interface eth2 other-config:enable-vlan-splinters=true
ovs-vsctl set interface eth3 other-config:enable-vlan-splinters=true

This will ensure the settings are applied to each network interface at every reboot and after each network service restart.

Network startup

Note – as mentioned for linux bridge networking – once all network startup scripts are in place and the network service is restarted you may lose connectivity to the host if there are any configuration errors in the files, hence make sure you have console access to rectify any issues.

To make the configuration live restart the network service:

# service network restart

To check the bridges use the ovs-vsctl command. The following shows the optional cloudbr100 on VLAN 100:

# ovs-vsctl show
27daed4e-52f3-4177-9827-550f0e7df452
Bridge "cloudbr1"
Port "vnet2"
Interface "vnet2"
Port "bond1"
Interface "eth3"
Interface "eth2"
Port "cloudbr1"
Interface "cloudbr1"
type: internal
Bridge "cloud0"
Port "cloud0"
Interface "cloud0"
type: internal
Port "vnet0"
Interface "vnet0"
Bridge "cloudbr0"
Port "cloudbr0"
Interface "cloudbr0"
type: internal
Port "vnet3"
Interface "vnet3"
Port "vnet1"
Interface "vnet1"
Port "cloudbr100"
tag: 100
Interface "cloudbr100"
type: internal
Port "bond0"
Interface "eth1"
Interface "eth0"
ovs_version: "2.3.1"

The bond status can be checked with the ovs-appctl command:

ovs-appctl bond/show bond0
---- bond0 ----
bond_mode: active-backup
bond may use recirculation: no, Recirc-ID : -1
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
lacp_status: off
active slave mac: 00:0c:xx:xx:xx:xx(eth0)

slave eth0: enabled
active slave
may_enable: true

slave eth1: enabled
may_enable: true

Ubuntu

First of all install the bonding kernel module and make sure this is added to /etc/modules such that it is loaded on startup:

# apt-get install ifenslave-2.6
# vi /etc/modules
# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

loop
lp
rtc
bonding

To ensure the linux bridge module isn’t loaded at boot time blacklist this module in /etc/modprobe.d/blacklist.conf:

# vim /etc/modprobe.d/blacklist.conf

Add the following line to the end of the file:

blacklist bridge

Ensure the correct hostname and FQDN are set in /etc/hostname and /etc/hosts respectively, and add the following lines to /etc/sysctl.conf:

# vi /etc/sysctl.conf
net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0

Same as for CentOS we first of all add the OVS bridges and bonds from command line using the ovs-vsctl command line tools:

# ovs-vsctl add-br cloudbr0
# ovs-vsctl add-br cloudbr1
# ovs-vsctl add-bond cloudbr0 bond0 eth0 eth1 bond_mode=active-backup other_config:bond-detect-mode=miimon other_config:bond-miimon-interval=100
# ovs-vsctl add-bond cloudbr1 bond1 eth2 eth3 bond_mode=active-backup other_config:bond-detect-mode=miimon other_config:bond-miimon-interval=100

As for linux bridge all network configuration is applied in /etc/network/interfaces:

# vi /etc/network/interfaces

Note in the following example all individual network interface configuration as commented out – which works in Ubuntu 14.04. For Ubuntu 12.04 the individual interface sections are required.

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
#auto eth0
#iface eth0 inet manual

#auto eth1
#iface eth1 inet manual

#auto eth2
#iface eth2 inet manual

#auto eth3
#iface eth3 inet manual

allow-cloudbr0 bond0
iface bond0 inet manual
ovs_bridge cloudbr0
ovs_type OVSBond
ovs_bonds eth0 eth1
ovs_options bond_mode=active-backup other_config:miimon=100

auto cloudbr0
allow-ovs cloudbr0
iface cloudbr0 inet static
address 172.16.152.39
netmask 255.255.255.0
gateway 172.16.152.2
dns-nameservers 8.8.8.8
dns-search mbplab.local
ovs_type OVSBridge
ovs_ports bond0

allow-cloudbr1 bond1
iface bond1 inet manual
ovs_bridge cloudbr1
ovs_type OVSBond
ovs_bonds eth2 eth3
ovs_options bond_mode=active-backup other_config:miimon=100

allow-ovs cloudbr1
iface cloudbr1 inet manual
ovs_type OVSBridge
ovs_ports bond1

Bridge startup in Ubuntu 12.04

The OpenVswitch implementation in Ubuntu 12.04 is slightly lacking compared to 14.04 and later. One thing which is missing is startup scripts for bringing the OVS bridges online. This can be accomplished by adding a custom bridge startup script similar to the following:

# openvswitch-bridgestartup
#
# Starts OVS bridges at startup
#
### BEGIN INIT INFO
# Provides: openvswitch-bridgestartup
# Required-Start: openvswitch-switch
# Required-Stop:
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Brings OVS bridges online at boot time
### END INIT INFO

IFCFG="/etc/network/interfaces"

start() {
BRIDGES=$(ovs-vsctl list-br)
for BRIDGE in $BRIDGES; do
if grep -q -w "$BRIDGE" $IFCFG; then
ifup "$BRIDGE"
fi
done
}

case $1 in
start)
start
;;
stop)
;;
*)
echo "Usage: $0 {start}"
exit 2
;;
esac

Configure the service as follows:

# update-rc.d openvswitch-bridges defaults 21

Internal bridge cloud0

In Ubuntu there is no requirement to add additional configuration for the internal cloud0 bridge, CloudStack manages this.

Optional tagged interface for storage traffic

Additional VLAN tagged interfaces are again accomplished by creating a VLAN tagged fake bridge on top of one of the cloud bridges. In this case we add it to cloudbr0 with VLAN 100:

# ovs-vsctl add-br cloudbr100 cloudbr0 100
# vi /etc/sysconfig/network-scripts/ifcfg-cloudbr100

Conclusion

As KVM is becoming more stable and mature, more people are going to start looking at using it rather that the more traditional XenServer or vSphere solutions, and we hope this article will assist in configuring host networking. As always we’re happy to receive feedback , so please get in touch with any comments, questions or suggestions.

About The Author

Dag Sonstebo is  a Cloud Architect at ShapeBlue, The Cloud Specialists. Dag spends most of his time designing, implementing and automating IaaS solutions based on on Apache CloudStack.

Understanding CloudStack’s Physical Networking Architecture

, , , ,

 

Understanding and configuring the physical connections of a host in a CloudStack deployment can at first be very confusing. While Software Defined Networking (SDN) is set to greatly simplify some aspects, its integration within CloudStack is not fully mature yet and it won’t be the right solution for everyone.

In this article, Paul Angus, Cloud Architect at ShapeBlue, the cloud specialists, unravels some of the areas which can cause confusion in CloudStack’s physical networking architecture.

Let’s Get Physical: Physical vs. Logical Hosts

One source of confusion is that when people refer to a ‘host’ in the CloudStack world they could be referring to two different things – the physical host and the ‘host’ that CloudStack communicates with. The physical host is understandably the physical box with processors, memory and network interfaces etc. The host CloudStack communicates with (over the management network) is the hypervisor within that physical host. So as an example, public traffic needs to be connected to the physical host (so the system VMs can connect out) but not to the logical host (the hypervisor).

The first determining factor in your physical network topology will be the type of zone you will be implementing. Let’s consider CloudStack’s ‘basic’ and ‘advanced’ zones from a physical networking point of view.

Basic Zones

In a basic zone you only have one ‘physical network’ because you have no VLAN separation, however you can still split traffic across multiple physical NICs (more on this later)

Basic zones can also only have one guest network and no public network.

 

Advanced Zones

In an advanced zone CloudStack allows for a public network and the creation of multiple guest networks, both physically and logically (usually using tagged VLANs).

Use cases for additional physical guest networks could be to create physically separate links from the hosts to a non-CloudStack network segment in a datacentre – for instance an MPLS network

Other use cases could be to enable dedicated guest networks which are only used by certain domains or accounts, or to provide higher performing guest networks, for example 10Gb links vs 1Gb links.

 

Network Labels

The most important part of this configuration are the network labels. For an ESXi host they refer to the names of the vSwitches on the hosts, for KVM hosts they refer to the bridges which you will have created and for XenServer hosts they simply refer to the network labels on the host for each of the interfaces or bonds.

The network labels tell CloudStack which physical network interfaces in the host to connect the various virtual interfaces on the guest and system instances to. The SSVM for instance has interfaces on the public, private, storage and management networks and it’s important that these are connected to the correct virtual networks within the hosts, which in turn connect to the physical networks. The actual names/labels can be anything you like – but it’s always good practice to call them something which reminds you which one is which.

Storage Traffic

Of the different CloudStack traffic types, storage traffic always causes the most confusion. The terms ‘storage traffic’ and ‘storage network’ are referring to secondary storage traffic, snapshots (backups), ISOs and templates are all transferred to/from secondary storage over this network.

By default, CloudStack primary and secondary storage traffic travels over the management network. Secondary storage traffic can be configured to travel over a separate network (the storage network) to the management network (there are ways to separate primary storage traffic at the hypervisor level – but that’s another blog).

The main benefits in separating secondary storage and management traffic (and therefore primary storage from secondary storage traffic) are seen when creating a guest instance, as the template is copied from secondary storage to primary storage and when taking a snapshot, as the disk image is copied from primary storage to secondary storage.

Basic Network Traffic

In a ‘basic’ network all of the traffic would go over the same interface(s) and each traffic type would have the same network label. This is not recommended.

 

 

 

The management, storage and guest traffic can be separated by having separate physical interfaces for them on the hosts and referencing them with different labels. You could now either have them connected to the same switch, different switches or the same switch but on different untagged VLANs. Connecting them to the same switch gives additional throughput to/from the hosts.

 

Connecting the interfaces to different VLANs (configured at the switch level) [figure 6] gives additional throughput and security as the guest traffic can be kept separated from the management and storage traffic. It’s considered best practice to have the guest traffic going through a separate interface on the hosts to the management traffic in a basic zone configuration – requiring a minimum of two interfaces per host.

 

Advanced Network Traffic

In an advanced network CloudStack can separate traffic logically through the use of VLAN tagging. In some ways separation is easier to configure than in basic networking, as you simply need to set the ports on your switches to trunk all of the VLANs you are using, en-masse.

However you would still want to separate traffic physically for increased throughput and potentially increased security. Again this is done through mapping the network label for each physical interface you will be using to the traffic type you wish to travel through that interface.

Inside the Host

Bringing this all together, a combined logical and physical diagram of the networking of an advanced zone host, which has two network interfaces, would look like this:

 

 

 

Summary

This article has explained the key concepts and terminology, essential to configuring physical networking within a CloudStack deployment for both ‘basic’ and ‘advanced’ zones. In particular this article has demonstrated how the various CloudStack traffic types travel within the physical network.

Look out for a future accompanying article covering the separation of primary storage traffic from other CloudStack traffic.

About the author

Paul Angus is a Senior Consultant & Cloud Architect at ShapeBlue, The Cloud Specialists. He has designed numerous CloudStack environments for customers across 4 continents, based on Apache Cloudstack and Citrix Cloudplatform.

When not building Clouds, Paul likes to create scripts that build clouds……..and he very occasionally can be seen trying to hit a golf ball.

 

CloudStack Networking Considerations

,

With the Acton release of CloudStack, the choices for networking were significantly enhanced. CloudStack 3.x now has more options than ever before which must be fully understood if the platform is to be correctly leveraged to build public or private cloud.

Geoff Higginbottom, CTO of ShapeBlue the strategic cloud consultancy gives an overview of the networking models in CloudStack and highlights the key networking decisions to be made when designing a cloud around CloudStack.

CloudStack has two Networking models, Basic and Advanced, which one you use will depend on your requirements and what you are using your cloud for. As a CloudStack deployment consists of multiple Availability Zones, PODs and Clusters (see https://www.shapeblue.com/2012/03/16/cloudstack-architecture-overview/ for more info) then you can have Zones running a Basic Network architecture, and Zones running an Advanced Network architecture, all within the same Cloud, and all managed by CloudStack.

Network Traffic Types

There are four types of network traffic within CloudStack:

Guest: When end users run VMs or Guest Instances as CloudStack refers to them, they generate guest traffic. The guest VMs communicate with each other over a network that can be referred to as the guest network.

Management: When CloudStack’s internal resources communicate with each other, they generate management traffic. This includes communication between Hosts, System VMs (VMs used by CloudStack to perform various tasks in the cloud), and any other component that communicates directly with the CloudStack Management Server.

Storage: Traffic between Primary and Secondary storage servers, such as VM templates and Snapshots.

Public: Public traffic is generated when VMs in the cloud access the Internet. Publicly accessible IPs must be allocated for this purpose. End users can use the CloudStack UI to acquire IPs to implement NAT and Load Balancing between their guest network and the public network.

Guest, Management and Storage are common to Basic and Advanced Networking, Public Traffic is only present in an Advanced Network configuration.

Basic Networking

Advanced Networking

It is highly recommended to assign the different traffic types to different dedicated NICs on the Hypervisor, and also to use NIC Bonding where possible.

IP Addressing Schemas

Basic Networking

In a Basic Networking setup, the Guest Instances use the same IP range as the underlying CloudStack and Hypervisor architecture. When designing your IP Schema, allow sufficient IP Addresses to cater for all of your CloudStack infrastructure including the Guest Instances. Consider using an IP Range such as 192.168.0.0/20 for each Zone, as this will provide over 4,000 addresses IPs per Zone. Warning:, If you use a /16 subnet for your first Zone, there would be no more Private IP Addresses left for additional Zones!

IP addresses will need to be reserved for each POD, and a Guest IP Range assigned during the initial configuration of the Zone. Every Host, System VM and Guest Instance within the whole Cloud must have a unique IP Address.

Each pod in a basic zone is a broadcast domain, and therefore each pod has a different IP range for the guest network.

Advanced Networking

In an Advanced Networking setup, each Account is allocated the following:

[slist]

  • Public IP, this is a live Internet Address and is assigned to their Virtual Router
  • Guest IP Range, for example 10.1.1.0/24
  • VLAN ID for the Isolated Guest Network

[/slist]

The default Guest IP Range is the same for all Accounts within a Zone, however it is possible for a Root Administrator to assign custom IP ranges to different Accounts.

Guests Instances within an Account communicate with each other and their Virtual Router via their own dedicated VLAN. Guest Instances can be running on any Host within the Zone, the VLANs on the Layer 2 Switches ensure they can always communicate by using VLAN Trunking.

Reserved System IP Addresses

When configuring a POD, IP addresses must be reserved for the System VMs, an allocation of 10 addresses  is normally sufficient for this purpose. The reserved addresses  are used by the Secondary Storage VM, and the Console Proxy VMs (on a busy system multiple Console Proxy VMs may be automatically created to deal with the workload)

Every Host and System VM throughout the entire Cloud must have a unique IP address, so when adding additional Zones, the configuration of any existing Zones must be taken into account.

Link-Local IPs

When using XenServer and KVM Link-Local addressses  are allocated to the System VMs and every Virtual Router, with over 65,000 addresses  available in the Link-Local Subnet it is unlikely you will ever run out of addresses. PODs containing only XenServer and KVM Clusters can be allocated a /24 CIDR for use by the Hosts and Storage.

When using VMware however IPs from the configured IP range for the POD are used by the System VMs Virtual Routers, so it is vital a suitable IP Schema is put in place when the system is configured.

For a VMware POD consider using a /21 address range, as this will allocate 2046 IPs for use by the Hosts, Storage, System VMs and Virtual Routers.

Guest Isolation

Guest Instances require isolation from other Instances running within the same Zone, this is achieved in two ways, Security Groups or VLANs.

Security Groups are available in both the Basic and Advanced Networking models, however VLAN isolation is only available with Advanced Networking.

Security Groups

When using Security Groups , each Account has a default Security Group which is automatically created. When Instances are created, they are assigned to one or more Security Groups.

Users can create additional Security Groups at any time, however existing Guest Instances cannot be assigned to newly created Security Groups, they have to be allocated to a Security Group when they are created.

Guest Instances within a Security Group are able to communicate with each other directly. Ingress and Egress rules on the Security Group control the flow of traffic, both in and out of the Group.

VLANs

VLANs are the default method for Guest Isolation when using Advanced Networking. When the first Guest Instance is created for an Account, an Isolated Network is also created, using the Guest CIDR configured for that Zone (the default 10.1.1.0/24)

If an additional Isolated Guest Network with a specific IP Range is required, a ROOT Administrator must create it and assign it to the Account, Users and Domain Administrators do not have the ability to create Isolated Guest Networks with a specific IP Range and VLAN ID .

Summary

This was an introduction to Networking in CloudStack 3.x, its aim was to give you an understanding of the two models, Basic and Advanced. There is lots more to CloudStack Networking as it is one of the key differentiators between simple Virtual Private Servers, and a true Cloud offering.

The next article in this series will look in more depth at the Advanced Networking model, and how to create and use custom Network Offerings using the CloudStack API.

About the Author

Geoff Higginbottom is CTO of ShapeBlue, the strategic cloud consultancy. Geoff spends most of his time designing  private & public cloud infrastructures for telco’s, ISP’s and enterprises based on CloudStack.