Tag Archive for: Networking (Infrastructure)

Troubleshooting CloudStack Virtual Routers

Cloudstack’s multi-tenant virtualised networking model is one of its strongest features. Abstracting complex networking concepts and allowing simple UI/API configuration of networks is something loved by users of Cloudstack clouds. But, as an operator/administrator of a Cloudstack cloud you’ll almost certainly have had to troubleshoot network  problems  – and that means troubleshooting CloudStack’s Virtual Routers (VRs) .

As Roy says in the picture , turning it off and on again can often resolve issues with a VR (or restarting a VR in Cloudstack language). But if that doesn’t work, administrators need to troubleshoot the VR. In this article, I will discuss some of the common approaches to such troubleshooting and look at some new CloudStack features that have been added to make this process much easier for administrators.

The problem

Now let’s try to imagine what could be wrong with our VR? The first and most obvious things are mistakes when defining IP range, netmask and gateway for our networks, or general human errors when building out infrastructure elements. CloudStack does provide some validation of the input you give, but it’s not guaranteed to pick up every mistake.

Then, most commonly we could have connectivity issues. The list is endless here, from inside the router itself to the Internet, other VPC/networks, Private Gateways and so on. To resolve such issues, we’ll need some serious networking skills and time to investigate, and most of the time it happens to be one tiny bit of configuration that we’ve missed or misconfigured.

Solution

To get to the bottom of these issues, it is often required to dig through all configurations, run connectivity troubleshooting in and out of a machine and so on, which can be painful and time consuming. CloudStack now offers two new features which will make our life easier and troubleshooting of the VR far more painless to the root admin. Let me introduce you to the “Run Diagnostics” and “Get Diagnostics” features from the VR. Using these two features an administrator will be able to get valuable information from the virtual router without even logging in to it. Furthermore, he’ll be able to operate from the inside of the router and execute scripts, commands etc. to determine what’s wrong or even fix it.

Let’s dive into details of these features.

Run diagnostics

The new “Run diagnostics” feature allows root administrators to execute connectivity diagnostics commands from the VR to a given target, which could be a host on the internet or some other internal element from our infrastructure that we need to make sure is reachable. This enables us to trace how the traffic goes in and out from the VR. The available commands are ping, arping and traceroute. Admins can execute those with any standard option and argument that they support under the VR operating system (Debian). CloudStack effectively logs in to the VR and executes the exact command with the given options and parameters, then it will display back the response within the management console UI.

Here’s a simple example of ping command being sent to google.com:

From Infrastructure section in the GUI, we select the VR and click on Run Diagnostics:

Then select the command you want to execute, add destination and any extra arguments and click OK.

This comes the back with the VR response:

Likewise, admins can do traceroute and arping.

Obviously, there’s a new API created for this which can be used with cloudmonkey. Here’s an example how to do that:

(localcloud) SBCM5> > run diagnostics targetid=f08d92b6-4839-4ca5-8924-bb2c59ce14c2 ipaddress=google.com type=ping params='-c 2'
{
"diagnostics": {
"exitcode": "0",
"stderr": "",
"stdout": "PING google.com (216.58.198.174): 56 data bytes\n64 bytes from 216.58.198.174: icmp_seq=0 ttl=50 time=7.664 ms\n64 bytes from 216.58.198.174: icmp_seq=1 ttl=50 time=7.645 ms\n--- google.com ping statistics ---\n2 packets transmitted, 2 packets received, 0% packet loss\nround-trip min/avg/max/stddev = 7.645/7.654/7.664/0.000 ms"
}
}

…where the targetId is the given ID of the VR in the test.

Run diagnostics is available as of 4.12 release and it is hypervisor agnostic.

Get Diagnostics Data

This feature, intended to be used by root administrators provides a way to retrieve any file from system VMs if the file path is known and specified as input. By default, the API gathers logs and configuration/property files and sends them as compressed tarball to a secondary storage pool within the same zone as the target system VM. A download URL is returned to the operator on successful file retrieval to allow him to download it to their local machines.

The API can be executed against all three types of system VMs, each of them having a separate default list of files and configurations it’ll gather. Here’s a list of the defaults for each system VM type:

• VR – ‘diagnostics.data.vr.defaults’ global setting:

“IPTABLES], [IFCONFIG], [ROUTE], /etc/dnsmasq.conf, /etc/resolv.conf, /etc/haproxy.conf, /etc/hosts.conf, /etc/dnsmaq-resolv.conf, /var/log/cloud.log, /var/log/routerServiceMonitor.log, /var/log/dnsmasq.log”

• CPVM – ‘diagnostics.data.cpvm.defaults’ global setting

“[IPTABLES], [IFCONFIG], [ROUTE], /usr/local/cloud/systemvm/conf/agent.properties, /usr/local/cloud/systemvm/conf/consoleproxy.properties, /var/log/cloud.log”

• SSVM – ‘diagnostics.data.ssvm.defaults’ global setting

“[IPTABLES], [IFCONFIG], [ROUTE], /usr/local/cloud/systemvm/conf/agent.properties, /usr/local/cloud/systemvm/conf/consoleproxy.properties, /var/log/cloud.log”

Please note that one could change the default to include custom files/scripts. To get the defaults from a System VM the admin simply calls the API just giving a target. To call a custom script the root administrator will also have to make sure the script is present at the “/usr/bin” directory on the system VM and can be executed. Once there, it’s name needs to be passed in square brackets, like this: [script]. It also accepts list of values separated by comma.

Here’s where to find it in the CloudStack Console:

Pick a VR and expand the quick view options, then you’ll be able to see ‘Get Diagnostics Data’ button and click on it:

After that, a pop-up would appear, taking one argument ‘Files’. Leave blank to get the defaults or fill in absolute paths to files or commands in [brackets] as custom values
Defaults:

or a command:

Once executed, SSVM will gather all the content in an archive and you’ll be given an URL to download it from:

And here’s the containing of the tar archive defaults for Virtual router:

Here’s an example how to use it directly calling the API from cloudmonkey:

(localcloud) SBCM5> > get diagnosticsdata targetid=1ce6de39-b4ed-412f-aced-3a421924c477 files=[ifconfig]
{
"diagnostics": {
"url": "https://10-1-36-2.sbcloud.uk/userdata/0ff2e4ae-8b55-49a0-815e-22185e45a7d1.tar"
}
}

The following global settings were introduced with Get Diagnostics FR, which let you control and configure the feature and most specifically how it uses the secondary storage of your datacenter. It’s a good practice to keep the garbage collection enabled and running so you don’t end-up with secondary storage occupied with diagnostics logs. Following is list of configurations that the admin can use.

SettingDescriptionDefault Value
diagnostics.data.gc.enable
Enable the garbage collector background task to delete old files from secondary storage.  Requires management server restart
True
diagnostics.data.gc.interval
The interval at which the garbage collector background tasks in seconds. Requires management server restart
86400 (Once a day)
diagnostics.data.retrieval.timeout
Overall system VM script execution time out in seconds. Does not require management server restart.
3600
diagnostics.data.max.file.age
Sets the maximum time in seconds a file can stay in secondary storage before it is deleted. 
86400 (1 day)
diagnostics.data.disable.threshold
Sets the secondary storage disk utilisation percentage for file retrieval. Used to look for suitable secondary storage  with enough space, otherwise an exception is thrown when no secondary store is found.
0.95 (95 %)

Conclusion

Get Diagnostics (cloudstack-#3350) has been submitted against master and against the 4.13 milestone, so hopefully it’ll make it in the next LTS release. Run Diagnostics (cloudstack-#2833) has been merged as of 4.12 release. Both are available from the UI and API, independent of the hypervisor used in CloudStack, they are handy, neat and can save tons of time accessing the VRs and finding what you need from them. Furthermore, they could be used to monitor and automate some of the processes on the VR if required.

About the author

Boris Stoyanov is Software Engineer in testing at ShapeBlue, the Cloud Specialists. Bobby spends his time testing features for the Apache CloudStack Community and for ShapeBlue clients.

/by

Securing CloudStack 4.11 with HTTPS/TLS

Apache CloudStack is generally considered secure out of the box, however it does have the capability of protecting both system VM traffic as well as management traffic with TLS certificates. Prior to version 4.11 CloudStack used Tomcat as the web server and servlet container. With 4.11 this has been changed to embedded Jetty web server and servlet container that makes CloudStack more secure and independent of distribution provided Tomcat version which could be prone to security issues. This changes the management server’s TLS configuration.

In this blog post we’ll go through how to implement native TLS to protect both the system VMs – CPVM and SSVM – as well as the management server with public TLS certificates.

Please note since SSL is now being deprecated we now refer to the replacement – TLS – throughout this post.

Overview

The TLS configuration in CloudStack provides different functionality depending on the system role:

  • TLS protecting the management server is a Jetty configuration which protects the main web GUI and API endpoint only. The configuration of this is handled in the underlying embedded Jetty configuration files.
  • System VMs:
    • The Console Proxy VM TLS configuration provides secure HTTPS connection between the main console screen (your browser) and the CPVM itself.
    • The Secondary Storage VM SSL configuration provides a secure HTTPS connection for uploads and downloads of templates / ISOs / volumes to secondary storage as well as between zones.
    • System VM SSL configuration is carried out through global settings as well as the CloudStack GUI.

System VM HTTPS configuration

Global settings

The following global setting are required configured to allow system VM TLS configuration:

Global settingValue
consoleproxy.url.domaindomain used for CPVM (see below)
consoleproxy.sslEnabledSwitches SSL configuration of the CPVM on / off
Values true | false
secstorage.ssl.cert.domaindomain used for SSVM (see below)
secstorage.encrypt.copySwitches SSL configuration of the SSVM on / off
Values true | false

The URL configurations can take three formats – and these also determine what kind of TLS certificate is required.

  • Blank: if left blank / unconfigured the URLs used for CPVM and SSVM will simply be passed as the actual public IP addresses of the system VMs.
  • Static URL: e.g. console.mydomain.com or ssvm.mydomain.com. In these cases CloudStack rely on external URL load balancing / redirection and/or DNS resolution of the URL to the IP address of the CPVM or SSVM. This can be achieved in a number of different ways through load balancing appliances or scripted DNS updates.
    This configuration relies on:

    • The same URL used for both CPVM and SSVM, or
    • a multi-domain certificate provided to cover both URLs if different ones are used for CPVM and SSVM.
  • Dynamic URL: e.g. *.mydomain.com. In this case CloudStack will redirect the connections to the CPVM / SSVM to the URL “a-b-c-d.mydomain.com” where a/b/c/d represent the IP address, i.e. a real world URL would be 192-168-34-145.mydomain.com.
    This relies on two things:

    • DNS name resolution configured for the full public system VM IP range, such that all combinations of “a-b-c-d.mydomain.com” can be resolved. Please note in CloudStack version 4.11 the public IP range used purely by system VMs can be limited by reserving a subrange of public IP addresses just for system use.
    • An TLS wildcard certificate covering the full “mydomain.com” domain name.

Configuration process – GUI

The first step is to configure the four global settings above, then restart the CloudStack management service to make these settings live.

Next upload the TLS root certificate chain, the actual TLS certificate as well as the PKCS8 formatted private key using the “TLS certificate” button in the zone configuration. In this example we use a wildcard certificate.

Configuration process – API / CloudMonkey

The uploadCustomCertificate API call can be used to upload the TLS certificates. Please note the upload process does require at least two API calls – more depending on how many intermediary certificates are used. If you use CloudMonkey the certificates can be uploaded in cleartext – otherwise they have to be URLencoded when passed as part of a normal HTTP GET API call.

  • In the first API call the combined root and intermediary certificates are uploaded. In this API call the following parameters are passed – note we don’t pass the private key:
    • id=1
    • name: Give the certificate a name.
    • certificate: the root / intermediary certificate in cleartext, with all formatting / line breaks in place. In the example below we pass a chain of root and intermediary certificates.
    • domainsuffix: provide the suffix used, e.g. *.mydomain.com.
  • The second API call in our example uploads the site certificate. In this case we do not give the certificate a name (CloudStack automatically names this):
    • id=2
    • certificate: the issued site certificate, again in cleartext.
    • privatekey: the private key, same cleartext format as certificate.
    • domainsuffix: as above.
cloudmonkey upload customcertificate id=1 name=RootCertificate certificate='-----BEGIN CERTIFICATE-----
MIIE0DCCsdf8HqjeIHgkqhkiG9w0BAQsFADCBgzELMAk
...8V3Idv7kaWKK3245lsoynJuh87/BKONvPi8BDAB
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIEfTCCA2WgAwIBAgIDG+cVMA8djwj1ldggKd9d9s
.....mw4TRfZHcYQFHfjDCmrw==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIEADCCAuigAwIBAgIBADANBgkqhkiG9w0BAQUFA
.....csbkqWletNw+vHX/bvZ8=
-----END CERTIFICATE-----'
domainsuffix='*.mydomain.com'

cloudmonkey upload customcertificate id=2 certificate='-----BEGIN CERTIFICATE-----
MIIGrjCCBZagAwIBAgIJAJ....
...xKjPTkOLfwMVWXc8Ul25t7lkyi0+a9jZxFAuDXFRgkQnbw==
-----END CERTIFICATE-----'
privatekey='-----BEGIN PRIVATE KEY-----
MIIEvEbidvik1gkqhkiG9w0BAQEFAASCBK
.....rEF5Qyuyserre87d234jj/Uddf
-----END PRIVATE KEY-----'
domainsuffix='*.mydomain.com'

 

System VM restart

Once uploaded the CPVM and SSVM will automatically restart to pick up the new certificates. If the system VMs do not restart cleanly they can be destroyed and will come back online with the TLS configuration in place.

Testing the TLS protected system VMs

To test the Console Proxy VM simply open the console to any user VM or system VM. The popup browser window will at this point show up as a non-secure website. The reason for this is the actual console session is provided in an inline frame – and the popup page itself is presented from the unsecured management servers. If you however look at the page source for the page the HTTPS link which presents the inline frame shows up in the expected format:

Once the management server is also TLS protected the CPVM console popup window will also show as secured:

The SSVM will also now utilise the HTTPS links for all browser based uploads as well as downloads from secondary storage:

Securing the CloudStack management server GUI with HTTPS

Protecting the management servers requires creating a PCKS12 certificate and updating the underlying Jetty configuration to take this into account. This is described in http://wiki.eclipse.org/Jetty/Howto/Configure_SSL#Configuring_Jetty

First of all combine the TLS private key file, certificate and root certificate chain into one file (in the specified order), then convert this into PKCS12 format and write it to a new CloudStack keystore. Enter a new keystore password when prompted:

# cat myprivatekey.key mycertificate.crt gd_bundle-g2-g1.crt > mycombinedcert.crt

# openssl pkcs12 -in mycombinedcert.crt -export -out mycombinedcert.pkcs12
Enter Export Password: ************
Verifying - Enter Export Password:************

# keytool -importkeystore -srckeystore mycombinedcert.pkcs12 -srcstoretype PKCS12 -destkeystore /etc/cloudstack/management/keystore.pkcs12 -deststoretype pkcs12
Importing keystore mycombinedcert.pkcs12 to /etc/cloudstack/management/keystore.pkcs12...
Enter destination keystore password:************
Re-enter new password:************
Enter source keystore password:************
Entry for alias 1 successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled

 

Next edit /etc/cloudstack/management/server.properties and update the following:

https.enable=true
https.keystore=/etc/cloudstack/management/keystore.pkcs12
https.keystore.password=<enter the same password as used for conversion>

 

In addition automatic redirect from HTTP/port 8080 to HTTPS/port 8443 can also be configured in /usr/share/cloudstack-management/webapp/WEB-INF/web.xml. Add the following section before the section around line 22:

  <security-constraint>
    <web-resource-collection>
      <web-resource-name>Everything</web-resource-name>;
      <url-pattern>*</url-pattern>
    </web-resource-collection>
    <user-data-constraint>
      <transport-guarantee>CONFIDENTIAL></transport-guarantee>
    </user-data-constraint>
  </security-constraint>

 

Lastly restart the management service:

# systemctl restart cloudstack-management

Conclusion

The TLS capabilities of CloudStack has been extended over the last few releases – and we hope this article helps to explain the new global settings and configuration procedure.

About The Author

Dag Sonstebo is  a Cloud Architect at ShapeBlue, The Cloud Specialists. Dag spends most of his time designing, implementing and automating IaaS solutions based on Apache CloudStack.



/by 
Working towards CloudStack zero downtime upgrades
As most people know, Apache CloudStack has gained a reputation as a solid, low maintenance dependable cloud orchestration platform. That’s why in last year’s Gartner Magic Quadrant so many leaders and challengers were organisations underpinning their services with Apache CloudStack. However, version upgrades – whilst being much simpler than many competing technologies – have always been the pain point for CloudStack operators. The irony is that upgrading CloudStack itself is usually relatively painless, but upgrading its distributed networking Virtual Routers often results in network downtime for users for a number of minutes, requiring user maintenance windows.


At ShapeBlue we have a vision that CloudStack based clouds – whatever their size and complexity – should be able to be upgraded with zero downtime. No maintenance windows, no service interruptions: zero downtime. Achieving this will allow all CloudStack users/operators to benefit from the vast array of new functionality being added by the CloudStack community in every release.


We set out on the journey towards zero downtime a number of months ago and have been working hard with the CloudStack community on the first steps (it is important to note that “we” includes many people in the CloudStack community who have contributed to this work). Below, I set out the detail of what we’ve achieved so far and what we hope to be achieving in the future, but if readers just want the headline: CloudStack 4.11.1 has up to an 80%+ reduction in network downtime during upgrades compared to CloudStack 4.9.3, and downtime is near eliminated when using redundant VRs.


What’s the problem when upgrading?


During upgrades, CloudStack’s virtual routers (VRs) have to be restarted and usually destroyed and recreated (this also sometimes has to be done during day-to-day operations, but is most apparent during upgrades). These restarts usually lead to downtime for users – in some cases up to several minutes. Whilst Redundant Virtual Routers can mitigate against this they do have some limitations with regards to backward compatibility and are therefore not always a solution to the problem.


Downtime reductions in CloudStack 4.11.1


With the changes made in 4.11.1 (described below) we have managed to achieve significant reductions in network downtime during VR restarts. Please note these improvements will vary from one environment to another, and will be dependent on hypervisor type, hypervisor version, storage backend as well as network bandwidth, so we suggest testing in your own environment to determine the benefits. We’ve tested with a typical VR configuration in our virtualised lab environment.


The testing setup used is as follows:


    

  • CloudStack 4.9.3 and 4.11.1 environments built in parallel. To maintain the same hypervisor versions across both tests the following hypervisor versions were used:

      

    • VMware vSphere 5.5
      • 

    • KVM on CentOS7
      • 

    • XenServer 7.0
      • 

    

    • 

  • Environment configuration: In each test we build a simple isolated network with:

      

    • 10 VMs
      • 

    • 10 IP addresses
      • 

    • Firewall rules configured on all IP addresses
      • 

    

    • 



Downtime was measured as follows:


    

  • For egress traffic we measured the total amount of time an outbound ping would fail from a hosted VM during the restart process.
    • 

  • For ingress traffic we assumed a hosted service on a CloudStack VM and measured the amount of time SSH would be unavailable for during the restart process.
    • 

  • In all tests we carried out a “restart network with cleanup” operation and measured the above times. Note – with the new parallel VR restart process (see below) we no longer care how long the overall process takes – we are only interested in how long the network is impacted for. As a result we’ve simply measured the sum of time services were unavailable for (note this time may in some cases be a sum of multiple downtime periods).
    • 

  • Tests were repeated multiple times and average number of seconds calculated for ingress and egress downtime across tests for each hypervisor. To illustrate our best case scenarios we’ve also included the shortest measured downtime figure.
    • 



Results are as follows:







	EnvironmentACS 4.9.3 avgACS 4.11.1 avg (lowest)Reduction avg (highest)






	VMware 5.5119s21s (12s)

82% (90%)




	KVM / CentOS744s26s (9s)40% (80%)




	XenServer 7.0181s33s (15s)82% (92%)







 


How these results were achieved


Existing improvements made in CloudStack 4.11


A number of changes were made in CloudStack 4.11 designed to improve VR restart performance:


    

  • The system VM has been upgraded from Debian 7 (init based) to Debian 9 (systemd based)
    • 

  • The patching process and boot times have been improved, and we have also eliminated reboots after patching
    • 

  • The system VM disk size has been reduced, leading to faster deployment time.
    • 

  • The VPN backend in the VR has been upgraded to Strongswan, which provides improved VPN performance
    • 

  • The redundant VR (RVR) mechanisms have been improved.
    • 

  • Code base has been refactored, and it is now easier to build and maintain VR code
    • 

  • A number of stability improvements made
    • 



 Changes  in CloudStack 4.11.1 – Parallel VR restarts


CloudStack 4.11.1 will ship with a new feature: Parallel VR Restarts, which  changes the behaviour of the “restart network with cleanup” option. In previous CloudStack versions this method would be a serial action where the original VR would be stopped and destroyed and then a new VR started. In CloudStack 4.11.1 this has now been changed to a parallel process where a “restart with cleanup” means:


    

  • A new VR is started in the background while the old one is still running and providing networking services.
    • 

  • Once the new VR is up and has checked in to CloudStack management the old VR is simply stopped and destroyed.
    • 

  • This is followed by a last configuration step where ARP caches at neighbours are updated.
    • 



With this method there is no negotiation between old and new VR, CloudStack simply orchestrates the parallel startup of the new VR. As a result this method does not have any pre-requisites around the version of the original VR – meaning it can be used for VR restarts after upgrade from considerable older CloudStack versions to 4.11.1.


It is worth noting that this 4.11.1 feature does not make large reductions in the actual  VR processing time itself – however with the parallel startup this doesn’t affect network downtime, and the downtime itself is more connected to the final handover of network processing from old to new VR.


In addition to the considerable reduction in normal VR restart downtime, this feature also introduces a much improved redundant VR restart – this comes close to eliminating network downtime when redundant VR networks are restarted, but does obviously mean the old and new VRs need to be version compatible. In our own testing we have seen downtime for redundant VR networks near eliminated.


Coming in future versions


Advanced  parallel restarts


The next step on the journey is to add further handshaking between old and new VR:


    

  • New VR will be started in parallel to old, but with some network services and / or network interfaces disabled.
    • 

  • Once new VR is up CloudStack management will do an external handover from old VR to new, i.e. handle VR connectivity via the hypervisor.
    • 



Fully negotiated redundant VR restarts


The last step on the journey will be aiming towards a fully redundant handover from old to new VR:


    

  • In this final step the end goal is to make all VRs redundant capable, which will reduce same version restart times as well as future upgrade restart times.
    • 

  • New VR will again be started in parallel to old, but will be configured with the redundancy options currently used in the RVR.
    • 

  • Once new VR is up the old and new VRs will internally negotiate the handover of all networking connectivity and services, before the old VR is shut down.
    • 



– – –


During this journey there are a number of tasks needing carried out – both to make the VR internal processing more efficient as well as improving the backend network restart mechanisms:


    

  • General speedup of IPtables rules application
    • 

  • Fix and improvement of the DNS / DHCP configuration to eliminate repetition of processing steps and cut down on processing time
    • 

  • Further improvements of redundant Virtual Router: VRRP2 configuration, and/ or move to a different VR HA solution
    • 

  • A move to make all VRs redundant capable by default
    • 

  • Move from python2 to python3
    • 

  • Consider a move from IPtables to NFtables
    • 

  • Converge and make network topologies flexible, refactor and merge VPC and non-VPC code base
    • 



Conclusion


With the changes implemented in 4.11.1 we have already made a huge step forward in reducing network downtime as part of VR restarts – whether this is during day to day operation or as part of a CloudStack version upgrade. With downtime reduced by up to 80% and average figures of less than 30 seconds this is a considerable improvement – and this is only the first step on the journey.


We have not yet achieved our goal of “zero downtime upgrades” but it is worth considering that the network interruptions that CloudStack can now achieve during an upgrade will be less than the timeouts for many applications and protocols.


In the coming CloudStack versions we hope to continue this development and further reduce the figures, working towards the ultimate goal of “zero downtime upgrades”. 


About The Author


Dag Sonstebo is a Cloud Architect at ShapeBlue, The Cloud Specialists. Dag spends his time designing, implementing and automating IaaS solutions based around Apache CloudStack.


 


/by 
Layer 2 Networks in CloudStack
Layer 2 (also known as the Data Link Layer) is the second in the seven-layer OSI reference model for network protocol design, and is equivalent to the link layer (the lowest layer) in the TCP/IP network model. Layer 2 is the network layer used to transfer data between adjacent network nodes in a wide area network, or between nodes on the same local area network.


We have recently introduced a new Layer 2 network offering in CloudStack. This enables all network services to be provided from network equipment external to CloudStack. As a result, the Virtual Router is not required, and VMs deployed to the network will get their IP addresses and other network services from these external appliances. This new network type is acting like a shared network, but it leaves all network services to be managed outside of CloudStack.


This article describes how to set up a Layer 2 network in the upcoming 4.11 CloudStack release.


The network offering


Following CloudStack’s architecture, there must first be a specific network offering enabled in order to deploy a L2 network. Such an offering can be created by selecting ‘Guest Type: L2’ (it also supports the ability to specify VLAN). When selected, the user will need to specify a VLAN id when creating the network, and the network will be implemented using a dynamically assigned VLAN. Otherwise, the administrator will have to manage this externally from CloudStack, by appliance or other VM. Once created, the network offering needs to be enabled in order to implement the L2 network.


 




The network


CloudStack Administrators are able to create Layer 2 networks simply by clicking ‘Network’, where a new button is introduced: ‘Add L2 Guest Network’. By clicking this button, a CloudStack administrator will have the ability to fill in the network specifics to implement it. Please note, that where the network offering requires a VLAN, CloudStack administrators must specify a particular VLAN id. When all details are complete the network can be saved by clicking ‘OK’. At this stage our Layer 2 network is ready to use and we can start deploying VMs in it.


Add network button:




Add network pop-up:




How it looks


As soon as we get our VMs into ‘Running’ state, we can see on the NICs section that there are no network details available (such as IP Address, gateway, etc.), but the VM is reachable and available through the Console Proxy VM. Since we don’t have any of these services, our network is running without a Virtual Router, which makes the deployment fast.




On the physical network side, the cloud administrators needs to setup all the resources managing the network services, since they are now external to CloudStack. They need to setup routers/switches, DHCP, additional load balancers and etc. Please note that once setup, all services are managed externally, and therefore CloudStack cannot handle IP assignments, port forwarding or other networking services that the Virtual Router normally does. Layer 2 type networks can only be managed by cloud administrators.


About the author


Boris Stoyanov is Software Engineer in testing at ShapeBlue, The Cloud Specialists. Bobby spends his time testing features for the Apache CloudStack Community and for our ShapeBlue clients.


/by 
A Beginners Guide To CloudStack Networking
Many people find it challenging to get started with CloudStack’s networking. There are some basic concepts, which although not overly complicated, are not especially obvious either. This blog will try to explain these underlying concepts, in order to make getting started with CloudStack networking models much easier.


Read more


/by 
Networking KVM for CloudStack
Introduction


KVM hypervisor networking for CloudStack can sometimes be a challenge, considering KVM doesn’t quite have the matured guest networking model found in the likes of VMware vSphere and Citrix XenServer. In this blog post we’re looking at the options for networking KVM hosts using bridges and VLANs, and dive a bit deeper into the configuration for these options. Installation of the hypervisor and CloudStack agent is pretty well covered in the CloudStack installation guide, so we’ll not spend too much time on this.


Network bridges


On a linux KVM host guest networking is accomplished using network bridges. These are similar to vSwitches on a VMware ESXi host or networks on a XenServer host (in fact networking on a XenServer host is also accomplished using bridges).


A linux network bridge is a Layer-2 software device which allows traffic to be forwarded between ports internally on the bridge and the physical network uplinks. The traffic flow is controlled by MAC address tables maintained by the bridge itself, which determine which hosts are connected to which bridge port. The bridges allows for traffic segregation using traditional Layer-2 VLANs as well as SDN Layer-3 overlay networks.


KVMnetworking41


Linux bridges vs OpenVswitch


The bridging on a KVM host can be accomplished using traditional linux bridge networking or by adopting an OpenVswitch back end. Traditional linux bridges have been implemented in the linux kernel since version 2.2, and have been maintained through the 2.4 and 2.6 kernels. Linux bridges provide all the basic Layer-2 networking required for a KVM hypervisor back end, but it lacks some automation options and is configured on a per host basis.


OpenVswitch was developed to address this, and provides additional automation in addition to new networking capabilities like Software Defined Networking (SDN). OpenVswitch allows for centralised control and distribution across physical hypervisor hosts, similar to distributed vSwitches in VMware vSphere. Distributed switch control does require additional controller infrastructure like OpenDaylight, Nicira, VMware NSX, etc. – which we won’t cover in this article as it’s not a requirement for CloudStack.


It is also worth noting Citrix started using the OpenVswitch backend in XenServer 6.0.


Network configuration overview


For this example we will configure the following networking model, assuming a linux host with four network interfaces which are bonded for resilience. We also assume all switch ports are trunk ports:


    

  • Network interfaces eth0 + eth1 are bonded as bond0.
    • 

  • Network interfaces eth1 + eth2 are bonded as bond1.
    • 

  • Bond0 provides the physical uplink for the bridge “cloudbr0”. This bridge carries the untagged host network interface / IP address, and will also be used for the VLAN tagged guest networks.
    • 

  • Bond1 provides the physical uplink for the bridge “cloudbr1”. This bridge handles the VLAN untagged public traffic.
    • 



 


The CloudStack zone networks will then be configured as follows:


    

  • Management and guest traffic is configured to use KVM traffic label “cloudbr0”.
    • 

  • Public traffic is configured to use KVM traffic label “cloudbr1”.
    • 



 


In addition to the above it’s important to remember CloudStack itself require internal connectivity from the hypervisor host to system VMs (Virtual Routers, SSVM and CPVM) over the link local 169.254.0.0/16 subnet. This is done over a host-only bridge “cloud0”, which is created by CloudStack when the host is added to a CloudStack zone.


 


KVMnetworking42




Linux bridge configuration


CentOS


In CentOS the linux bridge configuration is done with configuration files in /etc/sysconfig/network-scripts.


Each of the four individual NIC interfaces are configured as follows (eth0 / eth1 / eth2 / eth3 are all configured the same way):


# vi /etc/sysconfig/network-scripts/ifcfg-eth0



DEVICE=eth0
TYPE=Ethernet
USERCTL=no
MASTER=bond0
SLAVE=yes
BOOTPROTO=none
HWADDR=00:0C:12:xx:xx:xx
NM_CONTROLLED=no
ONBOOT=yes



The bond configurations are specificied in the equivalent ifcfg-bond scripts and specify bonding options as well as the upstream bridge name. In this case we’re just setting a basic active-passive bond (mode=1) with status monitoring every 100ms (miimon=100):


# vi /etc/sysconfig/network-scripts/ifcfg-bond0



DEVICE=bond0
ONBOOT=yes
BONDING_OPTS='mode=1 miimon=100'
BRIDGE=cloudbr0
NM_CONTROLLED=no



The same goes for bond1:


# vi /etc/sysconfig/network-scripts/ifcfg-bond1



DEVICE=bond1
ONBOOT=yes
BONDING_OPTS='mode=1 miimon=100'
BRIDGE=cloudbr1
NM_CONTROLLED=no



Cloudbr0 is configured in the ifcfg-cloudbr0 script. In addition to the bridge configuration we also specify the host IP address, which is tied directly to the bridge since it is on an untagged VLAN:


# vi /etc/sysconfig/network-scripts/ifcfg-cloudbr0



DEVICE=cloudbr0
ONBOOT=yes
TYPE=Bridge
IPADDR=192.168.100.20
NETMASK=255.255.255.0
GATEWAY=192.168.100.1
NM_CONTROLLED=no
DELAY=0



Cloudbr1 does not have an IP address configured hence the the configuration is simpler:


# vi /etc/sysconfig/network-scripts/ifcfg-cloudbr1



DEVICE=cloudbr1
ONBOOT=yes
TYPE=Bridge
NM_CONTROLLED=no
DELAY=0





Optional tagged interface for storage traffic


If a dedicated VLAN tagged IP interface is required for e.g. storage traffic this can be accomplished by created a VLAN tagged bond and tying this to a dedicated bridge. In this case we create a new bridge on bond0 using VLAN 100:


# vi /etc/sysconfig/network-scripts/ifcfg-bond.100



DEVICE=bond0.100
VLAN=yes
BOOTPROTO=static
ONBOOT=yes
TYPE=Unknown
BRIDGE=cloudbr100



The bridge can now be configured with the desired IP address for storage connectivity:


# vi /etc/sysconfig/network-scripts/ifcfg-cloudbr100



DEVICE=cloudbr100
ONBOOT=yes
TYPE=Bridge
VLAN=yes
IPADDR=10.0.100.20
NETMASK=255.255.255.0
NM_CONTROLLED=no
DELAY=0



Internal bridge cloud0


When using linux bridge networking there is no requirement to configure the internal “cloud0” bridge, this is all handled by CloudStack.


Network startup


Note – once all network startup scripts are in place and the network service is restarted you may lose connectivity to the host if there are any configuration errors in the files, hence make sure you have console access to rectify any issues.


To make the configuration live restart the network service:


# service network restart



To check the bridges use the brctl command:


# brctl show



bridge name bridge id STP enabled interfaces
cloudbr0 8000.000c29b55932 no bond0
cloudbr1 8000.000c29b45956 no bond1



The bonds can be checked with:


# cat /proc/net/bonding/bond0



Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth0
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0c:xx:xx:xx:xx
Slave queue ID: 0

Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: 00:0c:xx:xx:xx:xx
Slave queue ID: 0



Ubuntu


To use bonding and linux bridge networking in Ubuntu first install the following:


# apt-get install ifenslave-2.6 bridge-utils



Also add the bonding and bridge modules to the kernel modules to be loaded at boot time:


# vi /etc/modules



# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.
loop
lp
rtc
bonding
bridge



Before continuing also make sure the correct hostname and FQDN are set in /etc/hostname and /etc/hosts respectively. Also add the following lines to /etc/sysctl.conf:


# vi /etc/sysctl.conf



net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0



All interface, bond and bridge configuration are configured in /etc/network/interfaces. Same as for CentOS we are configuring basic active-passive bonds (mode=1) with status monitoring every 100ms (miimon=100), and configuring bridges on top of these. As before the host IP address is tied to cloudbr0:


# vi /etc/network/interfaces



# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
auto eth0
iface eth0 inet manual
bond-master bond0

auto eth1
iface eth1 inet manual
bond-master bond0

auto eth2
iface eth2 inet manual
bond-master bond1

auto eth3
iface eth3 inet manual
bond-master bond1

auto bond0
iface bond0 inet manual
bond-mode active-backup
bond-miimon 100
bond-slaves none

auto bond1
iface bond1 inet manual
bond-mode active-backup
bond-miimon 100
bond-slaves none

auto cloudbr0
iface cloudbr0 inet static
address 192.168.100.20
gateway 192.168.100.1
netmask 255.255.255.0
dns-nameservers 8.8.8.8 8.8.4.4
dns-domain mynet.local
bridge_ports bond0
bridge_fd 5
bridge_stp off
bridge_maxwait 1

auto cloudbr1
iface cloudbr1 inet manual
bridge_ports bond1
bridge_fd 5
bridge_stp off
bridge_maxwait 1



Optional tagged interface for storage traffic


Dedicated VLAN tagged IP interface for e.g. storage traffic is again accomplished by creating a VLAN tagged bond and tying this to a dedicated bridge. As above we add the following to /etc/network/interfaces  to create a new bridge on bond0 using VLAN 100:


auto bond0.100
iface bond0.100 inet manual
vlan-raw-device bond0

auto cloudbr100
iface cloudbr100 inet static
address 10.0.100.20
netmask 255.255.255.0
bridge_ports bond0.100
bridge_fd 5
bridge_stp off
bridge_maxwait 1



Internal bridge cloud0


When using linux bridge networking the internal “cloud0” bridge is again handled by CloudStack, i.e. there’s no need for specific configuration to be specified for this.


Network startup


Note – once all network startup scripts are in place and the network service is restarted you may lose connectivity to the host if there are any configuration errors in the files, hence make sure you have console access to rectify any issues.


To make the configuration live restart the network service:


# service networking restart



To check the bridges use the brctl command:


# brctl show



bridge name bridge id STP enabled interfaces
cloudbr0 8000.000c29b43c4d no bond0
cloudbr1 8000.000c29b43c61 no bond1
cloudbr100 8000.000c29b43c4d no bond0.100



To check the VLANs and bonds:


# cat /proc/net/vlan/config
VLAN Dev name | VLAN ID
Name-Type: VLAN_NAME_TYPE_RAW_PLUS_VID_NO_PAD
bond0.100 | 100 | bond0



# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.7.1 (April 27, 2011)

Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: eth1
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0

Slave Interface: eth1
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 10
Permanent HW addr: 00:0c:xx:xx:xx:xx
Slave queue ID: 0

Slave Interface: eth0
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 10
Permanent HW addr: 00:0c:xx:xx:xx:xx
Slave queue ID: 0



 


OpenVswitch bridge configuration


CentOS


To configure OVS bridges in CentOS we first of all add the bridges using the ovs-vctl command, then add bonds to these:


# ovs-vsctl add-br cloudbr0
# ovs-vsctl add-br cloudbr1
# ovs-vsctl add-bond cloudbr0 bond0 eth0 eth1
# ovs-vsctl add-bond cloudbr1 bond1 eth2 eth3



This will configure the bridges in the OVS database, but the settings will not be persistent. To make the settings persistent we need to configure the network configuration scripts in /etc/sysconfig/network-scripts/, similar to when using linux bridges.


Each individual network interface has a generic configuration – note there is no reference to bonds at this stage. The following ifcfg-eth script applies to all interfaces:


# vi /etc/sysconfig/network-scripts/ifcfg-eth0



DEVICE=eth0
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
HWADDR=00:0C:xx:xx:xx:xx
HOTPLUG=no



The bonds reference the interfaces as well as the upstream bridge. In addition the bond configuration specifies the OVS specific settings for the bond (active-backup, no LACP, 100ms status monitoring):


# vi /etc/sysconfig/network-scripts/ifcfg-bond0



DEVICE=bond0
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBond
OVS_BRIDGE=cloudbr0
BOOTPROTO=none
BOND_IFACES="eth0 eth1"
OVS_OPTIONS="bond_mode=active-backup lacp=off other_config:bond-detect-mode=miimon other_config:bond-miimon-interval=100"
HOTPLUG=no



# vi /etc/sysconfig/network-scripts/ifcfg-bond1



DEVICE=bond1
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBond
OVS_BRIDGE=cloudbr1
BOOTPROTO=none
BOND_IFACES="eth2 eth3"
OVS_OPTIONS="bond_mode=active-backup lacp=off other_config:bond-detect-mode=miimon other_config:bond-miimon-interval=100"
HOTPLUG=no



The bridges are now configured as follows. The host IP address is specified on the untagged cloudbr0 bridge:


# vi /etc/sysconfig/network-scripts/ifcfg-cloudbr0



DEVICE=cloudbr0
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
IPADDR=192.168.100.20
NETMASK=255.255.255.0
HOTPLUG=no



Cloudbr1 is configured without an IP address:


# vi /etc/sysconfig/network-scripts/ifcfg-cloudbr1



DEVICE=cloudbr1
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=none
HOTPLUG=no



Internal bridge cloud0


In addition to the above we also need to configure the internal only cloud0 bridge. This is required when using OVS bridging only, i.e. when the network bridge kernel module has been disabled. If the module is enabled CloudStack will configure the internal bridge using linux bridge, whilst allowing all other bridges to be configured using OVS. Note the CloudStack agent will create this bridge, hence there is no need to configure it using the ovs-vsctl command.

Since there is no routing involved for the internal bridge we simply configure this with IP address 169.254.0.1/16:


# vi /etc/sysconfig/network-scripts/ifcfg-cloud0



DEVICE=cloud0
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
HOTPLUG=no
IPADDR=169.254.0.1
NETMASK=255.255.0.0



Optional tagged interface for storage traffic


If a dedicated VLAN tagged IP interface is required for e.g. storage traffic this is accomplished by creating a VLAN tagged fake bridge on top of one of the cloud bridges. In this case we add it to cloudbr0 with VLAN 100:


# ovs-vsctl add-br cloudbr100 cloudbr0 100



# vi /etc/sysconfig/network-scripts/ifcfg-cloudbr100



DEVICE=cloudbr100
ONBOOT=yes
DEVICETYPE=ovs
TYPE=OVSBridge
BOOTPROTO=static
IPADDR=10.0.100.20
NETMASK=255.255.255.0
HOTPLUG=no



Additional OVS network settings


To finish off the OVS network configuration specify the hostname, gateway and IPv6 settings:


vim /etc/sysconfig/network



NETWORKING=yes
HOSTNAME=kvmhost1.mylab.local
GATEWAY=192.168.100.1
NETWORKING_IPV6=no
IPV6INIT=no
IPV6_AUTOCONF=no



VLAN problems when using OVS


Due to bugs in legacy network interface drivers there are in certain circumstances issues with getting VLAN traffic to propagate between KVM hosts. This is a known issue, and the OpenVswitch VLAN FAQ is a useful place to start any troubleshooting.


One workaround for this issue is to configure the “VLAN splinters” setting on the network interfaces. This is accomplished with the following command:


ovs-vsctl set interface eth0 other-config:enable-vlan-splinters=true



The problem arises when trying to make this setting persistent across reboots, as the command can not be ran as part of the normal ifcfg-eth script.


One way to make this persistent is to add the following lines to the end of the ifup-ovs script:


# vi /etc/sysconfig/network-scripts/ifup-ovs



if [ -x /sbin/ifup-localovs ]; then
/sbin/ifup-localovs
fi



Then add the commands to a new file ifup-localovs:


# vi /sbin/ifup-localovs



#!/bin/sh
ovs-vsctl set interface eth0 other-config:enable-vlan-splinters=true
ovs-vsctl set interface eth1 other-config:enable-vlan-splinters=true
ovs-vsctl set interface eth2 other-config:enable-vlan-splinters=true
ovs-vsctl set interface eth3 other-config:enable-vlan-splinters=true



This will ensure the settings are applied to each network interface at every reboot and after each network service restart.


Network startup


Note – as mentioned for linux bridge networking – once all network startup scripts are in place and the network service is restarted you may lose connectivity to the host if there are any configuration errors in the files, hence make sure you have console access to rectify any issues.


To make the configuration live restart the network service:


# service network restart



To check the bridges use the ovs-vsctl command. The following shows the optional cloudbr100 on VLAN 100:


# ovs-vsctl show



27daed4e-52f3-4177-9827-550f0e7df452
Bridge "cloudbr1"
Port "vnet2"
Interface "vnet2"
Port "bond1"
Interface "eth3"
Interface "eth2"
Port "cloudbr1"
Interface "cloudbr1"
type: internal
Bridge "cloud0"
Port "cloud0"
Interface "cloud0"
type: internal
Port "vnet0"
Interface "vnet0"
Bridge "cloudbr0"
Port "cloudbr0"
Interface "cloudbr0"
type: internal
Port "vnet3"
Interface "vnet3"
Port "vnet1"
Interface "vnet1"
Port "cloudbr100"
tag: 100
Interface "cloudbr100"
type: internal
Port "bond0"
Interface "eth1"
Interface "eth0"
ovs_version: "2.3.1"



The bond status can be checked with the ovs-appctl command:


ovs-appctl bond/show bond0
---- bond0 ----
bond_mode: active-backup
bond may use recirculation: no, Recirc-ID : -1
bond-hash-basis: 0
updelay: 0 ms
downdelay: 0 ms
lacp_status: off
active slave mac: 00:0c:xx:xx:xx:xx(eth0)

slave eth0: enabled
active slave
may_enable: true

slave eth1: enabled
may_enable: true



Ubuntu


First of all install the bonding kernel module and make sure this is added to /etc/modules such that it is loaded on startup:


# apt-get install ifenslave-2.6



# vi /etc/modules



# /etc/modules: kernel modules to load at boot time.
#
# This file contains the names of kernel modules that should be loaded
# at boot time, one per line. Lines beginning with "#" are ignored.

loop
lp
rtc
bonding



To ensure the linux bridge module isn’t loaded at boot time blacklist this module in /etc/modprobe.d/blacklist.conf:


# vim /etc/modprobe.d/blacklist.conf



Add the following line to the end of the file:


blacklist bridge



Ensure the correct hostname and FQDN are set in /etc/hostname and /etc/hosts respectively, and add the following lines to /etc/sysctl.conf:


# vi /etc/sysctl.conf



net.bridge.bridge-nf-call-ip6tables = 0
net.bridge.bridge-nf-call-iptables = 0
net.bridge.bridge-nf-call-arptables = 0



Same as for CentOS we first of all add the OVS bridges and bonds from command line using the ovs-vsctl command line tools:


# ovs-vsctl add-br cloudbr0
# ovs-vsctl add-br cloudbr1
# ovs-vsctl add-bond cloudbr0 bond0 eth0 eth1 bond_mode=active-backup other_config:bond-detect-mode=miimon other_config:bond-miimon-interval=100
# ovs-vsctl add-bond cloudbr1 bond1 eth2 eth3 bond_mode=active-backup other_config:bond-detect-mode=miimon other_config:bond-miimon-interval=100



As for linux bridge all network configuration is applied in /etc/network/interfaces:


# vi /etc/network/interfaces



Note in the following example all individual network interface configuration as commented out – which works in Ubuntu 14.04. For Ubuntu 12.04 the individual interface sections are required.


# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
#auto eth0
#iface eth0 inet manual

#auto eth1
#iface eth1 inet manual

#auto eth2
#iface eth2 inet manual

#auto eth3
#iface eth3 inet manual

allow-cloudbr0 bond0
iface bond0 inet manual
ovs_bridge cloudbr0
ovs_type OVSBond
ovs_bonds eth0 eth1
ovs_options bond_mode=active-backup other_config:miimon=100

auto cloudbr0
allow-ovs cloudbr0
iface cloudbr0 inet static
address 172.16.152.39
netmask 255.255.255.0
gateway 172.16.152.2
dns-nameservers 8.8.8.8
dns-search mbplab.local
ovs_type OVSBridge
ovs_ports bond0

allow-cloudbr1 bond1
iface bond1 inet manual
ovs_bridge cloudbr1
ovs_type OVSBond
ovs_bonds eth2 eth3
ovs_options bond_mode=active-backup other_config:miimon=100

allow-ovs cloudbr1
iface cloudbr1 inet manual
ovs_type OVSBridge
ovs_ports bond1



Bridge startup in Ubuntu 12.04


The OpenVswitch implementation in Ubuntu 12.04 is slightly lacking compared to 14.04 and later. One thing which is missing is startup scripts for bringing the OVS bridges online. This can be accomplished by adding a custom bridge startup script similar to the following:


# openvswitch-bridgestartup
#
# Starts OVS bridges at startup
#
### BEGIN INIT INFO
# Provides: openvswitch-bridgestartup
# Required-Start: openvswitch-switch
# Required-Stop:
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Brings OVS bridges online at boot time
### END INIT INFO

IFCFG="/etc/network/interfaces"

start() {
BRIDGES=$(ovs-vsctl list-br)
for BRIDGE in $BRIDGES; do
if grep -q -w "$BRIDGE" $IFCFG; then
ifup "$BRIDGE"
fi
done
}

case $1 in
start)
start
;;
stop)
;;
*)
echo "Usage: $0 {start}"
exit 2
;;
esac



Configure the service as follows:


# update-rc.d openvswitch-bridges defaults 21



Internal bridge cloud0


In Ubuntu there is no requirement to add additional configuration for the internal cloud0 bridge, CloudStack manages this.


Optional tagged interface for storage traffic


Additional VLAN tagged interfaces are again accomplished by creating a VLAN tagged fake bridge on top of one of the cloud bridges. In this case we add it to cloudbr0 with VLAN 100:


# ovs-vsctl add-br cloudbr100 cloudbr0 100



# vi /etc/sysconfig/network-scripts/ifcfg-cloudbr100



Conclusion


As KVM is becoming more stable and mature, more people are going to start looking at using it rather that the more traditional XenServer or vSphere solutions, and we hope this article will assist in configuring host networking. As always we’re happy to receive feedback , so please get in touch with any comments, questions or suggestions.


About The Author


Dag Sonstebo is  a Cloud Architect at ShapeBlue, The Cloud Specialists. Dag spends most of his time designing, implementing and automating IaaS solutions based on on Apache CloudStack.




/by 
Understanding CloudStack’s Physical Networking Architecture
 


Understanding and configuring the physical connections of a host in a CloudStack deployment can at first be very confusing. While Software Defined Networking (SDN) is set to greatly simplify some aspects, its integration within CloudStack is not fully mature yet and it won’t be the right solution for everyone.


In this article, Paul Angus, Cloud Architect at ShapeBlue, the cloud specialists, unravels some of the areas which can cause confusion in CloudStack’s physical networking architecture.


Let’s Get Physical: Physical vs. Logical Hosts


One source of confusion is that when people refer to a ‘host’ in the CloudStack world they could be referring to two different things – the physical host and the ‘host’ that CloudStack communicates with. The physical host is understandably the physical box with processors, memory and network interfaces etc. The host CloudStack communicates with (over the management network) is the hypervisor within that physical host. So as an example, public traffic needs to be connected to the physical host (so the system VMs can connect out) but not to the logical host (the hypervisor).


The first determining factor in your physical network topology will be the type of zone you will be implementing. Let’s consider CloudStack’s ‘basic’ and ‘advanced’ zones from a physical networking point of view.


Basic ZonesBasic Zones


In a basic zone you only have one ‘physical network’ because you have no VLAN separation, however you can still split traffic across multiple physical NICs (more on this later)


Basic zones can also only have one guest network and no public network.


 


Advanced ZonesAdvanced Zones


In an advanced zone CloudStack allows for a public network and the creation of multiple guest networks, both physically and logically (usually using tagged VLANs).


Use cases for additional physical guest networks could be to create physically separate links from the hosts to a non-CloudStack network segment in a datacentre – for instance an MPLS network


Other use cases could be to enable dedicated guest networks which are only used by certain domains or accounts, or to provide higher performing guest networks, for example 10Gb links vs 1Gb links.


 


Network LabelsNetwork Labels


The most important part of this configuration are the network labels. For an ESXi host they refer to the names of the vSwitches on the hosts, for KVM hosts they refer to the bridges which you will have created and for XenServer hosts they simply refer to the network labels on the host for each of the interfaces or bonds.


The network labels tell CloudStack which physical network interfaces in the host to connect the various virtual interfaces on the guest and system instances to. The SSVM for instance has interfaces on the public, private, storage and management networks and it’s important that these are connected to the correct virtual networks within the hosts, which in turn connect to the physical networks. The actual names/labels can be anything you like – but it’s always good practice to call them something which reminds you which one is which.


Storage Traffic


Of the different CloudStack traffic types, storage traffic always causes the most confusion. The terms ‘storage traffic’ and ‘storage network’ are referring to secondary storage traffic, snapshots (backups), ISOs and templates are all transferred to/from secondary storage over this network.


By default, CloudStack primary and secondary storage traffic travels over the management network. Secondary storage traffic can be configured to travel over a separate network (the storage network) to the management network (there are ways to separate primary storage traffic at the hypervisor level – but that’s another blog).


The main benefits in separating secondary storage and management traffic (and therefore primary storage from secondary storage traffic) are seen when creating a guest instance, as the template is copied from secondary storage to primary storage and when taking a snapshot, as the disk image is copied from primary storage to secondary storage.


Basic Network Traffic


Basic Network Traffic


In a ‘basic’ network all of the traffic would go over the same interface(s) and each traffic type would have the same network label. This is not recommended.


 


 


Basic Network


 


The management, storage and guest traffic can be separated by having separate physical interfaces for them on the hosts and referencing them with different labels. You could now either have them connected to the same switch, different switches or the same switch but on different untagged VLANs. Connecting them to the same switch gives additional throughput to/from the hosts.


 


Network Traffic


Connecting the interfaces to different VLANs (configured at the switch level) [figure 6] gives additional throughput and security as the guest traffic can be kept separated from the management and storage traffic. It’s considered best practice to have the guest traffic going through a separate interface on the hosts to the management traffic in a basic zone configuration – requiring a minimum of two interfaces per host.


 


Advanced Network TrafficAdvanced Network Traffic


In an advanced network CloudStack can separate traffic logically through the use of VLAN tagging. In some ways separation is easier to configure than in basic networking, as you simply need to set the ports on your switches to trunk all of the VLANs you are using, en-masse.


However you would still want to separate traffic physically for increased throughput and potentially increased security. Again this is done through mapping the network label for each physical interface you will be using to the traffic type you wish to travel through that interface.


Inside the HostInside the Host


Bringing this all together, a combined logical and physical diagram of the networking of an advanced zone host, which has two network interfaces, would look like this:


 


 


 


Summary


This article has explained the key concepts and terminology, essential to configuring physical networking within a CloudStack deployment for both ‘basic’ and ‘advanced’ zones. In particular this article has demonstrated how the various CloudStack traffic types travel within the physical network.


Look out for a future accompanying article covering the separation of primary storage traffic from other CloudStack traffic.


About the author


Paul Angus is a Senior Consultant & Cloud Architect at ShapeBlue, The Cloud Specialists. He has designed numerous CloudStack environments for customers across 4 continents, based on Apache Cloudstack and Citrix Cloudplatform.


When not building Clouds, Paul likes to create scripts that build clouds……..and he very occasionally can be seen trying to hit a golf ball.


 


/by 
CloudStack Networking Considerations
With the Acton release of CloudStack, the choices for networking were significantly enhanced. CloudStack 3.x now has more options than ever before which must be fully understood if the platform is to be correctly leveraged to build public or private cloud.


Geoff Higginbottom, CTO of ShapeBlue the strategic cloud consultancy gives an overview of the networking models in CloudStack and highlights the key networking decisions to be made when designing a cloud around CloudStack.


CloudStack has two Networking models, Basic and Advanced, which one you use will depend on your requirements and what you are using your cloud for. As a CloudStack deployment consists of multiple Availability Zones, PODs and Clusters (see https://www.shapeblue.com/2012/03/16/cloudstack-architecture-overview/ for more info) then you can have Zones running a Basic Network architecture, and Zones running an Advanced Network architecture, all within the same Cloud, and all managed by CloudStack.


Network Traffic Types


There are four types of network traffic within CloudStack:


• Guest: When end users run VMs or Guest Instances as CloudStack refers to them, they generate guest traffic. The guest VMs communicate with each other over a network that can be referred to as the guest network.


• Management: When CloudStack’s internal resources communicate with each other, they generate management traffic. This includes communication between Hosts, System VMs (VMs used by CloudStack to perform various tasks in the cloud), and any other component that communicates directly with the CloudStack Management Server.


• Storage: Traffic between Primary and Secondary storage servers, such as VM templates and Snapshots.


• Public: Public traffic is generated when VMs in the cloud access the Internet. Publicly accessible IPs must be allocated for this purpose. End users can use the CloudStack UI to acquire IPs to implement NAT and Load Balancing between their guest network and the public network.


Guest, Management and Storage are common to Basic and Advanced Networking, Public Traffic is only present in an Advanced Network configuration.








Basic Networking
Basic Networking



Advanced Networking
Advanced Networking







It is highly recommended to assign the different traffic types to different dedicated NICs on the Hypervisor, and also to use NIC Bonding where possible.


IP Addressing Schemas


Basic Networking


In a Basic Networking setup, the Guest Instances use the same IP range as the underlying CloudStack and Hypervisor architecture. When designing your IP Schema, allow sufficient IP Addresses to cater for all of your CloudStack infrastructure including the Guest Instances. Consider using an IP Range such as 192.168.0.0/20 for each Zone, as this will provide over 4,000 addresses IPs per Zone. Warning:, If you use a /16 subnet for your first Zone, there would be no more Private IP Addresses left for additional Zones!


IP addresses will need to be reserved for each POD, and a Guest IP Range assigned during the initial configuration of the Zone. Every Host, System VM and Guest Instance within the whole Cloud must have a unique IP Address.


Each pod in a basic zone is a broadcast domain, and therefore each pod has a different IP range for the guest network.


Advanced Networking


In an Advanced Networking setup, each Account is allocated the following:


[slist]


    

  • Public IP, this is a live Internet Address and is assigned to their Virtual Router
    • 

  • Guest IP Range, for example 10.1.1.0/24
    • 

  • VLAN ID for the Isolated Guest Network
    • 



[/slist]


The default Guest IP Range is the same for all Accounts within a Zone, however it is possible for a Root Administrator to assign custom IP ranges to different Accounts.


Guests Instances within an Account communicate with each other and their Virtual Router via their own dedicated VLAN. Guest Instances can be running on any Host within the Zone, the VLANs on the Layer 2 Switches ensure they can always communicate by using VLAN Trunking.


Advanced Networking Graphic


Reserved System IP Addresses


When configuring a POD, IP addresses must be reserved for the System VMs, an allocation of 10 addresses  is normally sufficient for this purpose. The reserved addresses  are used by the Secondary Storage VM, and the Console Proxy VMs (on a busy system multiple Console Proxy VMs may be automatically created to deal with the workload)


Every Host and System VM throughout the entire Cloud must have a unique IP address, so when adding additional Zones, the configuration of any existing Zones must be taken into account.


Link-Local IPs


When using XenServer and KVM Link-Local addressses  are allocated to the System VMs and every Virtual Router, with over 65,000 addresses  available in the Link-Local Subnet it is unlikely you will ever run out of addresses. PODs containing only XenServer and KVM Clusters can be allocated a /24 CIDR for use by the Hosts and Storage.


When using VMware however IPs from the configured IP range for the POD are used by the System VMs Virtual Routers, so it is vital a suitable IP Schema is put in place when the system is configured.


For a VMware POD consider using a /21 address range, as this will allocate 2046 IPs for use by the Hosts, Storage, System VMs and Virtual Routers.


Guest Isolation


Guest Instances require isolation from other Instances running within the same Zone, this is achieved in two ways, Security Groups or VLANs.


Security Groups are available in both the Basic and Advanced Networking models, however VLAN isolation is only available with Advanced Networking.


Security Groups


When using Security Groups , each Account has a default Security Group which is automatically created. When Instances are created, they are assigned to one or more Security Groups.


Users can create additional Security Groups at any time, however existing Guest Instances cannot be assigned to newly created Security Groups, they have to be allocated to a Security Group when they are created.


Guest Instances within a Security Group are able to communicate with each other directly. Ingress and Egress rules on the Security Group control the flow of traffic, both in and out of the Group.


VLANs


VLANs are the default method for Guest Isolation when using Advanced Networking. When the first Guest Instance is created for an Account, an Isolated Network is also created, using the Guest CIDR configured for that Zone (the default 10.1.1.0/24)


If an additional Isolated Guest Network with a specific IP Range is required, a ROOT Administrator must create it and assign it to the Account, Users and Domain Administrators do not have the ability to create Isolated Guest Networks with a specific IP Range and VLAN ID .


Summary


This was an introduction to Networking in CloudStack 3.x, its aim was to give you an understanding of the two models, Basic and Advanced. There is lots more to CloudStack Networking as it is one of the key differentiators between simple Virtual Private Servers, and a true Cloud offering.


The next article in this series will look in more depth at the Advanced Networking model, and how to create and use custom Network Offerings using the CloudStack API.


About the Author


Geoff Higginbottom is CTO of ShapeBlue, the strategic cloud consultancy. Geoff spends most of his time designing  private & public cloud infrastructures for telco’s, ISP’s and enterprises based on CloudStack.


 


/by