Many people find it challenging to get started with CloudStack’s networking. There are some basic concepts, which although not overly complicated, are not especially obvious either. This blog will try to explain these underlying concepts, in order to make getting started with CloudStack networking models much easier.
In order to make this blog suitable for beginners, in some places I will make generalisations which are true 95% of the time. However, there may be exceptions and workarounds to any given rule. Also, when talking about network isolation, I will generally refer to VLANs, but more advanced users may well use an SDN technology to provide network isolation.
The CloudStack Networks
CloudStack has a few underlying networking models, each having different requirements. These underlying models enable different feature sets, some of which are mutually exclusive.
We’ll start though, by explaining the different network planes which we have.
The Management Network (The Control Plane)
The management network (sometimes called the private network), is a range of IP addresses which the cloud builder reserves in their environment for CloudStack to give to the helper VMs (known as System VMs) which it creates. These IP addresses are then used for communication between the management server(s) and the system VMs. In reality it is not necessarily a single network – a unique range of IP addresses is required for each CloudStack pod.
The Guest Networks
The guest networks are really where the underlying network models vary the most, but for this summary, suffice to say that guest networks carry the network traffic between the guest instances, and between the guest instances and their respective gateways.
In isolated and VPC networks, CloudStack creates a virtual router which sits between the Guest network and the public network. The virtual router takes an IP address from the public range assigned by the cloud builder, and places it on the external face of the virtual router, thereby providing an externally accessible address. External users or devices can connect to the isolated network behind the virtual router using this address.
Although these are known as public network(s) they needn’t be publicly accessible networks. When referring to public networks we are generally referring to networks which are immediately North of the last gateway device which CloudStack has control of. In networks which do not have a gateway device which CloudStack controls, we say that there is no public network.
Public networks are zone wide, therefore IPs ranges assigned by the cloud builder will be assigned to a system VM in any pod.
Storage Data Plane
Finally, CloudStack can route long-term storage data (non-primary storage data) over a separate network plane. This network plane links secondary storage VM to the back-end secondary storage, and also links a number of physical host types directly to the back-end secondary storage. When ISOs, templates or volumes are imported or exported, the secondary storage VM uses this network. When a template is transferred from secondary to primary storage or a snapshot from primary to secondary storage, again, this network is employed.
If a storage network is not defined, then CloudStack will route all ‘long-term storage’ traffic over the management network.
Note: primary storage data can be transferred over dedicated networks but this is outside of CloudStack’s control.
The Networking Models
The advanced networking model is probably the most used in CloudStack. In this model, each tenant (account) can create networks which are isolated from each other (generally by VLAN). Therefore users in one account cannot access the networks (and therefore the VMs) in another. In the simplest of advanced networks, CloudStack creates a virtual router which NATs traffic between an isolated guest network and the public networks.
The above model is further enhanced in the VPC network, where a virtual router can be connected to multiple isolated guest networks for more advanced topologies.
In the advanced networking model, ‘shared’ networks can also be created. The router in these networks is not in the data-path, and simply provides services such as DNS and DHCP. Shared networks must be created by the admins rather than by users.
The basic networking model is very similar to the original Amazon Web Services security groups model, whereby isolation is provided at the virtual machine level (rather than per network). ‘Security groups’ are used to manage the ports and IP addresses from which traffic can flow to or from each guest instance.
Management Network (Basic and Advanced Networking)
The requirements of the management networks are the same for both basic and advanced networking. For each pod, the cloud builder must give CloudStack a range of IP addresses and a gateway which CloudStack can use for the system VMs that it creates. Generally, only a handful of IP addresses are required per pod, unless the pod contains vSphere or Hyper-V hosts. Then, additional IP addresses are required as each virtual router will use an address from the management network.
A global setting ‘check.pod.cidrs’ controls whether CloudStack will allow the management networks in pods to overlap. Meaning to allow pods one and two (for example) to share a gateway and subnet mask, while having different IP address ranges assigned to them, rather than forcing both pods to have completely unique subnets.
Management network ranges must be unique within any given region.
Guest Networks (Advanced Networking)
In advanced networking, each of the networks are isolated and therefore the same IP addresses can be reused in each network. However, to provide the isolation, CloudStack needs to know what VLANs it can use (i.e. ones which are reserved in the cloud builder’s infrastructure for CloudStack’s use alone). CloudStack assumes that each zone is isolated from the others, and therefore allows VLANs to be reused in different zones.
Guest Networks (Basic Networking)
In basic networking, VLANs are not required for isolation – each VM must have a different IP address in any given zone. Therefore, the cloud builder simply assigns a range of IP addresses in each pod, which CloudStack can assign to the virtual machines. The cloud builder is also required to specify the IP address of the gateway applicable to the range of IP addresses for each pod. Where there are multiple pods with different gateways, the gateways must be able to route traffic between each other, thereby allowing guest VMs in different pods to communicate with each other (users’ security groups permitting). Communication to/from a guest VM to the outside world is also via the gateway for the guest network in any given pod.
Public Networks (Advanced Networking)
For isolated and VPC networks, the cloud builder must provide one or more external (public) network ranges which will be assigned to the external interfaces of the virtual routers. The cloud builder must also provide CloudStack with the relevant gateway for each range, so that the public traffic can reach its intended external destinations.
In public clouds, the public networks will most likely be publicly routable IP ranges. However, for enterprises these may be either publicly routable or RFC 1918 (internal) IP ranges.
Public networks are ‘zone-wide’, therefore ranges assigned to public networks must be accessible via all hosts in a given zone.
Basic and Advanced Shared Guest Networks
Neither the shared network type in advanced networking or the guest network type in basic networking generally have a public network, as they are directly connected to external networks via the configured gateway of the network.
Armed with an understanding of the logical networks required, the cloud builder now needs to understand how to connect their hosts and switches to enable logical networks. In this section we will look at each logical network type (plane) in turn and look the physical requirements of these networks.
Control Plane (Management Networks)
When defining the management network for a pod, the cloud builder can specify a VLAN on which this traffic will travel. For simplicity, many cloud builders will have their hosts and system VMs share a management VLAN/CIDR. For enhanced security, the management traffic for physical hosts may be kept on a separate VLAN.
Converged networking can muddy the waters with respect to physical connections, however, normal practice is to have a single or a bonded pair of physical interfaces which carry management traffic into physical hosts. The physical host would have a management IP address in the management range.
Guest Traffic Plane (Advanced Networks)
In advanced networks, guest traffic traverses the physical networks via tagged VLANs (search IEEE 802.1Q for more information on tagged VLANs). To enable the movement of this traffic, a single or bonded pair of physical interfaces must be assigned to this traffic, and the corresponding switch interfaces set to ‘trunk’ tagged traffic. The switch ports are generally configured to only allow specific VLANs which have been reserved for CloudStack use through.
It is important that where multiple switches are used in an environment, the interconnects also trunk any VLANs required for guest traffic between them.
Guest Traffic Plane (Basic Networks)
In basic networks, a CIDR is assigned to each pod for the guest network traffic, and as with advanced networking, a single or bonded pair of physical interfaces must be assigned to this traffic. As it is normal practice to assign a subnet to a specific VLAN, the switch ports on the corresponding interfaces could be set as access ports for the VLAN which the subnet is on, therefore allowing traffic flow between the guest VMs in a pod, and the guest VMs and their external gateway. This gateway must be created and configured by the cloud builder. CloudStack needs to know the default gateway address it should give to the guest VMs in any given pod, but CloudStack has no control over this gateway itself.
Public Networks (Advanced Networks)
At this level, public networks are very similar to guest traffic. As public networks use tagged VLANs, a single or bonded pair of network interfaces must be assigned to this traffic, with the respective switch ports again set to trunk the public network VLANs. As with guest networks, where multiple switches are used in an environment, the interconnects must also trunk the VLANs required for public networks.
It is not uncommon for the guest and public network traffic to share interfaces with the respective switch ports set to trunk the VLANs for both traffic types.
Storage traffic is often given a separate interface or interfaces in order to segregate this high-volume traffic from other data types. These can be either trunked or access ports, depending on the cloud builder’s preference.
Advanced Network Trunking
Further information on the physical networks in CloudStack can be found in this previous blog: https://www.shapeblue.com/understanding-cloudstacks-physical-networking-architecture/
CloudStack is extremely versatile when it comes networking. However, that versatility comes at the cost of a little additional learning to get started. The aim of this blog is to explain the requirements of CloudStack’s networking, on both the physical infrastructure on which it is placed and also the logical requirements of ‘reserved’ IP address ranges.
About the author
Paul Angus is VP Technology & Cloud Architect at ShapeBlue, The Cloud Specialists. He has designed and implemented numerous CloudStack environments for customers across 4 continents, based on Apache Cloudstack.
Some say that when not building Clouds, Paul likes to create Ansible playbooks that build clouds. And that he’s actually read A Brief History of Time.