Posts

Ceph and CloudStack – Part 2

, , , , ,

In the previous article we covered some basics around Ceph and deployed a working Ceph cluster. In this article, we are going to finish the Ceph configuration needed for CloudStack and add it as a new Primary Storage pool. We are also going to deploy Ceph volumes via CloudStack and examine them. Finally, in part 3 (to be published soon), I will show you some examples of working with RBD images and will cover some Ceph specifics, both in general and related to the CloudStack.

Before proceeding with the actual work, let me first mention that CloudStack supports Ceph with KVM only, so most of the work we do below is KVM related. Let’s define the high-level steps to be done:

  • Create a dedicated RBD pool for CloudStack in which all RBD images (volumes) will be created
  • Create a dedicated authentication key for the previously created pool
  • Update / install required Ceph binaries on KVM nodes
  • Add Ceph as Primary Storage in CloudStack
  • Implement custom storage tag for Ceph Primary Storage
  • Create new Compute / Disk offerings with same storage tag in order to target Ceph

From any Ceph node…

Ceph groups RBD (RADOS block device) images in pools and manages authentication on a per pool level. Each image is collection of many RADOS objects, with each object having a default size of 4MB (configurable per image). At this moment we have no pools created. But before creating a pool, let’s go through some basics around the different kind of pools in Ceph.

There are 2 kind of pools, based on the way the objects are stored across cluster:

  • Replicated – makes sure that there are always total of N replicas/copies of an object
  • Erasure Coding – simplest way to think of this is a network RAID 5/6

Replicated pools are used for better performance at the expense of space consumption, and you can think of it as a network-based RAID 1, where we have n number of replicas of an object. On the other hand, erasure coding pools are usually used when using Ceph for S3 Object Storage purposes and for more space efficient storage where bigger latency and lower performance is acceptable, since it is similar to RAID 5 or RAID 6 (requires some computation power). Here, for example, we may have 4 chunks of actual data and 2 parity chunks (EC 4+2), with just 50% of space overhead, while (depending on the setup), we can still survive losing a Ceph node or even two.

So, let’s create a dedicated pool for CloudStack, set its replica size and finally initialize it:

ceph osd pool create cloudstack 64 replicated
ceph osd pool set cloudstack size 3
rbd pool init cloudstack

The commands above will create a replicated pool named “cloudstack” with total of 64 placement groups (more info on placement groups here) with a replica size of 3, which is recommended for a production cluster. Optionally, you can set replica size of 2 during testing, for somewhat increased performance and less space consumed on the cluster.

Next, let’s generate a dedicated authentication key for our CloudStack pool:

ceph auth get-or-create client.cloudstack mon 'profile rbd' osd 'profile rbd pool=cloudstack'

The command above will output a key to STDOUT only – please save the given key, since we will use it when adding Ceph to CloudStack later:

[client.cloudstack]
key = AQAFSZpc0t+BIBAAO95rOl+jgRwuOopojEtr/g==

Now that the pool for CloudStack is ready, we need to prepare KVM nodes with proper Ceph binaries as well as the write-back caching configuration.

From the Ceph admin node…

Starting from Centos 7.2  (and Ubuntu 14.04), libvirt / QEMU comes by default with support for RBD, so there’s no need to compile the binaries yourself. That being said, if we check KVM nodes with “rpm -qa | grep librbd1″ it will return an existing versions of ‘librbd1” package (version 10.2.5 in my case)  already installed, but most certainly it will not be the current version that corresponds to the cluster version we just installed (13.2.5 in this case). For the record, librbd is a user space Ceph client, to which the qemu / libvirt talks effectively.

Furthermore, if we run command “ceph features” from any Ceph node, it will return (in our fresh Mimic cluster) “luminous” as the minimum compatible release version for the client – that means that our Ceph client (librbd) needs to be of a minimum of “luminous” version (which translates to 12.2.0), but our current librbd version is 10.2.5 – so let’s upgrade it to same Mimic versions as the version of our cluster:

ceph-deploy install --cli kvm1  kvm2

The command above will add Mimic repo to my two KVM nodes and install only the cli binaries (“ceph-common” package). This will also trigger the upgrade of existing “librbd1” package to the correct version. In addition, please make sure that name resolution of the KVM nodes works from the Ceph admin node.

Optionally, if you don’t want to install Ceph cli tools on KVM nodes, you can just upgrade the “librbd1” package while having previously created a proper Ceph Mimic repository on each KVM node (i.e. clone repo file from any Ceph cluster node).

Some of you might want to be able to manage Ceph cluster from KVM nodes as well (beside being able to manage it from Ceph nodes) and to be able to interact with RBD images with via “rbd” or “qemu-img“ tools – in this case we need this “rbd” tool installed on KVM nodes (part of “ceph-common” package, already installed in previous step), then we need ceph.conf locally on KVM nodes in order for the “rbd” tool to know how to connect to cluster, which MONs to target, etc. and finally we need the admin authentication key – this is the file “ceph.client.admin.keyring” which was created on our Ceph admin node when we created our cluster initially (in folder /root/CEPH-CLUSTER, as mentioned in Part 1 of this article series).

Additionally, if we want to use qemu-img tool to examine RBD images, we can either have qemu-img installed on the Ceph cluster nodes or we have to provide the above mentioned ceph.conf and admin keys in their default location (/etc/ceph/) on the KVM nodes, where librbd (client) will pick them up automatically, so we don’t need to specify MON IP/URL and admin key on the command line.

If you don’t want to be able to manage your Ceph cluster from KVM nodes, simply don’t copy over the “ceph.client.admin.keyring” file to KVM nodes. The ceph.conf file is still a must due to RBD caching as explained later. I have decided to make my KVM nodes happy by providing them with ceph.conf and admin keys, as below:

ceph-deploy admin kvm1 kvm2

The command above will effectively just copy ceph.conf and ceph.client.admin.keyring files to /etc/ceph/ folder on KVM nodes. Actually, you can still operate RBD images and manage your cluster from KVM nodes even if you don’t have ceph.conf and admin key present locally – you can always pass required parameters on the command line to “rbd” or “qemu-img” tools, as shown later.

RBD caching

After we have pushed the ceph.conf file to KVM nodes, librbd will read it for any configuration directives under the “[client]” section of that file (beside the other sections), but that section is missing at this moment!

Before we proceed into configuring the RBD caching, let me do here a copy/paste from the original docs that is important to understand regarding RBD caching:

” The user space implementation of the Ceph block device (i.e., librbd) cannot take advantage of the Linux page cache, so it includes its own in-memory caching, called “RBD caching.” RBD caching behaves just like well-behaved hard disk caching. When the OS sends a barrier or a flush request, all dirty data is written to the OSDs. This means that using write-back caching is just as safe as using a well-behaved physical hard disk with a VM that properly sends flushes (i.e. Linux kernel >= 2.6.32). The cache uses a Least Recently Used (LRU) algorithm, and in write-back mode it can coalesce contiguous requests for better throughput. “

After digesting the above info, we can proceed into a brief configuration of caching. We can either fix it manually on each KVM node by adding the missing section in ceph.conf file, or we can do it in a more proper way by changing ceph.conf on the Ceph admin node and then pushing new file version to all KVM (and optionally Ceph cluster) nodes:

cat << EOM >> /root/CEPH-CLUSTER/ceph.conf
[client]
  rbd cache = true
  rbd cache writethrough until flush = true
EOM
 
ceph-deploy --overwrite-conf admin kvm1 kvm2

Please note the above “writethrough until flush = true”. This is a safety mechanism which will force writethrough cache mode until it receives the very first flush request from the VM OS (which means that the OS is sending proper flush requests to the underlying storage, i.e. kernel >= 2.6.32) and then cache mode will change to the write-back, which actually brings performance benefits.

If case you want to play more with RBD caching, please see here – where you can find some important default values which we didn’t explicitly configure i.e. default rbd cache size is 32 MB (this is per volume) – so in case of 50 VMs with 4 volumes each, that translates to 50 x 4 x 32MB =  6.4GB of additional RAM consumed on a KVM host – keep that in mind !

Finally, let’s add the Ceph to CloudStack as an additional Primary Storage – we can do it via GUI or optionally via CloudMonkey (API) as following:

 

Or via CloudMonkey:

create storagepool scope=zone zoneid=3c764ee1-6590-417d-b873-f073d0c550be hypervisor=KVM name=MyCephCluster provider=Defaultprimary url=rbd://cloudstack:AQAFSZpc0t-BIBAAO95rOl+jgRwuOopojEtr_g==@10.2.2.219/cloudstack tags=RBD

Most of the parameters are self-explanatory but let’s explain a few of them:

  • RADOS Monitor: This is the IP address (or DNS name) of the Ceph Monitor (MON) instance – in my case I have defined a very first MON instance (IP address of the Ceph1 node from my cluster) – but in production environment you will want to have an internal Round Robin DNS setup on some internal DNS server (i.e. single zone on Bind) – such that KVM nodes will resolve the ULR (i.e. mon.myceph.cluster) in a round robin fashion to multiple MON instances – this is the way to achieve high availability of Ceph MONs, though some manual DNS zone changes are needed in case of prolonged MON maintenance
  • RADOS Pool: This is the pool “cloudstack” which we created in the beginning of the article
  • RADOS User and RADOS Secret: This are the values from the authentication key which we generated in the beginning of the article, shown below again for your convenience

[client.cloudstack]
key = AQAFSZpc0t+BIBAAO95rOl+jgRwuOopojEtr/g==

The above command, used to add Ceph to CloudStack, will effectively do a few things:

  • On each KVM node, it will create a new storage pool in libvirt
  • The storage pool definition files (xml and the secret) will be written to /etc/libvirt/secrets/ folder as shown below
  • Every time CloudStack Agent is restarted, it will recreate the Ceph storage pool (even if you manually remove the files below)

[root@kvm1]# cat /etc/libvirt/secrets/ef9cfd17-abe1-343d-97a0-cee6c71a6dad.xml
<secret ephemeral='no' private='no'>
  <uuid>ef9cfd17-abe1-343d-97a0-cee6c71a6dad</uuid>
  <usage type='ceph'>
    <name>cloudstack@ceph1.local:6789/cloudstack</name>
  </usage>
</secret>

[root@kvm1]# cat /etc/libvirt/secrets/ef9cfd17-abe1-343d-97a0-cee6c71a6dad.base64
AQAFSZpc0t+BIBAAO95rOl+jgRwuOopojEtr/g==

If we check the libvirt pool created above, we can see that it’s not persistent and it doesn’t start automatically – i.e. when you restart libvirt alone, it will not recreate / start the Ceph storage pool in libvirt– the CloudStack agent is the one doing this for us:

virsh # pool-info ef9cfd17-abe1-343d-97a0-cee6c71a6dad
Name:           ef9cfd17-abe1-343d-97a0-cee6c71a6dad
UUID:           ef9cfd17-abe1-343d-97a0-cee6c71a6dad
State:          running
Persistent:     no
Autostart:      no
Capacity:       299.99 GiB
Allocation:     68.19 MiB
Available:      286.02 GiB

Note that in the example above, I was actually using DNS name for the Ceph MON (ceph1.local) instead of the IP – Ceph MON’s DNS name is resolved to IP both when you add Ceph to CloudStack and every time you start a VM or attach new volume, etc. – so DNS resolution needs to be fast and stable here.

Now that we added Ceph to CloudStack, let’s create a Data disk offering with tag “RBD” – this will make sure that any new volume from this offering is created on storage pool with tag “RBD” – which is Ceph in our case . Here, we are using storage tags to avoid messing up with your existing CloudStack installation – but it’s not required otherwise:

(localcloud) SBCM5> > create diskoffering name=5GB-Ceph displaytext=5GB-Ceph storagetype=shared provisioningtype=thin customized=false disksize=5 tags=RBD
{
  "diskoffering": {
    "created": "2019-03-26T19:27:32+0000",
    "disksize": 5,
    "displayoffering": true,
    "displaytext": "5GB-Ceph",
    "id": "2c74becc-c39d-4aa8-beec-195b351bdaf0",
    "iscustomized": false,
    "name": "5GB-Ceph",
    "provisioningtype": "thin",
    "storagetype": "shared",
    "tags": "RBD"
  }
}

Note the offering ID from above (2c74becc-c39d-4aa8-beec-195b351bdaf0) – and let’s create a disk from it:

(localcloud) SBCM5> > create volume diskofferingid=2c74becc-c39d-4aa8-beec-195b351bdaf0 name=MyFirstCephDisk zoneid=3c764ee1-6590-417d-b873-f073d0c550be
{
  "volume": {
    "account": "admin",
    "created": "2019-03-26T19:52:05+0000",
    "destroyed": false,
    "diskofferingdisplaytext": "5GB-Ceph",
    "diskofferingid": "2c74becc-c39d-4aa8-beec-195b351bdaf0",
    "diskofferingname": "5GB-Ceph",
    "displayvolume": true,
    "domain": "ROOT",
    "domainid": "401ce404-44c1-11e9-96c5-1e009001076e",
    "hypervisor": "None",
    "id": "47b1cfe5-6bab-4506-87b6-d85b77d9b69c",
    "isextractable": true,
    "jobid": "49a682ab-42f9-4974-8e42-452a13c97553",
    "jobstatus": 0,
    "name": "MyFirstCephDisk",
    "provisioningtype": "thin",
    "quiescevm": false,
    "size": 5368709120,
    "state": "Allocated",
    "storagetype": "shared",
    "tags": [],
    "type": "DATADISK",
    "zoneid": "3c764ee1-6590-417d-b873-f073d0c550be",
    "zonename": "ref-trl-1019-k-M7-apanic"
  }
}

Finally, since volume creation is a lazy provisioning process (i.e. volume is created in DB only, not really on storage pool), let’s attach the disk to a running VM (using volume ID “47b1cfe5-6bab-4506-87b6-d85b77d9b69c” from previous command output), which will trigger the actual disk creation on our Ceph cluster (output shortened for brevity):

(localcloud) SBCM5> > attach volume id=47b1cfe5-6bab-4506-87b6-d85b77d9b69c virtualmachineid=19a67e20-c747-43bb-b149-c2b2294002f9
{
  "volume": {
    …
    "jobstatus": 0,
    "name": "MyFirstCephDisk",
    "path": "47b1cfe5-6bab-4506-87b6-d85b77d9b69c",
    …  }
}

Note the “path” output field (which is usually the same as the ID of the volume, except in some special cases) – and let’s check our Ceph cluster if we can find this volume and check it’s properties.

From any KVM node…

[root@kvm1 ~]# rbd ls -p cloudstack
47b1cfe5-6bab-4506-87b6-d85b77d9b69c
 
[root@kvm1 ~]# rbd info cloudstack/47b1cfe5-6bab-4506-87b6-d85b77d9b69c
rbd image '47b1cfe5-6bab-4506-87b6-d85b77d9b69c':
        size 5 GiB in 1280 objects
        order 22 (4 MiB objects)
        id: d43b4c04a8af
        block_name_prefix: rbd_data.d43b4c04a8af
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
        op_features:
        flags:
        create_timestamp: Tue Mar 28 19:46:32 2019

We can also examine the Ceph RBD image with qemu-img tool:

[root@kvm1 ~]# qemu-img info rbd:cloudstack/47b1cfe5-6bab-4506-87b6-d85b77d9b69c
image: rbd:cloudstack/47b1cfe5-6bab-4506-87b6-d85b77d9b69c
file format: raw
virtual size: 5.0G (5368709120 bytes)
disk size: unavailable

As you can see in qemu-img command above, we did not specify any username and authentication keys, because we have our ceph.conf and the admin key files present in /etc/ceph/ folder. If you decided to opt-out of having these 2 files present on KMV nodes, you will have to use a cumbersome command as below:

qemu-img info rbd:cloudstack/47b1cfe5-6bab-4506-87b6-d85b77d9b69c:mon_host=10.2.2.219:auth_supported=Cephx:id=cloudstack:key=AQAFSZpc0t+BIBAAO95rOl+jgRwuOopojEtr/g==

In the above command we are specifying the MON IP address, username and key for authentication.

Now that you got the basics of consuming Ceph from CloudStack, feel free to also create Compute Offerings and System Offerings for Virtual Routers, Secondary Storage VM, Console Proxy VM and experiment with volume migration from i.e. NFS to Ceph. Be sure to have storage tags under control.

I hope that this article series has been interesting so far. In part 3 (which will be the final part), I will show you some examples of working with RBD images and will cover some Ceph specifics, both in general and related to CloudStack.

About the author

Andrija Panic is a Cloud Architect at ShapeBlue, the Cloud Specialists, and is a committer of Apache CloudStack. Andrija spends most of his time designing and implementing IaaS solutions based on Apache CloudStack.

Ceph and CloudStack – Part 1

, , , , , ,

As well as NFS and various block storage solutions for Primary Storage, CloudStack has supported Ceph with KVM for a number of years now. Thanks to some great Ceph users in the community lots of previously missing CloudStack storage features have been implemented for Ceph (and lots of bugs squashed), making it the perfect choice for CloudStack if you are looking for easy scaling of storage and decent performance.

In this and my next article, I am going to cover all steps needed to actually install a Ceph cluster from scratch, and subsequently add it to CloudStack. In this article I will cover installation and basic configuration of a standalone Ceph cluster, whilst in part 2 I will go into creating a pool for a CloudStack installation, adding Ceph to CloudStack as an additional Primary Storage and creating Compute and Disk offerings for Ceph. In part 3, I will also try to explain some of the differences between Ceph and NFS, both from architectural / integration point of view, as well as when it makes sense (or doesn’t) to use it as the Primary Storage solution.

It is worth mentioning that the Ceph cluster we build in this first article can be consumed by any RBD client (not just CloudStack). Although in part 2 we move onto integrating your new Ceph cluster into CloudStack, this article is about creating a standalone Ceph cluster – so you are free to experiment with Ceph.

Firstly, I would like to share some high-level recommendations from very experienced community members, who have been using Ceph with CloudStack for a number of years:

  • Make sure that your production cluster is at least 10 nodes so as to minimize any impact on performance during data rebalancing (in case of disk or whole node failure). Having to rebalance 10% of data has a much smaller impact (and duration) than having to rebalance 33% of data; another reason is improved performance as data is distributed across more drives and thus read / write performance is better
  • Use 10GB networking or faster – a separate network for client and replication traffic is needed for optimal performance
  • Don’t rely on cache tiering, unless you have a very specific IO pattern / use case. Moving data in and out of cache tier can quickly create a bottleneck and do more harm than good
  • If running an older version of Ceph cluster (eg. FileStore based OSD), you will probably place your journals on SSDs. If so, make sure that you properly benchmark SSD for the synchronous IO write performance (Ceph writes to journal devices with O_DIRECT and D_SYNC flags). Don’t try to put too many journals on single SSD; consumer grade SSDs are unacceptable, since their synchronous write performance is usually extremely bad and they have proven to be exceptionally unreliable when used in a Ceph cluster as journal device

Before we continue, let me state that this first article is NOT meant to be a comprehensive guide on Ceph history, theory, installation or optimization, but merely a simple step-by-step guide for a basic installation, just to get us going. Still, in order to be able to better follow the article, it’s good to define some basics around Ceph architecture.

Ceph has a couple of different components and daemons, which serves different purposes, so let’s mention some of these (relevant for our setup):

  • OSD (Object Storage Daemon) – usually maps to a single drive (HDD, SDD, NVME) and it’s the one containing user data. As can be concluded from it’s name, there is a Linux process for each OSD running in a node. A node hosting only OSDs can be considered as a Storage or OSD node in Ceph’s terminology.
  • MON (Monitor daemon) – holds the cluster map(s), which provides to Ceph Clients and Ceph OSD Daemons with the knowledge of the cluster topology. To clarify this further, in the heart of Ceph is the CRUSH algorithm, which makes sure that OSDs and clients can calculate the location of specific chunk of data in the cluster (and connect to specific OSDs for read/write of data), without a need to read it’s position from somewhere (as opposite to a regular file systems which have pointers to the actual data location on a partition).

A couple of other things are worth mentioning:

  • For cluster redundancy, it’s required to have multiple Ceph MONs installed, always aiming for an odd number to avoid a chance of split-brain scenario. For smaller clusters, these could be placed on VMs or even collocated with other Ceph roles (i.e. OSD nodes), though busier clusters will need a dedicated, powerful servers/VMs. In contrast to OSDs, there can be only one MON instance per server/VM.
  • For improved performance, you might want to place MON’s database (LevelDB) on dedicated SSDs (versus the defaults of being placed on OS partition).
  • There are two ways that OSDs can manage the data they store. Starting with the Luminous 12.2.z release, the new default (and recommended) backend is BlueStore. Prior to Luminous, the default (and only option) was FileStore. With FileStore, data is first written to a Journal (which can be collocated with the OSD on same device or it can be a completely separate partition on a faster, dedicated device) and then later committed to OSD. With BlueStore, there is no true Journal per se, but a RocksDB key/value database (for managing OSD’s internal metadata). FileStore OSD will use XFS on top of it’s partition, while BlueStore write data directly to raw device, without a need for a file system. With it’s new architecture, BlueStore brings big speed improvement over FileStore.
  • When building and operating a cluster, you will probably want to have a dedicated server/VM used as the deployment or admin node. This node will host your deployment tools (be it a basic ceph-deploy tool or a full blown ansible playbook), as well as cluster definition and configuration files, which can be changed on central place (this node) and then pushed to cluster nodes as required.

Armed with above knowledge (and against all recommendations given previously) we are going to deploy a very minimalistic installation of Ceph cluster on top of 3 servers (VMs), with 1 volume per node being dedicated for an OSD daemon, and Ceph MONs collocated with the Operating System on the system volume. The reason for choosing such a minimalistic setup is the ability to quickly build a test cluster on top of 3 VMs (which most people will do when building their very first Ceph cluster) and to keep configuration as short as possible. Remember, we just want to be able to consume Ceph from CloudStack, and currently don’t care about performance or uptime / redundancy (beside some basic things, which we will cover explicitly).

Our setup will be as following:

  • We will already have a working CloudStack 4.11.2 installation (i.e. we expect you to have a working CloudStack installation)
  • We will add Ceph storage as an additional Primary Storage to CloudStack and create offerings for it
  • CloudStack Management Server will be used as Ceph admin (deployment) node
  • Management Server and KVM nodes details:
    • CloudStack Management Server: IP 10.2.2.118
    • KVM host1: IP 10.2.3.135, hostname “kvm1”
    • KVM host2: IP 10.2.2.208, hostname “kvm2”
  • Ceph nodes details (dedicated nodes):
    • 2 CPU, 4GB RAM, OS volume 20GB, DATA volume 100GB
    • Single NIC per node, attached to the CloudStack Management Network – i.e. there is no dedicated network for Primary Storage traffic between our KVM hosts and the Ceph nodes
    • Node1: IP 10.2.2.119, hostname “ceph1”
    • Node2: IP 10.2.2.116, hostname “ceph2”
    • Node3: IP 10.2.3.159, hostname “ceph3”
    • Single OSD (100GB) running on each node
    • MON instance running on each node
    • Ceph Mimic (13.latest) release
    • All nodes will be running latest CentOS 7 release, with default QEMU and Libvirt versions on KVM nodes

As stated above Ceph admin (deployment) node will be on CloudStack Management Server, but as you can guess, you can use a dedicated VM/Server for this purpose as well.

Before proceeding with the actual work, let’s define the high-level steps required to deploy a working Ceph cluster

  • Building the Ceph cluster:
    • Setting time synchronization, host name resolution and password-less login
    • Setting up firewall and SELinux
    • Creating a cluster definition file and auth keys on the deployment node
    • Installation of binaries on cluster nodes
    • Provisioning of MON daemons
    • Copying over the ceph.conf and admin keys to be able to manage the cluster
    • Provisioning of Ceph manager daemons (Ceph Dashboard)
    • Provisioning of OSD daemons
    • Basic configuration

We will cover configuration of KVM nodes in second article.

Let’s start!

On all nodes…

It is critical that the time is properly synchronized across all nodes. If you are running on hypervisor, your VMs might already be synced with the host, otherwise do it the old-fashioned way:

ntpdate -s time.nist.gov
yum install ntp
systemctl enable ntpd
systemctl start ntpd

Make sure each node can resolve the name of each other node –  if not using DNS, make sure to populate /etc/hosts file properly across all 4 nodes (including admin node):

cat << EOM >> /etc/hosts
10.2.2.219 ceph1
10.2.2.116 ceph2
10.2.3.159 ceph3
EOM

On CEPH admin node…

We start by installing ceph-deploy, a tool which we will use to deploy our whole cluster later:

release=mimic
cat << EOM > /etc/yum.repos.d/ceph.repo
[ceph-noarch]
name=Ceph noarch packages
baseurl=https://download.ceph.com/rpm-$release/el7/noarch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://download.ceph.com/keys/release.asc
EOM
 
yum install ceph-deploy -y

Let’s enable password-less login for root account – generate SSH keys and seed public key into /root/.ssh/authorized_keys file on all Ceph nodes (in production environment, you might want to use a user with limited privileges with sudo escalation):

ssh-keygen -f $HOME/.ssh/id_rsa -t rsa -N ''
ssh-copy-id root@ceph1
ssh-copy-id root@ceph2
ssh-copy-id root@ceph3

On all CEPH nodes…

Before beginning, ensure that SELINUX is set to permissive mode and verify that firewall is not blocking required connections between Ceph components:

firewall-cmd --zone=public --add-service=ceph-mon --permanent
firewall-cmd --zone=public --add-service=ceph --permanent
firewall-cmd --reload
setenforce 0

Make sure that you make SELINUX changes permanent, by editing /etc/selinux.config and setting ‘SELINUX=permissive’

As for the firewall, in case you are using different distribution or don’t consume firewalld, please refer to the networking configuration reference at http://docs.ceph.com/docs/mimic/rados/configuration/network-config-ref/

On CEPH admin node…

Let’s create cluster definition locally on admin node:

mkdir CEPH-CLUSTER; cd CEPH-CLUSTER/
ceph-deploy new ceph1 ceph2 ceph3

This will trigger a ssh connection to each of above referenced Ceph nodes (to check for machine platform and IP addresses) and will then write a local cluster definition and the MON auth key in the current folder.  Let’s check the files generated:

# ls -la
-rw-r--r-- ceph.conf
-rw-r--r-- ceph-deploy-ceph.log
-rw------- ceph.mon.keyring

On Centos7, if you get the “ImportError: No module named pkg_resources” error message while running ceph-deploy tool, you might need to install missing packages:

yum install python-setuptools

In case that you have multiple network interfaces on Ceph nodes, you will be required to explicitly define public network (which accepts client’s connections) – in this case edit previously created ceph.conf on the local admin node to include public network setting:

echo "public network = 10.2.0.0/16" >> ceph.conf

If you only have one NIC in each Ceph node, the above line is not required.

Still on admin node, let’s start the installation of Ceph binaries across cluster nodes (no services started yet):

 ceph-deploy install ceph1 ceph2 ceph3 

Command above will also output the version of Ceph binaries installed on each node – make sure that you did not get a wrong Ceph version installed due to some other repos present (we are installing Mimic 13.2.5, which is latest as of the time of writing).

Let’s create (initial) MONs on all 3 Ceph nodes:

ceph-deploy mon create-initial

In order to be able to actually manage our Ceph cluster, let’s copy over the admin key and the ceph.conf files to all Ceph nodes:

ceph-deploy admin ceph1 ceph2 ceph3

On any CEPH node…

After previous step, you should be able to issue “ceph -s” from any Ceph node, and this will return the cluster health. If you are lucky enough, your cluster will be in HEALTH_OK state, but it might happen that your MON daemons will complain on time mismatch between the nodes, as following:

[root@ceph1 ~]# ceph -w
  cluster:
    id:     7f2d23c2-1f2e-4c03-821c-cab3d76f84fc
    health: HEALTH_WARN
            clock skew detected on mon.ceph1, mon.ceph3 

In this case, we should stop NTP daemon, force time update (a few times), and start NTP daemon again – and after doing this across all nodes, it would be required to restart Ceph monitors on each node, one by one (give it a few seconds between restart on different nodes) – below we are restarting all Ceph daemons – which effectively means just MONs since we deployed only MONs so far:

systemctl stop ntpd
ntpdate -s time.nist.gov; ntpdate -s time.nist.gov; ntpdate -s time.nist.gov
systemctl start ntpd
systemctl restart ceph.target

After time has been properly synchronized (with less then 0.05 seconds of time difference between the nodes), you should be able to see a cluster in HEALTH_OK state, as below:

[root@ceph1 ~]# ceph -s
  cluster:
    id:     7f2d23c2-1f2e-4c03-821c-cab3d76f84fc
    health: HEALTH_OK

On CEPH admin node…

Now that we are up and running with all Ceph monitors, let’s deploy Ceph manager daemon (Ceph dashboard, that comes with newer releases) on all nodes since they operate in active/standby configuration (we will configure it later):

ceph-deploy mgr create ceph1 ceph2 ceph3

Finally, let’s deploy some OSDs so our cluster can actually hold some data eventually:

ceph-deploy osd create --data /dev/sdb ceph1
ceph-deploy osd create --data /dev/sdb ceph2
ceph-deploy osd create --data /dev/sdb ceph3

Note in commands above, we reference /dev/sdb as the 100GB volume that is used for OSD.

As mentioned previously, newer versions of Ceph (as in our case) will use by default BlueStore as the storage backend, with (by default) collocating block data and RocksDB key/value database (for managing its internal metadata) on the same device (/dev/sdb in our case). In more complex setups, one can choose to separate RockDB DB on faster devices, while block data will remain on slower devices – somewhat similar with the older FileStore setups, where block data would be located on HDDs/SSDs devices, while Journals would be usually placed on SSD/NVME partitions.

On any CEPH node…

After previous step is done, we should get the output similar to below – confirming that we have a 300GB of space available:

[root@ceph1 ~]# ceph -s
  cluster:
    id:     7f2d23c2-1f2e-4c03-821c-cab3d76f84fc
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum ceph2,ceph1,ceph3
    mgr: ceph1(active)
    osd: 3 osds: 3 up, 3 in

  data:
    pools:   0 pools, 0 pgs
    objects: 0  objects, 0 B
    usage:   3.0 GiB used, 297 GiB / 300 GiB avail
    pgs:  

Finally, let’s enable the Dashboard manager and set the username/password for authentication (which will be encrypted and stored in monitor’s DB) to be able to access it.
In our lab, we will disable SSL connections and keep it simple – but obviously in production environment, you would want to force SSL connections and also install proper SSL certificate:

ceph config set mgr mgr/dashboard/ssl false
ceph mgr module enable dashboard
ceph dashboard set-login-credentials admin password

Let’s login to the Dashboard manager on the active node (ceph1 in our case, as can be seen in the output from “ceph -s” command above):

And there you go – you now have a working Ceph cluster, which concludes part 1 of this Ceph article series. In part 2 (published soon), we will continue our work by creating a dedicated RBD pool and authentication keys for our CloudStack installation, add Ceph to CloudStack, finally consuming it with dedicated Compute / Disk offerings.

It’s worth mentioning that Ceph itself does provided additional services – i.e. it supports S3 object storage (requires installation / configuration of Ceph Object Gateway) as well as POSIX-compliant file system CephFS (requires installation/configuration of Metadata Server), but for CloudStack, we only need Rados Block Device (RBD) services from Ceph.

About the author

Andrija Panic is a Cloud Architect at ShapeBlue, the Cloud Specialists, and is a committer of Apache CloudStack. Andrija spends most of his time designing and implementing IaaS solutions based on Apache CloudStack.

Securing CloudStack 4.11 with HTTPS/TLS

,

Apache CloudStack is generally considered secure out of the box, however it does have the capability of protecting both system VM traffic as well as management traffic with TLS certificates. Prior to version 4.11 CloudStack used Tomcat as the web server and servlet container. With 4.11 this has been changed to embedded Jetty web server and servlet container that makes CloudStack more secure and independent of distribution provided Tomcat version which could be prone to security issues. This changes the management server’s TLS configuration.

In this blog post we’ll go through how to implement native TLS to protect both the system VMs – CPVM and SSVM – as well as the management server with public TLS certificates.

Please note since SSL is now being deprecated we now refer to the replacement – TLS – throughout this post.

Overview

The TLS configuration in CloudStack provides different functionality depending on the system role:

  • TLS protecting the management server is a Jetty configuration which protects the main web GUI and API endpoint only. The configuration of this is handled in the underlying embedded Jetty configuration files.
  • System VMs:
    • The Console Proxy VM TLS configuration provides secure HTTPS connection between the main console screen (your browser) and the CPVM itself.
    • The Secondary Storage VM SSL configuration provides a secure HTTPS connection for uploads and downloads of templates / ISOs / volumes to secondary storage as well as between zones.
    • System VM SSL configuration is carried out through global settings as well as the CloudStack GUI.

System VM HTTPS configuration

Global settings

The following global setting are required configured to allow system VM TLS configuration:

Global settingValue
consoleproxy.url.domaindomain used for CPVM (see below)
consoleproxy.sslEnabledSwitches SSL configuration of the CPVM on / off
Values true | false
secstorage.ssl.cert.domaindomain used for SSVM (see below)
secstorage.encrypt.copySwitches SSL configuration of the SSVM on / off
Values true | false

The URL configurations can take three formats – and these also determine what kind of TLS certificate is required.

  • Blank: if left blank / unconfigured the URLs used for CPVM and SSVM will simply be passed as the actual public IP addresses of the system VMs.
  • Static URL: e.g. console.mydomain.com or ssvm.mydomain.com. In these cases CloudStack rely on external URL load balancing / redirection and/or DNS resolution of the URL to the IP address of the CPVM or SSVM. This can be achieved in a number of different ways through load balancing appliances or scripted DNS updates.
    This configuration relies on:

    • The same URL used for both CPVM and SSVM, or
    • a multi-domain certificate provided to cover both URLs if different ones are used for CPVM and SSVM.
  • Dynamic URL: e.g. *.mydomain.com. In this case CloudStack will redirect the connections to the CPVM / SSVM to the URL “a-b-c-d.mydomain.com” where a/b/c/d represent the IP address, i.e. a real world URL would be 192-168-34-145.mydomain.com.
    This relies on two things:

    • DNS name resolution configured for the full public system VM IP range, such that all combinations of “a-b-c-d.mydomain.com” can be resolved. Please note in CloudStack version 4.11 the public IP range used purely by system VMs can be limited by reserving a subrange of public IP addresses just for system use.
    • An TLS wildcard certificate covering the full “mydomain.com” domain name.

Configuration process – GUI

The first step is to configure the four global settings above, then restart the CloudStack management service to make these settings live.

Next upload the TLS root certificate chain, the actual TLS certificate as well as the PKCS8 formatted private key using the “TLS certificate” button in the zone configuration. In this example we use a wildcard certificate.

Configuration process – API / CloudMonkey

The uploadCustomCertificate API call can be used to upload the TLS certificates. Please note the upload process does require at least two API calls – more depending on how many intermediary certificates are used. If you use CloudMonkey the certificates can be uploaded in cleartext – otherwise they have to be URLencoded when passed as part of a normal HTTP GET API call.

  • In the first API call the combined root and intermediary certificates are uploaded. In this API call the following parameters are passed – note we don’t pass the private key:
    • id=1
    • name: Give the certificate a name.
    • certificate: the root / intermediary certificate in cleartext, with all formatting / line breaks in place. In the example below we pass a chain of root and intermediary certificates.
    • domainsuffix: provide the suffix used, e.g. *.mydomain.com.
  • The second API call in our example uploads the site certificate. In this case we do not give the certificate a name (CloudStack automatically names this):
    • id=2
    • certificate: the issued site certificate, again in cleartext.
    • privatekey: the private key, same cleartext format as certificate.
    • domainsuffix: as above.
cloudmonkey upload customcertificate id=1 name=RootCertificate certificate='-----BEGIN CERTIFICATE-----
MIIE0DCCsdf8HqjeIHgkqhkiG9w0BAQsFADCBgzELMAk
...8V3Idv7kaWKK3245lsoynJuh87/BKONvPi8BDAB
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIEfTCCA2WgAwIBAgIDG+cVMA8djwj1ldggKd9d9s
.....mw4TRfZHcYQFHfjDCmrw==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIEADCCAuigAwIBAgIBADANBgkqhkiG9w0BAQUFA
.....csbkqWletNw+vHX/bvZ8=
-----END CERTIFICATE-----'
domainsuffix='*.mydomain.com'

cloudmonkey upload customcertificate id=2 certificate='-----BEGIN CERTIFICATE-----
MIIGrjCCBZagAwIBAgIJAJ....
...xKjPTkOLfwMVWXc8Ul25t7lkyi0+a9jZxFAuDXFRgkQnbw==
-----END CERTIFICATE-----'
privatekey='-----BEGIN PRIVATE KEY-----
MIIEvEbidvik1gkqhkiG9w0BAQEFAASCBK
.....rEF5Qyuyserre87d234jj/Uddf
-----END PRIVATE KEY-----'
domainsuffix='*.mydomain.com'

 

System VM restart

Once uploaded the CPVM and SSVM will automatically restart to pick up the new certificates. If the system VMs do not restart cleanly they can be destroyed and will come back online with the TLS configuration in place.

Testing the TLS protected system VMs

To test the Console Proxy VM simply open the console to any user VM or system VM. The popup browser window will at this point show up as a non-secure website. The reason for this is the actual console session is provided in an inline frame – and the popup page itself is presented from the unsecured management servers. If you however look at the page source for the page the HTTPS link which presents the inline frame shows up in the expected format:

Once the management server is also TLS protected the CPVM console popup window will also show as secured:

The SSVM will also now utilise the HTTPS links for all browser based uploads as well as downloads from secondary storage:

Securing the CloudStack management server GUI with HTTPS

Protecting the management servers requires creating a PCKS12 certificate and updating the underlying Jetty configuration to take this into account. This is described in http://wiki.eclipse.org/Jetty/Howto/Configure_SSL#Configuring_Jetty

First of all combine the TLS private key file, certificate and root certificate chain into one file (in the specified order), then convert this into PKCS12 format and write it to a new CloudStack keystore. Enter a new keystore password when prompted:

# cat myprivatekey.key mycertificate.crt gd_bundle-g2-g1.crt > mycombinedcert.crt

# openssl pkcs12 -in mycombinedcert.crt -export -out mycombinedcert.pkcs12
Enter Export Password: ************
Verifying - Enter Export Password:************

# keytool -importkeystore -srckeystore mycombinedcert.pkcs12 -srcstoretype PKCS12 -destkeystore /etc/cloudstack/management/keystore.pkcs12 -deststoretype pkcs12
Importing keystore mycombinedcert.pkcs12 to /etc/cloudstack/management/keystore.pkcs12...
Enter destination keystore password:************
Re-enter new password:************
Enter source keystore password:************
Entry for alias 1 successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled

 

Next edit /etc/cloudstack/management/server.properties and update the following:

https.enable=true
https.keystore=/etc/cloudstack/management/keystore.pkcs12
https.keystore.password=<enter the same password as used for conversion>

 

In addition automatic redirect from HTTP/port 8080 to HTTPS/port 8443 can also be configured in /usr/share/cloudstack-management/webapp/WEB-INF/web.xml. Add the following section before the section around line 22:

  <security-constraint>
    <web-resource-collection>
      <web-resource-name>Everything</web-resource-name>;
      <url-pattern>*</url-pattern>
    </web-resource-collection>
    <user-data-constraint>
      <transport-guarantee>CONFIDENTIAL></transport-guarantee>
    </user-data-constraint>
  </security-constraint>

 

Lastly restart the management service:

# systemctl restart cloudstack-management

Conclusion

The TLS capabilities of CloudStack has been extended over the last few releases – and we hope this article helps to explain the new global settings and configuration procedure.

About The Author

Dag Sonstebo is  a Cloud Architect at ShapeBlue, The Cloud Specialists. Dag spends most of his time designing, implementing and automating IaaS solutions based on Apache CloudStack.


CloudStack upgrades – best practices

,

Introduction

Upgrading CloudStack can sometimes be a little daunting – but as the 5P’s proverb goes – Proper Planning Prevents Poor Performance. With planning, testing and the right strategy upgrades will have a high chance of success and have minimal impact on your CloudStack end users.

The CloudStack upgrade process is documented in the release notes for each CloudStack version (e.g. version 4.6 to 4.9 upgrade guide). These upgrade notes should always be adhered to in detail since they cover both the base CloudStack upgrade as well as the required hypervisor upgrade changes. This blog article does however aim to add some advise and best practices to make the overall upgrade process easier.

The following, outlines some best practice steps which can – and should – be carried out in advance of the actual CloudStack upgrade. This ensures all steps are done accurately and without the pressure of an upgrade change window.

Strategy – in-place upgrade vs. parallel builds

The official CloudStack upgrade guides are based on an in-place upgrade process. This process works absolutely fine  – it does however have some drawbacks:

  • It doesn’t easily lend itself to technology refreshes – if you want to carry out hardware or OS level upgrades or configuration changes to your CloudStack management and MySQL servers at the time of upgrade this adds complexity and time.
  • To ensure you have rollback point in case of problems with the upgrade you need to either take VM snapshots of the infrastructure (if your management and MySQL servers are Virtual Machines) or otherwise back up all configurations.
  • In case of a rollback scenario you also have to ensure all upgraded system CloudStack logs as well as the database is backed up before reverting to any snapshots / backups – such that a post-mortem investigation can be carried out.

To get around these issues it is easier and a less risky to simple prepare and build new CloudStack management and MySQL servers in advance of the actual upgrade:

  • This means you can carry out the upgrade on brand new servers, whilst leaving the old servers offline for a much quicker rollback.
  • This also means in the case of a rollback scenario the newly built servers can simply be disabled and left for further investigation whilst the original infrastructure is brought back online.
  • Since most people run their CloudStack management infrastructure as Virtual Machines the cost of adding a handful of new VMs for this purpose is negligible compared to the benefits of a more efficient and less risky upgrade process.

By using this parallel build process the upgrade process will include a few more steps – these will be covered later in this article:

  • Upgrade starts with stopping and disabling CloudStack management on the original infrastructure.
  • The original database is then copied to the new MySQL server(s), before MySQL is disabled.
  • Once completed the new CloudStack management servers are configured to point to the new MySQL server(s).
  • The upgrade is then run on the new CloudStack management servers, pointing to the new MySQL server(s).
  • As an additional step this does require the CloudStack global “host” to be reconfigured.

New CloudStack management infrastructure build

Following the standard documented build procedure for the new / upgrade version of CloudStack:

Build new management servers but without carrying out the following steps:

  • Do not carry out the steps for seeding system templates.
  • Do not carry our the cloudstack-setup-databases step.
  • Do not run cloudstack-setup-management.

Build new MySQL servers:

  • Ensure the cloud username / password is configured.
  • Create new empty databases for cloud / cloud_usage / cloudbridge.
  • If possible configure the MySQL master-slave configuration.

 Upgrade lab testing

As for any technical project testing the upgrade prior to upgrading is important. If you don’t have a production equal lab to do this in it is worth the effort to build one.

A couple of things to keep in mind:

  • Try to make the upgrade lab as close to production as possible.
  • Make sure you use the same OS’es, the same patch level and the same software versions (especially Java and Tomcat).
  • Use the same hardware model for the hypervisors if possible. These don’t have to be the same CPU / memory spec. but should otherwise match as closely as possible.
  • Use the same storage back-ends for primary and secondary storage. If not possible at least use the same protocols (NFS, iSCSI. FC, etc.)
  • Try to match your production cloud infrastructure with regards to number of zones.
  • If your production infrastructure has multiple clusters or pods also try to match this.

Once the lab has been built make sure it is prepared with a test workload similar to production:

  • Configure a selection of test domain and accounts + users.
  • Create VM infrastructure similar to production – e.g. configure isolated and shared networks, VPCs, VPN connections etc.
  • Create a number of VMs across multiple guest OS’es – at the very minimum try to have both Windows and Linux VMs running.
  • Make sure you also have VMs with additional disks attached.
  • Keep in mind your post-upgrade testing may be destructive – i.e. you may delete or otherwise break VMs. To cover for this it is good practice to prepare a lot more VMs than you think you may need.

For the test upgrade:

  • Ensure you have rollback points – you may want run the test upgrade multiple times. An easy way is to take VM snapshots and SAN/NAS snapshots.
  • Using the parallel build process create a new management server and carry out the upgrade (see below).
  • Investigate and fix any problems encountered during the upgrade process and add this to the production upgrade plan.

Once upgraded carry out thorough functional, regression and user acceptance testing. The nature of these depend very much on the type, complexity and nature of the production cloud, but should at the very minimum include:

User actions
  • Logins with passwords created prior to the upgrade.
  • Creation of new domains / accounts / users.
  • Deletion of old and new domains / accounts / users.
All VM lifecycle actions
  • Create / delete new VMs as well as deletion of existing VMs.
  • Start / stop of new and existing VMs.
  • Additions and removals of disks.
  • Additions and removals of networks to VMs.
Network lifecycle actions
  • Creation of new isolated / shared / VPC networks.
  • Deletion of existing as well as new isolated / shared / VPC networks.
  • Connectivity testing from VMs across all network types.
Storage lifecycle actions
  • Additions and deletions of disks.
Other
  • Test any integrated systems – billing portals, custom GUIs / portals, etc.

If any problems are encountered these need to be investigated and addressed prior to the production upgrade.

Production DB upgrade test

The majority of the CloudStack upgrade is done against the underlying databases. As a result the lab testing discussed above will not highlight any issues resulting from:

  • Schema changes
  • Database or table sizes
  • Internal database constraints

To test for this it is good practice to carry out the following test steps. This process should be carried out as close in time to the actual upgrade as possible:

  • Build a completely network isolated management server using the new upgrade version of CloudStack. For this test simply use the same server for management and MySQL.
  • Import a copy of the production databases.
  • Start the upgrade by running the standard documented cloudstack-setup-databases configuration command as well as cloudstack-setup-management. The latter will complete the server configuration and start the cloudstack-management service.
  • Monitor the CloudStack logs (/var/log/cloudstack/management/management-server.log).

This will highlight any issues with the underlying SQL upgrade scripts. If any problems are encountered these need to be investigated thoroughly and fixed.

Please note – if any problems are found and subsequently fixed the original database needs to be imported again and the full upgrade restarted, otherwise the upgrade process will attempt to carry out upgrade steps which have already been completed – which in most cases will lead to additional errors.

Once all issues have been resolved and the upgrade completes without errors the time taken for the upgrade should be noted. The database upgrade process can take from minutes up to multiple hours depending on the size of the production databases – something which needs to be taken into account when planning for the production upgrade. If the upgrade takes longer than expected it may also be prudent to consider database housekeeping of event and usage tables to cut down on size and thereby speeding up the upgrade process.

To prevent re-occurrence of problems during the production upgrade it is key that all database upgrade fixes and workarounds are documented and added to the production upgrade plan.

Install system VM templates

The official upgrade documentation specifies which system VM templates to upload prior to the upgrade.

Please note – whether you do the upgrade in-place or on a parallel build the system VM templates need to be uploaded on the original CloudStack infrastructure (i.e. it can not be done on the new infrastructure when using parallel builds).

In other words – the template upload can and should be done in the days prior to the upgrade. Uploading these will not have any adverse effect on the existing CloudStack instance – since the new templates are simply uploaded in a similar fashion to any other user template upload. Carrying this out in advance will also ensure the templates should be fully uploaded by the time the production upgrade is carried out, thereby cutting down the upgrade process time.

Other things to consider prior to upgrade

  • Always make sure the production CloudStack infrastructure is healthy prior to the upgrade – an upgrade will very seldom fix underlying issues – these will however quickly cause problems during the upgrade and lead to extended downtime and rollbacks.
  • Once the upgrade has been completed it is very difficult to backport any post-upgrade user changes back to the original database in case of a rollback scenario. In other words – it is prudent to disable user access to the CloudStack infrastructure until the upgrade has been deemed a success.
  • It is good practice to at the very minimum have the CloudStack MySQL servers configured in a master-slave configuration. If this has not been done on the original infrastructure it can be configured on the newly built MySQL servers prior to any database imports, again cutting down on processing time during the actual production upgrade.

Upgrade

Step 1 – review all documentation

The official upgrade documentation is updated for each version of CloudStack (e.g. CloudStack 4.6 to 4.9 upgrade guide) – and should always be taken into consideration during an upgrade.

Step 2 – confirm all system VM templates are in place

In the CloudStack GUI make sure the previously uploaded system VM templates are successfully uploaded:

  • Status: Download complete
  • Ready: Yes

Step 3 – stop all running CloudStack services

On all existing CloudStack servers stop and disable the cloudstack-management service:

# service cloudstack-management stop;
# service cloudstack-usage stop;
# chkconfig cloudstack-management off;
# chkconfig cloudstack-usage off;

Step 4 – back up existing databases and disable MySQL

On the existing MySQL servers back up all CloudStack databases:

# mysqldump -u root -p cloud > /root/cloud-backup.sql;
# mysqldump -u root -p cloud_usage > /root/cloud_usage-backup.sql;
# mysqldump -u root -p cloudbridge > /root/cloudbridge-backup.sql;

# service mysqld stop;
# chkconfig mysqld off;

Step 5 – copy and import databases on the new MySQL master server

# scp root@<original MySQL server IP>:/root/cloud*.sql;
# mysql –u root –p cloud < cloud-backup.sql;
# mysql –u root –p cloud_usage < cloud_usage-backup.sql;
# mysql –u root –p cloudbridge < cloudbridge-backup.sql;

Step 6 – update the “host” global setting

The parallel builds method introduces new management servers whilst disabling the old ones. If no load balancers are being used the “host” global setting requires to be updated to specify which new management server the hypervisors should check into:

# mysql –p;
# update cloud.configuration set value=”<new management server IP address | new loadbalancer VIP>” where name=”host”;

Step 7 – carry out all hypervisor specific upgrade steps

Following the official upgrade documentation upgrade all hypervisors as specified.

Step 8 – configure and start the first management server

On the first management server only carry out the following steps to configure connectivity to the new MySQL (master) server and start the management service. Note the “cloudstack-setup-databases” command is executed without the “–deploy-as” option:

# cloudstack-setup-databases <cloud db username>:<cloud db password>@<cloud db host>; 
# cloudstack-setup-management;
# service cloudstack-management status;

Step 9 – monitor startup

  • Monitor the CloudStack service startup:
    # tail –f /var/log/cloudstack/management/management-server.log
  • Open the CloudStack GUI and check and confirm all hypervisors check in OK.

Step 10 – update system VMs

  • Destroy the CPVM and make sure this is recreated with the new system VM template.
  • Destroy the SSVM and again makes sure this comes back online.
  • Restart guest networks “with cleanup” to ensure all Virtual Routers are recreated with the new template.
  • Checking versions of system VMs / VRs can be done by either:
    • Checking /etc/cloudstack-release locally on each appliance.
    • By checking the vm_instance / vm_templates table in the cloud database.
    • Updating the “minreq.sysvmtemplate.version” global setting to the new system VM template version – which will show which VRs require updated in the CloudStack GUI.

Step 11 – configure and start additional management servers

Following the same steps as in step 8 above configure and start the cloudstack-management service on any additional management servers.

Rollback

Rollbacks of upgrades should only be carried out in worst case scenarios. A rollback is not an exact science and may require additional steps not covered in this article. In addition any rollback should be carried out as soon as possible after the original upgrade – otherwise the new infrastructure will simply have too many changes to VMs, networks, storage, etc. which can not be easily backported to the original database.

Step 1 – disable new CloudStack management / MySQL servers

On the new CloudStack management servers stop and disable the management service:

# service cloudstack-management stop;
# chkconfig cloudstack-management off;

 

On the new MySQL server stop and disable MySQL:

# service mysqld stop;
# chkconfig mysqld off;

Step 2 – tidy up new VMs on the hypervisor infrastructure

Using hypervisor specific tools identify all VMs (system VMs and VRs) created since the upgrade. These instances need to be deleted since the old management infrastructure and databases have no knowledge of them.

Step 3 – enable the old CloudStack management / MySQL servers:

On the original MySQL servers:

# service mysqld start;
# chkconfig mysqld on;

 

On the original CloudStack management servers:

# service cloudstack-management start;
# chkconfig cloudstack-management on;

Step 4 – restart system VMs and VRs

System VMs and VRs may restart automatically (since the original instances are gone). If these do not come online follow the procedure described in upgrade step 10 to recreate these.

Conclusion

Many CloudStack users are reluctant to upgrade their platforms – however the risk of not upgrading and falling behind on technology and security tends to be a greater risk in the long run. Following these best practices should reduce the overall workload, risk and stress of upgrades by both allowing a lot of work to be carried out upfront, as well as providing a relatively straight forward rollback procedure.

As always we’re happy to receive feedback , so please get in touch with any comments, questions or suggestions.

About The Author

Dag Sonstebo is a Cloud Architect at ShapeBlue, The Cloud Specialists. Dag spends most of his time designing, implementing and automating IaaS solutions based on on Apache CloudStack.

Recovering From a vCenter Failure

, ,

While, in my opinion VMware’s vSphere is the best performing and most stable hypervisor available, vCenter obstinately remains a single point of failure when using vSphere and it’s no different when leveraging vCenter in a CloudStack environment.  Therefore, very occasionally there is a requirement to rebuild a vCentre server which was previously running in your CloudStack environment.  Working with one of our clients we found out (the hard way) how to do this.

Replacing a Pre-Existing vCenter Instance

Recreating a vCenter Instance

The recovery steps below apply to both the Windows-based vCenter server or an appliance.

The first step, which I won’t cover in detail here, is to build a new vCenter server and attach the hosts to it.  Follow VMware standard procedure when building your new vCenter server.  To save some additional steps later on, reuse the host name and IP address of the previous vCenter server. Ensure that the permissions have been reapplied allowing CloudStack to connect at the vCentre level with full administrative privileges and that datacenter and cluster names have been recreated accurately.  When you re-add the hosts, the VMs running on those hosts will automatically be pulled into the vCentre inventory.

Networking

The environment that we were working on was using standard vSwitches; therefore the configuration of those switches was held on each of the hosts independently and pulled into the vCenter inventory when the hosts were re-added to vCenter.  Distributed vSwitches (dvSwitches) have their configuration held in the vCenter.  We did not have to deal with this complication and so the recreation of dvSwitches is not covered here.

Roll-back

The changes that are to be made can easily be undone, as they make no change and trigger no change in the physical environment.  However, it is never a bad idea to have a backup, you shouldn’t ever need to roll back the whole database, but it’s good to have a record of your before/after states.

Update vCenter Password in CloudStack

If you have changed the vCenter password, it’ll need updating in the CloudStack database in its encrypted form.  The following instructions are taken from the CloudStack documentation:

  • Generate the encrypted equivalent of your vCenter password:
$ java -classpath /usr/share/cloudstack-common/lib/jasypt-1.9.0.jar org.jasypt.intf.cli.JasyptPBEStringEncryptionCLI encrypt.sh input="_your_vCenter_password_" password="`cat /etc/cloudstack/management/key`" verbose=false
  • Store the output from this step, we need to add this in cluster_details table and vmware_data_center tables in place of the plain text password
  • Find the ID of the row of cluster_details table that you have to update:
$ mysql -u <USERNAME> -p<PASSWORD>
select * from cloud.cluster_details;
  • Update the password with the new encrypted one
update cloud.cluster_details set value = '_ciphertext_from_step_1_' where id = _id_from_step_2_;
  • Confirm that the table is updated:
select * from cloud.cluster_details;
  • Find the ID of the correct row of vmware_data_center that you want to update
select * from cloud.vmware_data_center;
  • update the plain text password with the encrypted one:
update cloud.vmware_data_center set password = '_ciphertext_from_step_1_' where id = _id_from_step_5_;
  • Confirm that the table is updated:
select * from cloud.vmware_data_center;

Reallocating the VMware Datacenters to CloudStack

The next step is to recreate the cloud.zone property within vSphere which is used to track VMware datacenters which have been connected to CloudStack.   Adding a VMware datacenter to CloudStack creates this custom attribute and sets it to “true”.  CloudStack will check for this attribute and value and show an error on the management log if it attempts to connect to a vCenter server which does not have this attribute set.  While the property can be edited using the full vSphere client, it must be created using PowerCLI utility.

Using the PowerCLI utility do the following:

Connect-VIServer <VCENTER_SERVER_IP>

(you’ll be prompted for a username and password with sufficient permissions for the vCenter Server)

New-CustomAttribute -Name "cloud.zone" -TargetType Datacenter

Get-Datacenter <DATACENTER_NAME> | Set-Annotation -CustomAttribute "cloud.zone" -Value true

Reconnecting the ESXi Hosts to CloudStack

CloudStack will now be connected to the vCenter server, but the hosts will still probably all be marked as down.  This is because vCenter uses an internal identifier for hosts, so that names and IP addresses can change, but it still can keep track of which host is which.

This identifier appears in two places in CloudStack’s database; The ‘guid’ in the cloud.host table and also in the cloud.host_details table against the ‘guid’ value. The host table can be queried as follows:

SELECT id,name,status,guid FROM cloud.host WHERE hypervisor_type = 'VMware' AND status != 'Removed';

The guid takes the form:

HostSystem<HOST_ID>@<VCENTER_NAME>

You can find the new Host ID either though the PowerCLI again, or via a utility like Robware’s RVTools.  RVTools is a very light-weight way of viewing information from a vCenter. It also allows export to csv so is very handy for manual reporting and analysis. The screen grap below shows a view called vHosts; note the Object ID.  This is vCenter’s internal ID of the host.

rvtools

Through PowerCLI you would use:

Connect-VIServer <VCENTER_SERVER_IP>;

Get-VMHost | ft -Property Name,Id -autosize

Now you can update the CloudStack tables with the new host ids for each of your hosts.

cloud.host.guid for  host ‘10.0.1.32’ on vCenter ‘10.0.1.11’ might previously have read:

HostSystem:host-99@10.0.1.11

should be updated to:

HostSystem:host-10@10.0.1.11

Remember to check the host and host_details table.

Once that you have updated the entries for all of the active hosts in the zone, you can start the management service again now and all of the hosts should reconnect, as their pointers have been fixed in the CloudStack database.

Summary

This article explains how to re-insert a lost or failed vCenter server into CloudStack.  This blog describes the use of PowerCLI to both set and retrieve values from vCenter.

This process could be used to migrate an appliance based vCenter to a Windows based environment in order to support additional capacity as well as replacing a broken vCenter installation.

About The Author

Paul Angus is VP Technology & Cloud Architect at ShapeBlue, The Cloud Specialists. He has designed and implemented numerous CloudStack environments for customers across 4 continents, based on Apache CloudstackCitrix Cloudplatform and Citrix CloudPortal.

When not building Clouds, Paul likes to create Ansible playbooks that build clouds

Recovery of VMs to new CloudStack instance

, ,

We recently came across a very unusual issue where a client had a major security breach on their network. As well as lots of other damage their CloudStack infrastructure was maliciously damaged beyond recovery. Luckily the hackers hadn’t manage to damage the backend XenServer hypervisors so they were quite happily still running user VMs and Virtual Routers, just not under CloudStack control.  The client had a good backup regime so the CloudStack database was available to recover some key CloudStack information from. As building a new CloudStack instance was the only option the challenge was now how to recover all the user VMs to such a state they could be imported into the new CloudStack instance under the original accounts.

Mapping out user VMs and VHD disks

The first challenge is to map out all the VMs and work out which VMs belong to which CloudStack account, and which VHD disks belong to which VMs. To do this first of all recover the original CloudStack database and then query the vm_instance, service_offering, account and domain  tables.

In short we are interested in:

  • VM instance ID and names
  • VM instance owner account ID and account name
  • VM instance owner domain ID and domain name
  • VM service offering, which determines the VM CPU / memory spec
  • VM volume ID, name, size, path, type (root disk or data) and state for all VM disks – root or data.

At the same time we are not interested in:

  • System VMs
  • User VMs in state “Expunging”, “Expunged”, “Destroyed” or “Error”. The “Error” state would indicate the VM was not healthy on the original infrastructure.
  • VM disk volumes which are in state “Expunged” or “Expunging”. Both of these would indicate the VM was in the process of being deleted on the original CloudStack instance.

From a SQL point of view we do this as follows:

SELECT
cloud.vm_instance.id as vmid,
cloud.vm_instance.name as vmname,
cloud.vm_instance.instance_name as vminstname,
cloud.vm_instance.display_name as vmdispname,
cloud.vm_instance.account_id as vmacctid,
cloud.account.account_name as vmacctname,
cloud.vm_instance.domain_id as vmdomainid,
cloud.domain.name as vmdomname,
cloud.vm_instance.service_offering_id as vmofferingid,
cloud.service_offering.speed as vmspeed,
cloud.service_offering.ram_size as vmmem,
cloud.volumes.id as volid,
cloud.volumes.name as volname,
cloud.volumes.size as volsize,
cloud.volumes.path as volpath,
cloud.volumes.volume_type as voltype,
cloud.volumes.state as volstate
FROM cloud.vm_instance
right join cloud.service_offering on (cloud.vm_instance.service_offering_id=cloud.service_offering.id)
right join cloud.volumes on (cloud.vm_instance.id=cloud.volumes.instance_id)
right join cloud.account on (cloud.vm_instance.account_id=cloud.account.id)
right join cloud.domain on (cloud.vm_instance.domain_id=cloud.domain.id)
where
cloud.vm_instance.type='User'
and not (cloud.vm_instance.state='Expunging' or cloud.vm_instance.state='Destroyed' or cloud.vm_instance.state='Error')
and not (cloud.volumes.state='Expunged' or cloud.volumes.state='Expunging')
order by cloud.vm_instance.id;

This will return a list of VMs and disks like the following:

vmidvmnamevminstnamevmdispnamevmacctidvmacctnamevmdomainidvmdomnamevmofferingidvmspeedvmmemvolidvolnamevolsizevolpathvoltypevolstate
24rootvm1i-2-24-VMrootvm12admin1ROOT150051230ROOT-242147483648034c8b964-4ecb-4463-9535-40afc0bd2117ROOTReady
25ppvm2i-5-25-VMppvm25peterparker2SPDM Inc150051231ROOT-252147483648010c12a4f-7bf6-45c4-9a4e-c1806c5dd54aROOTReady
26ppvm3i-5-26-VMppvm35peterparker2SPDM Inc150051232ROOT-26214748364807046409c-f1c2-49db-ad33-b2ba03a1c257ROOTReady
26ppvm3i-5-26-VMppvm35peterparker2SPDM Inc150051236ppdatavol5368709120b9a51f4a-3eb4-4d17-a36d-5359333a5d71DATADISKReady

This now gives us all the information required to import the VM into the right account once the VHD disk file has been recovered from the original primary storage pool.

Recovering VHD files

Using the information from the database query above we now know that e.g. the VM “ppvm3”:

  • Is owned by the account “peterparker” in domain “SPDM Inc”.
  • Used to have 1 vCPU @ 500MHz and 500MB vRAM.
  • Had two disks:
    • A root disk with ID 7046409c-f1c2-49db-ad33-b2ba03a1c257.
    • A data disk with ID b9a51f4a-3eb4-4d17-a36d-5359333a5d71.

If we now check the original primary storage repository we can see these disks:

-rw-r--r-- 1 root root 502782464 Nov 17 12:08 7046409c-f1c2-49db-ad33-b2ba03a1c257.vhd
-rw-r--r-- 1 root root 13312 Nov 17 11:49 b9a51f4a-3eb4-4d17-a36d-5359333a5d71.vhd

This should in theory make recovery easy. Unfortunately due to the nature of XenServer VHD disk chains it’s not that straight forward. If we tried to import the root VHD disks as a template this would succeed, but as soon as we try to spin up a new VM from this template we get the “insufficient resources” error from CloudStack. If we trace this back in the CloudStack management log or XenServer SMlog we will most likely find an error along the lines of “Got exception SR_BACKEND_FAILURE_65 ; Failed to load VDI”. The root cause of this is we have imported a VHD differencing disk, or in common terms a delta or “child” disk. These reference a parent VHD disk – which we so far have not recovered.

To fully recover healthy VHD images we have two options:

  1. If we have access to the original storage repository from a running XenServer we can use the “xe” command line tools to export each VDI image. This method is preferable as it involves less copy operations and less manual work.
  2. If we have no access from a running XenServer we can copy the disk images and use the “vhd-util” utility to merge files.

Recovery using XenServer

VHD file export using the built in XenServer tools is relatively straight forward. The “xe vdi-export” tool can be used to export and merge the disk in a single operation. The first step in the process is to map an external storage repository to the XenServer (normally the same repository which is used for upload of the VHD images to CloudStack later on), e.g. an external NFS share.

We now use the vdi-export option as follows:

# xe vdi-export uuid=7046409c-f1c2-49db-ad33-b2ba03a1c257 format=vhd filename=ppvm3root.vhd --progress
[|] ######################################################&amp;gt; (100% ETA 00:00:00)
Total time: 00:01:12
# xe vdi-export uuid=b9a51f4a-3eb4-4d17-a36d-5359333a5d71 format=vhd filename=ppvm3data.vhd --progress
[\] ######################################################&gt; (100% ETA 00:00:00)
Total time: 00:00:00
# ll
total 43788816
-rw------- 1 root root 12800 Nov 18 2015 ppvm3data.vhd
-rw------- 1 root root 1890038784 Nov 18 2015 ppvm3root.vhd

If we now utilise vhd-util to scan the disks we see they are both dynamic disks with no parent:

# vhd-util read -p -n ppvm3root.vhd
VHD Footer Summary:
-------------------
Cookie : conectix
Features : (0x00000002) &lt;RESV&gt;
File format version : Major: 1, Minor: 0
Data offset : 512
Timestamp : Sat Jan 1 00:00:00 2000
Creator Application : 'caml'
Creator version : Major: 0, Minor: 1
Creator OS : Unknown!
Original disk size : 20480 MB (21474836480 Bytes)
Current disk size : 20480 MB (21474836480 Bytes)
Geometry : Cyl: 41610, Hds: 16, Sctrs: 63
: = 20479 MB (21474754560 Bytes)
Disk type : Dynamic hard disk
Checksum : 0xffffefb4|0xffffefb4 (Good!)
UUID : 4fc66aa3-ad5e-44e6-a4e2-b7e90ae9c192
Saved state : No
Hidden : 0

VHD Header Summary:
-------------------
Cookie : cxsparse
Data offset (unusd) : 18446744073709
Table offset : 2048
Header version : 0x00010000
Max BAT size : 10240
Block size : 2097152 (2 MB)
Parent name :
Parent UUID : 00000000-0000-0000-0000-000000000000
Parent timestamp : Sat Jan 1 00:00:00 2000
Checksum : 0xfffff44d|0xfffff44d (Good!)

# vhd-util read -p -n ppvm3data.vhd
VHD Footer Summary:
-------------------
Cookie : conectix
Features : (0x00000002) &lt;RESV&gt;
File format version : Major: 1, Minor: 0
Data offset : 512
Timestamp : Sat Jan 1 00:00:00 2000
Creator Application : 'caml'
Creator version : Major: 0, Minor: 1
Creator OS : Unknown!
Original disk size : 5120 MB (5368709120 Bytes)
Current disk size : 5120 MB (5368709120 Bytes)
Geometry : Cyl: 10402, Hds: 16, Sctrs: 63
: = 5119 MB (5368430592 Bytes)
Disk type : Dynamic hard disk
Checksum : 0xfffff16b|0xfffff16b (Good!)
UUID : 03fd60a4-d9d9-44a0-ab5d-3508d0731db7
Saved state : No
Hidden : 0

VHD Header Summary:
-------------------
Cookie : cxsparse
Data offset (unusd) : 18446744073709
Table offset : 2048
Header version : 0x00010000
Max BAT size : 2560
Block size : 2097152 (2 MB)
Parent name :
Parent UUID : 00000000-0000-0000-0000-000000000000
Parent timestamp : Sat Jan 1 00:00:00 2000
Checksum : 0xfffff46b|0xfffff46b (Good!)

# vhd-util scan -f -m'*.vhd' -p
vhd=ppvm3data.vhd capacity=5368709120 size=12800 hidden=0 parent=none
vhd=ppvm3root.vhd capacity=21474836480 size=1890038784 hidden=0 parent=none

These files are now ready for upload to the new CloudStack instance.

Note: using the Xen API it is also in theory possible to download / upload a VDI image straight from XenServer, using the “export_raw_vdi” API call. This can be achieved using a URL like:

https://<account>:<password>@<XenServer IP or hostname>/export_raw_vdi?vdi=<VDI UUID>&format=vhd

At the moment this method unfortunately doesn’t download the VHD file as a sparse disk image, hence the VHD image is downloaded in it’s full original disk size, which makes this very space hungry method. It is also a relatively new addition to the Xen API and is marked as experimental. More information can be found on http://xapi-project.github.io/xen-api/snapshots.html.

Recovery using vhd-util

If all we have access to is the original XenServer storage repository we can utilise the “vhd-util” binary which can be downloaded from http://download.cloud.com.s3.amazonaws.com/tools/vhd-util (note this is a slightly different version to the one built in to XenServer).

If we run this with the “read” option we can find out more information about what kind of disk this is and if it has a parent. For the root disk this results in the following information:

# vhd-util read -p -n 7046409c-f1c2-49db-ad33-b2ba03a1c257.vhd
VHD Footer Summary:
-------------------
Cookie : conectix
Features : (0x00000002) &amp;amp;lt;RESV&amp;amp;gt;
File format version : Major: 1, Minor: 0
Data offset : 512
Timestamp : Sun Nov 15 23:06:44 2015
Creator Application : 'tap'
Creator version : Major: 1, Minor: 3
Creator OS : Unknown!
Original disk size : 20480 MB (21474836480 Bytes)
Current disk size : 20480 MB (21474836480 Bytes)
Geometry : Cyl: 41610, Hds: 16, Sctrs: 63
: = 20479 MB (21474754560 Bytes)
Disk type : Differencing hard disk
Checksum : 0xffffefe6|0xffffefe6 (Good!)
UUID : 2a2cb4fb-1945-4bad-9682-6ea059e64598
Saved state : No
Hidden : 0

VHD Header Summary:
-------------------
Cookie : cxsparse
Data offset (unusd) : 18446744073709
Table offset : 1536
Header version : 0x00010000
Max BAT size : 10240
Block size : 2097152 (2 MB)
Parent name : cfe206b7-cacb-4291-a1c0-f248ff53ed4b.vhd
Parent UUID : f6f05652-20fa-4f5f-9784-d41734489b32
Parent timestamp : Fri Nov 13 12:16:48 2015
Checksum : 0xffffd82b|0xffffd82b (Good!)

From the above we notice two things about the root disk:

  • Disk type : Differencing hard disk
  • Parent name : cfe206b7-cacb-4291-a1c0-f248ff53ed4b.vhd

I.e. the VM root disk is a delta disk which relies on parent VHD disk cfe206b7-cacb-4291-a1c0-f248ff53ed4b.vhd.

If we run this against the data disk the story is slightly different:

# vhd-util read -p -n b9a51f4a-3eb4-4d17-a36d-5359333a5d71.vhd
VHD Footer Summary:
-------------------
Cookie : conectix
Features : (0x00000002) &amp;amp;lt;RESV&amp;amp;gt;
File format version : Major: 1, Minor: 0
Data offset : 512
Timestamp : Mon Nov 16 01:52:51 2015
Creator Application : 'tap'
Creator version : Major: 1, Minor: 3
Creator OS : Unknown!
Original disk size : 5120 MB (5368709120 Bytes)
Current disk size : 5120 MB (5368709120 Bytes)
Geometry : Cyl: 10402, Hds: 16, Sctrs: 63
: = 5119 MB (5368430592 Bytes)
Disk type : Dynamic hard disk
Checksum : 0xfffff158|0xfffff158 (Good!)
UUID : 7013e511-b839-4504-ba88-269b2c97394e
Saved state : No
Hidden : 0

VHD Header Summary:
-------------------
Cookie : cxsparse
Data offset (unusd) : 18446744073709
Table offset : 1536
Header version : 0x00010000
Max BAT size : 2560
Block size : 2097152 (2 MB)
Parent name :
Parent UUID : 00000000-0000-0000-0000-000000000000
Parent timestamp : Sat Jan 1 00:00:00 2000
Checksum : 0xfffff46d|0xfffff46d (Good!)

In other words the data disk is showing up with:

  • Disk type : Dynamic hard disk
  • Parent name : <blank>

This behaviour is typical for VHD disk chains. The root disk is created from an original template file, hence it has a parent disk, whilst the data disk was just created as a raw storage disk, hence has no parent.

Before moving forward with the recovery it is very important to make copies of both the differencing disks and parent disk to a separate location for further processing. 

The full recovery of the VM instance root disk relies on the differencing disk being coalesced or merged into the parent disk – but since the parent disk was a template disk it is used by a number of differencing disks the coalesce process will change this parent disk and render any other differencing disks unrecoverable.

Once we have copied the root VHD disk and it’s parent disk to a separate location we use the vhd-util “scan” option to verify we have all disks in the disk chain. The”scan” option will show an indented list of disks which gives a tree like view of disks and parent disks.

Please note if the original VM had a number of snapshots there might be more than two disks in the chain. If so use the process above to identify all the differencing disks and download them to the same folder.

# vhd-util scan -f -m'*.vhd' -p
vhd=cfe206b7-cacb-4291-a1c0-f248ff53ed4b.vhd capacity=21474836480 size=1758786048 hidden=1 parent=none
vhd=7046409c-f1c2-49db-ad33-b2ba03a1c257.vhd capacity=21474836480 size=507038208 hidden=0 parent=cfe206b7-cacb-4291-a1c0-f248ff53ed4b.vhd

Once all differencing disks have been copied we can now use the vhd-util “coalesce” option to merge the child difference disk(s) into the parent disk:

# ls -l
-rw-r--r-- 1 root root 507038208 Nov 17 12:45 7046409c-f1c2-49db-ad33-b2ba03a1c257.vhd
-rw-r--r-- 1 root root 1758786048 Nov 17 12:47 cfe206b7-cacb-4291-a1c0-f248ff53ed4b.vhd
# vhd-util coalesce -n 7046409c-f1c2-49db-ad33-b2ba03a1c257.vhd
# ls -l
-rw-r--r-- 1 root root 507038208 Nov 17 12:45 7046409c-f1c2-49db-ad33-b2ba03a1c257.vhd
-rw-r--r-- 1 root root 1863848448 Nov 17 13:36 cfe206b7-cacb-4291-a1c0-f248ff53ed4b.vhd

Note the vhd-util coalesce option has no output. Also note the size change of the parent disk cfe206b7-cacb-4291-a1c0-f248ff53ed4b.vhd.

Now the root disk has been merged it can be uploaded as a template to CloudStack to allow build of the original VM.

Import of VM into new CloudStack instance

We now have all details for the original VM:

  • Owner account details
  • VM virtual hardware specification
  • Merged root disk
  • Data disk

The import process is now relatively straight forward. For each VM:

  1. Ensure the account is created.
  2. In the context of the account (either via GUI or API):
    1. Import the root disk as a new template.
    2. Import the data disk as a new volume.
  3. Create a new instance from the uploaded template.
  4. Once the new VM instance is online attach the uploaded data disk to the VM.

In a larger CloudStack estate the above process is obviously both time consuming and resource intensive, but can to a certain degree be automated. As long as the VHD files were healthy to start off with it will however allow for successful recovery of XenServer based VMs between CloudStack instances.

About The Author

Dag Sonstebo is  a Cloud Architect at ShapeBlue, The Cloud Specialists. Dag spends most of his time designing and implementing IaaS solutions based on on Apache CloudStack.

 

System VM Upgrades

,

Recently we’ve seen a few clients being tripped up by the System VM upgrades during the CloudStack upgrades in multi-zone deployments.

The issue occurs when the System VM is pre-deployed individually to multiple zones, rather than being a single template deployed to multiple zones. This may be done in error or because the specific version of CloudStack/CloudPlatform which you are upgrading from does not allow you to provision the template as zone-wide.

The result is that only the system VMs in one zone recreate properly after the upgrade and the other zones do not recreate the SSVM or CPVM when they are destroyed.

The vm_template table will have entries like this for a given System VM template:

 SELECT id,name,type,cross_zones,state FROM cloud.vm_template WHERE name like '%systemvm-xenserver%' AND removed IS NULL; 

idnametypecross-zonesstate
634 systemvm-xenserver-4.5SYSTEM 0Active
635 systemvm-xenserver-4.5SYSTEM 0Active
636 systemvm-xenserver-4.5SYSTEM 0Active

And template_store_ref:

 SELECT template_store_ref.id,template_store_ref.store_id,template_store_ref.template_id,image_store.data_center_id,image_store.name FROM template_store_ref JOIN image_store ON template_store_ref.store_id=image_store.id WHERE template_store_ref.destroyed=0; 

idstore_idtemplate_iddata_center_idname
216341
326352
436363

The tables should actually show the same template ID in each store:

idnametypecross_zonesstate
634systemvm-xenserver-4.5SYSTEM1Active

and template_store_ref:

idstore_idtemplate_iddata_center_idname
21634 1
32634 2
43634 3

In the case of the first table, CloudStack will take the template with id of 636 as being THE region-wide System VM template, and will not be able to find it in zones 1 and 2. and therefore not be able to recreate System VMs in those zones.  The management server log will report that the zone is not ready as it can’t find the template.

The above example assumes that there are 3 zones, with 1 secondary storage pool in each zone, if there are multiple secondary storage pools, then only one in each zone will have initially received the template.

You could edit the path in the template_store_ref to point to the location where the template has been downloaded to on the secondary storage, however, this will leave the system in a non-standard state.

The safest way forward is to set one of the templates to cross_zones=1 and then coax CloudStack into creating an SSVM in each zone (one at a time), which will then download that cross-zone template.

Remediation

In this example I’ll use template 634 to be the ultimate region-wide template.

In order to download the cross_zone template we need to make one template the region-wide template at a time. So assuming that zone relating to secondary storage pool 3 upgraded properly, we first make template 635 the region-wide template. By changing the database to this:

UPDATE cloud.vm_template SET state='Inactive', cross_zones=1 WHERE id=634; UPDATE cloud.vm_template SET state='Active' WHERE id=635; UPDATE cloud.vm_template SET state='Inactive' WHERE id=636;

idnametypecross_zonesstate
634systemvm-xenserver-4.5SYSTEM 1Inactive
635systemvm-xenserver-4.5SYSTEM 0Active
636systemvm-xenserver-4.5SYSTEM 0Inactive

Zone 2 will now be able to recreate the CPVM and SSVM, while zone 3 CPVM and SSVM are already running so are unaffected.
Once the CPVM and SSVM in zone 2 are up, you can now make template 634 the region-wide system VM template, by editing the database like this.

UPDATE cloud.vm_template SET state='Active' WHERE id=634; UPDATE cloud.vm_template SET state='Inactive' WHERE id=635;

idnametypecross_zonestate
634systemvm-xenserver-4.5SYSTEM1Active
635systemvm-xenserver-4.5SYSTEM0Inactive
636systemvm-xenserver-4.5SYSTEM0Inactive

The CPVM and SSVM in zone 1 will now be able to start.

The SSVM in zones 2 and 3 will need their cloud service restarting in order to prompt the re-scanning of the templates table. This can be done by a restart or stopping then starting the cloud service on the SSVM – DO NOT DESTROY the SSVM in zones 2 or 3 as they don’t have the correct template yet. The SSVMs in zones 2 and 3 will now start downloading template id 634 to their secondary storage pools.

Once the template 634 has downloaded to zones 2 and 3 (it was already in zone 1), CloudStack will be able to recreate system VMs in any zones.

Summary

This article explains how to recover from the upgrade failure scenario when system VMs will not recreate in zones in a multi-zone deployment.

About The Author

Paul Angus is VP Technology & Cloud Architect at ShapeBlue, The Cloud Specialists. He has designed and implemented numerous CloudStack environments for customers across 4 continents, based on Apache Cloudstack ,Citrix Cloudplatform and Citrix Cloudportal.

When not building Clouds, Paul likes to create Ansible playbooks that build clouds

How to Deploy High Availability CloudStack with MariaDB Galera Cluster

,
As a cloud infrastructure scales to hundreds or thousands of servers, high availability becomes a key requirement of the production environments supporting multiple applications and services. Since the management servers use a MySQL database to store the state of all its objects, the database could become a single point of failure. The CloudStack manual recommends MySQL replication with manual failover in the event of database loss.

We have worked with Severalnines to produce what we believe is a better way.

In this blog post, we’ll show you how to deploy redundant CloudStack management servers with MariaDB Galera Cluster on CentOS 6.5 64bit. We will have two load balancer nodes fronting the management servers and the database servers. Since CloudStack relies on MySQL’s GET_LOCK and RELEASE LOCK, which are not supported by Galera, we will redirect all database requests to only one MariaDB node and automatically failover to another node in case the former goes down. So, we’re effectively getting the HA benefits of Galera clustering (auto-failover, full consistency between DB nodes, no slave lag), while avoiding the Galera limitations as we’re not concurrently accessing all the nodes. We will deploy a two-node Galera Cluster (plus an arbitrator on a separate ClusterControl node).

Our setup will look like this:

Note that this blog post does not cover the installation of hypervisor and storage hosts. Our setup consists of 4 servers:

  • lb1: HAproxy + keepalived (master)
  • lb2: HAproxy + keepalived (backup) + ClusterControl + garbd
  • mgm1: CloudStack Management + database server
  • mgm2: CloudStack Management + database server

 

Our main steps would be:

  1. Prepare 4 hosts
  2. Deploy MariaDB Galera Cluster 10.x with garbd onto mgm1, mgm2 and lb2 from lb2
  3. Configure Keepalived and HAProxy for database and CloudStack load balancing
  4. Install CloudStack Management #1
  5. Install CloudStack Management #2

 

Preparing Hosts

1. Add the following hosts definition in /etc/hosts of all nodes:

192.168.1.10		virtual-ip mgm.cloustack.local
192.168.1.11		lb1 haproxy1
192.168.1.12		lb2 haproxy2 clustercontrol
192.168.1.21		mgm1.cloudstack.local mgm1 mysql1
192.168.1.22		mgm2.cloudstack.local mgm2 mysql2

2. Install NTP daemon:

$ yum -y install ntp
$ chkconfig ntpd on
$ service ntpd start

3. Ensure each host is using a valid FQDN, for example on mgm1:

$ hostname --fqdn
mgm1.cloudstrack.local

Deploying MariaDB Galera Cluster

** The deployment of the database cluster will be done from lb2, i.e., the ClusterControl node.

1. To set up MariaDB Galera Cluster, go to the Severalnines Galera Configurator to generate a deployment package. In the wizard, we used the following values when configuring our database cluster (take note that we specified one of the DB nodes twice under Database Servers’ textbox):

Vendor                   : MariaDB
MySQL Version            : 10.x
Infrastructure           : none/on-premises 
Operating System         : RHEL6 - Redhat 6.4/Fedora/Centos 6.4/OLN 6.4/Amazon AMI 
Number of Galera Servers : 3
Max connections	     	 : 350
OS user                  : root
ClusterControl Server    : 192.168.1.12
Database Servers         : 192.168.1.21 192.168.1.22 192.168.1.22

At the end of the wizard, a deployment package will be generated and emailed to you.

2. Download and extract the deployment package:

$ wget http://www.severalnines.com/galera-configurator3/tmp/wb06494200669221809/s9s-galera-mariadb-3.5.0-rpm.tar.gzz
$ tar -xzf s9s-galera-mariadb-3.5.0-rpm.tar.gz

3. Before we proceed with the deployment, we need to perform some customization to fit the CloudStack database environment. Go to the deployment script’s MySQL configuration file at ~/s9s-galera-mariadb-3.5.0-rpm/mysql/config/my.cnf and ensure the following options exist under the [MYSQLD] section:

innodb_rollback_on_timeout=1
innodb_lock_wait_timeout=600
log-bin=mysql-bin

4. Then, go to ~/s9s-galera-mariadb-3.5.0-rpm/mysql/config/cmon.cnf.controller and remove the repeated value on mysql_server_addresses so it becomes as below:

mysql_server_addresses=192.168.1.21,192.168.1.22

5. Now we are ready to start the deployment:

$ cd ~/s9s-galera-codership-3.5.0-rpm/mysql/scripts/install/
$ bash ./deploy.sh 2>&1 | tee cc.log

6. The DB cluster deployment will take about 15 minutes, and once completed, the ClusterControl UI is accessible at https://192.168.1.12/clustercontrol .

7. It is recommended to run Galera on at least three nodes. So, install garbd, a lightweight arbitrator daemon for Galera on the ClusterControl node from the ClusterControl UI. Go to Manage > Load Balancer > Install Garbd > choose the ClusterControl node IP address from the dropdown > Install Garbd.

You will now see your MariaDB Galera Cluster with garbd installed and binlog enabled (master) as per below:

 

Load Balancer and Virtual IP

1. Before we start to deploy the load balancers, make sure lb1 is accessible using passwordless SSH from ClusterControl/lb2. On lb2, copy the SSH keys to 192.168.1.11:

$ ssh-copy-id -i ~/.ssh/id_rsa root@192.168.1.11

2. Login to ClusterControl, drill down to the database cluster and click Add Load Balancer button. Deploy HAProxy on lb1 and lb2 similar to below:

** Take note that for RHEL, ensure you check Build from source? to install HAProxy from source. This will install the latest version of HAProxy.

3. Install Keepalived on lb1(master) and lb2(backup) with 192.168.1.10 as virtual IP:

4. The load balancer nodes have now been installed, and are integrated with ClusterControl. You can verify this by checking out the ClusterControl summary bar:

5. By default, our script will configure the MySQL reverse proxy service to listen on port 33306 in active-active mode. We need to change this to active-passive multi master mode by declaring the second Galera node as backup, On lb1 and lb2, open /etc/haproxy/haproxy.cfg and append the word ‘backup’ into the last line:

	server 192.168.1.21 192.168.1.21:3306 check
	server 192.168.1.22 192.168.1.22:3306 check backup

6. We also need to add the load balancing definition for CloudStack. According to the documentation, we need to load balance port 8080 and 8025. To allow session stickiness, we will use source load balancing algorithm, where the same source address will be forwarded to the same management server unless it fails. On lb1 and lb2, open/etc/haproxy/haproxy.cfg and add the following lines:

frontend cloudstack_ui_8080
        bind *:8080
        mode http
        option httpchk OPTIONS /client
        option forwardfor
        option httplog
        balance source
        server mgm1.cloudstack.local 192.168.1.21:8080 maxconn 32 check inter 5000
        server mgm2.cloudstack.local 192.168.1.22:8080 maxconn 32 check inter 5000
frontend cloudstack_systemvm_8250
        bind *:8250
        mode tcp
        balance source
        server mgm1.cloudstack.local 192.168.1.21:8250 maxconn 32 check
        server mgm2.cloudstack.local 192.168.1.22:8250 maxconn 32 check

6. Restart HAProxy to apply the changes:

$ service haproxy restart

Or, you can just kill the haproxy process and let ClusterControl recover it.

7. Configure iptables to allow connections to port configured in HAProxy and Keepalived. Add the following lines:

$ iptables -I INPUT -m tcp -p tcp --dport 33306 -j ACCEPT
$ iptables -I INPUT -m tcp -p tcp --dport 8080 -j ACCEPT
$ iptables -I INPUT -m tcp -p tcp --dport 8250 -j ACCEPT
$ iptables -I INPUT -m tcp -p tcp --dport 80 -j ACCEPT
$ iptables -I INPUT -m tcp -p tcp --dport 443 -j ACCEPT
$ iptables -I INPUT -m tcp -p tcp --dport 9600 -j ACCEPT
$ iptables -I INPUT -i eth0 -d 224.0.0.0/8 -j ACCEPT
$ iptables -I INPUT -p 112 -i eth0 -j ACCEPT
$ iptables -I OUTPUT -p 112 -o eth0 -j ACCEPT

Save the iptables rules:

$ service iptables save

Installing CloudStack Management Server #1

** The following steps should be performed on mgm1.

1. Add the CloudStack repository, create /etc/yum.repos.d/cloudstack.repo and insert the following information.

[cloudstack]
name=cloudstack
baseurl=http://packages.shapeblue.com/cloudstack/upstream/centos/4.4/
enabled=1
gpgcheck=0

2. Install the CloudStack Management server:

$ yum -y install cloudstack-management

3. Create a root user with wildcard host for the CloudStack database setup. On mgm1, run the following statement:

$ mysql -u root -p

And execute the following statements:

MariaDB [(none)]> GRANT ALL PRIVILEGES ON *.* TO root@'%' IDENTIFIED BY 'password' WITH GRANT OPTION;
MariaDB [(none)]> FLUSH PRIVILEGES;

4. On mgm1, run the following command to configure the CloudStack databases:

$ cloudstack-setup-databases cloud:cloudpassword@192.168.1.10:33306 --deploy-as=root:password

5. Setup the CloudStack management application:

$ cloudstack-setup-management

** Allow some time for the CloudStack application to bootstrap on each startup. You can monitor the process at/var/log/cloudstack/management/catalina.out.

6. Open the CloudStack management UI at virtual IP, http://192.168.1.10:8080/client/ with default user ‘admin’ and password ‘password’. Configure your CloudStack environment by following the deployment wizard and let CloudStack build the infrastructure:

If completed successfully, you should then be redirected to the CloudStack Dashboard:

The installation of the first management server is now complete. We’ll now proceed with the second management server.

 

Installing CloudStack Management Server #2

** The following steps should be performed on mgm2

1. Add the CloudStack repository, create /etc/yum.repos.d/cloudstack.repo and insert the following information.

[cloudstack]
name=cloudstack
baseurl=http://cloudstack.apt-get.eu/rhel/4.4/
enabled=1
gpgcheck=0

2. Install the CloudStack Management server:

$ yum -y install cloudstack-management

3. Run the following command to setup the CloudStack database (note the absence of –deploy-as argument):

$ cloudstack-setup-databases cloud:cloudpassword@192.168.1.10:33306

4. Setup the CloudStack management application:

$ cloudstack-setup-management

** Allow some time for the CloudStack application to bootstrap on each startup. You can monitor the process at/var/log/cloudstack/management/catalina.out. At this point, this management server will automatically discover the other management server and form a cluster. Both management servers are load balanced and accessible via virtual IP, 192.168.1.10.

Lastly, change the management host IP address on every agent host at/etc/cloudstack/agent/agent.properties to the virtual IP address similar to below:

host=192.168.1.10

Restart the cloudstack agent service to apply the change:

$ service cloudstack-agent restart

Verify the Setup

1. Check the HAProxy statistics by logging into the HAProxy admin page at lb1 host port 9600. The default username/password is admin/admin. You should see the status of nodes from the HAProxy point-of-view. Our Galera cluster is in master-standby mode, while the CloudStack management servers are load balanced:

2. Check and observe the traffic on your database cluster from the ClusterControl overview page at https://192.168.1.12/clustercontrol

Reproduced with the kind permission of Severalnines
http://www.severalnines.com/blog/how-deploy-high-availability-cloudstack-mariadb-galera-cluster

 

How to Build a CloudStack Test Environment using VirtualBox

, , ,

Introduction

If you are new to Apache CloudStack and want to learn the concepts but do not have all the equipment required to stand-up a test environment, why not use your existing PC and VirtualBox.

VirtualBox is a cross platform virtualisation application which runs on OSX, Windows, Linux and Solaris, meaning no matter what OS you are running, you should be able to run VirtualBox.

The aim of this exercise is to build an Apache CloudStack environment which is as close to a Production deployment as possible, within the obvious constraints of running it all on a laptop. This deployment will support the following key functions of Apache CloudStack:

Production Grade Hypervisor: Citrix XenServer 6.2 with full VLAN support
Apache CloudStack on CentOS 6.5
NFS for Primary and Secondary Storage – each on a dedicated VLAN
Console Proxy and Secondary Storage VM
All Advanced Networking features such as Firewall, NAT, Port Forwarding, Load Balancing, VPC

To achieve all of this we need to deploy two VMs on VirtualBox, a CentOS VM for Apache Cloudstack, and a Citrix XenServer VM for our Hypervisor. The CloudStack VM will also act as our MySQL Server and NFS Server.

appliance

A key requirement of this test environment is to keep it completely self-contained so it can be used for training (insert link to Bootcamp) and demos etc.  To achieve this, and maintain the ability to deploy a new Zone and download the example CentOS Template to use in the system, we simulate the CloudStack Public Network and host the Default CentOS Template on the CloudStack Management Server VM using NGINX.

VirtualBox Configuration

Download and install the appropriate version from https://www.virtualbox.org/wiki/Downloads

Once VirtualBox is installed we need to configure it ready for this environment. The defaults are used where possible, but if you have been using VirtualBox already, you may have different settings which need to be adjusted.

We will be using three ‘Host Only’ networks, one ‘Nat’ network, and an ‘Internal’ network. By default VirtualBox has only one ‘Host Only’ network so we need to create two more.

  1. From the ‘file’ menu (windows) or VirtualBox menu (OSX), select ‘Preferences’ then ‘Network’ then ‘Host-only Networks’
  2. Add two more networks so you have at least 3 which we can use
  3. Setup the IP Schema for the 1st two networks as follows:

The naming conventions for Host Only Networks differs depending on the Host OS, I will simply refer to these as

‘Host Only Network 1’, 2 and 3 etc so please refer to the following comparison matrix to identify the correct Network.

This Guide Windows OSX
 Host Only Network 1    VirtualBox Host Only Ethernet Adapter  vboxnet0
 Host Only Network 2  VirtualBox Host Only Ethernet Adapter #2    vboxnet1
 Host Only Network 3  VirtualBox Host Only Ethernet Adapter #3  vboxnet2

Host Only Network 1:

IPv4 Address: 192.168.56.1
IPv4 Network Mask: 255.255.255.0

DHCP Server is optional as we don’t use it, but ensure the range does not clash with the static IPs we will be using which are 192.168.56.11 & 192.168.56.101

Host Only Network 2:

IPv4 Address: 172.30.0.1
IPv4 Network Mask: 255.255.255.0

By setting up these IP ranges, we ensure our host laptop has an IP on these Networks so we can access the VMs connected to them. We don’t need an IP on ‘Host Only Network 3’ as this will be used for storage and will also be running VLANs.

We use a NAT Network so that we can connect the CloudStack Management VM to the internet to enable the installation of the various packages we will be using.

Configure the VirtualBox ‘NatNetwork’ to use the following settings:

Network Name: NatNetwork
Network CIDR: 10.0.2.0/24

We disable DHCP as we cannot control the range to exclude our statically assigned IPs on our VMs.

Whilst this article focuses on creating a single CloudStack Management Server, you can easily add a second, and I have found that the DHCP allocated IPs from the NAT Network can change randomly, so setting up NAT Rules can be problematic, hence I always use statically assigned IPs.

The ‘Internal’ Network requires no configuration.

CloudStack VM

Create a VM for CloudStack Manager using the following Settings:

Name: CSMAN 4.4.1
Type: Linux
Version: Red Hat (64 bit)
RAM: 2048 (you cannot go lower than this for initial setup)
Hard Drive: VDI – Dynamic – 64 GB (we allocate this much as it will act as NFS Storage)

Note: VirtualBox seems to mix up the networks if you add them all at the same time so we add the 1st Network and install CentOS, then once fully installed, we add the additional networks, rebooting in-between. This appears to be a bug in the latest versions of VirtualBox (4.3.18 at the time of writing)

Modify the settings and assign ONLY the 1st network Adapter correct networks as follows:

csman-adapter-1

Install CentOS 6.5 64-bit minimal, set the Hostname to CSMAN, and IP address to 192.168.56.11/24 with a gateway of 192.168.56.1, and ensure the network is set to start on boot. Set DNS to public servers such as 8.8.8.8 & 8.8.4.4

Once the install is completed reboot the VM and confirm eth0 is active, then shutdown the VM and add the 2nd Network Adapter

csman-adapter-2

Boot the VM so it detects the NIC, then shut down and add the 3rd Adapter

csman-adapter-3

Boot the VM so it detects the NIC, then shut down and add the 4th Adapter

csman-adapter-4

Finally, boot the VM so it detects the last adapter and then we can configure the various interfaces with the correct IP schemas.

ifcfg-eth0

DEVICE=eth0
TYPE=Ethernet
IPADDR=192.168.56.11
PREFIX=24
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME=MGMT

ifcfg-eth1

DEVICE=eth1
TYPE=Ethernet
IPADDR=10.0.2.11
GATEWAY=10.0.2.1
PREFIX=24
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
DEFROUTE=yes
PEERROUTES=yes
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME=NAT

ifcfg-eth2

DEVICE=eth2
TYPE=Ethernet
IPADDR=172.30.0.11
PREFIX=24
ONBOOT=yes
NM_CONTROLLED=no
BOOTPROTO=none
IPV4_FAILURE_FATAL=yes
IPV6INIT=no
NAME=PUBLIC 

ifcfg-eth3

DEVICE=eth3
TYPE=Ethernet
BOOTPROTO=none
ONBOOT=yes
MTU=9000
VLAN=yes
USERCTL=no
MTU=9000

ifcfg-eth3.100

DEVICE=eth3.100
TYPE=Ethernet
IPADDR=10.10.100.11
PREFIX=24
ONBOOT=yes
BOOTPROTO=none
NAME=PRI-STOR
VLAN=yes
USERCTL=no
MTU=9000 

ifcfg-eth3.101

DEVICE=eth3.101
TYPE=Ethernet
IPADDR=10.10.101.11
PREFIX=24
ONBOOT=yes
BOOTPROTO=none
NAME=SEC-STOR
VLAN=yes
USERCTL=no
MTU=9000

Restart networking to apply the new settings, then apply all the latest updates

service networking restart
yum update -y

You can now connect via SSH using Putty to continue the rest of the configuration so you can copy and paste commands and settings etc

Installation and Configuration

With the base VM built we now need to install Apache CloudStack and all the other services this VM will be hosting. First we need to ensure the VM has the correct configuration.

Selinux

Selinux needs to be set to ‘permissive’, we can achieve this by running the following commands:

setenforce permissive
sed -i "/SELINUX=enforcing/ c\SELINUX=permissive" /etc/selinux/config

Hostname

The CloudStack Management Server should return its FQDN when you run hostname –fqdn, but as we do not have a working DNS installation it will probably return ‘unknown-host’ To resolve this we simply add an entry into the Hosts file, and while we are there, we may as well add one for the xenserver as well. Update /etc/hosts with the following, then reboot for it to take effect.

127.0.0.1 localhost localhost.cstack.local
192.168.56.11 csman.cstack.local csman
192.168.56.101 xenserver.cstack.local xenserver

 

Speed up SSH Connections

As you will want to use SSH to connect to the CloudStack VM its worth turning off the DNS Check to speed up the connection. Run the following commands

sed -i "/#UseDNS yes/ c\UseDNS no" /etc/ssh/sshd_config
service sshd restart

 

NTP

It’s always a good idea to install NTP so let’s add it now, and set it to start on boot (you can always configure this VM to act as the NTP Server for the XenServer, but that’s out of scope for this article)

yum install -y ntp
chkconfig ntpd on
service ntpd start

 

CloudStack Repo

Setup the CloudStack repo by running the following command:

echo "[cloudstack]
name=cloudstack
baseurl=http://packages.shapeblue.com/cloudstack/main/centos/4.4
enabled=1
gpgcheck=1" > /etc/yum.repos.d/cloudstack.repo

 

Import the ShapeBlue gpg release key: (Key ID 584DF93F, Key fingerprint = 7203 0CA1 18C1 A275 68B1 37C4 BDF0 E176 584D F93F)

yum install wget -y
wget http://packages.shapeblue.com/release.asc
sudo rpm --import release.asc

 

Install CloudStack and MySQL

Now we can install CloudStack and MySQL Server

yum install -y cloudstack-management mysql-server

 

Setup NFS Server

As the CSMAN VM will also be acting as the NFS Server we need to setup the NFS environment. Run the following commands to create the folders for Primary and Secondary Storage and then export them to the appropriate IP ranges.

mkdir /exports
mkdir -p /exports/primary
mkdir -p /exports/secondary
chmod 777 -R /exports
echo "/exports/primary 10.10.100.0/24(rw,async,no_root_squash)" > /etc/exports
echo "/exports/secondary 10.10.101.0/24(rw,async,no_root_squash)" >> /etc/exports
exportfs -a

 

We now need to update /etc/sysconfig/nfs with the settings to activate the NFS Server. Run the following command to update the required settings

sed -i -e '/#MOUNTD_NFS_V3="no"/ c\MOUNTD_NFS_V3="yes"' -e '/#RQUOTAD_PORT=875/ c\RQUOTAD_PORT=875' -e '/#LOCKD_TCPPORT=32803/ c\LOCKD_TCPPORT=32803' -e '/#LOCKD_UDPPORT=32769/ c\LOCKD_UDPPORT=32769' -e '/#MOUNTD_PORT=892/ c\MOUNTD_PORT=892' -e '/#STATD_PORT=662/ c\STATD_PORT=662' -e '/#STATD_OUTGOING_PORT=2020/ c\STATD_OUTGOING_PORT=2020' /etc/sysconfig/nfs

 

We also need to update the firewall settings to allow the XenServer to access the NFS exports so run the following to setup the required settings

sed -i -e "/:OUTPUT/ a\-A INPUT -p tcp -m tcp --dport 111 -j ACCEPT" /etc/sysconfig/iptables
sed -i -e "/:OUTPUT/ a\-A INPUT -p udp -m udp --dport 111 -j ACCEPT" /etc/sysconfig/iptables
sed -i -e "/:OUTPUT/ a\-A INPUT -p tcp -m tcp --dport 2049 -j ACCEPT" /etc/sysconfig/iptables
sed -i -e "/:OUTPUT/ a\-A INPUT -p udp -m udp --dport 2049 -j ACCEPT" /etc/sysconfig/iptables
sed -i -e "/:OUTPUT/ a\-A INPUT -p tcp -m tcp --dport 2020 -j ACCEPT" /etc/sysconfig/iptables
sed -i -e "/:OUTPUT/ a\-A INPUT -p tcp -m tcp --dport 32803 -j ACCEPT" /etc/sysconfig/iptables
sed -i -e "/:OUTPUT/ a\-A INPUT -p udp -m udp --dport 32769 -j ACCEPT" /etc/sysconfig/iptables
sed -i -e "/:OUTPUT/ a\-A INPUT -p tcp -m tcp --dport 892 -j ACCEPT" /etc/sysconfig/iptables
sed -i -e "/:OUTPUT/ a\-A INPUT -p udp -m udp --dport 892 -j ACCEPT" /etc/sysconfig/iptables
sed -i -e "/:OUTPUT/ a\-A INPUT -p tcp -m tcp --dport 875 -j ACCEPT" /etc/sysconfig/iptables
sed -i -e "/:OUTPUT/ a\-A INPUT -p udp -m udp --dport 875 -j ACCEPT" /etc/sysconfig/iptables
sed -i -e "/:OUTPUT/ a\-A INPUT -p tcp -m tcp --dport 662 -j ACCEPT" /etc/sysconfig/iptables
sed -i -e "/:OUTPUT/ a\-A INPUT -p udp -m udp --dport 662 -j ACCEPT" /etc/sysconfig/iptables
service iptables restart

 

Then we set the nfs service to autostart on boot, and also start it now

chkconfig nfs on
service nfs start

 

Setup MySQL Server

The following command will adjust the MySQL Configuration for this environment

sed -i -e '/datadir/ a\innodb_rollback_on_timeout=1' -e '/datadir/ a\innodb_lock_wait_timeout=600' -e '/datadir/ a\max_connections=350' -e '/datadir/ a\log-bin=mysql-bin' -e "/datadir/ a\binlog-format = 'ROW'" -e "/datadir/ a\bind-address = 0.0.0.0" /etc/my.cnf

 

Then we set the mysqld service to autostart on boot, and also start it now

chkconfig mysqld on
service mysqld start

 

It’s always a good idea to secure a default install of MySQL and there is a handy utility to do this for you. Run the following command, setting a new password when prompted, (the current password will be blank) and accept all of the defaults to remove the anonymous user, test database and disable remote access etc.

mysql_secure_installation

 

Now we will login into MySQL and assign all privileges to the root account, this is so it can be used to create the ‘cloud’ account in a later step

mysql -u root -p  (enter password when prompted)
mysql> GRANT ALL PRIVILEGES ON *.* TO 'root'@'%' WITH GRANT OPTION;
mysql> quit

 

Setup Databases

With MySQL configured we can now setup the CloudStack Databases by running the following two commands, substituting your root password you setup earlier

cloudstack-setup-databases cloud:<password>@127.0.0.1 --deploy-as=root:<password>
cloudstack-setup-management

 

Nginx

There is a default example template which gets downloaded from the cloud.com web servers, but as this test system has no real public internet access we need to provide a way for the Secondary Storage VM to download this template. We achieve this by installing NGINX on the CSMAN VM, and use it to host the Template on our simulated ‘Public’ network.

First create the NGINX repo by running the following command:

echo "[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/\$releasever/\$basearch/
gpgcheck=0
enabled=1" > /etc/yum.repos.d/nginx.repo

 

Then install NGINX by running the following command

yum install nginx -y

 

Now we download the example CentOS Template for XenServer by running the following two commands

cd /usr/share/nginx/html
wget -nc http://download.cloud.com/templates/builtin/centos56-x86_64.vhd.bz2

We need to add a firewall rule to allow access via port 80 so run the following two commands

sed -i -e "/:OUTPUT/ a\-A INPUT -p tcp -m tcp --dport 80 -j ACCEPT" /etc/sysconfig/iptables
service iptables restart

 

Finally we start the nginx service, then test it by accessing http://192.168.56.11/ from the Host laptop

service nginx start

nginx

XenServer vhd-util

As we will be using Citrix XenServer as our Hypervisor we need to download a special utility which will get copied to every XenServer when it is added to the system. Run the following lines to download the file and update the permissions.

cd /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/
wget http://download.cloud.com.s3.amazonaws.com/tools/vhd-util
chmod 755 /usr/share/cloudstack-common/scripts/vm/hypervisor/xenserver/vhd-util

 

Seed the CloudStack Default System VM Template

With now we need to seed the Secondary Storage with the XenServer System VM Template so run the following command

/usr/share/cloudstack-common/scripts/storage/secondary/cloud-install-sys-tmplt -m /exports/secondary -u http://packages.shapeblue.com/systemvmtemplate/4.4/4.4.1/systemvm64template-4.4.1-7-xen.vhd.bz2 -h xenserver -F

 

CloudStack Usage Server

An optional step is to install the CloudStack Usage Service, to do so run the following command

yum install cloudstack-usage -y
service cloudstack-usage start

 

Customise the Configuration

For this test system to work within the limited resources available on a 4GB RAM Laptop, we need to make a number of modifications to the configuration.

Firstly we need to enable the use of a non HVM enabled XenServer. When you install XenServer on VirtualBox it warns you that it will only support PV and not HVM. To get around this we run the following SQL update command to add a new line into the Configuration table in the Cloud Database (remember to substitute your own MySQL Cloud password you used when you setup the CloudStack Database)

mysql -p<password> cloud -e \ "INSERT INTO cloud.configuration (category, instance, component, name, value, description) VALUES ('Advanced', 'DEFAULT', 'management-server', 'xen.check.hvm', 'false', 'Shoud we allow only the XenServers support HVM');"

 

The following MySQL commands update various global settings, and change the resources allocated to the system VMs so they will work within the limited resources available.

mysql -u cloud -p<password>
UPDATE cloud.configuration SET value='8096' WHERE name='integration.api.port';
UPDATE cloud.configuration SET value='60' WHERE name='expunge.delay';
UPDATE cloud.configuration SET value='60' WHERE name='expunge.interval';
UPDATE cloud.configuration SET value='60' WHERE name='account.cleanup.interval';
UPDATE cloud.configuration SET value='60' WHERE name='capacity.skipcounting.hours';
UPDATE cloud.configuration SET value='0.99' WHERE name='cluster.cpu.allocated.capacity.disablethreshold';
UPDATE cloud.configuration SET value='0.99' WHERE name='cluster.memory.allocated.capacity.disablethreshold';
UPDATE cloud.configuration SET value='0.99' WHERE name='pool.storage.capacity.disablethreshold';
UPDATE cloud.configuration SET value='0.99' WHERE name='pool.storage.allocated.capacity.disablethreshold';
UPDATE cloud.configuration SET value='60000' WHERE name='capacity.check.period';
UPDATE cloud.configuration SET value='1' WHERE name='event.purge.delay';
UPDATE cloud.configuration SET value='60' WHERE name='network.gc.interval';
UPDATE cloud.configuration SET value='60' WHERE name='network.gc.wait';
UPDATE cloud.configuration SET value='600' WHERE name='vm.op.cleanup.interval';
UPDATE cloud.configuration SET value='60' WHERE name='vm.op.cleanup.wait';
UPDATE cloud.configuration SET value='600' WHERE name='vm.tranisition.wait.interval';
UPDATE cloud.configuration SET value='60' WHERE name='vpc.cleanup.interval';
UPDATE cloud.configuration SET value='4' WHERE name='cpu.overprovisioning.factor';
UPDATE cloud.configuration SET value='8' WHERE name='storage.overprovisioning.factor';
UPDATE cloud.configuration SET value='192.168.56.11/32' WHERE name='secstorage.allowed.internal.sites';
UPDATE cloud.configuration SET value='192.168.56.0/24' WHERE name='management.network.cidr';
UPDATE cloud.configuration SET value='192.168.56.11' WHERE name='host';
UPDATE cloud.configuration SET value='false' WHERE name='check.pod.cidrs';
UPDATE cloud.configuration SET value='0' WHERE name='network.throttling.rate';
UPDATE cloud.configuration SET value='0' WHERE name='vm.network.throttling.rate';
UPDATE cloud.configuration SET value='GMT' WHERE name='usage.execution.timezone';
UPDATE cloud.configuration SET value='16:00' WHERE name='usage.stats.job.exec.time';
UPDATE cloud.configuration SET value='true' WHERE name='enable.dynamic.scale.vm';
UPDATE cloud.configuration SET value='9000' WHERE name='secstorage.vm.mtu.size';
UPDATE cloud.configuration SET value='60' WHERE name='alert.wait';
UPDATE cloud.service_offering SET ram_size='128', speed='128' WHERE vm_type='domainrouter';
UPDATE cloud.service_offering SET ram_size='128', speed='128' WHERE vm_type='elasticloadbalancervm';
UPDATE cloud.service_offering SET ram_size='128', speed='128' WHERE vm_type='secondarystoragevm';
UPDATE cloud.service_offering SET ram_size='128', speed='128' WHERE vm_type='internalloadbalancervm';
UPDATE cloud.service_offering SET ram_size='128', speed='128' WHERE vm_type='consoleproxy';
UPDATE cloud.vm_template SET removed=now() WHERE id='2';
UPDATE cloud.vm_template SET url='http://192.168.56.11/centos56-x86_64.vhd.bz2' WHERE unique_name='centos56-x86_64-xen';
quit
service cloudstack-management restart

 

To enable access to the Un-Authenticated API which we have enabled on the default port of 8096, we need to add a firewall rule. Run the following commands to allow port 8096 through the firewall

sed -i -e "/:OUTPUT/ a\-A INPUT -p tcp -m tcp --dport 8096 -j ACCEPT" /etc/sysconfig/iptables
service iptables restart

 

Test the UI

Allow 1-2 mins for the cloudstack-management service to fully restart then login into the UI which should be accessible from the Host Laptop on http://192.168.56.11:8080/client/

The default credentials are

Username: admin
Password: password
Domain: <blank>

logon

Create Compute Offering

The default Compute Offerings are not suitable for this limited environment so we need to create a new compute offering using the following settings:

Name: Ultra Tiny
Description: Ultra Tiny – 1vCPU, 128MB RAM
Storage Type: Shared
Custom: No
# of CPU Cores: 1
CPU (in MHz): 500
Memory (in MB): 128
Network Rate (Mb/s): null
QoS Type: null
Offer HA: Yes
Storage Tags: null
Host Tags: null
CPU Cap: No
Public: Yes
Volatile: No
Deployment Planner: null
Planner mode: null
GPU: null

Reduce the amount of RAM

Following a successful login to the UI, the Databases will be fully deployed so now we can reduce the RAM to 1GB to free up memory for our XenServer VM. Shutdown the VM and change the settings to 1024 MB of RAM.

XenServer VM

To configure the XenServer you will need XenCenter running on your local Host if you are running Windows, but if your Host is running OSX or Linux, then you need to add a Windows VM which can run XenCenter. You can download XenCenter from http://downloadns.citrix.com.edgesuite.net/akdlm/8160/XenServer-6.2.0-XenCenter.msi

Create a VM for XenServer using the following settings:

Name: XenServer
Type: Linux
Version: Red Hat (64 bit)
vCPU: 2
RAM: 1536 (If your host has 8GB of RAM, consider allocating 3072)
Hard Drive: VDI – Dynamic – 24 GB

Note: VirtualBox seems to mix up the networks if you add them all at the same time so we add the 1st Network and install XenServer, then once fully installed, we add the additional networks, rebooting in-between. This appears to be a bug in the latest versions of VirtualBox (4.3.18 at the time of writing)

Modify the settings and assign ONLY the 1st network Adapter correct networks as follows:

xenserver-adapter-1

Note how we have set the ‘Promiscuous Mode’ to ‘Allow All’

Now install XenServer 6.2 by downloading the ISO from http://downloadns.citrix.com.edgesuite.net/akdlm/8159/XenServer-6.2.0-install-cd.iso and booting the VM.

The XenServer installation wizard is straightforward, but you will get a warning about the lack of hardware virtualisation support, this is expected as VirtualBox does not support this. Accept the warning and continue.

Choose the appropriate regional settings and enter the following details when prompted: (we enter the IP of the CSMAN VM for DNS and NTP, whilst this guide does not cover setting up these services on the CSMAN VM, this gives you the option of doing so at a later date)

Enable Thin Provisioning: Yes
Install source: Local media
Supplemental Packs: No
Verification: Skip
Password: <password>
Static IP: 192.168.56.101/24 (no gateway required)
Hostname: xenserver
DNS: 192.168.56.11
NTP: 192.168.56.11

Once the XenServer Installation has finished, detach the ISO and reboot the VM.

We now need to change the amount of RAM allocated to Dom0 to its minimum recommended amount which is 400MB, we do this by running the following command on the XenServer console

 /opt/xensource/libexec/xen-cmdline --set-xen dom0_mem=400M,max:400M 

XenServer Patches

It’s important to install XenServer Patches and whilst XenCenter will inform you of the required patches, as we are using the OpenSource version of XenServer we have to install Patches via the command line. Fortunately there are a number of ways of automating this process.

Personally I always use PXE to deploy XenServer and the installation of patches is built into my deployment process. However that is out of scope for this article, but Tim Mackey has produced a great blog article on how to do this: http://xenserver.org/discuss-virtualization/virtualization-blog/entry/patching-xenserver-at-scale.html

Whilst Tim’s method of rebooting after every patch install is best practice, it can take a long time to install all Patches so an alternative approach I use in these non-production test environments is detailed here https://github.com/amesserl/xs_patcher  This installs all patches and requires only a single reboot.

The configuration file ‘clearwater’ is now a little out of date, and should contain the following (and the cache folder should contain the associated patch files):

XS62E014|78251ea4-e4e7-4d72-85bd-b22bc137e20b|downloadns.citrix.com.edgesuite.net/8736/XS62E014.zip|support.citrix.com/article/CTX140052

XS62ESP1|0850b186-4d47-11e3-a720-001b2151a503|downloadns.citrix.com.edgesuite.net/8707/XS62ESP1.zip|support.citrix.com/article/CTX139788

XS62ESP1003|c208dc56-36c2-4e91-b8d7-0246575b1828|downloadns.citrix.com.edgesuite.net/9031/XS62ESP1003.zip|support.citrix.com/article/CTX140416

XS62ESP1005|1c952800-c030-481c-a0c1-d1b45aa19fcc|downloadns.citrix.com.edgesuite.net/9058/XS62ESP1005.zip|support.citrix.com/article/CTX140553

XS62ESP1009|a24d94e1-326b-4eaa-8611-548a1b5f8bd5|downloadns.citrix.com.edgesuite.net/9617/XS62ESP1009.zip|support.citrix.com/article/CTX141191

XS62ESP1013|b22d6335-823d-43a6-ba26-28793717125b|downloadns.citrix.com.edgesuite.net/9703/XS62ESP1013.zip|support.citrix.com/article/CTX141480

XS62ESP1014|4fc82e62-b938-407d-a2c6-68c8922f3ec2|downloadns.citrix.com.edgesuite.net/9708/XS62ESP1014.zip|support.citrix.com/article/CTX141486

Once you have your XenServer fully patched shut it down and then add the 2nd Adapter, again note how we have set the ‘Promiscuous Mode’ to ‘Allow All’

xenserver-adapter-2

Boot the VM and then using XenCenter perform a ‘Rescan’ on the NICs to detect this new NIC, then shutdown and add the 3rd Adapter, again note how we have set the ‘Promiscuous Mode’ to ‘Allow All’

xenserver-adapter-3

Boot the VM and then using XenCenter perform a ‘Rescan’ on the NICs to detect this new NIC, then shutdown and add the 4th Adapter, again note how we have set the ‘Promiscuous Mode’ to ‘Allow All’

xenserver-adapter-4

Boot the VM and then using XenCenter perform a ‘Rescan’ on the NICs to detect this final NIC, then one final reboot to make sure they are all activated and connected.

Configure XenServer Networks

Now we are ready to configure the XenServer Networks. We should have the following four networks present, and it’s worth just checking the MACs line up with the Adapters in VirtualBox.

xenserver-networks-1
We need to rename the networks using a more logical naming convention, and also create the two Storage Networks, and assign their VLANs etc.

First of all start by renaming them all setting the MTU of the Storage Network to 9000 (the rest remain at the default of 1500)

Network 0 – MGMT
Network 1 – GUEST
Network 2 – PUBLIC
Network 3 – STORAGE (and MTU of 9000)

xenserver-networks-2

Next we add the Primary Storage Network using the following settings:

Type: External Network
Name: PRI-STORAGE
NIC: NIC 3
VLAN: 100
MTU: 9000

Then the Secondary Storage Network:

Type: External Network
Name: SEC-STORAGE
NIC: NIC 3
VLAN: 101
MTU: 9000

xenserver-storage-networks

Finally we add the IP addresses for the Primary and Secondary Storage Networks so the XenServer can access them

Name: PRI-STOR
Network: PRI-STORAGE
IP address: 10.10.100.101
Subnet mask: 255.255.255.0
Gateway: <blank>

Name: SEC-STOR
Network: SEC-STORAGE
IP address: 10.10.101.101
Subnet mask: 255.255.255.0
Gateway: <blank>

xenserver-ips

That is all the configuration required for XenServer so now we can proceed with deploying our first Zone. However before we do, it’s worth taking a snapshot of both of the VMs so you can roll back and start again if required.

Zone Deployment

We now add an Advanced Zone by going to ‘Infrastructure/Zones/Add Zone’ and creating a new Zone of type ‘Advanced’ without Security Groups

Zone Name – Test
IPv4 DNS1 – 8.8.8.8
Internal DNS 1 – 192.168.56.11
Hypervisor – XenServer
Guest CIDR – 10.1.1.0/24

Next we need to setup the XenServer Traffic Labels to match the names we allocated to each Network on our XenServer, and we also need to add the optional Storage Network by dragging it onto the Physical Network.

xenserver-physical-networks

Edit each Traffic Type and set the following Labels:

Management Network – MGMT
Public Network – PUBLIC
Guest Network – GUEST
Storage Network – SEC-STORAGE

Then continue through the add zone wizard using the following settings

Public Traffic

Gateway – 172.30.0.1
Netmask – 255.255.255.0
VLAN – <blank>
Start IP – 172.30.0.21
End IP -172.30.0.30

POD Settings

POD Name – POD1
Reserved System Gateway – 192.168.56.1
Reserved System Netmask – 255.255.255.0
Start Reserved System IP – 192.168.56.21
End Reserved System IP – 192.168.56.30

Guest Traffic

VLAN Range – 600 – 699

Storage Traffic

Gateway – 10.10.101.1
Netmask – 255.255.255.0
VLAN – <blank>
Start IP – 10.10.101.21
End IP – 10.10.101.30

Cluster Settings

Hypervisor – XenServer
Cluster Name – CLU1

Host Settings

Host Name – 192.168.56.101
Username – root
Password – <password>

Primary Storage Settings

Name – PRI1
Scope – Cluster
Protocol – nfs
Server – 10.10.100.11
Path – /exports/primary
Provider: DefaultPrimary
Storage Tags: <BLANK>

Secondary Storage Settings

Provider – NFS
Name – SEC1
Server – 10.10.101.11
Path – /exports/secondary

At the end of it, activate the Zone, then allow approx. 5 minutes for the System VMs to deploy and the default CentOS Template to be ‘downloaded’ into the system. You are now ready to deploy your first Guest VM.

 

 

How to upgrade an Apache CloudStack Citrix XenServer Cluster

, , ,

So you have a Cluster of Citrix XenServers and you want to upgrade them to a new version, for example to go from XenServer 6.0.2 to XenServer 6.2, or simply apply the latest Hotfixes.  As this is a cluster that is being managed by CloudStack it is not as simple as using the Rolling Pool Upgrade feature in XenCenter – in fact this is the LAST thing you want to do, and WILL result in a broken Cluster.

This article walks you through the steps required to perform the upgrade, but as always you must test this yourself in your own test environment before attempting on a production system.

We need to change the default behaviour of CloudStack with respect to how it manages XenServer Clusters before continuing.  Edit /etc/cloudstack/management/environment.properties and add the following line:

# vi /etc/cloudstack/management/environment.properties

Add > manage.xenserver.pool.master=false

Now restart the CloudStack Management Service

# service cloudstack-management restart

Repeat for all CloudStack Management servers

It is vital that you upgrade the XenServer Pool Master first before any of the Slaves.  To do so you need to empty the Pool Master of all CloudStack VMs, and you do this by putting the Host into Maintenance Mode within CloudStack to trigger a live migration of all VMs to alternate Hosts (do not place the Host into Maintenance Mode using XenCenter as this will cause a new Master to be elected and we do not want that). 

Next you need to ‘Unmanage’ the Cluster, as this prevents users from being able to interact (stop/start) VMs you will need to arrange a ‘Maintenance Window’ but only long enough to update the Pool Master.  All Customer VMs will continue to run during the upgrade process unless you are using Local Storage, in which case VMs on the Hosts being upgraded will have to shut down.  After ‘Unmanaging’ the Cluster, all Hosts will go into a ‘Disconnected’ state, this is expected and is not a cause for concern.

Now you can upgrade your Pool Master, either upgrading to a newer version, or simply applying XenServer Hotfixes as required.  Once the Pool Master has been fully upgraded re-manage the Cluster and then wait for all of the Hosts in the Cluster to come back online within CloudStack. 

Monitor the status of your NFS Storage via XenCenter and wait for all Volumes to reconnect on the upgraded Host.  Once storage has reconnected and all Hosts are back online, take the Pool Master you just upgraded out of CloudStack Maintenance Mode.

Edit /etc/cloudstack/management/environment.properties and remove the following line which you added earlier:

# vi /etc/cloudstack/management/environment.properties

Delete > manage.xenserver.pool.master=false

Now restart the CloudStack Management Service

# service cloudstack-management restart

Repeat for all CloudStack Management servers

You can now upgrade each Slave by simply placing it into Maintenance Mode in CloudStack, apply the upgrade / Hotfixes and when completed, bringing out of Maintenance Mode before starting on the next Host.

About the Author

Geoff Higginbottom is CTO of ShapeBlue, the strategic cloud consultancy and an Apache CloudStack Committer. Geoff spends most of his time designing private & public cloud infrastructures for telco’s, ISP’s and enterprises based on CloudStack.