How to upgrade an Apache CloudStack Citrix XenServer Cluster

, , ,

So you have a Cluster of Citrix XenServers and you want to upgrade them to a new version, for example to go from XenServer 6.0.2 to XenServer 6.2, or simply apply the latest Hotfixes.  As this is a cluster that is being managed by CloudStack it is not as simple as using the Rolling Pool Upgrade feature in XenCenter – in fact this is the LAST thing you want to do, and WILL result in a broken Cluster.

This article walks you through the steps required to perform the upgrade, but as always you must test this yourself in your own test environment before attempting on a production system.

We need to change the default behaviour of CloudStack with respect to how it manages XenServer Clusters before continuing.  Edit /etc/cloudstack/management/environment.properties and add the following line:

# vi /etc/cloudstack/management/environment.properties

Add > manage.xenserver.pool.master=false

Now restart the CloudStack Management Service

# service cloudstack-management restart

Repeat for all CloudStack Management servers

It is vital that you upgrade the XenServer Pool Master first before any of the Slaves.  To do so you need to empty the Pool Master of all CloudStack VMs, and you do this by putting the Host into Maintenance Mode within CloudStack to trigger a live migration of all VMs to alternate Hosts (do not place the Host into Maintenance Mode using XenCenter as this will cause a new Master to be elected and we do not want that). 

Next you need to ‘Unmanage’ the Cluster, as this prevents users from being able to interact (stop/start) VMs you will need to arrange a ‘Maintenance Window’ but only long enough to update the Pool Master.  All Customer VMs will continue to run during the upgrade process unless you are using Local Storage, in which case VMs on the Hosts being upgraded will have to shut down.  After ‘Unmanaging’ the Cluster, all Hosts will go into a ‘Disconnected’ state, this is expected and is not a cause for concern.

Now you can upgrade your Pool Master, either upgrading to a newer version, or simply applying XenServer Hotfixes as required.  Once the Pool Master has been fully upgraded re-manage the Cluster and then wait for all of the Hosts in the Cluster to come back online within CloudStack. 

Monitor the status of your NFS Storage via XenCenter and wait for all Volumes to reconnect on the upgraded Host.  Once storage has reconnected and all Hosts are back online, take the Pool Master you just upgraded out of CloudStack Maintenance Mode.

Edit /etc/cloudstack/management/environment.properties and remove the following line which you added earlier:

# vi /etc/cloudstack/management/environment.properties

Delete > manage.xenserver.pool.master=false

Now restart the CloudStack Management Service

# service cloudstack-management restart

Repeat for all CloudStack Management servers

You can now upgrade each Slave by simply placing it into Maintenance Mode in CloudStack, apply the upgrade / Hotfixes and when completed, bringing out of Maintenance Mode before starting on the next Host.

About the Author

Geoff Higginbottom is CTO of ShapeBlue, the strategic cloud consultancy and an Apache CloudStack Committer. Geoff spends most of his time designing private & public cloud infrastructures for telco’s, ISP’s and enterprises based on CloudStack.