Overview

At the beginning of 2018 a number of vulnerabilities were discovered which allow malicious user space processes to read kernel memory and malicious code in VM guests to read hypervisor memory. These vulnerabilities affect most CPU manufacturers – Intel, AMD, ARM, MIPS, etc.

The vulnerabilities were nicknamed “Spectre” and “Meltdown” and are outlined in the following CVEs:

• Spectre variant 1 – Bounds Check Bypass: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5753
• Spectre variant 2 – Branch Target Injection: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5715
• Meltdown variant 3 – Rogue Data Cache Launch: https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-5754

From a CloudStack point of view the main affected components are the system VM templates. This advisory outlines the fix provided for Meltdown only in CloudStack 4.9 (no fixes are available for Spectre). CloudStack 4.11 system VM templates were patched at release time and are therefore not affected.

Affected components summary:

CVE	ACS 4.9 System VMs	ACS 4.11 System VMs	Hypervisors
Spectre 1	No fix available - upgrade to 4.11	Fix in release	Affected - consult with vendor
Spectre 2	No fix available - upgrade to 4.11	Fix in release	Affected - consult with vendor
Meltdown	Fixed - new system VM template	Fix in release	Affected - consult with vendor

Effect On CloudStack

The impact on CloudStack environments is two-fold since the vulnerabilities affect both the compute hypervisor hosts and the CloudStack system VMs.

Hypervisors

As these are low level CPU call vulnerabilities all hypervisors are affected. Hypervisor vendors have been providing patches – and may continue to do so as further analysis is carried out and potential fixes are developed. The issue with the hypervisor patches is they will potentially impact performance, something which may affect hypervisor VM density figures and/or VM guest performance. ShapeBlue therefore advise users to carry out thorough testing to determine each CloudStack environment impact before rolling these out to production. ShapeBlue can not provide further information or advise on these patches and we recommend all our community users and customers to discuss with the respective hypervisor vendors.

More information can be found in the following articles:

• VMware: https://kb.vmware.com/s/article/52245
• XenServer: https://support.citrix.com/article/CTX231390
• RedHat / KVM: https://access.redhat.com/security/vulnerabilities/speculativeexecution
• CentOS / KVM: https://blog.centos.org/2018/01/meltdown-and-spectre-the-response-from-centos/
• Ubuntu / KVM: https://wiki.ubuntu.com/SecurityTeam/KnowledgeBase/SpectreAndMeltdown

CloudStack System VMs

The CloudStack system VMs are also affected by Spectre and Meltdown. However since these vulnerabilities require local user access someone with malicious intent would first have to gain local access to the system VMs. Since these are locked down and secured in the first place the risk to CloudStack environments is considered low as long as general CloudStack security best practices are followed.

The CloudStack LTS branches system VMs are based on 64-bit Debian releases:

• CloudStack 4.11 utilises Debian 9 “Stretch” 64-bit system VMs
• CloudStack 4.9 utilises Debian 7 “Wheezy” 64-bit system VMs

CloudStack 4.11 was released in February 2018 at which point the Spectre and Meltdown fixes were already provided, and these were therefore included in the system VM templates.

However – CloudStack 4.9 utilise Debian 7 “Wheezy” system VM templates – and “Wheezy” went support end-of-life on May 31st 2018 (https://wiki.debian.org/LTS).  At this point the Debian community have only provided patches for Meltdown, and there are no indications Spectre fixes will be provided. As a result ShapeBlue have made the decision to provide new CloudStack 4.9 system VM templates with only the Meltdown patch included. Our overall recommendation if full patching of the vulnerabilities is required is to upgrade to CloudStack version 4.11.

Further information on the Debian fixes can be found in https://wiki.debian.org/DebianSecurity/SpectreMeltdown.

CloudStack 4.9 system VM templates / patching procedure

Whilst system VMs may be patched in-situ they will require reboots for the patches to take effect, and the ShapeBlue recommendation is therefore to update the system VM templates to ensure the Meltdown patch is permanently applied. ShapeBlue have built new system VM templates for CloudStack 4.9 for XenServer, VMware and KVM hypervisors. These can be downloaded from http://packages.shapeblue.com/systemvmtemplate/4.6/meltdown/. The new system VM templates have gone through the full test cycle and no regressions have been found.

The procedure for updating the system VM templates is as follows:

• For each hypervisor type in the CloudStack environment upload the new system VM template with the following information:
  • Name: use a descriptive name, e.g. systemvm-<hypervisor>-4.6-meltdown
  • Description: add template description
  • Zone: pick the correct zone(s)
  • Hypervisor: pick the correct hypervisor
  • Format: VHD (XenServer) / OVA (VMware) / QCOW2 (KVM)
  • OS Type: Debian GNU/Linux 7.0 (64-bit) (or the highest Debian release number available in the dropdown)
  • Extractable: no
  • Password Enabled: no
  • Public: no
  • Featured: no
  • Routing: yes
• Update the global settings for “router.template.<hypervisor>” to the same as the name configured during the template upload.
• Restart the management service on all management servers.
• Destroy SSVMs and CPVM instances – CloudStack management will recreate these with the new template.
• Restart all networks with the “cleanup” option, which will recreate all VRs with the new system VM template.

Further information

For ShapeBlue support customers, please contact the support team for further information.

For other CloudStack users, please use the community mailing lists.

Overview

Short Term Mitigation

Performing the following steps will restrict the API to use by your root-admin account users only.
USERS and DOMAIN ADMINS will NO LONGER be able to create or re-generate API keys themselves.

registerUserKeys=1

Save the file and restart your management server.

Users of CloudStack 4.9 who are using dynamic roles feature, can navigate to UI->Roles and go to Rules section of each of the non-admin role, and delete a configured ‘Allow’ rule for ‘registerUserKeys’.

 

Further information

For ShapeBlue support customers, please contact the support team for further information.

For other CloudStack users, please use the community mailing lists.

For users of commercial distribution of CloudStack, please contact your vendor

Overview

A vulnerability has been recently disclosed by Qualys that could result in a remote attacker being able to execute malicious instructions on vulnerable systems. The vulnerability affects Linux based operating systems.

This is better known as GHOST ‘glibc’ vulnerability (CVE-2015-0235): https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-0235

What is ShapeBlue Doing

ShapeBlue has analysed the impact of this issue on Apache CloudStack (ACS).  The download template functionality provided by the SSVM to the end user puts it at risk. Since it is a linux issue all the Apache CloudStack versions are affected.  An immediate fix would be to login into each SSVM and upgrade the glib package to the one that has the fix.  This is a temporary solution as on a reboot the template falls back to the configured version.  We have released an updated set of System VM Templates that have the fix applied as a permanent solution.  It is recommended that all CloudStack operators update their System VM Templates to a patched version following the instructions below.  It is also thought that this vulnerability affects commercial distributions of Apache CloudStack.

Virtual Appliance patching procedure

ShapeBlue has built new systemvm templates with glib fix for major CloudStack versions 4.3, 4.4 and 4.5 for XenServer, VMware and KVM hypervisors. We advise CloudStack users to upgrade to the  latest systemvm templates using the following steps:

1.   Register the new templates for all the hypervisors that you’re using in your CloudStack deployment.  If you’re a CloudStack 4.3 user please use the settings in this table, 4.4 users please scroll down the page.

 

Or, if you’re a CloudStack 4.4 user please use the following table:

2.  After registering the template, wait until the template gets downloaded and is installed. When the template is in READY state move to the next step.

3.  Stop the cloudstack-management service all the management server(s) and take a backup of the database.

service cloudstack-management stop

4.  Download ShapeBlue’s CloudStack SystemVM template upgrading tool for Ghost vulnerability:

wget http://packages.shapeblue.com/tools/ghost-systemvm-upgrader.py

5. This tool requires Python MySQLDB library, to install Python MySQLDB dependency run the following:

Debian based: apt-get install python-mysqldb

RHEL based: yum install MySQL-python

Or, Use pip:

Debian based: apt-get install build-essential python-dev libmysqlclient-dev

RHEL based: yum install mysql-devel python-devel

sudo pip install MySQL-python

6.  Run this tool against your database, it will automatically find and upgrade the registered template:

python ghost-systemvm-upgrader.py -c <Major CloudStack version, 4.3 or 4.4> -i <IP of the database host> -d cloud -u <Database user name, cloud> -p <Database user password>

e.g.   python ghost-systemvm-upgrader.py -c 4.4 -i 192.168.0.21 -d cloud -u cloud -p password

7.  Finally, restart the CloudStack Management service on all management servers and then destroy all of the SystemVMs

 service cloudstack-management start

 

Further information

For ShapeBlue support customers, please contact the support team for further information.

For other CloudStack users, please use the community mailing lists.

The realhostip.com service will be switched off on the 1^st October 2014. Paul Angus looks at what it did, what effect the retirement will have and what you need to do to carry on working if you’re affected.

What is realhostip.com?

When you connect to the Console Proxy system VM or download a disk or ISO from the secondary storage VM you connect over a secure (https) connection. This is particularly important when you put in your password.  In order for this to be secure you need to connect to a URL which has a FQDN and have a certificate to go with it.

The realhostip.com domain and DNS service was created to give an out of the box solution to this problem. By using the realhostip.com domain and a clever bit of magic, with hostnames and a dynamic DNS script, the data transmitted is encrypted.

The problem is that anyone can use this system, which means that just because a VM has a certificate that doesn’t mean you can trust it. Also, someone could extract the certificate from a CloudStack environment and then use that to decrypt others’ traffic. Its very unlikely but technically possible.

So that’s why the use of a publicly available certificate is being retired.

Who will this affect?

For clean installs of CloudStack 4.3 and later do not use realhostip and instead use HTTP (unencrypted) communications to the public interfaces – so you have to arrange the certificate yourself from the get go. So if you’re using 4.3 or later then the switching off of the service won’t affect you. However if you are using 4.3 or later and haven’t installed a certificate then you should install one for the security of your users. Now.

For users of CloudStack 4.3 environments which have been upgraded from earlier versions, you have the additional option of switching HTTPS off for the console proxy and secondary storage but you do so at your own risk and I can’t recommend it for anything which is public facing or not in the most isolated of test environments.

So for versions of CloudStack pre v4.3 or environments which have been upgraded from a previous version, admins need to replace realhostip.com with their own domain and certificate otherwise the console proxy service and downloading from the secondary storage VM will stop working on 1^st October 2014.

What to do to replace realhostip.com

There a number of sources which explain how to use your own domain for secure console proxy and secondary storage, so I’ll point you to the various sources and add some extra depth to the references.

The first thing you need is a domain to use to replace realhostip.com. Generally people use a subdomain of their corporate domain (such as console.supercloud.com) so that it can be managed separately from the parent domain. You don’t have to use a subdomain by any means, but I’m going to assume you have for the purposes of this blog. Then you need a DNS server or two which will act as the resolvers for your subdomain.  The resolvers need to resolve all addresses in your public ranges. Part of the cleverness of the realhostip solution is in the way CloudStack creates FQDNs for the secondary storage VMs and console proxy VMs which are easy to work with. For example, for the IP address 1.2.3.4 CloudStack would create an FQDN of 1-2-3-4.realhostip.com and for 4.3.2.1 it would be 4-3-2-1.realhostip.com.  It makes creating the zone files easy and even makes it possible to create a DNS server that will resolve anything that’s thrown at it in this format. Indeed that’s what the realhostip DNS servers did (source code is here https://github.com/ke4qqq/RHIP). You’re going to replace that with 1-2-3-4.console.supercloud.com

Things get trickier if your DNS is hosted by a 3^rd party and you have a lot of public IPs which the system VMs could use. Creating the entries for a zone file in a spreadsheet is easy, but if you had to fill in a web form for your ISP for each IP address it’s going to be a long, boring and error-prone task, so you’d probably look at creating your own DNS servers for your subdomain or trying to get your ISP to do the heavy lifting.

So we have a domain to use and some DNS servers to resolve it with, now we need a certificate to secure things with.  You’ll need a wildcard certificate for the subdomain as you need it to work for multiple hosts *. console.supercloud.com. A quick search for ‘wildcard certificate’ will bring back a bucket load of vendors of wildcard certificates.  I’ve found that large enterprises will likely have a system in place for obtaining certificates, so a call to corporate IT may save your credit card and a load of time.

To get a certificate you need to generate a private key and then a certificate signing request (CSR) which is used by the certificate provider to generate a certificate based on the private key.  A procedure for doing this is documented in the CloudStack documentation – you need openssl installed which is easy enough, but your CloudStack management server(s) will have it installed so you can always do it from there. http://docs.cloudstack.apache.org/en/latest/administration_guide.html#changing-the-console-proxy-ssl-certificate-and-domain

You can now go to the certificate vendor and request your wildcard certificate.

Once you get your wildcard certificate you’ll need the public certificate of root certificate authority (CA) and the public certificate(s) of intermediate CA(s) (if any). They may be sent with your wildcard certificate or you may need to go to the website of your certificate provider to download them. You need the whole chain so that a client can see that the certificate is bona fide.

Now is the most complicated bit – getting the everything in the correct format. Why this process always ends up being such a pain is beyond me, but anyone who’s tried to add a certificate to a NetScaler, IIS, RDS or CloudStack will testify that it’s rarely simple – the certificate and/or private key you have never seem to be in quite the right format or you’re missing an intermediate certificate, so stick with it – you’re not alone.

The openssl tool is pretty good for converting between formats.

So in all you should have the following:

1. Public certificate of root CA in PEM format
2. Public certificate(s) of intermediate CA(s) (if any) in PEM format
3. Wildcard domain certificate in PEM format
4. Private key in PKCS8 format
5. It should sound obvious but you must make sure that from the public internet your public IP addresses get resolved i.e. 1-2-3-4. console.supercloud.com -> 1.2.3.4 before you make the switchover

Now that you’re ready to make the change you’ll need to do the following steps

1. Update the CloudStack settings to use your new certificate
2. Upload the new certificate to the management server
3. Reboot the system VMs to use the new certificates
4. Verify the new certificate is working

These steps are well documented with additional help also on the CloudStack Wiki

Procedure to Replace realhostip.com with Your Own Domain Name
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Procedure+to+Replace+realhostip.com+with+Your+Own+Domain+Name

Things you should consider while changing the Realhost domain to custom Domain.
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=43188254

Troubleshooting – uploading custom domain certificate instead of using realhostip.com
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Troubleshooting+-+uploading+custom+domain+certificate+instead+of+using+realhostip.com

An extra pointer from our experience is that the system VMs should be stopped, then started, both from CloudStack management server, otherwise the new certificate will not be applied.

Chip Childers wrote this article a while back and the system for importing certificates has been improved since them but it may provide additional help
http://www.chipchilders.com/blog/2013/1/2/undocumented-feature-using-certificate-chains-in-cloudstack.html

 Summary

In this article we looked at what the realhostip.com service does and how its retirement will effect CloudStack/CloudPlatform clouds. We also gave information on the steps required to replace the realhostip.com service if you need to secure your cloud.

About the Author

Paul Angus is a Cloud Architect at ShapeBlue, The Cloud Specialists. He has designed and implemented numerous CloudStack environments for customers across 4 continents, based on Apache Cloudstack ,Citrix Cloudplatform and Citrix Cloudportal.

When not building Clouds, Paul likes to create scripts that build clouds……..and he very occasionally can be seen trying to hit a golf ball.