Posts

ShapeBlue Security Advisory – Spectre and Meltdown patches in CloudStack 4.9 and 4.11

Overview

At the beginning of 2018 a number of vulnerabilities were discovered which allow malicious user space processes to read kernel memory and malicious code in VM guests to read hypervisor memory. These vulnerabilities affect most CPU manufacturers – Intel, AMD, ARM, MIPS, etc.

The vulnerabilities were nicknamed “Spectre” and “Meltdown” and are outlined in the following CVEs:

From a CloudStack point of view the main affected components are the system VM templates. This advisory outlines the fix provided for Meltdown only in CloudStack 4.9 (no fixes are available for Spectre). CloudStack 4.11 system VM templates were patched at release time and are therefore not affected.

Affected components summary:

CVEACS 4.9 System VMsACS 4.11 System VMsHypervisors
Spectre 1No fix available - upgrade to 4.11Fix in releaseAffected - consult with vendor
Spectre 2No fix available - upgrade to 4.11Fix in releaseAffected - consult with vendor
MeltdownFixed - new system VM templateFix in releaseAffected - consult with vendor

Effect On CloudStack

The impact on CloudStack environments is two-fold since the vulnerabilities affect both the compute hypervisor hosts and the CloudStack system VMs.

Hypervisors

As these are low level CPU call vulnerabilities all hypervisors are affected. Hypervisor vendors have been providing patches – and may continue to do so as further analysis is carried out and potential fixes are developed. The issue with the hypervisor patches is they will potentially impact performance, something which may affect hypervisor VM density figures and/or VM guest performance. ShapeBlue therefore advise users to carry out thorough testing to determine each CloudStack environment impact before rolling these out to production. ShapeBlue can not provide further information or advise on these patches and we recommend all our community users and customers to discuss with the respective hypervisor vendors.

More information can be found in the following articles:

CloudStack System VMs

The CloudStack system VMs are also affected by Spectre and Meltdown. However since these vulnerabilities require local user access someone with malicious intent would first have to gain local access to the system VMs. Since these are locked down and secured in the first place the risk to CloudStack environments is considered low as long as general CloudStack security best practices are followed.

The CloudStack LTS branches system VMs are based on 64-bit Debian releases:

  • CloudStack 4.11 utilises Debian 9 “Stretch” 64-bit system VMs
  • CloudStack 4.9 utilises Debian 7 “Wheezy” 64-bit system VMs

CloudStack 4.11 was released in February 2018 at which point the Spectre and Meltdown fixes were already provided, and these were therefore included in the system VM templates.

However – CloudStack 4.9 utilise Debian 7 “Wheezy” system VM templates – and “Wheezy” went support end-of-life on May 31st 2018 (https://wiki.debian.org/LTS).  At this point the Debian community have only provided patches for Meltdown, and there are no indications Spectre fixes will be provided. As a result ShapeBlue have made the decision to provide new CloudStack 4.9 system VM templates with only the Meltdown patch included. Our overall recommendation if full patching of the vulnerabilities is required is to upgrade to CloudStack version 4.11.

Further information on the Debian fixes can be found in https://wiki.debian.org/DebianSecurity/SpectreMeltdown.

CloudStack 4.9 system VM templates / patching procedure

Whilst system VMs may be patched in-situ they will require reboots for the patches to take effect, and the ShapeBlue recommendation is therefore to update the system VM templates to ensure the Meltdown patch is permanently applied. ShapeBlue have built new system VM templates for CloudStack 4.9 for XenServer, VMware and KVM hypervisors. These can be downloaded from http://packages.shapeblue.com/systemvmtemplate/4.6/meltdown/. The new system VM templates have gone through the full test cycle and no regressions have been found.

The procedure for updating the system VM templates is as follows:

  • For each hypervisor type in the CloudStack environment upload the new system VM template with the following information:
    • Name: use a descriptive name, e.g. systemvm-<hypervisor>-4.6-meltdown
    • Description: add template description
    • Zone: pick the correct zone(s)
    • Hypervisor: pick the correct hypervisor
    • Format: VHD (XenServer) / OVA (VMware) / QCOW2 (KVM)
    • OS Type: Debian GNU/Linux 7.0 (64-bit) (or the highest Debian release number available in the dropdown)
    • Extractable: no
    • Password Enabled: no
    • Public: no
    • Featured: no
    • Routing: yes
  • Update the global settings for “router.template.<hypervisor>” to the same as the name configured during the template upload.
  • Restart the management service on all management servers.
  • Destroy SSVMs and CPVM instances – CloudStack management will recreate these with the new template.
  • Restart all networks with the “cleanup” option, which will recreate all VRs with the new system VM template.

Further information

For ShapeBlue support customers, please contact the support team for further information.

For other CloudStack users, please use the community mailing lists.

ShapeBlue Security Advisory – DNSMasq Vulnerabilities

, , , ,

A number of security flaws were recently found in the DNSMasq tool. This tool is used by many systems to provide DNS and DHCP services, including by the CloudStack System VMs.
This advisory explains their affect on CloudStack and how to patch CloudStack against these flaws.

Migration away from download.cloud.com to download.cloudstack.org may cause problems in exisiting cloudstack installations and versions

,

Background

Cloudstack relies on a fixed download site when it fetches the built-in guest VM templates.
That download site has historically been download.cloud.com and is being replaced by download.cloudstack.org.

Download.cloudstack.org is now fully functional. The retirement date of download.cloud.com is unknown but expected to be imminent

The issue & behaviour

After the retirement of download.cloud.com, the following issues may be experienced:

  • When installing CloudStack for the first time, failures will occur when downloading the built-in templates
  • For existing installations of CloudStack, if administrators or users attempt to re-download a template (for example when creating a new zone) failures will occur.

Versions affected

This issue affects Apache CloudStack version 4.9.2 and ALL PRIOR VERSIONS

CloudStack 4.10, due for release imminently, is NOT affected by this issue. Future versions should not be affected by this issue.

Resolution

The following steps will update an existing CloudStack installation to use the new download site. This process should also be followed (in advance of installation) when attempting to install a new instance of CloudStack for affected versions.

1. List the URLs to update
Locate the ‘cloud’ database and run this SQL command against it, replacing <user-id> and <your password>
NOTE: this requires having the MySQL client installed on the location that you are running the command from.

$ echo "SELECT id,url FROM vm_template WHERE url LIKE '%download.cloud.com%' AND NOT removed IS NULL\g" | mysql -h <cloud db server ip or hostname> -u <user-id> -p<your password> cloud

This will return all URLs that CloudStack uses for downloads that are pointing to download.cloud.com

A number of rows should be returned. The following is a sample output

id  url

11  http://download.cloud.com/templates/builtin/centos-7-x86_64.tar.gz

13  http://download.cloud.com/templates/4.2/systemvmtemplate-4.2-vh7.ova

If no rows are  returned you are not affected by this issue; you need to do nothing further. If rows are returned, proceed to step 2

2. Check that ALL templates are present on the new download site.

All templates that were previously located at download.cloud.com should be in an identical location at download.cloudstack.org. However, we advise that you confirm this  by to attempting to manually  download all of  the templates from the same directory at downloads.cloudstack.org

To do this, take every result returned at step 1 and attempt to manually  download them from the same location at downloads.cloudstack.org

Taking the above examples check that you are able to  download:
http://download.cloudstack.org/templates/builtin/centos-7-x86_64.tar.gz

http://download.cloudstack.org/templates/4.2/systemvmtemplate-4.2-vh7.ova

(the files don’t actually need to be downloaded at this stage, you are just checking for their existence)

If all templates are present, then continue to step 3. If any are missing (you will receive a 404 error), then contact users@cloudstack.org or your support provider

3. Update the URLs in the vm_template table

WARNING – Extreme care must always be taken when directly manipulating the database.  Ensure that you have a very recent backup.
Update any URL’s that point to download.cloud.com. This can be performed by running:

$ echo "UPDATE vm_template SET url = REPLACE(url, 'http://download.cloud.com', 'http://download.cloudstack.org') WHERE INSTR(url, "http://download.cloud.com") > 0 AND removed IS NULL;" | mysql -h <cloud db server ip or hostname> -u <user-id> -p<your password> cloud

4. Double check the update

Running the following command should return the same IDs as before, but with the new download location

$ echo "SELECT id,url FROM vm_template WHERE url LIKE '%download.cloudstack.org%' AND NOT removed IS NULL\g" | mysql -h <cloud db server ip or hostname> -u <user-id> -p<your password> cloud

Shapeblue Security Advisory For CVE-2016-6813: Apache CloudStack registerUserKeys authorization vulnerability

, , , ,

Overview

Apache CloudStack provides a registerUserKeys API that allows a user to create or recreate a secret key and an API key to use for authentication when using the CloudStack API. A malicious user can request this API action in conjunction with the ID of another CloudStack user/account.  The newly created or re-generated API keys for this other user would then be returned to the malicious user, giving them access the other user’s account and resources. The issue affects all users of CloudStack 4.1 and above.

NOTE: In order to exploit this vulnerability the malicious user must themselves have authenticated API access.

ShapeBlue have worked with the security team to create these instructions and security release(s), and all CloudStack operators are advised to follow these instructions or upgrade. It is thought that this vulnerability also affects commercial distributions of Apache CloudStack.

Credit: This vulnerability was reported by Marc-Aurèle Brothier at Exoscale.

Resolution

CloudStack operators should upgrade to one of the following security release versions, based on releases they are currently using: 4.5.2.2, 4.8.1.1, or 4.9.0.1. Please ensure that you upgrade to the relevant version.

These versions contain only security updates, and no other functionality change. The rpm and debian packages for these versions are available from ShapeBlue’s CloudStack repositories that can be used to upgrade the cloudstack-management package. Stopping the management service(s), upgrading all management servers and then restarting them is sufficient.

Short Term Mitigation

Performing the following steps will restrict the API to use by your root-admin account users only.
USERS and DOMAIN ADMINS will NO LONGER be able to create or re-generate API keys themselves.

Affected users can restrict this API to only the root admin user/accounts by setting the API to octet 1 (i.e. only root-admin accounts are allowed). To do this, open and edit the ‘commands.properties’ file (usually found at /etc/cloudstack/management/commands.properties or at /usr/share/cloudstack-management/webapps/client/WEB-INF/classes/commands.properties), then search and replace the API config to:
registerUserKeys=1

Save the file and restart your management server.

Users of CloudStack 4.9 who are using dynamic roles feature, can navigate to UI->Roles and go to Rules section of each of the non-admin role, and delete a configured ‘Allow’ rule for ‘registerUserKeys’.

 

Further information

For ShapeBlue support customers, please contact the support team for further information.

For other CloudStack users, please use the community mailing lists.

For users of commercial distribution of CloudStack, please contact your vendor

Shapeblue Security Advisory For CVE-2016-3085: Apache CloudStack Authentication Bypass Vulnerability

, , ,

Overview

Apache CloudStack contains an authentication module providing “single sign-on” functionality via the SAML data format. Under certain conditions, a user could manage to access the user interface without providing proper credentials. As the SAML plugin is disabled by default, this issue only affects installations that have enabled and use SAML-based authentication.

Mitigation:
Users of Apache CloudStack using the SAML plugin should upgrade to one of the following versions, based on which release they are currently using: 4.5.2.1, 4.6.2.1, 4.7.1.1, or 4.8.0.1. These versions contain only security updates, and no other functionality change.

Versions affected:
CloudStack versions 4.5.0 and newer with SAML authetication enabled.

 

What is ShapeBlue Doing

ShapeBlue has analysed the impact of this issue on Apache CloudStack (ACS). The issue only affects users of CloudStack version 4.5.0 and newer, who use CloudStack SAML plugin. The vulnerability allows an attacker to bypass SAML authentication and log in as a SAML user using any non-empty password. ShapeBlue discovered the issue, reported it to the CloudStack security team and created the necessary secuity pathces. We have since worked with the secuity team to create the security release(s) that all CloudStack operators are recommended to update to. It is also thought that this vulnerability affects commercial distributions of Apache CloudStack.

Security release upgrade procedure

Users of Apache CloudStack using the SAML plugin should upgrade to one of the following versions, based on which release they are currently using: 4.5.2.1, 4.6.2.1, 4.7.1.1, or 4.8.0.1. These versions contain only security updates, and no other functionality change. The rpm and debian packages are available from ShapeBlue’s CloudStack repositories that can be used by users to upgrade the cloudstack-management package and restart their management server(s) to fix the issue.

Further information

For ShapeBlue support customers, please contact the support team for further information.

For other CloudStack users, please use the community mailing lists.

For users of commerical distribution of Cloudstack, please contact your vendor

 

Shapeblue Security Advisory for CVE-2015-0235, aka the Ghost vulnerability

, , ,

Overview

A vulnerability has been recently disclosed by Qualys that could result in a remote attacker being able to execute malicious instructions on vulnerable systems. The vulnerability affects Linux based operating systems.

This is better known as GHOST ‘glibc’ vulnerability (CVE-2015-0235): https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-0235

What is ShapeBlue Doing

ShapeBlue has analysed the impact of this issue on Apache CloudStack (ACS).  The download template functionality provided by the SSVM to the end user puts it at risk. Since it is a linux issue all the Apache CloudStack versions are affected.  An immediate fix would be to login into each SSVM and upgrade the glib package to the one that has the fix.  This is a temporary solution as on a reboot the template falls back to the configured version.  We have released an updated set of System VM Templates that have the fix applied as a permanent solution.  It is recommended that all CloudStack operators update their System VM Templates to a patched version following the instructions below.  It is also thought that this vulnerability affects commercial distributions of Apache CloudStack.

Virtual Appliance patching procedure

ShapeBlue has built new systemvm templates with glib fix for major CloudStack versions 4.3, 4.4 and 4.5 for XenServer, VMware and KVM hypervisors. We advise CloudStack users to upgrade to the  latest systemvm templates using the following steps:

1.   Register the new templates for all the hypervisors that you’re using in your CloudStack deployment.  If you’re a CloudStack 4.3 user please use the settings in this table, 4.4 users please scroll down the page.

Hypervisor
Description for 4.3 Users
XenServer

Name: systemvm-xenserver-4.3-ghost
Description: systemvm-xenserver-4.3-ghost
URL: http://packages.shapeblue.com/systemvmtemplate/4.3/systemvm64template-4.3-xen.vhd.bz2
Zone: Choose the zone where this hypervisor is used
Hypervisor: XenServer
Format: VHD
OS Type: Debian GNU/Linux 7.0 (64-bit) (or the highest Debian release number available in the dropdown)
Extractable: no
Password Enabled: no
Public: no
Featured: no
Routing: no

KVM

Name: systemvm-kvm-4.3-ghost
Description: systemvm-kvm-4.3-ghost
URL: http://packages.shapeblue.com/systemvmtemplate/4.3/systemvm64template-4.3-kvm.qcow2.bz2
Zone: Choose the zone where this hypervisor is used
Hypervisor: KVM
Format: QCOW2
OS Type: Debian GNU/Linux 7.0 (64-bit) (or the highest Debian release number available in the dropdown)
Extractable: no
Password Enabled: no
Public: no
Featured: no
Routing: no

VMware
Name: systemvm-vmware-4.3-ghost
Description: systemvm-vmware-4.3-ghost
URL: Coming Soon !!
Zone: Choose the zone where this hypervisor is used
Hypervisor: VMware
Format: OVAOS
Type: Debian GNU/Linux 7.0 (64-bit) (or the highest Debian release number available in the dropdown)
Extractable: no
Password Enabled: no
Public: no
Featured: no
Routing: no

 

Or, if you’re a CloudStack 4.4 user please use the following table:

Hypervisor
Description for 4.4 Users
XenServer

Name: systemvm-xenserver-4.4-ghost
Description: systemvm-xenserver-4.4-ghost
URL: http://packages.shapeblue.com/systemvmtemplate/4.4/systemvm64template-4.4-xen.vhd.bz2
Zone: Choose the zone where this hypervisor is used
Hypervisor: XenServer
Format: VHD
OS Type: Debian GNU/Linux 7.0 (64-bit) (or the highest Debian release number available in the dropdown)
Extractable: no
Password Enabled: no
Public: no
Featured: no
Routing: no

KVM

Name: systemvm-kvm-4.4-ghost
Description: systemvm-kvm-4.4-ghost
URL: http://packages.shapeblue.com/systemvmtemplate/4.4/systemvm64template-4.4-kvm.qcow2.bz2
Zone: Choose the zone where this hypervisor is used
Hypervisor: KVM
Format: QCOW2
OS Type: Debian GNU/Linux 7.0 (64-bit) (or the highest Debian release number available in the dropdown)
Extractable: no
Password Enabled: no
Public: no
Featured: no
Routing: no

VMware
Name: systemvm-vmware-4.4-ghost
Description: systemvm-vmware-4.4-ghost
URL: Coming Soon !!
Zone: Choose the zone where this hypervisor is used
Hypervisor: VMware
Format: OVAOS
Type: Debian GNU/Linux 7.0 (64-bit) (or the highest Debian release number available in the dropdown)
Extractable: no
Password Enabled: no
Public: no
Featured: no
Routing: no

2.  After registering the template, wait until the template gets downloaded and is installed. When the template is in READY state move to the next step.

3.  Stop the cloudstack-management service all the management server(s) and take a backup of the database.

service cloudstack-management stop

4.  Download ShapeBlue’s CloudStack SystemVM template upgrading tool for Ghost vulnerability:

wget http://packages.shapeblue.com/tools/ghost-systemvm-upgrader.py

5. This tool requires Python MySQLDB library, to install Python MySQLDB dependency run the following:

Debian based: apt-get install python-mysqldb

RHEL based: yum install MySQL-python

Or, Use pip:

Debian based: apt-get install build-essential python-dev libmysqlclient-dev

RHEL based: yum install mysql-devel python-devel

sudo pip install MySQL-python

6.  Run this tool against your database, it will automatically find and upgrade the registered template:

python ghost-systemvm-upgrader.py -c <Major CloudStack version, 4.3 or 4.4> -i <IP of the database host> -d cloud -u <Database user name, cloud> -p <Database user password>

e.g.   python ghost-systemvm-upgrader.py -c 4.4 -i 192.168.0.21 -d cloud -u cloud -p password

7.  Finally, restart the CloudStack Management service on all management servers and then destroy all of the SystemVMs

 service cloudstack-management start

 

Further information

For ShapeBlue support customers, please contact the support team for further information.

For other CloudStack users, please use the community mailing lists.

Shellshock and CloudStack

, , ,

Shellshock is the family of bugs in the Unix Bash shell which allows an attacker to execute arbitrary commands on a vulnerable system potentially allowing an attacker to gain full access to that system. The bug (CVE-2014-6271) was first disclosed on 24 September 2014, upon closer inspection of the code, related vulnerabilities (CVE-2014-6277CVE-2014-6278CVE-2014-7169CVE-2014-7186, and CVE-2014-7187) were discovered. The bug is thought to have been in the Bash code since 1992.

Protecting Against Shellshock Attacks In a CloudStack Environment

The first line of defense is to keep all management functions in a private, firewalled network; denying would-be attackers to opportunity to reach vulnerable systems.

The next step is to patch all management servers (ie CloudStack Management servers, MySQL servers, BIND DNS servers etc.) running Linux OSes. Either yum update bash or apt-get update; apt-get install –only-upgrade bash will work on most Linux flavours.

The usual precautions should be taken when doing updates; ensuring you have good backups and taking systems to be patched off-line before commencing.

KVM compute hosts can also be patched in this way using yum or apt-get. Citrix have released a patch for XenServer https://support.citrix.com/article/CTX200223. This also applies to the open sourced versions of XenServer. VMware ESXi is not effected as it does not use bash, however other components of a vSphere environment may be effected so consult http://www.vmware.com/security/advisories/VMSA-2014-0010.html for details

Potentially the most complicated step is patching the system VMs as these can be rebuilt from the templates, so the templates must be patched as well.  As the system VMs are Debian based, then apt-get update; apt-get install –only-upgrade bash will update bash to a patched version.

The final step is to remind all creators/users of Linux based guest instances to patch their virtual machines.

Retirement of the realhostip.com Service

, , ,

The realhostip.com service will be switched off on the 1st October 2014. Paul Angus looks at what it did, what effect the retirement will have and what you need to do to carry on working if you’re affected.

What is realhostip.com?

When you connect to the Console Proxy system VM or download a disk or ISO from the secondary storage VM you connect over a secure (https) connection. This is particularly important when you put in your password.  In order for this to be secure you need to connect to a URL which has a FQDN and have a certificate to go with it.

The realhostip.com domain and DNS service was created to give an out of the box solution to this problem. By using the realhostip.com domain and a clever bit of magic, with hostnames and a dynamic DNS script, the data transmitted is encrypted.

The problem is that anyone can use this system, which means that just because a VM has a certificate that doesn’t mean you can trust it. Also, someone could extract the certificate from a CloudStack environment and then use that to decrypt others’ traffic. Its very unlikely but technically possible.

So that’s why the use of a publicly available certificate is being retired.

Who will this affect?

For clean installs of CloudStack 4.3 and later do not use realhostip and instead use HTTP (unencrypted) communications to the public interfaces – so you have to arrange the certificate yourself from the get go. So if you’re using 4.3 or later then the switching off of the service won’t affect you. However if you are using 4.3 or later and haven’t installed a certificate then you should install one for the security of your users. Now.

For users of CloudStack 4.3 environments which have been upgraded from earlier versions, you have the additional option of switching HTTPS off for the console proxy and secondary storage but you do so at your own risk and I can’t recommend it for anything which is public facing or not in the most isolated of test environments.

So for versions of CloudStack pre v4.3 or environments which have been upgraded from a previous version, admins need to replace realhostip.com with their own domain and certificate otherwise the console proxy service and downloading from the secondary storage VM will stop working on 1st October 2014.

What to do to replace realhostip.com

There a number of sources which explain how to use your own domain for secure console proxy and secondary storage, so I’ll point you to the various sources and add some extra depth to the references.

The first thing you need is a domain to use to replace realhostip.com. Generally people use a subdomain of their corporate domain (such as console.supercloud.com) so that it can be managed separately from the parent domain. You don’t have to use a subdomain by any means, but I’m going to assume you have for the purposes of this blog. Then you need a DNS server or two which will act as the resolvers for your subdomain.  The resolvers need to resolve all addresses in your public ranges. Part of the cleverness of the realhostip solution is in the way CloudStack creates FQDNs for the secondary storage VMs and console proxy VMs which are easy to work with. For example, for the IP address 1.2.3.4 CloudStack would create an FQDN of 1-2-3-4.realhostip.com and for 4.3.2.1 it would be 4-3-2-1.realhostip.com.  It makes creating the zone files easy and even makes it possible to create a DNS server that will resolve anything that’s thrown at it in this format. Indeed that’s what the realhostip DNS servers did (source code is here https://github.com/ke4qqq/RHIP). You’re going to replace that with 1-2-3-4.console.supercloud.com

Things get trickier if your DNS is hosted by a 3rd party and you have a lot of public IPs which the system VMs could use. Creating the entries for a zone file in a spreadsheet is easy, but if you had to fill in a web form for your ISP for each IP address it’s going to be a long, boring and error-prone task, so you’d probably look at creating your own DNS servers for your subdomain or trying to get your ISP to do the heavy lifting.

So we have a domain to use and some DNS servers to resolve it with, now we need a certificate to secure things with.  You’ll need a wildcard certificate for the subdomain as you need it to work for multiple hosts *. console.supercloud.com. A quick search for ‘wildcard certificate’ will bring back a bucket load of vendors of wildcard certificates.  I’ve found that large enterprises will likely have a system in place for obtaining certificates, so a call to corporate IT may save your credit card and a load of time.

To get a certificate you need to generate a private key and then a certificate signing request (CSR) which is used by the certificate provider to generate a certificate based on the private key.  A procedure for doing this is documented in the CloudStack documentation – you need openssl installed which is easy enough, but your CloudStack management server(s) will have it installed so you can always do it from there. http://docs.cloudstack.apache.org/en/latest/administration_guide.html#changing-the-console-proxy-ssl-certificate-and-domain

You can now go to the certificate vendor and request your wildcard certificate.

Once you get your wildcard certificate you’ll need the public certificate of root certificate authority (CA) and the public certificate(s) of intermediate CA(s) (if any). They may be sent with your wildcard certificate or you may need to go to the website of your certificate provider to download them. You need the whole chain so that a client can see that the certificate is bona fide.

Now is the most complicated bit – getting the everything in the correct format. Why this process always ends up being such a pain is beyond me, but anyone who’s tried to add a certificate to a NetScaler, IIS, RDS or CloudStack will testify that it’s rarely simple – the certificate and/or private key you have never seem to be in quite the right format or you’re missing an intermediate certificate, so stick with it – you’re not alone.

The openssl tool is pretty good for converting between formats.

So in all you should have the following:

  1. Public certificate of root CA in PEM format
  2. Public certificate(s) of intermediate CA(s) (if any) in PEM format
  3. Wildcard domain certificate in PEM format
  4. Private key in PKCS8 format
  5. It should sound obvious but you must make sure that from the public internet your public IP addresses get resolved i.e. 1-2-3-4. console.supercloud.com -> 1.2.3.4 before you make the switchover

Now that you’re ready to make the change you’ll need to do the following steps

  1. Update the CloudStack settings to use your new certificate
  2. Upload the new certificate to the management server
  3. Reboot the system VMs to use the new certificates
  4. Verify the new certificate is working

These steps are well documented with additional help also on the CloudStack Wiki

Procedure to Replace realhostip.com with Your Own Domain Name
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Procedure+to+Replace+realhostip.com+with+Your+Own+Domain+Name

Things you should consider while changing the Realhost domain to custom Domain.
https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=43188254

Troubleshooting – uploading custom domain certificate instead of using realhostip.com
https://cwiki.apache.org/confluence/display/CLOUDSTACK/Troubleshooting+-+uploading+custom+domain+certificate+instead+of+using+realhostip.com

An extra pointer from our experience is that the system VMs should be stopped, then started, both from CloudStack management server, otherwise the new certificate will not be applied.

Chip Childers wrote this article a while back and the system for importing certificates has been improved since them but it may provide additional help
http://www.chipchilders.com/blog/2013/1/2/undocumented-feature-using-certificate-chains-in-cloudstack.html

 Summary

In this article we looked at what the realhostip.com service does and how its retirement will effect CloudStack/CloudPlatform clouds. We also gave information on the steps required to replace the realhostip.com service if you need to secure your cloud.

About the Author

Paul Angus is a Cloud Architect at ShapeBlue, The Cloud Specialists. He has designed and implemented numerous CloudStack environments for customers across 4 continents, based on Apache Cloudstack ,Citrix Cloudplatform and Citrix Cloudportal.

When not building Clouds, Paul likes to create scripts that build clouds……..and he very occasionally can be seen trying to hit a golf ball.

How to Mitigate OpenSSL HeartBleed Vulnerability in Apache CloudStack

, ,

UPDATE: 09-Apr-2014 – The proper upgrade command is “apt-get install openssl libssl1.0.0”. If you’ve just updated openssl, please go back and update libssl as well.

UPDATE: 10-Apr-2014 – Added detailed verification steps / Apache CloudStack 4.0 – 4.1 are not vulnerable, they use older Debian/openssl.

Thanks to all involved for helping to put together and update this information

 

Earlier this week, a security vulnerability was disclosed in OpenSSL, one of the software libraries that Apache CloudStack uses to encrypt data sent over network connections. As the vulnerability has existed in OpenSSL since early 2012, System VMs in Apache CloudStack versions 4.0.0-incubating-4.3 are running software using vulnerable versions of OpenSSL. This includes CloudStack’s Virtual Router VMs, Console Proxy VMs, and Secondary Storage VMs.

The CloudStack community are actively working on creating updated System VM templates for each recent version of Apache CloudStack, and for each of the hypervisor platforms which Apache CloudStack supports. Due to testing and QA processes, this will take several days. In the meantime, a temporary workaround is available for currently running System VMs.

If you are running Apache CloudStack 4.0.0-incubating through the recent 4.3 release, the following steps will help ensure the security of your cloud infrastructure until an updated version of the System VM template is available:

Logon to each Secondary Storage VM, Console Proxy VM and Virtual Router and update openssl

XenServer & KVM

  1. Use the GUI to identify the Link Local IP and Host of the VM
  2. Connect to the Hypervisor Host using SSH
  3. From the Host, Connect to the VM using the following command, replacing n.n.n.n with the Link Local IP identified in step 1.
  4. On the System VM,When updating Secondary Storage VMs, run /etc/init.d/apache2 restart
    • run apt-get update
    • then run apt-get install openssl libssl1.0.0
    • If a dialog appears asking to restart programs, accept its request
  5. Log out of the System VM and host server
  6. Repeat for all Secondary Storage, Console Proxy and Virtual Router VMs

 

 VMware

  1. Use the GUI to identify the Management / Private IP of the VM
  2. SSH onto a CloudStack Management Server
  3. From the Management Server, connect to the VM using the following command, replacing n.n.n.n with the Managemnt/Private IP identified in step 1.
    • ​ssh -i /var/lib/cloud/management/.ssh/id_rsa -p 3922 root@n.n.n.n
  4. On the System VM,When updating Secondary Storage VMs, run /etc/init.d/apache2 restart
    • run apt-get update
    • then run apt-get install openssl libssl1.0.0
    • If a dialog appears asking to restart programs, accept its request
  5. Log out of the System VM and host server
  6. Repeat for all Secondary Storage, Console Proxy and Virtual Router VMs

 

Verification

On each System VM, you can test if it has non-vulnerable openssl packages installed by listing installed packages and looking at the installed versions of openssl and libssl. As in the example below, for a system to be non-vulnerable, the packages need to be at or above version 1.0.1e-2+deb7u6:

root@v-14-VM:~# dpkg -l|grep ssl
ii  libssl1.0.0:i386                1.0.1e-2+deb7u6                  i386         SSL shared libraries
ii  openssl                              1.0.1e-2+deb7u6                  i386         Secure Socket Layer (SSL) binary and related cryptographic tools

We realise that for larger installations where System VMs are being actively created and destroyed based on customer demand, this is a very rough stop-gap. The Apache CloudStack security team is actively working on a more permanent fix and will be releasing that to the community as soon as possible.

For Apache CloudStack installations that secure the web-based user-interface with SSL, these may also be vulnerable to HeartBleed, but that is outside the scope of this blog post. We recommend testing your installation with [1] to determine if you need to patch/upgrade the SSL library used by any web servers (or other SSL-based services) you use.

Information originally posted on https://blogs.apache.org/cloudstack/entry/how_to_mitigate_openssl_heartbleed

1: http://filippo.io/Heartbleed/

About the Author

Geoff Higginbottom is CTO of ShapeBlue, the strategic cloud consultancy and an Apache CloudStack Committer. Geoff spends most of his time designing private & public cloud infrastructures for telco’s, ISP’s and enterprises based on CloudStack.