Projects have proven to be a boon in organizing and grouping accounts and resources together, giving users in the same domain the ability to collaborate and share resources such as VMs, snapshots, volumes and IP addresses. However, there is a limitation. Only accounts can be added as members to projects, which can be an issue if we only want to add a single user of an account to a project. To address this, we’ve enhanced the way project membership is handled to facilitate addition of individual users.

Adding users to projects and assigning project-level roles

In order to restrict users in projects to a limited set of operations (adding further restrictions to those already defined by their account-level roles) we’ve brought in the concept of .
Project Roles are characterized by name and Project ID, and a project can have many project roles. Project Roles are then associated with Project Role Permissions which determine what operations users / accounts associated with a specific role can perform. It is crucial to understand that project-level permissions will not override those set at Account level.

Creation of Project Roles via the API:
$ create projectrole name=<projectRoleName> projectid=<project_uuid> description=<optional description>

Creation and association of a project role permission with a project role via the API:
$ create projectrolepermission projectid=<project_uuid> projectroleid=<project_role_id> permission=<allow/deny> rule=<API name/ wildcard> description=<optional description>

One can also create project roles and project role permissions from the UI:

1. Navigate to the specific project and enter its Details View

2. Go to the Project Roles Sub-tab and click on the Create Project Role button. Fill in the required details in the pop-up form and click OK:

3. To associate project role permissions to the created role, click on the + button on the left of the project role name and hit the ‘Save new Rule’ button:

The re-order button to the left of the rule name will invoke the ‘updateprojectRolePermission’ API as follows:

$ update projectrolepermission projectid=<project_uuid> projectroleid=<project_role_uuid> ruleorder=<list of project rule permission uuids that need to be moved to the top>

Other parameters that the updateProjectRolePermission API can take are:

4. One can also update the permission, namely Allow / Deny associated with the rule, by selecting the option from the drop-down list:

This invokes the ‘updateProjectRolePermission’ API, but passes the permission parameter instead of rule order, as follows:

$ update projectrolepermission projectid=<project_uuid> projectroleid=<project_role_uuid> projectrolepermissionid=<uuid of project role permission> permission=<allow/deny>

Now that we’ve seen how we create / modify the project roles and permissions, let’s understand how we associate them with users / accounts for them to take effect. When adding / inviting users or accounts to projects, we can now specify the project role:


The API call corresponding to this operation is ‘AddUserToProject’ or ‘AddAccountToProject’ and can be invoked as follows:

$ add userToProject username=<name of the user> projectid=<project_uuid> projectroleid=<project_role_uuid>

Project Admins

Regular users or accounts in a project can perform all management and provisioning tasks. A project admin can perform these tasks as well as administrative operations in a project such as create / update / suspend / activate project and add / remove / modify accounts. With this feature, we can have multiple users or accounts as project admins, providing more flexibility than before.

1. Creation of Projects with a user as the default project admin
The ‘createProject’ API has been extended to take user ID as an input along with account ID and domain ID:

$ create project name=<project name> displaytext=<project description> userid=<uuid of the user to be added as admin> accountid=<uuid of the account to which the user belongs to> domainid=<uuid of the domain in which the user exists>


2. Multiple Project Admins
Change the default ‘swap owner’ behaviour (single project admin allowed) to allow multiple project admins and promote / demote users to project admins / regular users respectively. Use 2. the ‘Promote’ (up arrow) or ‘Demote’ (down arrow) buttons to change the role of a user in a Project:

Please note:

1. Admins, Domain Admins or Project Admins permissions will never be affected by a changed project role
2. One cannot demote / delete the project admin if there is only one

If a role type isn’t specified while adding / inviting users to projects, then by default they become regular members. However, we can override this behaviour by passing the ‘roletype’ parameter to the ‘addAccountToProject’ or ‘addUserToProject’ APIs.

Upgrading CloudStack from any lower version to 4.15 will not affect existing projects and their members. Furthermore, in case we still want the swap owner feature we have the ‘swapowner’ parameter as part of ‘updateProject’ API (which by default is set to true for backward compatibility against the legacy UI). This parameter should be set to false if we want to promote or demote a particular member of the project.

In conclusion, this feature enhances the way Projects behave such that everything that happened at the Account level is now made possible at user level too. This feature will be available as part of CloudStack 4.15 LTS.

CloudStack has more than 600 APIs which can be allowed / disallowed in different combinations to create dynamic roles for the users. The aim of this feature is more effective use and management of these dynamic roles, allowing CloudStack users and operators to:

  1. Import and export roles (rule definitions) for the purpose of sharing.
  2. Create a new role from an existing role (clone and rename) to create a slightly different role.
  3. Use additional built-in roles (to quickly create read-only and support users and operators), such as:
    • Read-Only Admin role: an admin role in which an account is only allowed to perform any list / get / find APIs but not perform any other operation or changes to the infrastructure, configuration or user resources.
    • Read-Only User role: a user role in which an account is only allowed to perform list / get / find APIs who may only be interested in monitoring and usage for instance.
    • Admin-Support role: an admin role in which an admin account is limited to perform day-to-day tasks, such as creating offerings, but cannot change physical networks or add / remove hosts (but can put them in maintenance).
    • User-Support role: a user role in which an account cannot create or destroy resources (any create*, delete* etc. APIs are disallowed) but can view resources and perform operations such as start / stop VMs and attach / detach volumes, ISOs etc.

The existing role types (Admin, Domain Admin, Resource Admin, User) remain unchanged. This feature deals purely with the Dynamic Roles which filter the APIs which a user is allowed to call. The default roles and their permissions cannot be updated or deleted.

Cloning a role

An existing role can be used to create a new role, which will inherit the existing role’s type and permissions. A new parameter: roleid is introduced in the existing createRole API which takes the existing role id as the input, to clone from. The new role created can be modified to create a slightly different role later.
Example API call:

The Add Role dialog screen shown below is used to create a new role by selecting an existing role:

Import role and export rule definitions

A role can be imported with its rule definitions (rule, permission, description) using a new API: importRole with the following parameters:

  • name (Type: String, Mandatory) – role name
  • type (Type: String, Mandatory) – role type, any of the four role types: Admin, Resource Admin, Domain Admin, User
  • description (Type: String, Optional) – brief description of the role
  • rules (Type: Map, Mandatory) – rules set in the sort order, with key parameters: rule, permission and description
  • force (Type: Boolean, Optional)- whether to override any existing role (with same name and type) or not, “true” / “false”. Default is false

Example API call:

The import role option in the Roles section of the UI opens up the Import Role dialog screen below. Here you can specify the rules with a CSV file with rule, permission and description in the header row, followed by the rule values in each row:

The imported rule definitions are added to the rule set of the role. If a role already exists with the same name and role type, then the import will fail with a ‘role already exists’ message, unless it is forced to override the role by enabling the force option in the UI or setting the “force” parameter to true in the importRole API.

The ‘Export rules’ operation for a role is allowed at the UI level only, at the rules details as shown below. This operation fetches the rules for the selected role and exports to a CSV file. The exported rule definitions file can be thereafter used to import a role.

The rule definitions import / export file (CSV) contains details of role permissions. Each permission is defined in a row with comma-separated rule, permission (allow/deny) and description values. The row sequence of these permission details is considered to be the sort order, and the default export file name is “<RoleName>_<RoleType>.csv”.
Example CSV format:


…and so on, where:

  • Rule – Specifies the rule (API name or wildcard rule, in valid format)
  • Permission – Whether to “allow” or “deny”
  • Description – Brief description of the role permission (can be empty)

Example file (.csv), for TestUser with User role, TestUser_User.csv contains:

listVirtualMachines,allow,listing VMs
listVolumes,allow,volumes list
createNetworkACLList,deny,not allow acl
delete*,allow,delete permit

TestUser_User.csv shown in a spreadsheet (for clarity):








New built-in roles

New read-only and support roles (with pre-defined sets of permissions) for user and operator, namely Read-Only Admin, Read-Only User, Support Admin and Support User have been added to quickly create read only & support users and admins.

CloudStack doesn’t allow any modifications to built-in roles (new & existing), i.e. these default roles and their permissions cannot be updated, deleted or overridden. The image below shows new and existing built-in roles:

The following permissions are applicable for these roles:

  • Read-Only Admin: an admin role in which an account is only allowed to perform all list APIs, read-only get and quota APIs.
  • Read-Only User: a user role in which an account is only allowed to perform list APIs, read-only get and quota APIs which has user level access.
  • Support Admin: an admin role in which an account is only allowed to perform creating offerings, host/storage maintenance, start/stop VMs, Kubernetes Cluster, attach/detach volumes, ISOs.
  • Support User: a user role in which an account is only allowed to perform, start/stop VMs, Kubernetes Cluster, attach/detach volumes, ISOs.

Any of these roles can be selected to create an account:

This feature will be included in Apache CloudStack 4.15, which is an LTS release.




CloudStack supports sharing templates and ISOs between accounts and projects through the API ‘updateTemplatePermissions’ and sharing templates through the UI. However, prior to version 4.15, it was not possible to share ISOs from the UI. This feature introduces support for sharing ISOs through different accounts and / or projects via the UI.

With this feature, a user or administrator must be able to update the permissions for an ISO via API and UI, being able to:

  • Share the ISO with another account
  • Share the ISO with another project
  • Revoke the access to the ISO for an account
  • Revoke the access to the ISO for a project
  • Reset the ISO permissions to the default

A new button is added in the ISO details view: ‘Update ISO permissions’. This button is located at the top right corner of the ISO detail view:

Once clicked, the user is prompted with a dialogue like the one below:

The user or administrator must complete three fields:

  • Operation: Must be one of the following values: ‘Add’, ‘Remove’, ‘Reset’
  • Share With: It is not displayed if the Operation field is set to ‘Reset’. If the Operation field is set to ‘Add’ or ‘Remove’ then the possible values are: ‘Account’, ‘Project’
  • [Account/Project]: It is not displayed if the Operation field is set to ‘Reset’. The field label depends on the selected ‘Share With’ field type. In this field, the user or administrator must provide an account or project name to be added to or removed from the permitted list of the ISO.
    • When ‘allow.user.view.all.domain.accounts’ = true: the dialog box displays a list of accounts within the same domain, otherwise the user must specify a comma-separated list of account names instead of selecting the account names from a list.

The ISOs shared with a user are displayed under the ‘Shared’ section of the ISOs list view:

This feature Will be available from CloudStack 4.15.

As of 2021, CentOS 7 will be receiving maintenance updates only, and is end of life in 2024. Considering this, it is important that CloudStack supports CentOS 8 as a KVM hypervisor host and as a host for the management and usage servers. This support has been developed and will be included as of CloudStack 4.15.

CentOS 8 uses a more recent QEMU version, Python 3 by default and deprecates several networking tools (such as bridge-utils), therefore a number of changes have been made:

  • Python scripts related to setting up of management and usage servers and KVM agent have been migrated from Python 2 to Python 3.
  • Python 2 dependencies from cloudstack packages (cloudstack-common, cloudstack-management, cloudstack-usage and cloudstack-agent) have been removed.
  • Support for MySQL 8 (as CentOS 8 installs this by default).

With this feature, changes have also been made to the snapshot related codebase on KVM to support the newer version of the qemu-img utility. This should prevent issues with snapshot management on an OS with newer QEMU version.
KVM hosts and management / usage servers on CentOS 7 will continue to work as before and only new python 3 dependencies (python3, python3-pip and python3-setuptools) will be installed during upgrade.

XCP-ng (an open-source hypervisor based on XenServer) has been supported in CloudStack for some time, and support for XCP-ng 8.x will be available from CloudStack 4.15.

From an operational perspective, there is no visible change in the UI or API – XCP-ng 8.x hosts can be added to CloudStack during zone deployment or by using the addHost API / UI. With guest OS mappings added for XCP-ng 8.x, once added all VM operations can be performed on such hosts just like any older XCP-ng hosts unless they are functionally not available on XCP-ng 8.x. It should be noted that with XCP-ng 8.1, support for paravirtualized (PV) VMs has been removed, therefore PV guests and older guest OS will not work on 8.x hypervisor hosts. More information about supported and removed functionalities in XCP-ng 8.x can be found in the XCP-ng documentation.

Users and administrators sometimes need to change the boot order of a virtual machine (VM) so that they can (for example) force a VM to boot from an ISO. However, sometimes the proxied console session does not appear until after the opportunity to enter the BIOS menu has passed. This feature allows an admin to request that a VM boots into the BIOS or other hardware setup screen rather than into the operating system (OS). The nature of the setup depends on the underlying hardware version and the OS of the virtual machine, and can be BIOS or UEFI. This feature applies to VMware only (as XenServer and KVM do not provide a BIOS).

With this feature, when deploying a virtual machine the boot type can be chosen. It can be BIOS or UEFI, and if UEFI it can be Secure or Legacy. The choice made must be supported by the underlying hardware and the virtual machine’s OS and cannot be changed once a VM instance has been created.

In addition to the boot type options “bootintosetup” can be specified. The latter option is also available on reboot or starting a stopped machine.

Some examples from cloudmonkey:

New parameters for deploy vm:

     (localcloud) SBCM5> > deploy virtualmachine boot

     bootintosetup= bootmode=      boottype=

Boot into setup takes a Boolean value:

     (localcloud) SBCM5> > deploy virtualmachine bootintosetup=

     true   false

On trying to boot a VM into hardware setup will fail. As can be seen the option is only implemented for VMware:

     (localcloud) SBCM5> > reboot virtualmachine id=68ba4b83-12ec-4da1-aefa-8462d9135f00 bootintosetup=true


     "accountid": "1508af67-9c07-11ea-8893-1e000601081e",

     "cmd": "org.apache.cloudstack.api.command.admin.vm.RebootVMCmdByAdmin",

     "completed": "2020-05-22T16:33:29+0000",

     "created": "2020-05-22T16:33:29+0000",

     "jobid": "0137c77f-0a00-47b8-adcf-959315846000",

     "jobinstanceid": "68ba4b83-12ec-4da1-aefa-8462d9135f00",

     "jobinstancetype": "VirtualMachine",

     "jobprocstatus": 0,

     "jobresult": {

     "errorcode": 431,

     "errortext": " Booting into a hardware setup menu is not implemented on KVM"


     "jobresultcode": 530,

     "jobresulttype": "object",

     "jobstatus": 2,

     "userid": "150c8809-9c07-11ea-8893-1e000601081e"


     🙈 Error: async API failed for job 0137c77f-0a00-47b8-adcf-959315846000

This feature was included in Apache CloudStack 4.15, which is an LTS release.





CloudStack puts together many technologies to make managing large datacentres and its resources easy and efficient, but with many complex systems, it is not always evident what goes on behind the scenes. CloudStack has a verbose logging system built in that you can reference if you want to know what’s happening under the hood. This feature aims to make looking through those logs better for humans.

In CloudStack logs, alerts, and events, data values are displayed in bytes. When dealing with gigabytes and terabytes, these values are long strings of numbers which become ‘unreadable’ (i.e. very difficult to understand or compare). This feature changes these values so they can be shown in easily human readable form.

Looking at the management server and Secondary storage virtual machine (SSVM) logs, in various places byte size values are shown, for example:

If you are trying match exact byte values, one method would be by copying and pasting from the log into a text editor and then comparing the values, but that is cumbersome. If you are reading through the logs and want to know what the byte value is in kilobytes, megabytes, gigabytes or even terabytes, you will have to do the calculation manually (also cumbersome and time-consuming).

With this feature, byte values will be shown in human readable form in the management server and SSVM logs, and the description field in the usage records when using the list usage records API call. Original values are not lost as they may still be required, so both versions of the value will be shown.

For instance, a value currently displayed as: space available: 4541927915 bytes

Will be displayed as: space available: 4541927915 bytes (4.23GB): 


This feature will be on by default, but if for any reason you would like to turn it off, it can be disabled by setting the global setting “display.human.readable.sizes” to false:

This feature will be available as of Apache CloudStack 4.15, which will be an LTS release.

There are a plethora of ways to connect to a remote system, from command-line based protocols like SSH, to graphical user interfaces such as RDP. One of the simplest ways to connect to the GUI of a remote system is via VNC (Virtual Network Computing), which transmits keyboard and mouse events from one computer to another, relaying screen updates back by returning a sequence of pixels which when put together produce an image or ‘frame’. Simply put, it allows one computer to delegate its GUI display to another machine for the sake of convenience. VNC is built upon the Remote Frame Buffer (RFB) protocol, which works at the framebuffer level (a point in a system’s memory which contains the details for the GUI display). This allows it to work across most operating systems and applications.


(Old VNC-based console)


The RFB works in a client-server model. The remote user is the RFB client or viewer, and the endpoint where changes to the framebuffer originate (i.e. the system and applications) is known as the RFB server.

The efficiency and simplicity of this approach, along with its native support in KVM and XenServer, led to VNC’s adoption in CloudStack, allowing users to remotely access their virtual machines via a browser.

The user’s keyboard and mouse events are sent via API calls to CloudStack, which transmits them to the virtual machine. It then fetches the frames from the virtual machine, renders them into images, and via API calls displays these images in the browser as tiles, which when combined, renders the entire display.

Although this approach works well, network latency, as well as multiple continuous API calls, can reduce performance and User Experience. It leads to delays in events being propagated to the virtual machine as well as the partial rendering of the display.

Enter noVNC! noVNC is both an HTML VNC client JavaScript library, and an application built on top of that library, and runs well in any modern browser (including mobile browsers). noVNC follows the standard VNC protocol, but unlike other VNC clients, it leverages the bi-directional communication of WebSockets to do the heavy lifting and transfers the task of rendering the display to the client. Now all keyboard / mouse events are transmitted over this WebSocket (created specifically for communication to the virtual machine) and the frames to be displayed are transmitted back to the browser, eliminating all API calls to CloudStack. It also natively supports various keyboard layouts when configured in the remote machine. This reduces the application overhead, allows for real-time rendering of the virtual machine’s console, and provides a superior and uninterrupted user experience.


(no-VNC console)


noVNC will be introduced in the CloudStack 4.15 LTS release and will become the default console, controlled via the ‘novnc.console.default’ global setting. More information on noVNC can be found at

This feature allows admins to expose the hypervisor host name to a User VM instance either through Config Drive or Virtual Router, based on the user data provider chosen for the network offering on which the VM instance is deployed.

To expose this information, the new Global configuration ““ and the new Account scope configuration value “” must be set to true. Once set, the hypervisor host name will be visible after VM creation, migration or reboot.

If ConfigDrive is the user data provider, we need to firstly mount the config drive iso in the VM instance. After setting the config flags to true and performing an operation on a VM (create, migrate, etc.) a new file (hypervisor-host-name.txt) will be created under <mount_path>/cloudstack/metadata/ containing the hypervisor host name. If either of the configuration flags were set to false on mounting the iso in the user VM, then this file will not be created. To reflect the correct value of the hypervisor hostname, one must unmount and re-mount the iso.

If Virtual Router is the user data provider, a new file (hypervisor-host-name) will be created in the VR at /var/www/html/metadata/<VM_IPAddress>/ on setting the config flags to true. This information can also be accessed from the VM instance via http using: curl http://<Gateway_IP>/latest/meta-data/hypevisor-host-name. After having changed the configuration values, an operation on a VM (create, migrate, etc.) is required to expose the hypervisor host name in the VM.

This feature will be available as of Apache CloudStack 4.15, which will be an LTS release.



This feature allows CloudStack administrators to unmanage guest virtual machines (VMs) from their CloudStack infrastructure. Once unmanaged, CloudStack can no longer monitor, control, or manage provisioning and orchestration related operations on it. This feature is currently supported only on VMware.

An interesting use case of this feature (when used in conjunction with the VM ingestion feature) is being able to move guest VMs from one vCenter to another, by unmanaging it from one zone and then importing it into a different zone. It is also possible to perform any out-of-band operations on the VM (once unmanaged from CloudStack) directly through VMware vSphere.

Whilst unmanaging a guest VM from the CloudStack infrastructure, the process also removes and cleans up resources used by the VM on the CloudStack database and network resources. From CloudStack’s database’s point of view, unmanaging a VM is like expunging one, where records are removed from several different tables where the VM is referenced.

There is, however, a case in which this feature does not remove an associated record from a table – a MAC address from a VM’s NIC can be preserved after unmanaging. This behavior is disabled by default, but it can be enabled with the zone setting ‘unmanage.vm.preserve.nics’. When the setting is enabled the VM’s NICs and their MAC addresses are preserved when unmanaging a VM. Otherwise, NICS are removed and MAC addresses can be reassigned by CloudStack to different VMs.

Unmanaging a VM is possible through a new API available to administrators – ‘unmanageVirtualMachine’ – which requires the UUID of a guest VM as a parameter. The complete list of actions performed when unmanaging a VM is the following:

  • Clean up VM’s NICs and deallocate network resources used, such as IP addresses and DHCP entries on virtual routers.
    • If ‘unmanage.vm.preserve.nics’ = ‘false’ then the NICs are deallocated and removed from CloudStack
    • If ‘unmanage.vm.preserve.nics’ = ‘true’ then the NICs remain allocated and are not removed from the database. The NIC’s MAC address remains in the database and therefore cannot be assigned to any new NIC.
  • Clean up VM volumes in the CloudStack database
  • Clean up VM snapshots in the CloudStack database (if any)
  • Revoke host access to any managed volumes attached to the VM
  • Clean up the VM from the following:
    • Remove the VM from security groups (if any)
    • Remove the VM from instance groups (if any)
    • Remove firewall rules for the VM (if any)
    • Remove port forwarding rules for the VM (if any)
    • Remove load balancing rules for the VM (if any)
    • Disable static NAT (if the VM is assigned to it)
    • Remove the VM from affinity groups (if any)
  • Set VM details to ‘removed’ in the CloudStack database
  • Decrement the account resources count for volumes and VMs
  • Generate usage events:
    • For volumes destroyed, with type: ‘VOLUME.DELETE’
    • For VM snapshots destroyed (if any), with type: ‘VMSNAPSHOT.DELETE’ and ‘VMSNAPSHOT.OFF_PRIMARY’
    • For VM NICs destroyed: with type: ‘NETWORK.OFFERING.REMOVE’

For the VM being unmanaged: stopped and destroyed usage events (similar to the usage events generated when expunging a VM), with types: ‘VM.STOP’ and ‘VM.DESTROY’, unless the VM has already been stopped, in which case only the ‘VM.DESTROY’ event is generated.

The ‘unmanageVirtualMachine’ API has the following pre-conditions:

  • The VM must not be destroyed
  • The VM state must be ‘Running’ or ‘Stopped’
  • The VM must be a VMware VM

Assuming these pre-conditions are met, the API execution will perform the following pre-checks, failing if they are not met:

  • There are no volume snapshots associated with any of the VM volumes
  • There is no ISO attached to the VM

An additional check is performed prior to unmanaging a VM from CloudStack: the hypervisor returns checks that the VM exists, searching the VM by its instance name. If it is not found, then the operation fails.

Since the current UI is being replaced by Primate on the next CloudStack release, it is not possible to unmanage a guest VM from the current UI. However, a new button is displayed on the VM view in Primate, along with the allowed actions, enabling administrators to unmanage guest VMs from the UI. This button is displayed when:

  • The VM is running or stopped, and
  • The VM is a VMware VM

The introduction of the zone setting to preserve the VM NICs when unmanaging it makes it necessary to adapt the VM import functionality to this setting. Therefore, a new parameter has been introduced on the ‘importUnmanagedInstances’ API in which a new Boolean parameter was added: ‘forced’. The ‘forced’ parameter is false by default, but if set to true then a VM is imported despite some of its NIC’s MAC addresses being present. This parameter is false by default and prevents the importing of a VM which has a NIC containing a MAC address that has been previously assigned by CloudStack. If it is set to true, NICs with MAC addresses which already exist in the CloudStack database will have the current MAC addresses reassigned to its NICs. Also, the usage events generated when importing unmanaged VMs have been refactored:

  • The usage event with type: ‘VM.IMPORT’ has been replaced by the usage event with type: ‘VM.CREATE’.
  • If the imported VM is powered ON, then the VM import feature generates the usage event with type: ‘VM.START’

This feature will be available as of Apache CloudStack 4.15, which will be an LTS release.