Tag Archive for: CloudStack Feature First Look

Share ISO from UI | CloudStack Feature First Look

CloudStack supports sharing templates and ISOs between accounts and projects through the API ‘updateTemplatePermissions’ and sharing templates through the UI. However, prior to version 4.15, it was not possible to share ISOs from the UI. This feature introduces support for sharing ISOs through different accounts and / or projects via the UI.

With this feature, a user or administrator must be able to update the permissions for an ISO via API and UI, being able to:

  • Share the ISO with another account
  • Share the ISO with another project
  • Revoke the access to the ISO for an account
  • Revoke the access to the ISO for a project
  • Reset the ISO permissions to the default

A new button is added in the ISO details view: ‘Update ISO permissions’. This button is located at the top right corner of the ISO detail view:

Once clicked, the user is prompted with a dialogue like the one below:

The user or administrator must complete three fields:

  • Operation: Must be one of the following values: ‘Add’, ‘Remove’, ‘Reset’
  • Share With: It is not displayed if the Operation field is set to ‘Reset’. If the Operation field is set to ‘Add’ or ‘Remove’ then the possible values are: ‘Account’, ‘Project’
  • [Account/Project]: It is not displayed if the Operation field is set to ‘Reset’. The field label depends on the selected ‘Share With’ field type. In this field, the user or administrator must provide an account or project name to be added to or removed from the permitted list of the ISO.
    • When ‘allow.user.view.all.domain.accounts’ = true: the dialog box displays a list of accounts within the same domain, otherwise the user must specify a comma-separated list of account names instead of selecting the account names from a list.

The ISOs shared with a user are displayed under the ‘Shared’ section of the ISOs list view:

This feature Will be available from CloudStack 4.15.

/by

CentOS 8 Support | | CloudStack Feature First Look

,

As of 2021, CentOS 7 will be receiving maintenance updates only, and is end of life in 2024. Considering this, it is important that CloudStack supports CentOS 8 as a KVM hypervisor host and as a host for the management and usage servers. This support has been developed and will be included as of CloudStack 4.15.

CentOS 8 uses a more recent QEMU version, Python 3 by default and deprecates several networking tools (such as bridge-utils), therefore a number of changes have been made:

  • Python scripts related to setting up of management and usage servers and KVM agent have been migrated from Python 2 to Python 3.
  • Python 2 dependencies from cloudstack packages (cloudstack-common, cloudstack-management, cloudstack-usage and cloudstack-agent) have been removed.
  • Support for MySQL 8 (as CentOS 8 installs this by default).

With this feature, changes have also been made to the snapshot related codebase on KVM to support the newer version of the qemu-img utility. This should prevent issues with snapshot management on an OS with newer QEMU version.
KVM hosts and management / usage servers on CentOS 7 will continue to work as before and only new python 3 dependencies (python3, python3-pip and python3-setuptools) will be installed during upgrade.

/by

Support for XCP-ng 8.x | CloudStack Feature First Look

,

XCP-ng (an open-source hypervisor based on XenServer) has been supported in CloudStack for some time, and support for XCP-ng 8.x will be available from CloudStack 4.15.

From an operational perspective, there is no visible change in the UI or API – XCP-ng 8.x hosts can be added to CloudStack during zone deployment or by using the addHost API / UI. With guest OS mappings added for XCP-ng 8.x, once added all VM operations can be performed on such hosts just like any older XCP-ng hosts unless they are functionally not available on XCP-ng 8.x. It should be noted that with XCP-ng 8.1, support for paravirtualized (PV) VMs has been removed, therefore PV guests and older guest OS will not work on 8.x hypervisor hosts. More information about supported and removed functionalities in XCP-ng 8.x can be found in the XCP-ng documentation.

/by

Boot into Hardware Setup | CloudStack Feature First Look

,

Users and administrators sometimes need to change the boot order of a virtual machine (VM) so that they can (for example) force a VM to boot from an ISO. However, sometimes the proxied console session does not appear until after the opportunity to enter the BIOS menu has passed. This feature allows an admin to request that a VM boots into the BIOS or other hardware setup screen rather than into the operating system (OS). The nature of the setup depends on the underlying hardware version and the OS of the virtual machine, and can be BIOS or UEFI. This feature applies to VMware only (as XenServer and KVM do not provide a BIOS).

With this feature, when deploying a virtual machine the boot type can be chosen. It can be BIOS or UEFI, and if UEFI it can be Secure or Legacy. The choice made must be supported by the underlying hardware and the virtual machine’s OS and cannot be changed once a VM instance has been created.

In addition to the boot type options “bootintosetup” can be specified. The latter option is also available on reboot or starting a stopped machine.

Some examples from cloudmonkey:

New parameters for deploy vm:

     (localcloud) SBCM5> > deploy virtualmachine boot

     bootintosetup= bootmode=      boottype=

Boot into setup takes a Boolean value:

     (localcloud) SBCM5> > deploy virtualmachine bootintosetup=

     true   false

On trying to boot a VM into hardware setup will fail. As can be seen the option is only implemented for VMware:

     (localcloud) SBCM5> > reboot virtualmachine id=68ba4b83-12ec-4da1-aefa-8462d9135f00 bootintosetup=true

     {

     "accountid": "1508af67-9c07-11ea-8893-1e000601081e",

     "cmd": "org.apache.cloudstack.api.command.admin.vm.RebootVMCmdByAdmin",

     "completed": "2020-05-22T16:33:29+0000",

     "created": "2020-05-22T16:33:29+0000",

     "jobid": "0137c77f-0a00-47b8-adcf-959315846000",

     "jobinstanceid": "68ba4b83-12ec-4da1-aefa-8462d9135f00",

     "jobinstancetype": "VirtualMachine",

     "jobprocstatus": 0,

     "jobresult": {

     "errorcode": 431,

     "errortext": " Booting into a hardware setup menu is not implemented on KVM"

     },

     "jobresultcode": 530,

     "jobresulttype": "object",

     "jobstatus": 2,

     "userid": "150c8809-9c07-11ea-8893-1e000601081e"

     }

     🙈 Error: async API failed for job 0137c77f-0a00-47b8-adcf-959315846000

This feature was included in Apache CloudStack 4.15, which is an LTS release.

 

 

 

 

/by

Human readable byte sizes | CloudStack Feature First Look

,

CloudStack puts together many technologies to make managing large datacentres and its resources easy and efficient, but with many complex systems, it is not always evident what goes on behind the scenes. CloudStack has a verbose logging system built in that you can reference if you want to know what’s happening under the hood. This feature aims to make looking through those logs better for humans.

In CloudStack logs, alerts, and events, data values are displayed in bytes. When dealing with gigabytes and terabytes, these values are long strings of numbers which become ‘unreadable’ (i.e. very difficult to understand or compare). This feature changes these values so they can be shown in easily human readable form.

Looking at the management server and Secondary storage virtual machine (SSVM) logs, in various places byte size values are shown, for example:

If you are trying match exact byte values, one method would be by copying and pasting from the log into a text editor and then comparing the values, but that is cumbersome. If you are reading through the logs and want to know what the byte value is in kilobytes, megabytes, gigabytes or even terabytes, you will have to do the calculation manually (also cumbersome and time-consuming).

With this feature, byte values will be shown in human readable form in the management server and SSVM logs, and the description field in the usage records when using the list usage records API call. Original values are not lost as they may still be required, so both versions of the value will be shown.

For instance, a value currently displayed as: space available: 4541927915 bytes

Will be displayed as: space available: 4541927915 bytes (4.23GB): 

 

This feature will be on by default, but if for any reason you would like to turn it off, it can be disabled by setting the global setting “display.human.readable.sizes” to false:

This feature will be available as of Apache CloudStack 4.15, which will be an LTS release.

/by

noVNC Console Integration | CloudStack Feature First Look

,

There are a plethora of ways to connect to a remote system, from command-line based protocols like SSH, to graphical user interfaces such as RDP. One of the simplest ways to connect to the GUI of a remote system is via VNC (Virtual Network Computing), which transmits keyboard and mouse events from one computer to another, relaying screen updates back by returning a sequence of pixels which when put together produce an image or ‘frame’. Simply put, it allows one computer to delegate its GUI display to another machine for the sake of convenience. VNC is built upon the Remote Frame Buffer (RFB) protocol, which works at the framebuffer level (a point in a system’s memory which contains the details for the GUI display). This allows it to work across most operating systems and applications.

 

(Old VNC-based console)

 

The RFB works in a client-server model. The remote user is the RFB client or viewer, and the endpoint where changes to the framebuffer originate (i.e. the system and applications) is known as the RFB server.

The efficiency and simplicity of this approach, along with its native support in KVM and XenServer, led to VNC’s adoption in CloudStack, allowing users to remotely access their virtual machines via a browser.

The user’s keyboard and mouse events are sent via API calls to CloudStack, which transmits them to the virtual machine. It then fetches the frames from the virtual machine, renders them into images, and via API calls displays these images in the browser as tiles, which when combined, renders the entire display.

Although this approach works well, network latency, as well as multiple continuous API calls, can reduce performance and User Experience. It leads to delays in events being propagated to the virtual machine as well as the partial rendering of the display.

Enter noVNC! noVNC is both an HTML VNC client JavaScript library, and an application built on top of that library, and runs well in any modern browser (including mobile browsers). noVNC follows the standard VNC protocol, but unlike other VNC clients, it leverages the bi-directional communication of WebSockets to do the heavy lifting and transfers the task of rendering the display to the client. Now all keyboard / mouse events are transmitted over this WebSocket (created specifically for communication to the virtual machine) and the frames to be displayed are transmitted back to the browser, eliminating all API calls to CloudStack. It also natively supports various keyboard layouts when configured in the remote machine. This reduces the application overhead, allows for real-time rendering of the virtual machine’s console, and provides a superior and uninterrupted user experience.

 

(no-VNC console)

 

noVNC will be introduced in the CloudStack 4.15 LTS release and will become the default console, controlled via the ‘novnc.console.default’ global setting. More information on noVNC can be found at https://github.com/novnc/noVNC

/by

Send hypervisor host name via metadata | CloudStack Feature First Look

,

This feature allows admins to expose the hypervisor host name to a User VM instance either through Config Drive or Virtual Router, based on the user data provider chosen for the network offering on which the VM instance is deployed.

To expose this information, the new Global configuration “global.allow.expose.host.hostname“ and the new Account scope configuration value “account.allow.expose.host.hostname” must be set to true. Once set, the hypervisor host name will be visible after VM creation, migration or reboot.

If ConfigDrive is the user data provider, we need to firstly mount the config drive iso in the VM instance. After setting the config flags to true and performing an operation on a VM (create, migrate, etc.) a new file (hypervisor-host-name.txt) will be created under <mount_path>/cloudstack/metadata/ containing the hypervisor host name. If either of the configuration flags were set to false on mounting the iso in the user VM, then this file will not be created. To reflect the correct value of the hypervisor hostname, one must unmount and re-mount the iso.

If Virtual Router is the user data provider, a new file (hypervisor-host-name) will be created in the VR at /var/www/html/metadata/<VM_IPAddress>/ on setting the config flags to true. This information can also be accessed from the VM instance via http using: curl http://<Gateway_IP>/latest/meta-data/hypevisor-host-name. After having changed the configuration values, an operation on a VM (create, migrate, etc.) is required to expose the hypervisor host name in the VM.

This feature will be available as of Apache CloudStack 4.15, which will be an LTS release.

/by

Enable unmanaging of guest instances | CloudStack Feature First Look

,

This feature allows CloudStack administrators to unmanage guest virtual machines (VMs) from their CloudStack infrastructure. Once unmanaged, CloudStack can no longer monitor, control, or manage provisioning and orchestration related operations on it. This feature is currently supported only on VMware.

An interesting use case of this feature (when used in conjunction with the VM ingestion feature) is being able to move guest VMs from one vCenter to another, by unmanaging it from one zone and then importing it into a different zone. It is also possible to perform any out-of-band operations on the VM (once unmanaged from CloudStack) directly through VMware vSphere.

Whilst unmanaging a guest VM from the CloudStack infrastructure, the process also removes and cleans up resources used by the VM on the CloudStack database and network resources. From CloudStack’s database’s point of view, unmanaging a VM is like expunging one, where records are removed from several different tables where the VM is referenced.

There is, however, a case in which this feature does not remove an associated record from a table – a MAC address from a VM’s NIC can be preserved after unmanaging. This behavior is disabled by default, but it can be enabled with the zone setting ‘unmanage.vm.preserve.nics’. When the setting is enabled the VM’s NICs and their MAC addresses are preserved when unmanaging a VM. Otherwise, NICS are removed and MAC addresses can be reassigned by CloudStack to different VMs.

Unmanaging a VM is possible through a new API available to administrators – ‘unmanageVirtualMachine’ – which requires the UUID of a guest VM as a parameter. The complete list of actions performed when unmanaging a VM is the following:

  • Clean up VM’s NICs and deallocate network resources used, such as IP addresses and DHCP entries on virtual routers.
    • If ‘unmanage.vm.preserve.nics’ = ‘false’ then the NICs are deallocated and removed from CloudStack
    • If ‘unmanage.vm.preserve.nics’ = ‘true’ then the NICs remain allocated and are not removed from the database. The NIC’s MAC address remains in the database and therefore cannot be assigned to any new NIC.
  • Clean up VM volumes in the CloudStack database
  • Clean up VM snapshots in the CloudStack database (if any)
  • Revoke host access to any managed volumes attached to the VM
  • Clean up the VM from the following:
    • Remove the VM from security groups (if any)
    • Remove the VM from instance groups (if any)
    • Remove firewall rules for the VM (if any)
    • Remove port forwarding rules for the VM (if any)
    • Remove load balancing rules for the VM (if any)
    • Disable static NAT (if the VM is assigned to it)
    • Remove the VM from affinity groups (if any)
  • Set VM details to ‘removed’ in the CloudStack database
  • Decrement the account resources count for volumes and VMs
  • Generate usage events:
    • For volumes destroyed, with type: ‘VOLUME.DELETE’
    • For VM snapshots destroyed (if any), with type: ‘VMSNAPSHOT.DELETE’ and ‘VMSNAPSHOT.OFF_PRIMARY’
    • For VM NICs destroyed: with type: ‘NETWORK.OFFERING.REMOVE’

For the VM being unmanaged: stopped and destroyed usage events (similar to the usage events generated when expunging a VM), with types: ‘VM.STOP’ and ‘VM.DESTROY’, unless the VM has already been stopped, in which case only the ‘VM.DESTROY’ event is generated.

The ‘unmanageVirtualMachine’ API has the following pre-conditions:

  • The VM must not be destroyed
  • The VM state must be ‘Running’ or ‘Stopped’
  • The VM must be a VMware VM

Assuming these pre-conditions are met, the API execution will perform the following pre-checks, failing if they are not met:

  • There are no volume snapshots associated with any of the VM volumes
  • There is no ISO attached to the VM

An additional check is performed prior to unmanaging a VM from CloudStack: the hypervisor returns checks that the VM exists, searching the VM by its instance name. If it is not found, then the operation fails.

Since the current UI is being replaced by Primate on the next CloudStack release, it is not possible to unmanage a guest VM from the current UI. However, a new button is displayed on the VM view in Primate, along with the allowed actions, enabling administrators to unmanage guest VMs from the UI. This button is displayed when:

  • The VM is running or stopped, and
  • The VM is a VMware VM

The introduction of the zone setting to preserve the VM NICs when unmanaging it makes it necessary to adapt the VM import functionality to this setting. Therefore, a new parameter has been introduced on the ‘importUnmanagedInstances’ API in which a new Boolean parameter was added: ‘forced’. The ‘forced’ parameter is false by default, but if set to true then a VM is imported despite some of its NIC’s MAC addresses being present. This parameter is false by default and prevents the importing of a VM which has a NIC containing a MAC address that has been previously assigned by CloudStack. If it is set to true, NICs with MAC addresses which already exist in the CloudStack database will have the current MAC addresses reassigned to its NICs. Also, the usage events generated when importing unmanaged VMs have been refactored:

  • The usage event with type: ‘VM.IMPORT’ has been replaced by the usage event with type: ‘VM.CREATE’.
  • If the imported VM is powered ON, then the VM import feature generates the usage event with type: ‘VM.START’

This feature will be available as of Apache CloudStack 4.15, which will be an LTS release.

/by

KVM Rolling Maintenance | CloudStack Feature First Look

, , ,

In a typical scenario prior to this feature, the administrator had to automate the process of setting hosts in maintenance before upgrading, usually using external automation tools. This feature allows administrators to perform the entire process within CloudStack, providing a flexible framework that allows defining custom scripts to execute on each host.

CloudStack executes these scripts within the context of 4 stages:

  1. Pre-flight script runs on hosts before commencing the rolling maintenance. If pre-flight check scripts return an error from any host, then rolling maintenance will be cancelled with no actions taken, and an error returned. If there are no pre-flight scripts defined on a host, then no checks will be done.
  2. Pre-maintenance script runs before a specific host is put into maintenance. If no pre-maintenance script is defined, then no pre-maintenance actions will be taken, and the management server will move straight to putting the host in maintenance followed by requesting that the agent runs the maintenance script.
  3. Maintenance script runs after a host has been put into maintenance. If no maintenance script is defined, or if the pre-flight or pre-maintenance scripts determine that no maintenance is required (exit status 70), then the host will not be put into maintenance, and the completion of the pre-maintenance scripts will signal the end of all maintenance tasks and the KVM agent will hand the host back to the management server. Once the maintenance scripts have signalled that it has completed, the host will exit maintenance mode and any ‘information’ which was collected (such as processing times) will be returned to the management server.
  4. Post-maintenance script is expected to perform validation after the host exits maintenance. These scripts will help to detect any problem during the maintenance process, including reboots or restarts within scripts or (for example) can mark a host as successfully processed (i.e. create a “/processed.sucessfully” file, that may be in use during the PreFlight check), in case that the rolling maintenance is run again against a whole cluster (then some hosts will be skipped due to PreFlight script determining that it should not be processed again).

If you execute the startRollingMaintenance API against a Cluster, the PreFlight.sh script(s) will be executed against all hosts sequentially, and then will be processed one by one (PreMaintenance.sh, Maintenance.sh and PostMaintenance.sh scripts are executed for that single host). Only when a single host has been fully “processed” then the execution continues and the next host is processed. Scripts can have no file extension, or can have ‘.sh’ or ‘py’ extensions, and have to be executable.

To use the feature, the agent.properties file needs to be updated (restart required) to include settings:

  • maintenance.hooks.dir=/path/to/hooks/dir
  • maintenance.service.executor.disabled=false/true

Once the hosts are properly configured, simply start the rolling process by invoking the ‘startRollingMaintenance’ API method (or via UI) against the Zone(s), Pod(s), Cluster(s) or Host(s).

A new button ‘Start Rolling Maintenance’ is added, which opens a new dialogue invoking the startRollingMaintenance API method (all fields/parameters are optional):

If invoking the startRollingMaintenance API via GUI, keep in mind the results are not returned to the GUI (there is no querying of the job id) as it can take hours before the API / job completes. Doing the same via CLI (i.e. CloudMonkey) will return descriptive results, like in the example given below:

(localcloud) SBCM5> > start rollingmaintenance hostids=86c0b59f-89de-40db-9b30-251f851e869f
{
  "rollingmaintenance": {
    "details": "OK",
    "hostsskipped": [],
    "hostsupdated": [
      {
        "enddate": "2020-03-11'T'17:21:38+00:00",
        "hostid": "86c0b59f-89de-40db-9b30-251f851e869f",
        "hostname": "ref-trl-711-k-M7-apanic-kvm1",
        "output": "null ",
        "startdate": "2020-03-11'T'17:21:07+00:00"
      }
    ],
    "success": true
  }
}

The next example shows the output of startRollingMaintenance API when executed against 2 small ( i.e. demo) zones, with zone1 having 2 clusters with 3 hosts in each, while zone 2 having a single cluster with 3 hosts:

(localcloud) SBCM5> > start rollingmaintenance zoneids=6f3c9827-6e99-4c63-b7d5-e8f427f6dcff,ce831d12-c2df-4b11-bec9-684dcc292c18
{
  "rollingmaintenance": {
    "details": "OK",
    "hostsskipped": [],
    "hostsupdated": [
      {
        "enddate": "2020-03-12'T'12:41:24+00:00",
        "hostid": "86c0b59f-89de-40db-9b30-251f851e869f",
        "hostname": "ref-trl-711-k-M7-apanic-kvm1",
        "output": "",
        "startdate": "2020-03-12'T'12:40:44+00:00"
      },
      {
        "enddate": "2020-03-12'T'12:43:25+00:00",
        "hostid": "ef10dacd-ac4e-4ec0-bc8d-7fb5bb461c9d",
        "hostname": "ref-trl-711-k-M7-apanic-kvm2",
        "output": "",
        "startdate": "2020-03-12'T'12:41:45+00:00"
      },
      {
        "enddate": "2020-03-12'T'12:45:26+00:00",
        "hostid": "fcc8b96e-1c29-492e-a074-96babec70ecc",
        "hostname": "ref-trl-711-k-M7-apanic-kvm3",
        "output": "",
        "startdate": "2020-03-12'T'12:43:46+00:00"
      },
      {
        "enddate": "2020-03-12'T'12:47:27+00:00",
        "hostid": "4a732078-2f5d-4bf1-8425-2135004a6b1a",
        "hostname": "ref-trl-711-k-M7-apanic-kvm6",
        "output": "",
        "startdate": "2020-03-12'T'12:46:17+00:00"
      },
      {
        "enddate": "2020-03-12'T'12:49:28+00:00",
        "hostid": "8f27f11a-9c60-4c30-8622-0e1bce718adc",
        "hostname": "ref-trl-711-k-M7-apanic-kvm5",
        "output": "",
        "startdate": "2020-03-12'T'12:47:48+00:00"
      },
      {
        "enddate": "2020-03-12'T'12:51:29+00:00",
        "hostid": "adbbfc34-9369-4a15-93dc-7ed85756c24e",
        "hostname": "ref-trl-711-k-M7-apanic-kvm4",
        "output": "",
        "startdate": "2020-03-12'T'12:49:48+00:00"
      },
      {
        "enddate": "2020-03-12'T'12:53:00+00:00",
        "hostid": "59159ade-f5c3-4606-9174-e501301f59d4",
        "hostname": "ref-trl-714-k-M7-apanic-kvm3",
        "output": "",
        "startdate": "2020-03-12'T'12:52:19+00:00"
      },
      {
        "enddate": "2020-03-12'T'12:54:00+00:00",
        "hostid": "b0f54409-4874-4573-9c24-8efac5b07f6f",
        "hostname": "ref-trl-714-k-M7-apanic-kvm1",
        "output": "",
        "startdate": "2020-03-12'T'12:53:20+00:00"
      },
      {
        "enddate": "2020-03-12'T'12:55:01+00:00",
        "hostid": "02228e26-a0d6-4607-824d-501ae5ac8dab",
        "hostname": "ref-trl-714-k-M7-apanic-kvm2",
        "output": "",
        "startdate": "2020-03-12'T'12:54:21+00:00"
      }
    ],
    "success": true
  }
}
/by

Enable PVLAN support on L2 networks | CloudStack Feature First Look

Private VLANs have always been partially supported in CloudStack (for shared networks only), in versions prior to 4.14. Administrators could set up Isolated or Promiscuous PVLANs by creating their shared networks in which:

  • Primary VLAN ID = secondary VLAN ID, for Promiscuous PVLANs
  • Primary VLAN ID != secondary VLAN ID, for Isolated PVLANs

CloudStack 4.14 introduces some changes in the PVLAN support, by:

  • Extending the existing support for shared networks and L2 networks (initially supported for the VMware hypervisor when using dvSwitches)
  • Extending the PVLAN types to Isolated, Promiscuous and Community
  • Allowing the administrators to explicitly select the PVLAN type on network creation, as on the image below:

The following table summarizes the communication between these different PVLAN types:

Promiscuous Isolated Community 1 Community 2
Promiscuous ALLOW ALLOW ALLOW ALLOW
Isolated ALLOW DENY DENY DENY
Community 1 ALLOW DENY ALLOW DENY
Community 2 ALLOW DENY DENY ALLOW

Within an L2 network or shared network, it is possible to create:

  • 1 Promiscuous PVLAN
  • 1 Isolated PVLAN
  • Multiple Community PVLANs

Administrators must provide the PVLAN type and secondary VLAN ID as part of the ‘createNetwork’ API or through the UI.  If an admin requests a PVLAN which is not valid then a suitable error message will be returned, for example when:

  • A promiscuous PVLAN ID is not the same as the Primary VLAN ID
  • A community or isolated PVLAN ID which clashes with a PVLAN ID which is already in use on the same dvSwitch (i.e. the same physical network)
/by