CloudStack Test Automation with Trillian and Jenkins
In the previous post, we introduced and described Trillian that can build various environments in which we could deploy a CloudStack zone and run Marvin based integration tests. In this post, we’ll describe how we are using Jenkins and Trillian to test CloudStack builds in various environments.
Our build pipeline can be seen in the attached flow-diagram on the right, it consists of the following:
- The Cloudstack git repository.
- A Jenkins job for building CloudStack deb/rpm packages for Ubuntu 14.04, CentOS6 and CentOS7. Another Jenkins job for building CloudStack systemvm templates.
- A staging packages repository server for hosting the deb/rpm packages and the systemvm templates.
- A Jenkins job which uses Trillian to deploy a configurable integration environment on a parent CloudStack environment as a project. This currently supports deploying environments that are based on one of the three most popular hypervisors — on KVM (CentOS6, CentOS7, and Ubuntu 14.04 based hosts), XenServer (6.2 SP1 and 6.5 SP1) and VMware (5.5 U3 and 6.0 U1).
- A semi-automated Jenkins job to trigger testing of the build.
- Hipchat notification integration to receive updates from various Jenkins jobs.
Using Trillian with Jenkins
Trillian nests CloudStack environments in a parent CloudStack deployment. Within the parent CloudStack environment, it creates a project and it deploys the following VMs into it:
- A management server VM to running the CloudStack management server, the usage server and the MySQL server,
- One (1) or more VMs to running nested-hypervisors
- (Optional) A Marvin VM to execute integration tests.
Shared NFS storage is used to host primary and secondary storage pools at different paths for the CloudStack environment being tested.
When a build is kicked, the Jenkins job pulls the latest Trillian from the Trillian git repository, it then applies a cloudstack.ini file in Ansible directory that contains the url to the parent CloudStack management server, api key and secret key of the parent CloudStack environment. Next, specific to the parent CloudStack environment the Ansible/group_vars/all file is patched, this file contains configuration specific to the parent CloudStack environment such as the default credentials to use for various hypervisor hosts, mysql server, VM root password, networking related information, shared NFS storage related information, base CloudStack repository url, urls for systemvmtemplates, urls various other VM templates, Hipchat notification token, and various global settings for the management server being deployed.
Our Trillian Jenkins job is parameterized job that accepts some of the above options before the build job is started. These options are then used to first generate the Trillian host config and then this config is used to deploy the environment by running an ansible playbook with suitable options, for example:
$ ansible-playbook generate-cloudconfig.yml –extra-vars “env_name=$CONFIG_NAME env_version=$CLOUDSTACK_VERSION mgmt=$MS_COUNT db=0 $HYPERVISOR_OPTS hv=$HYPERVISOR_COUNT mgmt_os=$MS_OS env_accounts=all pri=$PRIMARY_STORAGE_COUNT sec=$SEC_STORAGE_COUNT baseurl_cloudstack=$CLOUDSTACK_REPO wait_till_setup=yes build_marvin=yes” -i localhost
$ ansible-playbook deployvms.yml -i ./hosts_$CONFIG_NAME –extra-vars “env_name=$CONFIG_NAME env_version=$CLOUDSTACK_VERSION”
We also have a Trillian-destroy Jenkins job that takes in the Trillian environment ID and runs an Ansible playbook to destroy the deployed VMs and related resources.
Testing CloudStack with Trillian
To make testing easier for CI systems such as Trillian and Bubble we have introduced two new CloudStack packages: cloudstack-marvin and cloudstack-integration-tests. By building Marvin and the integration-tests as a deb/rpm package we make it easier for CI systems to install Marvin and run the integration tests specific to that git tag/sha or github pull-request.
The Trillian Jenkins job allows us to deploy a test environment with various permutation and combination of compute, storage and network technologies from a specific CloudStack repository that was built from a specific git tag or SHA. The build pipeline described in previous section allows us to (1) start the building and testing process by providing it a git tag or SHA, which (2) triggers building of the CloudStack repository, which (3) on completion triggers deployment of a bunch of preset Trillian environments, which (4) on deployment starts a Jenkins job that runs the integration tests and gathers report.
About the author
Rohit Yadav is a Software Architect at ShapeBlue and an Apache CloudStack committer and PMC member. He has been contributing to CloudStack since 2012, some of the recent features he has developed for CloudStack includes metrics view, out-of-band management, dynamic roles. He is also the author and maintainer of CloudMonkey.