As a cloud infrastructure scales to hundreds or thousands of servers, high availability becomes a key requirement of the production environments supporting multiple applications and services. Since the management servers use a MySQL database to store the state of all its objects, the database could become a single point of failure. The CloudStack manual recommends MySQL replication with manual failover in the event of database loss.
We have worked with Severalnines to produce what we believe is a better way.
In this blog post, we’ll show you how to deploy redundant CloudStack management servers with MariaDB Galera Cluster on CentOS 6.5 64bit. We will have two load balancer nodes fronting the management servers and the database servers. Since CloudStack relies on MySQL’s GET_LOCK and RELEASE LOCK, which are not supported by Galera, we will redirect all database requests to only one MariaDB node and automatically failover to another node in case the former goes down. So, we’re effectively getting the HA benefits of Galera clustering (auto-failover, full consistency between DB nodes, no slave lag), while avoiding the Galera limitations as we’re not concurrently accessing all the nodes. We will deploy a two-node Galera Cluster (plus an arbitrator on a separate ClusterControl node).
Our setup will look like this:
Note that this blog post does not cover the installation of hypervisor and storage hosts. Our setup consists of 4 servers:
$ yum -y install ntp
$ chkconfig ntpd on
$ service ntpd start
3. Ensure each host is using a valid FQDN, for example on mgm1:
$ hostname --fqdn
Deploying MariaDB Galera Cluster
** The deployment of the database cluster will be done from lb2, i.e., the ClusterControl node.
1. To set up MariaDB Galera Cluster, go to the Severalnines Galera Configurator to generate a deployment package. In the wizard, we used the following values when configuring our database cluster (take note that we specified one of the DB nodes twice under Database Servers’ textbox):
Vendor : MariaDB
MySQL Version : 10.x
Infrastructure : none/on-premises
Operating System : RHEL6 - Redhat 6.4/Fedora/Centos 6.4/OLN 6.4/Amazon AMI
Number of Galera Servers : 3
Max connections : 350
OS user : root
ClusterControl Server : 192.168.1.12
Database Servers : 192.168.1.21 192.168.1.22 192.168.1.22
At the end of the wizard, a deployment package will be generated and emailed to you.
2. Download and extract the deployment package:
$ wget http://www.severalnines.com/galera-configurator3/tmp/wb06494200669221809/s9s-galera-mariadb-3.5.0-rpm.tar.gzz
$ tar -xzf s9s-galera-mariadb-3.5.0-rpm.tar.gz
3. Before we proceed with the deployment, we need to perform some customization to fit the CloudStack database environment. Go to the deployment script’s MySQL configuration file at ~/s9s-galera-mariadb-3.5.0-rpm/mysql/config/my.cnf and ensure the following options exist under the [MYSQLD] section:
7. It is recommended to run Galera on at least three nodes. So, install garbd, a lightweight arbitrator daemon for Galera on the ClusterControl node from the ClusterControl UI. Go to Manage > Load Balancer > Install Garbd > choose the ClusterControl node IP address from the dropdown > Install Garbd.
You will now see your MariaDB Galera Cluster with garbd installed and binlog enabled (master) as per below:
Load Balancer and Virtual IP
1. Before we start to deploy the load balancers, make sure lb1 is accessible using passwordless SSH from ClusterControl/lb2. On lb2, copy the SSH keys to 192.168.1.11:
$ ssh-copy-id -i ~/.ssh/id_rsa email@example.com
2. Login to ClusterControl, drill down to the database cluster and click Add Load Balancer button. Deploy HAProxy on lb1 and lb2 similar to below:
** Take note that for RHEL, ensure you check Build from source? to install HAProxy from source. This will install the latest version of HAProxy.
3. Install Keepalived on lb1(master) and lb2(backup) with 192.168.1.10 as virtual IP:
4. The load balancer nodes have now been installed, and are integrated with ClusterControl. You can verify this by checking out the ClusterControl summary bar:
5. By default, our script will configure the MySQL reverse proxy service to listen on port 33306 in active-active mode. We need to change this to active-passive multi master mode by declaring the second Galera node as backup, On lb1 and lb2, open /etc/haproxy/haproxy.cfg and append the word ‘backup’ into the last line:
server 192.168.1.21 192.168.1.21:3306 check
server 192.168.1.22 192.168.1.22:3306 check backup
6. We also need to add the load balancing definition for CloudStack. According to the documentation, we need to load balance port 8080 and 8025. To allow session stickiness, we will use source load balancing algorithm, where the same source address will be forwarded to the same management server unless it fails. On lb1 and lb2, open/etc/haproxy/haproxy.cfg and add the following lines:
option httpchk OPTIONS /client
server mgm1.cloudstack.local 192.168.1.21:8080 maxconn 32 check inter 5000
server mgm2.cloudstack.local 192.168.1.22:8080 maxconn 32 check inter 5000
server mgm1.cloudstack.local 192.168.1.21:8250 maxconn 32 check
server mgm2.cloudstack.local 192.168.1.22:8250 maxconn 32 check
6. Restart HAProxy to apply the changes:
$ service haproxy restart
Or, you can just kill the haproxy process and let ClusterControl recover it.
7. Configure iptables to allow connections to port configured in HAProxy and Keepalived. Add the following lines:
** Allow some time for the CloudStack application to bootstrap on each startup. You can monitor the process at/var/log/cloudstack/management/catalina.out.
6. Open the CloudStack management UI at virtual IP, http://192.168.1.10:8080/client/ with default user ‘admin’ and password ‘password’. Configure your CloudStack environment by following the deployment wizard and let CloudStack build the infrastructure:
If completed successfully, you should then be redirected to the CloudStack Dashboard:
The installation of the first management server is now complete. We’ll now proceed with the second management server.
Installing CloudStack Management Server #2
** The following steps should be performed on mgm2
1. Add the CloudStack repository, create /etc/yum.repos.d/cloudstack.repo and insert the following information.
** Allow some time for the CloudStack application to bootstrap on each startup. You can monitor the process at/var/log/cloudstack/management/catalina.out. At this point, this management server will automatically discover the other management server and form a cluster. Both management servers are load balanced and accessible via virtual IP, 192.168.1.10.
Lastly, change the management host IP address on every agent host at/etc/cloudstack/agent/agent.properties to the virtual IP address similar to below:
Restart the cloudstack agent service to apply the change:
$ service cloudstack-agent restart
Verify the Setup
1. Check the HAProxy statistics by logging into the HAProxy admin page at lb1 host port 9600. The default username/password is admin/admin. You should see the status of nodes from the HAProxy point-of-view. Our Galera cluster is in master-standby mode, while the CloudStack management servers are load balanced:
https://www.shapeblue.com/wp-content/uploads/2015/03/database-300x300.jpg300300Paul Angushttps://www.shapeblue.com/wp-content/uploads/2017/06/logo-340x156.pngPaul Angus2015-03-04 15:27:062017-11-06 17:38:15How to Deploy High Availability CloudStack with MariaDB Galera Cluster