Posts

Cloudstack’s multi-tenant virtualised networking model is one of its strongest features. Abstracting complex networking concepts and allowing simple UI/API configuration of networks is something loved by users of Cloudstack clouds. But, as an operator/administrator of a Cloudstack cloud you’ll almost certainly have had to troubleshoot network  problems  – and that means troubleshooting CloudStack’s Virtual Routers (VRs) .

As Roy says in the picture , turning it off and on again can often resolve issues with a VR (or restarting a VR in Cloudstack language). But if that doesn’t work, administrators need to troubleshoot the VR. In this article, I will discuss some of the common approaches to such troubleshooting and look at some new CloudStack features that have been added to make this process much easier for administrators.

The problem

Now let’s try to imagine what could be wrong with our VR? The first and most obvious things are mistakes when defining IP range, netmask and gateway for our networks, or general human errors when building out infrastructure elements. CloudStack does provide some validation of the input you give, but it’s not guaranteed to pick up every mistake.

Then, most commonly we could have connectivity issues. The list is endless here, from inside the router itself to the Internet, other VPC/networks, Private Gateways and so on. To resolve such issues, we’ll need some serious networking skills and time to investigate, and most of the time it happens to be one tiny bit of configuration that we’ve missed or misconfigured.

Solution

To get to the bottom of these issues, it is often required to dig through all configurations, run connectivity troubleshooting in and out of a machine and so on, which can be painful and time consuming. CloudStack now offers two new features which will make our life easier and troubleshooting of the VR far more painless to the root admin. Let me introduce you to the “Run Diagnostics” and “Get Diagnostics” features from the VR. Using these two features an administrator will be able to get valuable information from the virtual router without even logging in to it. Furthermore, he’ll be able to operate from the inside of the router and execute scripts, commands etc. to determine what’s wrong or even fix it.

Let’s dive into details of these features.

Run diagnostics

The new “Run diagnostics” feature allows root administrators to execute connectivity diagnostics commands from the VR to a given target, which could be a host on the internet or some other internal element from our infrastructure that we need to make sure is reachable. This enables us to trace how the traffic goes in and out from the VR. The available commands are ping, arping and traceroute. Admins can execute those with any standard option and argument that they support under the VR operating system (Debian). CloudStack effectively logs in to the VR and executes the exact command with the given options and parameters, then it will display back the response within the management console UI.

Here’s a simple example of ping command being sent to google.com:

From Infrastructure section in the GUI, we select the VR and click on Run Diagnostics:

Then select the command you want to execute, add destination and any extra arguments and click OK.

This comes the back with the VR response:

Likewise, admins can do traceroute and arping.

Obviously, there’s a new API created for this which can be used with cloudmonkey. Here’s an example how to do that:

(localcloud) SBCM5> > run diagnostics targetid=f08d92b6-4839-4ca5-8924-bb2c59ce14c2 ipaddress=google.com type=ping params='-c 2'
{
"diagnostics": {
"exitcode": "0",
"stderr": "",
"stdout": "PING google.com (216.58.198.174): 56 data bytes\n64 bytes from 216.58.198.174: icmp_seq=0 ttl=50 time=7.664 ms\n64 bytes from 216.58.198.174: icmp_seq=1 ttl=50 time=7.645 ms\n--- google.com ping statistics ---\n2 packets transmitted, 2 packets received, 0% packet loss\nround-trip min/avg/max/stddev = 7.645/7.654/7.664/0.000 ms"
}
}

…where the targetId is the given ID of the VR in the test.

Run diagnostics is available as of 4.12 release and it is hypervisor agnostic.

Get Diagnostics Data

This feature, intended to be used by root administrators provides a way to retrieve any file from system VMs if the file path is known and specified as input. By default, the API gathers logs and configuration/property files and sends them as compressed tarball to a secondary storage pool within the same zone as the target system VM. A download URL is returned to the operator on successful file retrieval to allow him to download it to their local machines.

The API can be executed against all three types of system VMs, each of them having a separate default list of files and configurations it’ll gather. Here’s a list of the defaults for each system VM type:

• VR – ‘diagnostics.data.vr.defaults’ global setting:

“IPTABLES], [IFCONFIG], [ROUTE], /etc/dnsmasq.conf, /etc/resolv.conf, /etc/haproxy.conf, /etc/hosts.conf, /etc/dnsmaq-resolv.conf, /var/log/cloud.log, /var/log/routerServiceMonitor.log, /var/log/dnsmasq.log”

• CPVM – ‘diagnostics.data.cpvm.defaults’ global setting

“[IPTABLES], [IFCONFIG], [ROUTE], /usr/local/cloud/systemvm/conf/agent.properties, /usr/local/cloud/systemvm/conf/consoleproxy.properties, /var/log/cloud.log”

• SSVM – ‘diagnostics.data.ssvm.defaults’ global setting

“[IPTABLES], [IFCONFIG], [ROUTE], /usr/local/cloud/systemvm/conf/agent.properties, /usr/local/cloud/systemvm/conf/consoleproxy.properties, /var/log/cloud.log”

Please note that one could change the default to include custom files/scripts. To get the defaults from a System VM the admin simply calls the API just giving a target. To call a custom script the root administrator will also have to make sure the script is present at the “/usr/bin” directory on the system VM and can be executed. Once there, it’s name needs to be passed in square brackets, like this: [script]. It also accepts list of values separated by comma.

Here’s where to find it in the CloudStack Console:

Pick a VR and expand the quick view options, then you’ll be able to see ‘Get Diagnostics Data’ button and click on it:

After that, a pop-up would appear, taking one argument ‘Files’. Leave blank to get the defaults or fill in absolute paths to files or commands in [brackets] as custom values
Defaults:

or a command:

Once executed, SSVM will gather all the content in an archive and you’ll be given an URL to download it from:

And here’s the containing of the tar archive defaults for Virtual router:

Here’s an example how to use it directly calling the API from cloudmonkey:

(localcloud) SBCM5> > get diagnosticsdata targetid=1ce6de39-b4ed-412f-aced-3a421924c477 files=[ifconfig]
{
"diagnostics": {
"url": "https://10-1-36-2.sbcloud.uk/userdata/0ff2e4ae-8b55-49a0-815e-22185e45a7d1.tar"
}
}

The following global settings were introduced with Get Diagnostics FR, which let you control and configure the feature and most specifically how it uses the secondary storage of your datacenter. It’s a good practice to keep the garbage collection enabled and running so you don’t end-up with secondary storage occupied with diagnostics logs. Following is list of configurations that the admin can use.

SettingDescriptionDefault Value
diagnostics.data.gc.enable
Enable the garbage collector background task to delete old files from secondary storage.  Requires management server restart
True
diagnostics.data.gc.interval
The interval at which the garbage collector background tasks in seconds. Requires management server restart
86400 (Once a day)
diagnostics.data.retrieval.timeout
Overall system VM script execution time out in seconds. Does not require management server restart.
3600
diagnostics.data.max.file.age
Sets the maximum time in seconds a file can stay in secondary storage before it is deleted. 
86400 (1 day)
diagnostics.data.disable.threshold
Sets the secondary storage disk utilisation percentage for file retrieval. Used to look for suitable secondary storage  with enough space, otherwise an exception is thrown when no secondary store is found.
0.95 (95 %)

Conclusion

Get Diagnostics (cloudstack-#3350) has been submitted against master and against the 4.13 milestone, so hopefully it’ll make it in the next LTS release. Run Diagnostics (cloudstack-#2833) has been merged as of 4.12 release. Both are available from the UI and API, independent of the hypervisor used in CloudStack, they are handy, neat and can save tons of time accessing the VRs and finding what you need from them. Furthermore, they could be used to monitor and automate some of the processes on the VR if required.

About the author

Boris Stoyanov is Software Engineer in testing at ShapeBlue, the Cloud Specialists. Bobby spends his time testing features for the Apache CloudStack Community and for ShapeBlue clients.

Hello all, this is Abhishek Kumar, currently the newest member of the ShapeBlue family. It’s been over a month since I started working as a Software Engineer on Apache CloudStack at ShapeBlue, and I’m here to tell you about how it’s gone.

2019 has been an exciting year for me as I moved from the application development domain to infrastructure development. I always knew it will be a challenging task but also a rewarding one.

The beginning

It was last year that I moved to Gurugram, India, to work for a major med-tech company dealing with navigated intra-operative products. Prior to that, I’d been freelancing as a desktop and mobile application developer. Moving to Gurugram meant getting back in touch with some of the friends and batchmates from college who were already living and working in the city. Late last year one of these friends suggested to me the idea of applying to ShapeBlue, the company he had been working at for a number of years. I had previously heard about ShapeBlue and Apache Cloudstack from him, and I was interested in how the company works with a distributed team and how they contribute to open-source while delivering for their customers – they are community leaders in a sense. Initially I was quite unsure as I had never worked on something like this but after some deliberation, I decided to go through the compact yet effective hiring process of ShapeBlue. It involved two interviews, a coding challenge and a knowledge test on a subject that was chosen because they KNEW it was new to me (they were testing my ability to pick up new concepts very quickly). The whole process only took a week or so before I was hired as a Software Engineer at ShapeBlue!

The learning

This was my first experience of being a developer with infrastructure software and working as part of a large, open-source project. To be honest, to start with, everything was a bit overwhelming as it was mostly new to me – and the people I’m working with are probably the champions of the field. Within this first month, I’ve transitioned from C++ to Java, and have learned complex concepts of networking topologies, hypervisors and many other new subjects. I’ve not only been learning the fundamentals of the Apache Cloudstack project and working on customer projects but I’m also starting to contribute to the open-source community. A large part of this learning can be credited to the awesome training program that ShapeBlue provides for a new joinee. It is a very well-structured training course (called the “hackerbook”) that constitutes several chapters that explain a particular topic and then require the trainee to do some coding exercises to test the acquired knowledge. During the training period, a mentor is assigned to the trainee to clear any doubts, review progress and even have 1-2-1 sessions on complex topics. This contrasts with what I’ve experienced with previous employers and most programmers experience as well, where they are given access to a codebase and some limited documentation and left to figure things out on their own.

The challenges

As expected there have been a number of challenges. Moving from developing consumer-centric small applications to working on massive, complex infrastructure orchestration software would never have been easy. Then there are always those regular things one faces when moving to a new job: onboarding on company infra, learning new services and technologies to do daily tasks, following new practices & policies, etc. With Apache Cloudstack being an open-source project, it adds another dimension as it is not just your own organization but the larger community that you are dealing with.

Apart from the technical aspect I also find the social aspect of onboarding with a new organization a bit testing personally. Being a reserved, quiet, person, gelling with new people isn’t always easy for me. However, over the last month at ShapeBlue I can safely say that all these have been exciting challenges. While the technical aspects were taken care of with well-structured training, the social aspect took care of itself due the intrinsic flat organizational structure at ShapeBlue where everyone has equal say and has the freedom to communicate with anybody else in the company irrespective of their position.

The joy

I have liked being able to jump between my training course and real-world customer facing development. I was able to use the concepts I learned, during this period, in the customer project I’m working on. Within this short span of time, even though I don’t have the expertise that my team has, I still feel like I can make a contribution to the project we are working on. I can still participate in the development of new features for customers and contribute to open-source community to some extent.

Conclusions

My time so far at ShapeBlue has been nothing less than amazing! I could not wish for a better mix of challenges and rewards. Most days I do have to work hard to make sense of a very large codebase or some complex network concepts, but with enough effort, I can work my way through and go home satisfied. Being a software developer in the infrastructure domain can be challenging and learning to become a better and more efficient one might be even harder, but so far, I’m enjoying this job and loving this journey with my new work family: ShapeBlue!

Our first meetup of 2019 saw us at a new venue – Ticketmaster’s London HQ, and if you’re a music lover it certainly takes the prize for coolest meeting venue yet! Walls covered with pictures of rock stars and a stage complete with guitars and Marshall amps (not to mention pinball machines and a bar) created a real buzz of excitement before the meeting had even started. Once everyone had met up with friends, taken photos and finished lunch, Giles Sirett (CSEUG chairman) called the meeting to order, and kicked the day off with CloudStack news.

Giles talked us through the current and upcoming releases of CloudStack, and the new release of Cloudmonkey (6.0), before ‘unofficially’ announcing the new VP of Apache CloudStack – our very own Paul Angus! Moving onto market news, Giles introduced a thought-provoking topic, starting by referencing an article titled ‘What happened to OpenStack?’, before moving onto the different marketing approaches taken by the technologies.

We then heard about upcoming events – the next CSEUG will be in Sofia in June (register here), and we are currently looking for speakers. The CSEUG returns to London in October (and we are working with Ceph on making this another collaboration event), and we have CloudStack Collaboration Conferences in April (Brazil) and September (Las Vegas). Again – the Call For Participation is open for Las Vegas. All the information provided by Giles can be found by watching his talk:

Giles then introduced our first guest speaker onto the stage – Mike Rowell (Director, Platform Infrastructure) of our hosts Ticketmaster, with a talk titled ‘Our journey to a next generation cloud’. Mike did indeed take us on a journey, first explaining what challenges they needed to overcome, and what solutions they initially implemented, before discussing their investigations into a scalable cloud solution. These investigations led them to Apache CloudStack, and Mike went on to share what issues he experienced, as well as what other tools they use, such as Ansible, Terraform and Prometheus in the stack. Mike finished his talk by expanding on some features he would like to see in CloudStack.

Next to the stage was Bobby Stoyanov (ShapeBlue), talking us through some of the new features in CloudStack. These new features include: more sophisticated options for specifying pod and cluster while deploying a VM; running and retrieving diagnostics on the VR; sending additional configuration to VMs; and adding options to cleanup additional data disks when destroying a VM. It’s always great to hear about new features, and to see evidence of the continuing innovation and commitment to the project from the community. Bobby ‘dived deep’ on each feature, so I recommend you watch his talk:

After a short break, we welcomed Wido den Hollander from PCextreme, who talked about flexible networking for scaling a cloud environment,. As Wido explained – regular layer 2 VLANs have their limitations when it comes to scalability, and VXLAN overcomes these limitations, making it easier to scale out your CloudStack deployment. As of CloudStack 4.12, VXLAN can use IPv6, and Wido talked about Advanced networking + IPv6 + VXLAN which he is putting into production right now with the 4.12 release. As usual, Wido covered his topic comprehensively, and if you want to hear more, watch his talk:

We then welcomed Boyan Ivanov (Storpool) with his talk ‘Latency: the #1 metric of your cloud’. As Boyan pointed out – no two clouds are the same. However, the leading clouds all have one thing in common: they deliver on metrics, which matter to the customer. In this session Boyan examined and presented his findings on leading clouds, demonstrating why low latency is the thing that makes a cloud stand out. Watch Boyan’s talk:

Towards the end of Boyan’s talk we weren’t sure whether there would be a fifth talk, or we would be enjoying the hospitality of the Ticketmaster bar a little sooner than anticipated. All day, Grégoire Lamodière (DIMSI) had been struggling to get to London from Paris, due to disruption to Eurostar services. We had already moved his talk to the final slot of the day, and with just a few minutes to spare, he arrived! Grégoire’s talk was ‘Using message broker to extend cloud features’. As he explained, many use cases involve communication between CloudStack admin (provider) and instances (end user) regarding configuration, build and management. Grégoire presented the DIMSI team’s communication framework that enables managing user infrastructure on Windows and Linux systems from a centralized panel. Grégoire’s full talk is on our channel:

After the final ‘official’ talk of the day, Mike (playing the part of bar tender) opened the bar and we enjoyed a couple of drinks and the unofficial discussions started. We were then truly spoiled as Computacenter led us to a nearby pub and carried on buying the drinks! As usual, a fantastic event made so by the CloudStack community. Great attendance from all over Europe (including a heroic effort from Grégoire), and varied, interesting talks (thanks to Mike, Bobby, Wido, Boyan, and Grégoire). Huge thanks to Ticketmaster for hosting and providing a very cool venue (have I mentioned the slide?), and thanks to Computacenter for their generosity. We are already planning the next CSEUG which will be in Sofia, Bulgaria, on June 13 (registration open) – we are looking for talks, so if you want to come along and give a talk, please let me know at steve.roles@shapeblue.com. See you soon!

All the day’s talks were recorded, and are available on the ShapeBlue YouTube channel.

 

Our presenter’s slides can be found on SlideShare:

Giles: https://www.slideshare.net/ShapeBlue/giles-sirett-cloudstack-news

Mike: https://www.slideshare.net/ShapeBlue/mike-rowell-our-journey-to-a-next-generation-cloud

Bobby: https://www.slideshare.net/ShapeBlue/boris-stoyanov-some-new-features-in-apache-cloudstack

Boyan: https://www.slideshare.net/ShapeBlue/boyan-ivanov-latency-the-1-metric-of-your-cloud

Andrija Panic shares some thoughts on joining the ShapeBlue team

Hi there, this is Andrija from… well, ShapeBlue! I’ve been working here for a month now and I thought that I’d share my views of working for the company.

Before I move to the actual topic, let me share just a little bit of background about myself.

Before joining ShapeBlue, I was working as a Cloud System Engineer for two different Swiss-based Public Cloud providers, both utilizing CloudStack to provide IaaS services for local (Swiss) and international customers – many of which (as you can probably guess) were serious financial institutions (Switzerland being considered a big privacy and security center). We even had customers connecting all the way from South America to their infrastructure for daily business, all managed by CloudStack – and it just worked flawlessly!

During my time with the Swiss guys, I had the pleasure (with my colleagues) to lead and build their CloudStack infrastructure from scratch. Here I gained some serious knowledge and experience on this topic. I also had the opportunity to work with some nice storage solutions, from NetApp SolidFire distributed All-Flash Storage (providing block-level storage to CloudStack VMs), to Cloudian Hyperstore S3 Object Storage solution providing (you can guess by its name…) S3 object storage with 100% Native S3 API compatibility. Both solutions had their challenges of integration into existing environment and I was lucky enough to pull the strings here and lead the thing myself. Really fun time! Did I mention CloudStack? Yes, we did quite a decent job here, we made a lot of tweaks and improvements, migrations and decent customer support.

But after 5 years with CloudStack in a service provider environment , it was time for me to move on and improve my cloud building skills even more, so my next logical step was to pull Giles Sirett, ShapeBlue CEO, for a quick coffee on the last CloudStack Conference (I even didn’t have to pay for the coffee – it was a free one!). The rest is pretty much history – I’m now paving my way into consultancy as a  Cloud Architect at ShapeBlue.

After spending a month here at ShapeBlue, I can honestly say that I’m nothing short of being impressed with both the people (colleagues) and the processes inside ShapeBlue. I was already used to Swiss guys being strict and very well organized, but my feeling is that ShapeBlue has moved this to a whole new level. When I joined the company, besides having a dedicated colleague as a mentor (hi there Dag – thanks for all your help!) helping me to find my way around the company, I also got proper training on many different tools and processes used in company, from some internal infrastructure stuff, to customer support tools, processes and SLAs, to many different things in general. In fact , this was a revelation when compared to the  old RTFM-it-yourself way (stands for Read The [Insert asterisks ***] Manual), in case you were wondering) that I’d experienced at previous companies. The people at ShapeBlue are supportive, the working atmosphere is just great, with tons of seriousness across the board but with a healthy dose of (mainly) British humor in the middle of hard work – to make you wake up and warm up during these cold winter days. From time to time we even get cats jumping from our Slack channel.

After being mostly in a technical leadership position in my previous jobs, I’m now, for the first time in my professional carrier, part of the team with a more experienced guys than me – and I’m really happy about that – it’s always nice to be able to get some help in case you need advice – but individual initiative and engagement is something that is strongly respected in ShapeBlue. One of the interesting things is, that the guys in the ShapeBlue Leadership Team do actually listen to engineers and take their advice / opinion – something you don’t necessarily find in every company. It’s a very collaborative and not authoritative environment – a thing that everybody respects here.

So far, I have been tasked with quite a few interesting things to work on: from  delivering the famous ShapeBlue Bootcamp to one of our new colleagues, playing around with some more interesting CloudStack setups (with different hypervisors) and been included in some customer projects and support stuff – all in all a good start!

In case you are still following me, here come a few personal things about me:

I’m based in Belgrade, Serbia (for all you techies, that is 44.0165° N, 21.0059° E ) – a country known for good cuisine, but mostly for ćevapi and šljivovica (national drink). Serbia is also home to Novak Djokovic, the world No. 1 in men’s singles tennis (this is the guy who regularly beats Roger Federer, for the record!).

In my free time I’m hanging around with my 3 princesses and sometimes I manage to squeeze some time for gym, music or very light electronic projects.

Talk to you later, Andrija.

There was a definite feel of Christmas in the air in London as we made our way to last Thursday’s (December 13) winter meetup of the Cloudstack European User Group (CSEUG), and that only increased as we arrived at the BT Centre near St. Paul’s and saw the big Christmas tree in reception!

A great turnout for this, the last meetup of 2018, and a great representation of the CloudStack community in Europe with people travelling from Germany, Serbia, Glasgow, Switzerland and Latvia to name but a few. After a quick lunch we took our seats, and Giles Sirett (chairman of the user group) welcomed everyone and got the event started with introductions and CloudStack news.

Firstly, Giles spoke about software updates and new releases. CloudStack 4.11 is an LTS (long term support) release and included more than 250 new capabilities and a big step towards zero downtime upgrades, 4.11.2 has just been released (including 71 fixes), 4.11.3 is coming soon and 4.12 is in planning. Giles then mentioned CloudStack events starting with the recent CloudStack Collaboration Conference in September (Montreal), and events for 2019 – the next CSEUG in March (London), and the next Collaboration Conference in September (Las Vegas). During Giles’ presentation, Maurice Nettisheim (Head of Cloud Compute for BT) took to the stage to say a few words about BT’s ongoing use of CloudStack in their IaaS platform and their continued support and involvement in the CloudStack community.

Giles slides contain much more information:

After Giles, Paul Angus gave us an update on ShapeBlue’s CloudStack Container Service (CCS), giving us a walkthrough of the recently released update.This update brings CCS bang up-to-date by running the latest version of Kubernetes (v1.11.3) on the latest version of Container Linux. CCS also now makes use of CloudStack’s new CA framework to automatically secure the Kubernetes environments it creates. Paul’s talks and slides are always packed with detail:

Olivier Lambert of XCP-ng & Xen Orchestra took the floor next to tell us about the current state of the project. For those that are not familiar, XCP-ng is an opensource, community powered hypervisor based on Xen. It is easy to upgrade from XenServer (keeping all VMs, settings etc.), 100% API compatible, requires no license and has no feature restrictions.

Please take a look through Olivier’s slides for much more on this fascinating subject:

After a short break, we welcomed Ingo Jochim and Andre Walter (itelligence) with their talk entitled ‘How our cloud works’. They talked through full automation with Ansible for all infrastructure components of the cloud with CloudStack, check_mk, LDAP and more, with all functionality available through a customer portal, also covering how the setup is fully scalable for larger landscapes.
Ingo and Andre’s slides right here:

Next up was Adam Dagnall (Cloudian) with ‘Advanced S3 compatible storage integration in CloudStack’. To provide tighter integration between the S3 compatible object store and CloudStack, Cloudian has developed a connector to allow users and their applications to utilize the object store directly from within the CloudStack platform in a single sign-on manner with self-service provisioning. Additionally, CloudStack templates and snapshots are centrally stored within the object store and managed through the CloudStack service. The object store offers protection of these templates and snapshots across data centres using replication or erasure coding. Adam went into the feature-set in great detail, and his slides provide much more information:

Last talk of the day, and the honours fell to Andrija Panic (Hiag Data) with ‘CloudStack – 5 years in production’. Andrija shared real world experience of designing, deploying and managing a CloudStack public cloud, explaining how high availability for the CloudStack management components was implemented and discussing different storage technologies and networking models used, as well as the challenges faced. Andrija also presented alternate methods for deploying CloudStack as regards to regions / zones / pods, and also touched on physical networking, finally looking at the different CloudStack guest networking models available (from Basic Zone / Shared Networks to all the Advanced Zone’s networking models) and when to use each of them.
Andrija went into a lot of detail and I encourage you to look through his slides:

After Andrija had finished answering questions, Giles wrapped things up and we moved to a local pub, where I am pleased to say that conversation and collaboration continued into the night, with what rapidly became the unofficial ‘CloudStack Christmas Party’! Huge thanks to BT for providing a first-rate venue and lunch, and to all our speakers, who make these events so interesting and such a success.

The next CloudStack User Group meetup will be on Thursday, March 14, and will be hosted by our friends at Ticketmaster here in London. Please register here!

All the talks were recorded and will be made available shortly on the ShapeBlue YouTube channel.

Thursday, September 13 saw us back at the Early Excellence Centre, Canada Water, for the (late) Summer meetup of the CloudStack European User Group. As usual, a great turnout and representation of the community and Europe – with attendees traveling from Germany, Switzerland, Bulgaria, Latvia, Poland, and further afield from Ukraine. There were even a few of us there from the UK!

After we’d caught up with old friends and greeted new ones, we had a bite to eat and took our seats for the talks. Giles Sirett (ShapeBlue CEO and chairman of the CloudStack European User Group) was first up, starting with introductions, a run through the day’s agenda, and CloudStack news – and this past few months has seen lots of activity and development, including the release of the latest LTS branch of CloudStack (4.11), with 4.11.2 due soon. CloudStack 4.11 included more than 250 new capabilities, such as new host HA framework and Prometheus integration, whilst the 4.11.1 release brought us a step closer to ‘near zero downtime upgrades’ with a major refactor of the virtual router. Speaking of activity – approximately 800 downloads of CloudStack per month (in the last 6 months) shows continued strong adoption of the technology.

Giles then looked to the future, talking through upcoming events… and we were in Montreal for the CloudStack Collaboration Conference just last week! It was a fabulous event in a great city, and please see my blog for a roundup and some more information. Of course Giles also mentioned our next user group meetup – London, December 13, hosted by our friends at BT (London). Giles finished up with a call for users of CloudStack to talk more about it. For more information on that, and everything Giles talked about, here are his slides:

Giles then introduced our first featured speaker of the day – Paul Angus (VP Technology at ShapeBlue), with his talk: Backup & Recovery in CloudStack. As Paul explained – CloudStack users currently only have snapshots as a form of VM backup. With the Backup and Recovery Framework, end users will be presented with the features and functions that they have come to expect outside of ‘the cloud’, while cloud providers will be able to leverage the advantages of using enterprise backup and recovery products. In this talk, Paul explained some features of the forthcoming backup and recovery feature, the user experience and demonstrated the Veeam plugin working with the backup and recovery framework. This is a highly anticipated feature, and Paul’s slides are a treasure trove of information and detail:

Following Paul, Dag Sonstebo took control of the laser pointer. Dag is a Cloud Architect here at ShapeBlue and had chosen as his topic the CloudStack usage service. Dag started by explaining that the usage service is used to track consumption of resources in Apache CloudStack for reporting and billing purposes, before giving an overview of how the service is installed and configured. Dag then dived deeper into how data is processed from the database into the different usage types (VMs, network usage, storage, etc.), before being aggregated into billable units or time slices in the usage database.

The talk included several examples on how to query and report on this usage data, and looked at general maintenance and troubleshooting of the service. This really was a deep dive, as evidenced by Dag’s extensive slides:

After a brief interlude to grab coffee and some fresh air, next up was Olivier Lambert, the creator of Xen Orchestra and XCP-ng. Starting by talking about Citrix XenServer, Olivier explained why he developed an alternative that is truly open-source. He talked us through Xen Orchestra, before moving onto XCP-ng – a fork of XenServer removing all restrictions that were put in place with the free Citrix version. This is an exciting project, already proven and widely adopted. Olivier and his team continue to develop new functionality with a fast-growing community and have an exciting roadmap in place for future development. Olivier’s slides from his presentation are right here:

After Olivier we welcomed Vladimir Melnik to the podium (all the way from Ukraine, and I think the person who traveled the furthest). Vladimir is a co-founder of the first IaaS provider in Ukraine – Tucha, and his talk was ‘Building a redundant CloudStack management cluster’. Starting with a brief history of Tucha, Vlad covered building and maintaining an open-source-driven clustered environment for the Apache CloudStack management server with GNU Linux, HAProxy, HeartBeat, Bind, OpenLDAP and other tools. Vladimir’s slides are both entertaining and very interesting:

The honour of the last talk of the day fell to Boyan Ivanov of Storpool, providing advice on building software-defined clouds. Boyan posed the question ‘why software defined?’ and went on to answer the question quite comprehensively! Infrastructure is becoming more and more ‘software defined’, and Boyan illustrated how this should mean increased profitability, putting forward the business case for a software defined stack. Boyan was then good enough to provide several tactical tips and a free reference design!

Take a look through Boyan’s slides:

Once Boyan had finished taking questions, we all headed out to the nearest hostelry, and conversation continued into the night. A nice touch (indicative of the great CloudStack community) was that when it was time to say goodbye most people said ‘see you in Montreal’!

Thanks to Early Excellence for providing a first-class venue and refreshments, and huge thanks to the day’s speakers – Paul, Dag, Olivier, Vlad and Boyan, all of whom were good enough to donate their time, and in most cases travel great distances to share their expertise.

The next meetup of the CloudStack European User Group will be in London, on Thursday, December 13 and you can register here. We are always looking out for speakers with interesting and relevant subjects, and if you are interested in talking, please contact us.

All talks were recorded in full and can be found on our ShapeBlue YouTube channel:

Giles Sirett: https://youtu.be/Ls_HakbyxUU

Paul Angus: https://youtu.be/ZVThUKPeC_w

Dag Sonstebo: https://youtu.be/I5I7eduWHRQ

Olivier Lambert: https://youtu.be/KWBCKvwvnUc

Vladimir Melnik: https://youtu.be/aBNMysDoi5w

Boyan Ivanov: https://youtu.be/wt4pqTZ57OY

Thanks for reading, and I hope to see you at the next event!


We’re here in Montreal for the CloudStack Collaboration Conference, and it’s been a fantastic event with more to come! We’ve had two full days of back to back talks over two tracks, with subjects ranging from storage, billing and diagnostics through to containers, automation and monitoring… and everything in between. Mike Tutkowski (CloudStack VP) set the tone with his keynote at the beginning of the first day, asking the question ‘why are we here?’ The answer? To learn, work together​, share ideas​ and share problems. These fundamentals are what makes for a great community, and what makes Apache CloudStack such a great product. We have never really known just how widely adopted CloudStack is, so we have (for the first time) undertaken some in-depth analysis which Mike shared. In the last 12 months CloudStack management server packages were downloaded 116,796 times from 21,202 different IP addresses. We think this means that worldwide there are about 20,000 CloudStack clouds in production! Mike also mentioned several organisations that have recently adopted CloudStack, including Ticketmaster, from whom we saw a talk illustrating how they deployed their global cloud environment using Apache CloudStack.

The CloudStack community is full of smart, committed, talented people passionate about what they do, and this is clear from the quality and delivery of the talks, and the collaboration before and after. They aren’t just repeating facts or reading what has been written for them – they are talking from first hand experience, often about features and functionality they have personally developed and committed to the project. Thanks to the community, CloudStack is constantly being improved and developed by these real-world users and operators.

So we’re into day three, which means no more CloudStack talks. However – as I said, the event is far from finished. Today (Wednesday) we have an all-day hackathon – a room full of people working together on shared goals and ideas, the sole purpose to talk and share new ideas, and make CloudStack even better!

Every time I attend a CloudStack conference, I am privileged to spend time with a community who genuinely enjoy what they do, and I come away having made new friends, and having learnt something new. I am already excited about next year’s event, and seeing some of our new friends in London at our next CloudStack meetup (December 13).

Sincere thanks to the Apache Software Foundation (our conference co-locates every year with Apachecon). It’s always a well organised and well attended event, and we are delighted to be associated with it. Thanks also to the city of Montreal – a beautiful city which I hope to visit again soon.

All the CloudStack talks were recorded and will be published to Apache.org and our YouTube channel very soon.

    

  

On Thursday, April 19 the CloudStack community joined up with the Ceph community for a combined event in London, and what an event it was! The meetup took place at the Early Excellence Centre at Canada Water, on a beautiful, sunny day (in fact the hottest day of the year so far), and registration started early with people enjoying coffee and pastries by the canal.

CloudStack & Ceph combined morning sessions

Once everyone had enjoyed breakfast, we all settled down as Wido den Hollander (of 42on) took to the podium to welcome everyone, explain a bit about how the event had come about, and talk through both technologies (CloudStack and Ceph), including how well they work together. Having set the scene Wido was ready to deliver the first presentation of the day, which was prepared to appeal to both communities – ‘Building a highly available cloud with Ceph and CloudStack’. In this talk, Wido talked about how he came across and started using Ceph, and how his company utilises both Ceph and CloudStack to offer services. To quote Wido… “We manage tens of thousands of Virtual Machines running with Ceph and CloudStack and we think it is a winning and golden combination”! This first talk really got the event off to a flying start, with plenty of questions and interaction from the floor, and Wido’s slides can be found here:

Next up was John Spray (RedHat) with a talk entitled ‘Ceph in Kubernetes’ John talked through Kubernetes, Rook, and why and how it all works together. Lot’s more detail in John’s slides:

After a short break, it was the turn of Phil Straw (SoftIron) to present. Phil’s talk was ‘Ceph on ARM64’, and covered all the fundamentals including some real world results from SoftIton’s appliance. Take a look through Phil’s slides:

Sebastien Bretschneider (intelligence) was last up before lunch, with ‘Our way to Ceph’ – another great talk for both communities as he talked about his experience using Ceph with CloudStack. Starting with some background, Sebastien talked through lessons learnt and how the platform looks now. More information in Sebastien’s slides:

CloudStack afternoon sessions

After lunch, the day split into two separate tracks, the first dedicated to CloudStack (the CloudStack European User Group – CSEUG), the second to Ceph. As with the morning, both rooms were very busy, with full agendas. As I focussed on CloudStack, I cover the CloudStack talks here (at the end of the article I include links to the Ceph presentations).

Giles Sirett (CSEUG chairman) brought the room to order and started with introductions and CloudStack news, which (as always) there is lots of, including new releases – 4.9.3, 4.10, and the latest LTS release – 4.11. Sticking with 4.11, Giles touched on some of the new features that have been introduced, such as new host HA framework, new CA framework and Prometheus integration (much more detail in Paul’s later presentation). Giles also mentioned that we are moving towards zero-downtime upgrades – a highly anticipated and much needed enhancement! What is exciting about this community is that nearly all of the 4.11 features were contributed or instigated by USERS… and speaking of new features – this was someone’s response to Citrix changing the XenServer licencing model: https://xcp-ng.org/

Giles then advised us about some dates or our diaries – lots of upcoming CloudStack events, and shared how CloudStack is growing in awareness and use – we are seeing 800 downloads per month, just from our (ShapeBlue) repo, and have several new, large scale private CloudStack clouds being deployed this year. Please see Giles slides for much more information:

Once Giles had finished taking questions, he introduced the first presentation, and this was both Wido den Hollander and Mike Tutkowski (SolidFire) for a brief retrospective of Wido’s year as VP
of the CloudStack project, and welcoming Mike as the new VP! This was a quick introduction, and next up was Antoine Coetsier (Exoscale) with his talk entitled ‘Billing the cloud’.

Starting with Exoscale’s background, Antoine talked through pros and cons of different approaches, and explained Exoscale’s solution. Antoine’s slides are here:

As mentioned earlier, next up was Paul Angus (ShapeBlue), with ‘What’s new in ACS 4.11’, which includes 100s of updates, over 30 new features and the best automated test coverage yet. Paul went through user features, operator features and integrations. This demonstrated just how much work and development is going into CloudStack, and if you are interested to see just how much work and development, I urge you to read through Paul’s comprehensive slides:

After a short break, Giles introduced the next talk of the day – Boyan Krosnov (Storpool), ‘Building a software-defined cloud’. After a little background, Boyan started by explaining how a hyper-converged cloud infrastructure looks, and then talked about how Storpool works with CloudStack.

Ivan’s talk was filmed: https://youtu.be/_d_xnqpmY0M, and his slides can be found here:

Next up was Mike Tutkowski (SolidFire), who talked us through managed storage in CloudStack, and then presented a live demo so we could see first-hand how it works. As Mike works for SolidFire this was the storage he used for his demo!

The end of this last talk of the CloudStack track coincided with the end of the last Ceph talk next door, so we all congregated together to hear some last thoughts and acknowledgements from Wido. This was the first collaborative day between Ceph and CloudStack, and it was a resounding success, thanks in no small part to the vibrant communities that support these technologies. Bringing the event to a close, we all moved the conversations to a nearby pub, where the event continued late into a beautiful London evening.

Thanks to our sponsors, without which these great events wouldn’t be possible – 42on, RedHat and ShapeBlue (us!) for providing the venue and refreshments, and to SoftIron for providing the evening drinks.

Ceph afternoon sessions

Wido den Hollander – 10 ways to break your Ceph cluster

Nick Fisk – low latency Ceph

Many people find it challenging to get started with CloudStack’s networking. There are some basic concepts, which although not overly complicated, are not especially obvious either. This blog will try to explain these underlying concepts, in order to make getting started with CloudStack networking models much easier.