In the ever-evolving realm of virtualization, hypervisors are a fundamental technology enabling the management of virtual machines. The choice of the underlying hypervisor for CloudStack-based clouds is one of the most important architectural decisions to make, impacting everything from resource utilisation to IT budgeting and the ability to adapt to future technological shifts. Unlike other solutions on the market, Apache CloudStack is a hypervisor-agnostic solution. This gives you the freedom to build your own flexible technology stack, optimise costs and ensure a more competitive price for your end products.

At ShapeBlue, we are often asked by our customers what’s going on in the hypervisor market and which hypervisor is the best fit for specific use cases. The truth is, there isn’t a simple answer, as each use case and technology stack is unique.

That said, there are now several main hypervisors in play – VMware vSphere, KVM, Microsoft Hyper-V, XCP-ng, Xen Server, Proxmox and Oracle VM. To help get a better understanding of where and how they fit, ShapeBlue brought together a set of experts who have deep expertise with these technologies and have had to make choices over their use to discuss where they see this all going and compare and contrast the different hypervisors.

The panel was made up of:

Olivier Lambert, Founder and CEO of Vates, the company which is behind both XCP-ng and Xen Orchestra. He’s been involved in open source for 20 years and is also contributing to open source upstream, like the Xen hypervisor. His deep experience with open source gives him a unique perspective on the role of open-source hypervisors in the market.

Lino Telera from Italy, a cloud architect with a focus on infrastructure automation. Lino is a Cloud Architect focused on Infrastructure automation and Cloud-Native. Coach at InfoCert S.p.A. He’s a vExpert x9, VMUG Leader, VMware {code} coach, TechFieldDay delegate, and HashCorp Ambassador 2022. He spends his spare time authoring his blog https://blog.linoproject.net, running an Italian podcast – “The Pipeline Guys”- and teaching coding to kids! His experience with infrastructure automation provides valuable insights into integrating hypervisors with other systems.

Wido den Hollander is CTO at Your.Online, a European domain and web hosting provider. Your.Online has a CloudStack infrastructure based on the KVM hypervisor running in the Netherlands and Spain. The foundation for this infrastructure was built in 2011 when Wido’s original company he founded in 2004, started to use CloudStack. Wido’s experience with KVM in a large-scale deployment offers a practical perspective on this hypervisor.

Kev Johnson – Technical Marketing Architect at Rubrik, Inc. Kev is a Tech Marketing Architect at Rubrik, where he builds demo videos, writes technical white papers, posts blogs, builds hands-on labs, and speaks at industry events. He has been in tech for almost 25 years, the most recent areas being focused on virtualisation, cloud, and security. Outside of work, he’s passionate about food, football, and fixing the ills of society. His broad technical marketing experience allows him to articulate complex concepts in an accessible way.

Finally, the panel was chaired by Giles Sirett, CEO of ShapeBlue, the CloudStack company.  Giles’ leadership role at a company focused on CloudStack gives him an in-depth view of the hypervisor market.

So, with the introductions over, let’s start by looking at the hypervisors up for discussion.

Get in touch

VMware – Is it still the market leader?

VMware has been the enterprise market leader for quite some time. It’s got a pretty mature ecosystem, there are a lot of integration points, and they have some good and some not-so-good APIs. They’ve also got OEM partnerships and products like Tanzu, which indicates they’re going after the container ecosystem, but there are other players in the market, like OpenShift. Finally, they’ve got all of the VMware Cloud offerings as well. So, for organisations that are looking to get to the cloud, they’ve got that tactical first step to getting there.

However, the choice of VMware isn’t always straightforward. Some organisations might find the cost prohibitive, while others might prefer the flexibility and control offered by open-source alternatives.

Lino – “Some 15 years ago, I helped build an open-source solution for a telco. Initially, our mission was to build from scratch our first cloud. We ultimately chose vSphere because of the operative model and its integration with other solutions, such as identity systems or the underlying switches and storage.”

“Today, that company still has that first solution in their data centre, instead of in the cloud. The operative model is the winning card! Being able to keep the same operative model on-premises and in the cloud without changing anything on the virtual machine and the organisation is the reason why VMware is still one of the leaders.”

Lino’s experience highlights the value of VMware’s operative model, which allows organisations to maintain the same workflows and processes whether they’re operating on-premises or in the cloud. This consistency can be a significant advantage for organisations that want to minimise disruption while transitioning to the cloud.

KVM – The major alternative for large-scale deployments

KVM is a good fit for those with large-scale deployments. As it is open-source, KVM allows you to modify the solution and the ecosystem around it, to make it work with your organisation. Ideally, you need to have engineers in-house to run and support KVM. But if you don’t have that level of resource, you will need to choose one of the Linux distributions Canonical, Red Hat or SUSE.

KVM’s open-source nature gives organisations the flexibility to tailor the hypervisor to their specific needs, which can be a significant advantage in large-scale deployments where standard solutions might not be sufficient. However, this flexibility comes with the requirement for more technical expertise, as KVM can be more complex to set up and manage than other hypervisors.

One of the reasons that VMware became so dominant was the ease of getting started with it. You don’t need to have a deep understanding of the Linux kernel. Consequently, it’s no surprise that organisations are moving to alternative hypervisors, such as KVM because, to a certain extent, the hypervisor has become commoditized to the point where pretty much anybody could set it up.

Xen and XCP-ng

There are two different aspects of utilising Xen. Firstly, you have Xen alone. That is the same situation as for KVM alone and is where you will take support from SUSE.

It was historically the first open-source hypervisor, so it got a lot of traction from the start, and then KVM came along and started to grow, thanks to Reddit. But now, people are not truly using Xen alone but are more into platforms.

The shift towards using platforms built around Xen reflects a broader trend in the industry towards integrated solutions that provide a full suite of capabilities rather than standalone hypervisors. These platforms can offer benefits such as improved ease of use, better integration with other systems, and the ability to leverage the strengths of Xen while mitigating some of its complexities.

This is important, as organisations are now building platforms using Xen as their core. Consequently, people are seeking solutions that are not just the hypervisor but the overall solution that just works.

It is worth also considering XCP-ng, which was a fork of Citrix Hypervisor.  The idea behind it was to bring back the old open-source version of XenServer, XCP (Xen Cloud Platform), and so XCP-ng stands for Xen Cloud Platform – a new generation. XCP-ng is a project in constant evolution and on several fronts at once. They are working on the platform’s security as part of the innovation contest and on a port to the new RISC-V open-source CPU architecture. The company behind XCP-ng is Vates. They also have work in progress with external vendors like LINBIT  to develop a hyperconverged solution compatible with XCP-ng and put lots of efforts in creating more and more integrations and improved releases for XCP-ng.

Why are organisations looking at alternative hypervisors?

So, with VMware being the dominant vendor in the datacentre space, why are we seeing many people using alternative hypervisors?

Wido – “One of the reasons why we chose KVM is because there are so many plugins for it. For example, we’re using Ceph as our storage deployment, and it’s natively integrated into KVM. This has made it very easy to deploy and run.

“But it’s also seen that KVM has a very stable track record over the last ten years and delivers what it needs to do – virtualization for our cloud environment. One of the downsides I personally see, and reasons why people are moving from VMware is because you’re tied into a single vendor, which dictates what kinds of things you can use.”

This vendor lock-in can limit an organisation’s flexibility and control over their own infrastructure. For example, they may be unable to use certain types of hardware or software that are not supported by VMware, or they may be forced to pay for features or capabilities that they don’t need. In contrast, alternative hypervisors like KVM offer greater flexibility, as they are not tied to a specific vendor and can be customised to meet the organisation’s specific needs.

“So, integration into a certain type of switch or a certain type of storage, if it’s supported by VMware, then it works. But with KVM, you need engineers who understand this stuff, but I also see it as a bonus because otherwise you’ll have a company potentially without any real knowledge internally that is dependent on the ongoing support of VMware.”

“One of the downsides I personally see, and reasons why people are moving from VMware, is because you’re tied into a single vendor, which dictates what kinds of things you can use.”

So away from VMware, you can make your own choices instead of having to think about whether the licensing model allows us to do this, or does the contract allow us to do that? You have the freedom to operate as you think necessary.

The benefits of Hyper-V – when it makes the most use?

Kev – “My background was in Windows for a long time, and from there, I went to VMware. The idea of SSH-ing into a box and managing it, using the CLI, would have been terrifying to me. And I think one of the things that Microsoft had done pretty well with Hyper-V is they’ve made it really easy for if you’re a two-person IT shop, you can go and set up the hypervisor, and you get it running.

Hyper-V‘s ease of use, particularly for those with a background in Windows, can be a significant advantage for small IT teams or organisations with limited technical resources. Its integration with other Microsoft products can also simplify management and support, making it a good fit for organizations that are heavily invested in the Microsoft ecosystem.

“And it will work, but when you start looking at things like scaling out to something like a cloud provider, that’s where it gets a little bit trickier. You’ve got tooling like SCVMM, which frankly isn’t the tool that vCenter is. You might find some PowerShell that will help you out but, I can’t see it ever becoming a massive underlying hypervisor. We’ve obviously got Azure, but I guess from below the hyperscale tiers, I can’t see Hyper-V ever really taking off.”

The impact of the Broadcom acquisition of VMware?

What is it about the Broadcom acquisition of VMware that is scaring people so much? Why are people worried about what is ultimately just a straightforward acquisition?

Olivier – “The Broadcom announcements with VMware truly triggered something in the community. In the months after it was announced, we tripled the number of people registered to our newsletter. People were asking how to migrate from VMware for many different reasons. So, maybe Broadcom was not the main reason. But, a lot of people realized they only had very limited budgets in terms of hiring people to manage their IT, and they counted only on managing VMware.”

The Broadcom acquisition has raised concerns about potential changes to VMware’s pricing, support, or product roadmap. These uncertainties can be unsettling for organizations that rely heavily on VMware for their virtualization needs and may prompt them to consider alternatives as a risk mitigation strategy.

Although organizations are thinking about migrating away from VMware, they are at the start of a long road and are likely to move in small steps. But for people considering open-source, Olivier has a word of warning.

“There is a gap in a lot of open-source projects regarding the way how easy or complex a program is to use. You always have the impression that it is DIY stuff. That’s how we have built our success in open-source projects by building the glue around to get something close to VMware in terms of the impression of usage.”

So, it’s a balance regarding what you need, something that just works out of the box or something more flexible depending on it requiring configuration. There are no silver bullets, but this is one of the underlying factors of concern over VMware currently.

Companies are now exploring alternatives to their virtualization platform purely because they’ve looked at what Broadcom has done historically and what their business model is. Typically, they acquire something, and they’ll focus on their top customers. Anything outside of that is not a priority.

This means that a responsible organisation must do a risk assessment and determine, ‘who do we want to partner with’? Do we want vendor lock-in, or do we want to explore alternatives? In the past, when Hyper-V was launched, many VMware users switched to Hyper-V without putting a great deal of thought into it. This time the Broadcom acquisition could cause many more to do the same.

The difference between virtualization and cloud

Many people don’t understand the difference between virtualization and a cloud environment. They believe that because they are running VMware and they have a number of hypervisors, with a few VMs running, they have a cloud.

While virtualization and cloud computing involve creating virtual machines, they serve different purposes and offer different capabilities. Virtualization is a technology that allows you to create multiple simulated environments or dedicated resources from a single, physical hardware system, while cloud computing is a service that delivers shared computing resources, software, or data on demand, via the internet.

They then ask, should I switch to CloudStack? But as they’re not running a cloud and looking for a VMware alternative, they might want to go to Proxmox. This is a solution built around the KVM hypervisor, and is super easy to deploy and run.

The choice between virtualization and cloud computing depends on an organisation’s specific needs. For example, an organisation that needs to maximise the use of its physical resources might choose virtualization, while an organisation that needs to scale its resources quickly and easily might choose cloud computing.

A lot of people are using VMware just for virtualization and are looking to make the step towards the cloud, but they actually don’t understand the difference.

A move to a multi-hypervisors solution

In the CloudStack world, as well as people wanting to build Iaas Clouds, we increasingly see people who simply want to  abstract away from hypervisor APIs, specifically the VMware API. They want an abstraction layer to give them that independence. They’re not looking to instantly migrate as this is a massive decision. Some of those considering it are quite traditional enterprise organisations, but they want something in between to give them a single API. So, if that hypervisor changes at some point in the future, they can change.

A multi-hypervisor approach can provide organisations with greater flexibility and resilience. By using multiple hypervisors, organisations can take advantage of the unique strengths and capabilities of each, while also avoiding the risks of vendor lock-in. This approach can also make migrating workloads between different environments easier, which can be a significant advantage in a dynamic and rapidly changing IT landscape.

Typically, when a company makes an acquifasition like Broadcom has with VMware, the first impact the community sees is budget cutting everywhere. Then they establish a new asset and a new organisation. But the feedback coming from the VMware community is different. It seems like Broadcom is not going to do the same as happened in the past with the other companies. Which can only be good news for the VMware community, as it looks like they will still be in the hypervisor space.

But there is still the question ‘Why should I stay with VMware if I can choose another operative model that is completely abstracted from the hypervisor?’ In this case, the multi-hypervisor solution can be the answer for a lot of companies. During the last ten years, companies have changed from the perspective of running their workload in a VM to today running them in a container. This has led a lot of companies to move on the cloud native space, especially Kubernetes.

Five to eight years ago, there was the feeling that virtualization as a concept was dead. Everything was going to be cloud native. But even though there has been massive growth in cloud-native adoption, there is still 16-17% compound growth to 2030 of the data center virtualization space because clouds have got to run somewhere.

The impact upon skills required 

These changes are profoundly impacting the skill sets of people working in the IT sector. We’re seeing a new generation of software developers that are growing up on open-source and so they don’t have a legacy of Windows. They’re happy with Linux and they’re building tooling and applications that tie in neatly with those open-source platforms.

This shift towards open-source and cloud-native technologies is creating a demand for new skills and expertise. IT professionals now need to be familiar with a wider range of technologies and platforms and be able to work effectively in more diverse and complex environments. This creates opportunities for those who can adapt and learn these new skills, but it can also be a challenge for those who are more comfortable with traditional, proprietary technologies.

Alongside this, the younger generation is used to working with the projects instead of building those projects – they are used to working with cloud data.  Building cloud environments and data applications is something they’re not being taught or actually seeking to learn anymore.

This shift in focus from building to using technologies is changing the nature of IT roles. Instead of being responsible for setting up and maintaining infrastructure, IT professionals are increasingly expected to be able to use existing platforms and services to deliver value to the business. This requires different skills, including a deeper understanding of business needs and a greater focus on integration and orchestration.

What this means is more users – as developers become users. They expect an API they can consume and build their application with instead of being the guys actually building the API and building the infrastructure underneath it.

This means the IT solutions companies will still need the skills to build an infrastructure where the platform runs. They need to seek engineers who can build a cloud environment.

The role of the cloud in the future

If you were starting a business right now and your business was in software where you were building an app, you’re probably going to build it to be cloud-native. You’re not going to go down the route of needing web servers, load balancers and databases, because the cloud provider can already give you that more quickly, simply and economically.

The cloud’s ability to provide scalable, on-demand resources makes it an attractive option for startups and other businesses that need to be able to scale quickly and efficiently. It also allows businesses to focus on their core competencies, rather than managing and maintaining their own infrastructure.

But, at the same time, there are many use cases where organisations do want that capability, as they need to be able to deliver it in their own datacentre because of regulatory or security reasons. They also want to be able to leverage that same interface to deploy things to a cloud provider. As a result, organisations are starting to repatriate workloads back out of the big hyperscalers because of cost. It’s become much cheaper for them to run it themselves.

This trend towards hybrid cloud models, where organisations use a mix of on-premises and cloud-based resources, reflects modern businesses’ complex and varied needs. It allows organisations to balance the flexibility and scalability of the cloud with the control and security of on-premises infrastructure.

Looking forward, the cloud helps the development of new tools, like infrastructure as code. But people tend to forget that you can now have those tools for even your own on-premises solution. Meaning that at some point, you can use them with the kind of similar flexibility that you could have in the cloud, but running on-premises or from a third-party hosting company.

Such shifting to open-source tooling was meant initially for the tool. But now you can replicate on-premises into racks. Although this will require maintenance, they are really inexpensive. The increasing amount of power you can put in one rack right now for a decent price is astonishing. You can fit on them 90% of whatever software you need to run and to scale up to two racks will be inexpensive without having to go to a public cloud to do that.

Hypervisor market in the next five years

Virtualization is not new. It’s more than 20 years old. Now you have more choices than ever, which is great. So, everyone has to do due diligence when making a decision. It’s not just a straight choice between VMware or Amazon. Now you have the spectrum of every possibility and for your own use case, you might find things that could be a good fit. There’s no universal solution anymore.

The increasing diversity of the hypervisor market reflects the growing complexity and diversity of IT environments. As organisations adopt a wider range of technologies and platforms, they need hypervisors that can support these diverse environments. This is driving innovation and competition in the hypervisor market and is likely to lead to the development of new and improved hypervisor technologies.

Having a choice means being able to make design decisions when you’re deploying workloads, whether you’re deploying them on premises or to the cloud. Whether they’re going to be containerized or virtual machines. Even if it’s something, you still need to run on bare metal for one of the legacy cases that still exist.

This flexibility is increasingly important as organisations move towards more agile and dynamic IT models. The ability to choose the right hypervisor for each workload can help organisations optimise their resource usage, improve performance, and reduce costs.

These are tricky decisions to make, and it’s easy to get into the mindset that there is only one correct solution. From a hypervisor’s perspective, CloudStack is abstracting that complexity away. So, you don’t have to worry about the hypervisor, you don’t need to worry about kernel patching, you don’t need to worry about any of that stuff. You need to figure out which APIs you need to consume to deliver and what you need to deliver.

The trend towards abstraction and automation is likely to continue, as organisations seek to simplify their IT operations and focus on delivering value to the business. This could lead to the development of more advanced and intelligent hypervisors that can automatically optimise resource usage, manage workloads, and adapt to changing conditions.

If, for example, you were an AWS user, the challenge is that every time you log into the management console, there are many new services that you need to understand quickly, especially the ones that directly (or indirectly) impact you. From an IaaS cloud perspective, this idea of abstraction gives you a degree of simplicity. This is what enabled VMware to become the dominant market force, by making their solution consumable by everybody.

Get in Touch

So where does this leave the hypervisor market over the next five years? Our panel felt that:

• We will see a more segmented market with VMware still having a large share but facing more competitors and companies moving away.
• All of the organisations will ideally be getting to the position of having universal tooling on top – like CloudStack. Also, having APIs that could use TerraForm and other solutions giving you the choice of the best tool, you need for your use case.
• A lot of applications will move naturally into the cloud because of the cost reduction and new operative models. A lot of companies are incentivized to put them there because of the Kubernetes space and also the evolution of the application in the cloud-native application.
• We will see an increase in the edge datacentre where the hypervisor will play an important role. Integration is better to manage all of the things that are running on the cloud on the edge.
• We’re going to see greater abstraction and more abstraction layers so that people can consume those hypervisors without actually having to care what those hypervisors are.
• Improved performance out of virtual machines through better disk and network access. VMware has some great integrations with NVMe over Ethernet or NVMe Fabrics, but that’s all tied to specific vendors. With the hypervisor we have open standards where we can talk to NVMe in a standardised way and get all the performance out of the NVMe. The same goes for 100gb networking, for 400gb networking, or even 25. We’re not getting the performance in the virtual machines yet, as we’re expecting on bare metal.

Conclusion

The landscape of hypervisors is evolving, driven by the shift towards cloud-native technologies, the growing complexity of IT environments, and the increasing demand for flexibility and efficiency. As organisations continue to diversify their technology stacks and adopt new platforms and services, the role of hypervisors is becoming more important than ever.

While VMware remains a dominant force in the market, the rise of alternative hypervisors and the increasing adoption of multi-hypervisor strategies create new opportunities and challenges. The recent acquisition of VMware by Broadcom has also sparked interest and concern, prompting many organisations to reassess their hypervisor strategies and consider other options.

In this context, solutions like CloudStack are gaining attention for their ability to abstract away the complexity of managing different hypervisors, providing a unified API and a simplified management interface. This abstraction and automation will likely become more prevalent as organisations seek to simplify their IT operations and focus on delivering value to their businesses.

Looking ahead, we expect to see continued innovation and competition in the hypervisor market. The trend towards abstraction and automation is likely to continue, leading to the development of more advanced and intelligent hypervisors. At the same time, the growing focus on cloud-native technologies and open-source platforms is likely to drive changes in the skills and expertise needed in the IT sector.

Ultimately, the future of the hypervisor market will be shaped by the needs and priorities of organisations. As they navigate the complex and rapidly changing IT landscape, they will need to make careful decisions about which hypervisors to use, how to manage and optimize their resources, and how to deliver value to their businesses. The choices they make will determine not only the success of their own IT operations, but also the direction and pace of innovation in the hypervisor market.

Get in Touch