Skip to main content

IBM R&D Labs in Israel

image: IBM and Haifa

Turtles all the way down

Decisive breakthrough from IBM researchers in Haifa introduces efficient nested virtualization for x86 hypervisors

What is nested virtualization and who needs it? Classical virtualization takes a physical computer and turns it into multiple logical, or virtual, computers. Each virtual machine can then interact independently, run its own operating environment, and basically behave like a separate physical resource. Hypervisor software is the secret sauce that makes virtualization possible by sitting in between the hardware and the operating system. It manages how the operating system and applications access the hardware.

IBM researchers found an efficient way to take one x86 hypervisor and run other hypervisors on top of it. For virtualization, this means that a virtual machine can be 'turned into' many machines, each with the potential to have its own unique environment, configuration, operating system, or security measures—which can in turn each be divided into more logical computers, and so on. With this breakthrough, x86 processors can now run multiple 'hypervisors' stacked, in parallel, and of different types.

This nested virtualization using one hypervisor on top of another is reminiscent of a tale popularized by Stephen Hawking. A little old lady argued with a lecturing scientist and insisted that the world is really a flat plate supported on the back of a giant tortoise. When the scientist asked what the tortoise is standing on, the woman answered sharply "But it's turtles all the way down!" Inspired by this vision, the researchers named their solution the Turtles Project: Design and Implementation of Nested Virtualization

They said it wasn't possible

"In systems research, sometimes the boundary between research and engineering gets a little fuzzy," explained Muli Ben Yehuda, research scientist and one of the leaders of the Turtles team. "We built something real that works, but it's also something that was thought to be impossible. We couldn't resist the challenge – it's got research written all over it."

Although virtualization was first introduced in the 1960s by IBM to boost utilization of large mainframe systems, the new distributed environments required for cloud computing have brought virtualization solutions back into the limelight.

The researchers are already seeing the first sparks of their work's influence and its impact on both industry and academia. The work already has numerous citations and computer science classes in several universities have included the Turtles paper in their course materials.

What can you do with nested virtualization? More than you think

"With nested virtualization for x86, it is possible to envision embedded hypervisors for Intel-based servers, much as IBM does for System P and System Z; the user can run their hypervisor of choice, on top of the embedded hypervisor," noted Abel Gordon, research scientist and another one of the leaders for this breakthrough.

Nested virtualization can be also used to enhance cloud offerings with new exciting capabilities:

  • A person who has their business running on a virtual machine in some provider's cloud environment, can now run their own hypervisor on top of the virtual machine. By adding as many virtual machines as they like, they can in essence create their own cloud.
  • By using one machine to build a small cloud with a number of hypervisors, we can simulate how code and software will work on a bigger cloud with many machines.
  • When data center prepare for moving to the cloud, nested virtualization can be used to replicate the customer's environment by using nested virtualization to represent all the different machines and configurations at the original site.

Where to now?

Gordon explained that the researchers began their work using the Linux Kernel-based Virtual Machine (KVM), core from Red Hat. Because of the lack of architectural support for nested virtualization, an x86 guest hypervisor cannot use the hardware virtualization support directly to run its own virtual machine. The Research team developed software extensions for KVM to multiplex multiple levels of virtualization (multiple hypervisors) on the single level of architectural support, thereby making efficient nested virtualization possible.

After a year of intense and ongoing iterations, handled by fellow researcher Nadav Har'el with the open source community, the IBM solution is now part of official Linux distributions. The team will continue to maintain the code, fix errors, and send extensions. "We're also looking forward to seeing this go out to IBM customers and service offerings," continued Gordon. "It opens a lot of exciting opportunities."


Awarded Best Paper at OSDI

The paper The Turtles Project: Design and Implementation of Nested Virtualization was awarded the "Jay Lepreau Best Paper" award at OSDI 2010, a PIC top-quality conference and one of the most prestigious computer science conferences in the world. This was one of the first Israeli papers ever accepted by the conference and the first IBM paper, with work done exclusively by IBMers, ever accepted to OSDI.

Going Open Source

Credit for having the Turtles solution incorporated as part of the new Linux release goes to Nadav Har'El, who serves as maintainer of the nested KVM support.

See Video

For more technical details on how the Turtles project accomplished its goal, see the YouTube video of Muli's talk at the 9th USENIX Symposium on Operating Systems Design and Implementation.

The Authors

Congratulations to authors: IBM Research - Haifa: Muli Ben-Yehuda, Zvi Dubitzky, Michael Factor, Nadav Har'El, Abel Gordon, Orit Wasserman, Ben-Ami Yassour. IBM Linux Technology Center: Anthony Liguori, Michael D. Day.


Some of the original approach derived from a project that Research did with the LTC (Black Walnut). Some of the Linux code written and validated by Research is modeled after KVM code that virtualizes AMD's SVM instructions without any of the performance improvements.

Your IBM expert

Muli Ben-Yehuda, Systems Researcher, IBM Research

Muli Ben-Yehuda: Systems Researcher, IBM Research

Abel Gordon, Research Staff Member, IBM Research

Abel Gordon: Research Staff Member, IBM Research

Michael Factor, Distinguished Engineer, Storage and Systems, IBM Research

Michael Factor: Distinguished Engineer, Storage and Systems, IBM Research

Nadav Har'el, Researcher, IBM Research

Nadav Har'el: Researcher, IBM Research