First experiments with Nutanix AHV Turbo.

One of  the improvements that came along with Nutanix  AOS 5.5 was IO path optimization feature called “AHV Turbo”.

While the marketing department at Nutanix might have fallen asleep and woken up in 1990s with god-awful name like this, it is actually a GOOD feature.  Every time I hear word “Turbo” in IT related matters, it just reminds me of my first ever PC, a 286 with a “Turbo” button, which supposedly boosted performance by increasing  frequency from 8 Mhz to 12 Mhz, but did actually do diddly-squat 🙂

What does AHV Turbo do?

In short the old IO path based on Qemu SCSI Data Path contained parts that were serialized, i.e some functions could be only executed one operation at a time. In AHV Turbo those parts are now upgraded to version where multiple operations can be issued at the same time, operations can now run in parallel. For more in-depth description of AHV Turbo, please check out: What’s .NEXT 2017 – AHV Turbo Mode.

What are the benefits of AHV Turbo?

By being capable of doing multiple things at the same time, you will get more work done and get better aggregate throughput (more IOPS) and response times (lower latency). This is where computers are different from human beings, with computers multitasking actually helps, most often with humans not so…

Is single Virtual Disk per VM enough to maximize performance?

According to Josh Odgers’ post above: “Another benefit of AHV and Turbo mode is that it eliminates the requirement for administrators to configure multiple PVSCSI adapters and spread virtual disks across those controllers.”

I think that above comment has generated some misunderstanding and I’ve heard from several sources that you only need one vdisk with AHV turbo to max out VM IO performance. Well that is not quite true, the post mentions that the need of multiple PVSCSI adapters is eliminated with AHV turbo, but does not say that you wont need multiple disks.

Testing single vDisk VM vs multi vDisk VM

To clear out confusion, I wanted to test if number of virtual disks had any relevance to max IO performance with AOS 5.5 (with Turbo mode). I created two Windows 2016 Server Virtual Machines with identical setup, except one point: E: drive that was holding test data  was configured differently between Server1 and Server2. Server1 had only on vdisk. While Server2 had five vdisks, which were combined together as striped dynamic volume to form a single E: drive under windows.

While Nutanix EULA prevents me to show actual numbers, I can tell you that with multiple vDisks Server2 was outperforming single vDisk Server1 by a lot, Server2 was doing easily 4-5 times more IO operations than Server1 while maintaining similar level of latency. For both servers throughput and latency were better than with the same tests run on previous AOS 5.1 version.

“Multiple vDisks required” point is also verified in Michael Webster’s post 1 Million IOPS in 1 VM – World First for HCI with Nutanix. Where he states “The VM used is configured with locally attached disk volumes, 33 in total (1 for OS, plus the rest for the IO load).”

To get the absolute maximum performance

Both Windows and Linux workloads benefit from AHV Turbo without making any tuning or changes at the operating system layer.

However to  squeeze out the absolute maximum out of  Windows workload, a new multiqueue driver is required. Current VirtIO-SCSI driver package does not contain this driver yet, but the future versions will do.

Linux kernels 4.14 or later have already built-in multiqueue driver. You need to set SCSI MQ parameter to get the maximum out of multiqueue features of AHV Turbo. More info: Enabling AHV Turbo on AOS 5.5

Conclusion

You will still need multiple virtual disks to drive maximum IO performance out of single virtual machine, even with AOS 5.5 AHV Turbo. In future multiqueue drivers will boost performance even more.

Call to action

Update your Nutanix cluster to AOS 5.5 to enjoy improved IO performance. This demonstrates the beauty of software defined architecture: Even the existing customers will get the benefits of improved architecture. If your architecture is heavily hardware dependant, you will have to upgrade your hardware (and spend some money) to enjoy similar benefits.