Nutanix Hybrid Selection Guide

In earlier post I went through some Nutanix spec sheet issues and while describing some disk size related issues, I made SSD/HDD capacity ratio calculations with Nutanix Hybrid (SSD+HDD) setups. I used only couple of SSD+HDD setups and Nutanix NX platforms as an example.

• Why not go through possible SSD/HDD ratio ranges for more NX platforms?
• Find the floor and the ceiling for SSD/HDD ratios?
• It might be useful in selecting NX Hybrid setup for workloads with certain requirements for SSD/HDD ratio?

 

Hybrid solution basics

First let’s cover some hybrid related basics, before diving into SSD/HDD ratio calculations.

• Why would you even consider using hybrid setups?
• Since all flash is sooooooooo much better?

HDD drives all dead?

Yep, most of the time in primary storage use cases, where not only your storage has to provide capacity, but also has to offer reasonable latency. In such use cases it is next to impossible to make a business case where HDD only setups could beat SSD only setups. That is, if using only HDD drives vs only SSD drives.

For secondary storage where latency is not that important, HDD is still the king, offering  better price / capacity ratio than SSD.

So if your use case is latency sensitive, you should not use HDDs at all?

Nope, you could still use HDD drives, combined with SSD drives. The “hot” latency sensitive data would go to “hot tier” made of SSD drives and the “cold” latency insensitive data would go to “cold tier” or “capacity tier” made of HDD drives.

Nutanix vs Traditional Storage

The storage part of Nutanix (hybrid) solution is much different than traditional storage solutions, even when compared to some scale-out storage solutions.

With Nutanix, portion of the SSD capacity is reserved as persistent storage, if sized correctly your active workloads can be served from the fast “SSD tier” most of the time. One could think “SSD tier” as kind of All Flash storage pool within the system, offering “All Flash like” performance. With default settings your workload might be evicted from this “SSD tier” to “HDD tier”, if it is considered “colder” than some other workloads and the capacity of the “SSD tier” is running low. You can prevent VMs or vDisks related to your workload being evicted to “HDD tier” by “pinning” the VM or vDisk to “SSD tier” permanently, by doing so guaranteeing “All Flash like” performance.

However, most of the time I would recommend using the automatic tiering between the “hot tier” and “cold tier”, it works quite well, doesn’t wait too long to make decisions and uses fairly small chunks of data when moving data between the tiers, so it is typically more efficient than “tiering ” with traditional storage solutions or even “Pinning”. If “pinning” is used too much, you might end up populating your “SSD tier” with cold data and losing some capacity and performance advantages of hybrid systems.

Usually with traditional hybrid storage solutions “flash tier” usually acts more like a cache and it is not used as a persistent storage tier, i.e eventually your workloads will up to “slow tier” and you cannot prevent that happening by “pinning”. Many times “flash tier” is used to speed up only read operations, sometimes they are able to cache writes as well, but sometimes not all write operations, maybe only subset of write operations like random overwrites.

Nutanix also benefits from distributing I/O operations related to a single workload throughout the cluster, the more nodes you have, the more load you can sustain.

With traditional storage solutions I/O load for a given workload is usually spread between one or two controllers and it is harder to scale performance in such cases and the performance of a single workload might be capped by the controller, even if the other controllers are not saturated.

Nutanix vs other HCI solutions

With Nutanix it is more feasible to use hybrid setups as using hybrid does not limit using some of the storage efficiency capabilities, like dedupe or compression. Unfortunately this is  not the case with all the HCI platforms, where you have to use All Flash in order to use some of the storage efficiency features.

And then there is “data locality” feature. Under normal circumstances Nutanix tries to keep data local to the node where the VM is running, and serves reads from directly-node-attached SSD drives, only writes traverse through the network. So if your read/write ratio typical is 70/30, it means that 70% of the I/O is served with local SSD latency without added network latency. This is a unique feature, not found with other HCI solutions, which typically act more like traditional storage systems and have to traverse through network while reading and writing data, sometimes there is some data locality, but it is more based on luck than design.

Finally under the hood some of the other HCI solutions act more like traditional storage systems and are NOT offering the full benefits of distributed nature of HCI scale-out solutions by NOT spreading the i/o load throughout the cluster for individual workloads. You can think of these HCI solutions as virtualized traditional storage solutions, yes, you get some benefits by virtualizing, but loose some HCI benefits by using similar structures and functions as traditional storage.

Money,money

With Nutanix you could actually save some money by using Hybrid (SSD+HDD) solution as with Nutanix Capacity- Based- Licensing (CBL) you don’t have to pay license fees for HDD capacity. Only flash capacity is licensed. Additionally the hardware portion of the storage capacity is much cheaper with HDD drives.

Why aren’t hybrid configurations used more often?

Sometimes it is a matter of convenience, it is easier to go with all flash solution.

Apples-to-Apples

Sometimes it is because of competitive situation. Your competitor is offering All Flash and to compare “apples-to-apples” you are forced to offer All Flash as well, even though your hybrid solution might be enough to serve the workloads at acceptable latency level and the competitor has use All Flash in order to fulfill the performance requirements.

Too specific RFP requirements

Sometimes RFPs are written in too specific terms. Instead of stating the business requirement, it is stated that you can only use SSD drives. Does it really matter what type of disks are used, if the business requirement can be fulfilled regardless of disk types used? One vendor could fulfill the business requirement using cheaper hybrid solution, but can’t because using HDD drives is not allowed.

Risks related to performance

With All Flash solution you are almost guaranteed to have better latency, with hybrid solution you might get close to the same latency level in some situations and in some situations your latency might be noticeably higher or worse.

All Flash might be also less risky, if the performance characteristics of your workloads change, you are less likely to hit performance issues with all SSD solutions in such situations. Hybrid solutions are less adaptable changes in performance requirements.

Extra sizing efforts required

With Hybrid setups you have to make more careful sizing.

“Working Set” with Hybrid setups

In order to keep the performance at acceptable level with hybrid setups, you have to size your “hot tier” correctly. The idea there is that all the “active” or “hot data”, also known as “working set” would be served from SSD and the HDD drives would only store “cold”, latency insensitive data.

One of the problems related to “working set” is that the size of the “working set” might be hard to find. Also “working set” size is not static, it might change over time, during different times of a day and so on.

If you are already on Nutanix, it is much easier as Nutanix keeps track of “working set” sizes, which can be seen from “Prism” user interface. With other platforms you might have to use third party tools to dig this information or even manually pull the data from the storage system and make some excel magic.

Let’s say you have done your “Working set” calculations and have a number for it.

If your working set is close to maybe 60%-100%, no need to entertain the idea of hybrid solutions, go with All Flash SSD solution or even All Flash NVMe solution. However in my experience you rarely find such high “working set” size in real life environments, typically the “working set” size is much lower and thus the “working set” might be also served with hybrid setup.

Calculating the SSD/HDD ratio

So let’s do some calculations with different NX platforms to see which kind of “working set” ranges they would be able to support.

You can’t just divide the raw SSD capacity with raw HDD capacity to get “room for working set” number.

Internal overheads

With Nutanix there are various I/O related “components” or “services” that reside on the SSD tier and the system reserves some capacity for these components. Additionally the system takes into account replication factor, which also consumes capacity as well reserves some capacity for failover in case one of the nodes in the cluster goes down.

Are these overheads a negative thing? Nope, they are there for good reasons and serve a valid purpose.

While all of the above is possible to calculate manually, the good news is that you won’t have to. I am using Nutanix capacity calculator, which does most of the heavy lifting.

Number of nodes used

To keep things simple I will use four node cluster with each of the NX platforms.

Capacity Calculator parameters

I am using following settings with the Nutanix capacity calculator:

• Replication Factor = RF2
• Storage Efficiency = None
• Failover plan = Standard (N+1)

Limiting the number of possible combinations

I am not going to make calculations for every possible combination of SSD and HDD sizes

• Pairing the smallest supported SSD with the largest supported HDD to get the lowest possible SSD/HDD ratio
• Pairing the largest supported SSD with the smallest supported HDD to get the highest possible SSD/HDD ratio

G7 generation only

Also while G6 generation is still supported, I am doing the calculations for the current G7 generation only.

Fully populated nodes only

To keep the number of permutations low, let’s fully populate the nodes with SSD/HDD drives. Some platforms support partially populated nodes as well, but let’s keep it simple.

Silly results?

With these parameters, I suspect that we will get some silly results with some of the platforms, but why not, it is allowed to have some fun while playing with Nutanix configurations… I might make additional calculations for some setups, if some of the high or low ratios end up being too silly.

Extent Store

From the capacity calculator you can get available “Extent Store” capacity for both SSD and HDD.

What is “Extent Store”?

It is the space where you can store data blocks persistently, after the internal overheads have been taken away. By dividing “SSD Extent Store” capacity with “HDD Extent Store” capacity you get a figure which would be roughly the size of “working set” that the setup can sustain while keeping the latency near All Flash level.

Why roughly?

One of the internal overheads using SSD is feature called “Oplog”, which acts similarly to battery backed NVRAM found in some traditional storage solutions, serving as a staging point, responding to individual write I/O operations at low (SSD) latency level and then coalescing the individual writes together and writing them to persistent (SSD or HDD) storage in more efficient manner as big sequential operations. Sort of cache, but since it is on SSD, you won’t lose data in case of power failure, like you would with pure (non-NV)RAM based caches.

So, not all of the internal overheads using SSD capacity are a pure loss in performance, with “Oplog” you will be able to support some additional slower “cold tier” storage while having a reasonable latency.

Stop yapping, let’s get started already… Okay, okay

Nutanix NX G7 Hybrid platform SSD/HDD ratios

4xNX-1175S-G7 Hybrid

Supported disk sizes & number of disks from the G7 spec sheet, page 8

Lowest SSD/HDD ratio, 4xNX-1175S-G7 Hybrid, 2×1.92TB SSD + 2x12TB HDD / node

• SSD Extent Store Capacity: 2,94 TiB
• HDD Extent Store Capacity: 28,89 TiB
• SSD / HDD Extent Store Ratio: (2,94 TiB / 28,89 TiB) * 100 ~ 10,17 %

Reasonable SSD/HDD ratio, 4xNX-1175S-G7 Hybrid, 2×3.84TB SSD + 2x8TB HDD / node

• SSD Extent Store Capacity: 8,08 TiB
• HDD Extent Store Capacity: 19,26 TiB
• SSD / HDD Extent Store Ratio: ( 8,08 TiB  / 19,26 TiB ) * 100 ~ 41,95 %

Highest SSD/HDD ratio, 4xNX-1175S-G7 Hybrid, 2×7.68TB SSD + 2x4TB HDD / node

• SSD Extent Store Capacity: 18,05 TiB
• HDD Extent Store Capacity: 9,63 TiB
• SSD / HDD Extent Store Ratio: (18,05 TiB / 9,63 TiB) * 100 ~ 187,43 %

Notes about Hybrid NX-1175S-G7 Hybrid SSD/HDD ratios

• The lowest ratio is okay
  •  Finding a ~10% “working set” size in real life environments is quite normal
• The highest ratio is just ridiculous
  • As would be any ratio over 100%
  • It is more like a All Flash solution backed by some extra HDD capacity
  • Most of the time not worth the trouble
  • Just go with All Flash
  • or
  • Select one of the disk size combinations offering more reasonable ratio

1xNX-1465S-G7 Hybrid

Supported disk sizes & number of disks from the G7 spec sheet, page 7

Lowest SSD/HDD ratio, 1xNX-1465-G7, 1×1.92TB SSD + 2x12TB HDD / node

• SSD Extent Store Capacity: 0,94 TiB
• HDD Extent Store Capacity: 28,89 TiB
• SSD / HDD Extent Store Ratio: (0,94 TiB / 28,89 TiB) * 100 ~ 3,25 %

Reasonable SSD/HDD ratio, 1xNX-1465-G7, 1×7.68TB SSD + 2x12TB HDD / node

• SSD Extent Store Capacity: 7,9 TiB
• HDD Extent Store Capacity: 28,89 TiB
• SSD / HDD Extent Store Ratio: (7,9 TiB / 28,89 TiB) *  ~ 27,35 %

Highest SSD/HDD ratio, 1xNX-1465-G7, 1×7.68TB SSD + 2x4TB HDD / node

• SSD Extent Store Capacity: 8,55 TiB
• HDD Extent Store Capacity: 9,63 TiB
• SSD / HDD Extent Store Ratio: (8,55 TiB / 9,63 TiB) * 100 ~ 88,78 %

Notes about Hybrid NX-1465-G7 Hybrid SSD/HDD ratios

• The lowest ratio is okay, as long as there is some positive number it is okay
  • It just means that the “working set” should be low compared to “cold” tier
• The highest ratio is very close to 100%
  • It is more like a All Flash solution backed by some extra HDD capacity
  • In most cases with this high “working set” I would probably go with All Flash
  • But still a feasible solution
    • If your “working set” is high enough
    • and/or
    • You want to store lots of snapshots (on cheaper media)
    • and/or
    • You want to “pin” some workloads to “SSD tier” permanently
  • Or
  • Go with SSD + HDD mixes that offer more reasonable SSD/HDD ratio

4xNX-3155G-G7 Hybrid

Supported disk sizes & number of disks from the G7 spec sheet, page 17

Lowest SSD/HDD ratio, 4xNX-3155G-G7, 2×1.92TB SSD + 4x12TB HDD / node

• SSD Extent Store Capacity: 1,96 TiB
• HDD Extent Store Capacity: 57,79 TiB
• SSD / HDD Extent Store Ratio: (1,96 TiB / 57,79 TiB) * 100 ~ 3,39 %

Reasonable SSD/HDD ratio, 4xNX-3155G-G7, 2×7.68TB SSD + 4x12TB HDD / node

• SSD Extent Store Capacity: 16,41 TiB
• HDD Extent Store Capacity: 57,79 TiB
• SSD / HDD Extent Store Ratio: (16,41 TiB / 57,79 TiB) * 100 ~ 28,4 %

Highest SSD/HDD ratio, 4xNX-3155G-G7, 2×7.68TB SSD + 4x6TB HDD / node

• SSD Extent Store Capacity: 17,4 TiB
• HDD Extent Store Capacity: 28,89 TiB
• SSD / HDD Extent Store Ratio: (17,4 TiB / 28,89 TiB) * 100 ~ 60,22 %

Notes about Hybrid NX-3155G-G7 Hybrid SSD/HDD ratios

• The lowest ratio is okay, as long as there is some positive number it is okay, it just means that the “working set” should be low compared to “cold” tier
• The highest ratio is fairly high
  • In most cases with this high “working set” I would probably go with All Flash
  • But still a feasible solution
    • If your “working set” is high enough
    • and/or
    • You want to store lots of snapshots (on cheaper media)
    • and/or
    • You want to “pin” some workloads to “SSD tier” permanently
  • Or
  • Go with SSD + HDD mixes that offer more reasonable SSD/HDD ratio

1xNX-3460-G7 Hybrid

Supported disk sizes & number of disks from the G7 spec sheet, page 17

Lowest SSD/HDD ratio, 1xNX-3460-G7, 2×1.92TB SSD + 4x2TB HDD / node

• SSD Extent Store Capacity: 3,59 TiB
• HDD Extent Store Capacity: 9,6 TiB
• SSD / HDD Extent Store Ratio: (3,59 TiB / 9,6 TiB) * 100 ~ 37,39 %

Highest SSD/HDD ratio, 1xNX-3460-G7, 2×7.68TB SSD + 4x2TB HDD / node

• SSD Extent Store Capacity: 18,05 TiB
• HDD Extent Store Capacity: 9,6 TiB
• SSD / HDD Extent Store Ratio: (18,05 TiB / 9,6 TiB) * 100 ~ 188,02 %

Notes about Hybrid NX-3460-G7 Hybrid SSD/HDD ratios

• The lowest ratio is okay
  • Though fairly high
  • This would be probably the only NX-3460-G7 Hybrid setup that I would actually recommend
    • As using larger SSDs with 2TB HDD would move the setup to All Flash territory
• The highest ratio is just ridiculous
  • As would be any ratio over 100%
  • It is more like a All Flash solution backed by some extra HDD capacity
  • Just go with All Flash
    • You don’t gain much by using 2TB drives
    • Better to save the disk slots for possible capacity upgrades
• NX-3060-G7 is the only G7 platform with 2.5″ drives that supports hybrid configurations

4xNX-8150-G7 Hybrid

• Hybrid = N/A
• NX-8150-G7 supports only SSD/SSD SED/NVMe 2.5″ drives
• Probably because the largest available 2TB HDD combined with SSD drives in hybrid configuration with NX-8150-G7 would not make much sense in most of the use cases
• If you need “cold tier” capacity, better to use 3.5″ platforms where you get much more “cold tier” capacity with minimal price increase

4xNX-8155-G7 Hybrid

Supported disk sizes & number of disks from the G7 spec sheet, page 41

Limits for the disk size combinations used with hybrid, G7 spec sheet, page 35

Lowest SSD/HDD ratio, 4×8155-G7, 2×1.92TB SSD + 10x8TB HDD / node

• SSD Extent Store Capacity: 0,92 TiB TiB
• HDD Extent Store Capacity: 96,32 TiB TiB
• SSD / HDD Extent Store Ratio: (0,92 TiB / 96,32 TiB) * 100 ~ 0,96 %

Second lowest SSD/HDD ratio, 4×8155-G7, 2×1.92TB SSD + 10x6TB HDD / node

• SSD Extent Store Capacity: 1,54 TiB TiB
• HDD Extent Store Capacity: 72,24 TiB TiB
• SSD / HDD Extent Store Ratio: (1,54 TiB / 72,24 TiB) * 100 ~ 2,13 %

Third lowest SSD/HDD ratio, NX-8155-G7, 2×3.84TB SSD + 10x12TB HDD / node

• SSD Extent Store Capacity: 3,83 TiB
• HDD Extent Store Capacity: 144,47 TiB
• SSD / HDD Extent Store Ratio: (3,83 TiB / 144,47 TiB) * 100 ~ 2,65 %

Reasonable SSD/HDD ratio, NX-8155-G7, 4×7.68TB SSD + 8x12TB HDD / node

• SSD Extent Store Capacity: 33,61 TiB
• HDD Extent Store Capacity: 115,58 TiB
• SSD / HDD Extent Store Ratio: (33,61 TiB / 115,58 TiB TiB) * 100 ~ 29,07 %

Highest SSD/HDD ratio, NX-8155-G7, 4×7.68TB SSD + 8x6TB HDD / node

• SSD Extent Store Capacity: 35,58 TiB
• HDD Extent Store Capacity: 57,79 TiB
• SSD / HDD Extent Store Ratio: (35,58 TiB / 57,79 TiB) * 100 ~ 61,57 %

Notes about Hybrid NX-8155-G7 Hybrid SSD/HDD ratios

• NX-8155-G7 is the most versatile NX platform for hybrid setups
  • Option to use 2xSSD + 4xHDD/6xHDD/8xHDD/10xHDD
  • Option to use 4xSSD + 8xSSD
  • Various SSD and HDD sizes supported
  • Possibility to fulfill many “working set” and capacity requirements at the same time
• The lowest ratio is okayish
  • as long as there is some positive number it is okay
  • It just means that in this case the “working set” should be really, really small compared to “cold” tier
  • Maybe reasonable candidate for some secondary storage type of workload
• Most likely the first reasonable low-end setup would be
  • Setup (2×1.92TB SSD + 10x6TB HDD) with second lowest ratio
  • or
  • Setup  (2×3.84TB+10x12TB HDD) with third lowest ratio
• The highest ratio with( 4×7.68TB SSD +8x6TB HDD) is again close to All Flash territory
• Maybe select something in the middle of the pack
  • Like 4×7.68TB SSD + 8x12TB HDD or 4×7.68TB SSD + 8x8TB HDD

4xNX-8170-G7 Hybrid

• Hybrid = N/A
• NX-8150-G7 supports only NVMe drives
• As seen with other 2.5″ platforms, using hybrid configurations do not make much sense most of the time
• However I would like to see this platform support also
  • SSD drives
  • SSD+NVMe combos
  • or
  • At least lesser number of NVMe drives
  • Similar to combinations that were supported with NX-3170-G6
  • IMHO with G7 generation NX appliances there is a 1U sized hole in the Nutanix lineup 🙂
  • Maybe some glorious day..

2xNX-8235-G7 Hybrid

Supported disk sizes & number of disks from the G7 spec sheet, page 28

Lowest SSD/HDD ratio, 2xNX-8235-G7, 2×1.92TB SSD + 4x12TB HDD / node

• SSD Extent Store Capacity: 1,96 TiB
• HDD Extent Store Capacity: 57,79 TiB
• SSD / HDD Extent Store Ratio: (1,96 TiB / 57,79 TiB) * 100 ~ 3,39 %

Reasonable SSD/HDD ratio, 2xNX-8235-G7, 2×7.68TB SSD + 4x12TB HDD / node

• SSD Extent Store Capacity: 16,41 TiB
• HDD Extent Store Capacity: 57,79 TiB
• SSD / HDD Extent Store Ratio: (16,41 TiB/ 57,79 TiB) * 100 ~ 28,4 %

Highest SSD/HDD ratio, 2xNX-8235-G7, 2×7.68TB SSD + 4x6TB HDD / node

• SSD Extent Store Capacity: 17,4 TiB
• HDD Extent Store Capacity: 28,89 TiB
• SSD / HDD Extent Store Ratio: (17,4 TiB / 28,89 TiB) * 100 ~ 60,22 %

Notes about Hybrid NX-8235-G7 Hybrid SSD/HDD ratios

• The lowest ratio is okay
  • as long as there is some positive number it is okay
  • it just means that the “working set” should be low compared to “cold” tier
• The highest ratio is fairly high
  • In most cases with this high “Working Set” I would probably go with All Flash
  • But still a feasible solution
    • If your “working set” is high enough
    • and/or
    • You want to store lots of snapshots (on cheaper media)
    • and/or
    • You want to “pin” some workload to “SSD tier” permanently
  • or
  • Select some disk size combinations which would lead to more reasonable ratio
• You might have noticed that NX-3155G-G7 and NX-8035-G7 offer the same capacities and “working sets”
  • Unless you need to use GPUs, I would go with NX-8035-G7 platform
  • as you can save some rack space with NX-8035-G7
    • With NX-3155G-G7 you will get 1 node per 2U
    • With NX-8035-G7 you will get 1 or 2 node(s) per 2U

Conclusion about Nutanix Hybrid (SSD+HDD) working set ranges

• As seen in the examples above, the Nutanix NX appliance hybrid lineup can support working set ranges from near 0% to all the way up to 100% or more
• It also seems that in most “hybrid” cases it makes sense to select platform with 3.5″ drives as there are very few cases where hybrid configurations with 2.5″ make sense
• Using the largest SSDs with the smallest HDDs rarely makes sense, typically in such cases a All Flash solution would be more viable option
• Typically the most suitable NX hybrid platforms are the ones with “middle-of-the-pack” SSD and HDD combinations
  • not too small to sustain reasonable “working sets”
  • and
  • not too large to run into All Flash territory
• Nutanix Hybrid platforms have technological advantages when
  • Compared to traditional hybrid storage platforms and other HCI hybrid platforms
  • Making Nutanix Hybrid platform a viable option
  • When traditional hybrid storage platforms or other HCI hybrid platforms might NOT be viable options
• Obviously that is not the complete truth
  • You have to consider the usable capacity requirement as well
    • You won’t always find combo that will match both
      • “working set” requirement
      • and
      • capacity requirement
    • Especially with small capacity requirements
    • With larger capacity requirements you can
      • Use more versatile NX-8155-G7 platform
      • or
      • mix and match different node types to better fulfill both capacity and “working set” requirements
    • But then again extra HDD capacity is fairly cheap
      • if the hybrid setup otherwise fulfills the business requirements, hybrid is still a viable option
      • Disk space is like money, you never have too much 🙂
  • Storage is only part of the equation
    • You have to also size the compute part, i.e select CPUs
  • Typically Nutanix sizings are done with Nutanix sizer tool
    • Workload parameters can be imported into the sizer tool
    • and
    • the sizer will size both your compute and storage at the same time and will make suggestions for suitable NX appliance candidates to be used to serve the workloads
• All in all hybrid is still valid option with Nutanix platforms as long as you are willing to
  • Put in more effort in sizing
  • Accept some risks related to performance and predicting future
  • In order to get more price friendly solution