Nutanix Spec Sheet Errata

The official Nutanix G6 & G7 generation NX appliance spec sheets are frequently updated and are getting better with every iteration. Unfortunately there are still issues with them. Some information in the NX appliance spec sheets is confusing and showing conflicting or wrong information and should be corrected. Some of the aspects of NX appliances are documented in fairly shallow manner and could benefit from more comprehensive documentation.

 

 

While building the DPTPB Nutanix Dynamic Visio shapes, the issues found in the spec sheets caused some headaches and extra work. Sometimes these issues went unnoticed and resulted in bugs with the dynamic shapes, which in turn required additional development cycles to address the issues. To avoid making the same mistakes again, I decided to write them down.

For average Nutanix end-user most of these issues are fairly minor, but finding errors in the spec sheets makes you wonder what other issues are there, still not found and undocumented.

I’ve notified Nutanix about few of these issues, but for some reason or other they haven’t (yet) updated the spec sheets. Hopefully this post will help Nutanix to correct some of the issues…

This blog post is based on the information found  in G6 and G7 generation single-node and multi-node spec sheets:

  • System-Specs-G6-Multinode, Feb 11, 2020
  • System-Specs-G6-Single-Node, Jan 25, 2020
  • System-Specs-G7-Multinode, Feb 26, 2020
  • System-Specs-G7-Single-Node, Feb 25, 2020

These spec sheets are available on the Nutanix support portal (login required).

Generic Issues

Wrong form factor NICs and PCIe Expansions Slots

Usually NICs and PCIe expansion slots come in two form factors, low-profile (LP) or full-height (FH). As NICs are installed into the PCIe expansion slots, naturally the form factor of the PCIe slot have to match with the form factor of the NIC or otherwise the NIC will not fit into the PCIe slot or will not fully cover the opening in the PCIe slot.

PCIe slot form factors

Unfortunately it is seems that Nutanix has overlooked this piece of information and states (wrongly) in the G6 & G7 spec sheets that ALL PCIe expansion slots are low-profile form factor.

Example 1: Expansion Slot for NX-1175S-G7, G7 spec sheet, page 9

nx-1175s-g7_expansion_slot

Example 2: Expansion Slot for NX-3155G-G7, G7 spec sheet, page 18

nx-3155G-g7_expansion_slot

Example 3: Expansion Slot for NX-8150-G7, G7 spec sheet, page 30

nx-8150-g7_expansion_slot

Back panel examples & PCIe expansion slots for NX appliances

By looking at “Back panel” examples one can clearly see that the PCIe expansions slots are not equally wide, meaning that the narrower ones are low-profile and the wider ones are full-height.

Example 4: Back panel NX-8150-G7, G7 spec sheet, page 26

NX-8150-G7_back_panel

Example 4: Notes

  • You can clearly see that
    • NIC1 is narrower than the rest
      • i.e it is low-profile form factor
    • NIC2 & NIC3 are wider than NIC1
      • i.e they are full-height form factor

The wrong usage of form factors in the spec sheets can be confirmed also by looking at Supermicro documentation for similar hardware as NX-8150-G7 (NX appliance hardware is supplied by Supermicro)

Example 5: Expansion slot form factors in Supermicro documentation

supermicro_expansion_slots

Example 5: Notes

  • Supermicro documentation confirms
  • NIC1/Slot4 is LP (low-profile) form factor
  • NIC2/Slot5 and NIC3/Slot7 are  FH (full-height) form factor

NIC form factors

No specification of NIC form factor in written format in the spec sheets, however in supporting images NICs are shown in both FH and LP form factors.

Example 6: NICs in both LP and FH form factor, G6 / G7 spec sheets

nx-8155-g7_sfp_qsfp_image

Example 6: Notes

  • First three NICs are Low-profile (LP) and have low-profile metal brackets
  • The last NIC (25GbE) is Full-Height (FH) and has full-height metal bracket
  • Typically you can acquire different form factor variants for NICs
    • The actual NIC hardware doesn’t usually change
    • Only different metal bracket is used

The actual NIC and PCIe slot form factors in use with NX appliances

Depending on NX model, it may be supported to use only LP form factor PCIe slots, only FH form factor PCIe slots or both LP and FH PCIe slots.

To make matters more confusing, some of the NX models (NX-3155/5155/8150/8155) share the same back panel and look exactly the same when viewed from behind without any NICs, but some of those models only support using FH PCIe slots, while others support using both LP and FH PCIe slots. Again not stated in the spec sheets in written form, you have to figure this out by looking at some of the images in the spec sheets.

  • NX Models with only LP PCIe slots in use
    • NX-1065-G6/G7
    • NX-3060-G6/G7
  • NX Models with only FH PCIe slots in use
    • NX-8035-G6/G7
    • NX-5155-G6
    • NX-8155-G6/G7
  • NX Models with both LP and FH PCIe slots in use
    • NX-1175S-G6/G7
    • NX-3155G-G6/G7
    • NX-3170-G6
    • NX-8150-G7
    • NX-8170-G7

NIC options

Unfortunately “NIC options” images used in the spec sheets show most of the time confusing information. Showing also FH NICs when the NX platform only supports LP NICs and the other way around showing also LP NICs when the NX platform only supports FH NICs. It seems that they are basically using a variation of the same image to document the NIC options, as seen in examples below.

Example 1: NX-3060-G7 NIC options, G7 spec sheet, page 14

nic_options_nx3060g7

  • NX-3060-G7 only supports NICs in Low-Profile (LP) form factor
  • Yet:
    • Quad-Port 10GbE spf+ is shown with Full-Height (FH) form factor
    • Dual-Port 25GbE spf28 is shown with Full-Height (FH) form factor

Example 2: NX-8035-G7 NIC options, G7 spec sheet, page 25

nic_options_nx8035g7

  • NX-8035-G7 only supports NICs in Full-Height (FH) form factor
  • Yet:
    • Dual-Port 10G BaseT is shown in LP form factor
    • Dual-Port 10GbE spf+ is shown in LP form factor
    • Dual-Port 40Gbe Qsfp28 is shown in LP form factor
  • Basically the same image as used with NX-3060-G7

Conclusion of wrong form factor NICs:

Wrong form factor NICs or NIC brackets are not usually problem with factory-installed units. Naturally when NX appliances are assembled at the factory, a correct form factor for NIC is selected or otherwise it wouldn’t fit in the designated PCIe slot…

But assuming wrong form factor might be a problem when NICs are ordered separately, either for expanding NX appliances with additional NICs or upgrading the existing NICs. With some NX platforms there are both LP and FH form factor PCIe slots in use and you might end up ordering wrong form-factor NICs.

Most of the time the NIC itself does not change when used with different form factors, but the metal bracket is different. I am not sure if separately ordered factory supplied NICs come with user changeable (LP and FH) form factor brackets or if they are sold with only a specific form factor bracket (LP or FH). If the latter, buyer be aware…

As I don’t have access to Nutanix Salesforce configurator any more, I can’t check available NIC options and their form factors, comments regarding this are welcome…

Nearly no NIC orientation information available

Once again one of those things that is not documented in written format, you can figure this out by looking at some of the supporting images found in the spec sheets.

When using multiple NICs with most NX platforms NICs run in two orientations.

  • 0° orientation
    • The PCIe connector is on the left side of the NIC to match the motherboard PCIe connector (when looking at the back panel )
  • 180° orientation
    • The PCIe connector is on the right side of the NIC to match the motherboard PCIe connector (when looking at the back panel )
  • Since there are no 0° and 180° variants of NICs available in real life, you make a 180° orientation NIC  from 0° orientation NIC by flipping the NIC by 180° when installing the NIC.
  • The orientation names “0°” and “180°” are just names, not something that are widely used or somehow “official” terms, I just picked them as they make sense and makes it easier to understand the rest of this blog post.

Example: NX-8155-G7 blow up image, G7 spec sheet, page 39

nx-8155-g7_blow_up

  • Since we are now looking the NX appliance from the front
    • left and right are swapped
    • 0° orientation
      • The PCIe connector is on the right side of the NIC to match the motherboard PCIe connector
    • 180° orientation
      • The PCIe connector is on the left side of the NIC to match the motherboard PCIe connector (when looking at the back panel )

NX Models with just one NIC orientation

  • NX-8035-Gx
    • All NICs are installed in 0° orientation

NX models with both 0° and 180° orientation NICs

  • All other NX models when using more than one NIC
  • there are no hard rules that NIC1 is always in 0°orientation or 180°
  • the orientation of specific NICn varies based on NX platform used

Example 1: NX-8035-G7 NIC orientation, G7 spec sheet, page 24

nx-8035-g7_nic_orientation

  • NICs are in the same orientation
  • All NICs are in FH form factor

Example 2: NX-8155-G7 NIC orientation, G7 spec sheet, page 37

nx-8155-g7-nic-orientation

 

  • NIC1 (on the left) is in different orientation
  • than
  • NIC2 (on the right) and NIC3 (in the middle)
  • All NICs are in FH form factor

Example 3: NX-8150-G7 NIC orientation, G7 spec sheet, page 26

nx-8150-g7_nic_orientation

  • NIC1 (on the left) is in different orientation
    • Little bit hard to figure this out, see the next example 3.1
  • than
  • NIC2 & NIC 3 (on the right)
  • While the back plane view for NX-8150-G7 and NX-8155-G7 is exactly the same when NOT populated with NICs, they have different PCIe slot usage order and assigned PCIe slots and they look different when populated with NICs
  • NX-8155-G7 uses only FH form factor NICs, while NX-8150-G7 uses both LP and FH form factor NICs

Example 3.1

nx-8150-g7_nic_orientation_DPTPB

  • Here is the same example as above using my DPTPB Dynamic shapes
  • Oops, it seems that Nutanix has used “Quad -port 10Gbe spf+” adapters in their example, which are not supported with NX-8150-G7 platform according to the spec sheet, maybe the image in the spec sheet should be updated with supported NICs
  • The automatic placement of NICs with DPTPB Dynamic shapes does not allow using unsupported NICs, but you can override the automation by manually placing also unsupported NICs in the PCI slots
  • The image used in the previous example is fit fuzzy it is fairly hard to see the orientation of the NICs
    • You can clearly see that NIC2 & NIC3 are in the same orientation
    • But what about NIC1?
      • If you look close enough, you can see that the status lights are on top of the ports with NIC1 and below the ports with NIC2 & NIC3
      • Meaning that NIC1 has different orientation than NIC2 & NIC3

So what is the actual problem with different NIC orientations and lack of documentation regarding this?

  • With 0° orientation NIC, port 1 is typically the LEFTMOST port
  • Once you flip the NIC by 180° when installing the NIC into 180° orientation PCIe slot
    • the port numbers flip as well
  • And with 180° orientation NIC, port 1 is typically the RIGHTMOST port
  • Or it might be the other way around
    • With 0° orientation port 1 is the RIGHTMOST port
    • but the important part is that the port numbers flip when the NIC is flipped by 180° when installing it to 180° orientation PCIe slot
    • And port 1 becomes the LEFTMOST port in 180° orientation
  • Using port numbering in the spec sheets also for NIC ports would solve this issue
  • For the rest of the documentation I am assuming that
    • With 0° orientation NIC, port 1 is the LEFTMOST port
    • With 0° orientation NIC, port 2 is the RIGHTMOST port
    • With 180° orientation NIC, port 1 is the RIGHTMOST port
    • With 180° orientation NIC, port 2 is the LEFTMOST port

Example 4: NX-8150-G7 with 2x10GbE sfp+ NIC / node, ports numbered

NX-8150-G7_2xNIC_port_numbers

  • With NIC1
    • LP form factor NIC on the left
    • port 1 is the rightmost port
    • port 2 is the leftmost port
    • NIC1 is in 180° orientation
  • With NIC2
    • FH form factor NIC on the right
    • port 1 is the LEFTmost port
    • port 2 is the RIGHTmost port
    • NIC2 is in 0° orientation
  • (Image made with DPTPB Dynamic Shapes, not available in the spec sheets)

Example 5: NX-8150-G7 with 2x10GbE sfp+ NIC / node

NX-8150-G7_2xNIC_v2

  • If this was factory installed unit, how would you know
    • which NIC port is which port number?
      • With some of the actual physical NICs the ports are numbered
      • But sometimes are not numbered
      • And sometimes it is hard to see port numbers in dimly lit data centers
    • what is the orientation of the NIC (0° or 180°)?
      • You would have to open the covers and find out where the motherboard PCI slot connectors are
      • With FH NICs ports are usually closer to PCI connector
        • i.e if the ports are on the left, it is a 0° NIC
        • i.e if the ports are on the right, it is a 180° NIC
      • With LP NICs is not so easy to figure out
  • (Image made with DPTPB Dynamic Shapes, not available in the spec sheets)

Example 6: Messy cabling

NX-8150-G7_2xNIC_port_numbers_messy_cabling

  • What if you
    • designed your system prior to having the system at your disposal?
    • and there was no documentation of NIC port numbers?
    • and you had no idea that the NIC orientation and port numbering might change when the NIC is installed?
    • and documented ports used for cabling based on those assumptions?
  • Well, you might mess up your cabling, while assuming that the left most NIC port is always port 1
  • And run cable from NIC1 port 2 to switch 1
  • when you were supposed to run NIC1 port 1 to switch 1
  • (Image made with DPTPB Dynamic Shapes, not available in the spec sheets)

Conclusion of Nearly no NIC orientation information available

With two possible NIC orientations (and resulting two NIC port number orientations), it is possible that you might run cables from nodes to switches using different NIC ports than you thought you were using.

Will it be a real problem in real life? Maybe, maybe not.

Most likely when everything is working and you have multiple links with load balancing, you would probably not notice that the port numbers are reverted with some of the NICs.

But consider a situation where:

  • you have only two links
  • one of your links goes down
  • to troubleshoot you are asked to replace a cable

Yes, there are status lights with NICs, but the meaning of NIC status lights vary from NIC model to NIC model (well documented in the spec sheets) and you might misinterpret the status light or might just rely on the fact that port 1 is usually the leftmost port. Since your data center is dimly lit, you also missed the stamped port number on the NIC.

As a result you might mistakenly:

  • Pull the cable from port 1
  • When you were supposed to pull the cable from port 2.
  • Pulling the cable from the only working link
  • Isolating your node totally from the network and the cluster….
  • Causing HA to kick in and move VMs to nodes that are still able to communicate
  • Your cluster might not be able to sustain other node loss…

It wouldn’t hurt to document the NIC orientations or at least NIC port numbers in the spec sheets. Some of the motherboard ports are already numbered, why not NIC ports as well?

Wrong NIC transceiver types used, sfp+ vs sfp28, qsfp+ vs qsfp28

While not a major issue, once again makes the documentation look sloppy…

Let’s recap correct usage of terms: sfp+/sfp28/qsfp+/qsfp28

Unfortunately the Nutanix spec sheets use these terms in various and very confusing combinations.

Example1: NX-8155-G6 ports, G6 spec sheet text, page 52

nx-8155-g6_sfp_qsfp

Example 1: Notes

  • 25 GbE should be SFP28yes, you can connect a sfp+ transceiver to sfp28 port, but it only operates at spf+ speed, 10GbE
  • 40 GbE should be QSFP+
    • The text used in the example above just plainly wrong

Example2: NX-8155-G7 ports, G7 spec sheet text, pages 37-38

  • Maybe G7 documentation is more accurate?

nx-8155-g7_sfp_qsfp

Example 2: Notes

  • Nope, the same errors with 25 Gbe and 40Gbe

Example 3: NX-8155-G7 ports, G7 spec sheet image, page 38

  • Maybe the image just BELOW the text is better

nx-8155-g7_sfp_qsfp_image

Example 3: Notes

  • Yes, they got it right in the image
  • But, somehow failed to fix the text which sits JUST ON TOP of this image…
  • As a side note, all NX-8155-G7 NICs are FH form factor, yet 3/4 NICs shown are LP form factor…

Example 4: NX-8035-G7 ports, G7 spec sheet text, page 25

nx8035g7_sfp_qsfp

Example 4: Notes

  • Nope, 40GbE is wrong
    • it should be qsfp+, not qsfp28

Example 5: NX-8035-G7 ports, G7 spec sheet image, page 25

nx8035g7_sfp_qsfp_image

Example 5: Notes

  • Nope, 40GbE is still wrong, should be qsfp+, instead of qsfp28
  • But they get brownie points for being consistent with both the text and the image

Example 6: NX-8150-G7, G7 spec sheet text, page 26

nx-8150-g7_sfp_qsfp

Example 6: Notes

  • Yihaa, no mistakes with sfp+/sfp28/qsfp+/qsfp28 in the text portion of the spec sheet
  • NX-8150-G7 all good?

Example 7: NX-8150-G7, G7 spec sheet image, page 26

nx-8150-g7_sfp_qsfp_image

Example 7: Notes

  • Yep, all good
  • Finally both the text and the image match and no mistakes
  • And NX-8150-G7 uses both LP and FH form factor NICs, so no mistake there either
  • There is hope…

Conclusion of Wrong NIC transceiver types used:

Typically using conflicting NIC connector types in the spec sheets is not problem with factory built NX appliances. If the transceivers are ordered with the appliance, the Salesforce configurator automatically picks matching transceivers for the NICs.

If you order the NICs with transceivers separately, you could get correct combinations. That is if your sales rep or SE has managed to use compatible part numbers as NICs and tranceivers are separate line items, if I recall correctly the Salesforce configurator does not force compatibility when ordering NICs and transceivers separately.

Also many customers order transceivers separately, sometimes from third party source.

Errors in spec sheets might cause confusion, if you are to order “25GbE sfp+” transceiver, which one would you get a “10Gbe spf+” or  a “25GbE sfp28” transceiver?

I would probably order “25GbE sfp28” transceiver, but somebody else might rely on “sfp+” bit and order “just spf+” transceivers instead of “25Gbe sfp28” transceivers.

Sure, I’ve made the same mistake multiple times and you can probably find examples of these mistakes in my blog, but anyhow I would like to see them fixed in the official spec sheets as using wrong terms leaves a sloppy impression and makes you question the quality of the rest of the documentation.

Supported disk sizes and disk combinations

Most of the time there is a valid technical or commercial reason for the difference in disk sizes and disk combinations supported for G6 and G7 generations.

With smaller drives it is quite natural to find smaller disk sizes that are supported with older G6 generation NX appliances and that these smaller size disks are not supported anymore with newer G7 generation NX appliances. A natural way of dropping smaller capacity drive support for newer hardware generations.

Typically using larger capacity drives requires extra qualification tests and it is normal for G7 generation NX appliances to support some larger disk sizes which are not supported with G6 generation NX appliances (without re-qualification?).

But then there are few cases where the differences found in the spec sheets are somewhat illogical or even questionable.

Example 1: NX-3060-G6 supports using 7.68TB SSDs with NVMe drives, but NX-3060-G7 does not

Example1.1 NX-3060-G6, Storage: SSD and NVMe, G6 spec sheet, page 17

nx-3060-g6_nvme_ssd_spec_sheet

Example1.2 NX-3060-G7, Storage: SSD and NVMe, G7 spec sheet, page 17

nx-3060-g7_nvme_ssd_spec_sheet

Yes, NX-3060-G6 also supports 960GB SSD, but NX-3060-G7 does not, I am alright with this one.

However, NX-3060-G6 supports 7.68TB SSD, while NX-3060-G7 does not, it doesn’t make much sense to me, if it was the other way around, maybe

Example 2: NX-5155-G6 and NX-8155-G6 share exactly the same disk specs, with only one exception.

With Hybrid SSD+HDD combinations NX-5155-G6 supports using 7.68TB SSD drives and NX-8155-G6 does not support using 7.68TB

Example 2.1 NX-5155-G6, Storage: Hybrid, G6 spec sheet, page 44

nx-5155-g6_hybrid_spec_sheet

Example 2.2 NX-8155-G6, Storage: Hybrid,G6 spec sheet, page 55

nx-8155-g6_hybrid_spec_sheet

Weird, most likely a typo, as I cannot imagine any valid reason for this difference. Maybe if 7.68TB was allowed with NX-8155-G6 (more powerful CPUs) and not allowed with NX-5155-G6 (less powerful CPUs), but not the other way around.

Example 3: With some NX platforms in Hybrid configurations small SSD drives are supported with large HDD drives, while with other platforms the same size combinations are not supported.

This might not be a problem at all, since there are other factors in play when Nutanix decides which SSD+HDD combos are supported. I’ve always wondered about some of the combinations, let’s see if there is a pattern or valid reason behind this or even some issues with these combinations.

For example with NX-8035-G7 it is supported to use 1.92TB SSD + 12TB HDD combo, while this combination is not supported with NX-8155-G7.

The difference is the number drives used, with NX-8035-G7 it is 2xSSD+4xHDD, while with NX-8155-G7 it is 2xSSD+10xHDD (or 4xSSD+8xHDD).

It comes down to keeping healthy ratio between SSD and HDD drives. While with quick math 2xSSD+4xHHD appears to give the same SSD/HDD ratio as 4xSSD+8xHDD, it is likely that Nutanix has decided the allowed combinations for NX-8155-G7 by using less favorable ratio with 2xSSD+10xHDD .

Let’s do some calculations to see what the actual usable SSD vs HDD ratios are for some of the NX platforms.

You can’t just divide the raw capacity of SSD drives with raw capacity of HDD drives, as there are some internal components that have to be factored in. Rather than doing all the calculations manually, let’s use Nutanix capacity calculator.

All calculations done with settings:

  • Replication factor: RF2
  • Storage Efficiency: None
  • Failover Plan: Standard (N+1)

Comparing available capacity after overheads. Using number from “Extent Store available” for both SSD and HDD = the space that is available to permanently store data blocks. You should have at least some “SSD extent store” capacity available, or you end up hitting HDDs with also active data and your Nutanix system will perform poorly because of higher i/o latency.

Example 3.1: 3xNX-8135-G7 with 2xSSD 1.92TB + 4xHDD 12TB

nx-8135-g7_2x192_4x12

  • SSD Extent Store available: 1,3 Tib
  • HDD Extent Store available: 38,53 TiB
  • SSD / HDD ratio: (1,3 TiB / 38,53 TiB) * 100 ~ 3,37%
  • supported configuration according to spec sheets
  • Looks okay to me, if your active data set is smallish when compared to the rest of data
  • (Image made with DPTPB Dynamic Shapes, not available in the spec sheets)

Example 3.2: 3xNX-8155-G7 with 4xSSD 1.92TB + 8xHDD 12TB

nx-8155-g7_4x192_8x12

  • SSD Extent Store available: 3,13 TiB
  • HDD Extent Store available: 77,05 TiB
  • SSD / HDD ratio: (3.13 TiB / 77,05 TiB) * 100 ~ 4,06%
  • Actually slightly better SSD/HDD ratio than you would expect by doing math in your head using just the number of SSD and HDD drives combined with raw capacities
    • Some of the internal overheads are fixed and are independent of the SSD size used. As result with larger total raw SSD capacity you get proportionally more capacity in the SSD Extent Store
  • Not actually supported according to spec sheets
    • Why?
    • See the next example
  • (Image made with DPTPB Dynamic Shapes, not available in the spec sheets)

Example 3.3: 3xNX-8155-G7 with 2xSSD 1.92TB + 10xHDD 12TB

nx-8155-g7_2x192_10x12

  • SSD Extent Store available: −0,2 TiB
    • The internal overheads would need to consume more SSD capacity than is available
    • Quite naturally such configuration is not supported…
    • And since using 1.92TB + 12TB could potentially cause issues, Nutanix has decided not to support this disk size combo with NX-8155-G7, even when some of those combinations with different number of drives look okay (to me)
  • (Image made with DPTPB Dynamic Shapes, not available in the spec sheets)

Example 3.4: 3xNX-8155-G6 with 2xSSD 0.96TB + 10xHDD 8TB

nx-8155-g6_2x960_10x8

  • According to spec sheets there are no disk size restrictions for SSD+HDD combinations with NX-8155-G6
  • SSD 960GB or 0.96TB is supported with NX-8155-G6
  • HDD 8TB is supported with NX-8155-G6
  • SSD Extent Store available: −0,59 TiB
    • The internal overheads would need to consume more SSD capacity than is available
    • Oops, maybe this combination should NOT be supported?
  • (Image made with DPTPB Dynamic Shapes, not available in the spec sheets)

Example: 3.5 3xNX-8135-G6 with 2xSSD 0.96TB + 4xHDD 12TB

nx-8135-g6_2x960_4x12

  • According to spec sheets there are no disk size restrictions for SSD+HDD combinations with NX-8135-G6
  • SSD 960GB or 0.96TB is supported with NX-8135-G6
  • HDD 12TB supported with NX-8135-G6
  • SSD Extent Store available: 0,07 TiB
  • HDD Extent Store available: 38,53 TiB
  • SSD / HDD ratio: (0,07 TiB / 38,53 TiB) * 100 ~ 0,18%
    • Very low SSD/HDD ratio
    • While technically possible combination, it is still questionable if such combination should be supported
  • (Image made with DPTPB Dynamic Shapes, not available in the spec sheets)

Conclusion of supported disk sizes and disk combinations:

According to the examples above there are some questionable hybrid configurations that are supported with G6 generation NX platforms, I didn’t find any such examples with G7 NX platforms, but then again I didn’t make the calculations for all the G7 NX appliance models. Update Mar 6, 2020: G7 is all good, for details: Nutanix Hybrid Selection Guide

I am not sure how Nutanix makes these calculations while deciding which combinations should be supported. The questionable combinations might be alright, if the calculations were made with “Failover Plan: Aggressive (N+0)”. I wouldn’t recommend doing so as you would jeopardize failover capabilities .

Also factoring in some “Storage Efficiency” ratios, like 1,5:1, might change the situation. Again something I don’t usually do when sizing systems for critical workloads, as “Storage Efficiency” savings are just estimates and if you apply too aggressive “Storage Efficiency” ratios, you might end up in deep poo-poo with your sizing.

Definitely a topic that Nutanix should revisit and correct possible issues in the spec sheets.

NX-3155G-Gx specific issues

GPU orientation

In the rear view diagram (below, G7 spec sheet page 14) NX-3155G-G7 is equipped with two Nvidia GPUs. According to the diagram both GPUs are using the same orientation (hint: the solid part of the bracket is on the left side with both GPUs).

backpanel-callouts-nx3155gg7

According to blow up diagram (below, G7 spec sheet, page 16 )

  • the PCIe connector for GPU1 (on the right) is on the left side of the GPU
  • the PCIe connector for GPU2 (on the left) is on the right side of the GPU
  • You can see the same GPUs as used in the previous diagram
    • but GPU2 has been flipped by 180° when compared to GPU1
    • Unlike in the previous example

riser-cards-nx3155gg7

Conclusion of GPU Orientation:

This is one of those issues which is mostly cosmetic. Issues in the spec sheet regarding GPU orientation doesn’t affect real life too much, if not all. When installing GPU2 with NX-3155G-Gx and trying to put GPU2 in the wrong orientation described in the spec sheet, you would very quickly realize that the PCIe connectors of the GPU2 and the PCIe slot in question wouldn’t line up and you would flip  GPU2 by 180° to get them to lineup.

Still, conflicting information found, I am pretty certain that the diagram on page 14 is wrong. GPU2 image should be flipped by 180° also in the diagram found on page 14.

Maybe something like this:

NX-3155G-G6_2xGPU_with_right_orientation

  • (Image made with DPTPB Dynamic Shapes, not available in the spec sheets)

 

PCIe Slot usage for GPUs

The same rear view diagram (below) as in the previous example. GPU1 on the right side, it takes the space of two PCIe slots. Likewise GPU2 on the left side, taking also space of two PCIe slots.

backpanel-callouts-nx3155gg7

The same blow up diagram (below) as in the previous example. (PCIe) slots are numbered for the slots on the right side. Counting from bottom to top, slot 1, slot 2, slot 3. According to the image a Double-Wide GPU (V100) goes to slot 3. This is somewhat confusing as you can clearly see that while the GPU takes also the space of slot 3, the GPU is actually connected to the fourth PCIe slot. Maybe it would be better to state that DW GPUs take the space of two PCIe slots and are connected to PCIe slot 3  in case of GPU1 and PCIe slot X in case of GPU2.

With G7 generation Nutanix added support for smaller form factor GPUs, Nvidia T4. As these GPUs are not so wide (or tall in horizontal orientation) as M10 or V100 GPUs, they only take the space of one PCIe slot to install. With these GPUs indicating that T4 should go to slot 3 is clearly a mistake. If T4 GPU is connected to the same PCIe slot as M10 or V100 GPUs, it is actually connected to the fourth PCIe slot from the bottom.

riser-cards-nx3155gg7

Conclusion of PCIe Slot usage of GPUs:

With double-wide GPUs like M10 or V100 it doesn’t really matter so much that the diagram indicates that slot3 is used with these GPUS as slot 4 is also occupied as well and could not be used for anything else.

That is if the GPU is actually connected to the top PCIe connector. If the GPU is connected to the third PCIe connector, the GPU would end up connected in the wrong PCIe slot and additionally the GPU would end up blocking also PCIe slot 2.

My bet is that this diagram was made prior to having support for smaller form factor T4 GPUs and later edited to have also T4 GPUs.

With T4 GPUs, if you follow the diagram and install the GPU to the third PCIe slot from the bottom, the GPU would end up in a wrong PCIe slot, occupying PCIe slot 3, when in fact the GPU should be placed to the fourth PCIe slot from bottom.

Maybe Nutanix should update the diagrams to avoid confusions.

Conclusion of Spec Sheet Issues

Some of the issues are just cosmetic with no real consequences, like wrong orientation for GPU2 with NX-3155-Gx.

As a single issue, most of the issues found in the spec sheets are fairly minor and would go unnoticed most of the time. However missing few these at the same time might lower the probability of successful implementation project or cause other problems with already running Nutanix cluster.

Unfortunately sometimes it is the small things that are overlooked and end up causing major issues.

I’ve personally witnessed quite a few times projects get delayed because of for example wrong NIC transceiver types ordered. While the actual transceivers are cheap(ish) to replace, the costly problem related to the wrong part is the time that it takes to sort out the situation. If the implementation project gets delayed by month, at minimum you are paying for support and facilities costs of the NX appliances while not gaining any business value from them. If you factor in people costs related to the delays, you might be unpleasantly surprised…

Finally, some of the supported G6 disk combinations are questionable and IMHO they should be removed from spec sheets.

Am I being mean to point out these issues?

I’d like quote one of the top technical guys at Nutanix with this one (Webscale Webster, in Twitter):

webscale_quote

  • Well said, I couldn’t have said it any better 🙂
  • Nope, Webscale Webster is NOT at fault here, I just liked his tweet
  • Happy to fix any mistakes I’ve made, just leave a comment
  • I would love to retract this post, because all of the issues have been fixed or counter argued…

Downloads

If using Visio to document Nutanix environments, it is highly advisable to use my unofficial DPTPB Nutanix dynamic Visio shapes.

I’ve addressed most of the spec sheet issues described in this post with my Visio shapes. For example with the built-in automation NICs are placed in correct form factor, to correct PCIe slots in correct orientation

With the official Nutanix Visio Stencil most of the issues described here are still present and you might end up with inaccurate or faulty documentation. How would you know? I’ve made them. Why aren’t they fixed? see: Why unofficial shapes?

Please visit the Nutanix stencil download page for the latest dynamic stencil package.

Since these are unofficial Visio shapes, they are not available on visiocafe.com, where the official stencil package resides.

DPTPB Dynamic Shape Documentation

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s