Nutanix + 10GbE Switches + DAC cables: Working Combinations

While Nutanix has only few types of cables to use and has way simpler cabling schematic than traditional 3-tier architectures, there are still cautions with cabling, especially with 10GbE cabling.

If I had to name one type of IT equipment that I really dislike, the award would definitely go to 10GbE DAC or TwinAx-cables. This dislike is not specific only towards Nutanix 10GbE DAC cabling, regardless of which vendor I happen to be working with, I usually feel pinch in my stomach, when someone suggest using DAC cables.

I’ve seen way too many IT projects delayed, simply because chosen 10GbE switch does not work with DAC cable supplied. Usually any cost savings made by choosing DAC over fiber are thrown out the window while waiting for the replacement cable. Your new and shiny “toy” is sitting idle, your IT staff is fiddling with their fingers, support period is running, the new devices are churning away electricity and producing heat, but are yet to produce any business value.

To stop flinching every time DAC is mentioned and to give my poor body a break, I decided to collect this information. You can help me out by leaving comments on known working / non-working combinations, my stomach will greatly appreciate.

Good news is that 10GbE DAC cables can be used with Nutanix and more so than with some other vendors. Nutanix uses 10GbE (intel) adapters that are compliant with spf+ Multi-Source Agreement or MSA, so as long as your switch and DAC cables are sfp+ MSA compliant you should be okay.

Unfortunately that is not the whole story.

Firstly DAC cable section of spf+ MSA is optional, so while a manufacturer might be compliant with general (fiber connected) section of spf+ MSA, they might NOT be compliant with DAC section of the spf+ MSA.

Additionally some major switch players, like Cisco and Hewlett Packard Enterprise (HPE), are NOT fully compliant with MSA and/or have crippled the MSA compatibility by adding DAC cable manufacturer EEPROM checks into their firmware and are raising alerts or downright disabling ports which use cables that are not sold/manufactured by them.

Non-compliant combinations of ports and DAC cables might work sometimes, they even might work all the time. But the problem arises when you have issues with networking of your device, switch manufacturer support might deny any further actions until unsupported DAC cables are replaced with supported DAC cables. In some cases even when the DAC cable is not at fault.

Check out my Visio by DPTPB documentation: 10 GbE cabling for additional thoughts on DAC cables.

Working switch + DAC cable combinations with Nutanix

Disclaimer: this is “tribal knowledge” and not official statement from any of the parties. Reported working combinations might turn into non-working combinations because of policy / firmare changes.

Any Switch Manufacturer:

• You are more likely to be successful if using DAC cables supplied by switch vendor
• Switch manufacturer DAC cables should work with Nutanix as long as they are MSA compliant
• If using passive DAC cables, limit lenght to max 5 meters (longer cables tend to be active anyway), see next point
• “DAC cables longer than 5M tend to work, but make sure you switch to active instead of passive cables. If you do that, you should be fine with distances that go over 5M in length.”
  • Comment by blog reader

Arista:

• Arista switch works well with Nutanix Molex cables. No issues at all.
  • Comment by blog reader

Cisco:

• Cisco switch with Cisco supplied DAC cable

HPE:

• HPE switch with HPE supplied DAC cable

Huawei:

• Works with Nutanix DAC cable
  • comment by blog reader

Brocade:

• Brocade ICX 7250 or VDX 6740 switches
  • Brocade Active DAC cable or third-party Active DAC cable
  • Cost savings with third-party cables are negligible, better to use Brocade cables

Juniper:

• Works with Nutanix supplied (Molex) DAC cable
  • comment by blog reader
• Note on Juniper support policy with 3rd party cables
  • If you face a problem running a Juniper Networks device that uses a third-party optic or cable, the Juniper Networks Technical Assistance Center (JTAC) can help you diagnose the source of the problem. Your JTAC engineer might recommend that you check the thirdparty optic or cable and potentially replace it with an equivalent Juniper Networks optic or cable that is qualified for the device.
  • Source : Panduit compatibility matrix

Zyxel:

• Works with Nutanix supplied (Molex) DAC cable
  • comment by blog reader

Extreme, IBM, DELL, Switch Vendor X:

• No experience: need more info, please leave a comment if using with Nutanix

Non-working switch + DAC cable combinations with Nutanix

Any Switch manufacturer:

• Check out Panduit compatibility matrix. If there are restrictions related to switch in use, those might apply with Nutanix as well (because switch manufacturer has set limitations on DAC cable usage)

Cisco:

• Cisco switch with any other DAC cable than Cisco branded DAC cable
  • Nexus Switches
    • You can force the switch to accept unsupported DAC cable, but the combination remains to be unsupported by Cisco
    • Use only if no other option is available and you have to get your system up and running
  • Catalyst Switches
    • Unsupported cables cannot be enabled by issuing commands
    • Link might not even come up with other DAC cables
• Nutanix supplied DAC cables are not supported by Cisco

HPE:

• HPE switch with any other DAC cable than HPE branded DAC cable
  • With HPE switches you are out of luck, AFAIK there is no command to disable DAC cable manufacturer check
  • IIRC there used to be such command, but not with the latest firmware versions
  • Link might not even come up with other DAC cables
• Nutanix supplied DAC cables do not work with HPE switches

Brocade:

• Brocade switch with passive DAC cables
  • Modern Brocade switches are MSA compliant, but their switches require active DAC cables
  • Can use third-party active cable, but most of time it makes sense to use Brocade supplied active DAC cables
• Passive DAC cables supplied by Nutanix might work, but are known to cause issues

Grounding caution when using DAC cables

• Warning: 10GSFP+Cu / 10 GbE DAC cables can only be used on systems with common grounds. Connecting systems with different ground potential with SFP+ direct attach cable results in a short and may cause damage.
• Source: ftp://ftp.seagate.com/sff/SFF-8431.PDF, page 101

Call-to-Action:

• If in doubt about DAC cabling compatibility
  • Check with switch manufacturer
  • or
  • use vendor specific sfp+ transceivers on both ends and connect with OM3/OM4 fiber
    • will be more expensive solution
    • but you won’t have a stomach ulcer as a result
• Please leave a comment on working / non-working combinations and I’ll update this post

Sources / More info:

DAC cable discussion on Nutanix community site:

http://next.nutanix.com/t5/Installation-Configuration/Nutanix-Cable-Support/m-p/1853

More info on sfp / sfp+

https://en.wikipedia.org/wiki/Small_form-factor_pluggable_transceiver

More info on Multi-source agreement (MSA):

http://www.commscope.com/Resources/Standards/Enterprise/Multi-source-agreements/

https://en.wikipedia.org/wiki/Multi-source_agreement

ftp://ftp.seagate.com/sff/SFF-8431.PDF

Panduit SFP+ Copper Cable Assemblies Compatibility:

http://www.panduit.com/heiler/TechnicalReferences/D-COTR104–WW-ENG-Rev5-SFPCprCblAsmbl.pdf