10GbE Cabling

10 Gigabit Ethernet (10GbE) cabling shapes are more generic and these shapes can also be used to document other solutions than just NetApp.

The Company I work for also distributes Nutanix gear and on occasion I’ve used these shapes to document Nutanix solutions. Documenting Nutanix cabling is quite straight forward as there are only few types of cables to run and all nodes run the cables same way. I attended Nutanix user conference a while go where they introduced a new slogan “Complex is Competent and Simple is Genius”. Maybe there is some truth to that, at least when it comes to cabling 🙂

10GbE cable shapes are divided into two categories:

Fiber

• In real life 10GbE cabling consist of three components
  • a sfp+ module in each end
  • fiber optic cable running in between
• For documentation purposes  10GbE fiber connection / cable is a single shape (no separate sfp+ modules)
• 10 GbE Fiber cables in my stencils are “Orange” in color
• Cable end has “Circle” shape

TwinAx or Direct Attached Copper (DAC)

• In real life DAC cable is one piece of hardware
  • a copper cable with soldered sfp+ modules in each end
• A single shape in stencils
• 10 GbE DAC cables in my stencils are “Black” in color
• Cable end has “Circle” shape

Additionally high-speed ethernet cables have few other properties.

Speed

Previous versions of my cable stencils had only 10GbE cables. With the latest HW/SW release NetApp also supports 40GbE speed. Additionally there are standards at least for 25GbE and 100Gbe. So while making new cable for 40GbE, I made 25GbE and 100GbE versions as well.

• The cable speed is now indicated with text “10G”,”25G”,”40G” or “100G” within “Circle” cable end shape

Manufacturer

<RANT>

One of the premises of TwinAX or DAC cabling was to lower 10GbE cabling costs. Manufacturers have fulfilled this promise quite well, a single 10GbE DAC connection can easily cost only 1/10th when compared to similar connection done with fiber and spf+ modules.

Positive side of fiber connection is compatibility. You can use sfp+ modules from different manufacturers in each end of the connection.

As an example to connect a HP server to NetApp, one would use HP branded/qualified sfp+ module for the server and NetApp branded/qualified spf+ module for the NetApp controller. Then connect sfp+ modules with supported OM3/OM4 standard fiber. As an end result all the components are qualified by the manufacturers to work and you end up with a solution that can be supported by both HP and NetApp (in reality there is typically a 10GbE switch in between, but the analogy is the same).

While DAC cables are considerably cheaper, the compatibility matrix with these is a “box-of-worms”. Since DAC cable is a single piece of hardware with soldered sfp+ modules in each end, you cannot change the spf+ module. This means that the sfp+ module has to be qualified by the manufacturer, so as an example HP would have to qualify NetApp cables and NetApp would have to qualify HP cables. Add third-party DAC cables into equation to be qualified and the number possible combinations becomes astronomical.

Extra amount of required work in qualifying the 10GbE DAC cables has led to situation where many manufacturers only support limited selection of 10GbE  DAC cables. The more established the manufacturer is, the more limited the number of supported cables are.

For example HPE and Cisco support only their own branded DAC cables. NetApp only supports Cisco branded DAC cables for host connections and only with Cisco branded switches. Less established players who don’t benefit in closing down the ecosystem are more open to third-party 10GbE cables. Examples of such companies are Brocade and Nutanix. There is standard for DAC cables, but it is not universally honored by different manufacturers, my guess that is done for commercial reasons to boost sales of their own gear.

In my mind the hassles related to DAC cables are not worth the trouble, most of the time DAC  cables are only usable in single-vendor environments or as ISL links between single-vendor switches. If you are lucky enough to have a supported combination of hosts, switches and storage controllers by all means use them (and shave some costs).

I highly recommend against using non-qualified 10 GbE (or 40GbE ) DAC cable and port combinations. The combinations might work when stars are aligned, but should you have any problems, you are out of luck, since the combination you are using is not supported by manufacturers.

</RANT OVER>

So since a Brand X 10GbE DAC cable cannot be used in every situation, I have to differentiate DAC cables some how. In earlier versions I had cables only for two manufacturers and used different line ends to indicate the manufacturer. There are only few usable end shapes, so in order to document more manufacturers I had to choose different way.

While looking to ways to differentiate 6 Gbps and 12 Gbps SAS cables, I learned how to make custom line ends with text inside the end shape for Visio line shapes. Now I am using this approach other cables as well.

I’ve added different cable ends for few more manufacturers. Manufacturer is indicated within cable end shape. Since real estate within end shape is limited, I am using “tickers” instead of Company names

• NetApp = NTAP
• Brocade = BRCD
• Cisco = CSCO
• Nutanix = NTNX

Additionally there are now 10GbE  and 40GbE DAC cables, so on top of manufacturer, the speed is indicated as well and there are different shapes for 10GbE and 40GbE DAC cables.

10 GbE cable variants

• All 10GbE (and faster cables) have at least five-legged versions
• With Primary / Pri (solid line) variant
• And Alternate / Alt (dotted line) variant

Example: 10GbE, 25GbE, 40GbE, 100GbE Fiber cables

Screenshot: X GbE Fiber cables in Visio Shapes Window

Example: 10GbE DAC cables

Example: 40GbE DAC Cables

Screenshot: 10 GbE & 40GbE DAC cables in Visio Shapes Window

Cabling Instructions / Cautions:

Generic usage:

• Each type has two subtypes:
  • Primary (Pri): solid line
  • Alternative (Alt): dotted line
• If you have two 10GbE cables for a controller
  • Run one as Pri
  • And the other one as Alt
• If you have more than two 10GbE connections alternate between Pri and Alt
• Space cables with one 800% grid column/row spacing
• Group together same type of cables running from one controller
• Leave one empty 800% row between groups

Fiber:

• With NetApp
  • Only alternative for any other brand than Cisco branded switch
    • Only requirement is that switch has SR optics
    • HP has also LMR short range optics which are not compatible with SR optics, due to different wavelength used
  • Most of the Cisco installations on the field also use fiber instead of DAC/Twinax cables
    • because of limited DAC cable length
  • No long wave optics/support available from NetApp
  • Max distance few hundred meters depending on fiber type OM3/OM4
  • Sfp+ (controller) + fiber + sfp+ (switch) solution 4-10x more expensive than using DAC/Twinax

NetApp DAC / TwinAx cable:

• Only supported usage at the moment is Cluster Interconnect cabling
  • From Controller to Cluster Interconnect Switch
  • Or
  • As ISL link between Cluster Interconnect Switches
• Cheaper alternative than using fiber + sfp+ optics for Cluster Interconnect connections
• Most of the NetApp cluster interconnect switches use DAC cable at least as ISL cable
• Most of the 2-Node installations use NetApp DAC cables running from controller to controller
• Max length 5 meters

Cisco DAC / TwinAx cable:

• With NetApp:
  • Only DAC / Twinax cable supported with 25xx/8000 series 10GbE adapters and motherboard ports for external / Front-End  / host connections
  • Twinax/DAC cables are vendor specific and may or may not work with switches from other vendors, but are definitely NOT supported by Netapp with any other than Cisco branded switches
  • Use only to connect to Cisco switches
  • Cannot be used for connections to CN1610 Cluster Interconnect Switch
  • Max length 5 meters
    • Active DAC/Twinax cables exceeding 5 meters are not supported
  • Cheaper alternative than using fiber + sfp+ optics for Front-End Cisco Switch connections

Brocade DAC / TwinAx cable:

• Cannot be used with NetApp
• Brocade switches require active DAC cables, they support some third-party DAC cables, but most of the time it will make sense to use Brocade cables
• Nutanix supports either their supplied cables or Brocade Cables

Nutanix DAC / TwinAx cable:

• Nutanix cables are passive, so don’t use them with Brocade switches
• Most of the time it is better use DAC cables from switch manufacturer, since most of them are supported by Nutanix, but not all switch manufacturers support Nutanix DAC cables.