Visio by DPTPB: 40GbE / 32 GB FC Switch shapes

Feb 10, 2017 update notes for Visio by DPTPB stencil collection, part 2

40GbE / 32G FC Switch General Notes

LAN & SAN switches are typically available in both “Fan Side Intake” / “Port Side Exhaust” and “Port Side Intake” / “Fan Side Exhaust” versions. If sourcing switches from NetApp typically correct airflow is selected, in case of sourcing switches directly from switch vendors, take care to order correct airflow option. With wrong airflow you might end up overheating your switch and cause it to shut down: been there, done that, no t-shirt 🙂

Normally switches housed in the same rack as hosts/storage controllers are using front-to-rear air flow  or “Fan Side Intake” / “Port Side Exhaust”, while network gear only racks are using reversed airflow rear-to-front or “Port Side Intake” / “Fan Side Exhaust”). For more info, see: Port side intake vs. Port side exhaust.

Typically switch power cord connectors are on the fan side and if the switch is placed within a NetApp rack, these connectors are on the other side of the rack than power supplies, this might require a 1U pass-trough / brush panel to leave some room to run power cables from back to front. This is more likely with narrow 19″ racks, wide variants of racks have more space to run cables and usually there is no need for pass-trough panels.

40GbE Cluster Interconnect Switch

At the moment only 40GbE switch certified as NetApp Cluster Interconnect Switch is Cisco Nexus 3132Q-V, a 32-port 40GbE 1U switch. For detailed specs, please see: Cisco Nexus 3100-V Platform Switches Data Sheet

According to NetApp Hardware Universe (hwu.netapp.com) this switch uses different DAC cables than 40GbE host-side switches. It made sense to use non-cisco DAC cables with the NetApp 10GbE Cluster Interconnect switch (CN 1610) since it wasn’t a Cisco switch.  I can’t quite figure out why different DAC cables are used with 40GbE fabric interconnect switch. Since it is a Cisco switch, why not use Cisco DAC cables?

I exchanged emails about this with NetApp TME and he verified that the recommended 40GbE DAC cable for Nexus 3132Q-V really is:

• PN 112-00542
• NOT Cisco 40GbE cable, PN 112-00604 (X-QSFP-H40G-CU3M-R6), which should be used only with host-side connections

In the past Cisco has been notoriously bad at supporting any other DAC cables than their own, so in a sense this is a welcome change 🙂

It is possible to use this switch also with nodes equipped with 10GbE ports by splitting 40GbE ports on the switch side into 4x10GbE ports by using breakout cables (in this case the cable should probably be called as breakin cable). One of the use cases for breakout / breakin cables is migration from 10GbE Cluster Interconnect Switch to 40GbE Cluster Interconnect Switch, for more info see NetApp library instructions for migration.

interesting note from the migration instructions:

NetApp twinax cables are not compatible with Cisco Nexus 3132Q-V switches.If your current CN1610 configuration uses NetApp twinax cables for cluster-node-to-switch connections or ISL connections and you want to continue using twinax in your environment, you need to procure Cisco twinax cables. Alternatively, you can use optical fiber cables for both the ISL connections and the cluster-node-to-switch connections.

My guess is that this note is referring to 10GbE NetApp DAC cables as there are four sfp+ ports on NX-3132Q-V, where you can connect 10GbE DAC cables. According to AFF A700s SE presentation correct breakout 40 GbE DAC cable for NX-3132Q-V is Non-Cisco version,  PN112-00545

40GbE Cluster Interconnect Switch shapes

The original switch Visio shapes are made by Cisco. Unfortunately they have the same “blue shape” bug as all the other Cisco Visio shapes, at 200% zoom level the shape turns into blue square (at least with Visio 2016). It seems to display correctly at 100%,400% and 800% zoom levels. This bug is somehow related to Visio themes, but I can’t figure out how to fix them, since my drawings/stencils are not using Visio themes…

Picture: NetApp 40G Cluster Interconnect Switches, Cisco Nexus 3132Q-V DPTPB shapes (front view)

Picture: NetApp 40G Cluster Interconnect Switches, Cisco Nexus 3132Q-V DPTPB shapes (rear view, rack version)

Picture: NetApp 40G Cluster Interconnect Switches, Cisco Nexus 3132Q-V DPTPB shapes (rear view, mid version)

I’ve also modified port naming convention used in port assignment boxes to match the 2-tuple / 3-tuple naming convention used by Cisco. Instead port “1” the first 40GbE port is “1/1”. If a 40GbE port is split into four 10GbE connections the port name would be “1/1/1″,”1/1/2″,”1/1/3” or “1/1/4”. In order to keep the port assignment box size unchanged, I had to trim down the first column indicating switch number, from  “SW 1/2:” to “1|2:”

With only 12 rows in the port assignment box, you can easily run out of space, specially if using multiple breakout cables. There isn’t simply enough space available to document very large configurations within Visio, in such case my suggestion is to leave port assignment boxes empty and document port assignments in a separate excel sheet.

Picture: Port assignment box for 40GbE switches

40GbE Switch shapes (host-side)

There is more freedom to use different 40GbE switches on the host-side with NetApp. Netapp obviously prefers Cisco due to their FlexPod affiliation. Since the company I work for is also Brocade distributor, I’ve included both Brocade and Cisco shapes. While there are multiple 40GbE switch models available, I’ve chosen to use switch models with 1U form factor to save some space in the drawings. I might add other switch models/vendors in the future.

Brocade VDX 6940-36Q shapes

A 36-port 40GbE 1U switch with QSFP+ connectors. For more info please see: Datasheet for Brocade VDX 6940

With NetApp this switch requires fiber connection, since NetApp only supports Cisco branded DAC cables for host-side connectivity.

Unfortunately Visio guy at Brocade has had a brain fart with the original shape: No connection points in the middle of ports, only two connection points on either side of the port. This may work with special QSFP+ connector shape built to match the port, but not with generic Visio cable shapes. I’ve modified the original shape to have a single connector point in the middle of each port so standard Visio cable shapes with single connector points will work with this switch shape. Additionally Brocade stencil only has a rear view shape, so I’ve used VDX 6740 front view shapes instead (likely to be very much like VDX 6940, both have two PSUs).

Picture:Brocade VDX 6940-36Q DPTPB visio  shape (front view)

Picture: Brocade VDX 6940-36Q DPTPB visio  shape (rear view, rack version)

Picture: Brocade VDX 6940-36Q DPTPB visio shape (rear view, mid version)

Cisco Nexus NX-5624Q shapes

A 24-port 40GbE 1U switch. Base model comes with 12 ports and the switch can be expanded with 12-port expander card. DPTPB shape comes equipped with expander card, so 24 ports in total. For more info, please see:  Cisco Nexus 5600 Platform 40-Gbps Switches Data Sheet

Since NetApp supports Cisco DAC cables for host-side connections this switch can be used with both DAC and Fiber cabling.

The original switch Visio shapes are made by Cisco. Unfortunately they have the same “blue shape” bug as all the other Cisco Visio shapes, at 200% zoom level the shape turns into blue square (at least with Visio 2016). It seems to display correctly at 100%,400% and 800% zoom levels.

Positive side with the original Cisco Visio shape is that there are separate front view shapes for different air-flows, blue fan latches for “Fan Side Intake / Port Side Exhaust” and red fan latches for “Port Side Intake / Fan Side Exhaust”. Since I normally draw switches in the same rack as the storage controller, I typically use the one with blue fan latches, but I’ve included both versions just in case, 5624P for Port Side Exhaust and 5624F for Fan Side Exhaust.

Picture: Cisco NX-5624Q “Fan Side Intake / Port Side Exhaust” DPTPB visio shape (front view)

Picture: Cisco NX-5624Q “Port Side Intake / Fan Side Exhaust” DPTPB visio shape (front view)

Picture: Cisco NX-5624Q DPTPB visio shape (rear view, rack)

Picture: Cisco NX-5624Q DPTPB visio shape (rear view, mid)

32G Fiber Channel Switch shapes

Brocade is one of the first Fibre Channel vendors to support 32G FC with G620 stand-alone switches and X6 series directors. These seem to be so new, that I couldn’t find Visio stencils. I’ll update DPTPB SAN stencil with 32G FC switches once Brocade publishes Visio stencils. Brocade G620 Switch Data Sheet

While 40GbE switches use QSFP+ connectors and MPO cabling, 32G FC is still using smaller form-factor SFP+ ports. ISL links can operate at 128Gbps, in such case connector type is QSFP+.