SAS port assignment boxes

In order to document rather than just illustrate, rear view drawings contain information about port assignments and cabling.

In the early version of my stencils these shapes were just boxes with rows and columns and editing was done by manually editing cells. To make life easier I made pre-populated port assignment boxes. This approach worked quite well with disk shelf SAS cabling as there is only limited number of ways to cable SAS shelves (and follow NetApp best practices).

SAS port assignment boxes

• Naming convention
  • Stack Ports
  • Number of Stacks (1x,2x,3x,4x,6x,8x)
  • Controller Type
  • Nth Stack
• Example:”Stack Ports 4xSAS 8040/2”
  • Four Stacks
  • FAS8040
  • 2nd SAS Stack

Stencil:

• DontPokeThePolarBear-NetApp-Stacks-Ports-Stencil-<DD-MM-YYYY>

Source of Original Shape:

• Well, Me 🙂

Screenshot: Port Assignment boxes in Visio Shape Data window

• There is only limited number of legal port combinations
• There are pre-populated SAS port assignment shapes for most typical use cases
• Normally you shouldn’t edit the information, use different shape instead
• 25xx series supports only one SAS stack
• FAS8020 has two SAS ports on the motherboard
  • That equals to 1 SAS stack
  • Add one 4-port SAS adapter = 3 SAS stacks
  • There are shapes for 1 or 3 SAS Stack 8020 environment
• FAS8040/8060/8080EX/FAS8200/AFF A300 have four SAS ports on the motherboard
  • 2 stacks
  • 4 stacks with one 4-port SAS adapter
  • 6 stacks with two 4-port SAS adapters
  • 8 stacks with three 4-port SAS adapters

However over time I started to use this approach for all port assignments

• Controller ports
• Switch ports
• Atto ports
• Etc

With this decision number of possible port combinations sky rocketed, so manually editing ports cell by cell was becoming too cumbersome. So instead of editing port information manually, possible port end-points are lists and can be manipulated in Shape Data window by using pull-down menus (see example below). Like with disk size rather than storing the values for the lists within the shape (or its shapesheet to be exact), the master shapesheet (or TheDoc) is used. This way If I have to add new end-points, I have to edit just few lists on document master shapesheet, rather than updating each shape individually.

Example of SAS Port Assignment Box manipulation with “Shape Data” window:

SAS Stack Port assignment box was designed with assumption that SAS port naming convention would be constant, i.e there would either “Square” or “Circle” ports. I have quite liked this naming convention, I am sad to see that it is now gone with new 12 Gbps/s SAS ports.

Why I liked the naming convention? “Square” and “Circle” are not just words, they describe a shape and that shape can be used to code information and give visual cues. In case of SAS cabling shape, there are two possible types of cable ends, “Square” and “Circle”. “Square” cable ends should connect to “Square” ports and so on. So the cable is somewhat self-documenting and you have visual cues about correct cable usage.

Example SAS cables:

With new IOM12 SAS module in disk shelves SAS ports are just numbered and SAS cable ends are not so descriptive as they used to be. IOM12 SAS has four ports: 1,2,3,4. If normal dual-path HA SAS cabling is used port “1” is equivalent to “Square” port and  port “3” is equivalent to “Circle” port. I have to update my  SAS port shapes, in the mean while, just use old boxes and map port numbers either to “Square” or “Circle”.

With four ports per IOM module you can now use new cabling schema “Quad-Path HA”. This will give more bandwidth per controller, which might be required as SSD media is getting faster with new technologies like “NVMe” and “3d crosspoint”. With this schema “2” ports are essentially “Square2” ports and “4” ports are “Circle2” ports and should be cabled with similar cabling rules like “Square” and “Circle” ports. Quad-Path HA requires also new port assignment boxes, making them will take some time.  I’ll hold on with these   until I have more experience/insight on Quad Path HA.

NetApp best practice for SAS cabling

• a/c ports from SAS adapter should connect to “Square” connector on SAS shelf IOMx module
• b/d ports from SAS adapter should connect to “Circle” connector on SAS shelf IOMx module
• Square / Circle naming convention was used with DS2246,DS4246 and DS4486 shelves
• New DS224C adn DS2112C shelves have more SAS ports per shelf and ports are numbered instead
  • With New shelves:
    • Ports 1 & 2 can be considered as Square ports
    • Ports 3 & 4 can be considered as Circle ports
  • Visual cues with new shelves are not self-evident
• SAS stack should also be symmetrical, i.e if 0a port is used on node1, then node2 should use 0a port to connect to the same stack and shelf
• MyAutosupport and Config Advisor will complain about “Best Practice” breach if rules above are not followed

Port usage according to NetApp Universal SAS Cabling Guide

• First use all “a” ports starting from first adapter / motherboard to Square connections
• Once “a” ports are used use “c” ports for following stacks
• Skip first adapter’s “b” port and use “b” ports for remaining adapters Circle connections
• Then use all “d” ports for following stacks starting from first adapter
• Then use remaining “b” port from the first adapter (and C port the last adapter ) to connect to the last stack

• NetApp best practice is not optimal solution in certain situations
• Lets take 8020 with one 4-port SAS adapter = 3 stacks
  • According to NetApp BP
  • Available a/c ports and order of use: 0a,1a,1c
  • Available b/d ports and order of use: 1b,1d,0b
  • Only two SAS stacks in use
    • Stack1: 0a & 1b
  • Stack2: 1a & 1d
    • Both ports on the same adapter, if we loose this adapter, we loose all connections to the stack2 from one controller -> takeover
  • Stack3: 1c & 0b

With slight modification we can avoid all stack connections from one adapter for first two stacks

• Available a/c ports and order of use: 0a,1a,1c
• Available b/d ports and order of use: 1d,0b,1b
• Only two SAS stacks in use
  • Stack1: 0a & 1d
  • Stack2: 1a & 0b
  • Stack3: 1c & 1b
    • Third stack is not in use and failure of SAS adapter doesn’t take down all connections to one stack
• We have modified port orders slightly to avoid sub-optimal configurations
• Config Advisor or Myautosupport will not complain when using this revised port order
• Adding SAS adapters and SAS stacks after initial installation will throw off port assignments, SAS adapters are cheap, put extra ones in to the initial configuration for future expansions, if there are PCI slots available

Examples of SAS assignment port shapes

SAS port Assignment boxes for new FAS9000 / AFF A700 are still under development. In the meanwhile use existing FAS8200 port assignment boxes and adjust ports as needed, SAS Adapters are most likely in different slots than with FAS8200, check slot  assignment for FAS9000 from hwu.netapp.com

Update:

SAS Port Assignment boxes for FAS8200 / AFF A300

• FAS8200 / AFF A300
  • 4 ports on motherboard = 2 SAS Stacks
  • + SAS adapter in slot 1 = 4 SAS Stacks
  • + SAS adapter in slot 2 = 6 SAS Stacks

Example: FAS8200 / AFF A300 SAS Port Assignment boxes

Screenshot: FAS8200 / AFF A300 SAS Port Assignment box shapes in Visio Shapes Window