NetApp is moving towards 12Gbps SAS or SAS-3 connectivity for disk shelves. DS224C shelves with IOM12 (12Gbps SAS) modules have been available for a while and now new generation controllers have onboard 12Gbps SAS ports as well separate 12Gbps capable SAS adapters.

I was somewhat surprised to learn that new 12 Gbps SAS ports are behind single ASIC chip.

What does that mean?

In the past 4-port SAS adapters had two ASIC chips, one controlling ports “a” and “b”,  while the other ASIC chip took care of ports “c” and “d”. This had impact on best practices for SAS cabling.

• To protect against ASIC chip failure:
  • Run one connection from controller port “a” to disk shelf IOMA Square port
  • Run the other connection from controller port “d” to disk shelf IOMB Circle port
  • Since “a” and “d” are behind different ASIC chips and IOMA & IOMB are separate modules you would be protected against ASIC failure
• A single ASIC failure:
  • with old dual-ASIC SAS adapter would NOT lead to other controller taking over, since there would still be a path to disk shelf / SAS stack trough surviving ASIC
  • with new single-ASIC SAS adapters the same event could be more dramatic
    • if using only motherboard SAS ports
      • Or single SAS adapter
      • Both paths to disks shelf / SAS stack would go down
      • Controller would panic
      • and other controller would take-over the workload
      • while take over is transparent with most protocols, there would be slight pause in IO while take-over is happening
    • if using multiple SAS adapters
      • Or combination of onboard SAS ports + SAS adapters
      • There would be access to disks trough surviving adapter / ASIC
      • No need for take-over

Hold on! And this is related to Visio how?

My Visio drawings / stencils are not just “pretty pictures”, they are tools which I am using during solution design phase. Likewise blog entries in “Visio by DBTPB” series are not just about Visio or how to create Visio drawings, these posts also contain insights / reasoning behind certain design / drawing decisions. Even if you are not interested in Visio, you might get some ideas or hints for designing or setting up systems.

DPTPB stencils have a shape type, pre-populated SAS port assignment box. Individual port assignments in these shapes are based on (slightly modified) NetApp SAS Universal Cabling Guide cabling sequence / rules. One of the premises behind those rules is to protect against ASIC failure with dual ASIC SAS controllers.

Now that SAS ports are behind single ASIC chips, there is no point of following SAS cabling rules, right?

OMG NO!

With single ASIC SAS adapters there is one less safeguard against failures and it is even more important to use consistent cabling sequence and follow SAS cabling rules. By following these rules you make sure that the other controller in HA-pair is able to access the partner disks and is capable of taking over in case of ASIC or other failures.

By using ad-hoc cabling sequence you could make a mistake and end up with SAS cabling where both controller A and it’s HA-partner controller B would lose connectivity to a disk shelf or complete SAS stack as a result of single ASIC failure.

Extreme example of such faulty cabling would be both controllers terminating all (four) SAS cables to single a four-port IOM12 module. With such cabling, you might get your storage system up and running, but losing a single ASIC (in IOM12 module) would be a catastrophic event. Both controllers losing all connectivity to disk back-end , which in turn would lead to both controllers panicking and the whole storage system would shut down.

Additionally NetApp configuration check tools, like MyAutoSupport and Config Advisor are checking SAS cabling against these rules. By deviating too much from the rules you will at minimum get unnecessary alerts. This is also reason, why I am considering dropping my modifications to cabling sequence, and revert back to using original cabling sequence used in NetApp Universal SAS cabling guide, a topic for future post.

So while following SAS cabling rules with single-ASIC SAS adapters does not protect against ASIC failures, there is still valid reasons for following the cabling rules.

Call-to-action:

To follow advice from one of my favourite public speakers, Matt Watts / NetApp, I am not closing with a “Thank You” note. Instead he advised ending your presentations with call-to-action.

So here is mine:

• Run Config Advisor against your system / check SAS cabling alerts from MyAutoSupport.
• If you get alerts about SAS cabling
• Consult your NetApp service provider and consider fixing your cabling to follow cabling rules during next maintenance break .
• You will have fewer headaches if something goes wrong