Overview of SMB Multichannel

imageI had mentioned about the new capabilities of SMB 3.0 in my previous blog. Lets review what multichannel buys us. With multichannel, we can increase throughput by using multiple NICs and with RSS we can use multiple cores, have fault tolerance and automatic end-to-end failover.

Sample configurations:

Single NIC

image

This is a common configuration we face everyday; left hand side of the picture shows RSS capable client and server communicating over 10GbE, but can only use one TCP/IP connection and only one CPU core with no failover capability ,i.e., if the link/path/NIC goes down, client will lose connectivity. On the other hand, with SMB multichannel and RSS for the same configuration, now the client is able to create multiple TCP/IP connections for the same session, and server can distribute the load on the cores, and hence can fully utilize 10Gbps link. However, we still do not have fault tolerance on the NICs.

Multi NIC

image

Here we have multi NIC configuration with and without RSS capable NICs. With SMB multichannel, we can have client utilize both NICs with multiple TCP/IP connections on RSS capable NICs, and one connection per NIC when RSS is not available. With multi-NICs, we also have automatic NIC failover and combined NIC bandwidth.

NIC Teaming

image

As you might know Windows 2012 now has teaming built in. With SMB multichannel over NIC teaming, we can have multiple TCP/IP connections same as previous case (but notice 4 connections, limits are mentioned below, 2 connections per team member), NIC failover (in the team), and combined bandwidth.

SMB Direct

image

Here, SMB detects RDMA and create multi RDMA connections. By the way, do not team RDMA NICs as they will show as non-RDMA capable when teamed.

Performance

It scales linearly depending on the I/O size (Small I/O’s consume more CPU)

image

Notes:

  • Single Non-RSS NICs cannot benefit from multichannel.
  • Cannot use different speed NICs, faster NIC is chosen first…

Ref: http://channel9.msdn.com/Events/TechEd/NorthAmerica/2012/WSV314

PowerShell/Troubleshooting:

Some useful PowerShell cmdlets to view/change configurations:

[ps]
Get-NetAdapter
Get-SmbServerNetworkInterface
Get-SmbClientNetworkInterface
Get-SmbServerNetworkInterface
Get-SmbClientNetworkInterface
Get-SmbMultichannelConnection
Get-SmbConnection
[/ps]

 

SMB multichannel is enabled by default and can be turned off:

[ps]
Server: Set-SmbServerConfiguration -EnableMultiChannel $false
Client: Set-SmbClientConfiguration -EnableMultiChannel $false
[/ps]

There is one more PowerShell which can be handy when you want to manually select the NICs that will be used for SMB for a target file server:

[ps]
New-SmbMultichannelConstraint -ServerName FILE_SERVER_NAME -InterfaceIndex 1,2
[/ps]

image

 

Debug Logs: In the Applications and Services Log>Microsoft>Windows> SMB Client, right click on ObjectStateDiagnostic and enable logging for troubleshooting.

 

Perfmon Counters:

  • imageSMB Server Shares
  • SMB Server Sessions
  • SMB Client Shares

Connection Limits

First, the limit is 8 connections from client to server. The distribution/max connections per NIC is:

RSS-capable: 4 TCP/IP connections per NIC

RDMA-capable: 2 RDMA connections per NIC

For others: 1 TCP/IP connection per NIC

    The defaults are recommended but it can be changed for special configurations or for troubleshooting:

[ps]
Set-SmbClientConfiguration –MaximumConnectionCountPerServer
Set-SmbClientConfiguration -ConnectionCountPerRssNetworkInterface
[/ps]

For RDMA capable NICs, DWORD ConnectionCountPerRdmaNetworkInterface needs to be adjusted. It is at HKLM > SYSTEMCurrentControlSetServicesLanmanWorkstationParameters


For resources on Windows Server 2012 File Server and SMB 3.0, please visit

http://blogs.technet.com/b/josebda/archive/2012/05/03/updated-links-on-windows-server-2012-file-server-and-smb-3-0.aspx


Bulent Tolu

Bulent Tolu

Sr. Systems Engineer at VMware
Bulent is an IT professional with Master's in MIS and 10-years of experience in broad range of Information Technologies. He is exposed to engineering/architecting, implementation/integration, and administration of various high-available IT systems and infrastructure. He has a passion to continually research, test and evaluate new technologies and follow industry best practices to secure and optimize IT systems. Currently, he lives in Istanbul and works as a Sr. Systems Engineer at VMware.
Bulent Tolu

One Thought on “Overview of SMB Multichannel

  1. Tester on 22 September 2014 at 09:38 said:

    don’t forget about Update-SmbMultichannelConnection

Leave a Reply

Your email address will not be published. Required fields are marked *

Post Navigation

Share
Translate »