Overview of SMB Multichannel
I had mentioned about the new capabilities of SMB 3.0 in my previous blog. Lets review what multichannel buys us. With multichannel, we can increase throughput by using multiple NICs and with RSS we can use multiple cores, have fault tolerance and automatic end-to-end failover.
This is a common configuration we face everyday; left hand side of the picture shows RSS capable client and server communicating over 10GbE, but can only use one TCP/IP connection and only one CPU core with no failover capability ,i.e., if the link/path/NIC goes down, client will lose connectivity. On the other hand, with SMB multichannel and RSS for the same configuration, now the client is able to create multiple TCP/IP connections for the same session, and server can distribute the load on the cores, and hence can fully utilize 10Gbps link. However, we still do not have fault tolerance on the NICs.
Here we have multi NIC configuration with and without RSS capable NICs. With SMB multichannel, we can have client utilize both NICs with multiple TCP/IP connections on RSS capable NICs, and one connection per NIC when RSS is not available. With multi-NICs, we also have automatic NIC failover and combined NIC bandwidth.
As you might know Windows 2012 now has teaming built in. With SMB multichannel over NIC teaming, we can have multiple TCP/IP connections same as previous case (but notice 4 connections, limits are mentioned below, 2 connections per team member), NIC failover (in the team), and combined bandwidth.
Here, SMB detects RDMA and create multi RDMA connections. By the way, do not team RDMA NICs as they will show as non-RDMA capable when teamed.
It scales linearly depending on the I/O size (Small I/O’s consume more CPU)
- Single Non-RSS NICs cannot benefit from multichannel.
- Cannot use different speed NICs, faster NIC is chosen first…
Some useful PowerShell cmdlets to view/change configurations:
SMB multichannel is enabled by default and can be turned off:
Server: Set-SmbServerConfiguration -EnableMultiChannel $false
Client: Set-SmbClientConfiguration -EnableMultiChannel $false
There is one more PowerShell which can be handy when you want to manually select the NICs that will be used for SMB for a target file server:
New-SmbMultichannelConstraint -ServerName FILE_SERVER_NAME -InterfaceIndex 1,2
Debug Logs: In the Applications and Services Log>Microsoft>Windows> SMB Client, right click on ObjectStateDiagnostic and enable logging for troubleshooting.
First, the limit is 8 connections from client to server. The distribution/max connections per NIC is:
RSS-capable: 4 TCP/IP connections per NIC
RDMA-capable: 2 RDMA connections per NIC
For others: 1 TCP/IP connection per NIC
- The defaults are recommended but it can be changed for special configurations or for troubleshooting:
For RDMA capable NICs, DWORD ConnectionCountPerRdmaNetworkInterface needs to be adjusted. It is at HKLM > SYSTEMCurrentControlSetServicesLanmanWorkstationParameters
For resources on Windows Server 2012 File Server and SMB 3.0, please visit