Document ID: 290098
http://support.veritas.com/docs/290098 E-Mail Colleague IconE-Mail this document to a colleague

Slow performance when backing up remote servers with the TCP Offload Engine (TOE) Feature enabled on network cards and/or Windows 2003 SP2 Scalable Networking Pack (SNP) is enabled.
Details:

Windows 2003 Service Pack 2 introduced a new update called SNP (Scalable Networking Pack) . This utilizes a feature in newer network cards called TCP/IP Offload Engine (TOE ) or TCP Chimney.  Some servers that have this feature enabled experience throughput or connectivity issues when used in conjunction with certain NICs (Network Interface Cards) between the media server and remote machines.  

TCP Offload Engine is an emerging technology which is designed to offload TCP stack handling from the main system CPU to a processor built into NIC cards.  This technology is still relatively new, and when engaged, has been known to cause unstable connections.  This results in dropped sockets, dropped packets, packet reordering, packet retransmits and ultimately slow or failed backups.
There are two types of TCP offloads - stateful and stateless.
Stateful offload:
TCP Chimney is a stateful offload. TCP Chimney offload enables TCP/IP processing to be offloaded to network adapters that can handle the TCP/IP processing in hardware.
The Windows 2003 TCP Chimney setting and several Network Card (NIC) settings have been known to interfere with successful inter process communication and connectivity over TCP/IP networks.
Stateless offload
RSS and NetDMA are stateless offloads. Where multiple CPUs reside in a single computer, the Windows networking stack limits "receive" protocol processing to a single CPU. RSS resolves this issue by enabling the packets that are received from a network adapter to be balanced across multiple CPUs.
The following has been observed in cases when this issue occurs:

<1> Backup of Local Media Server to B2D (Backup to Disk) runs at a normal throughtput rate.
<2> Backup of Local Media Server to tape runs at a normal throughtput rate.
<3> Backup of Remote Server (One or All) to B2D runs at a normal throughtput rate in most instances, but runs slower in some network configurations.
<4> Backup of Remote server (One or All) to tape is consistently slow or looses connection all together during a backup or restore.

Please note:  If this occurs only while backing up one remote server, TOE is most likely enabled on that Server's NIC.  If this is happening for all remote servers , then TOE Chimney is most likely enabled on the Media Server's NIC.  To determine if the TOE Feature is possibly causing the performance issue, disable the feature per the following and run additional remote backups to test:
1) Modify the network adapter (NIC) properties to disable Receive Side Scaling or the TOE Option.  Contact the Hardware Provider for specific instructions.

2) Temporarily disable Windows 2003 SP2 Scalable Networking Pack TCP Chimney Offload feature by issuing the following command at a windows command prompt (Start, Run, CMD). No reboot is required

Netsh int ip set chimney DISABLED

To check the current setting, observe this registry key:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Tcpip\Parameters
EnableTCPChimney
Type: REG_DWORD
Values: 1 (enabled) 0 (disabled)

Note:  This is a temporary work-around to disable the TOE Feature and will not persist after a reboot.  Also, if the command completes successfully but no improvement is seen in the performance, then it most likely is not being caused by the TOE Feature.

3) Disable the Windows 2003 SP2 Scalable Networking Pack features (TCP Chimney Offload, Receive Side Scaling , Network Direct Memory Access) by applying the update per Microsoft KB 948496  
An update to turn off default SNP features is available for Windows Server 2003-based and Small Business Server 2003-based computers
 http://support.microsoft.com/kb/948496/

This issue is currently under investigation by Symantec Software. Pending the outcome of the investigation, this issue may be resolved by way of a patch or hotfix in current or future revisions of the software. However, this particular issue is not currently scheduled for any release. If you feel this issue has a direct business impact for you and your continued use of the product, please contact your Symantec Sales Representative or the Symantec Sales group to discuss these concerns. For information on how to contact Symantec Sales, please see:  www.symantec.com
For additional information on the TOE Feature, review the following:
Windows 2003 Scalable Networking pack and its possible effects on Exchange
 http://msexchangeteam.com/archive/2007/07/18/446400.aspx
The Microsoft Windows Server 2003 Scalable Networking Pack release
 http://support.microsoft.com/kb/912222
You cannot host TCP connections when Receive Side Scaling is enabled in Windows Server 2003 with Service Pack 2
 http://support.microsoft.com/default.aspx?scid=kb;EN-US;927695
TCP traffic stops after you enable both receive-side scaling and Internet Connection Sharing in Windows Vista or in Windows Server 2003 with Service Pack 1
 http://support.microsoft.com/default.aspx?scid=kb;EN-US;927168

How to enable or disable TCP Offload at the hardware/driver level:
In addition to what can be done through the Windows OS, individual network cards have attributes associated with their drivers which can toggle TCP offload settings from the main CPU to the processor on the NIC.

To list out which TCP offloads are enabled per network card, run this command:
Netsh int ip show offload

Here are examples of settings from 2 different NIC cards:

Example 1:Output from Netsh int ip show offload:
Offload Options for interface "Local Area Connection" with index: 10005:
TCP Transmit Checksum
IP Transmit Checksum
TCP Receive Checksum
IP Receive Checksum
TCP Large Send

In order to toggle these settings, go to Windows Device Manager > Network Adapters > (NIC) > Advanced tab

Example 1 Advanced tab settings:
 
Example 1 map :
TCP Transmit Checksum = Offload Transmit TCP Checksum
IP Transmit Checksum     = Offload Transmit IP Checksum
TCP Receive Checksum   = Offload Receive TCP Checksum
IP Receive Checksum      = Offload Receive IP Checksum
TCP Large Send              = Offload TCP Segmentation

Example 2 Output from Netsh int ip show offload:
Offload Options for interface "Local Area Connection" with index: 20003:
TCP Transmit Checksum
IP Transmit Checksum
TCP Receive Checksum
IP Receive Checksum
TCP Large Send
Example 2 Advanced tab settings:
 
Example 2 map :
TCP Transmit Checksum = Checksum Offload
IP Transmit Checksum     = Checksum Offload
TCP Receive Checksum   = Checksum Offload
IP Receive Checksum      = Checksum Offload
TCP Large Send              = Large Send Offload

Symantec Support recommends setting all these values to Off/Disabled/None  (or whatever value means "off")

NOTE: As can be seen above, different NIC manufacturers and drivers may name these attributes differently.  Experimentation can be done to identify which settings map to which names by changing the value in the Attribute section of the NIC properties and then rerunning Netsh int ip show offload to see what changed.

NOTE: Modifying any of these settings will temporarily disconnect the NIC from the network.  Care should be taken when making this adjustment so as not to disrupt any critical activity.  Also, consider directly interfacing with the machine rather then using Remote Desktop Control.

NOTE: Modifying these settings for most NIC's does not require a reboot to engage the setting.  But if this is a great concern, check with your NIC vendor to see if a reboot is necessary, or experiment on a non-production machine before modifying a production machine.

How to observe the Offload State of a TCP connection:
The command netstat with the switch "-t"  can be used to show the Offload State of a network connection.

Example Output from the netstat -ab -t command:
Proto Local Address Foreign Address State      PID Offload State
TCP   server:epmap  server:0        LISTENING  816 InHost

Proto Local Address  Foreign Address  State       Offload State
TCP   Server:netbios name:2151        ESTABLISHED Offloaded

Flow Control Recommendations for NICs:
What is Flow Control?
"Flow Control" is a parameter that allows the user to enable or disable the receipt or transmission of PAUSE frames. PAUSE frames enable the adapter and the switch to control the transmit rate. The side that is receiving the PAUSE frame temporarily stops transmitting.

It is recommended to set Flow Control on the NIC to "Generate and Respond".  Flow Control can also be modified within the Network Card Properties > Advanced tab.  See Example 3.
 






Supplemental Material:

System: Ref.#Description
ETrack: 1107687 Slow Remote Agent Backups when the TCP Offload Engine (TOE) Feature is enabled on a network interface card

Acknowledgements
Microsoft
Products Applied:
 Backup Exec for Windows Servers 10.0, 10d (10.1), 11d (11.0), 12.0
Last Updated: June 26 2009 05:41 PM GMT
Expires on: 365 days from publish date
Subscribe Via E-Mail IconSubscribe to receive critical updates about this document

Subjects:
 Backup Exec for Windows Servers
   Application: Backup, Restore, Troubleshooting
   Application Basics: Backup-Restore, Troubleshoot
   Device Management: Troubleshoot

Languages:
 Russian, English (US), French, German, Spanish, Italian, Japanese, Chinese, Korean

Operating Systems:
Windows Server 2003

DataCenter SP1, DataCenter SP2, Enterprise SP2, Enterprise ServerSP1, Standard Server SP1, Standard Server SP2, Storage Server SP1, Storage Server SP2, Web Server SP1, Web Server SP2