This technote is specifically aimed the AIX operating environment.

In an IBM virtual I/O (VIO) environment, the maximum transfer unit (MTU) values for interfaces, as reported by the AIX kernel and the interface card/switch may differ.

The kernel default value is set much higher and the interface and/or the switch could be set to a nominal value of  for instance 1500 bytes. Since LLT gets its value from the kernel, there is an attempt to push up to 65000 bytes through the switch that can handle only 1500 bytes. The problem exhibits only with large packets as when main.cf is shipped across. Smaller packets like HB will go through, and so it appears that GAB and LLT are working fine. So this is normally seen any time when main.cf is sent between the nodes.

Example:  When starting VERITAS Cluster Server, the second node will hang in remote build. This problem will also occur during the initial configuration through the installer (installsfrac), where configuration steps of the installer will hang while trying to "haconf -makerw".  

The VCS /etc/llttab requires the LLT configuration change when using LLT across a switch in a Virtual I/O (VIO) environment on AIX

Explanation:

LLT queries the AIX native DLPI layer to get the MTU size for each private network interface. It then takes the least of all the MTUs,  which is then treated as the "overall MTU" that LLT will use for communicating with its peers.

Example:
If you have one private link with jumbo frames and the other at normal MTU, then the "overall MTU" will be normal (1500). Use the command lltstat -c to get the "overall mtu" value:

# lltstat -c  | grep overall
   overall mtu: 1460

If the LLT private network links are *virtual Ethernet devices*, the AIX DLPI layer returns an MTU size of *65354* for each link. The "overall mtu", as a result, is *65354*. This value does not match with the maximum MTU that the external switch can handle. As a result, any LLT packets greater than 1500 bytes will be dropped by the switch.

Let's consider a scenario with two private links and one public link:
Even if both of the private links are lost, the MTU used by LLT will not change. The MTU will remain at a size that will work on all links. If the llttab has defined that LLT use an MTU of 1500 for the private links, then if the public link is ever promoted from a low-pri to high-pri LLT link, it will also use MTU of 1500Bytes for LLT.


Errors:

2008/08/08 14:31:57 VCS ERROR V-16-1-10116 GabHandle: open failed errno = 16
2008/08/08 14:31:57 VCS ERROR V-16-1-11033 GAB open failed. Exiting

How to confirm VIO Interfaces are in use:

# lscfg -v | grep ent
 ent3             U9119.595.029445E-V45-C967-T1                                    Virtual I/O Ethernet Adapter (l-lan)

Workaround:

1. If the LLT private links are over virtual Ethernet devices, modify the /etc/llttab entry to restrict the MTU value to 1500

Sample llttab file restricting the MTU size to 1500:

# more /etc/llttab
set-node thoribm90mp8
set-cluster 90
link en1 /dev/dlpi/en:1 - ether - 1500
link en2 /dev/dlpi/en:2 - ether - 1500

OR

2. Connect the interfaces with crossover cables instead of a switch. This, however, is restrictive.

OR

3. Modify the switch MTU settings to something larger may help, but if kernel is reporting 64k, then the LLT packets will still not arrive at other nodes across the switch.