I’ve stumbled onto another potential trap for Cisco Nexus users using Fibre Channel / FCoE and VMWare ESX. I was asked to look into a stability problem with a customer running ESX hosts with CNA cards on the Nexus. Now I know that Fibre Channel and FCoE payload MTU is 2112 bytes across the switch. If you add the additional 46 bytes for things like FCoE header, FC header, CRC and, FCoE EOF that totals 2158 bytes.
So that would explain this built-in QoS profile:
nexus5k# show policy-map type network-qos
Type network-qos policy-maps
policy-map type network-qos default-nq-policy
class type network-qos class-fcoe
class type network-qos class-default
Another interesting point about the Nexus is you can’t configure an MTU per interface like you can with other platforms. It needs to be done globally using a “system qos” policy. Even stranger is the MTU you see with the “show interface” is cosmetic and wrong. There is a case with Cisco raised for this:
CSCsl21529, Symptom: An incorrect MTU value is displayed in the show interface command output. The Cisco Nexus 5000 Series switch only supports class-based MTU. Per-interface level MTU configuration is not supported. The switch supports jumbo frames by default. However, the show interface command output currently displays an incorrect MTU value of 1500 bytes.
If you don’t have a system QoS policy configured the built-in network-qos policy above is applied. Now Ethernet has an MTU of 1500 bytes by default. So even though FCoE uses 2158 bytes, it does not align very well. You’ll see hundreds of thousands of jumbo frames recorded on the interfaces connected to your storage cards.
It’s also worth mentioning that if you’re running VMWare ESX/ESXi a fairly significant change was make post version 4. The ESX hosts now support jumbo frames up to 9k. I checked our ESX hosts and they were configured with an MTU of 1500. So why were we seeing a large amount of jumbo frames still? It looks like VLAN tagging might be causing the frames to exceed the MTU.
If you would like to increase your MTU on your ESX hosts this is how you do it:
- Putty onto VMA. Username = vi-admin Password = use usual ESX password
- Get status with:
vicfg-snmp -server -s
- Enable with:
vicfg-snmp -server -E
- Set community with:
vicfg-snmp -server -c
NOTE:would you your ESX FQDN.
While searching for more information on the problem we were experiencing with the CNA cards I found something very interesting in Scot Lowe’s blog:
Gen2 requires ‘switchport mode trunk’ on the 5K. Gen1 doesn’t. Also make sure FCoE VLANs are allowed on the trunk.
• If you are using a Gen2 CNA, you must put the Ethernet interface in VLAN trunk mode.
• If you are using a Gen1 CNA, the Ethernet interface may be in either access mode or trunk mode.
• If the interface is in trunk mode, be sure that you have allowed the FCoE VLAN via the switchport trunk allowed vlan command.
• If the interface is in access mode, be sure that you have placed the interface in the FCoE VLAN via the switchport access vlan command.
I seem to being going off on tangents here… in summary there is a solution to the problem. You need to increase the MTU on your Nexus switches. This isn’t very well documented but you’ll find more information on this in the Cisco and VMWare forums. The configuration you need to apply is the following:
Setting a per interface MTU (maximum transmission unit) is not supported on the Nexus 5000/2000 series switches.
If a Jumbo packet is required to traverse a Nexus 5000 series switch , the jumbo MTU must be set in a policy-map and applied to the ‘Sytem QOS’.
Configuration, PRE NX-OS 4.1:
Configuration with POST NX-OS 4.1:
policy-map type network-qos support-jumbo-frames
class type network-qos class-default
service-policy type network-qos support-jumbo-frames
All this information I’ve pieced together while troubleshooting and researching how to fix this. I thought it would be worth posting some interesting findings.
Please visit our website: