TIP: In FCoE world there are no HBAs, but there
are CNAs instead. All the Storage concepts stay the same.
TIP: If you want to use the FCoE, you first
need to enable the feature using the command “feature
fcoe”. Also, have in mind that Max Length of the FCoE is 2240 bytes, and
Jumbo Frames are required.
FCoE is a great model because it uses the Ethernet, and
therefore enables the convergence of SAN and LAN (Unified Fabric). FCoE is an encapsulation of the FC frames into
Ethernet. The only
difference between FC and FCoE is the L1 and L2 TRANSPORT.
In the simplest FCoE topology, a server with CNA (converged network adapter; a card that
can send both Ethernet and FCoE traffic over the same gigabit Ethernet uplink)
is connected to an FCoE-enabled switch, which has a direct connection into the
legacy FC network.
The idea is to make Ethernet lossless, because Storage
requires the lossless flow. FCoE is fully defined in FC-BB-5 standard, and it
is defined by the following 3 criteria:
-
PFC
(Lossless Ethernet), 802.1Qbb. It enables Flow Control using the Priority
Bit using the Pause Frames (802.1p). The link is devided into the 8 “lanes”
using 3 bits, and only one lane is dedicated to the FCoE. This enables FCoE to
create the lossless environment. When too many frames are sent – the PAUSE is
sent on the lane basis, not on the entire link.
-
ETS
(Priority Grouping), 802.1Qaz. This is a way to guarantee the BW, because
it can assign BWs percentages to groups.
-
DCBX (Data Center Bridging Excange, used for the
Configuration Verification), 802.1Qaz
It requires Jumbo Frames and 10Gbps Links. FCoE requires
that the FC IDs be mapped to the Ethernet MAD Addresses.
FIP (FCoE Initialization Protocol) is a Control
Plane on FCoE, and it will be running between the FCoE Initiators and FCoE
Switch, known as the FCF (FCoE Forwarder). FLOGI happens inside of FIP within
the FCoE.
The port roles in FCoE are the following:
-
ENode is the FCoE Node Port. ENode gets
a Fabric Provided MAC Address (FPMA)
for FCoE, which is ultimately used for the Data Plane as a SAN-MAC (apart from
the FCID, that it also gets assigned). It still uses the MAC address for the
LAN traffic, and it´s 6 bytes (3 bytes of FCID and 3 bytes of FC-MAP). This
FC-MAP is a mechanism that separates the VSANs, so that one VSAN can never go
through a switch it´s not meant to (The Switch descarts MACs that are not part
of a particular fabric).
-
Virtual Fibre Channel (VFC) Interface,
the logical representation of the FC interface to the Ethernet world. The FIP
runs between the ENode and the VFC. They form a p2p logical link between them.
-
Virtual Port Types:
- VN_Port, like the N port in the FC
- VF_Port, like the F port in the FC
- VE_Port, like the E port in the FC (normally these are actually TE ports, running Trunking)
FCF (Fiber Channel Forwarders) are like a
switching devices for FCoE, do Forwarding of the Information and the Encapsulation and Decapsulation of the FC. These are the combination of the FCoE
Termination Functions and Fiber Channel stack on Ethernet Switches, asa Dual
Stack Switches.
End Nodes represent Virtual FC Interfaces, called VN ports,
or the virtual N Ports. The Virtual Fabric ports are the VF ports
on the switch.
Configure FCoE
You need to be sure you need the release notes for the exact
CNA card, and the exact release of your NX-OS to be sure what is supported.
Step 1: First of
all you need to create a VSAN:
(config)# vsan
database
(config-vlan)# vsan
1010
(config-vlan)# vsan
1010 name VSAN_1010
TIP: A link going from the Switch to the CNA
has to be a Trunk Link, and the FCoE VLAN has to be on the allowed list. The
FCoE VLAN must not be a native VLAN, and don’t use the VLAN 1. If you want the CNA to have an
Access VLAN, that has to be a NATIVE VLAN.
Step 2: Map a
VLAN to VSAN:
(config)# vlan
1010
(config-vlan)# fcoe
vsan 1010
If you want FLOGI not to fail on the FCoE ports directly
connected to the CNAs, you need to make it a EDGE port, because the FLOGI
timeout will expire cause it´s shorter then STP timer:
(config-if)# spanning-tree
portfast edge trunk
Step 3: Configure
the vFC and bind the VSAN to the
physical interface where the CNA is connected:
(config)# interface
vfc1010
(config-if)# bind
interface gi1/1
(config-if)# switchport
trunk allowed vsan 1010
(config-if)# no
shut
Step 4: vFC (virtual Fiber Channel) Interface should
be associated to only one VSAN. You need to bind these:
(config)# vsan
database
(config-vlan)# vsan
VSAN_1010 interface vfc1010
Step 5: Configure
the Ethernet Link (facing the Server) as a Trunk:
(config)# interface
Gi1/1
(config-if)# shut
(config-if)# spanning-tree
portfast edge trunk
(config-if)# switchport
trunk native vlan 10
(config-if)# switchport
trunk allowed vlan 10,1010
FCoE QoS
All Fibre Channel and FCoE control and data traffic is
automatically classified into the FCoE system class, which provides no-drop service. There are two
protocols here:
- FCoE is in charge of the Data Plane
- FIP (FC Initialization Protocol) handles the Logins and the Control Plane.
- This class is created automatically when the system starts up (the class is named class-fcoe in the CLI).
You cannot delete this class, and you can only modify the IEEE
802.1p CoS value to associate with this class. This class is identified by qos-group 1. In the N5k Switch you will see the following
in the default running
configuration:
# class-map type qos class-fcoe
# class-map type queuing
class-fcoe
# match
qos-group 1
Setting up an FCoE connection on the host
or storage requires one or more supported converged network adapters (CNAs)
connected to a supported FCoE switch. The CNA is a consolidation point and
effectively serves as both an FC HBA and an Ethernet adapter
The CNA is presented to the host and
target as both an FCoE Initiator HBA and a 10-Gb Ethernet adapter. The FCoE
Initiator HBA portion of the CNA handles the FCoE traffic when traffic is sent
and received as FC frames mapped into Ethernet packets (FC over Ethernet). The
Ethernet adapter portion of the CNA handles the standard Ethernet IP traffic,
such as iSCSI, CIFS, NFS, and HTTP, for the host. Both the FCoE and standard
Ethernet portions of the CNA communicate over the same Ethernet port, which
connects to the FCoE switch.
The FCoE target adapter is also sometimes
called a "unified target adapter" or UTA. Like the CNA, the UTA
supports both FCoE and regular Ethernet traffic. FCoE frames have the MAC
Addresses (hop-by-hop) and the FC addresses (end-to-end).
You should configure jumbo frames (MTU =
9000) for the Ethernet adapter portion of the CNA. You cannot
change the MTU for the FCoE portion of the adapter.
SAN Port-Channels like Port-Channels, but to
aggregate SAN PCs using the PCP (Port Channeling Protocol).
(config)# interface
san-port-channel-1
(config-pc)# switchport
mode E
(config-pc)# channel mode active
(config-pc)# switchport
trunk allowed vsan 1
(config-pc)# switchport
speed 4000
