Unified Fabric is a term for all of the equipment that makes LAN and
SAN possible. There are two different networks (LAN as Front-end and SAN
as Back-end) that we are trying to “converge”, and 10G is an enabler of that
convergence.
SAN (Storage Area Network) has 3 high level
components:
-
Storage
Arrays, where the Physical Disks are located. These can be JBOD (Just a
Bunch of Disks) or RAID.
-
SAN
Switches, that connect
the Servers (Initiators) to Storage (Target) can do protocol conversions
between FC and iSCSI, FC and FCoE, FC and FCIP.
-
HBA (Host
Bus Adapters) – which is a NIC card for SAN. There are 3 types of HBA:
o
Fibre
Channel HBAs (SAN traffic only, 1/2/4/8/16 Gbps).
o
iSCSI HBA
(1/10Gbps LAN plus iSCSI hardware offload).
o
FCoE
Converged Network Adapter (CNA) (10Gbps LAN plus FCoE hardware offload).
TIP: When you change the mode of the port
(Ethernet/FC), you need to reboot the switch. On Nexus 5548UP you can assign
the FC ports, but you need to start from the END of the ports in the module.
For example, a module of 32 ports, you assign the ports 28-32 as FC, and 1-27
as native Ethernet.
iSCSI: IP based protocol capable of carrying
SCSI commands over TCP/IP to and from the Storage. SCSI is the standard that
actually defines the communication between the Initiator (Server/Host) and the
Target (Disk/Storage). LUN is the 64-bit
field that SCSI uses for addressing.
HBA (Host Bus
Adapter) represents a NIC card for the FC host. Ports have a specific role
in the Fibre Channel. Fibre Channel defines the Port-to-Port Flow Control, and
also End to End flow control. The HBA will negotiate 1/2/4/8/12/16Gbps with the
Switch Port.
Fibre Channel Switch is not like the LAN Switch, because FC
is a connection oriented protocol. This means that end-to-end “awareness” must
exist between all the Servers and the Disks that will communicate. The ports
will adapt to what is connected to it. The most important Fibre Channel port
types are:
- N_Port
(Node Port) is the Fibre Channel port on the Host (HBA on a Server or a target
port on a Storage Array).
- F_Port
(Fabric Port) is the Fibre Channel Switch Port where the Host (Servers HBA or
Storage Targer port) plugs in. There is also NL and FL mode, which are the Loop
modes (they include a type of a loop prevention, but these are not used a lot)
- E_Port
(Expansion Port) is the port of the interconnection with another Switch. The connection between two E_Ports forms an
Inter-Switch Link (ISL). E_Port
needs to be in
a DEDICATED RATE MODE, and this MUST be set on the MDS (switchport rate-mode dedicated), while on the Nexus it´s
automatical.
- TE_Port
(Trunking E_Port), interconnection with another Switch where tagged frames from
various VSANs are carried. It is a sort of Enchanced ISL link, and it´s
analougous to dot1q trunk, but with VSANs. There is NO “switchport mode te”
command
- VN_Port
(Virtual N_Port), created on a FCoE node (Server or Storage) to enable FCoE
communication with a FCoE switch.
- VF_Port (Virtual
F_port) on a switch, created as needed to establish connection with an end-node
(N_Port)
- VE_Port (Virtual
E_port), created on an FCoE switch to link it with another FCoE switch
- * When you set the FC Mode to AUTO on the interface – it
will automatically negotiate the Port Type. If you want to force one type of
Type, use the command:
- (config-pc)# switchport mode [F | E | NP …]
WWN (World Wide Names): In the FC, the WORD (4 bytes) entoded in the Frames
are exchanged between the INITIATOR and the TARGET. EXCHANGE is the series of WORDS exchanged between INITIATOR
(Server) and the TARGET (Storage Array), and it´s like the TCP Session.
FCNS (Fibre Channel Name Server) is
used to resolve the WWN.
In order for the Fiber Channel device to talk to the
destination device, there has to be a Fabric Registration 3-step process.
Step 1:
The host N_port must log in to it´s F port (FLOGI, F Port Login
process). Switch identifies the WWN of the HBA and assigned the Fibre Channel
ID (FCID).
Step 2:
The host N_port also has to log in to the final N port (PLOGI – End
Port Login process). FCNS is used here to handle all the WWNs,
and we will have the list of all the Initiators and all the Disks.
Step 3: Process
login (PRLI) process must also occur, which handles the upper
layer protocols.
IMPORTANT: Using the
command “fcping” you can PING the
remote FCID or WWN.
You need to start the configuration by
following these:
Activate the FC interfaces, from last port of the module going down & Reload the Nexus.
Activate the FCoE feature (even if you're using the native FC only). No need for additional
reload.
Check with “show license usage”
If you're using
NPV, you also need to activate that feature.
FC Auto
Negotiation in ON, but don’t forget to un-shut the FC ports (interface fc1/20, not ethernet). You should, however, manually
configure the ports (Port Type, Trunk Mode if it´s a trunk, Speed). You can
verify all this using the “show interface fcx/y brief”.
To configure a port as a FC port, we need
to do the following, and then RELOAD the Nexus.
Nexus-5k(conf)# slot 1
Nexus-5k(conf-slot)# port 24-32 type fc <- We need to choose the
LAST group
of ports on
the module
To check the interface configuration you
can use the “show interface” or “show tranciever”:
N5k# show
interface brief
-------------------------------------------------------------------------------
Interface Vsan
Admin Admin Status Oper
Oper Port-channel
-------------------------------------------------------------------------------
fc2/5 1
E on trunking TE 2
port-channel 2
fc2/6 1
E on trunking TE 2
port-channel 2
fc2/7 1
E on down -- --
--
fc2/8 1
auto on fcotAbsent --
-- --
fc2/9 3
E off up E 2 --
fc2/12 3
E on down -- --
port-channel 4
fc3/14 1
SD -- up SD 1 --
fc9/1 1
auto on fcotAbsent --
-- --
fc9/9 1
auto auto up FL <- Since
here we have the FL, the other side is NL
N5k# show interface transceiver
part number is QFBR-5796L
serial number is A00156980
nominal bitrate is 8000 Mbit/sec <-
installed SFP
is 8Gbps
basic id fields (bytes 0-63)
To verify all the FLOGI processes on the
Nexus (just the Local Devices connected to the Switch directly) and check all
the Disks/Servers (including the FCID, Port Name or pWWN and Node Name or nWWN)
that have done the Login to the Nexus/MDS Switch:
Nexus-5k# show flogi database
---------------------------------------------------------------------------
INTERFACE
VSAN FCID PORT NAME NODE NAME
---------------------------------------------------------------------------
sup-fc0
2 0xb30100 10:00:00:05:30:00:49:63 20:00:00:05:30:00:49:5e
fc9/13
1 0xb200e2 21:00:00:04:cf:27:25:2c 20:00:00:04:cf:27:25:2c
fc9/13
1 0xb200e1 21:00:00:04:cf:4c:18:61 20:00:00:04:cf:4c:18:61
fc9/13
1 0xb200d1 21:00:00:04:cf:4c:18:64 20:00:00:04:cf:4c:18:64
fc9/13
1 0xb200ce 21:00:00:04:cf:4c:16:fb 20:00:00:04:cf:4c:16:fb
fc9/13
1 0xb200cd 21:00:00:04:cf:4c:18:f7 20:00:00:04:cf:4c:18:f7
Total
number of flogi = 6.
To configure the TE (trunk extension) Port, you need to set the port mode to “E”, and set the Trunking ON and
define the allowed VSAN. On MDS you also need to set the Rate-Mode to
DEDICATED:
(config-if)# switchport mode e
(config-if)# switchport trunk mode on
(config-if)# switchport trunk allowed vsan [add] X
(config-if)# switchport rate-mode dedicated
To configure the Domain ID:
(config)# fcdomain domainid 0x51
NPIV (N_Port ID Virtualization) is used to assign multiple FCIDs to a single N_Port.
Normally, an N_Port would have a single N_Port_ID associated with it; this N_Port_ID is a 24-bit address
assigned by the Fibre Channel switch during the FLOGI process. The
N_Port_ID is not the same as the World Wide Port Name (WWPN), although there is typically a one-to-one relationship between WWPN and
N_Port_ID. Thus, for any given physical N_Port, there would be exactly one WWPN
and one N_Port_ID associated with it.
NPIV allows multiple FCIDs to be assigned
to a single N port on a host (allows a single physical N_Port to have multiple
WWPNs). This means that different VMs using different WWNs
can use the same HBA.
The Fibre
Channel switch must also support NPIV, as the F_Port on
the other end of the link would “see” multiple WWPNs and multiple N_Port_IDs
coming from the host and must know how to handle this behavior.
NPV (N _port Virtualization) is the extension of the NPIV, and they define the NP ports (N Proxy) and the Edge switch doesn’t require the separate Domain ID
to receive connectivity to the Fabric. The device aggregates the localy
connected N ports into the one or more Uplinks to the Core Switch, which should
support NPIV.
Credit Based Flow Control ensures that the
sender keeps track of the free buffers
on the receiver side. It´s a very conservative strategy to ensure we don´t drop frames (VERY important in the Storage world).
Fiber Channel defines both Flow Controls:
- Port to Port Flow Control, which is Buffer to
Buffer Flow Control
- End to End Flow Control
Buffer-to-Buffer
(B2B) Credit is negotiated to ensure that the FC flow is lossless. Credit gets decreased with each
packet placed on the wire, and gets increased every time “transfer ready” (a
sort of an ACK) is received. There is no similarity with this in the
Ethernet world.
FSPF (Fabric Shortest
Path First) is a FC link-state protocol similar to OSPF, and it does the
routing based on the Domain ID.
RSCN (Registered
State Change Notification) is a message that is communicated to the End
Hosts when something changes. This in most cases means that one path (SAN A for
example) is down, and the other SAN should be used.
Fiber Channel Addressing and Routing
To understand how the FC “routing” work, we need to
understand the WWN (Hardware identifier, like a MAC address in the NIC) and
FCID, which is actually used for the FC “routing”. Fibre Channel routing does
not use the flooding, because the FC routing is built on a separate control
plane process (that includes all the LOGIN-s).
FC needs to have the WWN
(World Wide Names) assigned to every Port and every Node. This is similar
to MAC addresses on the Ethernet. There are:
- Node WWN (nWWN) to identify
Devices (every Disk Drive, HBA, Switch has a single unique nWWN)
- Port WWN (pWWN) to identify Ports.
This means that HBA with 2 ports will be assigned one N and
two P WWNs.
FC address (Identifies the Switch in the Fabric. It can be
assigned Automatically (by a principal Switch) or Manualy), or FC ID has 3
bytes (1 byte for each, Domain ID, Area ID and Port ID):
- Domain ID
(8 bits), to identify the Switch in the FC Network, and used for the Storage
Routing. There are only 239 available (01 - EF) so you can only have 239
Switches in your FC Network. FSPS is a Storage routing protocol that routes
between the Domain IDs. The Principal Switch is the one that assigns the Domain
IDs (if we don’t do it manyally), and it´s the Switch with the lowest WWN. You
can modify the Principal Switch using the command “fcdomain priority 1 vsan X”.
- Area ID
(8 bits), to identify Groups of Ports within a Switch.
- Port ID
(8 bits), to identify the individual devices on the particular port.
To configure the Domain ID you can use the Preferred per
VSAN (you tell the FCNS the ID you´d like, and if it´s not available – you take
the one it assignes), or STATIC (you tell the FCNS the ID you´d like, and if
it´s not available – you don’t get the Domain ID and you´re unable to
communicate):
(config)# fcdomain domain 0x51 preferred vsan 50
VSAN (Virtual SAN) is a technology that
separates the FC Services. VSAN Switches are interconnected with the Trunks
that carry various VSANs (TE_Ports, or the Trunking Expansion ports).
The MDS Switches use TE
(Trunking Expansion) Ports, and we create the EISL (Enhanced Inter-Switch Links), and these can pass the Tagged
VSAN traffic. EISL will also carry the control traffic.
VSANs are like VLANs, and they cannot talk to each other. We
need to implement the IVR (Inter VSAN
routing) to enable the access from one VSAN to another. This can also be
done used the ZONING.
To create a new VSAN and assign it to the FC interface:
(config)# vsan database
(config-db)# vsan 10
(config-db)# vsan 10 interface fc1/1
And add the VSAN to a TRUNK link if needed:
(config)# interface fc1/2
(config-if)# switchport trunk allowed vlan add 10
Zoning is defining which SERVER (Source) can
access which STORAGE (Destination Disk), like the bidirectional Access Lists. FCNS provides everyone the WWNs of all
the Initiators (Servers) and all the Targets (Disks), and if they wanted to
talk to each other – they can. This is what we want to avoid, because we don´t
want Windows server to mount the Linux partition, because the Data Corruption
will occure. Within the Zoning you associate the WWNs and FCIDs and create
aliases.
Default zone policy is to deny. This could
be change to permit as:
(config)#
zone
default-zone permit vsan 1
(config)#
system
default zone
To activate a zone set, you must first create the zone and a
zone set.
FULL zone set: Configured, but not activated,
so – it´s not to be used.
ACTIVE zone set: Can be used. The active zone
set should be the same on all the Fabric Switches.
N5k# show zoneset active
zoneset
name ZoneSet1 vsan 1
zone name zone1 vsan 1
fcid 0x080808
fcid 0x090909
fcid 0x0a0a0a
To configure a zone and assign a zone name, follow these
steps:
Step 1: Set a
Zone NAME and assign it to a VSAN:
N5k(config)#
zone name MYZONE vsan 3
Step 2: Configure
a member for the specified zone (MYZONE) based on the type (pWWN, fabric pWWN,
FC ID, or FC alias) and value specified.
N5k(config-zone)#
member pwwn 10:00:00:23:45:67:89:ab
Fabric pWWN example:
N5k(config-zone)#
member fwwn 10:01:10:01:10:ab:cd:ef
FC ID
example:
switch(config-zone)#
member fcid 0xce00d1
FC alias example (Aliases can be configured using
the “fcalias” command in the global config mode):
switch(config-zone)# member fcalias Payroll
Zone-Set is a set of Zones, and it´s used for
separating Initiators from the Targets, or separating the sensitive networks.
Zones can overlap, but only one Zone Set can be active at a time. Zoning can be
enforced in two ways: soft and hard.
- Soft
Zoning, as in Software Based, handled by FCNS (Fibre Channel Name Server). If
an end device somehow knows the FC ID of a device outside its zone, it can
access that device.
- Hard
Zoning, or Hardware Based, where the membership is enforced using the ACLs.
As frames enter the switch, source-destination IDs are compared with permitted
combinations to allow the frame at wirespeed.
- Zone
Membership, which can be enforced using the various parameters.
Single Tier Extended Fabric is made of the N7K
or N5K as a “Parent”, and N2K as a fabric
extender (FEX) with a function of a ToR (Top of the Rack) Switch. VM-FEX (Fabrix
Extender) Technology enables the VM traffic flow to be identified
within the entire flow form ToR to the Parent switch. VIC Card, such as UCS
P8IE helps with VM-FEX.
