Cisco Data Center: UCS Basics

TIP: Be sure to use the HUU (Host Upgrade Utility) when adding the Blade to the UCS architecture, to make sure you have the latest drivers and the Firmware.

UCS (Unified Computing System) is a BladeCenter chassis that integrates compute, networking and Storage (FCoE) at backplane level whilst adding management capabilities through UCSM (USC Manager). UCSM seats on top of the blade chassis and are responsible for the overall management of the UCS stack.

UCS Upgrade Sequence
Download –> Update –> Activate. Start the upgrade from the UCS Manager, and then the Subordinate FI and later the Primary FI and the other components. The other way is the Host Firmware Package, which Updates using the Service Profiles.

VN Link is a Cisco invention to connect the VM directly to the Physical port without it being handled by VDS/1000v. One physical port can actually be logically separated into various vEthernet ports, where each one connects with exactly one vNIC on the Blade. This is especially comfortable to accommodate the vMotion. There is this new concept called the Interface Virtualizer, where there are special interfaces (VIC), where there is a VN-Tag, which is created by the vNIC when sent to the Physical Network (Host Interface on FEX device, and logically represented as the LIF on the Parent device), and removed by the Interface Virtualizer and the traffic is forwarded to the corresponding vNIC/VM. Nexus Switches support this technology, so the traffic tagged with the VN tag can go through FEX from the ToR to the Parent Switch (such as N5k or N7k).

UCS has 4 types of Ports:
  • Cluster Ports, 4 dual 10/100/1000 Ethernet clustering ports, which are used for connecting two UCSs together, they do sync and hearbeat. You direct connect these with a standard Ethernet cable, and they cannot be used for other purpose.
  • Management Port, Dedicated port for Out-of-Band Management.
  • SFP+ Ports may be used to connect to the 2100 Fabric Extenders (FeX) modules inside the 5100 Chassis (that contains the blades). They may also be used to connect up to your datacenter switching core or aggregation point.
  • Expansion Modules, used to provide further external connectivity. There are 3 types available: Ethernet, Fibre Channel plus Ethernet and Fibre Channel.

CIMC (Cisco Integrated Management Controller)

CIMC forms a part of the Blade, and the Blade connects with the IOM (Input Output Module) using a single link, or a port channel.

As long as the operating system installed on the UCS blades can understand and handle multipathing, it not only can achieve a direct logical connection to the LAN cloud, but it can also achieve an active/active connection via both Fabrics A and B, shown on the diagram below:

Configure the UCS Cluster

In the initial Setup the Wizard asks you if you´re the A or the B in the UCS Cluster.

UCS C-Series (Rack Mount)

A typical example (and the model that Cisco uses in the CCNP DC exam) is a C200 M2 rackable server (up to 2 5500/5600 Xeon CPUs, up to 192GB of RAM, 2 PCI 2.0 Slots).  These are more economical. Blades of this series are also equipped with the IMC (Management Chip), and don’t forget to press F8 during the boot and assign an IP from your management VLAN.
NIC on the C series UCS can be set to one of the 3 modes:
  • Management
  • Shared LoM (LAN on Motherboard, default)
  • Cisco Card

SCOPE command it used to check the state of the LAN adapter:
# scope chassis
chassis# scope adapter 1

There is a possibility of Local Storage on the C series, and there is a RAID Controller. There might be some problems with the Drivers and with the RAID modes you want to choose, but I wont get deep into that here.

RBAC – Role Based Access Control (Management of the User Privileges)

Cisco allows another level of flexibility regarding the AAA. The options for Authentication, or PROTOCOL REALMS are:
  • LDAP
  • Radius and TACACS+
  • Local Authentication (Native Authentication, tied to the local database)

Inside these REALMS you create the PROVIDER GROUPS, which is where you put the Individual Servers. Now we have the Provider Groups where each user chooses the Provider Group in order to authenticate against the correct method.

Authorization (under USER SERVICES) is provided using the ROLES, meaning providing the particular privileges, which is done creating the ORGANIZATIONS, and then relating them to LOCALE (Organizations and Locale are optional). Locales are based on the ORGANIZATIONS, which is how they get into the organizational structure (Operations, Finance or other).

UCS B-Series

The compute hardware managed by the UCS Manager software on the Fabric Interconnects can be B-Series (blades), C-Series (rackmount) or a combination of the two. B series are inside the Chassis, and then they connect with the rest of the Network Infrastructure.

If we use 32 ports for the Virtual Machines on the IOM (Input/Output Module), and one 10Gb Uplink port to the Fabric Interconnect, so we are having the 32:1 Oversubscription.
There are 2 important commands in UCS, which are:
  • CONNECT [nxos, iom…], for read only TS tools, such as Logs.

B Series automatically comments with the LAN and with the SAN.

B Series LAN Connectivity

Fabric Interconnect ports can be configured with different Port Personalities, but remember to configure the Fiber Interconnect as Ethernet or Fiber Channel before you continue with this. You can configure the Fabric Interconnect as:
  • Uplink Port (Border Links), going Northbound to the LAN Equipment, and these should be configured as Port Channels (1-256) using LACP only.
  • Server Port, going to the UCS IOM modules.
  • FCoE Port
  • FCoE Storage Ports
  • Appliance Port

The default mode is called End Host Virtualization (EHV) mode, and it doesn’t participate the Spanning Tree. There are two types of links in the Fabric Interconnect:
  • Server links, connecting to the UCS IOMs, which lead to the Blade servers.
  • Border Ports or Border Links, connecting to the Upstream LAN. These links don’t participate the STP and they do NOT forward traffic between each other. Server Links are being PINNED to the Border Links.

B Series SAN (Storage Area Network) Connectivity

The ustream Storage switches need (VSAN or some other) to be in the NPIV mode, so that they support the FCIDs. The default mode of the Fabric Interconnect for the SAN connectivity is the NPV. All the switching will be done in the upstream Switches. There are two types of Fabric Interconnect ports:
  • F Ports, to the Servers (Blades).
  • NP ports in Proxy Mode, leading to the Network Equipment, like VSANs.
  • The PINNING also occurs between the F ports and the Virtual SANs (VSANs), and it will happen in the Round Robin fashion.

FCoE needs it's own VLAN, and it cannot overlap the already used VLANs. To connect the FCoE link to SAN there is also a special switch that understands the Fiber Channel Switching and the Eternet Switching, and its called FCF.
Fabric Interconnect is being treated as a Fibre Channel Mode in this case. Each and every server link is pinned to ONE Uplink. In the storage world there are also Port Personalities, exactly like in the B Series LANs. vNIC is used with LAN, while with the storage we have the concept of vHBA (virtual Host Bus Adapter). In SAN environment we are not using the MAC addressing, but the WWN (World Wide Names), and use them in the Service Profiles.
FCoE VLAN si used for carrying FCoE traffic, and it cannot overlap with any other VLANs. We can use the SAN pin group in order to pin the particular servers to particular ports.

Fabric Interconnect

Fabric Interconnect is basically the L2 Switch, so it’s really similar to the in/host implementation of the N5k. By default it´s not in the Ethernet Switching mode.
Devices are interconnected using two 1Gbps CAT6 links. This redundancy works with Fabric Interconnect, and the UCS Manager Controller App is in charge of the Cluster Redundancy, and handles the Active – Standby roles. UCS needs to Initialize and do the Discovery before it´s usable. UCS is built with Data Management Engine (DME). SSO (Stateful SwitchOver) is also implemented between the Fabric Interconnects. In order for these 2 options to work, we need to cable the FI devices properly, so L1 and L2 are used for the Heartbeats exchange. There are also Management ports on each FI where we assign the management IP address, and there are 8 IOM ports. There are 2 cool commands that you should use to check the discovery process:
# show cluster state
# show platform software cmcctrl dmclient all

By default, the UCS’s main components, the Fabric Interconnects, operate in End-Host mode. By doing this, the UCS system literally looks like a big computer with a bunch of ports to the northbound LAN switches. With this, as long as a company has a standards-compliant network, they are able to simply “drop in” the UCS solution to their existing infrastructure. Because their existing network sees the UCS as just a “host”, the switches don’t think they’re connecting to another switch and therefore, no switching changes are needed.
The Fabric Interconnect uses the EHV (End Host Virtualization) Mode, so that the LAN switches see it as an End Host and not as a Switch, so the Spanning Tree isn´t blocking any ports, all ports are actually ACTIVE. In this mode Server to Server traffic is switched by the FI. The other mode that can be set is the Ethernet Switching Mode, where the Spanning Tree is activated and the Fabric Interconnect starts to act as a Switch.

Service Profiles

This concept is really important for both, C and B Series. Service Profile is a Logical profile of a Blade Server. It´s used to define the policies, and there are 2 groups of policies:
  • Configuration Policies.
  • Operation Policies (Power consumption, firmware and so on).

The important Policy in all the troubleshooting tasks is the Maintenance Policy, which allows you to restart the Blade immediately, User Acknowledged or Scheduled.
Power Policy is set within the Inventory tab, and it can be set to:
  • Non Redundant, as in – exactly the combined power we need
  • N+1, which means the needed ones, and an additional one in a Standby mode
  • Grid, which means the redundancy at a grid level, meaning that we have 2 Power Supplies at one company, and 2 on another.

To see all the ports and the Roles and the Review, go to: Equipment -> Fabric Interconnects -> FI A, and go to “LAN Uplink Manager”:

And for the end, a great Home Lab instruction blog:

Most Popular Posts