Software
BIOS: version 07.17
NXOS: version 6.1(2)I3(3a)
BIOS compile time: 09/10/2014
NXOS image file is: bootflash:///n9000-dk9.6.1.2.I3.3a.bin
NXOS compile time: 1/26/2015 11:00:00 [01/26/2015 19:45:44]
Hardware
cisco Nexus9000 C9372PX chassis
Intel(R) Core(TM) i3-3227U C with 16402544 kB of memory.
Processor Board ID SAL1935N8A2
Device name: switch
bootflash: 51496280 kB
Kernel uptime is 0 day(s), 0 hour(s), 7 minute(s), 0 second(s)
Last reset
Reason: Unknown
System version: 6.1(2)I3(3a)
Service:
plugin
Core Plugin, Ethernet Plugin
By default your Leaf Switches will be in NX-OS mode. On the bootflash: of each Switch we will find the ACI image, NX-OS image and the EPLD file. If there is no ACI image, we will have to download it from Cisco website. Before we proceed with switching the operational mode from NX-OS to ACI, first we need to apply the EPLD upgrade:
switch# show install all impact epld bootflash:n9000-epld.6.1.2.I3.3a.img
Compatibility check:
Module Type Upgradable Impact Reason
------ ----------------- ---------- ---------- ------
1 SUP Yes disruptive Module Upgradable
Retrieving EPLD versions... Please wait.
Images will be upgraded according to following table:
Module Type EPLD Running-Version New-Version Upg-Required
------ ---- ------------- --------------- ----------- ------------
1 SUP MI FPGA 0x12 0x11 Yes
1 SUP IO FPGA 0x06 0x05 Yes
switch# install epld bootflash:n9000-epld.6.1.2.I3.3a.img module all
Compatibility check:
Module Type Upgradable Impact Reason
------ ----------------- ---------- ---------- ------
1 SUP Yes disruptive Module Upgradable
Retrieving EPLD versions... Please wait.
Images will be upgraded according to following table:
Module Type EPLD Running-Version New-Version Upg-Required
------ ---- ------------- --------------- ----------- ------------
1 SUP MI FPGA 0x12 0x11 Yes
1 SUP IO FPGA 0x06 0x05 Yes
The above modules require upgrade.
The switch will be reloaded at the end of the upgrade
Do you want to continue (y/n) ? [n] y
Proceeding to upgrade Modules.
Starting Module 1 EPLD Upgrade
Module 1 : MI FPGA [Programming] : 100.00% ( 64 of 64 sectors)
Module 1 : IO FPGA [Programming] : 100.00% ( 64 of 64 sectors)
Module 1 EPLD upgrade is successful.
Module Type Upgrade-Result
------ ------------------ --------------
1 SUP Success
EPLDs upgraded.
Once you have the entire fabric up and running in the same version of Firmware, and in the ACI mode, you can start configuring the APIC devices. It is of most importance that you decide and label each APIC Controller with a Number in the cluster, and that the Username and the Password you define match. Don’t be surprised we´re giving so much importance to this, because if you get the initial APIC configuration wrongly, it will be difficult to recover.
Start by assigning the Out-of-Bound Management IP addresses to all the Switchs and all the APICs. In our case, as an example, we set up a simple 1 Spine – 2 Leaf – 1 APIC architecture as a PoC kit, but the same principles apply regardless of the number of devices you bought. Besides the management IPs you will need:
- IP range for your VTEPs (tunnel endpoints). By default its 10.0.0.0/8.
- DNS and NTP Servers.
- Dedicated infrastructure VLAN.
ACI can be upgraded before you build the entire fabric and perform a Fabric Discovery from the APIC Cluster, in which case you would have to separately upgrade every switch and every APIC controller manually (use a TFTP or SCP server or a USB stick, and copy a new image to each device and boot form an ACI image), or you can start by building a fabric and perform an orchestrated upgrade controlling it all from the APIC SSH line. I personally prefer the second option, even more so knowing that the future upgrades will be performed that way.
The Upgrade Procedure is to start with the Switches Upgrade, and then, when the entire fabric is in the new version, you may proceed with the APIC Upgrade.
STEP 1: Upgrade Leaf and Spine Switches. Before we begind, lets check the Firmware version on the entire ACI architecture:
admin@APIC:/> firmware upgrade status
Node-Id Role Current-Firmware Target-Firmware Upgrade-Status
------------------------------------------------------------------------------------------
1 controller apic-1.0(3f) completeok 100
101 leaf n9000-11.0(3f) notscheduled 0
102 leaf n9000-11.0(3f) notscheduled 0
201 spine n9000-11.0(3f) notscheduled 0
You will notice in the output above that the APIC controller has the “complete OK” status in the Upgrade Status column. This is because the APIC had been turned on for the first time out-of-the-box.
Start by upgrading ONE of the Leaf Switches. In my case, I used Leaf1, or a Node 101 to a version 11.2m (compatible with the Dec2016 Brazos release of ACI). First we make sure that the new image is in the repository, and then we execute the upgrade:
admin@APIC:fwrepo> pwd
/firmware/fwrepos/fwrepo
admin@APIC:fwrepo> ls
aci-catalog-dk9.1.0.3f.bin aci-n9000-dk9.11.2.1m.bin boot md5sum
admin@APIC:/> firmware upgrade switch node 101 aci-n9000-dk9.11.2.1m.bin
Firmware Installation on Switch Scheduled
To check the upgrade status, use 'firmware upgrade status node <node-id>.
admin@APIC:/> firmware upgrade status node 101
Node-Id Role Current-Firmware Target-Firmware Upgrade-Status Progress-
-----------------------------------------------------------------------------------------------------
101 leaf n9000-11.0(3f) n9000-11.2(1m) inprogress 5
You should repeat this procedure for all the Leafs and Spines. If you´re in a production environment, be sure to use the High Availability you´ve previously taken care of (I hope), and update a Leaf Switch at a time, then a Spine Switches one by one, and after a while:
admin@APIC:pam.d> firmware upgrade status node 101
Node-Id Role Current-Firmware Target-Firmware Upgrade-Status Progress-
-----------------------------------------------------------------------------------------------------
101 leaf n9000-11.2(1m) n9000-11.2(1m) completeok 100
admin@APIC:pam.d> firmware upgrade status node 102
Node-Id Role Current-Firmware Target-Firmware Upgrade-Status Progress-
-----------------------------------------------------------------------------------------------------
102 leaf n9000-11.2(1m) n9000-11.2(1m) completeok 100
admin@APIC:pam.d> firmware upgrade status node 201
Node-Id Role Current-Firmware Target-Firmware Upgrade-Status Progress-
-----------------------------------------------------------------------------------------------------
201 spine n9000-11.2(1m) n9000-11.2(1m) completeok 100
STEP 2: Upgrade the APIC controller. In this example I´m doing the upgrade from ACI 1.0.3f to ACI 11.2.1m (Jan2016 version of the ACI release called Brazos). The first step is to copy the new firmware to the APIC Controller:
$ scp /Users/iCloud-MJ/Downloads/aci-n9000-dk9.11.2.1m.bin admin@10.20.70.92:
Application Policy Infrastructure Controller
admin@10.20.70.92's password:
aci-n9000-dk9.11.2.1m.bin 100% 532MB 6.4MB/s 01:23
IMPORTANT: When you copy your firmware files using the SCP, make sure that you have the correct privileges in the destination folder. If you don´t specify the destination folder on the APIC, it will be /home/admin/:
admin@APIC:~> pwd
/home/admin
admin@APIC:~> ls
aci aci-apic-dk9.1.2.1m.iso aci-n9000-dk9.11.2.1m.bin debug mit
Add the newly copied Firmware to your Firmware Repository:
admin@APIC:~> firmware add aci-n9000-dk9.11.2.1m.bin
Firmware Image aci-n9000-dk9.11.2.1m.bin is added to the repository
Be sure all the images have been correctly added to the List before you proceed. Notice that the catalog image will auto add to the Firmware Repository when you add the NEXUS 9k and APIC Upgrade Firmware images.
IMPORTANT: You will notice the CATALOG images in the below output. These are generated automatically once you have the Fabric and the Controller image correctly synchronized.
admin@APIC:~> firmware list
Name : aci-n9000-dk9.11.2.1m.bin
Type : switch
Version : 11.2(1m)
Size(Bytes) : 558351658
Release-Date : 2016-01-29T07:07:15.000+01:00
Download-Date : 2016-02-04T09:56:40.833+01:00
Name : aci-apic-dk9.1.2.1m.bin
Type : controller
Version : 1.2(1m)
Size(Bytes) : 3936555008
Release-Date : 2016-01-29T01:57:59.000+01:00
Download-Date : 2016-02-03T19:19:42.110+01:00
Name : aci-catalog-dk9.1.2.1m.bin
Type : catalog
Version : 1.2(1m)
Size(Bytes) : 25358
Release-Date : 2016-01-29T00:19:57.000+01:00
Download-Date : 2016-02-03T19:19:44.034+01:00
Name : aci-catalog-dk9.1.0.3f.bin
Type : catalog
Version : 1.0(3f)
Size(Bytes) : 18064
Release-Date : 2015-02-10T01:27:12.000+01:00
Download-Date : 2016-02-02T08:29:32.530+01:00
As you can see below, the APIC controller is still in the old 1.0(3f) version:
admin@APIC:~> firmware upgrade status
Node-Id Role Current-Firmware Target-Firmware Upgrade-Status
-----------------------------------------------------------------------------------
1 controller apic-1.0(3f) completeok 100
101 leaf n9000-11.2(1m) n9000-11.2(1m) completeok 100
102 leaf n9000-11.2(1m) n9000-11.2(1m) completeok 100
201 spine n9000-11.2(1m) n9000-11.2(1m) completeok 100
Start the APIC Upgrade, and check the status:
admin@APIC:~> firmware upgrade controllers aci-apic-dk9.1.2.1m.bin
Firmware Upgrade on Controllers has been scheduled.
The upgrade will be performed on one controller at a time in the background.
admin@APIC:~> firmware upgrade status
Node-Id Role Current-Firmware Target-Firmware Upgrade-Status Progress-
----------------------------------------------------------------------------------------------
1 controller apic-1.0(3f) apic-1.2(1m) inprogress 0
101 leaf n9000-11.2(1m) n9000-11.2(1m) completeok 100
102 leaf n9000-11.2(1m) n9000-11.2(1m) completeok 100
201 spine n9000-11.2(1m) n9000-11.2(1m) completeok 100
In a certain moment you will get this message:
admin@APIC:~>
Broadcast message from root@APIC
(unknown) at 10:57 ...
The system is going down for reboot NOW!
Once you get the control back, do not panic, because now the commands have changed, but as you will see from the “show version” output, the entire ACI architecture has now been upgraded:
Application Policy Infrastructure Controller
admin@10.20.70.92's password:
APIC# firmware upgrade status
Error: Invalid argument 'status '. Please check syntax in command reference guide
APIC#
APIC# show ver
Role Id Name Version
---------- ---------- ------------------------ --------------------
controller 1 APIC 1.2(1m)
leaf 101 Leaf1 n9000-11.2(1m)
leaf 102 Leaf2 n9000-11.2(1m)
spine 201 Spine n9000-11.2(1m)
You can now SSH into any of the Nodes from ACI. Nexus Switches in ACI mode do have CLI, but it's different. For example, the “?” won’t work, but the Double-ESC will (quickly press the “escape” key twice). Also the “include” and “begin” commands won’t work, but “grep” will :)
What happens with the VTEPs within the Fabric? During the initial ACI configuration I defined the 172.1.0.0/16 range for the VTEPs. Lets first connect to one of the Leaf Switches, and check the Local interfaces that belong to the VTEP IP range, and the routing table in the Overlay-1 VRF (VRF that is internally used by the fabric for VTEP routing):
Leaf2# show ip interface brief | grep 172
vlan7 172.1.0.30/27 protocol-up/link-up/admin-up
lo0 172.1.0.93/32 protocol-up/link-up/admin-up
lo1023 172.1.0.32/32 protocol-up/link-up/admin-up
From the output above we can clearly see that the loopbacks are in fact the VTEP interfaces. They are all /32, exactly as the VTEPs should be.
Leaf2# show vrf all
VRF-Name VRF-ID State Reason
black-hole 3 Up --
overlay-1 4 Up --
Leaf2# show ip route vrf overlay-1
IP Route Table for VRF "overlay-1"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
172.1.0.0/27, ubest/mbest: 1/0, attached, direct
*via 172.1.0.30, vlan7, [1/0], 20:56:08, direct
172.1.0.1/32, ubest/mbest: 1/0
*via 172.1.0.94, eth1/49.1, [115/12], 05:02:39, isis-isis_infra, L1
172.1.0.30/32, ubest/mbest: 1/0, attached
*via 172.1.0.30, vlan7, [1/0], 20:56:08, local, local
172.1.0.32/32, ubest/mbest: 2/0, attached, direct
*via 172.1.0.32, lo1023, [1/0], 20:54:12, local, local
*via 172.1.0.32, lo1023, [1/0], 20:54:12, direct
172.1.0.93/32, ubest/mbest: 2/0, attached, direct
*via 172.1.0.93, lo0, [1/0], 20:54:20, local, local
*via 172.1.0.93, lo0, [1/0], 20:54:20, direct
172.1.0.94/32, ubest/mbest: 1/0
*via 172.1.0.94, eth1/49.1, [115/2], 05:02:39, isis-isis_infra, L1
172.1.0.95/32, ubest/mbest: 1/0
*via 172.1.0.94, eth1/49.1, [115/3], 05:02:39, isis-isis_infra, L1
172.1.208.64/32, ubest/mbest: 1/0
*via 172.1.0.94, eth1/49.1, [115/2], 05:02:39, isis-isis_infra, L1
172.1.208.65/32, ubest/mbest: 1/0
*via 172.1.0.94, eth1/49.1, [115/2], 05:02:39, isis-isis_infra, L1
172.1.208.66/32, ubest/mbest: 1/0
*via 172.1.0.94, eth1/49.1, [115/2], 05:02:39, isis-isis_infra, L1
172.1.216.65/32, ubest/mbest: 1/0
*via 172.1.0.94, eth1/49.1, [115/2], 05:02:39, isis-isis_infra, L1
172.1.216.66/32, ubest/mbest: 1/0
*via 172.1.0.94, eth1/49.1, [115/2], 05:02:39, isis-isis_infra, L1
172.1.216.67/32, ubest/mbest: 1/0
*via 172.1.0.94, eth1/49.1, [115/2], 05:02:39, isis-isis_infra, L1
Leaf2#
The internal protocol of the Spine-Leaf Fabric of ACI is IS-IS, and as we can see in the Routing Table above - on the Leaf Switch all the VTEPs
Now lets check the Spine Switch:
Spine# show ip interface brief | grep 172
lo0 172.1.0.94/32 protocol-up/link-up/admin-up
lo1 172.1.208.65/32 protocol-up/link-up/admin-up
lo2 172.1.216.65/32 protocol-up/link-up/admin-up
lo3 172.1.208.64/32 protocol-up/link-up/admin-up
lo4 172.1.216.66/32 protocol-up/link-up/admin-up
lo5 172.1.216.67/32 protocol-up/link-up/admin-up
lo6 172.1.208.66/32 protocol-up/link-up/admin-up
In the above output we can see 7 VTEP interfaces on the Spine created so far. Important thing to notice at this point is that there are NO VLANs on the Spine Switch at this point, and on the Leaf there is only one automatically provisioned VLAN that is used for the APIC connection (APIC is plugged to e1/1 port of each of the 2 Leafs):
Spine# show vlan brief
VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
Spine#
Leaf1# show vlan br
VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
7 infra:default active Eth1/1
Leaf1#
Leaf2# show vlan br
VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
7 infra:default active Eth1/1
Leaf2#
