Cisco ACI Guide for Humans, Part 2: Upgrade Cisco ACI

First time we “unpack” ACI, we will find a certain number of potential Spine and potential Leaf switches, and hopefully 3 (or 5) APIC Controllers. We will rack the entire fabric, interconnect every Spine to every Leaf with a single 40G cable, and connect every APIC to 2 Leaf Switches. We would power on the devices, and before we even start configuring the APIC Cluster, we need to console to each Switch and verify if its running ACI mode or a NX-OS mode by executing the “show version” command. These are the details of the Fabric we used in our Lab:

Software
  BIOS: version 07.17
  NXOS: version 6.1(2)I3(3a)
  BIOS compile time:  09/10/2014
  NXOS image file is: bootflash:///n9000-dk9.6.1.2.I3.3a.bin
  NXOS compile time:  1/26/2015 11:00:00 [01/26/2015 19:45:44]

Hardware
  cisco Nexus9000 C9372PX chassis
  Intel(R) Core(TM) i3-3227U C with 16402544 kB of memory.
  Processor Board ID SAL1935N8A2

  Device name: switch
  bootflash:   51496280 kB
Kernel uptime is 0 day(s), 0 hour(s), 7 minute(s), 0 second(s)

Last reset
  Reason: Unknown
  System version: 6.1(2)I3(3a)
  Service:

plugin
  Core Plugin, Ethernet Plugin


By default your Leaf Switches will be in NX-OS mode. On the bootflash: of each Switch we will find the ACI image, NX-OS image and the EPLD file. If there is no ACI image, we will have to download it from Cisco website. Before we proceed with switching the operational mode from NX-OS to ACI, first we need to apply the EPLD upgrade:

switch# show install all impact epld bootflash:n9000-epld.6.1.2.I3.3a.img
Compatibility check:
Module        Type         Upgradable        Impact   Reason
------  -----------------  ----------    ----------   ------
     1            SUP           Yes       disruptive   Module Upgradable

Retrieving EPLD versions... Please wait.

Images will be upgraded according to following table:
Module  Type   EPLD              Running-Version   New-Version  Upg-Required
------  ----  -------------      ---------------   -----------  ------------
     1   SUP  MI FPGA                   0x12        0x11            Yes
     1   SUP  IO FPGA                   0x06        0x05            Yes
switch# install epld bootflash:n9000-epld.6.1.2.I3.3a.img module all
Compatibility check:
Module        Type         Upgradable        Impact   Reason
------  -----------------  ----------    ----------   ------
     1            SUP           Yes       disruptive   Module Upgradable

Retrieving EPLD versions... Please wait.

Images will be upgraded according to following table:
Module  Type   EPLD              Running-Version   New-Version  Upg-Required
------  ----  -------------      ---------------   -----------  ------------
     1   SUP  MI FPGA                   0x12        0x11            Yes
     1   SUP  IO FPGA                   0x06        0x05            Yes
The above modules require upgrade.
The switch will be reloaded at the end of the upgrade
Do you want to continue (y/n) ?  [n] y

Proceeding to upgrade Modules.

 Starting Module 1 EPLD Upgrade

Module 1 : MI FPGA [Programming] : 100.00% (     64 of      64 sectors)
Module 1 : IO FPGA [Programming] : 100.00% (     64 of      64 sectors)
Module 1 EPLD upgrade is successful.
Module        Type  Upgrade-Result
------  ------------------  --------------
     1         SUP         Success


EPLDs upgraded.


Once you have the entire fabric up and running in the same version of Firmware, and in the ACI mode, you can start configuring the APIC devices. It is of most importance that you decide and label each APIC Controller with a Number in the cluster, and that the Username and the Password you define match. Don’t be surprised we´re giving so much importance to this, because if you get the initial APIC configuration wrongly, it will be difficult to recover.
Start by assigning the Out-of-Bound Management IP addresses to all the Switchs and all the APICs. In our case, as an example, we set up a simple 1 Spine – 2 Leaf – 1 APIC architecture as a PoC kit, but the same principles apply regardless of the number of devices you bought. Besides the management IPs you will need:


  • IP range for your VTEPs (tunnel endpoints). By default its 10.0.0.0/8.
  • DNS and NTP Servers.
  • Dedicated infrastructure VLAN.


ACI can be upgraded before you build the entire fabric and perform a Fabric Discovery from the APIC Cluster, in which case you would have to separately upgrade every switch and every APIC controller manually (use a TFTP or SCP server or a USB stick, and copy a new image to each device and boot form an ACI image), or you can start by building a fabric and perform an orchestrated upgrade controlling it all from the APIC SSH line. I personally prefer the second option, even more so knowing that the future upgrades will be performed that way.

The Upgrade Procedure is to start with the Switches Upgrade, and then, when the entire fabric is in the new version, you may proceed with the APIC Upgrade.

STEP 1: Upgrade Leaf and Spine Switches. Before we begind, lets check the Firmware version on the entire ACI architecture:

admin@APIC:/> firmware upgrade status
Node-Id    Role            Current-Firmware     Target-Firmware      Upgrade-Status  
------------------------------------------------------------------------------------------
1          controller      apic-1.0(3f)                              completeok      100
101        leaf            n9000-11.0(3f)                            notscheduled    0
102        leaf            n9000-11.0(3f)                            notscheduled    0
201        spine           n9000-11.0(3f)                            notscheduled    0

You will notice in the output above that the APIC controller has the “complete OK” status in the Upgrade Status column. This is because the APIC had been turned on for the first time out-of-the-box.

Start by upgrading ONE of the Leaf Switches. In my case, I used Leaf1, or a Node 101 to a version 11.2m (compatible with the Dec2016 Brazos release of ACI). First we make sure that the new image is in the repository, and then we execute the upgrade:

admin@APIC:fwrepo> pwd
/firmware/fwrepos/fwrepo
admin@APIC:fwrepo> ls
aci-catalog-dk9.1.0.3f.bin  aci-n9000-dk9.11.2.1m.bin  boot  md5sum

admin@APIC:/> firmware upgrade switch node 101 aci-n9000-dk9.11.2.1m.bin
Firmware Installation on Switch Scheduled


To check the upgrade status, use 'firmware upgrade status node <node-id>.

admin@APIC:/> firmware upgrade status node 101
Node-Id    Role            Current-Firmware     Target-Firmware      Upgrade-Status  Progress-
-----------------------------------------------------------------------------------------------------
101        leaf            n9000-11.0(3f)       n9000-11.2(1m)       inprogress      5


You should repeat this procedure for all the Leafs and Spines. If you´re in a production environment, be sure to use the High Availability you´ve previously taken care of (I hope), and update a Leaf Switch at a time, then a Spine Switches one by one, and after a while:

admin@APIC:pam.d> firmware upgrade status node 101
Node-Id    Role            Current-Firmware     Target-Firmware      Upgrade-Status  Progress-
-----------------------------------------------------------------------------------------------------
101        leaf            n9000-11.2(1m)       n9000-11.2(1m)       completeok      100


admin@APIC:pam.d> firmware upgrade status node 102
Node-Id    Role            Current-Firmware     Target-Firmware      Upgrade-Status  Progress-
-----------------------------------------------------------------------------------------------------
102        leaf            n9000-11.2(1m)       n9000-11.2(1m)       completeok      100


admin@APIC:pam.d> firmware upgrade status node 201
Node-Id    Role            Current-Firmware     Target-Firmware      Upgrade-Status  Progress-
-----------------------------------------------------------------------------------------------------
201        spine           n9000-11.2(1m)       n9000-11.2(1m)       completeok      100


STEP 2: Upgrade the APIC controller. In this example I´m doing the upgrade from ACI 1.0.3f to ACI 11.2.1m (Jan2016 version of the ACI release called Brazos). The first step is to copy the new firmware to the APIC Controller:

$ scp /Users/iCloud-MJ/Downloads/aci-n9000-dk9.11.2.1m.bin admin@10.20.70.92:
Application Policy Infrastructure Controller
admin@10.20.70.92's password:
aci-n9000-dk9.11.2.1m.bin                                                   100%  532MB   6.4MB/s   01:23


IMPORTANT: When you copy your firmware files using the SCP, make sure that you have the correct privileges in the destination folder. If you don´t specify the destination folder on the APIC, it will be /home/admin/:

admin@APIC:~> pwd
/home/admin
admin@APIC:~> ls
aci  aci-apic-dk9.1.2.1m.iso  aci-n9000-dk9.11.2.1m.bin  debug mit

Add the newly copied Firmware to your Firmware Repository:

admin@APIC:~> firmware add aci-n9000-dk9.11.2.1m.bin
Firmware Image aci-n9000-dk9.11.2.1m.bin is added to the repository

Be sure all the images have been correctly added to the List before you proceed. Notice that the catalog image will auto add to the Firmware Repository when you add the NEXUS 9k and APIC Upgrade Firmware images.

IMPORTANT: You will notice the CATALOG images in the below output. These are generated automatically once you have the Fabric and the Controller image correctly synchronized.

admin@APIC:~> firmware list
Name                 : aci-n9000-dk9.11.2.1m.bin
Type                 : switch
Version              : 11.2(1m)
Size(Bytes)          : 558351658
Release-Date         : 2016-01-29T07:07:15.000+01:00
Download-Date        : 2016-02-04T09:56:40.833+01:00

Name                 : aci-apic-dk9.1.2.1m.bin
Type                 : controller
Version              : 1.2(1m)
Size(Bytes)          : 3936555008
Release-Date         : 2016-01-29T01:57:59.000+01:00
Download-Date        : 2016-02-03T19:19:42.110+01:00

Name                 : aci-catalog-dk9.1.2.1m.bin
Type                 : catalog
Version              : 1.2(1m)
Size(Bytes)          : 25358
Release-Date         : 2016-01-29T00:19:57.000+01:00
Download-Date        : 2016-02-03T19:19:44.034+01:00

Name                 : aci-catalog-dk9.1.0.3f.bin
Type                 : catalog
Version              : 1.0(3f)
Size(Bytes)          : 18064
Release-Date         : 2015-02-10T01:27:12.000+01:00
Download-Date        : 2016-02-02T08:29:32.530+01:00


As you can see below, the APIC controller is still in the old 1.0(3f) version:

admin@APIC:~> firmware upgrade status
Node-Id    Role            Current-Firmware     Target-Firmware      Upgrade-Status  
-----------------------------------------------------------------------------------
1          controller      apic-1.0(3f)                              completeok      100
101        leaf            n9000-11.2(1m)       n9000-11.2(1m)       completeok      100
102        leaf            n9000-11.2(1m)       n9000-11.2(1m)       completeok      100
201        spine           n9000-11.2(1m)       n9000-11.2(1m)       completeok      100


Start the APIC Upgrade, and check the status:

admin@APIC:~> firmware upgrade controllers aci-apic-dk9.1.2.1m.bin
Firmware Upgrade on Controllers has been scheduled.
The upgrade will be performed on one controller at a time in the background.

admin@APIC:~> firmware upgrade status
Node-Id    Role            Current-Firmware     Target-Firmware      Upgrade-Status  Progress-
----------------------------------------------------------------------------------------------
1          controller      apic-1.0(3f)         apic-1.2(1m)         inprogress      0
101        leaf            n9000-11.2(1m)       n9000-11.2(1m)       completeok      100
102        leaf            n9000-11.2(1m)       n9000-11.2(1m)       completeok      100
201        spine           n9000-11.2(1m)       n9000-11.2(1m)       completeok      100


In a certain moment you will get this message:
admin@APIC:~>
Broadcast message from root@APIC
(unknown) at 10:57 ...

The system is going down for reboot NOW!


Once you get the control back, do not panic, because now the commands have changed, but as you will see from the “show version” output, the entire ACI architecture has now been upgraded:

Application Policy Infrastructure Controller
admin@10.20.70.92's password:
APIC# firmware upgrade status
Error: Invalid argument 'status '. Please check syntax in command reference guide
APIC#
APIC# show ver
 Role        Id          Name                      Version
 ----------  ----------  ------------------------  --------------------
 controller  1           APIC                      1.2(1m)
 leaf        101         Leaf1                     n9000-11.2(1m)
 leaf        102         Leaf2                     n9000-11.2(1m)
 spine       201         Spine                     n9000-11.2(1m)


You can now SSH into any of the Nodes from ACI. Nexus Switches in ACI mode do have CLI, but it's different. For example, the “?” won’t work, but the Double-ESC will (quickly press the “escape” key twice). Also the “include” and “begin” commands won’t work, but “grep” will :)
What happens with the VTEPs within the Fabric? During the initial ACI configuration I defined the 172.1.0.0/16 range for the VTEPs. Lets first connect to one of the Leaf Switches, and check the Local interfaces that belong to the VTEP IP range, and the routing table in the Overlay-1 VRF (VRF that is internally used by the fabric for VTEP routing):

Leaf2# show ip interface brief | grep 172
vlan7                172.1.0.30/27        protocol-up/link-up/admin-up
lo0                  172.1.0.93/32        protocol-up/link-up/admin-up
lo1023               172.1.0.32/32        protocol-up/link-up/admin-up


From the output above we can clearly see that the loopbacks are in fact the VTEP interfaces. They are all /32, exactly as the VTEPs should be.

Leaf2# show vrf all
 VRF-Name                           VRF-ID State    Reason
 black-hole                              3 Up       --
 overlay-1                               4 Up       --

Leaf2# show ip route vrf overlay-1
IP Route Table for VRF "overlay-1"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>

172.1.0.0/27, ubest/mbest: 1/0, attached, direct
    *via 172.1.0.30, vlan7, [1/0], 20:56:08, direct
172.1.0.1/32, ubest/mbest: 1/0
    *via 172.1.0.94, eth1/49.1, [115/12], 05:02:39, isis-isis_infra, L1
172.1.0.30/32, ubest/mbest: 1/0, attached
    *via 172.1.0.30, vlan7, [1/0], 20:56:08, local, local
172.1.0.32/32, ubest/mbest: 2/0, attached, direct
    *via 172.1.0.32, lo1023, [1/0], 20:54:12, local, local
    *via 172.1.0.32, lo1023, [1/0], 20:54:12, direct
172.1.0.93/32, ubest/mbest: 2/0, attached, direct
    *via 172.1.0.93, lo0, [1/0], 20:54:20, local, local
    *via 172.1.0.93, lo0, [1/0], 20:54:20, direct
172.1.0.94/32, ubest/mbest: 1/0
    *via 172.1.0.94, eth1/49.1, [115/2], 05:02:39, isis-isis_infra, L1
172.1.0.95/32, ubest/mbest: 1/0
    *via 172.1.0.94, eth1/49.1, [115/3], 05:02:39, isis-isis_infra, L1
172.1.208.64/32, ubest/mbest: 1/0
    *via 172.1.0.94, eth1/49.1, [115/2], 05:02:39, isis-isis_infra, L1
172.1.208.65/32, ubest/mbest: 1/0
    *via 172.1.0.94, eth1/49.1, [115/2], 05:02:39, isis-isis_infra, L1
172.1.208.66/32, ubest/mbest: 1/0
    *via 172.1.0.94, eth1/49.1, [115/2], 05:02:39, isis-isis_infra, L1
172.1.216.65/32, ubest/mbest: 1/0
    *via 172.1.0.94, eth1/49.1, [115/2], 05:02:39, isis-isis_infra, L1
172.1.216.66/32, ubest/mbest: 1/0
    *via 172.1.0.94, eth1/49.1, [115/2], 05:02:39, isis-isis_infra, L1
172.1.216.67/32, ubest/mbest: 1/0
    *via 172.1.0.94, eth1/49.1, [115/2], 05:02:39, isis-isis_infra, L1
Leaf2#


The internal protocol of the Spine-Leaf Fabric of ACI is IS-IS, and as we can see in the Routing Table above - on the Leaf Switch all the VTEPs
Now lets check the Spine Switch:

Spine# show ip interface brief | grep 172
lo0                  172.1.0.94/32        protocol-up/link-up/admin-up
lo1                  172.1.208.65/32      protocol-up/link-up/admin-up
lo2                  172.1.216.65/32      protocol-up/link-up/admin-up
lo3                  172.1.208.64/32      protocol-up/link-up/admin-up
lo4                  172.1.216.66/32      protocol-up/link-up/admin-up
lo5                  172.1.216.67/32      protocol-up/link-up/admin-up
lo6                  172.1.208.66/32      protocol-up/link-up/admin-up

In the above output we can see 7 VTEP interfaces on the Spine created so far. Important thing to notice at this point is that there are NO VLANs on the Spine Switch at this point, and on the Leaf there is only one automatically provisioned VLAN that is used for the APIC connection (APIC is plugged to e1/1 port of each of the 2 Leafs):

Spine# show vlan brief

 VLAN Name                             Status    Ports
 ---- -------------------------------- --------- -------------------------------
Spine#


Leaf1# show vlan br

 VLAN Name                             Status    Ports
 ---- -------------------------------- --------- -------------------------------
 7    infra:default                    active    Eth1/1
Leaf1#

Leaf2# show vlan br

 VLAN Name                             Status    Ports
 ---- -------------------------------- --------- -------------------------------
 7    infra:default                    active    Eth1/1
Leaf2#


Most Popular Posts