The GUI based updates suck and never work. In place upgrades suck and never work. I can’t imagine trying to do this with an HA scenario. I do an outage-requiring command line update because it’s the only one I had the patience to deal with, and my wireless environment can have scheduled downtime. Even TAC admits their upgrades suck, This was their statement to me last time I was trying to do in place upgrades:
ISSU is not optimal to perform EWC/WLC upgrades, we in TAC know it can be unreliable.
Even their docs suck. For example, the following page is how you perform such an upgrade, but it goes from setting the download profile to verifying the post-install upgrade completed successfully:
CLI Configuration:
9120-EWC(config)#wireless profile image-download default
9120-EWC(config-wireless-image-download-profile)#image-download-mode sftp
9120-EWC(config-wireless-image-download-profile-sftp)#sftp-image-server <SFTP-Server>
9120-EWC(config-wireless-image-download-profile-sftp)#sftp-image-path <path>
9120-EWC(config-wireless-image-download-profile-sftp)#sftp-username <user>
9120-EWC(config-wireless-image-download-profile-sftp)#sftp-password 0 <password>Verify
The CAPWAP State Machine logs in the APs flow as you normally would expect for any other AP Image Download process.
Huh? We never started the AP image download process, so how are we verifying that it worked?
Anyway, I have a 9130AXI acting as the embedded wireless controller (EWC), some 9120AXI’s, a 9214AXI outside AP, and a few older AP2802i devices.
Download the latest 9130AXI image zip, for example, C9800-AP-universalk9.17.15.01.zip. Stick it on your tftp server; an Ubuntu vm makes this easy with the tftp-hpa package. Your tftp server MUST be accessible to all of the AP’s in your environment, because they will download their hardware-specific images after the EWC upgrades itself and starts rolling updates through the environment.
Move the zip file to that tftp server under /srv/tftp/ and unzip it, set all files to world readable. Should leave you with a bunch of hardware-specific access point images along with the EWC software:
ap1g4
ap1g5
ap1g6
ap1g6a
ap1g7
ap1g8
ap3g3
C9800-AP-iosxe-wlc.bin
C9800-AP-universalk9.17.15.01.zip
controller_version.info
readme.txt
version.info
Code language: CSS (css)SSH to your EWC and check the status of your AP’s and images, just to ensure nothing is stuck in a weird state from a prior failed update, etc:
########################################################################################################
# #
# Welcome to the Cisco Catalyst 9800-AP Embedded Wireless Controller command line interface. #
# #
# Please see command reference guide for the complete list of supported commands for this release: #
# https://www.cisco.com/c/en/us/td/docs/wireless/embedded_wireless_controller_configuration_guide.html #
# #
########################################################################################################
ewc1#show ap image
Total number of APs : 10
Number of APs
Initiated : 0
Downloading : 0
Predownloading : 0
Completed downloading : 0
Completed predownloading : 0
Not Supported : 0
Failed to Predownload : 0
Predownload in progress : No
AP Name Primary Image Backup Image Predownload Status Predownload Version Next Retry Time Retry Count Method
------------------------------------------------------------------------------------------------------------------------------------------------------------------
ap-SEA-conf 17.14.0.79 17.11.0.155 None 0.0.0.0 0 0 N/A
ap-SEA-office 17.14.0.79 17.12.1.5 None 0.0.0.0 0 0 N/A
ap-SEA-outside 17.14.0.79 17.12.1.5 None 0.0.0.0 0 0 N/A
ap-SEA-warehouse 17.14.0.79 17.12.1.5 None 0.0.0.0 0 0 N/A
...
Code language: PHP (php)Looks good. Clear out old images to make room, old install config, old install stats, and the prior profile with download location:
ewc1#clear ap predownload statistics
ewc1#install remove profile default
ewc1#install remove inactive
install_remove: START Fri Aug 30 15:56:24 UTC 2024
Cleaning up unnecessary package files
Scanning boot directory for packages ... done.
Preparing packages list to delete ...
done.
The following files will be deleted:
[chassis 1/R0]:
/backup_image/C9800-AP-iosxe-wlc.bin
Do you want to remove the above files? [y/n]y
[chassis 1/R0]:
Deleting file backup_image:C9800-AP-iosxe-wlc.bin ... done.
SUCCESS: Files deleted.
--- Starting Post_Remove_Cleanup ---
Performing Post_Remove_Cleanup on all members
[1] Post_Remove_Cleanup package(s) on chassis 1/R0
[1] Finished Post_Remove_Cleanup on chassis 1/R0
Checking status of Post_Remove_Cleanup on [1/R0]
Post_Remove_Cleanup: Passed on [1/R0]
Finished Post_Remove_Cleanup
SUCCESS: install_remove Fri Aug 30 15:58:14 UTC 2024Code language: PHP (php)Well, I tried the above and got an error:
ewc1#install remove profile default
install_remove: START Fri Aug 30 16:48:59 UTC 2024
--- Starting Cleanup ---
Performing Cleanup on all members
[1] Cleanup package(s) on chassis 1/R0
ERROR: Invalid: File backup_image:C9800-AP-iosxe-wlc.bin is Invalid, please specify a valid filename
[1] Finished Cleanup on chassis 1/R0
Checking status of Cleanup on [1/R0]
Cleanup: Passed on []. Failed on [1/R0]
Finished Cleanup
FAILED: install_remove exit(1) Fri Aug 30 16:49:05 UTC 2024
Code language: PHP (php)Can’t find anything online about this other than other people having the similar error and also ignoring it. For example.
Config mode to replace upgrade config and set the default image download profile:
wireless profile image-download default
description "default image download profile"
image-download-mode tftp
tftp-image-path /
tftp-image-server 192.0.2.1
end
Example:
ewc1(config)#wireless profile image-download default
ewc1(config-wireless-image-download-profile)#description "default image download profile"
ewc1(config-wireless-image-download-profile)#image-download-mode tftp
ewc1(config-wireless-image-download-profile-tftp)# tftp-image-path /
ewc1(config-wireless-image-download-profile-tftp)# tftp-image-server 192.0.2.1
ewc1(config-wireless-image-download-profile-tftp)#end
Code language: PHP (php)Now let’s get out of config mode and perform the image download and upgrade as one outage-causing command (THIS WILL CAUSE DOWNTIME):
ewc1#install add file tftp://192.0.2.1/C9800-AP-iosxe-wlc.bin activate commit
install_add_activate_commit: START Fri Aug 30 16:11:40 UTC 2024
Downloading file tftp://192.0.2.1/C9800-AP-iosxe-wlc.bin
Finished downloading file tftp://192.0.2.1/C9800-AP-iosxe-wlc.bin to backup_image:C9800-AP-iosxe-wlc.bin
install_add: Adding PACKAGE
install_add: Checking whether new add is allowed ....
install_add_activate_commit: Adding PACKAGE
install_add_activate_commit: Checking whether new add is allowed ....
INSTALL_EXIT_CODE|26|
FAILED: install_add_activate_commit : Super package already added. Add operation not allowed. 'install remove profile default' can be used to discard added packagesCode language: PHP (php)It failed. Know why? A bug from 16.x, shown in this 2022 forum post, yet here I am running “Cisco IOS XE Software, Version 17.14.01” and experiencing the same error trying to go to 17.15.01, while having not experienced this going to 17.14.01. Bug ID CSCvq07097 suggests this is ‘fixed’. Guess not.
So, no idea what to try next. Maybe we copy the file local instead and install from there?
ewc1#copy tftp://192.0.2.1/C9800-AP-iosxe-wlc.bin bootflash:
Destination filename [C9800-AP-iosxe-wlc.bin]?
Accessing tftp://192.0.2.1/C9800-AP-iosxe-wlc.bin...
Loading C9800-AP-iosxe-wlc.bin from 192.0.2.1 (via GigabitEthernet0): !!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
[OK - 294958104 bytes]
294958104 bytes copied in 72.714 secs (4056414 bytes/sec)
Code language: PHP (php)Try the install:
ewc1#install add file bootflash:C9800-AP-iosxe-wlc.bin activate commit
install_add_activate_commit: START Fri Aug 30 16:34:47 UTC 2024
install_add_activate_commit: Adding PACKAGE
install_add_activate_commit: Checking whether new add is allowed ....
--- Starting initial file syncing ---
Info: Finished copying bootflash:C9800-AP-iosxe-wlc.bin to the selected chassis
Finished initial file syncing
--- Starting Add ---
Performing Add on all members
[1] Add package(s) on chassis 1/R0
[1] Finished Add on chassis 1/R0
Checking status of Add on [1/R0]
Add: Passed on [1/R0]
Finished Add
Image added. Version: 17.15.01.0.126
install_add_activate_commit: Activating PACKAGE
FAILED: install_add_activate_commit : Predownload of 17.15.01.0.126 has not finished
Code language: PHP (php)Well that’s lovely. What next? This forum post had someone else running into this in 2021 and it was suggested they isolate the access point by shutting down every other AP. Are you kidding? Why does Cisco break this shit in some new way with every release, then break shit again years later in the exact same way?
I decided lets start over again, after a reboot since one thread I found with similar errors said just reboot the damn thing. I’m also going to install add but not yet activate. So I did all the first steps of clearing out everything, then reloaded, then install added my image back:
ewc1#reload
Reload command is being issued on Active unit, this will reload the whole stack
Proceed with reload? [confirm]y
Connection to 192.168.0.100 closed by remote host.
ewc1#install add file tftp://192.168.0.1/C9800-AP-iosxe-wlc.bin
install_add: START Fri Aug 30 17:09:38 UTC 2024
Downloading file tftp://192.168.0.1/C9800-AP-iosxe-wlc.bin
Finished downloading file tftp://192.168.0.1/C9800-AP-iosxe-wlc.bin to backup_image:C9800-AP-iosxe-wlc.bin
install_add: Adding PACKAGE
install_add: Checking whether new add is allowed ....
--- Starting initial file syncing ---
Info: Finished copying backup_image:C9800-AP-iosxe-wlc.bin to the selected chassis
Finished initial file syncing
--- Starting Add ---
Performing Add on all members
[1] Add package(s) on chassis 1/R0
[1] Finished Add on chassis 1/R0
Checking status of Add on [1/R0]
Add: Passed on [1/R0]
Finished Add
Image added. Version: 17.15.01.0.126
SUCCESS: install_add Fri Aug 30 17:11:52 UTC 2024
Code language: PHP (php)Let’s see what’s going on. Joy! It seems to have taken the image and is predownloading it to the AP’s:
ewc1#show ap image
Total number of APs : 10
Number of APs
Initiated : 3
Downloading : 0
Predownloading : 0
Completed downloading : 0
Completed predownloading : 0
Not Supported : 0
Failed to Predownload : 0
Predownload in progress : No
AP Name Primary Image Backup Image Predownload Status Predownload Version Next Retry Time Retry Count Method
------------------------------------------------------------------------------------------------------------------------------------------------------------------
ap-SEA-conf 17.14.0.79 17.11.0.155 Initiated 17.15.1.6 0 0 N/A
ap-SEA-office 17.14.0.79 17.12.1.5 None 0.0.0.0 0 0 N/A
ap-SEA-outside 17.14.0.79 17.12.1.5 Initiated 17.15.1.6 0 0 N/A
ap-SEA-warehouse 17.14.0.79 17.12.1.5 Initiated 17.15.1.6 0 0 N/A
...Code language: PHP (php)Wait for the status to go complete to all AP’s and then you should be safe to cause an outage to complete the activation with minimal reload time:
ewc1#install activate
install_activate: START Fri Aug 30 18:45:40 UTC 2024
FAILED: install_activate : System configuration has been modified. Please save configuration and resubmit command.
ewc1#wr mem
Building configuration...
[OK]
ewc1#install activate
install_activate: START Fri Aug 30 18:45:58 UTC 2024
install_activate: Activating PACKAGE
This operation requires a reload of the system. Do you want to proceed? [y/n]y
--- Starting Activate ---
Performing Activate on all members
[1] Activate package(s) on chassis 1/R0
[1] Finished Activate on chassis 1/R0
Checking status of Activate on [1/R0]
Activate: Passed on [1/R0]
Finished Activate
Send model notification for install_activate before reload
Install will reload the system now!
SUCCESS: install_activate Fri Aug 30 18:47:56 UTC 2024
Code language: PHP (php)