Tech Tips For Cisco Devices

Monday, June 22, 2015

Cisco 3750x FRULink 10G service module in pass-thru mode

%PLATFORM_SM10G-3-SW_VERSION_MISMATCH: The FRULink 10G Service Module (C3KX-SM-10G) in switch 1 has a software version that is incompatible with the IOS software version. Please update the software. Module is in pass-thru mode.

Switch#sh switch service-modules

Switch/Stack supports service module CPU version: 03.00.78

Temperature CPU

Switch# H/W Status (CPU/FPGA) CPU Link Version

-----------------------------------------------------------------

1 OK 75C/74C ver-mismatch 03.00.65

2 OK 69C/70C ver-mismatch 03.00.65

Switch#archive download-sw flash:c3kx-sm10g-tar.150-2.SE7.tar
examining image...
extracting info (100 bytes)
extracting c3kx-sm10g-mz.150-2.SE7/info (499 bytes)
extracting info (100 bytes)

Stacking Version Number: 1.49

System Type: 0x00010002
Ios Image File Size: 0x017BDA00
Total Image File Size: 0x017BDA00
Minimum Dram required: 0x08000000
Image Suffix: sm10g-150-2.SE7
Image Directory: c3kx-sm10g-mz.150-2.SE7
Image Name: c3kx-sm10g-mz.150-2.SE7.bin
Image Feature: IP|LAYER_3|MIN_DRAM_MEG=128
FRU Module Version: 03.00.78

Updating FRU Module on switch 1...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
Updating FRU FPGA image...

FPGA image update complete.

Updating FRU Module on switch 2...
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
All software images installed.

Switch#sh switch service-modules

Switch/Stack supports service module CPU version: 03.00.78

Temperature CPU

Switch# H/W Status (CPU/FPGA) CPU Link Version

-----------------------------------------------------------------

1 OK 75C/74C ver-mismatch 03.00.65

2 OK 69C/70C ver-mismatch 03.00.65

Switch#switch 1 frulink reload

This will cause the FRULink's traffic and links to be disrupted. Are you sure? (yes/[no]): yes

FRULink on Switch 1 reloaded.

TPB-12thFl-3750x#sh switch service-modules

Switch/Stack supports service module CPU version: 03.00.78

Temperature CPU

Switch# H/W Status (CPU/FPGA) CPU Link Version

-----------------------------------------------------------------

1 OK 73C/73C connected 03.00.78

2 OK 71C/70C ver-mismatch 03.00.65

Switch#switch 2 frulink reload

This will cause the FRULink's traffic and links to be disrupted. Are you sure? (yes/[no]): yes

FRULink on Switch 2 reloaded.

TPB-12thFl-3750x#sh switch service-modules

Switch/Stack supports service module CPU version: 03.00.78

Temperature CPU

Switch# H/W Status (CPU/FPGA) CPU Link Version

-----------------------------------------------------------------

Switch#sh switch service-modules

Switch/Stack supports service module CPU version: 03.00.78

Temperature CPU

Switch# H/W Status (CPU/FPGA) CPU Link Version

-----------------------------------------------------------------

1 OK 73C/73C connected 03.00.78

2 OK 71C/70C notconnected N/A

TPB-12thFl-3750x#sh switch service-modules

Switch/Stack supports service module CPU version: 03.00.78

Temperature CPU

Switch# H/W Status (CPU/FPGA) CPU Link Version

-----------------------------------------------------------------

1 OK 73C/73C connected 03.00.78

2 OK 71C/71C checking-ver N/A

TPB-12thFl-3750x#sh switch service-modules

Switch/Stack supports service module CPU version: 03.00.78

Temperature CPU

Switch# H/W Status (CPU/FPGA) CPU Link Version

-----------------------------------------------------------------

1 OK 73C/73C connected 03.00.78

2 OK 69C/71C connected 03.00.78

Tuesday, June 9, 2015

Helpful IOS-XE Commands

We are currently suffering from a couple bugs (one internal which causes the switch to reboot and one cosmetic bug that just fills up our syslog server). Consequently, we are upgrading a total of 92 Cisco 3850 switches over the course of 2 nights.

I decided to set up constant pings to our devices and use the "reload at" command to reload all of the switch stacks on all 6 floors of one building at the same time.

We are currently on code version 03.03.05 and are upgrading to 03.07.00 at the advice of Cisco TAC.

I couldn't find much good documentation on the "reload at" command so I used my test switch stack to figure it out (using the question mark on the CLI is kind of ambiguous).

Below shows what I typed into the CLI:

Switch#reload ?

/noverify Don't verify file signature before reload.

/verify Verify file signature before reload.

LINE Reason for reload

at Reload at a specific time/date

cancel Cancel pending reload

in Reload after a time interval

slot Slot number card

standby-cpu Standby RP

<cr>

Switch#reload at ?

hh:mm Time to reload (hh:mm)

Switch#reload at 21:05 ?

<1-31> Day of the month

LINE Reason for reload

MONTH Month of the year

<cr>

Switch#reload at 21:05 PDT ?

LINE <cr>

Switch#reload at 21:05 PDT Jun

Switch#reload at 21:05 PDT Jun ?

LINE <cr>

Switch#reload at 21:05 ?

<1-31> Day of the month

LINE Reason for reload

MONTH Month of the year

<cr>

Switch#reload at 21:05 ?

<1-31> Day of the month

LINE Reason for reload

MONTH Month of the year

<cr>

Switch#reload at 21:05 Upgrade ?

LINE <cr>

That is what I mean by ambiguous... It keeps giving me that "LINE" command.

After some trial and error, I figured out the following command works:

Switch#reload at 21:05 6/09/15 Upgrade

Reload scheduled for 21:05:00 PDT Tue Jun 9 2015 (in 6 hours and 36 minutes) by console

Reload reason: 6/09/15 UpgradeReload command is being issued on Active unit, this will reload the whole stack

Proceed with reload? [confirm]

Switch#

Jun 9 21:28:27.309: %SYS-5-SCHEDULED_RELOAD: Reload requested for 21:05:00 PDT Tue Jun 9 2015 at 14:28:18 PDT Tue Jun 9 2015 by console. Reload Reason: 6/09/15 Upgrade.

Jun 9 21:28:27.309: %SYS-5-SCHEDULED_RELOAD: Reload requested for

Switch#sh ver

Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.03.02SE RELEASE SOFTWARE (fc2)

Technical Support: http://www.cisco.com/techsupport

Compiled Thu 20-Feb-14 21:17 by prod_rel_team

Output truncated..

Switch Ports Model SW Version SW Image Mode

------ ----- ----- ---------- ---------- ----

* 1 32 WS-C3850-24P 03.03.02SE cat3k_caa-universalk9 INSTALL

2 32 WS-C3850-24P 03.03.02SE cat3k_caa-universalk9 INSTALL

Switch 02

---------

Switch uptime : 1 week, 5 days, 3 hours, 39 minutes

Base Ethernet MAC Address :

Motherboard Assembly Number :

Motherboard Serial Number :

Model Revision Number : K0

Motherboard Revision Number : C0

Model Number : WS-C3850-24P

System Serial Number :

Configuration register is 0x102

Reload scheduled for 21:05:00 PDT Tue Jun 9 2015 (in 6 hours and 36 minutes) by console

Reload reason: 6/09/15 Upgrade

As you can see, this shows I am using code version 03.03.02. This is my non-production stack.

You can also issue the following command to reload on a specific day:
Switch#reload at 20:00 07 Dec Upgrade
or a specific time:
Switch#reload in 120 (reloads the switch in 2 hours)

Update - 6/10/2015
Reload/Upgrade successful!

Here is what the switch does when you are logged in to it.

Reload scheduled for 21:05:00 PDT Tue Jun 9 2015 (in 1 minute and 20 seconds) by xxxxxxxxx on vty0 (x.x.x.x)
Reload reason: 6/09/15 Upgrade
Swtich#sh clo
21:03:56.858 PDT Tue Jun 9 2015
Switch#

***
*** --- SHUTDOWN in 0:01:00 ---
***

Switch#

***
*** --- SHUTDOWN NOW ---

***

Update - 4/29/2016

We are no longer on IOS 03.07.00 and have actually downgraded to 03.06.04 which is the recommended release. We were having some "bug"gy behavior.

Monday, December 1, 2014

Cisco 3850 stack with a switch in the V-Mismatch state

FYI, this is done in a non-production environment. I just booted a 2-member stack and got the following message on a 'show switch' command:

Switch#sh sw
Switch/Stack Mac Address : 00e1.6d09.ae00 - Local Mac Address
Mac persistency wait time: Indefinite
H/W Current
Switch# Role Mac Address Priority Version State
------------------------------------------------------------
*1 Active 00e1.6d09.ae00 1 M0 Ready
2 Member 00e1.6d51.6780 1 0 V-Mismatch

In my case, in order to save some time, I copied the .bin file from flash: to flash-2: using the following command:

copy flash:cat3k_caa-universalk9.SPA.03.03.02.SE.150-1.EZ2.bin flash-2:

Then, you use the following command to install the correct version to the switch that has the version mismatch:

Switch#software auto-upgrade

% Auto upgrade has been initiated for the following incompatible switches: 2

INFO level system messages will be generated to provide status information during
the auto upgrade process

Switch#sh log

*Dec 1 10:29:47.546: %INSTALLER-6-AUTO_UPGRADE_SW_INITIATED: 1 installer: Auto upgrade initiated for switch 2
*Dec 1 10:29:47.547: %INSTALLER-6-AUTO_UPGRADE_SW: 1 installer: Searching stack for software to upgrade switch 2
*Dec 1 10:29:47.548: %INSTALLER-6-AUTO_UPGRADE_SW: 1 installer: Found donor switch 1 to auto upgrade switch 2
*Dec 1 10:29:47.548: %INSTALLER-6-AUTO_UPGRADE_SW: 1 installer: Upgrading switch 2 with software from switch 1
*Dec 1 10:31:03.973: %INSTALLER-6-AUTO_UPGRADE_SW: 1 installer: Finished installing software on switch 2
*Dec 1 10:31:03.973: %INSTALLER-6-AUTO_UPGRADE_SW: 1 installer: Reloading switch 2 to complete the auto upgrade
*Dec 1 10:31:04.005: %STACKMGR-1-RELOAD_REQUEST: 1 stack-mgr: Received reload request for switch 2, reason Auto upgrade
*Dec 1 10:31:04.468: %STACKMGR-1-STACK_LINK_CHANGE: 1 stack-mgr: Stack port 2 on switch 1 is down
*Dec 1 10:31:04.470: %STACKMGR-6-SWITCH_REMOVED: 1 stack-mgr: Switch 2 has been removed from the stack.
*Dec 1 10:31:04.474: Starting SWITCH-DELETE sequence, switch 2
*Dec 1 10:31:04.482: SWITCH-DELETE sequence complete, switch 2
*Dec 1 10:36:42.741: %STACKMGR-1-STACK_LINK_CHANGE: 1 stack-mgr: Stack port 2 on switch 1 is up
*Dec 1 10:36:57.017: %STACKMGR-6-SWITCH_ADDED: 1 stack-mgr: Switch 2 has been added to the stack.
*Dec 1 10:37:15.914: %STACKMGR-6-SWITCH_READY: 1 stack-mgr: Switch 2 is ready.
*Dec 1 10:37:15.918: Starting SWITCH-ADD sequence, switch 2
*Dec 1 10:37:16.044: %NGWC_USB_CONSOLE-6-CONFIG_ENABLE: Switch 2: Console media-type changed to default
*Dec 1 10:37:16.942: %NGWC_PLATFORM_FEP-6-FRU_PS_OIR: Switch 2: FRU power supply A inserted
*Dec 1 10:37:16.946: %NGWC_PLATFORM_FEP-6-FRU_PS_OIR: Switch 2: FRU power supply B inserted
*Dec 1 10:37:35.029: SWITCH-ADD sequence complete, switch 2

Switch#sh sw

Switch/Stack Mac Address : 00e1.6d09.ae00 - Local Mac Address

Mac persistency wait time: Indefinite

H/W Current

Switch# Role Mac Address Priority Version State

------------------------------------------------------------

*1 Active 00e1.6d09.ae00 1 M0 Ready

2 Standby 00e1.6d51.6780 1 M0 Ready

As you can see, switch 2 reloaded after the current version of software (in this case, 03.03.02) was installed and it only took about 6 minutes. The other option is to remove the switch from the stack, put a USB in it with the correct IOS version, copy the IOS version onto the switch and then do the 'software install file flash:' command and wait for it to reload. We've had 3850's deployed for about 6 months now and are really liking them.