ACI: Spine replacement in a single Spine fabric

A colleague has exchanged a spine at a customer. This is a lab fabric which runs in a minimal setup, 1 x APIC, 1 x Spine, 4 x Leaf.

In principle, the exchange of a Spine within an ACI fabric is a very simple undertaking, ACI does all the heavy lifting for you :)

However, in a scenario where a Leaf has only one path to the APIC (thus connected to a single Spine), the exchange of the Spine leads to problems.

1
2
3
4
5


┌────────┐    ┌────────┐    ┌────────┐    ┌────────┐
│        │    │        │    │        │    │        │
│  APIC  ├────┤  L111  ├────┤  S101  ├────┤  L112  │
│        │    │        │    │        │    │        │
└────────┘    └────────┘    └────────┘    └────────┘

Sample Topology

After replaceing the Spine, L112 wasn’t joined in the fabric anymore. On the switch itself it was then pretty easy to spot the issue.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24


L112# show discoveryissues
Checking the platform type................LEAF!
Check01 - System state - in-service                [ok]
Check02 - DHCP status                [ok]
    TEP IP: 10.0.200.66 Node Id: 111 Name: LEAF111
Check03 - AV details check                [ok]
Check04 - IP rechability to apic                [FAIL]
    Ping from switch to 10.0.0.1 with MTU 1450 failed. Check Communication issues.
Check05 - infra VLAN received                [ok]
    infra vLAN:4093
Check06 - LLDP Adjacency                [FAIL]
    Error: wiring issue found unapproved-serialnumber on eth1/54
Check07 - Switch version                [ok]
    version: n9000-14.2(7f)  and apic version: 4.2(7f)
Check08 - FPGA/BIOS out of sync test                [ok]
Check09 - SSL check                [check]
    SSL certificate details are valid
Check10 - Downloading policies                [ok]
Check11 - Checking time                [ok]
    2022-02-09 10:33:24
Check12 - Checking modules, power and fans                [ok]
Check13 - Checking isis adjacencies                [FAIL]
    No isis adjacencies found
    Ignore if first leaf/spine in an ACI fabric or remote leaf 

show discoveryissues from L112

Error: wiring issue found unapproved-serialnumber on eth1/54

Indicates that from the Leaf perspective the new Spine is not approved to the fabric and because the Leaf has no path to the APIC anymore, this information can’t be updated.

Easiest fix we found, was to simply reset the leaf.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17


L112# setup-clean-config.sh
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
In progress
Done

And then do a reload.

1
2


L112# reload
This command will reload the chassis, Proceed (y/n)? [n]: y

Thanks to Jonas for the story & the outputs.