Even if every APIC in the Cluster goes down, there will be no impact to the production traffic. This is because the Forwarding plane is handled by the nodes, and the control plane will be offline with APIC.
Unlike CatOS, where everything is allowed by default.
Nexus must be configured to allow all destinations.
Usually a 9500 is used for a spine
Usually a 9300 is used as a Leaf
Leafs can only be connected to spines, not between them
When APIC is connected to a Leaf, LLDP is used for discovery of neighbours. Turn off CDP
Unlike a router by default allowing all traffic, ACI requires an access list to allow traffic
The default is to block all
The ACI Fabric is a Layer 3 Fabric where a routing is being performed by IS-IS.
—Changing an interface and default gateway
configure
controller api1
interface mgmt0
ip address 192.168.1.101/24 gateway 192.168.1.112
—Converting NXOS to ACI
Step1. TFTP the aci-n9000 image
Note: You must have the correct egl.img boot image
Step2. Conft#no boot nsos
Conft#boot aci bootflash:aci-n9000
Step3. copy running-config Startup-config
Step4. Reload
—Tenants are a grouping of objects. There are 3 default tenants which cannot be deleted.
COMMON – shared settings which can be used by all for example SNMP
INFRA – handles all the communication in the fabric between the spine, leaves and nodes
MGT – inband and out of band settings management traffic
—VXLAN
Virtual Extensible Local area network, is Layer 3 tunneling. protocol that can bridge 2 VLANs which are separated from
Layer 3 broadcast domain.
—Spine
This is a core Nexus 9000 usually not connected to a secondary spine.
Adding a Spine, Fabric -> Fabric Membership -> Tools -> Add Device. Spines are not connected directly to other spines they are connected to each Leaf
A spine carries the global proxy table which is complete list of nodes or endpoints
—Leaf
This is an access switch Nexus 9000K, it is connected to the spine
Adding a Spine, Fabric -> Fabric Membership -> Tools -> Add Device. Leafs are connected to each other, they are connected to each spine.
The Leaf does not have a routing table, but a default route to the spines.
—Bridge Domains
Isolate broadcast and multicast traffic for (most commonly) 1 VLAN, but can support multiple VLANs.
VRF’s hold bridge domains, bridge domains contain a list of subnets(VLANS) which can VXLAN.
When communication takes place between bridge domains packets are routed between them using COOP.
Layer2 Flooding
—-Remote Site or Datacentre
These connection go to the Spine NOT the Leafs
Connect to Leaf from APIC
bash
attach [node-name]
Ping from Leaf
iping [-dDFLnqRrv] [-V vrf] [-c count] [-i wait] [-p pattern] [-s packetsize] [-t timeout] [-S source ip/interface] host
Clustering User Commands
<controller> – shows the current cluster size and state of APICs
<cd /aci/system/controllers/1/cluster> <moset administrative-cluster-size (#)> <moconfig commit>– changes the size of the cluster
<controller -d -t (ID)> – Decommissions the APIC of the given ID
<eraseconfig setup> – Factory resets APIC and after reboot will load into setup script
<reload [controller|switch] (nodeID)> – Reboots the APIC of the given ID
<acidiag rvread> – shows replica which are not healthy
<acidiag rvread (svc) (shard) (replica)> – shows the state of one replica
<avread> – large output which will show cluster size, chassisID, if node is active, and summary of replica health
<acidiag fnvread> – shows fabric node vector
<acidiag avread> – shows appliance vector
<acidiag verifyapic> – verifies APIC hardware
<acidiag reboot> – Reload APIC
<ip link> – shows link status
<cat /proc/net/bonding/(ID)> – shows the status of bond link
<show dhcp internal info client> – shows dhcp client information to confirm dhcp address from APIC
<fabricnode (nodeID) [commission|decommission|wipeout]> – commissions, decommissions, or wipes out given node. wipeout will completely wipeout the node including configuration. Use sparingly.
SSL Troubleshooting
<openssl s_client -connect (IP):12151> – tries to connect ssl between APIC and Node and gives output of SSL information
<zgrep SSL svc_ifc_appliancedirector.bin.log*> -shows logging of DME-logs for node
<zgrep SSL svc_ifc_policyelem.log*> – shows policy-element logs for SSL connectivity
Can also check logs in the /var/log/dme/log directory
Switch Cert Verification
<openssl asn1parse < /securedata/ssl/server.crt> – Next to PRINTABLESTRING, it will list Insieme or Cisco Manufacturing CA. Cisco means new secure certs are installed, Insieme means old unsecure are installed
<openssl x509 -noout -issuer -subject -dates -in /securedata/ssl/server.crt> – Shows start and end dates of certificate. Must be within range for APIC to accept
<act_util key_pair show (#)> – Shows keypairs of specified cert
Switch Diagnostics
<show module internal event-history module (#)> – shows bootup tests and diagnostics of given module
<show diagnostic content module (ID)> – shows ongoing tests of given module
<show diagnostic result module [all|(moduleID)]> – shows diagnostic result of given module or all modules
<show diagnostic result module (moduleID) test (testID) detail> – shows diagnostic result of given test on given module
<show diagnostic internal [diagmgr|diagclient|port_lb]> – show debug information for the diagnostic modules
Debug Commands
<debug platform internal emon [heartbeat|kfsm|stats|traffic]> – shows debug output of given argument
<debug platform internal emon [heartbeat|kfsm|stats|traffic] [enable|disable]> – enables/disables given argument on all modules
<debug platform internal emon [heartbeat|kfsm|stats|traffic] interval get> – gets the interval of given argument
<debug platform internal emon stats get (ID)> – EPC mon statistics
<debug platform internal emon kfsm state get (ID)> – EPC mon statistics
<debug platform internal marvell switch [0|1] status> – EOBC/EPC switch status (0: EOBC, 1: EPC)
<debug platform internal broadcom switch status> – SC card broadcom switch status
Insieme ELTM VRF, VLAN, Interface Commands
<debug system internal [eltm|eltmc] trace output file> – dumps ELTM trace to output file
<show system internal [eltm|eltmc] info trace> – dumps eltm trace to console
<show system internal [eltm|eltmc] info vrf (vrf)> – shows vrf table of given vrf
<show platform internal ns forwarding segments> –
<show platform internal ns forwarding epgs> –
<cat summary> – vrf summary, shows ID, pcTag, scope
<show system internal eltmc info vlan brief> – shows vlan information. Can substitute (brief) for a vlan ID
<show sytem internal eltmc info interface (interface ID)> –
OSPF CLI Commands
<show ip ospf neighbors vrf (vrf|all)> – shows OSPF neighbors of given vrf
<show ip ospf route vrf (vrf|all)> – shows OSPF routes of given vrf
<show ip ospf interface vrf (vrf|all)> – shows ospf interfaces of given vrf
<show ip ospf vrf (vrf|all)> – shows ospf information of given vrf
<show ip ospf traffic vrf (vrf|all)> – shows ospf traffic of given vrf
External Connectivity
<show ip arp vrf (vrf)> – shows arp entries for given vrf
<show ip ospf neighbors vrf (vrf)> – shows ospf neighbors for given vrf
<show bgp sessions vrf (vrf)> – shows bgp sessions/peers for given vrf
<show ip ospf route vrf (vrf)> – shows ospf routes for given vrf
<show bgp ipv4 unicast vrf (vrf)> – shows bgp unicast routes for given vrf
<show ip static-route vrf (vrf)> – shows static routes for given vrf
<show ip route vrf (vrf)> – shows routes for given vrf
<l3 defip show> – shows external LPMs
<l3 egress show> – shows next hops towards NorthStar ASIC or external router
<show platform internal ns table mth_lux_slvd_DHS_HigigDstMapTable_memif_data ingress> – HigigDstMapTable Indexed using DMOD/DPORT coming from T2. Provides a pointer to DstEncapTable.
<show platform internal ns table mth_lux_slvg_DHS_DstEncapTable_memif_data ingress> – DstEncapTable Indexed using the HigigDstMapTable’s result. Gives tunnel forwarding data.
<show platform internal ns table mth_rwx_slva_DHS_RwEncapTable_memif_data ingress> – RwEncapTable Indexed using the HigigDstMapTable’s result. Gives tunnel encap data.
ISIS Fabric Unicast Debugging
<show isis protocol> – shows ISIS statistics
<show isis adjacency [detail] vrf (vrf)> – shows ISIS adjacencies for given vrf. Can also add detail
<show lldp neighbor> – shows lldp neigbor status
<show interface (interface ID)> – shows interface status information and statistics
<show isis database [detail] vrf (vrf)> – shows isis database, can also add detail
<show isis route vrf (vrf)> – shows isis route information
<show isis traffic vrf (vrf)> – shows isis traffic information
<show isis dtep vrf (vrf)> – shows all discovered tunnel end points
<show isis statistics (vrf)> – shows isis statistics of given vrf
<show isis event-history [detail]> – shows isis event history
<show isis internal mem-stats [detail]> – shows isis memory statistics
<show tech-support-service isis> – provides isis tech-support output for TAC
ASIC Platform Commands
<show platform internal [ns|alp] mac asic [0|1]> – shows the MAC port status
<show platform internal [ns|alp] counters mac asic [0|1]> – shows the MAC port counters
<show platform internal [ns|alp] counters asic-block [all|bax|lbx|lux|prx|qsx|rwx|scx|top]> – shows ASIC block counters for given ASIC. Can also add [detail] for more details
<show platform internal [ns|alp] interrupts> – shows interrupts for given ASIC
ASIC Platform Commands – T2 Specific
<show c rpkt> – shows receive counters for T2
<show c tpkt> – shows transmit counters for T2
<show c xe12> – shows per port packet type counters
<g chg ing_event_debug> – shows ingress drop counters
<g chg_egr_drop_vector> – shows egress drop counters
<s RDBGC(#)_SELECT bitmap=(hex)> & <g chg RDBGC(#)_SELECT> – setting register to specific trigger. 9 registers per port (0-8)
<s RDBGC3_SELECT bitmap=0x2000> <g chg RDBGC3_SELECT> – sets 4th register to select RFILDR selector (bit 13)
<cstat xe17> – checking the stats for above command
ASIC Platform Commands – NS Specific
<show platform internal counters port> – shows port counters
<show platform internal counters port internal> – shows internal port counters
<show platform internal counters vlan> – shows vlan counters
<show platform internal counters tep> – shows per-tunnel counters
<show platform internal ns counters asic-block all> – shows ASIC block counters
<show platform internal ns forwarding list> – shows well-defined tables
Fabric Multicast – General
<show isis internal mcast routes ftag> – shows currecnt state of FTAG, cost, root port, OIF list
<show isis database mgroup detail vrf (vrf)> – shows GM-LSP database
<show isis internal mcast routes gipo> – shows GIPO routes, Local/transit, OIF list
<show isis internal mcast statistics> – shows topology and compute stats, MRIB update stats, Sync+Ack packet stats, Object store stats
<show isis event-history mcast> – shows isis multicast event history logs
<show isis event-history mcast-convergence> – more detailed than above command, specifically dealing with forwarding events and forwarding updates
Fabric Multicast Debugging – MFDM
<show forwarding distribution l2 multicast> – flood/OMF/GIPi membership
<show forwarding distribution l2 multicast vlan (vlanID)> per BD
<show forwarding distribution l2 multicast gipi> – GIPi membership
<show forwarding distribution l2 multicast gipi (IP)> – specific
<show forwarding distribution l2 multicast gipi vlan (vlanID)> – per BD
<show forwarding distribution l2 multicast gipi (IP) vlan (vlanID)> – specific per BD
<show forwarding distribution l2 multicast flood> – flood membership
<show forwarding distribution l2 multicast flood vlan (vlan ID)> – per BD
<show forwarding distribution l2 multicast omf> – OMF membership
<show forwarding distribution l2 multicast omf vlan (vlan ID)> – per BD
<show system internal forwarding distribution multicast ipmc> – IPMC membership
<show system internal forwarding distribution multicast ipmc 0x3> – specific IPMC
<show forwarding distribution multicast ipmc-sw>
<show forwarding distribution multicast ipmc-sw (ID)>
Fabric Multicast Debugging – L2 Multicast
<show system internal forwarding l2 multicast> – flood/OMF/GIPi membership
<show system internal forwarding l2 multicast vlan (vlanID)> – per BD
<show system internal forwarding l2 multicast gipi> – GIPi membership
<show system internal forwarding l2 multicast gipi (IP)> – specific
<show system internal forwarding l2 multicast gipi vlan (vlanID)> – per BD
<show system internal forwarding l2 multicast gipi (IP) vlan (vlanID)> – specific per BD
<show system internal forwarding l2 multicast flood> – flood membership
<show system internal forwarding l2 multicast flood bd (bdID)> – per BD
<show system internal forwarding l2 multicast met> – MET membership
<show system internal forwarding l2 multicast met (ID)> – specific MET
<show system internal forwarding l2 multicast met flood> – flood MET
<show system internal forwarding l2 multicast met gipi> – GIPi MET
<show system internal forwarding l2 multicast met gipi bd (bdID)> – per BD
<show system internal forwarding l2 multicast met gipi (IP) bd (bdID)> – specific per BD
<show system internal forwarding l2 multicast ipmc> – IPMC membership
<show system internal forwarding l2 multicast ipmc (ID)> – specific IPMC
Fabric Multicast Debugging – MRIB
<show ip mroute vrf (vrf)> – shows IP multicast routing table for given vrf
Fabric Multicast Debugging – MFIB
<show ip fib mroute ftag> – shows FTAGs
<show forwarding vrf all multicast route> – shows GIPO routes
Fabric Multicast Debugging – IGMP
<show ip igmp snooping groups> – shows multicast route information in IGMP
<show ip igmp snooping mrouter> – shows multicast router information IGMP
<show ip igmp snooping encap-db> – FD to BD vlan mapping. IGMP gets FD and G from Istack. It needs to know the BD to create (BD, G)
<show ip igmp snooping vlan (vlanID)> – verify BD membership of a port in IGMP. Only when ports are part of BD joins are processed
<show ip igmp snooping vtep-if-db> – verify the tunnel to IF mapping in IGMP. IGMP uses this to get the groups on VPC and only sync them.
Fabric Multicast Debugging – MFDM
<show forwarding distribution ip multicast route vrf (vrf)> – shows IPv4 multicast routing table for given vrf
<show forwarding distribution multicast vlan_db> – Verify FD to BD vlan mapping. MFDM gets (FD,port) memberships from vlan_mgr and uses this information go create BD floodlists.
<show forwarding distribution multicast bd_gipo> – BD to GIPO mapping. GIPO is used by Mcast in Fabric
<show forwarding distribution multicast epg_gipo_prime> – FD-vxlan to GIPO mapping
<show forwarding distribution multicast vtep_if_db> – tunnel to phy mapping
Fabric Multicast Debugging – M2rib
<show l2 mroute> – shows multicast route information in M2rib
<show l2 mroute omf> – shows multicast route informatino in M2rib
Fabric Multicast Debugging – PIXM
<show system internal pixm info ltl-range start-ltl 0x0 ltl-cnt 4000> – RID to IPMC mapping. IFIDX is RID and LTL is IPMC
Fabric Multicast Debugging – VNTAG Mgr
<show system internal vntag dvif-allocation> – IPMC to DVIF mapping. LTL is IPMC
EP Announce – Debugging
<show system internal epm announce>
<show system internal epm counters announce>
<show system internal epm vlan (vlanID) detail>
<show system internal epm vrf (vrf) detail>
<show system internal epm periodic>
<show system internal epm endpoint all>
iBash CLI
<show mac address-table>
<show endpoint [summary|address|interface|vlan|vrf]> – show endpoint information
BCM Table Dump
<bcm-shell-hw “l2 show”>
<bcm-shell-hw “l3 l3table show”>
Fabric QoS Debugging – CoPP CLI
<show copp policy>
<show system inernal aclqos brcm coppp entries unit 0> – CoPP statistics (red = dropped, green = allowed)
<show system internal qos classes> – shows QoS classes configured
<show system internal qos vlan all> – shows QoS classes/policices configured per vlan
<show system internal qos ppf [pinst|nodes]> – shows ppf details
<show system internal aclqos qos classes> – shows QoS classes configured in hardware
<show system internal aclqos qos vlan (vlanID)> – shows the QoS DSCP/dot1p policy configured for a vlan in HW
<show system internal aclqos qos policy summary> – shows QoS DSCP/dot1p policy summary
<show system internal aclqos qos policy detail> – shows QoS DSCP/dot1p policy in detail
<show system internal aclqos brcm tcam entries unit 0 group [efp-bpdu|efp-ctrl-pol|efp-mark|ifp-ctrl|ifp-dscp|ifp-elmc-vleaf|ifp-span-port-vlan|ifp-span-port-vlan-egress|ifp-span-vlan-egress|ifp-vni-udf|vfp-vni]> – shows T2 TCAM entries for specified group
<show platform internal counters port (#)> – shows QoS counters on each port
<show platform internal counters port internal (#)> – shows QoS counters on each port (internal)
<show platform internal counters class (#)> – shows QoS counters for each class for all ports
MCP CLI
<show mcp internal info global> – shows the edge port config on the HIF (FEX) ports, the internal VLAN mapping and the STP TCN packet statistics received on the fabric ports
<show mcp internal info interface [all|interfaceID]> – shows mcp information by interface
<show mcp internal info stats interface> – shows stats for all interfaces
<show mcp internal info vlan [all|vlanID]> – shows mcp information per vlan
<show mcp internal stats vlan> – shows stats for all vlans
<show mcp internal info msti [all|(region name) (instance ID)]> – shows mcp information per msti region
<show mcp internal info stats msti> – shows stats for all msti regions
iTraceroute CLI
<itraceroute (destinationIP) (pld-size)> – node traceroute
<itraceroute (destinationIP) vrf (vrf) encap vlan (vlan-encap) payload (pld-size)> – Tenant traceroute for vlan encapped source EP
<itraceroute (destinationIP) vrf (vrf) encap vxlan (vxlan-encap) dst-mac (dst-mac) payload (pld-size)> – Tenant traceroute for vxlan encapped source EP
ELAM Setup and debugging (follow commands in order)
<debug platform internal ns elam asic (#)> – starts ELAM on given ASIC
<trigger init ingress in-select 3 out-select 0> – sets trigger for ELAM
<set outer l2 dst_mac (destination mac) src_mac (source mac)> – sets source and destination mac addresses
<start> – Starts capture
<status> – shows capture status
<report> – shows report of the capture
VMM Troubleshooting
<show vmware controllers> – shows VM controllers and their attributes such as IP/hostname, state, model, serial number
<show vmware domain mininet (name) inventory> – shows hypervisor inventory of given VM controller
<show vmware domain mininet (name) [inventory|policy|status]>
<show vmware domain mininet (name) inventory [hypervisors|portgroups|virtual-machines|virtual-switches]>
TOR Sync Troubleshooting
<netstat -tp | grep epm>
<tcpdump -i kpm_inb>
<show system internal epm vpc>
<show system internal epm counters vpc>
<show system internal epm counter zmq>
<show system internal epm announce>
<show system internal epm counters announce>
<show system internal epm vlan (vlanID) [detail]> – can see which VLAN is learn disable
<show system internal epm vrf (vrf) [detail]> – can see which VLAN is learn disable
<show system internal epm periodic> – see if timer is attached on the VLAN/vrf
<show system internal epm counters all>
<show system internal epmc counters all>
OpFlex Debugging
<vemcmd show openflex> – shows if OpFlex is online (status = 12 means OpFlex is online, remoteIP is anycast IP, intra vlan is vlan used by VTEP, FTEP IP is the iLeaf’s IP)
<vem status> – check if DPA is running
<vemcmd show sod>
<vemcmd show port> – uplinks and vtep should be in forwarding state. PC-LTL of uplink port should be non-zero
<vemcmd show pc> – Check port channel type
<vemcmd show lacp> – if port channel type is LACP, can use this command to see the individual uplink LACP state
<esxcfg-vmknic -l> – verify if the VTEP received a valid DHCP IP address
SPAN Debugging
<vemcmd show span>
BPDU Debugging
<vemcmd show card> – shows if BPDU Guard/Filter is enabled or disabled
<vemcmd show bpdu-stats> – check if the bpdu-drop stats are incrementing on the uplinks/virtual ports
VEM Misc Commands
<vemcmd show openflex> – show channel status
<vemcmd show port> – check port status
<vemcmd show bd> – check per EPG flood lists
<vemcmd show epp multicast> – check vLeaf multicast membership
<vemcmd show stats> – show packet stats
<vemcmd show packets> – show packet counters
<vemlog debug sfport all> – debug vxlan packet path
<vemlog debug sflayer2 all> – debug vxlan packet path
<vemlog show all> – show above logging output
<vempkt capture [egress|pre-ingress]>
<vempkt clear>
<vempkt start>
<vempkt stop>
<vempkt display brief all>
<vempkt display detail entry (#)>
<vempkt cancel capture all>
FEX Troubleshooting
<show fex> – shows all FEXs and their states
<show fex (#) [detail]> – gives detailed stats of given FEX
<show environment fex> – gives environmental stats of FEX
<show fex transceiver>
<show fex version> – shows FEX version
<show interface fex-fabric> – shows FEX fabric interface information
<show logging level fex> – shows logging information for FEX
<show interface transceiver fex-fabric> – shows transceiver information for FEX
<show system reset-reason fex> – show FEX reset reason
<show module fex> – shows FEX module information
<show system internal fex log | grep (anything)> – shows debugging information and you can grep to find what you want
<show system internal fex internal event-history msgs> – use to find out which service is failing the sequence and you can debug that process further