Troubleshooting cases
L1
Fiber Power levels -67 to -30 dBm (-30 dBm is best and -90dBm is no signal)
Types of SFP's.
CWDM - 1470 to 1610 nm(8 colors) (soft coded Wavelength with high spacing bw channels.)
DWDM - (32 colors) (hardcoded, long distance comm. sfp's with denser channel spacing)
types of cables and SFP's.
o Twin aux
o aqua, yellow, orange, SFP, QSFP, CSFP
L2
L-2 lag address hashing issue caused lacp mac missing on hw prog and resulting in register counter drops,
intermittent arp-resolution issues and bgp flaps.
VRRP and STP mismatch causing root bridge and primary router mismatch and dropping traffic because of
forwarded and blocked pots in stp on primary router.
o reason somebody added lowest IP address on the second bridge with same root bridge Priority.
VPC Same Domain ID configuration caused mac-address duplication that caused issues.
Broadcast storm issue with catalyst switches.
L2-Secure Vlan with Hospira.
Route Recursive resolution similarly next-hop recursive resolution.
BGP
BGP ECMP next hop link flap, recursive resolution with ldp causing index not reset to up.
BGP prefix states "Accepted DeletePending" under extensive hidden routers supposed to be under hidden table
but still active when BGP route flap damping is enabled.(PR1556656)
BGP RPKi Route validation to avoid route leak and route hijack with the following states
"Valid/invalid/unknown" periodic serial notify state routs may get deleted and added back without bgp peer
flap. (PR1556656)
o ROV - Route Origin Validation
o Lab Validator - Routinator
Change in RPKI timers leads to records timeout (1594597)
Refresh/record/hold timers have to be config in proper ratios for right operational behavior.
BGP three way establishment mismatch because of duplicated loopback ip address.
Network design with campus network at Hospira call center.
ISIS:
ISIS Protocol flap on the interface as a result of Multicast/TTL/resolve ddos.
ISIS will configure push/pop 0 label instead of regular label causing traffic interruption.
ISIS RSVP/LSP micro loops frr resolution(More details below)(2021-0223-0106)
OSPF/BGP Flap (More details Below) (2021-0828-250693)
Interface Packet loss may be the cause of OSPF stuck in the Exchange state.
ISIS/OSPF forwarding and traffic loss
o This issue might be seen if the following conditions are met:
* The forwarding member nexthop of unilist nexthop is down while there is another active forwarding
member nexthop.
* "load-balance per-packet" is configured
* Host generated traffic (e.g. BGP) is forwarded
Juniper Business Use Only
Scripting:
Python:
RSVP hop program tracing using event-options triggering.
nhdb next hop index monitoring
fpc reboot device health check data collection triggered by even script
regular data collection before fru reboot
Pre-Post snapshot script
RSVP LSP build script
RSVP lsp heath monitoring scripts.
Decommission Script
Auditing scripts
bulk load config scripts
trigger bugs on lab environments
Silly email notification script pre Maintenance Window.
Linux:
Interface flap
day to day investigation
host cron job for triggering scripts
trigger bugs in lab environments
Parse through PR's and old cases to find the keywords and find repetitive issues
Trigger bugs in lab environments
Juniper Business Use Only