Sometimes we get lost in the ideation of lofty ideals and futuristic networking at a high level, especially in architecture and design conversations. However, no matter how advanced the technology gets, we must still do the basic things — troubleshoot when issues arise. The front-line support engineer is often the hero in many daily activities — everything from simple firewall ports being opened to solving a connectivity issue with a user. But life in the trenches of front-line support is stressful, and you must rely heavily on the tools afforded to you.
A Tale from the Front Lines: Troubleshooting Intermittent Connectivity
Now, a true story: A trouble ticket has come in, and we must assist a user who says he’s experiencing intermittent connectivity to the corporate network. First, we need to determine how he’s connecting, wired or wireless, and if he’s getting an IP address. The corporate policy allows pinging for troubleshooting (something that took an arm and leg to get done). We determine that the end user has an IP address assigned that matches the corporate wireless for the campus (~2,000 users), and we have the MAC address. Next, like any good path, we need a destination. We are informed he’s trying to access Application X. However, to verify basic connectivity, we have him test to the domain controller and the internet, and perform a handful of other steps.
Life in the trenches of front-line support is stressful, and you must rely heavily on the tools afforded to you.
During this process, we’ve brought up NetBrain and input the source & destination IP addresses and mapped it as an A-to-B path. Now we can see that the user is connecting via wireless to AP-18, to the 4th-floor switch seven via PoE, and from there traffic is working as expected. We map out all paths on a single map and flip on monitor mode so we can see utilization, CPU, and other variables. No issue here. This must mean there is a user issue, right? Good ’ol PEBKAC – Problem Exists Between Keyboard And Chair.
NetBrain maps any path across the network, with just the source and destination addresses.
During this time of our testing, the user again attempts Application X several times, and to his dismay and frustration, it is still disconnecting. Now we bring up Application X into the map, and it looks excellent, reliable, stable. Or does it? While we’re reviewing the map in monitor mode, we see that the app changes switching path from DC switch 2B to 2A for a moment and then back, and then back again, and then back yet again. If I were doing a “show mac address-table | i abcd.ef01.2345” on the CLI, we could have missed this. However, NetBrain visually showed us with live data how the path was changing.
NetBrain visually showed us with live data how the path was changing. Instead of punting the user to the application team, we were able to provide first-line troubleshooting that showed an issue occurring with IP connectivity.
The user initially stated the issue as “unable to connect to the corporate network.” Quickly, we discerned that was not the case — the user could get on the corporate network. He really had an issue with access to a specific application. Instead of punting the user to the application team, we were able to provide first-line troubleshooting that showed an issue occurring with IP connectivity. All this quick information allowed us to present data to the virtualization team to resolve the end-user connectivity problems (which was also affecting others) while we remained on the call.
NetBrain breaks down data silos, with all information easily shared on a Dynamic Map.
Conclusion
NetBrain allowed us to go beyond our network management system at the time. The issue was a MAC flap of a few different VMs; it wasn’t a spanning-tree port flap or other layer one physical issue. A virtual NIC change caused a cascading failure across a subset of VMs that was only alerting and visible within the reporting structure of the virtualization team.
Depending on your organization, you too might feel the effects of siloed support teams. Visibility is everything. Your end-user doesn’t like being punted around from team to team, just like you don’t like being put on hold when you call for support on your personal items.
When you need something resolved quickly, you want expert technicians and knowledgeable support personnel. What you need is what we did – empower your support teams with NetBrain and elevate your troubleshooting and quick resolution game.