NetBrain helped to cut mean-time-to-repair (MTTR) by as much as half for a global financial firm

This company is the world’s second largest financial information provider.  The 5,000-node private network delivers real-time financial data to thousands of financial institutes world-wide. The Network Operation Center (NOC) in Boston and Bangalore monitors the network 24/7 and troubleshoots 95% of reported problems. The major challenge the company consistently faces is to resolve network outages as fast as possible to minimize negative revenue impact.

Three NetBrain Appliances were deployed in the company’s Boston, New York and London datacenters, and over 50 network engineers were equipped with NetBrain Workstation. In this environment, NetBrain was also integrated with the NetCool alarm system, Opsware configuration management solution and Vitalnet’s performance trending solution.  Via these integrations, an alarm reported by HP OpenView is instantly translated to a map inside NetBrain Workstation.

NetBrain continues to offer value to this customer in three areas:

• NetBrain on-demand network mapping effectively removes dependencies on manual network diagrams which are often inconsistent and error-prone.
• Network performance diagnosis via smart maps enables lower level engineers to troubleshoot advanced problems with less escalation
• Engineers share information via NetBrain for collaborative troubleshooting sessions

The following are some war stories reported by this customer:

• Monitor application performance during major network degradation

  In 2006, an earthquake in Taiwan caused an undersea cable to be cut. This resulted in a prolonged period of network down-time between North America and Asia.  This customer used NetBrain to map out all impacted applications and visualize real-time performance metrics on visual screens.  This allowed the NOC to accurately analyze and report the business impacts.

• Visualize and troubleshoot network instability caused by malicious attack

  On one Friday afternoon, the NetCool alarm system was suddenly filled with thousands of alarms, like a “sea of red.”   Over 10 engineers worked together to ascertain the breadth of the problem before having to field an inevitable flood of customer phone calls.  Based on the source of the alarm, one engineer created a NetBrain map of the Boston-NY core and launched diagnostic monitoring from the map.  He instantly visualized that a large stream of traffic was oscillating between two fault-tolerant WAN links connecting the Boston and New York datacenters. Using NetBrain, he was able to dive-in, locate the offending machine, and counterattack the virus that caused the machine to malfunction and attack the network.     

• Catch routing loop while it was happening

  An alarm was raised about an inaccessible router. The technician assigned to the problem used NetBrain to create a live map between the NetBrain Appliance and the missing router. While NetBrain was mapping out the live traffic path, a routing loop was caught which was then traced back to a wrongly configured static route.

NetBrain saves time when time is critical.  As the director of the NOC reported, “NetBrain is one of the most advanced tools we have ever used.  It often cuts our network troubleshooting time by a half or more.”

Quick Links:

» Network Discovery and Network Documentation

» Data Center Migration