Traceroute — and its little brother, ping — is one of the most commonly used tools in network troubleshooting. As helpful as this tool is, there are a few critical challenges when using it to handle today’s much more complex problems. Remember that traceroute was created in the late 1980s, when networks were far simpler.
Everything was physical, point-to-point was all the rage, and there were fewer protocols to deal with. Switches were commonly referred to as bridges, with LAN-to-WAN routers traversing from one building to the next. (Remember, this was half a dozen years before the Internet existed!) And who remembers ‘Leased T1 Lines’? The good ol’ days, so to speak.
So, how do antique tools like this hold up in today’s software-defined and virtualized world? Surely, there must be a more modern way to help troubleshoot today’s networks. Yes, there is, so keep reading.
To get started, let’s learn more about the traceroute command. Then, we’ll discuss traceroute errors, limitations, and more.
The source (SRC) machine typically sends three probe packets toward the destination IP address, starting with Time to Live (TTL) set to 1. People mistakenly think the probe packet type is always ICMP. In reality, it depends on the device originating the traceroute. Windows operating systems often use ICMP, while Unix and Routing devices commonly use UDP messages to ephemeral ports (ports greater than 1024 that are not well-known services like DNS, SMTP, WEB, etc.)
As each Layer-3 device receives the probe packet, it decrements the TTL. When TTL reaches 0, the receiving device sends an ICMP message from the receiving interface: “TTL Expired.” This is how traceroute knows each hop along the way.
In the picture below, 172.16.2.1 and 10.3.2.2 would be the addresses returned in the traceroute results.
The default for a traceroute is to report the path between points A and B. The opposite route requires a second traceroute. Let’s discuss why this can be a limitation.
Asymmetric routing is common in modern networks. Many routing protocols are in play here, including:
Depending on the algorithm used for hashing, traffic will likely take a different path in each direction. These different paths are tough to detect with a single execution of a traceroute command. Therefore, it’s difficult to determine where the delay result comes from.
As you can see above, the delay value increases significantly on this particular hop. Where is the delay, though? Is it on the forward or return path?
If we look at the above traceroute again, the only information provided is:
To investigate hop six, we would need to establish a Telnet or SSH connection with the router and determine which interface is attached to this particular IP address. “Show IP route 129.259.2.41” was my favorite way to do this. However, you could show the IP interface and pipe the output to a parser… “Show IP interface brief | in 129.150.2.41.” You’d need a follow-up lookup to identify the egress (outgoing) interface. For example, “show IP route 192.41.37.40.”
ICMP messaging sometimes reports a more significant delay than that experienced by traffic, mainly because it uses slow path processing rather than fast path.
What is a slow path? It’s when the router needs to process the packet contents. Virtually any device-destined packet is handled this way. The internal mechanisms of newer routers prioritize which packets they process.
They process routing protocol updates first, and then ICMP messages, which increases the issue. Many systems use UDP messages. However, generating the ICMP TTL expired message is a lower priority. Then, the return path will be an ICMP message. As a result, the delay metrics provided in the results are difficult to accept as real problems.
TIP: When a traceroute jumps at a particular step and the subsequent results are low, it’s often a sign that the “slow” router experiences a processing delay. If the path jumps and subsequent steps are incremented, this typically means some form of queuing due to congestion.
Now we know the path toward a specific destination, but what if we want to further investigate additional hops along the path? If a user uses the Telnet or SSH console, they’ll need to log into each device. The traceroute response only provides the interface IP address. In many networks, this creates challenges.
Security policies require users to use Telnet or SSH to access the management IP address. The specific interfaces being reported in the traceroute may block Telnet, which would require a manual lookup to determine the device’s management interface with that interface address. If you can’t use DNS resolution, this may be almost impossible. When there is an outage, every second can count against you.
As mentioned earlier, equal or unequal cost multi-path routes are commonplace in most networks today. Traceroute will only report on the specific path that was part of the inquiry probe messages. Most common traceroute implementations send three probe packets with different UDP destination ports. As a result, it’s possible that, in an ECMP environment, each hop will have up to three different IP addresses reported. Keeping track of which response is part of which path can become cumbersome.
As we now understand, traceroute decrements the TTL value in an IP packet and generates TTL-expired ICMP messages. Only Layer-3 devices decrement TTL. As a result, there’s no built-in visibility or function for easily determining the Layer-2 path taken from the result. Most enterprise and data center environments have a Layer-2 access switch used for aggregating end stations.
Some designs feature a Layer-2 distribution layer before reaching a core router. The router performs the first Layer-3 routing function, which can report back to a traceroute request.
If packet loss presents in the traceroute output, it could be due to a:
You can only discover some issues by inspecting the Layer-2 elements along the path. Finding this information would require:
This process can be very time-consuming, especially if CDP/LLDP is not enabled in the network or the documentation is outdated.
The results you see with a traceroute are the current state. There is no way to determine what path was used when traffic was successful (yesterday, for example). Consider the traceroute below. It would be good to understand what hop four was before things changed to help isolate the possible issue.
Note: It has been our experience that based on the output of the above traceroute command, engineers will assume that hop three is the problem. This is often not the case. A user should first log into the device at hop three and check if it has a routing entry to the destination. If so, the problem device is often the next hop in the routing entry. To be thorough, checking the egress interface toward the routing next hop is useful to verify interface-level performance and ACL configuration.
When I run into a traceroute command issue, every now and then, I get a character in the output that surprises me. The table below is from Cisco’s Implementation of traceroute. One of the traceroute limitations is that they may seem straightforward, but understanding what’s happening generally requires additional follow-up digging.
Say, for example, I receive the letter A in the traceroute response. I can understand that an access-list is blocking the traceroute, but which access-list is it? To determine this, I’ll need to:
This process can be lengthy, and we’re always concerned about time when troubleshooting issues. Therefore, a better way to gain access to this information would be invaluable.
NOTE: In our experience, these traceroute error messages are only ever provided based on the ingress interface. Once a packet is received and then forwarded to the next hop along the path, if there is an ACL on the egress port, you simply get a *** for the next hop. That means taking additional steps to ensure that the egress interface is known and any ACLs validated on that port as well.
NetBrain includes many automation and visualization features required to maintain modern networks. It understands software-defined, virtualized, and even cloud networks. It continuously communicates with every device on the end-to-end network and builds a real-time digital twin of the network, which is an exact replica of every device’s details.
It can visualize real-time networks and includes a modern replacement for traceroute, the A/B Path Calculator. This tool helps engineers dynamically map a detailed network path between any two points in the network.
It supports mapping through underlays, overlays, and modern technologies, including:
Simultaneously, it considers advanced protocols like:
Once deployed, the visualized structure is the perfect platform for no-code automation of every task, big and small!
Enable your teams to troubleshoot and diagnose application performance at the traffic-flow level with NetBrain’s Application Assurance (AAM). It:
Prevent application performance degradation with NetBrain’s no-code Intent-based automation. NetBrain AAM protects the user experience by: