While troubleshooting a totally unrelated issue, one of my colleagues noticed that they were seeing packets in a tcpdump that were neither destined for nor sourced from the server. This is odd, when plugged into a switch, so we started digging.
Server 1, was sending a stream of packets to Server 2 - in a different subnet somewhere. Sometimes, although rarely, these packets could be seen on Server 3.
By examining the captures, we established these facts:
- Capturing on Server 3, we see traffic sourced from the IP of Server 1, destined to the IP of Server 2.
- The packet at layer 2 is sourced from the MAC address of Server 1, and destined to the MAC of the default gateway (the router's MAC address).
- There were only 6 packets in the capture, which were over a 400 msec period, and it took half a day to capture them – that means this doesn’t happen very often, and when it does it doesn’t last very long.
A network switch is intelligent – when it has a fully populated MAC address it “routes” frames – i.e. it only sends frames out of the ports that it knows needs them. It dynamically learns MAC addresses and adds them to a MAC address table, by examining the source MAC address field in received frames. This means that Server 3 should not see traffic that is not sourced or destined to it, except in the following 4 circumstances (faulty hardware and software bugs aside!):
- Our switch is actually a hub (I’m reasonably confident that it’s not this… ;o) )
- The traffic is broadcast traffic – we can see from the captures that the traffic is not broadcast.
- The traffic is multicast traffic and IGMP snooping is disabled or not functioning – we can see from the captures that the traffic is not broadcast.
- The traffic is “unknown” unicast – we can see from the captures that it is definitely unicast.
Unicast traffic is sourced from one specific host and destined to one specific host. It is “unknown” unicast when the switch doesn’t know which port the destination MAC address is based on. When the switch has to forward unknown unicast frames, it follows a simple rule: Forward the frame out of every active (forwarding) port, in the same VLAN, except the one it was received on.
There are a number of reasons why the switch might not know where the destination is:
- The server has never sent any frames, since it’s network interface came up. (unlikely on a server that hasn’t been rebooted)
- The server has not sent any frames in the last 300 seconds (very unlikely…5 minutes is a long time without a single frame). This would cause it to age out of the MAC address table.
- There has been a topology change on the switch which caused a Spanning-Tree Protocol (STP) Topology Change Notification (TCN), AND the server didn’t send any traffic for 15 seconds.
It is perfectly normal behaviour when a TCN is generated for switches to shorten their MAC address table timers to 15 seconds. Any host which doesn’t send traffic within 15 seconds would then be flushed from the MAC address table and have to be relearned. This shortened timer lasts for a period of 45 seconds, after which time it reverts to 300 seconds.
We proved this by manually flapping a port while running tcpdumps on various servers in the same VLAN. For periods of a few hundred milliseconds, some traffic does get unknown unicasted. It's not much, but it is there. This is one of many reasons why ports should be designated as edge ports in RSTP (portfast!)
All in all, this is perfectly normal network behaviour – it is how the network topology is learned and stabilised, and is fully expected.
Unfortunately I couldn't post the pcap's as they had some info in them that I can't publish, and I don't have the time to lab it as I would have to use physical kit to do it properly.