I often find myself searching for facts to dispel “hearsay” and guesswork in troubleshooting discussions between the Network and Windows VM teams.  This article is intended to show just how straightforward it is to use the VMware distributed switch SPAN port capability, to put a virtual sniffer on a virtual port on a virtual distributed switch.

RSPAN and ERSPAN – not today

To avoid unnecessary over-excitement, this post will not explain how to configure RSPAN or ERSPAN which are considerably more “troublesome” due to the added requirement of physical network switch configuration changes.  This configuration is within a single ESXi host, for someone with control of a VM, and permissions to add/remove a vmnic via vCenter.  Fairly common capabilities for a virtual network team resource.  If you need to monitor a port on another ESX host in the same cluster, I’d recommend vMotioning your monitoring VM to the same host to keep the process clean, avoiding network changes.

I like the simplicity of this approach, with all traffic forwarded by the hypervisor within the same host.  Having served time mucking about with multi-hop RSPAN sessions to a sniffer, on interfaces close to bandwidth saturation, it is all too easy to miss key frames or cause unexpected traffic patterns along the source to destination path.  Ask any fulltime network resource how reliable RSPAN is … but I digress.

Our landscape for our simple configuration looks like this…

vDS SPAN port with Wireshark-v0.6

Key points:

  1. There are two hosts ESX5.5 Host 1 and ESX5.5 Host 2, both of which have access to the same distributed switch called “GANDALF”
  2. The port we wish to monitor is in the ProdB distributed port group, assigned to VM1 on ESX5.5 Host 1
  3. All VMs are using vSwitch 802.1q VLAN tagging i.e. the VM guest Operating Systems do not tag their frames – Wireshark will see no 802.1q frames.
  4. The target VM we wish to monitor is VM1, which happens to be a Windows 2008 R2 DHCP server – we’re going to watch the DHCP ARP requests
  5. Wireshark v1.12.4-0-gb4861da is installed on VM2, which is running Windows 7 64bit
  6. We assume that we use vCenter direct console to access VM1, VM2 and VM3 – we ignore the VMKernel mgmt ports (they’re just noise for this explanation)

The stopwatch starts now.

STEP 1

Login to vCenter and add a new vmnic to our Wireshark host, VM2.  Be sure to connect it to the “ProdB” distributed port group, which is where VM1 sends and receives the traffic we are interested in.  At this point, we have TWO vmnics on VM2, both in the ProdB port group.  One is being used for normal traffic in and out of the VM.  The other is brand new, and will start to DHCP when it is configured, if we let it … here’s how it looks in the vCenter web client.

vDS SPAN Blog 2

Network adapter 1 and network adapter 3 are both in the ProdB vDS Port Group.  One is carrying normal traffic, the other will be our SPAN interface.

STEP 2

Login to Windows on VM2, and remove ALL the protocol bindings from the new NIC interface.

vDS SPAN Blog 3

I renamed mine to be VLAN -SPAN to make it’s configuration intention clear.

vDS SPAN Blog 1

We’re on about 90 seconds so far I hope!

STEP 3

In the vCenter web client, from the vCenter Home page, select Networking.

vDS SPAN Blog 4

 

Then select the Distributed Switch where you need the SPAN session, and click on the Manage tab.

vDS SPAN Blog 5

Click on settings, and then “Port mirroring”.  This is all somewhat pedantic, but we’re getting there.

STEP 4

Now we’re finally getting to the configuration of the SPAN session.  Click on “New” then you’ll see the following options.

vDS SPAN Blog 7

 

Take the first option – “Distributed Port Mirroring”.  Most of the following steps are fairly self-explanatory, but I’ll screenshot them here for completeness.

vDS SPAN Blog 9 - after

A useful note here is the “disallowed” setting for the destination port.  While we want VM2 to get all the packets that VM1 gets (ingress and egress), we don’t want any RDP or other normal VM traffic to pollute the feed.  By disabling all the Windows protocols on our target VM2 interface, we avoid generating any distracting traffic, and can use the VLAN -SPAN Windows interface purely for Wireshark.

Click on Next.

STEP 5 – Source and Destination

Now we are presented with a dialogue that asks us to select our sources.  We can select from the menu, or by port number – both have the same outcome, namely adding a source port for the mirroring session.

vDS-SPAN-Blog-10

Go with what is easiest for you.

Here I have selected VM1 as my source port, which happened to have Port ID 100.

vDS SPAN Blog 11

After “Next” I selected VM2 as the destination port – making sure I got the correct Port number, as there are TWO connected to this distributed Port Group remember!

vDS SPAN Blog 12

At the final screen you are presented with a summary, and the Finish button.  Here’s what mine looked like:

vDS SPAN Blog 13

STEP 6 – Wireshark

At last, we can fire up Wireshark on our Windows VM2 image and see the fruits of our labour.

vDS SPAN Blog 15

I’ve redacted MAC addresses and IP addresses from this dump, but it is a real one, from the SPAN just configured on the esx006 host in my lab environment.  Both REQ and ACK packets are visible, telling us ingress and egress traffic is being provided to the SPAN interface for Wireshark’s consumption.

I hope this was useful, without being too long winded.

Give or take a click or two, it’s a 2 minute exercise to setup a SPAN port on the same ESX host as the Wireshark image.  Learning Wireshark is something for the weekend if you’re not already familiar with this excellent packet analysis tool.

The Architect’s View

When troubleshooting complicated, highly concurrent, SOA infrastructure for ESB brokering for example, knowing exactly where your network traffic is coming from and going to, and where it isn’t going to, can be complex to understand.  This is one situation where I didn’t want the complexity of multi-hop RSPAN sessions alongside the multicast traffic path inside the vSwitches.

Reducing the complexity of IT service architectures and the tools used to troubleshoot them is highly recommended.  The KISS principle is our friend.  Complexity is our enemy.

Wireshark is available here:  https://www.wireshark.org/download.html

 

Written by Rob Lyle