Traffic Capture with VMWare ESXi

by
Reading Time: 5 minutes

My plan is to blog more about solutions and how to configure them and show how the packets look like. As I haven’t plenty of hardware I will use VMWare ESXi to simulate most of the stuff using HPE VSR routers. Therefore I was looking for a way to capture traffic of a specific VM or VM network adapter and found something. As it is very helpful for me, I will share this with the community.

Traffic Capture with VMWare ESXi

Since ESXi version 5.5, VMWare included a traffic capturing tool into ESXi, pktcap–uw. This tool is working like tcpdump on the command line and is able to capture traffic from different locations. For a full list refer to the VMWare knowledge base article here:
https://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2051814The interesting part for me is to capture traffic from specific vswitch port to get the traffic for this virtual machine, to be able to analyze the protocols. This can be achieved by using the switchport option for the pktcap–uw command. This option requires the switchport number, a virtual machine nic is connected to as a parameter. Unfortunately, this information is not available in the GUI and you have to use another CLI command:

[root@esxi:~] net-stats -l
PortNum          Type SubType SwitchName       MACAddress         ClientName
33554434            4       0 vSwitch0         50:65:f3:2f:cf:89  vmnic0
33554436            3       0 vSwitch0         50:65:f3:2f:cf:89  vmk0
33554437            5       9 vSwitch0         00:0c:29:f1:fb:49  Infrastructure
33554438            5       9 vSwitch0         00:0c:29:89:27:02  AirWave
33554439            5       9 vSwitch0         00:0c:29:68:12:be  CPPM

The command which is used, was “net-stats” with the “-l” option. This will get you the output above, where the first column represents the port number. The port number will only be displayed for running virtual machines. For machines, which are not running, there will be no entry. If your virtual machine has more than one nic, you can use the mac address to match the correct port to listen on.

To start the traffic capture, I will use the port number and start the capture process:

[root@localhost:~] pktcap-uw --switchport 50331651 --dir 2
The switch port id is 0x03000003
No server port specifed, select 35814 as the port
Output the packet info to console.
Local CID 2
Listen on port 35814
Accept...Vsock connection from port 1025 cid 2
14:17:09.962773[1] Captured at PortInput point, TSO not enabled, Checksum not offloaded and not verified, length 60.
        Segment[0] ---- 60 bytes:
        0x0000:  ffff ffff ffff 000c 29d2 6ca5 0806 0001
        0x0010:  0800 0604 0001 000c 29d2 6ca5 c0a8 0114
        0x0020:  0000 0000 0000 c0a8 0101 0000 0000 0000
        0x0030:  0000 0000 0000 0000 0000 0000
14:17:11.939401[2] Captured at PortInput point, TSO not enabled, Checksum not offloaded and not verified, length 60.
        Segment[0] ---- 60 bytes:
        0x0000:  ffff ffff ffff 000c 29d2 6ca5 0806 0001
        0x0010:  0800 0604 0001 000c 29d2 6ca5 c0a8 0114
        0x0020:  0000 0000 0000 c0a8 0101 0000 0000 0000
        0x0030:  0000 0000 0000 0000 0000 0000
14:17:12.164320[3] Captured at PortInput point, TSO not enabled, Checksum not offloaded and not verified, length 60.
        Segment[0] ---- 60 bytes:
        0x0000:  ffff ffff ffff 000c 29d2 6ca5 0806 0001
        0x0010:  0800 0604 0001 000c 29d2 6ca5 c0a8 0114
        0x0020:  0000 0000 0000 c0a8 0101 0000 0000 0000
        0x0030:  0000 0000 0000 0000 0000 0000
14:17:14.365208[4] Captured at PortInput point, TSO not enabled, Checksum not offloaded and not verified, length 60.
        Segment[0] ---- 60 bytes:
        0x0000:  ffff ffff ffff 000c 29d2 6ca5 0806 0001
        0x0010:  0800 0604 0001 000c 29d2 6ca5 c0a8 0114
        0x0020:  0000 0000 0000 c0a8 0101 0000 0000 0000
        0x0030:  0000 0000 0000 0000 0000 0000
14:17:14.939098[5] Captured at PortInput point, TSO not enabled, Checksum not offloaded and not verified, length 60.
        Segment[0] ---- 60 bytes:
        0x0000:  ffff ffff ffff 000c 29d2 6ca5 0806 0001
        0x0010:  0800 0604 0001 000c 29d2 6ca5 c0a8 0114
        0x0020:  0000 0000 0000 c0a8 0101 0000 0000 0000
        0x0030:  0000 0000 0000 0000 0000 0000
14:17:16.566210[6] Captured at PortInput point, TSO not enabled, Checksum not offloaded and not verified, length 60.
        Segment[0] ---- 60 bytes:
        0x0000:  ffff ffff ffff 000c 29d2 6ca5 0806 0001
        0x0010:  0800 0604 0001 000c 29d2 6ca5 c0a8 0114
        0x0020:  0000 0000 0000 c0a8 0101 0000 0000 0000
        0x0030:  0000 0000 0000 0000 0000 0000
14:17:17.939357[7] Captured at PortInput point, TSO not enabled, Checksum not offloaded and not verified, length 60.
        Segment[0] ---- 60 bytes:
        0x0000:  ffff ffff ffff 000c 29d2 6ca5 0806 0001
        0x0010:  0800 0604 0001 000c 29d2 6ca5 c0a8 0114
        0x0020:  0000 0000 0000 c0a8 0101 0000 0000 0000
        0x0030:  0000 0000 0000 0000 0000 0000
Join with dump thread failedDestroying session 1

Dumped 7 packet to console, dropped 0 packets.
Done.

As you can see above, the capture captured some packets and printed them to the console. If you are very good in reading hex code, this is what you want, but if you like the Wireshark interface, you can also dump the output to a Wireshark compatible file by using another option.

*******UPDATE March 13th 2020*******

Since ESXi version 6.7 you also have to specify the capture direction with “–dir”. The default value is “0” which only captures traffic to the vSwitch. A value of “1” captures the traffic from the vSwitch and the value “2” is what you want (in most cases), which captures traffic in both directions.

*******END UPDATE*******


The “-o” option will allow you to specify a file as the destination for the capture. This file can then be opened with Wireshark. Here is an example:

[root@localhost:~] pktcap-uw --switchport 50331651 --dir 2 -o /tmp/test.pcap
The switch port id is 0x03000003
The output file is /tmp/test.pcap
No server port specifed, select 36776 as the port
Local CID 2
Listen on port 36776
Accept...Vsock connection from port 1026 cid 2
Dump: 8, broken : 0, drop: 0, file err: 0Join with dump thread failedDestroying session 2

Dumped 8 packet to file /tmp/test.pcap, dropped 0 packets.
Done.

As you can see, the output was now dumped to a file, which can be downloaded from the ESXi server and then opened with Wireshark. To make it simpler, you could also mount an external share with the ESXi server and store the dump there.

You can also run multiple capture sessions by using the “&” sign between multiple versions of the capture command. This is very helpful if you would like to capture on both ends of the connection, assuming that both ends are on the same ESXi server.

The command would look like this:

[root@localhost:~] pktcap-uw --switchport 50331651 --dir 2 -o /tmp/test1.pcap & pktcap-uw --switchport 50331652 -o /tmp/test2.pcap &

Make sure to kill the capturing with CTRL-C multiple times to make sure it is really killed. To check if there is a capture still running, you can use this command:

sof |grep pktcap-uw |awk '{print $1}'| sort -u
38202

To kill captures still running, use this command:

kill $(lsof |grep pktcap-uw |awk '{print $1}'| sort -u)

This will kill all capture sessions.

From my point of view, this is really helpful to get traces from specific virtual machines. I will use this to provide insight of the traffic flow of the solutions, I will write about in future posts.

If you have any questions please use the comment function below. If you would like to provide feedback please contact me or use the comment function as well.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.