VXLAN WAN and IPSec

I did this post for two reasons. First I would like to demonstrate the possibility to extend a layer 2 network with VXLAN over a WAN connection and second to show the configuration of two VSR routers creating an IPSec tunnel with one router being on a dynamic IP address.

The reason for me, to implement this kind of scenario is the need for more virtualization space. My server at home has reached his limit and I needed more but couldn’t afford a new server for my home lab. But I still have my server in the cloud running, which has more than enough power to run the needed AirWave instance. So I decided to interconnect my home network with the internal server network .
As my provider provided router is not able to work with routes, I needed to create a layer 2 connection between my networks to be able to interconnect them. And it would be a good practice for my last VXLAN post and a very good example on how to use VXLAN, even if this should not be the main scenario for VXLAN.

Build the IPSec Tunnel

The first step is to build up the IPSec tunnel. This time I will use two VSR instances for the connection. So both ends will work with Comware 7 and from my last post on the IPSec topic I’ve got the question on how to set up the connection using Comware 7. This post will answer this question. Here is the scenario I need to setup:

The lab network is extended through the IPSec tunnel using VXLAN to the remote server. As described above, this is the only solution, but I will get a full-featured Comware router (hopefully) in the next months and than, I will be able to work with a fully routed network setup.

The two VSR routers are running on the ESXi Lab server and on the DC server. The router on the DC server will be the server router. This is just for my own terminology, as this router has a public static IP address, where as the VSR router on the ESXi lab server has to use the dynamically assigned IP address from the provider. This will bring some differences in the configuration.

Let’ start with the configuration of the VSR on the DC server, the one with a public static IP address.
The first step is to create a loopback interface, which will be used for the encapsulated protocol as the source interface. Even this did not belong to the IPSec configuration itself, it is helpful to create this interface now, as we need to define the traffic, which will be allowed to enter the IPSec tunnel and is it very easy, to include the source IP address in this definition.

interface LoopBack0
 description VXLAN-TUNNEL-ENDPOINT
 ip address 172.16.1.1 255.255.255.255

This will create a simple loopback interface with an IP address. No more is needed.

The next step is to create the rule, which is used to define the permitted traffic for the IPSec tunnel:

acl advanced 3000
 rule 10 permit udp source 172.16.1.1 0 destination 172.16.1.2 0

This creates an ACL, which allow udp traffic, as VXLAN is udp based, from the created loopback interface to the loopback interface on the other end of the tunnel. We will create this one on the other VSR router later on.

We will now create the transformation set:

ipsec transform-set tran1
 esp encryption-algorithm aes-cbc-128 3des-cbc 
 esp authentication-algorithm sha1

This will create a list of supported encryption and authentication algorithm, which will be offered to the other party. Both endpoints must support at least one of the same encryption and authentication algorithm to be able to create the tunnel.

The same rule applies to the IKE proposal:

ike proposal 100
 encryption-algorithm aes-cbc-128
 dh group2

Make sure, that both endpoints support the same encryption algorithm and are using the same dh group.

The next step is to secure the connection with a key. I use the keychain for this, with a single password. I do not have the need to use certificates.

ike keychain keychain1
 pre-shared-key hostname flomain.local key simple top_secret

We will now put everything together:

ike profile profile1
 keychain keychain1
 exchange-mode aggressive
 local-identity fqdn flomain.de
 match remote identity fqdn flomain.local
 proposal 100

This profile includes the keychain, I created with the password, the local identity is defined and the ike proposal is selected, if the correct remote identity is trying to connect to the VSR.

As the remote party will use a dynamically assigned IP address, the use of a simple IPSec policy, is not possible, instead, we need to use a policy template:

ipsec policy-template template1 1
 transform-set tran1 
 local-address xxx.xxx.xxx.xxx
 ike-profile profile1

This template will bring together the transformation set and the IKE profile. It will also need the local IP address, which is used by the remote party to connect to the router.

To be able to apply this to an interface, we need to connect the template to a profile:

ipsec policy home-vpn 10 isakmp template template1

Assign this profile to an interface:

interface GigabitEthernet2/0
 port link-mode route
 ip address xxx.xxx.xxx.xxx xxx.xxx.xxx.xxx
 ipsec apply policy home-vpn

Make sure, that this interface will be used if the router tries to reach the destination IP of the tunnel, which will be created later on the other VSR router.

Even not necessary, but good practice, define some global variables to make sure, that they are the same on both routers:

ipsec anti-replay window 1024
ipsec sa global-duration time-based 86400
ike invalid-spi-recovery enable
ike keepalive interval 30
ike keepalive timeout 90
ike identity fqdn flomain.de

Only the one on line 6, needs to be unique and should be equal to the one used on the IKE profile above.

The VSR router on the DC server is now configured to accept IPSec connections. We will now configure the VSR on the ESXi lab server. This is the VSR router, which has a dynamic assigned IP address.

Again, the first step is to create the loopback interface, which will be used as the tunnel source interface, of course, using a different IP address :

interface LoopBack0
 description VXLAN-TUNNEL-ENDPOINT
 ip address 172.16.1.2 255.255.255.255

We will also configure the acl to permit traffic for the IPSec tunnel:

acl advanced 3000
 rule 10 permit udp source 172.16.1.2 0 destination 172.16.1.1 0

The next step is to create the transformation set:

ipsec transform-set tran1
 esp encryption-algorithm aes-cbc-128 3des-cbc 
 esp authentication-algorithm sha1

Create the IKE proposal:

ike proposal 100
 encryption-algorithm aes-cbc-128
 dh group2

We will now create the keychain. To identify for which remote host the keychain should be used we include the public static IP of the VSR router on the DC server:

ike keychain keychain1
 pre-shared-key address xxx.xxx.xxx.xxx 0.0.0.0 key simple top_secret

Create the IKE proposal:

ike profile profile1
 keychain keychain1
 exchange-mode aggressive
 local-identity fqdn flomain.local
 match remote identity fqdn flomain.de
 proposal 100

Compared to the VSR on the DC server, only match remote identity and local-identity are swapped. The rest stays the same, due to the fact, that we are using the same profile names.

As the remote server is uniquely defined by its IP address, we did not need an IPSec template and can use the profile without the template:

ipsec policy home-vpn 10 isakmp
 transform-set tran1 
 security acl 3000 
 remote-address 89.163.185.211
 ike-profile profile1

The last step is to apply this profile to an interface:

interface GigabitEthernet1/0
 port link-mode route
 ip address dhcp-alloc
 ipsec apply policy home-vpn

We will also apply some global parameters:

ipsec anti-replay window 1024
ipsec sa global-duration time-based 86400
ike invalid-spi-recovery enable
ike keepalive interval 30
ike keepalive timeout 90
ike identity fqdn flomain.local

After this, the IPSec tunnel should be created and can be used, but only udp traffic is allowed, which means no traffic will go through the IPSec tunnel until we create the VXLAN tunnel.

Build the VXLAN Tunnel

Creating a VXLAN tunnel is already described in this post. To get detailed information about the implementation, please read this post before. In this section, I will only cover the configuration itself to have it consistent with the section above. I will not explain what the meaning behind the parameter is.

The first step is to enable L2VPN and define the VXLAN, which will be used for the control plane function:

l2vpn enable
reserved vxlan 1000

This is the same for both routers. If there is a difference in the configuration I will mention this explicitly.

The next step is to create the VSI:

vsi ngy
 description DC-NGY
 arp suppression enable
 vxlan 1

This will create the VSI “ngy” with arp supression enabled, as we will solve this later by using IS-IS to distribute the MAC addresses through the tunnel. This VSI will include VXLAN 1.

Interface Gigabit 1/0 is mapped to the internal lab network on the VSR router and therefore, the VSI is mapped to this interface:

interface GigabitEthernet1/0
 port link-mode route
 xconnect vsi ngy

Every packet, arriving untagged on this port will be mapped to the VSI and VXLAN 1. If you like to encapsulate multiple VLAN’s into a VXLAN tunnel, the xconnect command should look like this:

xconnect vsi vsi-name access-mode ethernet

This is currently not supported within the VSR router. On the VSR router you need to create subinterfaces for each VLAN and map them to the VSI.

As I’m lazy, I will use ENDP to create the VXLAN tunnel:

interface Tunnel0 mode nve
 source 172.16.1.1
 network-id 1
 vxlan neighbor-discovery server enable

This will create the ENDP server. The other host needs to be the client:

interface Tunnel0 mode nve
 source 172.16.1.2
 network-id 1
 vxlan neighbor-discovery client enable 172.16.1.1

The last step is to enable the IS-IS control plane:

vxlan-isis 
 negotiate-vni enable
 mac-synchronization enable

This will create a control plane and ensure that MAC addresses gets synchronized on all VTEP’s. This reduces the broadcast on the tunnel.

This will create a layer 2 extension of the network to the remote host, using VXLAN, secured with IPSec. As I have only a normal WAN connection, without a larger MTU size, I ran into some issues, when sending large packages to the remote site of the connection, as the max MTU in the tunnel is lower than in the network MTU in the local network. If you need those kind of connections, make sure, that you get a larger MTU from your provider.

From my personal point of view, VXLAN can be used to interconnect two sites. If you need to connect more than two sites, you can create loops, as VXLAN is only a layer 2 tunneling protocol without a build in loop prevention, but I’m pretty sure, that we will see something like this in the near future.

If you have any questions please use the comment function below. If you would like to provide feedback please contact me or use the comment function as well.