Speed Up vMotion

Nowadays, most of servers have been virtualized and I can’t imagine work without VMotion in my datacenter. Using VMotion, I can move (migrate) VMs between ESXi hosts without interruption.

Increasing configured RAM of VMs causes VMotion needs time to be completed. So in this article we discuss how to improve/speed up VMotion.

Best Practices for vMotion Networking:

Dedicate at least one gigabit ethernet adapter for vMotion

Provision at least one additional physical NIC as a failover NIC

vMotion traffic should not be routed, as this may cause latency issues

We can speed up VMotion by various ways:

10Gbit network

The easiest way to accelerate vMotion is using 10Gbit connection. This method provides not only better bandwidth but also more concurrent VMotions. If a vMotion is configured with a 1GB line speed, it is possible four concurrent vMotion, while with a 10GB link speed – eight concurrent vMotions per host. Dedicated 10Gbit links just for vMotion are not often scenario, so if you have a Distributed Switch, you can use Network IO Control (NetIOC)  to prioritize vMotion traffic.

Distributed Switch and LACPv2

If you are an administrator of VMware infrastructure with Enterprise Plus license installed, you are lucky. Enterprise + lets you use all features such Distributed Switch (VMware or Cisco Nexus 1000v). Why is it useful for VMotion? Distributed switch supports the Link Aggregation Control Protocol (LACP) (since version 5.1) or LACPv2 (since version 5.5). LACPv2 lets you use following balancing alghoritms:

Destination IP address

Destination IP address and TCP/UDP port

Destination IP address and VLAN

Destination IP address, TCP/UDP port and VLAN

Destination MAC address

Destination TCP/UDP port

Source IP address

Source IP address and TCP/UDP port

Source IP address and VLAN

Source IP address, TCP/UDP port and VLAN

Source MAC address

Source TCP/UDP port

Source and destination IP address

Source and destination IP address and TCP/UDP port

Source and destination IP address and VLAN

Source and destination IP address, TCP/UDP port and VLAN

Source and destination MAC address

Source and destination TCP/UDP port

Source port ID

VLAN

Distributed Switch in vSphere 5.1 supports only one algorithm:

Source and destination IP address

Pros:

Even if you have two hosts VMware cluster with Distributed Switch 5.5 configured (LACPv2), two or more vMotions (running at the same time) between two ESXi hosts should use two or more uplinks (one uplink per VMotion).

Cons:

Required Enterprise Plus license

LACPv2 should be supported by Physical Switch

Standard Switch (vSwitch) and Etherchannel

If your VMware hosts are licensed lower than Ent+, you can speed up VMotion by using static Etherchannel. It provides fault-tolerance and high-speed links between switches and servers by grouping two to eight physical Ethernet links to create a logical Ethernet link with additional failover links.

Pros:

Any ESXi license is supported

Easy to configure

Cons:

It supports only Source and Destination IP address (hash). Example, two VMotions  between ESXi A and ESXi B would use the same one uplink because IP Source and IP Destination are the same.

Multi-NIC vMotion

Multi-NIC vMotion was introduced in vSphere 5.0. This feature provides load balancing the vMotion network traffic over multiple network adapters. It means that one VMotion session is balanced between all available vmknic.

Let’s assume that we have two ESXi hosts and we want to migrate a VM with 512GB RAM configured. To show how Multi-NIC vMotion exactly works, there are below two scenarios:Multi-NIC vMotion - Scenario AThe scenario A: Host A with 1 x 10Gbit NIC and Host B with 4 x 1Gbit NICs

Multi-NIC vMotion - ScenarioThe scenario B: Host A with 2 x 10Gbit NICs and Host B with 1 x 10Gbit and 3 x 1 Gbit NICs

When a migration is initiated, VMkernel will pair source and destination vMotion NICs based on link speed, pairing multiple NICs to a single NIC as needed to fully utilize the link. VMkernel opens a TCP connection per network adapter pair and transparently load balances the migration traffic over all the connections.

In scenario A, VMkernel will pair the first 10GbE NIC on the ESXi A host  with the four 1GbE NICs on the ESXi B host, thereby resulting in a total of four TCP connections.

In scenario B, VMkernel will pair the first 10GbE NIC on the ESXi A host with the sole 10GbE NIC on the ESXi B host, then VMkernel will pair the second 10GbE NIC on the ESXi A host with the two 1GbE NICs on the ESXi B host, thereby resulting in a total of four TCP connections.

Pros:

No special license required

Easy to configure

Cons:

None :)

To configure Multi-NIC vMotion, please follow steps mentioned in post here.

Even you have Distributed Switch, I recommend to use Multi-NIC vMotion. If you use vSwitch and vDS, VMkernel port group may be present on a standard switch and the other one on a distributed switch.

Configuration Multi-NIC vMotion with Cisco Nexus 1000v requires some steps mentioned in my another article here.

Conclusion

I used to configure Static Etherchannel to improve vMotion till release of vSphere 5.0 and introduce of Multi-NIC vMotion. Nowadays, almost my VMware designs are based on Multi-NIC vMotion. It’s simple to configure and works perfect without additional hardware requirements and costs.