How ML2/VXLAN works


April 1, 2014 by kimizhang

My setup:
1 controller node + 2 compute nodes
RDO Havana 2013.2.2,
CentOS 6.5, OpenVSwitch 1.11.0

VXLAN local IPs:


1. Setup VXLAN with ML2.

After packstack installation, ML2 is not installed by default, we have to configure it manually.

On controller node:

yum install  openstack-neutron-ml2 python-pyudev

Edit  /etc/neutron/neutron.conf

core_plugin =neutron.plugins.ml2.plugin.Ml2Plugin,

Change plugin.ini link

#unlink /etc/neutron/plugin.ini

#ln -s /etc/neutron/plugins/ml2/ml2_conf.ini /etc/neutron/plugin.ini

Edit  /etc/neutron/plugin.ini

type_drivers = vxlan
tenant_network_types = vxlan
mechanism_drivers = openvswitch
vni_ranges = 1001:2000
vxlan_group =
sql_connection = mysql://neutron:83105f1d6ded47cc@
firewall_driver = dummy_value_to_enable_security_groups_in_server

Edit /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini on every node

local_ip=      #Use 102 for compute-1, 103 for compute-2
tunnel_types = vxlan

Database creation:

mysql -e “drop database if exists neutron_ml2;”
mysql -e “create database neutron_ml2 character set utf8;”
mysql -e “grant all on neutron_ml2.* to ‘neutron’@’%’;”
neutron-db-manage –config-file /usr/share/neutron/neutron-dist.conf –config-file /etc/neutron/neutron.conf –config-file /etc/neutron/plugin.ini upgrade head

Restart neutron services

#service neutron-server restart
#service neutron-openvswitch-agent restart    #On every node

Check if tunnel is established on controller:

[root@controller ~]# ovs-vsctl show

Bridge br-tun
Port patch-int
Interface patch-int
type: patch
options: {peer=patch-tun}
Port br-tun
Interface br-tun
type: internal
 Port “vxlan-”
Interface “vxlan-″
type: vxlan
options: {in_key=flow, local_ip=”″, out_key=flow, remote_ip=””}
 Port “vxlan-”
Interface “vxlan-″
type: vxlan
options: {in_key=flow, local_ip=”″, out_key=flow, remote_ip=””}

We see 2 tunnels created between controller to compute-1 and compute-2.

Check if tunnel is established on compute (compute-1 as example):

[root@compute ~]# ovs-vsctl show

Bridge br-tun
Port “vxlan-”
Interface “vxlan-″
type: vxlan
options: {in_key=flow, local_ip=”″, out_key=flow, remote_ip=””}
Port “vxlan-”
Interface “vxlan-″
type: vxlan
options: {in_key=flow, local_ip=”″, out_key=flow, remote_ip=””}

The same, 2 tunnels created between compute-1 and controller and compute-2.

Try to launch some VM instances, to test how VM traffic go through VXLAN tunnels.

nova boot –flavor 1 –image cirros –num-instances 2 –nic net-id=<net-uuid> vm

Each compute node should take one VM:

#nova list –fields name,status,power_state,host,networks
| ID | Name | Status | Power State | Host | Networks |
| 2bc01296-f8d4-48ce-a600-5acf83ee2bbf | vm-2bc01296-f8d4-48ce-a600-5acf83ee2bbf | ACTIVE | Running | compute2 | testnet= |
| 4f63feaf-e92a-4045-8e45-d3160c99fb84 | vm-4f63feaf-e92a-4045-8e45-d3160c99fb84 | ACTIVE | Running | compute | testnet= |

2. Unicast packets of VM traffic

Send an ICMP packet from controller (L3 agent router namespace) to a VM. Capture the packet to see the VXLAN packet.

[root@controller ~]# ip netns exec qrouter-28f6fe53-1f94-4355-81e3-85a2aad7b665 ping -c 1192.168.1.4


We can see the Outer IP header + VXLAN header with VNI 1000, then it’s Inner IP header plus ICMP payload. We can see the VXLAN packet is unicast between controller local IP to compute-1 local IP

3. Broadcast or Multicase packet of VM traffic

Neutron and OpenvSwitch handle VM broadcast and multicast in the same way, here takes broadcast as example.

Send one ARP request broadcast packet from controller (L3 agent router namespace) to the VM network. Capture the packet to see the VXLAN packet.

[root@controller ~]# ip netns exec qrouter-28f6fe53-1f94-4355-81e3-85a2aad7b665 arping -c 1 -I qr-f3d1a9ea-9a








We can see the broadcast is actually sent as 2 unicast packets from controller to compute-1 and compute-2. Then compute-1 holding the VM with IP replies the ARP request.

The picture from official Openstack documentation explains this situation:


This kind of packet flood is obvious not ideal especially in large size of deployment. However, Openstack support better mechanisms to handle this. See following chapters.

4. VXLAN uses multicast between VTEP for VM broadcast/multicast traffic

According to VXLAN specification draft, chapter 4.2:

If VM traffic type is broadcast or multicast, VXLAN VTEPs use multicast to send them among.

Linux bridge support this way of working.

However, OpenvSwitch does not support this way, as I saw in last chapter, it uses multiple unicast between VTEPs.

See OpenvSwitch Q&A:;a=blob_plain;f=FAQ;hb=HEAD

Q: How much of the VXLAN protocol does Open vSwitch currently support?

A: Open vSwitch currently supports the framing format for packets on the
wire. There is currently no support for the multicast aspects of VXLAN.
To get around the lack of multicast support, it is possible to
pre-provision MAC to IP address mappings either manually or from a

So the configuration “vxlan_group = <a multicast ip, e.g>” in /etc/neutron/plugins/ml2/ml2_conf.ini only applies to Linux Bridge mechanism driver.

5. ML2 L2 population mechanism driver

From the scenario of VM broadcast/multicast traffic handling above, we know there are many useless broadcast emulation(unicast) packets flying around VTEPs. L2 population is introduced to solving the flooding.

When using the ML2 plugin with tunnels and a new port goes up, ML2 sends a update_port_postcommit notification which is picked up and processed by the l2pop mechanism driver. l2 pop then gathers the IP and MAC of the port, as well as the host that the port was scheduled on; It then sends an RPC notification to all layer 2 agents.

So every agent can learn the MAC-IP-VTEP mappings, when broadcast packet comes from VM, VTEP only sends it to related VTEPs, no multicast emulation needed.
This picture explains:


Let’s go through how it works.

Add l2population in mechanism driver list in /etc/neutron/plugins/ml2/ml2_conf.ini on controller node.

mechanism_drivers = openvswitch,l2population

Enable l2population on every openvswitch agent node, in /etc/neutron/plugins/openvswitch/ovs_neutron_plugin.ini

l2_population = True

Restart all neutron-server and openvswitch agent services on controller and compute nodes.

service neutron-server restart    #on controller
service neutron-openvswitch-agent restart   #on all nodes

Now L2 population should start to work.

…To be continued with detailed packet level analysis…


2 thoughts on “How ML2/VXLAN works

  1. alvin says:

    thanks for your sharing. it’s useful.
    but, i am encountering a problem, when i set multi-nodes.
    there’s maybe something wrong in my answer file.
    could u share your answer file for setting multi-nodes


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: