Neutron L2 Gateway + HP 5930 switch OVSDB integration, for VXLAN bridging and routing

Neutron L2 Gateway is a new Openstack project working as a Neutron service plugin. It supports only one use case at the moment, which is to bridge Neutron VxLAN tenant network to physical VLAN network via OVSDB Hardware VTEP enabled physical switch.

This blog will go through all steps to configure Openstack and physical switch to make Neutron L2-GW working.

HP 5930 switch is used in this integration.

Red Hat Openstack Platform 7 (Kilo) is used as Openstack environment.

L2 population mechanism driver needs to be enabled to make l2-GW working.

Here's network diagram:

Neutron L2-GW to HP 5930 switch

HP 5930 switch configuration

First, let's configure HP 5930 switch.

Get into system-view:

<VGP_LCC_EOR_02>system-view  
[VGP_LCC_EOR_02]

Enable l2vpn:

l2vpn enable  

Configure a passive TCP connection for OVSDB, we use port 6632 here:

ovsdb server ptcp port 6632  

Enable OVSDB server:

ovsdb server enable  

Enable VTEP process:

vtep enable  

Configure 10.7.31.1 as VTEP source IP:

tunnel global source-address 10.7.31.1  

Configure vtep access port, in our example, we use Bridge-Aggregation1:

interface Bridge-Aggregation1  
 vtep access port

Now HP switch part configuration is done, we can dump the OVSDB to see what it looks like, run ovsdb-client from any Linux machine:

[stack@under ~(stackrc)]$ ovsdb-client dump --pretty tcp:10.7.31.1:6632
Arp_Sources_Local table  
_uuid locator src_mac  
----- ------- -------

Arp_Sources_Remote table  
_uuid locator src_mac  
----- ------- -------

Global table  
_uuid                                managers switches  
------------------------------------ -------- --------------------------------------
58708319-c85e-44a5-b221-4c2d3cde6004 []       [8f3d25db-f219-443e-9771-1d8e5219b5d9]

Logical_Binding_Stats table  
_uuid bytes_from_local bytes_to_local packets_from_local packets_to_local  
----- ---------------- -------------- ------------------ ----------------

Logical_Router table  
_uuid description name static_routes switch_binding  
----- ----------- ---- ------------- --------------

Logical_Switch table  
_uuid description name tunnel_key  
----- ----------- ---- ----------

Manager table  
_uuid inactivity_probe is_connected max_backoff other_config status target  
----- ---------------- ------------ ----------- ------------ ------ ------

Mcast_Macs_Local table  
MAC _uuid ipaddr locator_set logical_switch  
--- ----- ------ ----------- --------------

Mcast_Macs_Remote table  
MAC _uuid ipaddr locator_set logical_switch  
--- ----- ------ ----------- --------------

Physical_Locator table  
_uuid dst_ip encapsulation_type  
----- ------ ------------------

Physical_Locator_Set table  
_uuid locators  
----- --------

Physical_Port table  
_uuid                                description name                  port_fault_status vlan_bindings vlan_stats  
------------------------------------ ----------- --------------------- ----------------- ------------- ----------
28d3da91-7666-492e-9c20-fc960249a7b6 ""          "Bridge-Aggregation1" [UP]              {}            {}

Physical_Switch table  
_uuid                                description management_ips name             ports                                  switch_fault_status tunnel_ips    tunnels  
------------------------------------ ----------- -------------- ---------------- -------------------------------------- ------------------- ------------- -------
8f3d25db-f219-443e-9771-1d8e5219b5d9 ""          []             "VGP_LCC_EOR_02" [28d3da91-7666-492e-9c20-fc960249a7b6] []                  ["10.7.31.1"] []

Tunnel table  
_uuid bfd_config_local bfd_config_remote bfd_params bfd_status local remote  
----- ---------------- ----------------- ---------- ---------- ----- ------

Ucast_Macs_Local table  
MAC _uuid ipaddr locator logical_switch  
--- ----- ------ ------- --------------

Ucast_Macs_Remote table  
MAC _uuid ipaddr locator logical_switch  
--- ----- ------ ------- --------------

We can see this switch in Physical_Switch table and a port in Physical_Port table.

L2-GW installation and integration with Neutron

On controller node, fetch latest networking-l2gw codes from stable/kilo branch

git clone -b stable/kilo  https://github.com/openstack/networking-l2gw.git  

Install it by pip:

pip install ~/networking-l2gw/  

Add l2-gw service plugin to neutron server:

[root@overcloud-controller-0 ~]# grep ^service_pl /etc/neutron/neutron.conf
service_plugins =router,networking_l2gw.services.l2gateway.plugin.L2GatewayPlugin  

Copy l2-gw example config files to /etc/neutron/:

[root@overcloud-controller-0 ~]#cp  /usr/etc/neutron/l2g* /etc/neutron/

Add HP 5930 OVSDB connection to l2gateway_agent.ini, ovsdb1 is user defined identifier of physical switch.

[root@overcloud-controller-0 ~]# grep ^ovsdb /etc/neutron/l2gateway_agent.ini
ovsdb_hosts = 'ovsdb1:10.7.31.1:6632'  

Make neutron-l2gateway-agent.service systemd unit file, so that we can control the service by systemctl:

[root@overcloud-controller-0 system]# cat /usr/lib/systemd/system/neutron-l2gateway-agent.service
[Unit]
Description=OpenStack Neutron L2 Gateway Agent  
After=syslog.target network.target

[Service]
Type=simple  
User=neutron  
ExecStart=/usr/bin/neutron-l2gateway-agent   --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/l2gateway_agent.ini  
PrivateTmp=false  
KillMode=process

[Install]
WantedBy=multi-user.target  

Load the unit file we created:

[root@overcloud-controller-0]# systemctl daemon-reload

Stop neutron services:

pcs resource disable neutron-server-clone  

Update neutron database schema:

neutron-l2gw-db-manage --config-file /etc/neutron/neutron.conf upgrade head  

Start neutron services:

pcs resource enable neutron-server-clone  

Start neutron-l2-gateway service:

systemctl start neutron-l2gateway-agent.service  

Now neutron l2-GW is enable, we should be able to see the L2 Gateway agent from neutron agent-list:

[root@overcloud-controller-0 ~]# neutron agent-list
+--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+
| id                                   | agent_type         | host                               | alive | admin_state_up | binary                    |
+--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+
| 097320d1-9e7c-468a-bcee-948c560b65c9 | Metadata agent     | overcloud-controller-0.localdomain | :-)   | True           | neutron-metadata-agent    |
| 0ef52d27-9b91-40d4-b9dd-d24ff4f86ef7 | Open vSwitch agent | overcloud-controller-0.localdomain | :-)   | True           | neutron-openvswitch-agent |
| 304a3e9b-3a6f-4ae6-b2af-02891fa63d0f | L2 Gateway agent   | overcloud-controller-0.localdomain | :-)   | True           | neutron-l2gateway-agent   |
| 4f81e156-e679-4d76-a816-9b976cf9d62f | Open vSwitch agent | overcloud-compute-0.localdomain    | :-)   | True           | neutron-openvswitch-agent |
| 71f7e395-045f-47a2-9575-9976604eac7c | L3 agent           | overcloud-controller-0.localdomain | :-)   | True           | neutron-l3-agent          |
| 81f254a4-dbf2-48e2-afb1-9456201c2631 | DHCP agent         | overcloud-controller-0.localdomain | :-)   | True           | neutron-dhcp-agent        |
+--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+

Also we can see the physical switch is registered into neutron DB:

MariaDB [ovs_neutron]> select * from physical_switches;  
+--------------------------------------+----------------+-----------+------------------+---------------------+
| uuid                                 | name           | tunnel_ip | ovsdb_identifier | switch_fault_status |
+--------------------------------------+----------------+-----------+------------------+---------------------+
| 8f3d25db-f219-443e-9771-1d8e5219b5d9 | VGP_LCC_EOR_02 | 10.7.31.1 | ovsdb1           | NULL                |
+--------------------------------------+----------------+-----------+------------------+---------------------+

L2-GW usage

Create a neutron network (VxLAN overlay):

[root@overcloud-controller-0 ~(admin)]# neutron net-create test-l2gw
Created a new network:  
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| id                        | 49485e07-77db-4d43-8b20-068597f744a5 |
| mtu                       | 0                                    |
| name                      | test-l2gw                            |
| provider:network_type     | vxlan                                |
| provider:physical_network |                                      |
| provider:segmentation_id  | 100                                  |
| router:external           | False                                |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tenant_id                 | 7128bb366ffd47ffb445965daab45751     |
+---------------------------+--------------------------------------+

We can see neutron randomly allocates VNI ID 100 to this tenant network.
Let's create subnet for it:

neutron subnet-create --allocation-pool start=10.7.30.201,end=10.7.30.210 test-l2gw 10.7.30.0/24  

Launch a VM in this network:

nova boot --image cirros --flavor m1.tiny --nic net-id=49485e07-77db-4d43-8b20-068597f744a5 test-vm  

Check VM launching status and IP address allocated:

[root@overcloud-controller-0 ~(admin)]# nova list
+--------------------------------------+---------+--------+------------+-------------+-----------------------+
| ID                                   | Name    | Status | Task State | Power State | Networks              |
+--------------------------------------+---------+--------+------------+-------------+-----------------------+
| 15e9aefb-cddd-4be6-a74e-a3098e17f26c | test-vm | ACTIVE | -          | Running     | test-l2gw=10.7.30.202 |
+--------------------------------------+---------+--------+------------+-------------+-----------------------+

We can see VM is running, with IP 10.7.30.202.

We can check VXLAN tunnels created on controller and compute nodes:

[root@overcloud-controller-0 ~(admin)]# ovs-vsctl show | grep vxlan -A1
        Port "vxlan-0a14699a"
            Interface "vxlan-0a14699a"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="10.20.105.153", out_key=flow, remote_ip="10.20.105.154"}

[root@overcloud-compute-0 ~]# ovs-vsctl show | grep vxlan -A1
        Port "vxlan-0a146999"
            Interface "vxlan-0a146999"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="10.20.105.154", out_key=flow, remote_ip="10.20.105.153"}

So we can see at the moment, both controller and compute node has only 1 tunnel created to each other. This makes VM to DHCP server traffic working.

Now let's create a l2-gateway using neutron-l2gw client:

neutron-l2gw l2-gateway-create --device name="VGP_LCC_EOR_02",interface_names="Bridge-Aggregation1"  gw1  

Then we can now create a l2-gateway-connection, to bridge test-l2gw tenant network to gw1 for VLAN 2000:

neutron-l2gw l2-gateway-connection-create --default-segmentation-id 2000 gw1 test-l2gw  

We can try to ping hosts in VLAN 2000, from controller dhcp namespace and VM.

[root@overcloud-controller-0 ~(admin)]# ip netns exec qdhcp-49485e07-77db-4d43-8b20-068597f744a5 ping 10.7.30.32
PING 10.7.30.32 (10.7.30.32) 56(84) bytes of data.  
64 bytes from 10.7.30.32: icmp_seq=61 ttl=64 time=0.493 ms  
64 bytes from 10.7.30.32: icmp_seq=62 ttl=64 time=0.400 ms  
64 bytes from 10.7.30.32: icmp_seq=63 ttl=64 time=0.443 ms  
...

cirros$ ping 10.7.30.33  
PING 10.7.30.33 (10.7.30.33): 56 data bytes  
64 bytes from 10.7.30.33: seq=0 ttl=64 time=2.007 ms  
64 bytes from 10.7.30.33: seq=1 ttl=64 time=1.034 ms  
...

cirros$ ping 10.7.30.32  
PING 10.7.30.32 (10.7.30.32): 56 data bytes  
64 bytes from 10.7.30.32: seq=0 ttl=64 time=1.824 ms  
64 bytes from 10.7.30.32: seq=1 ttl=64 time=1.007 ms  
...

Pings work!!!

Let's try to check what's making that working under the hood.

Check VXLAN tunnels on controller and compute nodes, we will see tunnels created between them and HP 5930 switch.

[root@overcloud-controller-0 ~(admin)]# ovs-vsctl show | grep vxlan -A1
        Port "vxlan-0a14699a"
            Interface "vxlan-0a14699a"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="10.20.105.153", out_key=flow, remote_ip="10.20.105.154"}
        Port "vxlan-0a071f01"
            Interface "vxlan-0a071f01"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="10.20.105.153", out_key=flow, remote_ip="10.7.31.1"}

[root@overcloud-compute-0 ~]# ovs-vsctl show | grep vxlan -A1
        Port "vxlan-0a146999"
            Interface "vxlan-0a146999"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="10.20.105.154", out_key=flow, remote_ip="10.20.105.153"}
        Port "vxlan-0a0713f4"
            Interface "vxlan-0a0713f4"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="10.20.105.154", out_key=flow, remote_ip="10.7.31.1"}

Dump the OVSDB of physical switch:

[stack@under ~(stackrc)]$ ovsdb-client dump --pretty tcp:10.7.31.1:6632
Arp_Sources_Local table  
_uuid locator src_mac  
----- ------- -------

Arp_Sources_Remote table  
_uuid locator src_mac  
----- ------- -------

Global table  
_uuid                                managers switches  
------------------------------------ -------- --------------------------------------
58708319-c85e-44a5-b221-4c2d3cde6004 []       [8f3d25db-f219-443e-9771-1d8e5219b5d9]

Logical_Binding_Stats table  
_uuid bytes_from_local bytes_to_local packets_from_local packets_to_local  
----- ---------------- -------------- ------------------ ----------------

Logical_Router table  
_uuid description name static_routes switch_binding  
----- ----------- ---- ------------- --------------

Logical_Switch table  
_uuid                                description name                                   tunnel_key  
------------------------------------ ----------- -------------------------------------- ----------
c08ad894-886f-452a-a528-216f36d48958 "test-l2gw" "49485e07-77db-4d43-8b20-068597f744a5" 100

Manager table  
_uuid inactivity_probe is_connected max_backoff other_config status target  
----- ---------------- ------------ ----------- ------------ ------ ------

Mcast_Macs_Local table  
MAC         _uuid                                ipaddr locator_set                          logical_switch  
----------- ------------------------------------ ------ ------------------------------------ ------------------------------------
unknown-dst aeaab3ec-a0ae-4937-8cda-14ca48039be8 ""     e542b5fe-70c4-4872-a741-f334d1bf7021 c08ad894-886f-452a-a528-216f36d48958

Mcast_Macs_Remote table  
MAC _uuid ipaddr locator_set logical_switch  
--- ----- ------ ----------- --------------

Physical_Locator table  
_uuid                                dst_ip          encapsulation_type  
------------------------------------ --------------- ------------------
de8e7621-023a-4dfe-bc79-1bbd97962c71 "10.20.105.153" "vxlan_over_ipv4"  
6047ff6f-7662-44d0-8e49-b1c6afb51e68 "10.20.105.154" "vxlan_over_ipv4"  
7e83e09d-2ee3-4c8d-8784-efae6b780c15 "10.7.31.1"     "vxlan_over_ipv4"

Physical_Locator_Set table  
_uuid                                locators  
------------------------------------ --------------------------------------
e542b5fe-70c4-4872-a741-f334d1bf7021 [7e83e09d-2ee3-4c8d-8784-efae6b780c15]

Physical_Port table  
_uuid                                description name                  port_fault_status vlan_bindings                               vlan_stats  
------------------------------------ ----------- --------------------- ----------------- ------------------------------------------- ----------
28d3da91-7666-492e-9c20-fc960249a7b6 ""          "Bridge-Aggregation1" [UP]              {2000=c08ad894-886f-452a-a528-216f36d48958} {}

Physical_Switch table  
_uuid                                description management_ips name             ports                                                                        switch_fault_status tunnel_ips    tunnels  
------------------------------------ ----------- -------------- ---------------- ---------------------------------------------------------------------------- ------------------- ------------- ----------------------------------------------------------------------------
8f3d25db-f219-443e-9771-1d8e5219b5d9 ""          []             "VGP_LCC_EOR_02" [28d3da91-7666-492e-9c20-fc960249a7b6] []                  ["10.7.31.1"] [340ae640-ad8c-4dc7-9c61-1f84c9c462aa, 68629911-a3d0-4793-8a06-583b544dd827]

Tunnel table  
_uuid                                bfd_config_local                                               bfd_config_remote                                       bfd_params                                                                                                            bfd_status                                                                           local                                remote  
------------------------------------ -------------------------------------------------------------- ------------------------------------------------------- --------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------ ------------------------------------ ------------------------------------
68629911-a3d0-4793-8a06-583b544dd827 {bfd_dst_ip="10.7.31.1", bfd_dst_mac="188:234:250:108:94:141"} {bfd_dst_ip="10.20.105.153", bfd_dst_mac="0:0:0:0:0:0"} {check_tnl_key="0", cpath_down="0", decay_min_rx="100", enable="1", forwarding_if_rx="1", min_rx="100", min_tx="100"} {diagnostic="", forwarding="0", remote_diagnostic="", remote_state=down, state=down} 7e83e09d-2ee3-4c8d-8784-efae6b780c15 de8e7621-023a-4dfe-bc79-1bbd97962c71  
340ae640-ad8c-4dc7-9c61-1f84c9c462aa {bfd_dst_ip="10.7.31.1", bfd_dst_mac="188:234:250:108:94:141"} {bfd_dst_ip="10.20.105.154", bfd_dst_mac="0:0:0:0:0:0"} {check_tnl_key="0", cpath_down="0", decay_min_rx="100", enable="1", forwarding_if_rx="1", min_rx="100", min_tx="100"} {diagnostic="", forwarding="0", remote_diagnostic="", remote_state=down, state=down} 7e83e09d-2ee3-4c8d-8784-efae6b780c15 6047ff6f-7662-44d0-8e49-b1c6afb51e68

Ucast_Macs_Local table  
MAC                 _uuid                                ipaddr locator                              logical_switch  
------------------- ------------------------------------ ------ ------------------------------------ ------------------------------------
"2c:60:0c:c0:97:c9" 1b74b46b-5afb-41ad-ab84-faf1331d0c72 ""     7e83e09d-2ee3-4c8d-8784-efae6b780c15 c08ad894-886f-452a-a528-216f36d48958
"2c:60:0c:c0:a6:c1" 13996ca9-3de0-4bea-b815-036a07aa54d4 ""     7e83e09d-2ee3-4c8d-8784-efae6b780c15 c08ad894-886f-452a-a528-216f36d48958
"2c:60:0c:c0:a9:3b" 9d1c3c28-8ced-4e74-a1a9-bb3a27a41267 ""     7e83e09d-2ee3-4c8d-8784-efae6b780c15 c08ad894-886f-452a-a528-216f36d48958

Ucast_Macs_Remote table  
MAC                 _uuid                                ipaddr        locator                              logical_switch  
------------------- ------------------------------------ ------------- ------------------------------------ ------------------------------------
"fa:16:3e:96:3b:ce" 18dd6787-d5dc-42dc-97eb-285faf5b23ff "10.7.30.202" 6047ff6f-7662-44d0-8e49-b1c6afb51e68 c08ad894-886f-452a-a528-216f36d48958
"fa:16:3e:d1:44:85" 109c752e-1b97-4169-b939-6ada0a8ec703 "10.7.30.201" de8e7621-023a-4dfe-bc79-1bbd97962c71 c08ad894-886f-452a-a528-216f36d48958

From Tunnel table, we can see 2 VXLAN tunnels created between physical switch and controller and compute nodes.
From Logical_Switch table, a logical switch is created with VNI ID(tunnel_key) 100, then from Physical_Port table, we can see VLAN 2000 is bound to that logical switch.
From Ucast_Macs_Remote table, we can see MAC of 10.7.30.201 and 10.7.30.202, which are VM and dhcp server's MACs.

Check tunnel interfaces from HP 5930 console:

[VGP_LCC_EOR_02]dis interface Tunnel
Tunnel0  
Current state: UP  
Line protocol state: UP  
Description: created by ovsdb  
Bandwidth: 64 kbps  
Maximum transmission unit: 1464  
Internet protocol processing: Disabled  
Last clearing of counters: Never  
Tunnel source unknown, destination 10.20.105.153  
Tunnel protocol/transport UDP_VXLAN/IP

Tunnel1  
Current state: UP  
Line protocol state: UP  
Description: created by ovsdb  
Bandwidth: 64 kbps  
Maximum transmission unit: 1464  
Internet protocol processing: Disabled  
Last clearing of counters: Never  
Tunnel source unknown, destination 10.20.105.154  
Tunnel protocol/transport UDP_VXLAN/IP  

2 Tunnels are shown there as well.

VXLAN routing problem

On HP 5930 switch, for VLAN 2000, we have a L3 IP address working as gateway of this network:

[VGP_LCC_EOR_02]display current-configuration interface Vlan-interface 2000
interface Vlan-interface2000  
 ip address 10.7.30.1 255.255.255.0

After l2-gateway-connection is created, this default GW IP can not be reached from VM, it's because of limitation of TRIDENT 2 chipset in HP 5930 switch, details can be checked from this blog: LAYER-3 SWITCHING OVER VXLAN REVISITED

Workaround of VXLAN routing problem by adding a loopback cable in the switch

To workaround the HW limitation, we could make a loopback cable on HP 5930 switch, bridge out VXLAN to VLAN first, then through the loopback cable, reaching in the L3 GW as normal VLAN routing.

The diagram is:

Neutron-L2-GW-to-HP-5930-Switch-VXLAN-Routing-w-Loopback-Cable

Loopback cable is connected between FGE1/0/27 and FGE1/0/28, here we configure FGE1/0/27 as vtep access port, and FGE1/0/28 as normal VLAN trunk port. Dotted red line shows the packet transit path between VM and L3 GW.

On HP 5930 console:

[VGP_LCC_EOR_02]dis current-configuration interface FortyGigE 1/0/27
#
interface FortyGigE1/0/27  
 port link-mode bridge
 port link-type trunk
 undo port trunk permit vlan 1
 vtep access port
#
return

[VGP_LCC_EOR_02]dis current-configuration interface FortyGigE 1/0/28
#
interface FortyGigE1/0/28  
 port link-mode bridge
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 2000
#
return  

We need to remove previously created l2-gateway-connection and l2-gateway:

neutron-l2gw l2-gateway-connection-delete 4fb1c9eb-de29-442c-a946-190f5f99cf17  
neutron-l2gw l2-gateway-delete gw1  

Re-create l2-gateway gw1 with interface FortyGigE1/0/27:

neutron-l2gw l2-gateway-create --device name="VGP_LCC_EOR_02",interface_names="FortyGigE1/0/27"  gw1  

Create l2-gateway-connection, to bridge test-l2gw tenant network to gw1 for VLAN 2000

neutron-l2gw l2-gateway-connection-create --default-segmentation-id 2000 gw1 test-l2gw  

Now try to ping L3 GW 10.7.30.1 from VM.

$ ping 10.7.30.1
PING 10.7.30.1 (10.7.30.1): 56 data bytes  
64 bytes from 10.7.30.1: seq=1 ttl=64 time=1.463 ms  
64 bytes from 10.7.30.1: seq=2 ttl=64 time=2.606 ms  
64 bytes from 10.7.30.1: seq=3 ttl=64 time=1.036 ms  
...

It works now!!!