Neutron L2 Gateway + HP 5930 switch OVSDB integration, for VXLAN bridging and routing

Neutron L2 Gateway is a new Openstack project working as a Neutron service plugin. It supports only one use case at the moment, which is to bridge Neutron VxLAN tenant network to physical VLAN network via OVSDB Hardware VTEP enabled physical switch.

This blog will go through all steps to configure Openstack and physical switch to make Neutron L2-GW working.

HP 5930 switch is used in this integration.

Red Hat Openstack Platform 7 (Kilo) is used as Openstack environment.

L2 population mechanism driver needs to be enabled to make l2-GW working.

Here's network diagram:

Neutron L2-GW to HP 5930 switch

HP 5930 switch configuration

First, let's configure HP 5930 switch.

Get into system-view:

<VGP_LCC_EOR_02>system-view
[VGP_LCC_EOR_02]

Enable l2vpn:

l2vpn enable 

Configure a passive TCP connection for OVSDB, we use port 6632 here:

ovsdb server ptcp port 6632

Enable OVSDB server:

ovsdb server enable

Enable VTEP process:

vtep enable

Configure 10.7.31.1 as VTEP source IP:

tunnel global source-address 10.7.31.1

Configure vtep access port, in our example, we use Bridge-Aggregation1:

interface Bridge-Aggregation1
 vtep access port

Now HP switch part configuration is done, we can dump the OVSDB to see what it looks like, run ovsdb-client from any Linux machine:

[[email protected] ~(stackrc)]$ ovsdb-client dump --pretty tcp:10.7.31.1:6632
Arp_Sources_Local table
_uuid locator src_mac
----- ------- -------

Arp_Sources_Remote table
_uuid locator src_mac
----- ------- -------

Global table
_uuid                                managers switches
------------------------------------ -------- --------------------------------------
58708319-c85e-44a5-b221-4c2d3cde6004 []       [8f3d25db-f219-443e-9771-1d8e5219b5d9]

Logical_Binding_Stats table
_uuid bytes_from_local bytes_to_local packets_from_local packets_to_local
----- ---------------- -------------- ------------------ ----------------

Logical_Router table
_uuid description name static_routes switch_binding
----- ----------- ---- ------------- --------------

Logical_Switch table
_uuid description name tunnel_key
----- ----------- ---- ----------

Manager table
_uuid inactivity_probe is_connected max_backoff other_config status target
----- ---------------- ------------ ----------- ------------ ------ ------

Mcast_Macs_Local table
MAC _uuid ipaddr locator_set logical_switch
--- ----- ------ ----------- --------------

Mcast_Macs_Remote table
MAC _uuid ipaddr locator_set logical_switch
--- ----- ------ ----------- --------------

Physical_Locator table
_uuid dst_ip encapsulation_type
----- ------ ------------------

Physical_Locator_Set table
_uuid locators
----- --------

Physical_Port table
_uuid                                description name                  port_fault_status vlan_bindings vlan_stats
------------------------------------ ----------- --------------------- ----------------- ------------- ----------
28d3da91-7666-492e-9c20-fc960249a7b6 ""          "Bridge-Aggregation1" [UP]              {}            {}

Physical_Switch table
_uuid                                description management_ips name             ports                                  switch_fault_status tunnel_ips    tunnels
------------------------------------ ----------- -------------- ---------------- -------------------------------------- ------------------- ------------- -------
8f3d25db-f219-443e-9771-1d8e5219b5d9 ""          []             "VGP_LCC_EOR_02" [28d3da91-7666-492e-9c20-fc960249a7b6] []                  ["10.7.31.1"] []

Tunnel table
_uuid bfd_config_local bfd_config_remote bfd_params bfd_status local remote
----- ---------------- ----------------- ---------- ---------- ----- ------

Ucast_Macs_Local table
MAC _uuid ipaddr locator logical_switch
--- ----- ------ ------- --------------

Ucast_Macs_Remote table
MAC _uuid ipaddr locator logical_switch
--- ----- ------ ------- --------------

We can see this switch in Physical_Switch table and a port in Physical_Port table.

L2-GW installation and integration with Neutron

On controller node, fetch latest networking-l2gw codes from stable/kilo branch

git clone -b stable/kilo  https://github.com/openstack/networking-l2gw.git

Install it by pip:

pip install ~/networking-l2gw/

Add l2-gw service plugin to neutron server:

[[email protected] ~]# grep ^service_pl /etc/neutron/neutron.conf
service_plugins =router,networking_l2gw.services.l2gateway.plugin.L2GatewayPlugin

Copy l2-gw example config files to /etc/neutron/:

[[email protected] ~]#cp  /usr/etc/neutron/l2g* /etc/neutron/

Add HP 5930 OVSDB connection to l2gateway_agent.ini, ovsdb1 is user defined identifier of physical switch.

[[email protected] ~]# grep ^ovsdb /etc/neutron/l2gateway_agent.ini
ovsdb_hosts = 'ovsdb1:10.7.31.1:6632'

Make neutron-l2gateway-agent.service systemd unit file, so that we can control the service by systemctl:

[[email protected] system]# cat /usr/lib/systemd/system/neutron-l2gateway-agent.service
[Unit]
Description=OpenStack Neutron L2 Gateway Agent
After=syslog.target network.target

[Service]
Type=simple
User=neutron
ExecStart=/usr/bin/neutron-l2gateway-agent   --config-file /etc/neutron/neutron.conf --config-file /etc/neutron/l2gateway_agent.ini
PrivateTmp=false
KillMode=process

[Install]
WantedBy=multi-user.target

Load the unit file we created:

[[email protected]]# systemctl daemon-reload

Stop neutron services:

pcs resource disable neutron-server-clone

Update neutron database schema:

neutron-l2gw-db-manage --config-file /etc/neutron/neutron.conf upgrade head

Start neutron services:

pcs resource enable neutron-server-clone

Start neutron-l2-gateway service:

systemctl start neutron-l2gateway-agent.service

Now neutron l2-GW is enable, we should be able to see the L2 Gateway agent from neutron agent-list:

[[email protected] ~]# neutron agent-list
+--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+
| id                                   | agent_type         | host                               | alive | admin_state_up | binary                    |
+--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+
| 097320d1-9e7c-468a-bcee-948c560b65c9 | Metadata agent     | overcloud-controller-0.localdomain | :-)   | True           | neutron-metadata-agent    |
| 0ef52d27-9b91-40d4-b9dd-d24ff4f86ef7 | Open vSwitch agent | overcloud-controller-0.localdomain | :-)   | True           | neutron-openvswitch-agent |
| 304a3e9b-3a6f-4ae6-b2af-02891fa63d0f | L2 Gateway agent   | overcloud-controller-0.localdomain | :-)   | True           | neutron-l2gateway-agent   |
| 4f81e156-e679-4d76-a816-9b976cf9d62f | Open vSwitch agent | overcloud-compute-0.localdomain    | :-)   | True           | neutron-openvswitch-agent |
| 71f7e395-045f-47a2-9575-9976604eac7c | L3 agent           | overcloud-controller-0.localdomain | :-)   | True           | neutron-l3-agent          |
| 81f254a4-dbf2-48e2-afb1-9456201c2631 | DHCP agent         | overcloud-controller-0.localdomain | :-)   | True           | neutron-dhcp-agent        |
+--------------------------------------+--------------------+------------------------------------+-------+----------------+---------------------------+

Also we can see the physical switch is registered into neutron DB:

MariaDB [ovs_neutron]> select * from physical_switches;
+--------------------------------------+----------------+-----------+------------------+---------------------+
| uuid                                 | name           | tunnel_ip | ovsdb_identifier | switch_fault_status |
+--------------------------------------+----------------+-----------+------------------+---------------------+
| 8f3d25db-f219-443e-9771-1d8e5219b5d9 | VGP_LCC_EOR_02 | 10.7.31.1 | ovsdb1           | NULL                |
+--------------------------------------+----------------+-----------+------------------+---------------------+

L2-GW usage

Create a neutron network (VxLAN overlay):

[[email protected] ~(admin)]# neutron net-create test-l2gw
Created a new network:
+---------------------------+--------------------------------------+
| Field                     | Value                                |
+---------------------------+--------------------------------------+
| admin_state_up            | True                                 |
| id                        | 49485e07-77db-4d43-8b20-068597f744a5 |
| mtu                       | 0                                    |
| name                      | test-l2gw                            |
| provider:network_type     | vxlan                                |
| provider:physical_network |                                      |
| provider:segmentation_id  | 100                                  |
| router:external           | False                                |
| shared                    | False                                |
| status                    | ACTIVE                               |
| subnets                   |                                      |
| tenant_id                 | 7128bb366ffd47ffb445965daab45751     |
+---------------------------+--------------------------------------+

We can see neutron randomly allocates VNI ID 100 to this tenant network.
Let's create subnet for it:

neutron subnet-create --allocation-pool start=10.7.30.201,end=10.7.30.210 test-l2gw 10.7.30.0/24

Launch a VM in this network:

nova boot --image cirros --flavor m1.tiny --nic net-id=49485e07-77db-4d43-8b20-068597f744a5 test-vm

Check VM launching status and IP address allocated:

[[email protected] ~(admin)]# nova list
+--------------------------------------+---------+--------+------------+-------------+-----------------------+
| ID                                   | Name    | Status | Task State | Power State | Networks              |
+--------------------------------------+---------+--------+------------+-------------+-----------------------+
| 15e9aefb-cddd-4be6-a74e-a3098e17f26c | test-vm | ACTIVE | -          | Running     | test-l2gw=10.7.30.202 |
+--------------------------------------+---------+--------+------------+-------------+-----------------------+

We can see VM is running, with IP 10.7.30.202.

We can check VXLAN tunnels created on controller and compute nodes:

[[email protected] ~(admin)]# ovs-vsctl show | grep vxlan -A1
        Port "vxlan-0a14699a"
            Interface "vxlan-0a14699a"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="10.20.105.153", out_key=flow, remote_ip="10.20.105.154"}

[[email protected] ~]# ovs-vsctl show | grep vxlan -A1
        Port "vxlan-0a146999"
            Interface "vxlan-0a146999"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="10.20.105.154", out_key=flow, remote_ip="10.20.105.153"}

So we can see at the moment, both controller and compute node has only 1 tunnel created to each other. This makes VM to DHCP server traffic working.

Now let's create a l2-gateway using neutron-l2gw client:

neutron-l2gw l2-gateway-create --device name="VGP_LCC_EOR_02",interface_names="Bridge-Aggregation1"  gw1

Then we can now create a l2-gateway-connection, to bridge test-l2gw tenant network to gw1 for VLAN 2000:

neutron-l2gw l2-gateway-connection-create --default-segmentation-id 2000 gw1 test-l2gw

We can try to ping hosts in VLAN 2000, from controller dhcp namespace and VM.

[[email protected] ~(admin)]# ip netns exec qdhcp-49485e07-77db-4d43-8b20-068597f744a5 ping 10.7.30.32
PING 10.7.30.32 (10.7.30.32) 56(84) bytes of data.
64 bytes from 10.7.30.32: icmp_seq=61 ttl=64 time=0.493 ms
64 bytes from 10.7.30.32: icmp_seq=62 ttl=64 time=0.400 ms
64 bytes from 10.7.30.32: icmp_seq=63 ttl=64 time=0.443 ms
...

cirros$ ping 10.7.30.33
PING 10.7.30.33 (10.7.30.33): 56 data bytes
64 bytes from 10.7.30.33: seq=0 ttl=64 time=2.007 ms
64 bytes from 10.7.30.33: seq=1 ttl=64 time=1.034 ms
...

cirros$ ping 10.7.30.32
PING 10.7.30.32 (10.7.30.32): 56 data bytes
64 bytes from 10.7.30.32: seq=0 ttl=64 time=1.824 ms
64 bytes from 10.7.30.32: seq=1 ttl=64 time=1.007 ms
...

Pings work!!!

Let's try to check what's making that working under the hood.

Check VXLAN tunnels on controller and compute nodes, we will see tunnels created between them and HP 5930 switch.

[roo[email protected] ~(admin)]# ovs-vsctl show | grep vxlan -A1
        Port "vxlan-0a14699a"
            Interface "vxlan-0a14699a"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="10.20.105.153", out_key=flow, remote_ip="10.20.105.154"}
        Port "vxlan-0a071f01"
            Interface "vxlan-0a071f01"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="10.20.105.153", out_key=flow, remote_ip="10.7.31.1"}

[[email protected] ~]# ovs-vsctl show | grep vxlan -A1
        Port "vxlan-0a146999"
            Interface "vxlan-0a146999"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="10.20.105.154", out_key=flow, remote_ip="10.20.105.153"}
        Port "vxlan-0a0713f4"
            Interface "vxlan-0a0713f4"
                type: vxlan
                options: {df_default="true", in_key=flow, local_ip="10.20.105.154", out_key=flow, remote_ip="10.7.31.1"}

Dump the OVSDB of physical switch:

[[email protected] ~(stackrc)]$ ovsdb-client dump --pretty tcp:10.7.31.1:6632
Arp_Sources_Local table
_uuid locator src_mac
----- ------- -------

Arp_Sources_Remote table
_uuid locator src_mac
----- ------- -------

Global table
_uuid                                managers switches
------------------------------------ -------- --------------------------------------
58708319-c85e-44a5-b221-4c2d3cde6004 []       [8f3d25db-f219-443e-9771-1d8e5219b5d9]

Logical_Binding_Stats table
_uuid bytes_from_local bytes_to_local packets_from_local packets_to_local
----- ---------------- -------------- ------------------ ----------------

Logical_Router table
_uuid description name static_routes switch_binding
----- ----------- ---- ------------- --------------

Logical_Switch table
_uuid                                description name                                   tunnel_key
------------------------------------ ----------- -------------------------------------- ----------
c08ad894-886f-452a-a528-216f36d48958 "test-l2gw" "49485e07-77db-4d43-8b20-068597f744a5" 100

Manager table
_uuid inactivity_probe is_connected max_backoff other_config status target
----- ---------------- ------------ ----------- ------------ ------ ------

Mcast_Macs_Local table
MAC         _uuid                                ipaddr locator_set                          logical_switch
----------- ------------------------------------ ------ ------------------------------------ ------------------------------------
unknown-dst aeaab3ec-a0ae-4937-8cda-14ca48039be8 ""     e542b5fe-70c4-4872-a741-f334d1bf7021 c08ad894-886f-452a-a528-216f36d48958

Mcast_Macs_Remote table
MAC _uuid ipaddr locator_set logical_switch
--- ----- ------ ----------- --------------

Physical_Locator table
_uuid                                dst_ip          encapsulation_type
------------------------------------ --------------- ------------------
de8e7621-023a-4dfe-bc79-1bbd97962c71 "10.20.105.153" "vxlan_over_ipv4"
6047ff6f-7662-44d0-8e49-b1c6afb51e68 "10.20.105.154" "vxlan_over_ipv4"
7e83e09d-2ee3-4c8d-8784-efae6b780c15 "10.7.31.1"     "vxlan_over_ipv4"

Physical_Locator_Set table
_uuid                                locators
------------------------------------ --------------------------------------
e542b5fe-70c4-4872-a741-f334d1bf7021 [7e83e09d-2ee3-4c8d-8784-efae6b780c15]

Physical_Port table
_uuid                                description name                  port_fault_status vlan_bindings                               vlan_stats
------------------------------------ ----------- --------------------- ----------------- ------------------------------------------- ----------
28d3da91-7666-492e-9c20-fc960249a7b6 ""          "Bridge-Aggregation1" [UP]              {2000=c08ad894-886f-452a-a528-216f36d48958} {}

Physical_Switch table
_uuid                                description management_ips name             ports                                                                        switch_fault_status tunnel_ips    tunnels
------------------------------------ ----------- -------------- ---------------- ---------------------------------------------------------------------------- ------------------- ------------- ----------------------------------------------------------------------------
8f3d25db-f219-443e-9771-1d8e5219b5d9 ""          []             "VGP_LCC_EOR_02" [28d3da91-7666-492e-9c20-fc960249a7b6] []                  ["10.7.31.1"] [340ae640-ad8c-4dc7-9c61-1f84c9c462aa, 68629911-a3d0-4793-8a06-583b544dd827]

Tunnel table
_uuid                                bfd_config_local                                               bfd_config_remote                                       bfd_params                                                                                                            bfd_status                                                                           local                                remote
------------------------------------ -------------------------------------------------------------- ------------------------------------------------------- --------------------------------------------------------------------------------------------------------------------- ------------------------------------------------------------------------------------ ------------------------------------ ------------------------------------
68629911-a3d0-4793-8a06-583b544dd827 {bfd_dst_ip="10.7.31.1", bfd_dst_mac="188:234:250:108:94:141"} {bfd_dst_ip="10.20.105.153", bfd_dst_mac="0:0:0:0:0:0"} {check_tnl_key="0", cpath_down="0", decay_min_rx="100", enable="1", forwarding_if_rx="1", min_rx="100", min_tx="100"} {diagnostic="", forwarding="0", remote_diagnostic="", remote_state=down, state=down} 7e83e09d-2ee3-4c8d-8784-efae6b780c15 de8e7621-023a-4dfe-bc79-1bbd97962c71
340ae640-ad8c-4dc7-9c61-1f84c9c462aa {bfd_dst_ip="10.7.31.1", bfd_dst_mac="188:234:250:108:94:141"} {bfd_dst_ip="10.20.105.154", bfd_dst_mac="0:0:0:0:0:0"} {check_tnl_key="0", cpath_down="0", decay_min_rx="100", enable="1", forwarding_if_rx="1", min_rx="100", min_tx="100"} {diagnostic="", forwarding="0", remote_diagnostic="", remote_state=down, state=down} 7e83e09d-2ee3-4c8d-8784-efae6b780c15 6047ff6f-7662-44d0-8e49-b1c6afb51e68

Ucast_Macs_Local table
MAC                 _uuid                                ipaddr locator                              logical_switch
------------------- ------------------------------------ ------ ------------------------------------ ------------------------------------
"2c:60:0c:c0:97:c9" 1b74b46b-5afb-41ad-ab84-faf1331d0c72 ""     7e83e09d-2ee3-4c8d-8784-efae6b780c15 c08ad894-886f-452a-a528-216f36d48958
"2c:60:0c:c0:a6:c1" 13996ca9-3de0-4bea-b815-036a07aa54d4 ""     7e83e09d-2ee3-4c8d-8784-efae6b780c15 c08ad894-886f-452a-a528-216f36d48958
"2c:60:0c:c0:a9:3b" 9d1c3c28-8ced-4e74-a1a9-bb3a27a41267 ""     7e83e09d-2ee3-4c8d-8784-efae6b780c15 c08ad894-886f-452a-a528-216f36d48958

Ucast_Macs_Remote table
MAC                 _uuid                                ipaddr        locator                              logical_switch
------------------- ------------------------------------ ------------- ------------------------------------ ------------------------------------
"fa:16:3e:96:3b:ce" 18dd6787-d5dc-42dc-97eb-285faf5b23ff "10.7.30.202" 6047ff6f-7662-44d0-8e49-b1c6afb51e68 c08ad894-886f-452a-a528-216f36d48958
"fa:16:3e:d1:44:85" 109c752e-1b97-4169-b939-6ada0a8ec703 "10.7.30.201" de8e7621-023a-4dfe-bc79-1bbd97962c71 c08ad894-886f-452a-a528-216f36d48958

From Tunnel table, we can see 2 VXLAN tunnels created between physical switch and controller and compute nodes.
From Logical_Switch table, a logical switch is created with VNI ID(tunnel_key) 100, then from Physical_Port table, we can see VLAN 2000 is bound to that logical switch.
From Ucast_Macs_Remote table, we can see MAC of 10.7.30.201 and 10.7.30.202, which are VM and dhcp server's MACs.

Check tunnel interfaces from HP 5930 console:

[VGP_LCC_EOR_02]dis interface Tunnel
Tunnel0
Current state: UP
Line protocol state: UP
Description: created by ovsdb
Bandwidth: 64 kbps
Maximum transmission unit: 1464
Internet protocol processing: Disabled
Last clearing of counters: Never
Tunnel source unknown, destination 10.20.105.153
Tunnel protocol/transport UDP_VXLAN/IP

Tunnel1
Current state: UP
Line protocol state: UP
Description: created by ovsdb
Bandwidth: 64 kbps
Maximum transmission unit: 1464
Internet protocol processing: Disabled
Last clearing of counters: Never
Tunnel source unknown, destination 10.20.105.154
Tunnel protocol/transport UDP_VXLAN/IP

2 Tunnels are shown there as well.

VXLAN routing problem

On HP 5930 switch, for VLAN 2000, we have a L3 IP address working as gateway of this network:

[VGP_LCC_EOR_02]display current-configuration interface Vlan-interface 2000
interface Vlan-interface2000
 ip address 10.7.30.1 255.255.255.0

After l2-gateway-connection is created, this default GW IP can not be reached from VM, it's because of limitation of TRIDENT 2 chipset in HP 5930 switch, details can be checked from this blog: LAYER-3 SWITCHING OVER VXLAN REVISITED

Workaround of VXLAN routing problem by adding a loopback cable in the switch

To workaround the HW limitation, we could make a loopback cable on HP 5930 switch, bridge out VXLAN to VLAN first, then through the loopback cable, reaching in the L3 GW as normal VLAN routing.

The diagram is:

Neutron-L2-GW-to-HP-5930-Switch-VXLAN-Routing-w-Loopback-Cable

Loopback cable is connected between FGE1/0/27 and FGE1/0/28, here we configure FGE1/0/27 as vtep access port, and FGE1/0/28 as normal VLAN trunk port. Dotted red line shows the packet transit path between VM and L3 GW.

On HP 5930 console:

[VGP_LCC_EOR_02]dis current-configuration interface FortyGigE 1/0/27
#
interface FortyGigE1/0/27
 port link-mode bridge
 port link-type trunk
 undo port trunk permit vlan 1
 vtep access port
#
return

[VGP_LCC_EOR_02]dis current-configuration interface FortyGigE 1/0/28
#
interface FortyGigE1/0/28
 port link-mode bridge
 port link-type trunk
 undo port trunk permit vlan 1
 port trunk permit vlan 2000
#
return

We need to remove previously created l2-gateway-connection and l2-gateway:

neutron-l2gw l2-gateway-connection-delete 4fb1c9eb-de29-442c-a946-190f5f99cf17
neutron-l2gw l2-gateway-delete gw1

Re-create l2-gateway gw1 with interface FortyGigE1/0/27:

neutron-l2gw l2-gateway-create --device name="VGP_LCC_EOR_02",interface_names="FortyGigE1/0/27"  gw1

Create l2-gateway-connection, to bridge test-l2gw tenant network to gw1 for VLAN 2000

neutron-l2gw l2-gateway-connection-create --default-segmentation-id 2000 gw1 test-l2gw

Now try to ping L3 GW 10.7.30.1 from VM.

$ ping 10.7.30.1
PING 10.7.30.1 (10.7.30.1): 56 data bytes
64 bytes from 10.7.30.1: seq=1 ttl=64 time=1.463 ms
64 bytes from 10.7.30.1: seq=2 ttl=64 time=2.606 ms
64 bytes from 10.7.30.1: seq=3 ttl=64 time=1.036 ms
...

It works now!!!