How metadata service work by default (in router namespace)

In Openstack, by default, you need L3 agent to make metadata service working, which means you need to attach the tenant network to a neutron router. In router namespace, a metadata proxy process is running to handle metadata request, iptables rule redirect metadata request to metadata proxy.

neutron-ns-metadata-proxy process in router namespace, listening on 9697 port:

[[email protected] ~]# ip netns exec qrouter-28120946-9f6a-4638-9747-603977b49816 netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:9697            0.0.0.0:*               LISTEN      28521/python2

[[email protected] ~]# ps -f --pid 28521 | fold -s -w 100
UID        PID  PPID  C STIME TTY          TIME CMD
neutron  28521     1  0 03:56 ?        00:00:00 /usr/bin/python2 /bin/neutron-ns-metadata-proxy
--pid_file=/var/lib/neutron/external/pids/28120946-9f6a-4638-9747-603977b49816.pid
--metadata_proxy_socket=/var/lib/neutron/metadata_proxy
--router_id=28120946-9f6a-4638-9747-603977b49816 --state_path=/var/lib/neutron --metadata_port=9697
--metadata_proxy_user=998 --metadata_proxy_group=996 --debug
--log-file=neutron-ns-metadata-proxy-28120946-9f6a-4638-9747-603977b49816.log
--log-dir=/var/log/neutron

iptables rule to redirect metadata request sent to http://169.254.169.254:80 to 9697 port

[[email protected] ~]# ip netns exec qrouter-28120946-9f6a-4638-9747-603977b49816 iptables-save | grep REDIRECT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697

Metadata service in DHCP namespace

Openstack has now option to enable metadata proxy service in DHCP namespace, thus metadata service can work in isolated tenant network, no need to attach tenant network to any neutron router.

To enable this, we need to configure /etc/neutron/dhcp_agent.ini:

# The DHCP server can assist with providing metadata support on isolated
# networks. Setting this value to True will cause the DHCP server to append
# specific host routes to the DHCP request. The metadata service will only
# be activated when the subnet does not contain any router port. The guest
# instance must be configured to request host routes via DHCP (Option 121).
# This option doesn't have any effect when force_metadata is set to True.
enable_isolated_metadata = True

Restart neutron-dhcp-agent on every controller node:

systemctl restart neutron-dhcp-agent

Now let's try to create a network, the comment section above says metadata service will only be activated when the subnet does not contain any router port, so we need to create a network without gateway.

neutron net-create test-metadata-in-dhcp
neutron subnet-create --no-gateway --name test-metadata-in-dhcp test-metadata-in-dhcp 192.168.111.0/24

We can see in newly created dhcp namespace, there's metadata server IP 169.254.169.254 configured:

[[email protected] ~]# ip netns exec qdhcp-18b28e2a-30a2-4374-83e1-54bdfeda66a3 ip -4 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
19: tap25fabc7f-c2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    inet 192.168.111.1/24 brd 192.168.111.255 scope global tap25fabc7f-c2
       valid_lft forever preferred_lft forever
    inet 169.254.169.254/16 brd 169.254.255.255 scope global tap25fabc7f-c2
       valid_lft forever preferred_lft forever

And there's a metadata-proxy running for that namespace:

[[email protected] ~]# ps -ef | grep meta | grep 18b28e2a-30a2-4374-83e1-54bdfeda66a3 | fold -s -w 100
neutron   6422     1  0 06:35 ?        00:00:00 /usr/bin/python2 /bin/neutron-ns-metadata-proxy
--pid_file=/var/lib/neutron/external/pids/18b28e2a-30a2-4374-83e1-54bdfeda66a3.pid
--metadata_proxy_socket=/var/lib/neutron/metadata_proxy
--network_id=18b28e2a-30a2-4374-83e1-54bdfeda66a3 --state_path=/var/lib/neutron --metadata_port=80
--metadata_proxy_user=998 --metadata_proxy_group=996
--log-file=neutron-ns-metadata-proxy-18b28e2a-30a2-4374-83e1-54bdfeda66a3.log
--log-dir=/var/log/neutron

The metadata-proxy is listening on port 80 in that namespace:

[[email protected] ~]# ip netns exec qdhcp-18b28e2a-30a2-4374-83e1-54bdfeda66a3 netstat -ntlp | grep 6422
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      6422/python2

From dhcp server static routes option, we can see a static route 169.254.169.254 to dhcp server IP in place, VM instances receive this static route via dhcp client, thus metadata request can be routed to dhcp namespace, metadata-proxy running there then can handle the request.

[[email protected] ~]# ps -ef | grep dnsmasq | grep 18b28e2a-30a2-4374-83e1-54bdfeda66a3 |fold -s -w 100
nobody    6398     1  0 06:35 ?        00:00:00 dnsmasq --no-hosts --no-resolv --strict-order
--bind-interfaces --interface=tap25fabc7f-c2 --except-interface=lo
--pid-file=/var/lib/neutron/dhcp/18b28e2a-30a2-4374-83e1-54bdfeda66a3/pid
--dhcp-hostsfile=/var/lib/neutron/dhcp/18b28e2a-30a2-4374-83e1-54bdfeda66a3/host
--addn-hosts=/var/lib/neutron/dhcp/18b28e2a-30a2-4374-83e1-54bdfeda66a3/addn_hosts
--dhcp-optsfile=/var/lib/neutron/dhcp/18b28e2a-30a2-4374-83e1-54bdfeda66a3/opts
--dhcp-leasefile=/var/lib/neutron/dhcp/18b28e2a-30a2-4374-83e1-54bdfeda66a3/leases
--dhcp-range=set:tag0,192.168.111.0,static,86400s --dhcp-lease-max=256
--conf-file=/etc/neutron/dnsmasq-neutron.conf --domain=openstacklocal

[[email protected] ~]# cat /var/lib/neutron/dhcp/18b28e2a-30a2-4374-83e1-54bdfeda66a3/opts
tag:tag0,option:classless-static-route,169.254.169.254/32,192.168.111.1
tag:tag0,249,169.254.169.254/32,192.168.111.1
tag:tag0,option:router
tag:tag0,option:dns-server,192.168.111.1,192.168.111.3,192.168.111.2

As mentioned above, this works without gateway defined in subnet, what if we have gateway defined, let's try it out.

Create a network with gateway defined(by default):

neutron net-create test-metadata-in-dhcp-with-gw
neutron subnet-create --name test-metadata-in-dhcp-with-gw test-metadata-in-dhcp-with-gw 172.31.0.0/24

Check IP interfaces in dhcp namespace:

[[email protected] ~]# ip netns exec qdhcp-f673b2b9-360f-4815-b308-362e4725fd85 ip -4 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
20: tap3da34c9a-1d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    inet 172.31.0.4/24 brd 172.31.0.255 scope global tap3da34c9a-1d
       valid_lft forever preferred_lft forever
    inet 169.254.169.254/16 brd 169.254.255.255 scope global tap3da34c9a-1d
       valid_lft forever preferred_lft forever

Aha, we still see 169.254.169.254 IP is configured, what about metadata-proxy service?

[[email protected] ~]# ps -ef | grep meta | grep f673b2b9-360f-4815-b308-362e4725fd85   | fold -s -w 100
neutron  14475     1  0 07:15 ?        00:00:00 /usr/bin/python2 /bin/neutron-ns-metadata-proxy
--pid_file=/var/lib/neutron/external/pids/f673b2b9-360f-4815-b308-362e4725fd85.pid
--metadata_proxy_socket=/var/lib/neutron/metadata_proxy
--network_id=f673b2b9-360f-4815-b308-362e4725fd85 --state_path=/var/lib/neutron --metadata_port=80
--metadata_proxy_user=998 --metadata_proxy_group=996
--log-file=neutron-ns-metadata-proxy-f673b2b9-360f-4815-b308-362e4725fd85.log
--log-dir=/var/log/neutron

[[email protected] ~]# ip netns exec qdhcp-f673b2b9-360f-4815-b308-362e4725fd85  netstat -ntlp|grep 14475
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      14475/python2

Metadata-proxy is also running, then how about static options of dhcp server ?

[[email protected] ~]# ps -ef | grep dnsmasq | grep f673b2b9-360f-4815-b308-362e4725fd85  |fold -s -w 100
nobody   14289     1  0 07:15 ?        00:00:00 dnsmasq --no-hosts --no-resolv --strict-order
--bind-interfaces --interface=tap3da34c9a-1d --except-interface=lo
--pid-file=/var/lib/neutron/dhcp/f673b2b9-360f-4815-b308-362e4725fd85/pid
--dhcp-hostsfile=/var/lib/neutron/dhcp/f673b2b9-360f-4815-b308-362e4725fd85/host
--addn-hosts=/var/lib/neutron/dhcp/f673b2b9-360f-4815-b308-362e4725fd85/addn_hosts
--dhcp-optsfile=/var/lib/neutron/dhcp/f673b2b9-360f-4815-b308-362e4725fd85/opts
--dhcp-leasefile=/var/lib/neutron/dhcp/f673b2b9-360f-4815-b308-362e4725fd85/leases
--dhcp-range=set:tag0,172.31.0.0,static,86400s --dhcp-lease-max=256
--conf-file=/etc/neutron/dnsmasq-neutron.conf --domain=openstacklocal

[[email protected] ~]# cat /var/lib/neutron/dhcp/f673b2b9-360f-4815-b308-362e4725fd85/opts
tag:tag0,option:classless-static-route,169.254.169.254/32,172.31.0.4,0.0.0.0/0,172.31.0.1
tag:tag0,249,169.254.169.254/32,172.31.0.4,0.0.0.0/0,172.31.0.1
tag:tag0,option:router,172.31.0.1
tag:tag0,option:dns-server,172.31.0.3,172.31.0.2,172.31.0.4

Interesting, everything just looks like the case without gateway defined, let's see if we attach the network to a router will change things or not:

neutron router-create testrouter
neutron router-interface-add testrouter test-metadata-in-dhcp-with-gw

Let's check static route options of dhcp server:

[[email protected] ~]# cat /var/lib/neutron/dhcp/f673b2b9-360f-4815-b308-362e4725fd85/opts
tag:tag0,option:classless-static-route,169.254.169.254/32,172.31.0.4,0.0.0.0/0,172.31.0.1
tag:tag0,249,169.254.169.254/32,172.31.0.4,0.0.0.0/0,172.31.0.1
tag:tag0,option:router,172.31.0.1
tag:tag0,option:dns-server,172.31.0.3,172.31.0.2,172.31.0.4

The 169.254.169.254 static route still presents, let's check router namespace as well:

[[email protected] ~]# ip netns  exec qrouter-4cc7cb93-8b8b-4859-93b1-628031b04f4c iptables-save | grep REDIRECT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697

In router namespace, metadata-proxy is also running, which means now both router and dhcp namespace can handle metadata request.

If now I restart neutron-dhcp-agent on controller where active router is running:

systemctl restart neutron-dhcp-agent

Then check static route options of dhcp server again:

[[email protected] ~]# cat /var/lib/neutron/dhcp/f673b2b9-360f-4815-b308-362e4725fd85/opts
tag:tag0,option:router,172.31.0.1
tag:tag0,option:dns-server,172.31.0.2,172.31.0.4,172.31.0.3

Funny thing is the 169.254.169.254 static route is gone. But 169.254.169.254 IP is still in dhcp namespace, and metadata-proxy is still runnning.

[[email protected] ~]# ip netns exec qdhcp-f673b2b9-360f-4815-b308-362e4725fd85 ip -4  a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
28: tap5fa39073-72: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
    inet 172.31.0.3/24 brd 172.31.0.255 scope global tap5fa39073-72
       valid_lft forever preferred_lft forever
    inet 169.254.169.254/16 brd 169.254.255.255 scope global tap5fa39073-72
       valid_lft forever preferred_lft forever

[[email protected] ~]# ps -ef | grep metadata-proxy | grep f673b2b9-360f-4815-b308-362e4725fd85 |fold -s -w 100
neutron   8459     1  0 07:15 ?        00:00:00 /usr/bin/python2 /bin/neutron-ns-metadata-proxy
--pid_file=/var/lib/neutron/external/pids/f673b2b9-360f-4815-b308-362e4725fd85.pid
--metadata_proxy_socket=/var/lib/neutron/metadata_proxy
--network_id=f673b2b9-360f-4815-b308-362e4725fd85 --state_path=/var/lib/neutron --metadata_port=80
--metadata_proxy_user=998 --metadata_proxy_group=996
--log-file=neutron-ns-metadata-proxy-f673b2b9-360f-4815-b308-362e4725fd85.log
--log-dir=/var/log/neutron

This behaviour looks weird, however both router and dhcp namespace have metadata-proxy running to handle metadata request, but 169.254.169.254 is routed to dhcp server before you restarting neutron-dhcp-agent, after restart, 169.254.169.254 will be routed to router gateway IP via default route.