Metadata service in DHCP namespace
How metadata service work by default (in router namespace)
In Openstack, by default, you need L3 agent to make metadata service working, which means you need to attach the tenant network to a neutron router. In router namespace, a metadata proxy process is running to handle metadata request, iptables rule redirect metadata request to metadata proxy.
neutron-ns-metadata-proxy process in router namespace, listening on 9697 port:
[root@overcloud-controller-1 ~]# ip netns exec qrouter-28120946-9f6a-4638-9747-603977b49816 netstat -ntlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address Foreign Address State PID/Program name
tcp 0 0 0.0.0.0:9697 0.0.0.0:* LISTEN 28521/python2
[root@overcloud-controller-1 ~]# ps -f --pid 28521 | fold -s -w 100
UID PID PPID C STIME TTY TIME CMD
neutron 28521 1 0 03:56 ? 00:00:00 /usr/bin/python2 /bin/neutron-ns-metadata-proxy
--pid_file=/var/lib/neutron/external/pids/28120946-9f6a-4638-9747-603977b49816.pid
--metadata_proxy_socket=/var/lib/neutron/metadata_proxy
--router_id=28120946-9f6a-4638-9747-603977b49816 --state_path=/var/lib/neutron --metadata_port=9697
--metadata_proxy_user=998 --metadata_proxy_group=996 --debug
--log-file=neutron-ns-metadata-proxy-28120946-9f6a-4638-9747-603977b49816.log
--log-dir=/var/log/neutron
iptables rule to redirect metadata request sent to http://169.254.169.254:80 to 9697 port
[root@overcloud-controller-1 ~]# ip netns exec qrouter-28120946-9f6a-4638-9747-603977b49816 iptables-save | grep REDIRECT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
Metadata service in DHCP namespace
Openstack has now option to enable metadata proxy service in DHCP namespace, thus metadata service can work in isolated tenant network, no need to attach tenant network to any neutron router.
To enable this, we need to configure /etc/neutron/dhcp_agent.ini:
# The DHCP server can assist with providing metadata support on isolated
# networks. Setting this value to True will cause the DHCP server to append
# specific host routes to the DHCP request. The metadata service will only
# be activated when the subnet does not contain any router port. The guest
# instance must be configured to request host routes via DHCP (Option 121).
# This option doesn't have any effect when force_metadata is set to True.
enable_isolated_metadata = True
Restart neutron-dhcp-agent on every controller node:
systemctl restart neutron-dhcp-agent
Now let's try to create a network, the comment section above says metadata service will only be activated when the subnet does not contain any router port, so we need to create a network without gateway.
neutron net-create test-metadata-in-dhcp
neutron subnet-create --no-gateway --name test-metadata-in-dhcp test-metadata-in-dhcp 192.168.111.0/24
We can see in newly created dhcp namespace, there's metadata server IP 169.254.169.254 configured:
[root@overcloud-controller-0 ~]# ip netns exec qdhcp-18b28e2a-30a2-4374-83e1-54bdfeda66a3 ip -4 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
19: tap25fabc7f-c2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
inet 192.168.111.1/24 brd 192.168.111.255 scope global tap25fabc7f-c2
valid_lft forever preferred_lft forever
inet 169.254.169.254/16 brd 169.254.255.255 scope global tap25fabc7f-c2
valid_lft forever preferred_lft forever
And there's a metadata-proxy running for that namespace:
[root@overcloud-controller-0 ~]# ps -ef | grep meta | grep 18b28e2a-30a2-4374-83e1-54bdfeda66a3 | fold -s -w 100
neutron 6422 1 0 06:35 ? 00:00:00 /usr/bin/python2 /bin/neutron-ns-metadata-proxy
--pid_file=/var/lib/neutron/external/pids/18b28e2a-30a2-4374-83e1-54bdfeda66a3.pid
--metadata_proxy_socket=/var/lib/neutron/metadata_proxy
--network_id=18b28e2a-30a2-4374-83e1-54bdfeda66a3 --state_path=/var/lib/neutron --metadata_port=80
--metadata_proxy_user=998 --metadata_proxy_group=996
--log-file=neutron-ns-metadata-proxy-18b28e2a-30a2-4374-83e1-54bdfeda66a3.log
--log-dir=/var/log/neutron
The metadata-proxy is listening on port 80 in that namespace:
[root@overcloud-controller-0 ~]# ip netns exec qdhcp-18b28e2a-30a2-4374-83e1-54bdfeda66a3 netstat -ntlp | grep 6422
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 6422/python2
From dhcp server static routes option, we can see a static route 169.254.169.254 to dhcp server IP in place, VM instances receive this static route via dhcp client, thus metadata request can be routed to dhcp namespace, metadata-proxy running there then can handle the request.
[root@overcloud-controller-0 ~]# ps -ef | grep dnsmasq | grep 18b28e2a-30a2-4374-83e1-54bdfeda66a3 |fold -s -w 100
nobody 6398 1 0 06:35 ? 00:00:00 dnsmasq --no-hosts --no-resolv --strict-order
--bind-interfaces --interface=tap25fabc7f-c2 --except-interface=lo
--pid-file=/var/lib/neutron/dhcp/18b28e2a-30a2-4374-83e1-54bdfeda66a3/pid
--dhcp-hostsfile=/var/lib/neutron/dhcp/18b28e2a-30a2-4374-83e1-54bdfeda66a3/host
--addn-hosts=/var/lib/neutron/dhcp/18b28e2a-30a2-4374-83e1-54bdfeda66a3/addn_hosts
--dhcp-optsfile=/var/lib/neutron/dhcp/18b28e2a-30a2-4374-83e1-54bdfeda66a3/opts
--dhcp-leasefile=/var/lib/neutron/dhcp/18b28e2a-30a2-4374-83e1-54bdfeda66a3/leases
--dhcp-range=set:tag0,192.168.111.0,static,86400s --dhcp-lease-max=256
--conf-file=/etc/neutron/dnsmasq-neutron.conf --domain=openstacklocal
[root@overcloud-controller-0 ~]# cat /var/lib/neutron/dhcp/18b28e2a-30a2-4374-83e1-54bdfeda66a3/opts
tag:tag0,option:classless-static-route,169.254.169.254/32,192.168.111.1
tag:tag0,249,169.254.169.254/32,192.168.111.1
tag:tag0,option:router
tag:tag0,option:dns-server,192.168.111.1,192.168.111.3,192.168.111.2
As mentioned above, this works without gateway defined in subnet, what if we have gateway defined, let's try it out.
Create a network with gateway defined(by default):
neutron net-create test-metadata-in-dhcp-with-gw
neutron subnet-create --name test-metadata-in-dhcp-with-gw test-metadata-in-dhcp-with-gw 172.31.0.0/24
Check IP interfaces in dhcp namespace:
[root@overcloud-controller-0 ~]# ip netns exec qdhcp-f673b2b9-360f-4815-b308-362e4725fd85 ip -4 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
20: tap3da34c9a-1d: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
inet 172.31.0.4/24 brd 172.31.0.255 scope global tap3da34c9a-1d
valid_lft forever preferred_lft forever
inet 169.254.169.254/16 brd 169.254.255.255 scope global tap3da34c9a-1d
valid_lft forever preferred_lft forever
Aha, we still see 169.254.169.254 IP is configured, what about metadata-proxy service?
[root@overcloud-controller-0 ~]# ps -ef | grep meta | grep f673b2b9-360f-4815-b308-362e4725fd85 | fold -s -w 100
neutron 14475 1 0 07:15 ? 00:00:00 /usr/bin/python2 /bin/neutron-ns-metadata-proxy
--pid_file=/var/lib/neutron/external/pids/f673b2b9-360f-4815-b308-362e4725fd85.pid
--metadata_proxy_socket=/var/lib/neutron/metadata_proxy
--network_id=f673b2b9-360f-4815-b308-362e4725fd85 --state_path=/var/lib/neutron --metadata_port=80
--metadata_proxy_user=998 --metadata_proxy_group=996
--log-file=neutron-ns-metadata-proxy-f673b2b9-360f-4815-b308-362e4725fd85.log
--log-dir=/var/log/neutron
[root@overcloud-controller-0 ~]# ip netns exec qdhcp-f673b2b9-360f-4815-b308-362e4725fd85 netstat -ntlp|grep 14475
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 14475/python2
Metadata-proxy is also running, then how about static options of dhcp server ?
[root@overcloud-controller-0 ~]# ps -ef | grep dnsmasq | grep f673b2b9-360f-4815-b308-362e4725fd85 |fold -s -w 100
nobody 14289 1 0 07:15 ? 00:00:00 dnsmasq --no-hosts --no-resolv --strict-order
--bind-interfaces --interface=tap3da34c9a-1d --except-interface=lo
--pid-file=/var/lib/neutron/dhcp/f673b2b9-360f-4815-b308-362e4725fd85/pid
--dhcp-hostsfile=/var/lib/neutron/dhcp/f673b2b9-360f-4815-b308-362e4725fd85/host
--addn-hosts=/var/lib/neutron/dhcp/f673b2b9-360f-4815-b308-362e4725fd85/addn_hosts
--dhcp-optsfile=/var/lib/neutron/dhcp/f673b2b9-360f-4815-b308-362e4725fd85/opts
--dhcp-leasefile=/var/lib/neutron/dhcp/f673b2b9-360f-4815-b308-362e4725fd85/leases
--dhcp-range=set:tag0,172.31.0.0,static,86400s --dhcp-lease-max=256
--conf-file=/etc/neutron/dnsmasq-neutron.conf --domain=openstacklocal
[root@overcloud-controller-0 ~]# cat /var/lib/neutron/dhcp/f673b2b9-360f-4815-b308-362e4725fd85/opts
tag:tag0,option:classless-static-route,169.254.169.254/32,172.31.0.4,0.0.0.0/0,172.31.0.1
tag:tag0,249,169.254.169.254/32,172.31.0.4,0.0.0.0/0,172.31.0.1
tag:tag0,option:router,172.31.0.1
tag:tag0,option:dns-server,172.31.0.3,172.31.0.2,172.31.0.4
Interesting, everything just looks like the case without gateway defined, let's see if we attach the network to a router will change things or not:
neutron router-create testrouter
neutron router-interface-add testrouter test-metadata-in-dhcp-with-gw
Let's check static route options of dhcp server:
[root@overcloud-controller-0 ~]# cat /var/lib/neutron/dhcp/f673b2b9-360f-4815-b308-362e4725fd85/opts
tag:tag0,option:classless-static-route,169.254.169.254/32,172.31.0.4,0.0.0.0/0,172.31.0.1
tag:tag0,249,169.254.169.254/32,172.31.0.4,0.0.0.0/0,172.31.0.1
tag:tag0,option:router,172.31.0.1
tag:tag0,option:dns-server,172.31.0.3,172.31.0.2,172.31.0.4
The 169.254.169.254 static route still presents, let's check router namespace as well:
[root@overcloud-controller-1 ~]# ip netns exec qrouter-4cc7cb93-8b8b-4859-93b1-628031b04f4c iptables-save | grep REDIRECT
-A neutron-l3-agent-PREROUTING -d 169.254.169.254/32 -i qr-+ -p tcp -m tcp --dport 80 -j REDIRECT --to-ports 9697
In router namespace, metadata-proxy is also running, which means now both router and dhcp namespace can handle metadata request.
If now I restart neutron-dhcp-agent on controller where active router is running:
systemctl restart neutron-dhcp-agent
Then check static route options of dhcp server again:
[root@overcloud-controller-1 ~]# cat /var/lib/neutron/dhcp/f673b2b9-360f-4815-b308-362e4725fd85/opts
tag:tag0,option:router,172.31.0.1
tag:tag0,option:dns-server,172.31.0.2,172.31.0.4,172.31.0.3
Funny thing is the 169.254.169.254 static route is gone. But 169.254.169.254 IP is still in dhcp namespace, and metadata-proxy is still runnning.
[root@overcloud-controller-1 ~]# ip netns exec qdhcp-f673b2b9-360f-4815-b308-362e4725fd85 ip -4 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
28: tap5fa39073-72: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN
inet 172.31.0.3/24 brd 172.31.0.255 scope global tap5fa39073-72
valid_lft forever preferred_lft forever
inet 169.254.169.254/16 brd 169.254.255.255 scope global tap5fa39073-72
valid_lft forever preferred_lft forever
[root@overcloud-controller-1 ~]# ps -ef | grep metadata-proxy | grep f673b2b9-360f-4815-b308-362e4725fd85 |fold -s -w 100
neutron 8459 1 0 07:15 ? 00:00:00 /usr/bin/python2 /bin/neutron-ns-metadata-proxy
--pid_file=/var/lib/neutron/external/pids/f673b2b9-360f-4815-b308-362e4725fd85.pid
--metadata_proxy_socket=/var/lib/neutron/metadata_proxy
--network_id=f673b2b9-360f-4815-b308-362e4725fd85 --state_path=/var/lib/neutron --metadata_port=80
--metadata_proxy_user=998 --metadata_proxy_group=996
--log-file=neutron-ns-metadata-proxy-f673b2b9-360f-4815-b308-362e4725fd85.log
--log-dir=/var/log/neutron
This behaviour looks weird, however both router and dhcp namespace have metadata-proxy running to handle metadata request, but 169.254.169.254 is routed to dhcp server before you restarting neutron-dhcp-agent, after restart, 169.254.169.254 will be routed to router gateway IP via default route.