NFS backend for Openstack Glance/Cinder/Instance-store
In this post, let’s go through how to configure NFS as unified storage backend for Openstack Glance, Cinder and shared instance-store, also we look at how it works under the hood.
Setup: 1 controller and 2 compute nodes. Controller acts as a NFS server as well.
OS+Openstack: RHEL7 + Juno
Controller: 192.168.255.1 HPDL36
Compute: 192.168.255.2 HPDL37
Compute: 192.168.255.3 HPDL38
Setup NFS server on controller server
Create 3 folders as shared source for instance-store, glance and cinder, grant enough access right:
mkdir /nfsshare; chmod 777 /nfsshare
mkdir /nfsshare_glance; chmod 777 /nfsshare_glance
mkdir /nfsshare_cinder; chmod 777 /nfsshare_cinder
Create /etc/exports
/nfsshare *(rw,no_root_squash)
/nfsshare_cinder *(rw,no_root_squash)
/nfsshare_glance *(rw,no_root_squash)
Start NFS server
systemctl start rpcbind
systemctl start nfs
systemctl start nfslock
Setup NFS clients
Glance
Mount NFS share on controller node for glance:
mount HPDL36:/nfsshare_glance /var/lib/glance/images
Nova instance-store
Mount NFS share on 2 compute nodes for shared instance-store
mount HPDL36:/nfsshare /var/lib/nova/instances
Cinder
Cinder-volume service will handle the mounting, we don’t need to do manual mounting here.
Setup Openstack
Since Glance and Nova uses those NFS mounted folder just as local filesystem, default Openstack configuration will work. Only Cinder needs special configs for NFS backend:
Create NFS shares entries into a file /etc/cinder/nfsshare
HPDL36:/nfsshare_cinder
Change owership and access right of the file:
chown root:cinder /etc/cinder/nfsshare
chmod 0640 /etc/cinder/nfsshare
Configure /etc/cinder.conf
nfs_shares_config=/etc/cinder/nfsshare
volume_driver=cinder.volume.drivers.nfs.NfsDriver
Restart cinder services
systemctl restart openstack-cinder-api
systemctl restart openstack-cinder-scheduler
systemctl restart openstack-cinder-volume
Check mounted Cinder NFS share
[[email protected] ~(keystone_admin)]# mount | grep cinder
HPDL36:/nfsshare_cinder on /var/lib/cinder/mnt/2bc8688d1bab3cab3b9a974b3f99cb82 type nfs4 (rw,relatime,vers=4.1,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.255.1,local_lock=none,addr=192.168.255.1)
Testing
Create Glance image
[[email protected] ~(keystone_admin)]# glance image-create –name cirros –disk-format qcow2 –container-format bare –is-public true –file cirros-0.3.1-x86_64-disk.qcow2
We can see the image is created and stored on /var/lib/glance/images
[[email protected] ~(keystone_admin)]# glance image-list
+————————————–+——–+————-+——————+———-+——–+
| ID | Name | Disk Format | Container Format | Size | Status |
+————————————–+——–+————-+——————+———-+——–+
| d3fd5cb6-1a88-4da8-a0af-d83f7728e76b | cirros | qcow2 | bare | 13147648 | active |
+————————————–+——–+————-+——————+———-+——–+
[[email protected] ~(keystone_admin)]# ls -lah /var/lib/glance/images/
total 13M
drwxrwxrwx 2 root root 49 Feb 11 23:29 .
drwxr-xr-x 3 glance nobody 19 Feb 11 13:38 ..
-rw-r—– 1 glance glance 13M Feb 11 23:29 d3fd5cb6-1a88-4da8-a0af-d83f7728e76b
Launch a VM:
[[email protected] ~(keystone_admin)]# nova boot –flavor m1.tiny –image cirros –nic net-id=8a7032de-e041-4e5b-a282-51534b38b15f testvm
[[email protected] ~(keystone_admin)]# nova list –fields name,status,power_state,host,networks
+————————————–+——–+——–+————-+——–+———————+
| ID | Name | Status | Power State | Host | Networks |
+————————————–+——–+——–+————-+——–+———————+
| f17ecb86-04de-44c9-9466-47ff6577b7d8 | testvm | ACTIVE | Running | HPDL37 | network=192.168.0.7 |
+————————————–+——–+——–+————-+——–+———————+
From the compute node HPDL37, we could see the VM related files are created under /var/lib/nova/instances
[[email protected] ~]# virsh list
setlocale: No such file or directory
Id Name State
—————————————————-
7 instance-0000005d running
[[email protected] ~]# ls -lah /var/lib/nova/instances/
total 4.0K
drwxrwxrwx 5 root root 129 Feb 11 23:47 .
drwxr-xr-x 9 nova nova 93 Feb 11 15:35 ..
drwxr-xr-x 2 nova nova 100 Feb 11 23:47 _base
-rw-r–r– 1 nova nova 57 Feb 11 23:43 compute_nodes
drwxr-xr-x 2 nova nova 69 Feb 11 23:47 f17ecb86-04de-44c9-9466-47ff6577b7d8
-rw-r–r– 1 nfsnobody nfsnobody 0 Feb 11 13:22 glance-touch
drwxr-xr-x 2 nova nova 143 Feb 11 23:47 locks
-rw-r–r– 1 nfsnobody nfsnobody 0 Feb 11 13:42 nova-touch
Live-Migration
Since we have shared instance-store, let’s try a live-migration:
[[email protected] ~(keystone_admin)]# nova live-migration testvm
[[email protected] ~(keystone_admin)]# nova list –fields name,status,power_state,host,networks
+————————————–+——–+——–+————-+——–+———————+
| ID | Name | Status | Power State | Host | Networks |
+————————————–+——–+——–+————-+——–+———————+
| f17ecb86-04de-44c9-9466-47ff6577b7d8 | testvm | ACTIVE | Running | HPDL38 | network=192.168.0.7 |
+————————————–+——–+——–+————-+——–+———————+
It works, now the VM is live-migrated to HPDL38 compute node.
We could do a measurement when the VM has no load, how fast the migration can be. From controller, I ping the VM every 1ms for 10000 times, which last 10000ms (10s), during the ping, I do the live-migration, then we check the result, how many packet lost we get:
[[email protected] ~(keystone_admin)]# ip netns exec qrouter-02ca3bdc-999a-4d3a-8485-c7ffd4600ebc ping 192.168.0.7 -i 0.001 -c 10000 -W 0.001
…
…
— 192.168.0.7 ping statistics —
10000 packets transmitted, 9942 received, 0% packet loss, time 10526ms
rtt min/avg/max/mdev = 0.113/0.167/1.649/0.040 ms
We lost actually 58 packets, which means basically live-migration takes only 58ms!
Create a Cinder volume
[[email protected] ~(keystone_admin)]# cinder create –display-name 5gb 5
[[email protected] ~(keystone_admin)]# cinder list
+————————————–+———–+————–+——+————-+———-+————-+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+————————————–+———–+————–+——+————-+———-+————-+
| 6e408336-43a8-453a-a5e5-928c12cdd3a1 | available | 5gb | 5 | None | false | |
+————————————–+———–+————–+——+————-+———-+————-+
[[email protected] ~(keystone_admin)]# ls -lah /var/lib/cinder/mnt/2bc8688d1bab3cab3b9a974b3f99cb82/
total 0
drwxrwxrwx 2 root root 56 Feb 12 00:16 .
drwxr-xr-x 4 cinder cinder 84 Feb 11 16:16 ..
-rw-rw-rw- 1 root root 5.0G Feb 12 00:16 volume-6e408336-43a8-453a-a5e5-928c12cdd3a1
We could see the 5GB volume is stored on the mounted Cinder NFS share.
Attach volume to an instance
[[email protected] ~(keystone_admin)]# nova volume-attach testvm 6e408336-43a8-453a-a5e5-928c12cdd3a1
+———-+————————————–+
| Property | Value |
+———-+————————————–+
| device | /dev/vdb |
| id | 6e408336-43a8-453a-a5e5-928c12cdd3a1 |
| serverId | f17ecb86-04de-44c9-9466-47ff6577b7d8 |
| volumeId | 6e408336-43a8-453a-a5e5-928c12cdd3a1 |
+———-+————————————–+
Let’s check on compute node:
[[email protected] ~]# virsh list
setlocale: No such file or directory
Id Name State
—————————————————-
7 instance-0000005d running
[[email protected] ~]# virsh domblklist 7
setlocale: No such file or directory
Target Source
————————————————
vda /var/lib/nova/instances/f17ecb86-04de-44c9-9466-47ff6577b7d8/disk
vdb /var/lib/nova/mnt/2bc8688d1bab3cab3b9a974b3f99cb82/volume-6e408336-43a8-453a-a5e5-928c12cdd3a1
We see the VM get the volume attached as vdb, and the source is /var/lib/nova/mnt/2bc8688d1bab3cab3b9a974b3f99cb82/volume-6e408336-43a8-453a-a5e5-928c12cdd3a1, which is actually the volume file on the Cinder NFS share.
[[email protected] ~]# mount |grep cinder
HPDL36:/nfsshare_cinder on /var/lib/nova/mnt/2bc8688d1bab3cab3b9a974b3f99cb82 type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.255.3,local_lock=none,addr=192.168.255.1)
[[email protected] ~]# ls -lah /var/lib/nova/mnt/2bc8688d1bab3cab3b9a974b3f99cb82
total 0
-rw-rw-rw- 1 qemu qemu 5.0G Feb 12 00:16 volume-6e408336-43a8-453a-a5e5-928c12cdd3a1
So from compute node, it also mount the Cinder NFS share to /var/lib/nova/mnt/2bc8688d1bab3cab3b9a974b3f99cb82 , then directly expose the volume file to KVM.
Now we know how it works, compute node directly mount Cinder NFS share, access the volume file, not like LVM Cinder backend, Cinder-volume service exposes volume via iSCSI target.
Live-migration with a volume attached
Will live-migration working for a VM with a volume attached?
[[email protected] ~(keystone_admin)]# nova live-migration testvm
[[email protected] ~(keystone_admin)]# nova list –fields name,status,power_state,host,networks
+————————————–+——–+——–+————-+——–+———————+
| ID | Name | Status | Power State | Host | Networks |
+————————————–+——–+——–+————-+——–+———————+
| f17ecb86-04de-44c9-9466-47ff6577b7d8 | testvm | ACTIVE | Running | HPDL37 | network=192.168.0.7 |
+————————————–+——–+——–+————-+——–+———————+
The answer is YES!
Let’s check the compute node HPDL37.
[[email protected] ~]# virsh list
setlocale: No such file or directory
Id Name State
—————————————————-
11 instance-0000005d running
[[email protected] ~]# virsh domblklist 11
setlocale: No such file or directory
Target Source
————————————————
vda /var/lib/nova/instances/f17ecb86-04de-44c9-9466-47ff6577b7d8/disk
vdb /var/lib/nova/mnt/2bc8688d1bab3cab3b9a974b3f99cb82/volume-6e408336-43a8-453a-a5e5-928c12cdd3a1
[[email protected] ~]# mount | grep cinder
HPDL36:/nfsshare_cinder on /var/lib/nova/mnt/2bc8688d1bab3cab3b9a974b3f99cb82 type nfs4 (rw,relatime,vers=4.0,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,port=0,timeo=600,retrans=2,sec=sys,clientaddr=192.168.255.2,local_lock=none,addr=192.168.255.1)
[[email protected] ~]# ls -lah /var/lib/nova/mnt/2bc8688d1bab3cab3b9a974b3f99cb82
total 0
-rw-rw-rw- 1 qemu qemu 5.0G Feb 12 00:16 volume-6e408336-43a8-453a-a5e5-928c12cdd3a1
Advanced Cinder feature testing
Create volume from Glance image
[[email protected] ~(keystone_admin)]# cinder create –image-id d3fd5cb6-1a88-4da8-a0af-d83f7728e76b –display-name vol-from-image 1
[[email protected] ~(keystone_admin)]# cinder list
+————————————–+———–+—————-+——+————-+———-+————————————–+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+————————————–+———–+—————-+——+————-+———-+————————————–+
| 209840f0-0559-4de5-ab64-bd4a8249ffd4 | available | 1gb | 1 | None | false | |
| 6e408336-43a8-453a-a5e5-928c12cdd3a1 | in-use | 5gb | 5 | None | false | f17ecb86-04de-44c9-9466-47ff6577b7d8 |
| 6fda4ea7-8f97-4c62-8df0-f04a36860d30 | available | vol-from-image | 1 | None | true | |
+————————————–+———–+—————-+——+————-+———-+————————————–+
[[email protected] ~(keystone_admin)]# ls -lh /var/lib/cinder/mnt/2bc8688d1bab3cab3b9a974b3f99cb82/
total 18M
-rw-rw-rw- 1 root root 1.0G Feb 12 00:43 volume-209840f0-0559-4de5-ab64-bd4a8249ffd4
-rw-rw-rw- 1 qemu qemu 5.0G Feb 12 00:16 volume-6e408336-43a8-453a-a5e5-928c12cdd3a1
-rw-rw-rw- 1 root root 1.0G Feb 12 00:45 volume-6fda4ea7-8f97-4c62-8df0-f04a36860d30
Create Glance image from volume
[[email protected] ~(keystone_admin)]# cinder upload-to-image 209840f0-0559-4de5-ab64-bd4a8249ffd4 image-from-vol
[[email protected] ~(keystone_admin)]# glance image-list
+————————————–+—————-+————-+——————+————+——–+
| ID | Name | Disk Format | Container Format | Size | Status |
+————————————–+—————-+————-+——————+————+——–+
| d3fd5cb6-1a88-4da8-a0af-d83f7728e76b | cirros | qcow2 | bare | 13147648 | active |
| 85d4ddb0-6159-40f6-b7ce-653eabea7142 | image-from-vol | raw | bare | 1073741824 | active |
+————————————–+—————-+————-+——————+————+——–+
[[email protected] ~(keystone_admin)]# ls -lh /var/lib/glance/images/
total 1.1G
-rw-r—– 1 glance glance 1.0G Feb 12 01:05 85d4ddb0-6159-40f6-b7ce-653eabea7142
-rw-r—– 1 glance glance 13M Feb 11 23:29 d3fd5cb6-1a88-4da8-a0af-d83f7728e76b
Boot an instance from a bootable volume
[[email protected] ~(keystone_admin)]# nova boot –flavor m1.small –block-device-mapping vda=6fda4ea7-8f97-4c62-8df0-f04a36860d30:::0 vm-boot-from-vol
[[email protected] ~(keystone_admin)]# nova list –fields name,status,power_state,host,networks
+————————————–+——————+——–+————-+——–+———————+
| ID | Name | Status | Power State | Host | Networks |
+————————————–+——————+——–+————-+——–+———————+
| f17ecb86-04de-44c9-9466-47ff6577b7d8 | testvm | ACTIVE | Running | HPDL37 | network=192.168.0.7 |
| 01c20971-7876-474c-8a38-93d39b78cc98 | vm-boot-from-vol | ACTIVE | Running | HPDL38 | network=192.168.0.8 |
+————————————–+——————+——–+————-+——–+———————+
[[email protected] ~]# virsh list
setlocale: No such file or directory
Id Name State
—————————————————-
8 instance-0000005e running
[[email protected] ~]# virsh domblklist 8
setlocale: No such file or directory
Target Source
————————————————
vda /var/lib/nova/mnt/2bc8688d1bab3cab3b9a974b3f99cb82/volume-6fda4ea7-8f97-4c62-8df0-f04a36860d30
Boot an instance from image (by creating new volume)
[[email protected] ~(keystone_admin)]# nova boot –flavor m1.tiny –block-device source=image,id=d3fd5cb6-1a88-4da8-a0af-d83f7728e76b,dest=volume,size=6,shutdown=preserve,bootindex=0 vm-boot-from-image-create-new-vol
[[email protected] ~(keystone_admin)]# nova list –fields name,status,power_state,host,networks
+————————————–+———————————–+——–+————-+——–+———————+
| ID | Name | Status | Power State | Host | Networks |
+————————————–+———————————–+——–+————-+——–+———————+
| f17ecb86-04de-44c9-9466-47ff6577b7d8 | testvm | ACTIVE | Running | HPDL37 | network=192.168.0.7 |
| 6cacf835-2adb-4730-ac11-cceacf1d0915 | vm-boot-from-image-create-new-vol | ACTIVE | Running | HPDL38 | network=192.168.0.9 |
| 01c20971-7876-474c-8a38-93d39b78cc98 | vm-boot-from-vol | ACTIVE | Running | HPDL37 | network=192.168.0.8 |
+————————————–+———————————–+——–+————-+——–+———————+
[[email protected]HPDL36 ~(keystone_admin)]# cinder list
+————————————–+———–+—————-+——+————-+———-+————————————–+
| ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |
+————————————–+———–+—————-+——+————-+———-+————————————–+
| 209840f0-0559-4de5-ab64-bd4a8249ffd4 | available | 1gb | 1 | None | false | |
| 6e408336-43a8-453a-a5e5-928c12cdd3a1 | in-use | 5gb | 5 | None | false | f17ecb86-04de-44c9-9466-47ff6577b7d8 |
| 6fda4ea7-8f97-4c62-8df0-f04a36860d30 | in-use | vol-from-image | 1 | None | true | 01c20971-7876-474c-8a38-93d39b78cc98 |
| 7d8a4fc5-5f75-4273-a2ac-3e36521be37c | in-use | | 6 | None | true | 6cacf835-2adb-4730-ac11-cceacf1d0915 |
+————————————–+———–+—————-+——+————-+———-+————————————–+
[[email protected] ~]# virsh list
setlocale: No such file or directory
Id Name State
—————————————————-
9 instance-0000005f running
[[email protected] ~]# virsh domblklist 9
setlocale: No such file or directory
Target Source
————————————————
vda /var/lib/nova/mnt/2bc8688d1bab3cab3b9a974b3f99cb82/volume-7d8a4fc5-5f75-4273-a2ac-3e36521be37c
Volume snapshot related features
Not supported at the moment, coming from Kilo: https://blueprints.launchpad.net/cinder/+spec/nfs-snapshots
Volume clone
Not supported at the moment