/ HA

Openstack Heat try-out

What’s Heat ?

Heat is a service to orchestrate multiple composite cloud applications using the AWS CloudFormation template format, it’s Openstack’s implementation of CloudFormation and CloudWatch.

Heat integrates other core components of OpenStack into a one-file template system. The templates allow creation of most OpenStack resource types, the main functions of Heat cover:

  • vAPP stack creation/update/deletion (suspend/resume will be added in Havana)
  • High Availability
  • Autoscaling
  • Manual scale in/out(in Havana)

How to install Heat and make it ready to use?


->Heat has been included in RDO repo, installation can be done by Yum:

yum install openstack-heat-api openstack-heat-api-cfn openstack-heat-api-cloudwatch \
openstack-heat-cli openstack-heat-common openstack-heat-engine \
python-heatclient python-heatclient-doc heat-cfntools heat-jeos

->Comment out following line since we use RabbitMQ instead of QPID as message queue. Do this for heat-api-cfn.conf, heat-api-cloudwatch.conf, heat-api.conf, heat-engine.conf under /etc/heat.


->Configure [filter:authtoken] section of heat-api-cfn-paste.ini  heat-api-cloudwatch-paste.ini  heat-api-paste.ini under /etc/heat

paste.filter_factory = heat.common.auth_token:filter_factory  
service_protocol = http  
service_host = api-vip  
service_port = 5000  
auth_host = api-vip  
auth_port = 35357  
auth_protocol = http  
auth_uri = http://api-vip:35357/v2.0
admin_tenant_name = service  
admin_user = heat  
admin_password = password

->Configure [filter:ec2authtoken] section of  heat-api-cfn-paste.ini  heat-api-cloudwatch-paste.ini under /etc/heat

paste.filterfactory = heat.api.aws.ec2token:EC2Tokenfilterfactory 
authuri = http://api-vip:5000/v2.0 
keystoneec2uri = http://api-vip:5000/v2.0/ec2tokens

->Update /etc/heat/heat-engine.conf as below:

heatmetadataserverurl = 
heatwaitconditionserverurl = 
heatwatchserverurl = 
sqlconnection = mysql://heat:[email protected]/heat

The IP for metadata, watchcondition and watch_server should be the host IP of heat services are running on, and should be reachable from VM instance.

->Setup database for heat:

heat-db-setup rpm -y -r <root password of mysql>

->Setup keystone entries for heat:


->Start heat services:

service openstack-heat-api start 
service openstack-heat-api-cfn start 
service openstack-heat-api-cloudwatch start 
service openstack-heat-engine start

2.Prepare JEOS(Just enough OS)

JEOS image is basic OS image with cloud-init  and heat-cfn-tools pre-installed, they are the keys for Heat and VM exchanging information and doing post and regular jobs.

Pre-built JEOS of Fedora 17 and 18 can be downloaded from: http://fedorapeople.org/groups/heat/prebuilt-jeos-images/

To make our own JEOS on RHEL, following the steps:

a. Assume we have a RHEL 6.3 image registered in Glance already, launch a VM instance from it.
b. Log into the VM, install cloud-init and heat-cfn-tools

yum -y update 
export http_proxy=   #This is needed if lab has no direct internet access 
curl -O http://ftp.ps.pl/pub/Linux/fedora-epel/6/i386/epel-release-6-7.noarch.rpm 
rpm -Uvh epel-release-6-7.noarch.rpm 
yum -y install perl python python-setuptools cloud-init python-pip 
pip-python install argparse ‘boto==2.5.2’ heat-cfntools 
cfn-create-aws-symlinks –source /usr/bin 
rm -rf /etc/udev/rules.d/70-persistent-net.rules

c. From Horizon GUI or CLI, make a snapshot for this VM, name it as “rhel63-x86_64-cfntools”.  This snapshot actually becomes a valid JEOS image.

Use cases

1.Deploy a single WordPress blog server with a floating IP associated.

This templates deploy a single VM instance with a associated floating IP, then install mysql-server, httpd and wordpress packages, initial database, integrate each other and finally start up wordpress web service.

->Download the template file

wget https://github.com/abckey/heat-templates/raw/master/WordPress_Single_Instance_With_EIP.template

->Source a tenant credential

source /root/nceprc

->Create a key-pair

nova  keypair-add ncepkey > ncepkey.pem
chmod 400 ncepkey.pem

->Deploy this template by heat

heat stack-create wordpress  -f WordPress_Single_Instance_With_EIP.template \
 -P “InstanceType=m1.medium;DBUsername=wordpress;DBPassword=worldpress;KeyName=ncepkey;LinuxDistribution=RHEL-6.3”

Parameters followed after “-P” are needed for instance type selection, DB access setup, key-pair selection and image selection.

->Check stack creation process

Once the stack-create is issued, we should see the stack in “CREATE_IN_PROGRESS” status from output of “heat list”

Once creation is complete, we can see the status will change to “CREATE_COMPLETE”.

[[email protected] heat(keystone_ncep)]# heat list  
| ID | Name | Status | Created |  
| a6740599-2c98-41b8-a55e-490f28f6f6f8 | wordpress | CREATE_COMPLETE | 2013-08-05T06:57:40Z |  

We can also see detailed resource status in the stack by:

[[email protected] heat(keystone_ncep)]# heat resource-list wordpress

| Name | Type | Status | Updated | 
| IPAddress | AWS::EC2::EIP | CREATECOMPLETE | 2013-08-05T06:57:40Z | 
| WebServerSecurityGroup | AWS::EC2::SecurityGroup | CREATECOMPLETE | 2013-08-05T06:57:41Z | 
| IPAssoc | AWS::EC2::EIPAssociation | CREATECOMPLETE | 2013-08-05T06:57:52Z | 
| WebServer | AWS::EC2::Instance | CREATECOMPLETE | 2013-08-05T06:57:52Z | 

Also heat stack-show wordpress can be used to show detailed information of the stack

->Try to access the wordpress web UI

From nova list, we could see a VM named wordpress.WebServer is created with internal IP and floating IP:

[[email protected] heat(keystone_ncep)]# nova list 
| ID | Name | Status | Networks | 
| 02da7185-a0a6-44cf-8076-875754163dac | wordpress.WebServer | ACTIVE | ncep-net=, | 

Try to access the wordpress website by its floating IP

You should see the login page in the browser!!!

2.Deploy wordpress blog server with 2 VM instances(front-end + back-end topology)

This template deploy a 2 VM wordpress solution, one VM running web server, 2nd VM running Mysql DB.

->Download the template:

wget https://github.com/abckey/heat-templates/raw/master/WordPress_2_Instances.template

->Deploy the template by heat

heat stack-create wp-2-vm  -f WordPress_2_Instances.template -P \n

->From heat resource-list wp-2-vm, we can see 2 VM instances are created, also a floating IP and association with Webserver is there

[[email protected] heat(keystonencep)]# heat resource-list wp-2-vm 
| Name | Type | Status | Updated | 
| DatabaseServer | AWS::EC2::Instance | CREATECOMPLETE | 2013-08-05T07:14:04Z | 
| IPAddress | AWS::EC2::EIP | CREATECOMPLETE | 2013-08-05T07:14:13Z | 
| WebServer | AWS::EC2::Instance | CREATECOMPLETE | 2013-08-05T07:14:13Z | 
| IPAssoc | AWS::EC2::EIPAssociation | CREATE_COMPLETE | 2013-08-05T07:14:14Z | 

->Try wordpress website with floating IP of WebServer VM instance

[[email protected] heat(keystone_ncep)]# nova list 
| ID | Name | Status | Networks | 
| 4879bb03-bc20-4c33-91e2-d4a7e197d2e5 | wp-2-vm.DatabaseServer | ACTIVE | ncep-net= | 
| 8e1dc2c7-bc34-4e5d-b833-b64dc8a12c9b | wp-2-vm.WebServer | ACTIVE | ncep-net=, | 

3.Deploy a wordpress blog server with EBS volume as mysql DB storage

This template deploy a VM instance with httpd, wordpress and mysql-server installed, a EBS volume is attached to instance as Mysql DB storage point.

->Download the template

wget https://github.com/abckey/heat-templates/raw/master/WordPress_Single_Instance_With_EBS_EIP.template
heat stack-create wp-ebs -f WordPress_Single_Instance_With_EBS_EIP.template -P “VolumeSize=5;InstanceType=m1.medium;DBUsername=wordpress;DBPassword=worldpress;KeyName=ncepkey;LinuxDistribution=RHEL-6.3”
#Here we need use parameter “VolumeSize” to specify the volume size we intend to use as DB storage.
->From resource list , we can see it has volume and volume attachment there
[[email protected] heat(keystone_ncep)]# heat resource-list wp-ebs  
  | Name | Type | Status | Updated |  
  | DataVolume | AWS::EC2::Volume | CREATE_COMPLETE | 2013-08-05T07:35:48Z |  
  | WikiServerSecurityGroup | AWS::EC2::SecurityGroup | CREATE_COMPLETE | 2013-08-05T07:35:48Z |  
  | IPAddress | AWS::EC2::EIP | CREATE_COMPLETE | 2013-08-05T07:35:59Z |  
  | WikiServer | AWS::EC2::Instance | CREATE_COMPLETE | 2013-08-05T07:35:59Z |  
  | IPAssoc | AWS::EC2::EIPAssociation | CREATE_COMPLETE | 2013-08-05T07:36:00Z |  
  | MountPoint | AWS::EC2::VolumeAttachment | CREATE_COMPLETE | 2013-08-05T07:36:02Z |  
->Check volume and attachment
[[email protected] heat(keystone_ncep)]# cinder list  
  | ID | Status | Display Name | Size | Volume Type | Bootable | Attached to |  
  | 16c9c84f-9bf3-43c7-853d-453ec161b170 | in-use | wp-ebs.DataVolume | 5 | None | false | 0738154d-8d58-4a84-a5fc-71774228105d |  

Inside vm, we can check mounts:

[[email protected] ~]# df -h  
  Filesystem Size Used Avail Use% Mounted on  
  /dev/vda1 5.0G 1.6G 3.1G 34% /  
  tmpfs 1.9G 0 1.9G 0% /dev/shm  
  /dev/vdb1 938M 39M 853M 5% /var/lib/mysql
->Try to access wordpress by floating IP.

4.Deploy a wordpress blog server with HA protection enabled

This template deploy a VM instance with wordpress server installed, and HA is enabled to monitor httpd/mysqld services to make sure they are already running.
->Download template:
wget https://github.com/abckey/heat-templates/raw/master/WordPress_Single_Instance_With_HA.template
->Assign admin role to the tenant user by `keystone user-role-add` command.

To use HA, waitcondition resource is needed, which needs to create an ec2signed url and to do that it creates a restricted user in keystone.

keystone user-role-add –user <ncep user id> –role <admin role id>  –tenant-id <ncep tenant id>
heat stack-create wp-ha -f WordPress_Single_Instance_With_HA.template -P “HupPollInterval=1;InstanceType=m1.medium;DBUsername=wordpress;DBPassword=worldpress;KeyName=ncepkey;LinuxDistribution=RHEL-6.3”
#Parameter `HupPollInterval` is needed to specify the cfn-hup process running interval in minutes, cfn-hup is to check services status, and restart them if return code of “service status” is not “0”.
->Check resource
[[email protected] heat(keystone_ncep)]# heat resource-list wp-ha  
  | Name | Type | Status | Updated |  
  | CfnUser | AWS::IAM::User | CREATE_COMPLETE | 2013-08-05T07:53:30Z |  
  | WaitHandle | AWS::CloudFormation::WaitConditionHandle | CREATE_COMPLETE | 2013-08-05T07:53:30Z |  
  | WebServerKeys | AWS::IAM::AccessKey | CREATE_COMPLETE | 2013-08-05T07:53:30Z |  
  | WaitCondition | AWS::CloudFormation::WaitCondition | IN_PROGRESS | 2013-08-05T07:53:41Z |  
  | WikiDatabase | AWS::EC2::Instance | CREATE_COMPLETE | 2013-08-05T07:53:41Z |  

Here WaitCondition actually is waiting VM instance is properly started and configured, then goes to next step

Whole resource list after create_complete:

 [[email protected] heat(keystone_ncep)]# heat resource-list wp-ha  
  | Name | Type | Status | Updated |  
  | CfnUser | AWS::IAM::User | CREATE_COMPLETE | 2013-08-05T07:53:30Z |  
  | WebServerKeys | AWS::IAM::AccessKey | CREATE_COMPLETE | 2013-08-05T07:53:30Z |  
  | WaitHandle | AWS::CloudFormation::WaitConditionHandle | CREATE_COMPLETE | 2013-08-05T07:54:41Z |  
  | WikiDatabase | AWS::EC2::Instance | CREATE_COMPLETE | 2013-08-05T07:54:41Z |  
  | HttpFailureAlarm | AWS::CloudWatch::Alarm | CREATE_COMPLETE | 2013-08-05T07:54:47Z |  
  | WaitCondition | AWS::CloudFormation::WaitCondition | CREATE_COMPLETE | 2013-08-05T07:54:47Z |  
  | WebServerRestartPolicy | OS::Heat::HARestarter | CREATE_COMPLETE | 2013-08-05T07:54:47Z |  

Resource OS::Heat::HARestart takes care of service HA.

->Try to access wordpress web by floating IP

Login page should show up.

->Try HA by stopping httpd service

In the VM, do service httpd stop.

Check cfn-hup.log

 [[email protected] ~]# tail -f /var/log/cfn-hup.log
 DEBUG [2013-08-05 16:28:02,552] Running command: /sbin/service httpd status  
  DEBUG [2013-08-05 16:28:02,609] Return code of 3 after executing: ‘[‘su’, ‘root’, ‘-c’, u’/sbin/service httpd status’]’  
  WARNING [2013-08-05 16:28:02,610] Restarting service httpd  
  DEBUG [2013-08-05 16:28:02,610] Running command: /sbin/service httpd start  
  DEBUG [2013-08-05 16:28:02,732] Running command: /etc/cfn/notify-on-httpd-restarted  
  DEBUG [2013-08-05 16:28:03,634] Running command: /sbin/service mysqld status  
  DEBUG [2013-08-05 16:28:03,714] Running command: /sbin/service crond status

We can see once cfn-hup found httpd service has problem, it tried to start httpd service.

->Check wordpress web access again, it should work again.