In a previous blog, I discussed how cloud-init can be used to customize a CoreOS image deployed as an instance on Eucalyptus – which happens to work in the same fashion on AWS. This is a follow-up blog to demonstrate how to use Eucalyptus Cloudformation (which is in Tech Preview in Eucalyptus 4.0.0/4.0.1) to deploy a CoreOS cluster on Eucalyptus, customizing each instance using the cloud-config service. This setup will allow cloud users to test out CoreOS clusters on Eucalyptus, just as CoreOS recommends on AWS EC2.
Prerequisites
Just as in the previous blog discussing the use of CoreOS, using Eucalyptus IAM is highly recommended. In addition, to the prerequisites mentioned in that blog, the following service API actions need to be allowed (at a minimum) in the IAM policy for the user(s) that want to utilize this blog:
- CreateAutoScalingGroup
- CreateLaunchConfiguration
- DescribeAutoScalingInstances
- CreateStack
- DescribeStackEvents
- DescribeStackResource
- DescribeStackResources
- DescribeStacks
- ListStackResources
- ListStacks
In addition to having the correct IAM policy actions authorized, the cloud user needs to be using the latest version of euca2ools with Eucalyptus 4.0.1. Once these prerequisites are met, the Eucalyptus cloud needs to be prepared with the correct EMI for the deployment.
Adding CoreOS Image To Eucalyptus
In order to deploy an CoreOS cluster on Eucalyptus, the CoreOS image needs to be bundled, uploaded and registered. To obtain the CoreOS image, download the image from the CoreOS Beta Release site. For example:
# wget -q http://beta.release.core-os.net/amd64-usr/current/coreos_production_ami_image.bin.bz2 # bunzip2 -d coreos_production_ami_image.bin.bz2 # qemu-img info coreos_production_ami_image.bin image: coreos_production_ami_image.bin file format: raw virtual size: 4.4G (4699717632 bytes) disk size: 4.4G
Once the image has been downloaded and user credentials have been sourced, use euca-install-image to bundle, upload and register the image as an instance store-backed HVM image to be used with the Cloudformation template. In addition, note the EC2_USER_ID value present in the eucarc file as it will be used with the Cloudformation template as well.
# euca-install-image -b coreos-production-ami -i coreos_production_ami_image.bin --virtualization-type hvm -n coreos-hvm -r x86_64 .... /var/tmp/bundle-WsLdGB/coreos_production_ami_image.bin.part.19 100% |=================================================================| 6.08 MB 12.66 MB/s Time: 0:00:00 /var/tmp/bundle-WsLdGB/coreos_production_ami_image.bin.manifest.xml 100% |============================================================| 6.28 kB 2.66 kB/s Time: 0:00:02 IMAGE emi-DAB316FD
CoreOS etcd Discovery Service Token
CoreOS uses a service called etcd on each machine to handle coordination of services in a cluster. To make sure the machines know that they are part of the same cluster, a discovery token needs to be generated and shared with each instance using the cloud-config service. To generate a custom token, open a browser and go to the following URL:
https://discovery.etcd.io/new
The URL similar to the example below should show up in the browser:
https://discovery.etcd.io/7b67f765e2f264cf65b850a849a7da7e
Take note of the URL because it will be needed later.
Select VM Type and Availability Zone on Eucalyptus
Before deploying the CoreOS cluster on Eucalyptus, the user needs to determine the instance type, and the availability zone (Eucalyptus Cluster). In order to do this, use euca-describe-instance-types to show the instance types, availability zone(s), and the capacity for each instance type available in the availability zone(s).
# euca-describe-instance-types --show-capacity --by-zone AVAILABILITYZONE SirLuciousLeftFoot INSTANCETYPE Name CPUs Memory (MiB) Disk (GiB) Used / Total Used % INSTANCETYPE t1.micro 1 256 5 0 / 6 0% INSTANCETYPE m1.small 1 512 10 0 / 6 0% INSTANCETYPE m1.medium 1 1024 10 0 / 6 0% INSTANCETYPE c1.xlarge 2 2048 10 0 / 3 0% INSTANCETYPE m1.large 2 1024 15 0 / 3 0% INSTANCETYPE c1.medium 1 1024 20 0 / 6 0% INSTANCETYPE m1.xlarge 2 1024 30 0 / 3 0% INSTANCETYPE m2.2xlarge 2 4096 30 0 / 3 0% INSTANCETYPE m3.2xlarge 4 4096 30 0 / 1 0% INSTANCETYPE m2.xlarge 2 2048 40 0 / 3 0% INSTANCETYPE m3.xlarge 2 2048 50 0 / 3 0% INSTANCETYPE cc1.4xlarge 8 3072 60 0 / 0 INSTANCETYPE m2.4xlarge 8 4096 60 0 / 0 INSTANCETYPE hi1.4xlarge 8 6144 120 0 / 0 INSTANCETYPE cc2.8xlarge 16 6144 120 0 / 0 INSTANCETYPE cg1.4xlarge 16 12288 200 0 / 0 INSTANCETYPE cr1.8xlarge 16 16384 240 0 / 0 INSTANCETYPE hs1.8xlarge 48 119808 24000 0 / 0
AVAILABILITYZONE ViciousLiesAndDangerousRumors INSTANCETYPE Name CPUs Memory (MiB) Disk (GiB) Used / Total Used % INSTANCETYPE t1.micro 1 256 5 4 / 12 33% INSTANCETYPE m1.small 1 512 10 4 / 12 33% INSTANCETYPE m1.medium 1 1024 10 4 / 12 33% INSTANCETYPE c1.xlarge 2 2048 10 2 / 6 33% INSTANCETYPE m1.large 2 1024 15 2 / 6 33% INSTANCETYPE c1.medium 1 1024 20 4 / 12 33% INSTANCETYPE m1.xlarge 2 1024 30 2 / 6 33% INSTANCETYPE m2.2xlarge 2 4096 30 0 / 2 0% INSTANCETYPE m3.2xlarge 4 4096 30 0 / 2 0% INSTANCETYPE m2.xlarge 2 2048 40 2 / 6 33% INSTANCETYPE m3.xlarge 2 2048 50 2 / 6 33% INSTANCETYPE cc1.4xlarge 8 3072 60 0 / 0 INSTANCETYPE m2.4xlarge 8 4096 60 0 / 0 INSTANCETYPE hi1.4xlarge 8 6144 120 0 / 0 INSTANCETYPE cc2.8xlarge 16 6144 120 0 / 0 INSTANCETYPE cg1.4xlarge 16 12288 200 0 / 0 INSTANCETYPE cr1.8xlarge 16 16384 240 0 / 0 INSTANCETYPE hs1.8xlarge 48 119808 24000 0 / 0
For this blog, the availability zone ‘ViciousLiesAndDangerousRumors’ and the instance type ‘c1.medium’ will be used as a parameter for the Cloudformation template. Now, Eucalyptus Cloudformation is ready to be used.
Deploying the CoreOS Cluster
Final Preparations
Before using the Cloudformation template for the CoreOS cluster, a keypair needs to be created. This keypair will also be used as a parameter for the Cloudformation template.
To obtain the template, download the template from coreos-cloudformation-template bucket on AWS S3. Once the file has been downloaded, the following edits need to happen.
The first edit is to define the ‘AvailabilityZones’ in the ‘Properties’ section of the ‘CoreOsGroup’ resource. For example, ‘ViciousLiesAndDangerousRumors’ has been placed as the value for ‘AvailabilityZones’:
"CoreOsGroup" : {
"Type" : "AWS::AutoScaling::AutoScalingGroup",
"Properties" : {
"AvailabilityZones" : [ "ViciousLiesAndDangerousRumors" ],
"LaunchConfigurationName" : { "Ref" : "CoreOsLaunchConfig" },
"MinSize" : { "Ref" : "ClusterSize" },
"MaxSize" : { "Ref" : "ClusterSize" }
}
},
The second and final edit, is to update the ‘UserData’ property to have the correct value for the discovery token that was provided earlier in this blog. For example:
"UserData" : { "Fn::Base64" : { "Fn::Join" : ["",[
"#cloud-config","\n",
"coreos:","\n",
" etcd:","\n",
" discovery: https://discovery.etcd.io/7b67f765e2f264cf65b850a849a7da7e","\n",
" addr: $private_ipv4:4001","\n",
" peer-addr: $private_ipv4:7001","\n",
" units:","\n",
Now that these values have been updated, the CoreOS cluster can be deployed.
Create the Stack
To deploy the cluster, use euform-create-stack with the parameter values filled in appropriately. For example:
# euform-create-stack --template-file cfn-coreos-as.json --parameter "CoreOSImageId=emi-DAB316FD" --parameter "UserKeyPair=account1-user01" --parameter "AcctId=408396244283" --parameter "ClusterSize=3" --parameter "VmType=c1.medium" CoreOSClusterStack arn:aws:cloudformation:bigboi:408396244283:stack/CoreOSClusterStack/43d53adb-68f2-4317-bd2b-3da661977ebc
The ‘ClusterSize’ parameter is completely dependent upon how big of a CoreOS cluster the user would like to have based upon the instance types supported on the Eucalyptus cloud. Please refer to the CoreOS documentation regarding optimal cluster sizes to see what would best suit the use case of the cluster.
Check Out The Stack Resources
After deploying the Cloudformation stack, after a few minutes, use euform-describe-stacks to check the status of the stack. The status of the stack should return with CREATE_COMPLETE.
# euform-describe-stacks STACK CoreOSClusterStack CREATE_COMPLETE Complete! Deploy CoreOS Cluster 2014-08-28T22:31:02.669Z OUTPUT AutoScalingGroup CoreOSClusterStack-CoreOsGroup-G7Y7YVWI4DOPG
To check out the resources associated with the Cloudformation stack, use euform-describe-stack-resources:
# euform-describe-stack-resources -n CoreOSClusterStack --region account1-user01@ RESOURCE CoreOsSecurityGroupIngress2 CoreOsSecurityGroupIngress2 AWS::EC2::SecurityGroupIngress CREATE_COMPLETE RESOURCE CoreOsLaunchConfig CoreOSClusterStack-CoreOsLaunchConfig-FFSTY76SDQAWB AWS::AutoScaling::LaunchConfiguration CREATE_COMPLETE RESOURCE CoreOsSecurityGroup CoreOSClusterStack-CoreOsSecurityGroup-D3WCUH0SKHYVC AWS::EC2::SecurityGroup CREATE_COMPLETE RESOURCE CoreOsSecurityGroupIngress1 CoreOsSecurityGroupIngress1 AWS::EC2::SecurityGroupIngress CREATE_COMPLETE RESOURCE CoreOsGroup CoreOSClusterStack-CoreOsGroup-G7Y7YVWI4DOPG AWS::AutoScaling::AutoScalingGroup CREATE_COMPLETE
Check the status of the instances by using the value returned for ‘AutoScalingGroup’ from the euform-describe-stacks output:
# euscale-describe-auto-scaling-groups CoreOSClusterStack-CoreOsGroup-G7Y7YVWI4DOPG --region account1-user01@ AUTO-SCALING-GROUP CoreOSClusterStack-CoreOsGroup-G7Y7YVWI4DOPG CoreOSClusterStack-CoreOsLaunchConfig-FFSTY76SDQAWB ViciousLiesAndDangerousRumors 3 33 Default INSTANCE i-E6FB62D0 ViciousLiesAndDangerousRumors InService Healthy CoreOSClusterStack-CoreOsLaunchConfig-FFSTY76SDQAWB INSTANCE i-2AC4CC35 ViciousLiesAndDangerousRumors InService Healthy CoreOSClusterStack-CoreOsLaunchConfig-FFSTY76SDQAWB INSTANCE i-442C4692 ViciousLiesAndDangerousRumors InService Healthy CoreOSClusterStack-CoreOsLaunchConfig-FFSTY76SDQAWB
Check the Status of the CoreOS Cluster
In order to check the status of the CoreOS cluster, SSH into one of the instances (the port was opened in the security group as part of the Cloudformation template), and use the fleetctl command:
# euca-describe-instances i-E6FB62D0 i-2AC4CC35 i-442C4692 --region account1-user01@ RESERVATION r-AF98046C 408396244283 CoreOSClusterStack-CoreOsSecurityGroup-D3WCUH0SKHYVC INSTANCE i-2AC4CC35 emi-DAB316FD euca-10-104-6-233.bigboi.acme.eucalyptus-systems.com euca-172-18-223-111.bigboi.internal running account1-user01 0 c1.medium 2014-08-28T22:15:48.043Z ViciousLiesAndDangerousRumors monitoring-enabled 10.104.6.233 172.18.223.111 instance-store hvm d88cac3d-ce92-4c3b-98ee-7e507afc26cb_ViciousLiesAndDangerousR_1 sg-31503C69 x86_64 TAG instance i-2AC4CC35 aws:autoscaling:groupName CoreOSClusterStack-CoreOsGroup-G7Y7YVWI4DOPG RESERVATION r-A24611A2 408396244283 CoreOSClusterStack-CoreOsSecurityGroup-D3WCUH0SKHYVC INSTANCE i-442C4692 emi-DAB316FD euca-10-104-6-235.bigboi.acme.eucalyptus-systems.com euca-172-18-223-227.bigboi.internal running account1-user01 0 c1.medium 2014-08-28T22:15:48.056Z ViciousLiesAndDangerousRumors monitoring-enabled 10.104.6.235 172.18.223.227 instance-store hvm 1281a747-69a7-4f26-8fe2-2dea6b8b858d_ViciousLiesAndDangerousR_1 sg-31503C69 x86_64 TAG instance i-442C4692 aws:autoscaling:groupName CoreOSClusterStack-CoreOsGroup-G7Y7YVWI4DOPG RESERVATION r-089053BE 408396244283 CoreOSClusterStack-CoreOsSecurityGroup-D3WCUH0SKHYVC INSTANCE i-E6FB62D0 emi-DAB316FD euca-10-104-6-232.bigboi.acme.eucalyptus-systems.com euca-172-18-223-222.bigboi.internal running account1-user01 0 c1.medium 2014-08-28T22:15:38.146Z ViciousLiesAndDangerousRumors monitoring-enabled 10.104.6.232 172.18.223.222 instance-store hvm c0dc6cca-5fa3-4614-a4ec-8a902bf6ff66_ViciousLiesAndDangerousR_1 sg-31503C69 x86_64 TAG instance i-E6FB62D0 aws:autoscaling:groupName CoreOSClusterStack-CoreOsGroup-G7Y7YVWI4DOPG
# ssh -i account1-user01/account1-user01.priv core@euca-10-104-6-232.bigboi.acme.eucalyptus-systems.com Last login: Thu Aug 28 15:32:34 2014 from 10.104.10.55 CoreOS (beta) core@euca-172-18-223-222 ~ $ fleetctl list-machines -full=true MACHINE IP METADATA 6f4e3de463490a7644e3d7c80d826770 172.18.223.227 - 929c1f121860c63b506c0b951c19de7b 172.18.223.222 - a08155346fb55f9b53b154d6447af0fa 172.18.223.211 - core@euca-172-18-223-222 ~ $
The cluster status can also be checked by going to the discovery token URL that was placed in the Cloudformation template.
Conclusion
Just as on AWS, Cloudformation can be used to deploy a CoreOS cluster on Eucalyptus. Users will be able to test out different use cases, such as Cluster-Level Container Development with fleet, or get more familiar with CoreOS by going through the CoreOS documentation. As always, feel free to ask any questions. Feedback is always welcome.
Enjoy!
More Mind Spew-age from Harold Spencer Jr. 