Updated CoreOS Cluster Cloudformation Template for HPE Helion Eucalyptus 4.2 VPC Deployments

In 2014, I created a series of blog posts that have discussed using CoreOS on Eucalyptus cloud infrastructures.  This blog post is an updated version of the entry which discussed how to deploy a CoreOS cluster using a cloudformation template on Eucalyptus 4.0.1.  It will cover how to deploy a CoreOS cluster using Cloudformation on a HPE Helion Eucalyptus 4.2 VPC environment.

In HPE Helion Eucalyptus 4.1, VPC (Virtual Private Cloud) was in technical preview state.  With the release of Eucalyptus 4.2, VPC was upgraded to stable release.  HPE Helion Eucalyptus VPC provides similar features as AWS VPC.  For more information about what is currently supported in Eucalyptus VPC, please refer to the online documentation.

Prerequisites

Prerequisites for this blog entry are listed in the following previous blogs:

Please note the information regarding HPE Helion Eucalyptus IAM and how to obtain the CoreOS Beta AMI image in the previous listed blog entries.

CoreOS ETCD Discovery Service Token

When setting up the CoreOS cluster, the method used to handle cluster membership is using etcd Discovery.  This provides a unique discovery URL that will show all the members of the cluster.  To obtain a token for the size of the cluster you desire, use the following URL and add the value for the size of the cluster.  For example, if the cluster will have five members, using curl – the request URL will look like the following:

curl https://discovery.etcd.io/new?size=5

The value returned will look similar to the following:

https://discovery.etcd.io/fdd7d8ac203d2cac0c27ead148ad83ed

This URL can be referenced to see if all the members of the cluster registered successfully.

Deploying the Cluster on HPE Helion Eucalyptus VPC

When deploying the cluster on a Eucalyptus VPC environment, there are additional variables that have to be taken into account.  To download the example template, use the following URL:

https://s3-us-west-1.amazonaws.com/cfn-coreos-deployment/cfn-coreos-as-vpc.json

After downloading the template, use either euca2ools or AWS CLI to validate the template.  This will display the arguments that need to be passed when creating the cloudformation stack on Eucalyptus.  For example:

# euform-validate-template --template-file cfn-coreos-as.json 
DESCRIPTION Deploy CoreOS Cluster on Eucalyptus VPC
PARAMETER VpcId false VpcId of your existing Virtual Private Cloud (VPC)
PARAMETER Subnets false The list of SubnetIds in your Virtual Private Cloud (VPC)
PARAMETER AZs false The list of AvailabilityZones for your Virtual Private Cloud (VPC)
PARAMETER CoreOSImageId false CoreOS Image Id
PARAMETER UserKeyPair true User Key Pair
PARAMETER ClusterSize false Desired CoreOS Cluster Size
PARAMETER VmType false Desired VM Type for Instances

Notice the template requires unique variables associated with HPE Helion Eucalyptus VPC.

Now that the template has been downloaded, create the CoreOS stack using euca2ools.  For example:

# euform-create-stack CoreOSCluster --template-file cfn-coreos-as.json --parameter Subnets=subnet-0814e7aa,subnet-5d816215,subnet-c3755d6c --parameter AZs=euca-east-1c,euca-east-1b,euca-east-1a --parameter CoreOSImageId=emi-dfa27782 --parameter UserKeyPair=devops-admin --parameter ClusterSize=5 --parameter VmType=m1.large --parameter VpcId=vpc-d7fcff27

Once the cluster has been deployed, confirm that the cloudformation stack deployed successfully:

# euform-describe-stacks
STACK CoreOSCluster CREATE_COMPLETE Complete! Deploy CoreOS Cluster on Eucalyptus VPC 2016-01-01T21:09:10.965Z
PARAMETER VpcId vpc-d7fcff27
PARAMETER Subnets subnet-0814e7aa,subnet-5d816215,subnet-c3755d6c
PARAMETER AZs euca-east-1c,euca-east-1b,euca-east-1a
PARAMETER CoreOSImageId emi-dfa27782
PARAMETER UserKeyPair ****
PARAMETER ClusterSize 5
PARAMETER VmType m1.large
OUTPUT AutoScalingGroup CoreOSCluster-CoreOsGroup-JTKMRINKKMYDI

Check the discovery URL using curl, wget or any browser to confirm that the cluster membership completed:

# curl https://discovery.etcd.io/fdd7d8ac203d2cac0c27ead148ad83ed
{"action":"get","node":{"key":"/_etcd/registry/fdd7d8ac203d2cac0c27ead148ad83ed","dir":true,"nodes":[{"key":"/_etcd/registry/fdd7d8ac203d2cac0c27ead148ad83ed/d0a4c6d73d0d8d17","value":"8981923b54d7d7f46fabc527936a7dcf=http://172.31.4.17:2380","modifiedIndex":953833155,"createdIndex":953833155},{"key":"/_etcd/registry/fdd7d8ac203d2cac0c27ead148ad83ed/12b6e6e78c9cb70c","value":"33a3209006d2be1d5be0da6eaea007c5=http://172.31.19.215:2380","modifiedIndex":953833156,"createdIndex":953833156},{"key":"/_etcd/registry/fdd7d8ac203d2cac0c27ead148ad83ed/d5c5d93e360ba87","value":"e71b1fefcd65c43a0fbacc7103efbc2b=http://172.31.22.157:2380","modifiedIndex":953833162,"createdIndex":953833162},{"key":"/_etcd/registry/fdd7d8ac203d2cac0c27ead148ad83ed/cffd4985c990f872","value":"f047b9ff24f3d0c4e74c660709103b36=http://172.31.6.166:2380","modifiedIndex":953833167,"createdIndex":953833167},{"key":"/_etcd/registry/fdd7d8ac203d2cac0c27ead148ad83ed/8e6ccfef42f98260","value":"c48b163558b61733c1aa44dccb712406=http://172.31.47.175:2380","modifiedIndex":953833339,"createdIndex":953833339}],"modifiedIndex":953831075,"createdIndex":953831075}}

To confirm the health of the cluster, SSH into one of the cluster nodes, and use fleetctl and etcdctl:

# ssh -i devops-admin-key core@euca-10-116-131-230.eucalyptus.c-05.autoqa.qa1.eucalyptus-systems.com
Last login: Sat Jan 2 23:53:25 2016 from 10.111.1.71
CoreOS beta (877.1.0)
core@euca-172-31-22-157 ~ $ fleetctl list-machines
MACHINE IP METADATA
33a32090... 10.116.131.107 purpose=coreos-cluster,region=euca-us-east-1
8981923b... 10.116.131.121 purpose=coreos-cluster,region=euca-us-east-1
c48b1635... 10.116.131.213 purpose=coreos-cluster,region=euca-us-east-1
e71b1fef... 10.116.131.230 purpose=coreos-cluster,region=euca-us-east-1
f047b9ff... 10.116.131.197 purpose=coreos-cluster,region=euca-us-east-1
core@euca-172-31-22-157 ~ $ etcd
etcd etcd2 etcdctl
core@euca-172-31-22-157 ~ $ etcdctl cluster-health
member d5c5d93e360ba87 is healthy: got healthy result from http://10.116.131.230:2379
member 12b6e6e78c9cb70c is healthy: got healthy result from http://10.116.131.107:2379
member 8e6ccfef42f98260 is healthy: got healthy result from http://10.116.131.213:2379
member cffd4985c990f872 is healthy: got healthy result from http://10.116.131.197:2379
member d0a4c6d73d0d8d17 is healthy: got healthy result from http://10.116.131.121:2379
cluster is healthy
core@euca-172-31-22-157 ~ $ etcdctl member list
d5c5d93e360ba87: name=e71b1fefcd65c43a0fbacc7103efbc2b peerURLs=http://172.31.22.157:2380 clientURLs=http://10.116.131.230:2379
12b6e6e78c9cb70c: name=33a3209006d2be1d5be0da6eaea007c5 peerURLs=http://172.31.19.215:2380 clientURLs=http://10.116.131.107:2379
8e6ccfef42f98260: name=c48b163558b61733c1aa44dccb712406 peerURLs=http://172.31.47.175:2380 clientURLs=http://10.116.131.213:2379
cffd4985c990f872: name=f047b9ff24f3d0c4e74c660709103b36 peerURLs=http://172.31.6.166:2380 clientURLs=http://10.116.131.197:2379
d0a4c6d73d0d8d17: name=8981923b54d7d7f46fabc527936a7dcf peerURLs=http://172.31.4.17:2380 clientURLs=http://10.116.131.121:2379

Thats it! The CoreOS cluster has been successfully deployed.  Given HPE Helion Eucalyptus’s AWS compatibility, this template can be used on AWS as well.

As always, please let me know if there are any questions.  Enjoy!

Updated CoreOS Cluster Cloudformation Template for HPE Helion Eucalyptus 4.2 VPC Deployments

Using Eucalyptus 4.0.1 CloudFormation to Deploy a CoreOS (Docker) Cluster

In a previous blog, I discussed how cloud-init can be used to customize a CoreOS image deployed as an instance on Eucalyptus – which happens to work in the same fashion on AWS.  This is a follow-up blog to demonstrate how to use Eucalyptus Cloudformation (which is in Tech Preview in Eucalyptus 4.0.0/4.0.1) to deploy a CoreOS cluster on Eucalyptus, customizing each instance using the cloud-config service.  This setup will allow cloud users to test out CoreOS clusters on Eucalyptus, just as CoreOS recommends on AWS EC2.

Prerequisites

Just as in the previous blog discussing the use of CoreOS, using Eucalyptus IAM is highly recommended.  In addition, to the prerequisites mentioned in that blog, the following service API actions need to be allowed (at a minimum) in the IAM policy for the user(s) that want to utilize this blog:

In addition to having the correct IAM policy actions authorized, the cloud user needs to be using the latest version of euca2ools with Eucalyptus 4.0.1.  Once these prerequisites are met, the Eucalyptus cloud needs to be prepared with the correct EMI for the deployment.

Adding CoreOS Image To Eucalyptus

In order to deploy an CoreOS cluster on Eucalyptus, the CoreOS image needs to be bundled, uploaded and registered.  To obtain the CoreOS image, download the image from the CoreOS Beta Release site. For example:

# wget -q http://beta.release.core-os.net/amd64-usr/current/coreos_production_ami_image.bin.bz2
 # bunzip2 -d coreos_production_ami_image.bin.bz2
 # qemu-img info coreos_production_ami_image.bin
 image: coreos_production_ami_image.bin
 file format: raw
 virtual size: 4.4G (4699717632 bytes)
 disk size: 4.4G

Once the image has been downloaded and user credentials have been sourced, use euca-install-image to bundle, upload and register the image as an instance store-backed HVM image to be used with the Cloudformation template. In addition, note the EC2_USER_ID value present in the eucarc file as it will be used with the Cloudformation template as well.

# euca-install-image -b coreos-production-ami -i coreos_production_ami_image.bin --virtualization-type hvm -n coreos-hvm -r x86_64
 ....
 /var/tmp/bundle-WsLdGB/coreos_production_ami_image.bin.part.19 100% |=================================================================| 6.08 MB 12.66 MB/s Time: 0:00:00
 /var/tmp/bundle-WsLdGB/coreos_production_ami_image.bin.manifest.xml 100% |============================================================| 6.28 kB 2.66 kB/s Time: 0:00:02
 IMAGE emi-DAB316FD

CoreOS etcd Discovery Service Token

CoreOS uses a service called etcd on each machine to handle coordination of services in a cluster.  To make sure the machines know that they are part of the same cluster, a discovery token needs to be generated and shared with each instance using the cloud-config service.  To generate a custom token, open a browser and go to the following URL:

https://discovery.etcd.io/new

The URL similar to the example below should show up in the browser:

https://discovery.etcd.io/7b67f765e2f264cf65b850a849a7da7e

Take note of the URL because it will be needed later.

Select VM Type and Availability Zone on Eucalyptus

Before deploying the CoreOS cluster on Eucalyptus, the user needs to determine the instance type, and the availability zone (Eucalyptus Cluster). In order to do this, use euca-describe-instance-types to show the instance types, availability zone(s), and the capacity for each instance type available in the availability zone(s).

# euca-describe-instance-types --show-capacity --by-zone
 AVAILABILITYZONE SirLuciousLeftFoot
 INSTANCETYPE Name CPUs Memory (MiB) Disk (GiB) Used / Total Used %
 INSTANCETYPE t1.micro 1 256 5 0 / 6 0%
 INSTANCETYPE m1.small 1 512 10 0 / 6 0%
 INSTANCETYPE m1.medium 1 1024 10 0 / 6 0%
 INSTANCETYPE c1.xlarge 2 2048 10 0 / 3 0%
 INSTANCETYPE m1.large 2 1024 15 0 / 3 0%
 INSTANCETYPE c1.medium 1 1024 20 0 / 6 0%
 INSTANCETYPE m1.xlarge 2 1024 30 0 / 3 0%
 INSTANCETYPE m2.2xlarge 2 4096 30 0 / 3 0%
 INSTANCETYPE m3.2xlarge 4 4096 30 0 / 1 0%
 INSTANCETYPE m2.xlarge 2 2048 40 0 / 3 0%
 INSTANCETYPE m3.xlarge 2 2048 50 0 / 3 0%
 INSTANCETYPE cc1.4xlarge 8 3072 60 0 / 0
 INSTANCETYPE m2.4xlarge 8 4096 60 0 / 0
 INSTANCETYPE hi1.4xlarge 8 6144 120 0 / 0
 INSTANCETYPE cc2.8xlarge 16 6144 120 0 / 0
 INSTANCETYPE cg1.4xlarge 16 12288 200 0 / 0
 INSTANCETYPE cr1.8xlarge 16 16384 240 0 / 0
 INSTANCETYPE hs1.8xlarge 48 119808 24000 0 / 0
AVAILABILITYZONE ViciousLiesAndDangerousRumors
 INSTANCETYPE Name CPUs Memory (MiB) Disk (GiB) Used / Total Used %
 INSTANCETYPE t1.micro 1 256 5 4 / 12 33%
 INSTANCETYPE m1.small 1 512 10 4 / 12 33%
 INSTANCETYPE m1.medium 1 1024 10 4 / 12 33%
 INSTANCETYPE c1.xlarge 2 2048 10 2 / 6 33%
 INSTANCETYPE m1.large 2 1024 15 2 / 6 33%
 INSTANCETYPE c1.medium 1 1024 20 4 / 12 33%
 INSTANCETYPE m1.xlarge 2 1024 30 2 / 6 33%
 INSTANCETYPE m2.2xlarge 2 4096 30 0 / 2 0%
 INSTANCETYPE m3.2xlarge 4 4096 30 0 / 2 0%
 INSTANCETYPE m2.xlarge 2 2048 40 2 / 6 33%
 INSTANCETYPE m3.xlarge 2 2048 50 2 / 6 33%
 INSTANCETYPE cc1.4xlarge 8 3072 60 0 / 0
 INSTANCETYPE m2.4xlarge 8 4096 60 0 / 0
 INSTANCETYPE hi1.4xlarge 8 6144 120 0 / 0
 INSTANCETYPE cc2.8xlarge 16 6144 120 0 / 0
 INSTANCETYPE cg1.4xlarge 16 12288 200 0 / 0
 INSTANCETYPE cr1.8xlarge 16 16384 240 0 / 0
 INSTANCETYPE hs1.8xlarge 48 119808 24000 0 / 0

For this blog, the availability zone ‘ViciousLiesAndDangerousRumors’ and the instance type ‘c1.medium’ will be used as a parameter for the Cloudformation template.  Now, Eucalyptus Cloudformation is ready to be used.

Deploying the CoreOS Cluster

Final Preparations

Before using the Cloudformation template for the CoreOS cluster, a keypair needs to be created.  This keypair will also be used as a parameter for the Cloudformation template.

To obtain the template, download the template from coreos-cloudformation-template bucket on AWS S3.  Once the file has been downloaded, the following edits need to happen.

The first edit is to define the ‘AvailabilityZones’ in the ‘Properties’ section of the ‘CoreOsGroup’ resource.  For example, ‘ViciousLiesAndDangerousRumors’ has been placed as the value for ‘AvailabilityZones’:

"CoreOsGroup" : {
 "Type" : "AWS::AutoScaling::AutoScalingGroup",
 "Properties" : {
 "AvailabilityZones" : [ "ViciousLiesAndDangerousRumors" ],
 "LaunchConfigurationName" : { "Ref" : "CoreOsLaunchConfig" },
 "MinSize" : { "Ref" : "ClusterSize" },
 "MaxSize" : { "Ref" : "ClusterSize" }
 }
 },

The second and final edit, is to update the ‘UserData’ property to have the correct value for the discovery token that was provided earlier in this blog.  For example:

"UserData" : { "Fn::Base64" : { "Fn::Join" : ["",[
 "#cloud-config","\n",
 "coreos:","\n",
 " etcd:","\n",
 " discovery: https://discovery.etcd.io/7b67f765e2f264cf65b850a849a7da7e","\n",
 " addr: $private_ipv4:4001","\n",
 " peer-addr: $private_ipv4:7001","\n",
 " units:","\n",

Now that these values have been updated, the CoreOS cluster can be deployed.

Create the Stack

To deploy the cluster, use euform-create-stack with the parameter values filled in appropriately.  For example:

# euform-create-stack --template-file cfn-coreos-as.json --parameter "CoreOSImageId=emi-DAB316FD" --parameter "UserKeyPair=account1-user01" --parameter "AcctId=408396244283" --parameter "ClusterSize=3" --parameter "VmType=c1.medium" CoreOSClusterStack
 arn:aws:cloudformation:bigboi:408396244283:stack/CoreOSClusterStack/43d53adb-68f2-4317-bd2b-3da661977ebc

The ‘ClusterSize’ parameter is completely dependent upon how big of a CoreOS cluster the user would like to have based upon the instance types supported on the Eucalyptus cloud.  Please refer to the CoreOS documentation regarding optimal cluster sizes to see what would best suit the use case of the cluster.

Check Out The Stack Resources

After deploying the Cloudformation stack, after a few minutes, use euform-describe-stacks to check the status of the stack. The status of the stack should return with CREATE_COMPLETE.

# euform-describe-stacks
 STACK CoreOSClusterStack CREATE_COMPLETE Complete! Deploy CoreOS Cluster 2014-08-28T22:31:02.669Z
 OUTPUT AutoScalingGroup CoreOSClusterStack-CoreOsGroup-G7Y7YVWI4DOPG

To check out the resources associated with the Cloudformation stack, use euform-describe-stack-resources:

# euform-describe-stack-resources -n CoreOSClusterStack --region account1-user01@
 RESOURCE CoreOsSecurityGroupIngress2 CoreOsSecurityGroupIngress2 AWS::EC2::SecurityGroupIngress CREATE_COMPLETE
 RESOURCE CoreOsLaunchConfig CoreOSClusterStack-CoreOsLaunchConfig-FFSTY76SDQAWB AWS::AutoScaling::LaunchConfiguration CREATE_COMPLETE
 RESOURCE CoreOsSecurityGroup CoreOSClusterStack-CoreOsSecurityGroup-D3WCUH0SKHYVC AWS::EC2::SecurityGroup CREATE_COMPLETE
 RESOURCE CoreOsSecurityGroupIngress1 CoreOsSecurityGroupIngress1 AWS::EC2::SecurityGroupIngress CREATE_COMPLETE
 RESOURCE CoreOsGroup CoreOSClusterStack-CoreOsGroup-G7Y7YVWI4DOPG AWS::AutoScaling::AutoScalingGroup CREATE_COMPLETE

Check the status of the instances by using the value returned for ‘AutoScalingGroup’ from the euform-describe-stacks output:

# euscale-describe-auto-scaling-groups CoreOSClusterStack-CoreOsGroup-G7Y7YVWI4DOPG --region account1-user01@
 AUTO-SCALING-GROUP CoreOSClusterStack-CoreOsGroup-G7Y7YVWI4DOPG CoreOSClusterStack-CoreOsLaunchConfig-FFSTY76SDQAWB ViciousLiesAndDangerousRumors 3 33 Default
 INSTANCE i-E6FB62D0 ViciousLiesAndDangerousRumors InService Healthy CoreOSClusterStack-CoreOsLaunchConfig-FFSTY76SDQAWB
 INSTANCE i-2AC4CC35 ViciousLiesAndDangerousRumors InService Healthy CoreOSClusterStack-CoreOsLaunchConfig-FFSTY76SDQAWB
 INSTANCE i-442C4692 ViciousLiesAndDangerousRumors InService Healthy CoreOSClusterStack-CoreOsLaunchConfig-FFSTY76SDQAWB

Check the Status of the CoreOS Cluster

In order to check the status of the CoreOS cluster, SSH into one of the instances (the port was opened in the security group as part of the Cloudformation template), and use the fleetctl command:

# euca-describe-instances i-E6FB62D0 i-2AC4CC35 i-442C4692 --region account1-user01@
 RESERVATION r-AF98046C 408396244283 CoreOSClusterStack-CoreOsSecurityGroup-D3WCUH0SKHYVC
 INSTANCE i-2AC4CC35 emi-DAB316FD euca-10-104-6-233.bigboi.acme.eucalyptus-systems.com euca-172-18-223-111.bigboi.internal running account1-user01 0 c1.medium 2014-08-28T22:15:48.043Z ViciousLiesAndDangerousRumors monitoring-enabled 10.104.6.233 172.18.223.111 instance-store hvm d88cac3d-ce92-4c3b-98ee-7e507afc26cb_ViciousLiesAndDangerousR_1 sg-31503C69 x86_64
 TAG instance i-2AC4CC35 aws:autoscaling:groupName CoreOSClusterStack-CoreOsGroup-G7Y7YVWI4DOPG
 RESERVATION r-A24611A2 408396244283 CoreOSClusterStack-CoreOsSecurityGroup-D3WCUH0SKHYVC
 INSTANCE i-442C4692 emi-DAB316FD euca-10-104-6-235.bigboi.acme.eucalyptus-systems.com euca-172-18-223-227.bigboi.internal running account1-user01 0 c1.medium 2014-08-28T22:15:48.056Z ViciousLiesAndDangerousRumors monitoring-enabled 10.104.6.235 172.18.223.227 instance-store hvm 1281a747-69a7-4f26-8fe2-2dea6b8b858d_ViciousLiesAndDangerousR_1 sg-31503C69 x86_64
 TAG instance i-442C4692 aws:autoscaling:groupName CoreOSClusterStack-CoreOsGroup-G7Y7YVWI4DOPG
 RESERVATION r-089053BE 408396244283 CoreOSClusterStack-CoreOsSecurityGroup-D3WCUH0SKHYVC
 INSTANCE i-E6FB62D0 emi-DAB316FD euca-10-104-6-232.bigboi.acme.eucalyptus-systems.com euca-172-18-223-222.bigboi.internal running account1-user01 0 c1.medium 2014-08-28T22:15:38.146Z ViciousLiesAndDangerousRumors monitoring-enabled 10.104.6.232 172.18.223.222 instance-store hvm c0dc6cca-5fa3-4614-a4ec-8a902bf6ff66_ViciousLiesAndDangerousR_1 sg-31503C69 x86_64
 TAG instance i-E6FB62D0 aws:autoscaling:groupName CoreOSClusterStack-CoreOsGroup-G7Y7YVWI4DOPG
# ssh -i account1-user01/account1-user01.priv core@euca-10-104-6-232.bigboi.acme.eucalyptus-systems.com
 Last login: Thu Aug 28 15:32:34 2014 from 10.104.10.55
 CoreOS (beta)
 core@euca-172-18-223-222 ~ $ fleetctl list-machines -full=true
 MACHINE IP METADATA
 6f4e3de463490a7644e3d7c80d826770 172.18.223.227 -
 929c1f121860c63b506c0b951c19de7b 172.18.223.222 -
 a08155346fb55f9b53b154d6447af0fa 172.18.223.211 -
 core@euca-172-18-223-222 ~ $

The cluster status can also be checked by going to the discovery token URL that was placed in the Cloudformation template.

CoreOS etcd discovery cluster listing

Conclusion

Just as on AWS, Cloudformation can be used to deploy a CoreOS cluster on Eucalyptus.  Users will be able to test out different use cases, such as Cluster-Level Container Development with fleet, or get more familiar with CoreOS by going through the CoreOS documentation.  As always, feel free to ask any questions.  Feedback is always welcome.

Enjoy!

Using Eucalyptus 4.0.1 CloudFormation to Deploy a CoreOS (Docker) Cluster

12 Steps To EBS-Backed EMI Bliss on Eucalyptus

In previous posts, I shared how to use Ubuntu Cloud Images and eustore with Eucalyptus and AWS.  This blog entry will focus on how to use these assets to create EBS-backed EMIs in 12 steps.   These steps can be used on AWS as well, but instead of creating an instance store-backed AMI first, Ubuntu has already provided AMIs that can be used as the building block instance on AWS.  Let’s get started.

Prerequisites

On Eucalyptus and AWS, it is required the user has the appropriate IAM policy in order to perform these steps.  The policy should contain the following EC2 Actions at a minimum:

  • RunInstances
  • AttachVolume
  • AuthorizeSecurityGroupEgress
  • AuthorizeSecurityGroupIngress
  • CreateKeyPair
  • CreateSnapshot
  • CreateVolume
  • DescribeImages
  • DescribeInstances
  • DescribeInstanceStatus
  • DescribeSnapshots
  • DetachVolume
  • RegisterImage

In addition, the user needs an access key ID and secret key.  For more information, check out the following resources:

This entry also assumes Eucalyptus euca2ools are installed on the client machine.

The 12 Steps

Although the Ubuntu Cloud Image used in this entry is Ubuntu Precise (12.04) LTS, any of of the maintained Ubuntu Cloud images can be used.

  1. Use wget to download tar-gzipped precise-server-cloudimg:
    $ wget http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64.tar.gz
  2. After setting the EC2_ACCESS_KEY, EC2_SECRET_KEY, and EC2_URL, use eustore-install-image to an instance stored-backed EMI:
    $ eustore-install-image -t precise-server-cloudimg-amd64.tar.gz \
    -b ubuntu-latest-precise-x86_64 --hypervisor universal \
    -s "Ubuntu Cloud Image - Precise Pangolin - 12.04 LTS"
  3. Create a keypair using euca-create-keypair, then use euca-run-instances to launch an instance from the EMI returned from eustore-install-image. For example:
    $ euca-run-instances -t m1.medium \
    -k account1-user01 emi-5C8C3909
  4. Use euca-create-volume to create a volume based upon the size of how big you want the root filesystem to be.  The availability zone (-z option) will be based on if you are using Eucalyptus or AWS:
    $ euca-create-volume -s 6 \
    -z LayinDaSmackDown
  5. Using euca-attach-volume, attach the resulting volume to the running instance. For example:
    $ euca-attach-volume -d /dev/vdd \
    -i i-839E3FB0 vol-B5863B3B
  6. Use euca-authorize to open SSH access to the instance, SSH into the instance, then use wget to download the Ubuntu Precise Cloud Image (qcow2 format):
    $ ssh -i account1-user01.priv ubuntu@euca-10-104-7-10.eucalyptus.euca-hasp.eucalyptus-systems.com
    # sudo -s
    # wget http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img
  7. Install qemu-utils:
    # apt-get install -y qemu-utils
  8. Use qemu-img to convert image from qcow2 to raw:
    # qemu-img convert \
    -O raw precise-server-cloudimg-amd64-disk1.img precise-server-cloudimg-amd64-disk1-raw.img
  9. dd raw image to block device where volume is attached (use dmesg to figure that out easily):
    # dmesg | tail
    [ 7026.943212] virtio-pci 0000:00:05.0: using default PCI settings
    [ 7026.943249] pci 0000:00:07.0: no hotplug settings from platform
    [ 7026.943251] pci 0000:00:07.0: using default PCI settings
    [ 7026.945964] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)
    [ 7026.955143] virtio-pci 0000:00:07.0: PCI INT A -> Link[LNKC] -> GSI 10 (level, high) -> IRQ 10
    [ 7026.955180] virtio-pci 0000:00:07.0: setting latency timer to 64
    [ 7026.955429] virtio-pci 0000:00:07.0: irq 45 for MSI/MSI-X
    [ 7026.955456] virtio-pci 0000:00:07.0: irq 46 for MSI/MSI-X
    [ 7026.986990] vdb: unknown partition table
    [10447.093426] virtio-pci 0000:00:07.0: PCI INT A disabled
    # dd if=/mnt/precise-server-cloudimg-amd64-disk1-raw.img of=/dev/vdb bs=1M
  10. Log out the instance, and use euca-detach-volume to detach the volume:
    $ euca-detach-volume vol-B5863B3B
  11. Use euca-create-snapshot to create a snapshot of the volume:
    $ euca-create-snapshot vol-B5863B3B
  12. Use euca-register to register the resulting snapshot to create the EBS-backed EMI:
    $ euca-register --name ebs-precise-x86_64-sda \
    --snapshot snap-EFDB40A1 --root-device-name /dev/sda

Thats it!  You have successfully created an EBS-backed EMI/AMI.  As mentioned earlier, these steps can be used on AWS just as well (just skip steps 1 & 2, and use one of the Ubuntu Cloud Images in the AWS region of your choice).  Enjoy!

12 Steps To EBS-Backed EMI Bliss on Eucalyptus

Getting Started with EucaLobo

Nice tool for AWS/Eucalyptus hybrid utilization.

Testing Clouds at 128bpm

Initial Setup

In my previous post, I described the story behind EucaLobo, a graphical interface for managing workloads on AWS and Eucalyptus clouds through a <cliche>single pane of glass</cliche>. The tool is built using Javascript and the XUL framework allowing it to be used on Linux, Windows, and Mac for the following APIs:

  • EC2
  • EBS
  • S3
  • IAM
  • CloudWatch
  • AutoScaling
  • Elastic Load Balancing

To get started download the binary for your platform:

Once installation is complete and EucaLobo starts for the first time you will be prompted to enter an endpoint. My esteemed colleague Tony Beckham has created a great intro video showing how to create and edit credentials and endpoints. The default values have been set to the Eucalyptus Community Cloud, a free and easy way to get started using Eucalyptus and clouds in general. This is a great resource for users who want to get a…

View original post 653 more words

Getting Started with EucaLobo

Demo of Eucalyptus hotness: 3.3 milestone 6

Eucalyptus 3.3 Demo: More AWS Compatibility Goodness..:-)

Greg DeKoenigsberg Speaks

Our demo day for milestone 6 was yesterday, and it was choice. We’re at feature completeness at this point, and we’re now on final approach for release sometime Soon-ish, as soon as we shake out all the code nasties. We’ve got some good stuff to show off on Vimeo. The basic transcript:

  • 0:00 Eric Choi, Product Mktg Manager, with agenda/housekeeping.
  • 1:25 Tim Cramer, VP of Engineering, sets the table for demos.
  • 3:45 Yours Truly, VP of Community, talks (a little too long) about Eucalyptus compatibility with the AWS Ruby SDK.
  • 16:00 Vic Iglesias, QA Lead, talks about EucaLobo, an amazing fork of ElasticWolf that provides an autoscaling UI for both Eucalyptus and AWS. (Watch this one twice; it’s really the star of the show.)
  • 24:40 Vic Iglesias talks about Micro-QA, the self-contained environment for running automated testing on Eucalyptus installations.
  • 31:30 Chris Grzegorczyk, Co-founder and Chief Architect, shows Asgard…

View original post 59 more words

Demo of Eucalyptus hotness: 3.3 milestone 6

What’s new in Ansible 1.1 for AWS and Eucalyptus users?

Ansible providing more AWS/Eucalyptus support!

Take that to the bank and cash it!

I thought the Ansible 1.0 development cycle was busy but 1.1 is crammed full of orchestration goodness.  On Tuesday, 1.1 was released and you can read more about it here: http://blog.ansibleworks.com/2013/04/02/ansible-1-1-released/

For those working on AWS and Eucalyptus, 1.1 brings some nice module improvements as well as a new cloudformation and s3 module.  It’s great to see the AWS-related modules becoming so popular so quickly.  Here are some more details about the changes but you can find info in the changelog here: https://github.com/ansible/ansible/blob/devel/CHANGELOG.md

Security group ID support

It’s now possible to specify the security group by its ID.  This is quite typically behaviour in EC2 and Eucalyptus will support this with the pending 3.3 release.  The parameter is optional.

VPC subnet ID

VPC users can now specify a subnet ID associated with their instance.

Instance state wait timeout

In 1.0 there was no way to specify how long to wait…

View original post 300 more words

What’s new in Ansible 1.1 for AWS and Eucalyptus users?

UCSB Sweethearts of Cloud Computing: AppScale and Eucalyptus

AppScale and Eucalyptus: Great Tag Team for On-Premise Cloud Computing

shaon's blog

Cloud computing is the use of computing resources (hardware and software) that are delivered as a service over a network (typically the Internet). – Wikipedia

According to Wikipedia currently there are few popular service models exist.

1. Infrastructure as a service (IaaS)
2. Platform as a service (PaaS)
3. Software as a service (SaaS)

So, I have an Eucalyptus cloud, which is great, serves as AWS-like IaaS platform. But now I want PaaS. And right here Appscale comes into play with full compatibility of Google App Engine (GAE) applications. In this post, we will install the popular open source PaaS framework Appscale on Eucalyptus, the AWS compatible open source IaaS platform.

Agenda
0. Introduction
1. Resize Lucid image
2. Install Appscale from source
3. Install Appscale Tool
4. Bundle Appscale image
5. Run Appscale
6. Run an application on Appscale

Eucalyptus
Eucalyptus Cloud platform is open source software for building…

View original post 541 more words

UCSB Sweethearts of Cloud Computing: AppScale and Eucalyptus