Using Boto’s connect_to_region function with Eucalyptus 4.1

Typically, when documenting how to use the Python interface boto with Eucalyptus clouds, the function connect_<service> is always leveraged (<service> being ec2, s3, iam, sts, elb, cloudformation, cloudwatch, or autoscale).  The focus of this blog entry is to demonstrate how to use the connect_to_region function associated with each AWS service supported by Eucalyptus.  This will be very important when using boto with federated Eucalyptus clouds, which will be available in the release of HP Helion Eucalyptus 4.2.0.

Prerequisites

To get started, the following requirements need to be met:

The Setup

Once the prerequisites are met, now we can focus on setting up the configuration file for boto.  Below is an example of a boto configuration file:

[Credentials]
aws_access_key_id = <Eucalyptus Access Key ID>
aws_secret_access_key = <Eucalyptus Secret Access Key>
[Boto]
is_secure = False
endpoints_path = /root/boto-qa-setup1-endpoints.json

Notice the file boto-qa-setup1-endpoints.json.  This is a JSON file that contains the Eucalyptus service API endpoints correlating to the Eucalyptus cloud that would like to be accessed.  Here are the contents of the boto-qa-setup1-endpoints.json file:

{
 "autoscaling": {
 "eucalyptus": "autoscaling.h-01.autoqa.qa1.eucalyptus-systems.com"
 },
 "cloudformation": {
 "eucalyptus": "cloudformation.h-01.autoqa.qa1.eucalyptus-systems.com"
 },
 "cloudwatch": {
 "eucalyptus": "cloudwatch.h-01.autoqa.qa1.eucalyptus-systems.com"
 },
 "ec2": {
 "eucalyptus": "compute.h-01.autoqa.qa1.eucalyptus-systems.com"
 },
 "elasticloadbalancing": {
 "eucalyptus": "loadbalancing.h-01.autoqa.qa1.eucalyptus-systems.com"
 },
 "iam": {
 "eucalyptus": "euare.h-01.autoqa.qa1.eucalyptus-systems.com"
 },
 "s3": {
 "eucalyptus": "objectstorage.h-01.autoqa.qa1.eucalyptus-systems.com"
 },
 "sts": {
 "eucalyptus": "tokens.h-01.autoqa.qa1.eucalyptus-systems.com"
 },
 "swf": {
 "eucalyptus": "simpleworkflow.h-01.autoqa.qa1.eucalyptus-systems.com"
 }
}

As you can see, each AWS service implemented by Eucalyptus is defined in this file.  With the boto configuration file and endpoints json file defined, the connect_to_region function in boto can be easily utilized.

Example

To show how this setup works, ipython will be used as the demonstration environment.  Below is an example that shows how to use the connect_to_region function with the Compute service (EC2) against a Eucalyptus 4.1 cloud.

# ipython
Python 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
Type "copyright", "credits" or "license" for more information.
IPython 0.13.2 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.
In [1]: import boto.ec2
In [2]: ec2_connection = boto.ec2.connect_to_region('eucalyptus', port=8773)
In [3]: ec2_connection.get_all_instance_types()
Out[3]:
[InstanceType:m1.small-1,256,5,
 InstanceType:t1.micro-1,256,5,
 InstanceType:m1.medium-1,512,10,
 InstanceType:c1.medium-2,512,10,
 InstanceType:m1.large-2,512,10,
 InstanceType:m1.xlarge-2,1024,10,
 InstanceType:c1.xlarge-2,2048,10,
 InstanceType:m2.xlarge-2,2048,10,
 InstanceType:m3.xlarge-4,2048,15,
 InstanceType:m2.2xlarge-2,4096,30,
 InstanceType:m3.2xlarge-4,4096,30,
 InstanceType:cc1.4xlarge-8,3072,60,
 InstanceType:m2.4xlarge-8,4096,60,
 InstanceType:hi1.4xlarge-8,6144,120,
 InstanceType:cc2.8xlarge-16,6144,120,
 InstanceType:cg1.4xlarge-16,12288,200,
 InstanceType:cr1.8xlarge-16,16384,240,
 InstanceType:hs1.8xlarge-48,119808,24000]

Here is an example of describing the volumes and snapshots on a given Eucalyptus cloud:

In [3]: ec2_connection.get_all_volumes()
Out[3]:
[Volume:vol-563802cd,
 Volume:vol-274c3628,
 Volume:vol-7074a3a8,
 Volume:vol-172cdb42,
 Volume:vol-d00e53e9,
 Volume:vol-4f370899]
In [4]: ec2_connection.get_all_snapshots()
Out[4]:
[Snapshot:snap-6d874d5a,
 Snapshot:snap-e41c6adc,
 Snapshot:snap-63700417,
 Snapshot:snap-6055a378,
 Snapshot:snap-91d70d6b,
 Snapshot:snap-7b39eca8,
 Snapshot:snap-80a4f3e2]

Just like the EC2 tutorial by boto demonstrates how to use connect_to_region with AWS EC2, this function can also be used against Eucalyptus 4.1 clouds as well.  As mentioned earlier, when region support (multiple Eucalyptus clouds) becomes available in Eucalyptus 4.2, this function will be very useful.

Enjoy!

Using Boto’s connect_to_region function with Eucalyptus 4.1

Using AWS CodeDeploy with Eucalyptus Cloudformation for On-Premise Application Deployments

Background

Recently, Amazon Web Services (AWS) announced that their CodeDeploy service supports on-premise instances.  This is extremely valuable – especially for developers and administrators to allow utilization of existing on-premise resources.

For teams who are using HP Helion Eucalyptus 4.1 (or who want to use Eucalyptus), this is even better news.  This feature – along with HP Helion Eucalyptus 4.1 Cloudformation – developers can deploy applications within a private cloud environment of HP Helion Eucalyptus.  This makes it even easier for developers and administrators to separate out and maintain production (AWS) versus development (HP Helion Eucalyptus) environments (or vice versa).  In addition, since HP Helion Eucalyptus strives for AWS compatibility, the Cloudformation templates used on Eucalyptus, can be used with AWS – with just a couple of modifications.

The Setup

To leverage on-premise instances with AWS CodeDeploy, please reference the AWS documentation entitled “Configure Existing On-Premises Instances by Using AWS CodeDeploy“.  To use these steps with an HP Helion Eucalyptus cloud, a slight change had to be done to the AWS CLI tools.  When using the ‘aws deploy register’ command, AWS CLI checks to see if the instance is running on an AWS environment by confirm if the instance metadata is present.  For on-premise cloud environments that provide the same service, this will cause the on-premise instance registration to fail.  To resolve this issue, I updated the AWS CLI tools with a patch that checks the instance metadata variable ‘AMI ID’ – which on AWS will begin with ‘ami’.  All images on Eucalyptus start with ’emi’ (i.e. Eucalyptus Machine Images).  With this patch, on-premise instance registration completes without a problem.

In addition to the patch, the following is needed on HP Helion Eucalyptus 4.1 cloud environments:

  1. Ubuntu Server 14.04 LTS EMI (EBS-backed or Instance Store-Backed)
  2. Eucalyptus IAM access policy actions that allow the user to use CloudFormations, AutoScaling and EC2 actions.  (Along with the Eucalyptus documentation, reference the AWS IAM documentation as well.)

Once these requirements have been met on the HP Helion Eucalyptus 4.1 environment, developers can use their AWS credentials in the Eucalyptus Cloudformation templates to leverage the on-premise instances with AWS CodeDeploy.

Using Eucalyptus Cloudformation For Instance Deployment

To help get started, I provided the following example Cloudformation templates:

Each template has specific parameters that need values.  The key parameters are the following:

  • UserKeyPair -> Eucalyptus EC2 Key Pair
  • UbuntuImageId -> Ubuntu 14.04 Cloud Image (EMI)
  • SSHLocation -> IP address range that can SSH into the Eucalyptus instances

Once there are values for these parameters, the Cloudformation templates can be utilized to deploy the on-premise instances.

Configure Existing On-Premises Instances by Using AWS CodeDeploy

After the AWS IAM prerequisites have been met for AWS CodeDeploy, use the example Cloudformation templates with HP Helion Eucalyptus.  Below is an example output of both templates being used on a given HP Helion Eucalyptus 4.1 cloud:

# euform-describe-stacks --region account2-admin@eucalyptus-cloud
STACK UbuntuCodeDeployTest CREATE_COMPLETE Complete! Eucalyptus Cloudformation Example => Deploy an instance that is configured and registered as an on-premise instance with AWS CodeDeploy 2015-04-14T02:42:01.325Z
PARAMETER UbuntuImageId emi-759e12a3
PARAMETER UserKeyPair account2-admin
OUTPUT InstanceId i-df9af6f5
OUTPUT AZ thugmotivation101
OUTPUT PublicIP 10.111.75.103
STACK UbuntuCodeDeployAutoScalingTest CREATE_COMPLETE Complete! Eucalyptus CloudFormation Sample Template AutoScaling-Single AZ for AWS CodeDeploy on-premise instances. The autoscaling group is configured to span in one availability zone (one cluster) and is Auto-Scaled based on the CPU utilization of the servers. In addition, each instance will be registered as an on-premise instance with AWS CodeDeploy. Please refer to http://docs.aws.amazon.com/codedeploy/latest/userguide/how-to-configure-on-premises-host.html for additional information. 2015-04-14T02:41:44.733Z
PARAMETER InstanceType m1.xlarge
PARAMETER UbuntuImageId emi-759e12a3
PARAMETER UserKeyPair account2-admin
PARAMETER MinSize 2
PARAMETER MaxSize 4
PARAMETER Zone theinspiration
OUTPUT AutoScalingGroup UbuntuCodeDeployAutoScalingTest-ServerGroup-211FTERKLII6T

Since both Eucalyptus Cloudformation stacks have successfully deployed, let’s check out the instances:

# euca-describe-instances --region account2-admin@eucalyptus-cloud
RESERVATION r-feeb1023 968367465792 UbuntuCodeDeployTest-CodeDeploySecurityGroup-HP5L5HRU3WI98
INSTANCE i-df9af6f5 emi-759e12a3 euca-10-111-75-103.eucalyptus.a-35.autoqa.qa1.eucalyptus-systems.com euca-10-111-75-107.eucalyptus.internal running account2-admin 0 m1.xlarge 2015-04-14T02:42:11.346Z thugmotivation101 monitoring-disabled 10.111.75.103 10.111.75.107 instance-store hvm sg-422ed69a x86_64
TAG instance i-df9af6f5 aws:cloudformation:logical-id CodeDeployInstance
TAG instance i-df9af6f5 aws:cloudformation:stack-id arn:aws:cloudformation::968367465792:stack/UbuntuCodeDeployTest/b210c81a-7e34-476f-9c59-7ea69ac9647a
TAG instance i-df9af6f5 aws:cloudformation:stack-name UbuntuCodeDeployTest
RESERVATION r-10df526e 968367465792 UbuntuCodeDeployAutoScalingTest-InstanceSecurityGroup-B2OVH0XWAFN5S
INSTANCE i-9b2b14e3 emi-759e12a3 euca-10-111-75-97.eucalyptus.a-35.autoqa.qa1.eucalyptus-systems.com euca-10-111-75-106.eucalyptus.internal running account2-admin 0 m1.xlarge 2015-04-14T02:42:05.939Z theinspiration monitoring-enabled 10.111.75.97 10.111.75.106 instance-store hvm d739a9eb-ba3c-4f16-940c-366a516cebfe_theinspiration_1 sg-556b10ce x86_64
TAG instance i-9b2b14e3 Name UbuntuCodeDeployAutoScalingTest
TAG instance i-9b2b14e3 aws:autoscaling:groupName UbuntuCodeDeployAutoScalingTest-ServerGroup-211FTERKLII6T
TAG instance i-9b2b14e3 aws:cloudformation:logical-id ServerGroup
TAG instance i-9b2b14e3 aws:cloudformation:stack-id arn:aws:cloudformation::968367465792:stack/UbuntuCodeDeployAutoScalingTest/2a5aefc6-c5c3-41e8-a9b4-a9ca095c1696
TAG instance i-9b2b14e3 aws:cloudformation:stack-name UbuntuCodeDeployAutoScalingTest
RESERVATION r-6c8a9642 968367465792 UbuntuCodeDeployAutoScalingTest-InstanceSecurityGroup-B2OVH0XWAFN5S
INSTANCE i-12f1a3a3 emi-759e12a3 euca-10-111-75-101.eucalyptus.a-35.autoqa.qa1.eucalyptus-systems.com euca-10-111-75-111.eucalyptus.internal running account2-admin 0 m1.xlarge 2015-04-14T02:42:05.872Z theinspiration monitoring-enabled 10.111.75.101 10.111.75.111 instance-store hvm 16a61ee7-d143-4f08-b926-c711ce335a1a_theinspiration_1 sg-556b10ce x86_64
TAG instance i-12f1a3a3 Name UbuntuCodeDeployAutoScalingTest
TAG instance i-12f1a3a3 aws:autoscaling:groupName UbuntuCodeDeployAutoScalingTest-ServerGroup-211FTERKLII6T
TAG instance i-12f1a3a3 aws:cloudformation:logical-id ServerGroup
TAG instance i-12f1a3a3 aws:cloudformation:stack-id arn:aws:cloudformation::968367465792:stack/UbuntuCodeDeployAutoScalingTest/2a5aefc6-c5c3-41e8-a9b4-a9ca095c1696
TAG instance i-12f1a3a3 aws:cloudformation:stack-name UbuntuCodeDeployAutoScalingTest

As we can see above, the Eucalyptus Cloudformation instances are tagged just as if they were running on AWS – again demonstrating the AWS compatibility desired by HP Helion Eucalyptus.

Now, look in the AWS Management Console, under the AWS CodeDeploy service.  In the dropbox under ‘AWS CodeDeploy’, select ‘On-Premise Instances’:

Displaying the dropdown box options under the AWS CodeDeploy title
Displaying the dropdown box options under AWS CodeDeploy

Once that has been selected, the on-premise instances running on HP Helion Eucalyptus should show up as ‘Registered’:

Display of Registered On-Premise Instances for AWS CodeDeploy
Display of Registered On-Premise Instances for AWS CodeDeploy

Now developers can proceed with remaining steps of using AWS CodeDeploy to do an application deployment.

Conclusion

As demonstrated, the new feature in AWS CodeDeploy allows developers to gain a true sense of a hybrid cloud environment.  This feature – along with HP Helion Eucalyptus’s AWS compatibility – makes it easy for developers and administrators to use the same toolset to deploy, manage and maintain both public and private cloud environments.  Don’t forget – using AWS CodeDeploy with on-premise instances does have an AWS pricing cost associated with it.  Check out AWS CodeDeploy Pricing for more details.

Enjoy!

Using AWS CodeDeploy with Eucalyptus Cloudformation for On-Premise Application Deployments

Eucalyptus and CEPH for Elastic Block Storage

Originally posted on A sysadmin born in the cloud:

Today I did my first install of CEPH, which I used as backend system for the Elastic Block Storage (EBS) in Eucalyptus. The advantage of CEPH is that it is a distributed system which is going to provide you replication (persistence for data), redundancy (we are using a pool of resources, not a single target) and scalability : the easiest way to add capacity is to add nodes which will be used for storage, and those nodes can be simple machines.

I won’t go too deep into CEPH installation. The official documentation is quite good for anyone to get up to speed quickly. I personally had no troubles using CentOS7 (el7).
Also, I won’t go too deep into Eucalyptus installation, I will simply share with you the CEPH config files and my NC configuration files which have some values non-indicated in Eucalyptus docs.

I will simply spend some time to configure a…

View original 321 more words

Eucalyptus and CEPH for Elastic Block Storage

Deploying the Eucalyptus Management Console on Eucalyptus

hspencer77:

More Eucalyptus Cloudformation goodness..this time discussing how to deploy Eucalyptus Management Console. Solid work here!

Originally posted on Coders Like Us:

The Eucalyptus Management Console can be deployed in a variety of ways, but we’d obviously like it to be scalable, highly available and responsive. Last summer, I wrote up the details of deploying the console with Auto Scaling coupled with Elastic Load Balancing. The Cloud Formations service ties this all together by putting all of the details of how to use these services together in one template. This post will describe an example of how you can do this which works well on Eucalyptus (and AWS) and may guide you with your own application as well.

Let’s tackle a fairly simple deployment for the first round. For now, we’ll setup a LaunchConfig, AS group and ELB. We’ll also set up a security group for the AS group and allow access only to the ELB. Finally, we’ll set up a self signed SSL cert for the console. In another post, we’ll add…

View original 375 more words

Deploying the Eucalyptus Management Console on Eucalyptus

EDGE Networking in Eucalyptus

Originally posted on A sysadmin born in the cloud:

We are just about to have Eucalyptus 4.1 released with VPC implementations and some new features, but I think that it is quite important to take a few time to dig into EDGE networking and networking modes in General with Eucalyptus.

For years, we had 3 mainly used modes :

  • MANAGED
    • AWS Security Groups supported and VLANs created to give L2 separation
    • All traffic goes via the Cluster Controller for cross-groups communication
    • Requires a DHCP clean-environment
  • MANAGED-NOVLAN
    • AWS SG supported but no L2 separation
    • All traffic goes via the Cluster Controller for cross-groups communication
    • Requires a DHCP clean environment
  • SYSTEM
    • Customer DHCP server will assign IP addresses to instances
    • No AWS SG compatibility

As we can figure out, if you needed the AWS compatibility you also had to deal with the CC handling all the instances traffic. But the problem is that you also had to dedicate a physical machine to…

View original 1,008 more words

EDGE Networking in Eucalyptus

The Case for a Policy Decision Point inside the LDAP Server

hspencer77:

Great insight as to the importance of Policy Decision Points with regards to security processes.

Originally posted on iamfortress:

Why on earth would you do that?

We all understand that runtime characteristics change as processes get moved around the network.  Having problems with network io?  Move the database daemon to the same tier as the client process.  Problems with file io?  Store the data in memory as opposed to disk.  etc…

These same techniques apply for system architecture and security.  Location of policy enforcement, decision, and database processes hugely impact the overall welfare of your organization’s computational systems.

With these kinds of thoughts, what happens when security processes get moved around the network?

But first, we must define the security processes:

1. Policy Enforcement Point (PEP)

The gatekeeper component.  It enforces the security policy on the client program.  PEPs come in many shapes and sizes.  Often times it’s a small block of code that gets embedded directly into a client program.

2. Database (DB)

The database is used by PDPs to house…

View original 637 more words

The Case for a Policy Decision Point inside the LDAP Server