Using Boto’s connect_to_region function with Eucalyptus 4.1

June 4, 2015June 4, 2015 hspencer77boto, boto config, cloud computing, connect_to_region, region support Eucalyptus 4.2Leave a comment

Typically, when documenting how to use the Python interface boto with Eucalyptus clouds, the function connect_<service> is always leveraged (<service> being ec2, s3, iam, sts, elb, cloudformation, cloudwatch, or autoscale). The focus of this blog entry is to demonstrate how to use the connect_to_region function associated with each AWS service supported by Eucalyptus. This will be very important when using boto with federated Eucalyptus clouds, which will be available in the release of HP Helion Eucalyptus 4.2.0.

Prerequisites

To get started, the following requirements need to be met:

Python 2.6 or Higher
boto 2.34.0 or higher (latest version at the time of this blog entry was 2.38.0)
Eucalyptus 4.1.0 or Higher Cloud with Eucalyptus DNS enabled
Access Key ID and Secret Access Key for a user under a Eucalyptus IAM account
eucarc file containing Eucalyptus service API endpoints
Eucalyptus IAM policy that provides access to given Eucalyptus service API resources applied to the user account

The Setup

Once the prerequisites are met, now we can focus on setting up the configuration file for boto. Below is an example of a boto configuration file:

[Credentials]
aws_access_key_id = <Eucalyptus Access Key ID>
aws_secret_access_key = <Eucalyptus Secret Access Key>
[Boto]
is_secure = False
endpoints_path = /root/boto-qa-setup1-endpoints.json

Notice the file boto-qa-setup1-endpoints.json. This is a JSON file that contains the Eucalyptus service API endpoints correlating to the Eucalyptus cloud that would like to be accessed. Here are the contents of the boto-qa-setup1-endpoints.json file:

{
 "autoscaling": {
 "eucalyptus": "autoscaling.h-01.autoqa.qa1.eucalyptus-systems.com"
 },
 "cloudformation": {
 "eucalyptus": "cloudformation.h-01.autoqa.qa1.eucalyptus-systems.com"
 },
 "cloudwatch": {
 "eucalyptus": "cloudwatch.h-01.autoqa.qa1.eucalyptus-systems.com"
 },
 "ec2": {
 "eucalyptus": "compute.h-01.autoqa.qa1.eucalyptus-systems.com"
 },
 "elasticloadbalancing": {
 "eucalyptus": "loadbalancing.h-01.autoqa.qa1.eucalyptus-systems.com"
 },
 "iam": {
 "eucalyptus": "euare.h-01.autoqa.qa1.eucalyptus-systems.com"
 },
 "s3": {
 "eucalyptus": "objectstorage.h-01.autoqa.qa1.eucalyptus-systems.com"
 },
 "sts": {
 "eucalyptus": "tokens.h-01.autoqa.qa1.eucalyptus-systems.com"
 },
 "swf": {
 "eucalyptus": "simpleworkflow.h-01.autoqa.qa1.eucalyptus-systems.com"
 }
}

As you can see, each AWS service implemented by Eucalyptus is defined in this file. With the boto configuration file and endpoints json file defined, the connect_to_region function in boto can be easily utilized.

Example

To show how this setup works, ipython will be used as the demonstration environment. Below is an example that shows how to use the connect_to_region function with the Compute service (EC2) against a Eucalyptus 4.1 cloud.

# ipython
Python 2.6.6 (r266:84292, Jan 22 2014, 09:42:36)
Type "copyright", "credits" or "license" for more information.

IPython 0.13.2 -- An enhanced Interactive Python.
? -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help -> Python's own help system.
object? -> Details about 'object', use 'object??' for extra details.

In [1]: import boto.ec2

In [2]: ec2_connection = boto.ec2.connect_to_region('eucalyptus', port=8773)

In [3]: ec2_connection.get_all_instance_types()
Out[3]:
[InstanceType:m1.small-1,256,5,
 InstanceType:t1.micro-1,256,5,
 InstanceType:m1.medium-1,512,10,
 InstanceType:c1.medium-2,512,10,
 InstanceType:m1.large-2,512,10,
 InstanceType:m1.xlarge-2,1024,10,
 InstanceType:c1.xlarge-2,2048,10,
 InstanceType:m2.xlarge-2,2048,10,
 InstanceType:m3.xlarge-4,2048,15,
 InstanceType:m2.2xlarge-2,4096,30,
 InstanceType:m3.2xlarge-4,4096,30,
 InstanceType:cc1.4xlarge-8,3072,60,
 InstanceType:m2.4xlarge-8,4096,60,
 InstanceType:hi1.4xlarge-8,6144,120,
 InstanceType:cc2.8xlarge-16,6144,120,
 InstanceType:cg1.4xlarge-16,12288,200,
 InstanceType:cr1.8xlarge-16,16384,240,
 InstanceType:hs1.8xlarge-48,119808,24000]

Here is an example of describing the volumes and snapshots on a given Eucalyptus cloud:

In [3]: ec2_connection.get_all_volumes()
Out[3]:
[Volume:vol-563802cd,
 Volume:vol-274c3628,
 Volume:vol-7074a3a8,
 Volume:vol-172cdb42,
 Volume:vol-d00e53e9,
 Volume:vol-4f370899]

In [4]: ec2_connection.get_all_snapshots()
Out[4]:
[Snapshot:snap-6d874d5a,
 Snapshot:snap-e41c6adc,
 Snapshot:snap-63700417,
 Snapshot:snap-6055a378,
 Snapshot:snap-91d70d6b,
 Snapshot:snap-7b39eca8,
 Snapshot:snap-80a4f3e2]

Just like the EC2 tutorial by boto demonstrates how to use connect_to_region with AWS EC2, this function can also be used against Eucalyptus 4.1 clouds as well. As mentioned earlier, when region support (multiple Eucalyptus clouds) becomes available in Eucalyptus 4.2, this function will be very useful.

Enjoy!

Adding Eucalyptus Load Balancer Access Logging for Eucalyptus Cloud Users

December 11, 2014December 18, 2014 hspencer77access logging, acls, AWS ELB, AWS S3, boto, cloud computing, elb, eucalyptus 4.0.2, eucalyptus load balancer, haproxy logging, s3cmdLeave a comment

Preface

Eucalyptus continues to strive as the best on-premise AWS-compatible Infrastructure as a Service (IaaS). One of the great things about Eucalyptus being an open source platform, is that if there is an AWS feature that any cloud administrator/developer wants to add, they have the ability to do it. This blog entry will cover how to enable cloud users to have access to the Eucalyptus Load Balancer access logs – similar to how this is accomplished with Amazon Web Services Elastic Load Balancer service.

Before we dive in, I would like to give special thanks to the following members of the Eucalyptus Engineering Team. Without their hard work, this blog would not be possible:

Special thanks to these individuals for their continued contributions to the Eucalyptus software.

Overview

Currently, when a cloud user launches a Eucalyptus Load Balancer, they will see something similar to the following:

# eulb-create-lb hasp-euca-lb --listener "lb-port=80, protocol=http, instance-port=8888, instance-protocol=http" --availability-zone Honest
# eulb-describe-lbs
LOAD_BALANCER hasp-euca-elb hasp-euca-elb-325271821652.eulb.future.euca-hasp.cs.prc.eucalyptus-systems.com 2014-12-11T23:34:35.397Z

Notice the DNS name of the load balancer. It has the following format:

{load balancer name}-{Account ID}.{Load Balancer DNS Subdomain}.{Eucalyptus Cloud DNS Domain}

The “{load balancer name}-{Account ID}” string is the important information in this value.

From the cloud administrator’s perspective, the load balancer is an AutoScaling group. More information can be found in the following resources:

If the cloud administrator describes the instances running under the ‘eucalyptus‘ account and the load balancer above is running, the following would be displayed:

# euca-describe-instances 
RESERVATION r-278c161e 094999295155 euca-internal-325271821652-hasp-euca-elb
INSTANCE i-135b4b0a emi-7a4367b8 euca-10-104-7-21.future.future.euca-hasp.cs.prc.eucalyptus-systems.com euca-172-17-156-121.future.internal running euca-elb 0 c1.medium 2014-12-11T23:34:44.428Z Honest monitoring-enabled 10.104.7.21 172.17.156.121 instance-store hvm c4946e25-64ed-4453-808c-9ff2ab831b47_Honest_1 sg-da911c98 arn:aws:iam::094999295155:instance-profile/internal/loadbalancer/loadbalancer-vm-325271821652-hasp-euca-elb x86_64
TAG instance i-135b4b0a Name loadbalancer-resources
TAG instance i-135b4b0a aws:autoscaling:groupName asg-euca-internal-elb-325271821652-hasp-euca-elb
TAG instance i-135b4b0a euca:node 10.104.1.218

Notice the ‘RESERVATION’ line that contains the security group that the instance is using. If the ‘euca-internal-‘ prefix is removed, the security group has the following format:

{Account ID}-{load balancer Name}

This information matches the Load Balancer launched by the cloud user and will be the base for the solution.

Building the Foundation

In order to get started, the solution needs to be applied from the Cloud Administrator (i.e. admin user in the ‘eucalyptus’ account) perspective. This solution can not be applied by any other type of cloud user. In addition to cloud administrator user requirement, the following is needed:

Eucalyptus 4.0.2 Cloud Deployment
Eucalyptus DNS enabled
Eucalyptus Load Balancer enabled
Cloud property ‘loadbalancing.loadbalancer_vm_keyname‘ is set to a keypair created by the Cloud Administrator

Once these requirements are met, the environment is ready to go.

Create ELB Access Log User

A user (e.g. ‘elb-osg-logger’) needs to be created under the ‘eucalyptus’ account which will be used with the custom python script to store the load balancer access logs to the OSG bucket. To create the user, after sourcing the cloud administrator credentials, use euare-usercreate:

# euare-usercreate -u elb-osg-logger -k 
AKILXXXXXXXXXXXXXX
PS6nXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Store these credentials in a safe place. Next, customizing the load balancer instance.

Customize the Load Balancer

To begin, a Eucalyptus Load Balancer needs to be launched in order to modify it. The goal here is to build an image from this instance using euca-bundle-instance. We will start with the load balancer mentioned earlier:

# euca-describe-instances 
RESERVATION r-5e1d4d17 094999295155 euca-internal-325271821652-hasp-euca-lb
INSTANCE i-315dd646 emi-7a4367b8 euca-10-104-7-9.future.future.euca-hasp.cs.prc.eucalyptus-systems.com euca-172-17-177-235.future.internal running euca-elb 0 c1.medium 2014-12-11T04:23:04.441Z Honest monitoring-enabled 10.104.7.9 172.17.177.235 instance-store hvm b134a0bc-cfc4-4c6e-84ba-4fd1df160407_Honest_1 sg-b6cc605e arn:aws:iam::094999295155:instance-profile/internal/loadbalancer/loadbalancer-vm-325271821652-hasp-euca-lb x86_64
TAG instance i-315dd646 Name loadbalancer-resources
TAG instance i-315dd646 aws:autoscaling:groupName asg-euca-internal-elb-325271821652-hasp-euca-lb
TAG instance i-315dd646 euca:node 10.104.1.218

To access the load balancer, authorize SSH to the instance:

# euca-authorize -P tcp -p 22 euca-internal-325271821652-hasp-euca-elb

Next, SSH into the ELB instance:

# ssh -i euca-elb.priv root@euca-10-104-7-9.future.future.euca-hasp.cs.prc.eucalyptus-systems.com

Once inside the instance, install the EPEL package repository:

# yum localinstall --nogpgcheck https://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm -y

After the package has been installed, use yum to install the python-pip package:

# yum install python-pip -y

Next, use pip to upgrade and install the ‘boto‘ and ‘argparse‘ modules:

# pip install --upgrade boto argparse

Now, its time to add the custom python script.

Add Access Logs Script

The Access Log Script performs the following actions:

Creates a bucket with READ bucket ACL for Account ID which launches the Eucalyptus Load Balancer
- bucket created with the following format – s3://access_logs-{LB name}_{public-IPV4 of LB}_{LB instance numeric ID}
Places a copy of “/var/log/load-balancer-access.log.1” in the bucket with READ object ACL for Account ID which owns the Eucalyptus Load Balancer
- the file in the bucket will have the following naming format – elb-access-{timestamp DDMMYY-HourMinSec}.log
Bonus – since Eucalyptus 4.0.0, OSG has supported object lifecycle management. If lifecycle value is passed and its greater than 0, the object lifecycle is applied to all objects in the bucket.

To add the script to the instance, use curl:

# curl http://euca-elb-access-log-blog.s3.amazonaws.com/access-log-transfer-s3.py -o access-log-transfer-s3.py

Once the script has been downloaded, edit the script and add the ‘elb-osg-logger’ user credentials, the S3_URL and EC2_URL to the script in the following locations:

 EC2Connection.DefaultRegionEndpoint = '<EC2_URL - Eucalyptus Cloud Compute API DNS Name>'
 ec2conn = EC2Connection(aws_access_key_id="<elb-osg-logger user Access Key ID>",
 aws_secret_access_key="<elb-osg-logger user Secret Access Key>",
 is_secure=False, port="8773")

 s3 = S3Connection(aws_access_key_id="<elb-osg-logger user Access Key ID>",
 aws_secret_access_key="<elb-osg-logger user Secret Access Key>",
 host="<S3_URL - Eucalyptus Cloud OSG API DNS Name>",
 is_secure=False, port=8773, calling_format=OrdinaryCallingFormat())

Set the script to be executable using chmod:

# chmod a+x /root/access-log-transfer-s3.py

Now its time to configure HAProxy to log information.

Enable HAProxy Logging

The Eucalyptus Load Balancer uses haproxy to perform load balancing. To enable logging, the following files need to be edited:

/etc/load-balancer-servo/haproxy_template.conf
- under the ‘global’ section add – log 127.0.0.1 local3 info
- under the ‘default’ section add – log global
/usr/lib/python2.6/site-packages/servo/haproxy/haproxy_conf.py
- change the following section:

 if protocol == 'http' or protocol == 'https':
 self.__content_map[section_name].append('log-format httplog\ %f\ %b\ %s\ %ST\ %ts\ %Tq\ %Tw\ %Tc\ %Tr\ %Tt') 
 elif protocol == 'tcp' or protocol == 'ssl':
 self.__content_map[section_name].append('log-format tcplog\ %f\ %b\ %s\ %ts\ %Tw\ %Tc\ %Tt')

if protocol == 'http' or protocol == 'https':
 self.__content_map[section_name].append('log-format httplog\ %f\ %b\ %s\ %ST\ %ts\ %Tq\ %Tw\ %Tc\ %Tr\ %Tt\ %{+Q}r\ %ci:%cp\ %fi:%fp\ %si:%sp\ req_size=%U\ resp_size=%B')
 elif protocol == 'tcp' or protocol == 'ssl':
 self.__content_map[section_name].append('log-format tcplog\ %f\ %b\ %s\ %ts\ %Tw\ %Tc\ %Tt\ %{+Q}r\ %ci:%cp\ %fi:%fp\ %si:%sp\ req_size=%U\ resp_size=%B')

For more information about the log-format in HAProxy, reference the HAProxy documentation on log format. The information that can be logged is highly customizable. Reference the AWS ELB documentation regarding Access Log Entries to get a better sense of the logging experience on AWS.

Logging for HAProxy is complete. Next, rsyslog and logrotate need to be configured.

Log Management

Storing the HAProxy logs, and rotating them is very important to this solution. The script takes the rotated log, and stores it in the OSG bucket for the access logs. The purpose of this is to make sure the file is not being written to when its being sent to the OSG bucket. To start out, download the load-balancer.conf file to use with logrotate using curl:

# curl http://euca-elb-access-log-blog.s3.amazonaws.com/load-balancer.conf -o load-balancer.conf

This is the logrotate configuration file that the cronjob script will call to rotate the log file, then execute the access-log-transfer-s3.py script with a 1 day object lifecycle. To change the lifecycle, just change the value of the –lifecycle option in the load-balancer.conf file.

Next, update rsyslog to make sure the latest is running on the instance:

# yum upgrade rsyslog -y

After this has completed, add the following to the /etc/rsyslog.d/load-balancer.conf file:

local3.*       /var/log/load-balancer-access.log

Follow this step up by uncommenting and adding the following lines in /etc/rsyslog.conf:

$ModLoad imudp
$UDPServerAddress 127.0.0.1
$UDPServerRun 514

To wrap up, we need to add a script that will be kicked off by the cronjob.

Cronjob Script

To kick off the log rotation, add the ‘elb-logrotate‘ script to the instance using curl:

# curl http://euca-elb-access-log-blog.s3.amazonaws.com/elb-logrotate -o elb-logrotate

Using ‘crontab -e’, set up a cron for 5 minutes (or however often the access log information would like to be uploaded to the bucket):

*/5 * * * * /root/elb-logrotate

Clean Up

After completing all the customizations, the instance needs to be prepared for bundling. Run the following commands to prepare the instance:

# echo "" > /etc/udev/rules.d/70-persistent-net.rules
# echo "" > /lib/udev/rules.d/75-persistent-net-generator.rules

If PERSISTENT_DHCLIENT is not in the /etc/sysconfig/network-scripts/ifcfg-eth0 file, then add it:

# grep PERSISTENT_DHCLIENT /etc/sysconfig/network-scripts/ifcfg-eth0
# echo "PERSISTENT_DHCLIENT=yes" >> /etc/sysconfig/network-scripts/ifcfg-eth0

Now we can exit out the instance.

Creating the New Eucalyptus Load Balancer EMI

After finishing with the instance customizations, the instance is ready to be bundled and registered. First, use euca-bundle-instance to bundle and upload the instance. Use euca-describe-bundle-tasks to check on the status of the bundling operation. Once the bundling operation has been completed, use euca-register to register the new ELB EMI:

# euca-bundle-instance -b load-balancer-access-logs -p eucalyptus-load-balancer-image-access-log i-315dd646
BUNDLE bun-315dd646 i-315dd646 load-balancer-access-logs eucalyptus-load-balancer-image-access-log 2014-12-11T04:07:59.835Z 2014-12-11T04:07:59.835Z pending 0 load-balancer-access-logs/eucalyptus-load-balancer-image-access-log.manifest.xml
....
# euca-describe-bundle-tasks
BUNDLE bun-315dd646 i-315dd646 load-balancer-access-logs eucalyptus-load-balancer-image-access-log 2014-12-11T04:07:59.835Z 2014-12-11T04:09:57.671Z complete 0 load-balancer-access-logs/eucalyptus-load-balancer-image-access-log.manifest.xml
# euca-register -a x86_64 -n load-balancer-access-logs load-balancer-access-logs/eucalyptus-load-balancer-image-access-log.manifest.xml --virtualization-type hvm
IMAGE emi-7a4367b8

Now that the new Eucalyptus Load Balancer EMI is register, update the cloud property ‘loadbalancing.loadbalancer_emi‘ to display the new ELB EMI:

# euca-modify-property -p loadbalancing.loadbalancer_emi=emi-7a4367b8
PROPERTY loadbalancing.loadbalancer_emi emi-7a4367b8 was emi-cf4fb988

Now, lets test out the changes.

Testing Out the ELB with Access Logging

To test it out, you can use either the Cloud Administrator, or a user from a ‘non-eucalyptus’ account. In the example below, a user from a ‘non-eucalyptus’ account was used. If a ‘non-eucalyptus’ account user is used, make sure the user has the appropriate IAM access policies for EC2 (Compute), S3 (OSG), and ELB (Eucalyptus Load Balancer).

First, create the Eucalyptus Load Balancer:

# eulb-create-lb hasp-euca-lb --listener "lb-port=80, protocol=http, instance-port=80, instance-protocol=http" --availability-zone Honest --region account2-user11@

Next, launch an instance that has a web service running on port 80. In this example, I used a cloud-init configuration file to install nginx on an Ubuntu 14.04 (Trusty Tahr) Cloud Image:

# euca-run-instances -k account2-user11 -t m1.medium emi-59a742d0 --user-data-file nginx-cloudinit.config --region account2-user11@
....
# euca-describe-instances --region account2-user11@
RESERVATION r-5c16c716 325271821652 default
INSTANCE i-45c1ebd1 emi-59a742d0 euca-10-104-7-29.future.future.euca-hasp.cs.prc.eucalyptus-systems.com euca-172-17-248-189.future.internal running account2-user11 0 m1.medium 2014-12-05T21:53:51.197Z Honest monitoring-disabled 10.104.7.29 172.17.248.189 instance-store hvm sg-6ef9907f x86_64

# eulb-register-instances-with-lb --instances i-45c1ebd1 hasp-euca-lb --region account2-user11@
INSTANCE i-45c1ebd1

Generate some traffic to the ELB using curl or some other tool to populate the HAProxy log file. Based upon how often the cronjob was set to execute, use s3cmd to see the bucket created in the ‘eucalyptus’ account (i.e. Cloud Administrator) for the access logs. For information regarding s3cmd configuration files, refer to my previous blog:

# ./s3cmd/s3cmd --config=.s3cfg-cloud-admin ls
2014-09-18 02:59 s3://51c700-download-manifests
2014-12-11 04:31 s3://access_logs-hasp-euca-lb_10.104.7.9_315dd646
2014-09-18 02:43 s3://centos-6.5-x86_64-20140917
2014-09-18 02:46 s3://centos-7-x86_64-20140917
2014-11-05 22:05 s3://centos6.4-kernel
2014-11-05 21:54 s3://centos6.4-ramdisk
2014-11-05 22:08 s3://centos6.4-test
2014-09-18 02:52 s3://debian-7-x86_64-20140917

# ./s3cmd/s3cmd --config=.s3cfg-cloud-admin ls s3://access_logs-hasp-euca-lb_10.104.7.9_315dd646
2014-12-11 04:31 817 s3://access_logs-hasp-euca-lb_10.104.7.9_315dd646/elb-access-11122014-043122.log
2014-12-11 05:13 78764 s3://access_logs-hasp-euca-lb_10.104.7.9_315dd646/elb-access-11122014-051353.log
2014-12-11 05:20 58202 s3://access_logs-hasp-euca-lb_10.104.7.9_315dd646/elb-access-11122014-052002.log

Once that has been confirmed, create another s3cmd configuration file for the ‘non-eucalyptus’ user, and confirm the user can list the contents of the bucket:

# ./s3cmd/s3cmd --config=.s3cfg-acct2-user11 ls s3://access_logs-hasp-euca-lb_10.104.7.9_315dd646
2014-12-11 04:31 817 s3://access_logs-hasp-euca-lb_10.104.7.9_315dd646/elb-access-11122014-043122.log
2014-12-11 05:13 78764 s3://access_logs-hasp-euca-lb_10.104.7.9_315dd646/elb-access-11122014-051353.log
2014-12-11 05:20 58202 s3://access_logs-hasp-euca-lb_10.104.7.9_315dd646/elb-access-11122014-052002.log

After that has been confirmed, download one of the log files and confirm the contents:

# ./s3cmd/s3cmd --config=.s3cfg-acct2-user11 get s3://access_logs-hasp-euca-lb_10.104.7.9_315dd646/elb-access-11122014-051353.log .
s3://access_logs-hasp-euca-lb_10.104.7.9_315dd646/elb-access-11122014-051353.log -> ./elb-access-11122014-051353.log [1 of 1]
 78764 of 78764 100% in 0s 238.84 kB/s done
 
# cat elb-access-11122014-051353.log
Dec 11 04:32:11 localhost haproxy[1070]: httplog http-80 backend-http-80 http-80 200 -- 0 0 0 1 1 "HEAD / HTTP/1.1" 10.5.1.70:49960 172.17.177.235:80 10.104.7.29:8888 req_size=142 resp_size=241
Dec 11 05:05:35 localhost haproxy[1070]: httplog http-80 backend-http-80 http-80 200 -- 0 0 0 1 1 "HEAD / HTTP/1.1" 10.5.1.70:50216 172.17.177.235:80 10.104.7.29:8888 req_size=142 resp_size=241
Dec 11 05:05:36 localhost haproxy[1070]: httplog http-80 backend-http-80 http-80 200 -- 4 0 1 1 6 "HEAD / HTTP/1.1" 10.5.1.70:50217 172.17.177.235:80 10.104.7.29:8888 req_size=142 resp_size=241
Dec 11 05:05:38 localhost haproxy[1070]: httplog http-80 backend-http-80 http-80 200 -- 5 0 0 1 6 "HEAD / HTTP/1.1" 10.5.1.70:50218 172.17.177.235:80 10.104.7.29:8888 req_size=142 resp_size=241
Dec 11 05:05:39 localhost haproxy[1070]: httplog http-80 backend-http-80 http-80 200 -- 0 0 0 1 1 "HEAD / HTTP/1.1" 10.5.1.70:50219 172.17.177.235:80 10.104.7.29:8888 req_size=142 resp_size=241
Dec 11 05:05:40 localhost haproxy[1070]: httplog http-80 backend-http-80 http-80 200 -- 0 0 0 1 1 "HEAD / HTTP/1.1" 10.5.1.70:50220 172.17.177.235:80 10.104.7.29:8888 req_size=142 resp_size=241
Dec 11 05:05:41 localhost haproxy[1070]: httplog http-80 backend-http-80 http-80 200 -- 0 0 0 1 1 "HEAD / HTTP/1.1" 10.5.1.70:50221 172.17.177.235:80 10.104.7.29:8888 req_size=142 resp_size=241
Dec 11 05:05:42 localhost haproxy[1070]: httplog http-80 backend-http-80 http-80 200 -- 4 0 1 1 6 "HEAD / HTTP/1.1" 10.5.1.70:50222 172.17.177.235:80 10.104.7.29:8888 req_size=142 resp_size=241
.....

How is this ‘non-eucalyptus’ user able to see and download the contents of this bucket? This is because of the script that creates the access log bucket, and uploads the logs to the bucket. By grabbing the account ID from the instance metadata ‘security group’ category, the script adds bucket and object READ ACLs for the account ID. The only issue here is that the cloud administrator will still need to communicate the bucket that the cloud user can access for the logs. With the extra bonus of using the object lifecycle, the cloud administrator doesn’t have to worry about managing the buckets. The objects will remove themselves after the define period of time.

Conclusion

Even though the solution isn’t exactly like AWS ELB Access Logs feature, it does provide a solution that is very similar to it. The only thing missing is the service API interaction to enable/disable the access logging feature, set the interval and define the bucket that will be used. Hopefully, this will be a feature we will see in the not too distant feature. Thanks for hanging in there with me. I hope you enjoy! Feedback is always welcome.

Cheers!

Eucalyptus 4.0 IAM Feature: Cross-Account API Access Using IAM Roles

December 7, 2014December 11, 2014 hspencer77aws compatibility, boto, cloud computing, cross-account API access, Euare, euca2ools, eucalyptus, eucalyptus 4.0.2, IAM, rolesLeave a comment

As mentioned in my past blog, Eucalyptus has the concept of EUARE (IAM) Roles which pretty much work the same way as AWS IAM Roles. This blog with discuss how to set up cross-account API access using EUARE (IAM) Roles in Eucalyptus 4.0.2. Cross-account API access is very useful for use cases where there are multiple Eucalyptus EUARE accounts that need to access the same resources. A great example of this can be found in the AWS IAM documentation regarding this same feature.

To establish cross-account access, in the trusting account (Account A), you create an IAM role that designates Account B as the trusted account. The role also defines permissions that allow Account B access to specific resources in Account A. Account B can then delegate this access to its own users (IAM users or federated users). Account B cannot delegate more access to its users than the permissions that it has been granted by Account A.

To demonstrate this feature, a user in Account A will set up a role to allow DescribeInstances API access for a user in Account B.

Prerequisites

Before getting started, the following resources are needed:

Eucalyptus 4.0.2 Cloud with Eucalyptus DNS enabled
Two non-‘eucalyptus‘ accounts created (for example, AccountA and AccountB)
Eucalyptus Machine Images (EMIs) available to be launched as instances
Running instance(s) under AccountA
Latest version of euca2ools

Once these prerequisites are met, the cross-account role use case can be deployed.

NOTE: All actions are assumed to be performed by a user under the given account. All IAM actions can be done by the cloud administrator (i.e. user under the ‘eucalyptus’ account) if so desired.

Account A EUARE (IAM) Setup

Before setting up the role in Account A, the minimal EUARE (IAM) access policy needed by the user in Account A in order to create the role. The access policy can be found here:

http://cross-account-blog.s3.amazonaws.com/euare-accountA-access.policy

This policy allows the user to perform the following EUARE (IAM) API actions:

Create a role
List roles
Upload a policy to a role

Using IAM best practices, apply this policy to the group that has this user as a member under Account A, using the euare-groupuploadpolicy command.

Trust Policy for Role in Account A

In order for the user in Account B (in this example, the user is ‘user12’) to be able to assume the role under Account A, a trust policy needs to be associated with the role. Below is an example of the trust policy (which can be downloaded from here):

{
 "Version": "2012-10-17",
 "Statement": [{
 "Effect": "Allow",
 "Principal": {"AWS": "arn:aws:iam::325271821652:user/user12"},
 "Action": "sts:AssumeRole"
 }]
}

This states that ‘user12’ under the Account ID ‘325271821652‘ (which maps to Account B) can assume this role. To create the role, use the euare-rolecreate command as follows:

# euare-rolecreate --role-name cross-describe-instances --file-name euare-accountA-role-trust.policy

IAM Access Policy for Role under Account A

After creating the role, now the API access (in this case – DescribeInstances) needs to be associated by uploading an IAM access policy for the role. Here is an example of the policy:

{
 "Statement": [
 {
 "Sid": "Stmt1417530823611",
 "Action": [
 "ec2:DescribeInstances"
 ],
 "Effect": "Allow",
 "Resource": "*"
 }
 ]
}

This policy can be downloaded from here. Use euare-roleuploadpolicy to upload the access policy to the role. For example:

# euare-roleuploadpolicy --role-name cross-describe-instances --policy-name describe-instances --policy-document euare-accountA-role-access.policy

Once this has been completed, all work is done under Account A. Next, the user under Account B needs to be allowed to assume the role. The only important information needed from Account A, is the ARN for the role – which in this use case is the following:

arn:aws:iam::290122656840:role/cross-describe-instances

Account B EUARE (IAM) Setup

Before applying the policy to allow assuming the role to the group where ‘user12’ is a member, the user needs to be able to do the following at a minimum:

Generate Access Keys
List Access Keys
List groups
Upload IAM policies to groups

An example IAM access policy of these actions can be found here:

http://cross-account-blog.s3.amazonaws.com/euare-accountB-access.policy

As mentioned above, apply this policy to the group that has ‘user12’ as a member under Account B, using the euare-groupuploadpolicy command.

Assume Role Access

To allow ‘user12’ under Account B to assume the role under Account A, we need to upload another policy. This can be done to the group that has ‘user12’ as a member, or directly to ‘user12’ by using euare-useruploadpolicy. The key information needed in the IAM policy is the ARN of the role (arn:aws:iam::290122656840:role/cross-describe-instances). The policy should look something similar to this:

{
 "Statement": [
 {
 "Sid": "Stmt1417531456446",
 "Action": [
 "sts:AssumeRole"
 ],
 "Effect": "Allow",
 "Resource": "arn:aws:iam::290122656840:role/cross-describe-instances"
 }
 ]
}

This policy can be downloaded from here:

http://cross-account-blog.s3.amazonaws.com/euare-accountB-assume-role.policy

To keep consistent with IAM best practices, euare-groupuploadpolicy will be used to upload the policy to the group that has ‘user12’ as a member:

# euare-groupuploadpolicy --policy-name assume-role-run-describe-instances --group-name Testers --policy-document euare-accountB-assume-role.policy

Once the policy is uploaded, we just need to test out the cross-account access.

Testing with Python Boto

To confirm that the cross-account access works, we will use the boto – a python interface to AWS, which also works with Eucalyptus. Below is an example script with performs the following actions:

Accesses the Eucalyptus STS service to obtain a temporary access key, secret key and token of the assumed role
Performs the DescribeInstances API call

#!/bin/env python
import boto
from boto.sts import STSConnection
from boto.ec2.connection import EC2Connection
if __name__ == "__main__":
 """
 Assuming 'describe-instances' role by AccountB, User12 user
 """
 STSConnection.DefaultRegionEndpoint = "tokens.future.euca-hasp.cs.prc.eucalyptus-systems.com"
 sts_connection = STSConnection(aws_access_key_id="<Account B User Access Key>",
                                aws_secret_access_key="<Account B User Secret Key>",
                                is_secure=False, port="8773")

 assumedRoleObject = sts_connection.assume_role(
                     role_arn="arn:aws:iam::290122656840:role/cross-describe-instances",
                     role_session_name="Acct2User12DescribeInstancesSession")

 EC2Connection.DefaultRegionEndpoint = "compute.future.euca-hasp.cs.prc.eucalyptus-systems.com"
 ec2_connection = EC2Connection(aws_access_key_id=assumedRoleObject.credentials.access_key,
                                aws_secret_access_key=assumedRoleObject.credentials.secret_key,
                                security_token=assumedRoleObject.credentials.session_token,
                                is_secure=False, port="8773")

 reservations = ec2_connection.get_all_instances()
 if len(reservations) > 0:
     print "Instance Information:"
     for reservation in reservations:
         for instance in reservation.instances:
             print "\t--------------------------------------"
             print "\tInstance ID:\t" + instance.id
             print "\tInstance State:\t" + instance.state
             print "\tPublic DNS Name:\t" + instance.public_dns_name
             print "\tLaunch Time:\t" + instance.launch_time
 else:
     print "No Instance Reservations Available."

The script can be downloaded from here:

http://cross-account-blog.s3.amazonaws.com/describe-instance-script.py

To use the script, the following information is needed:

For STSConnection.DefaultRegionEndpoint, use the Eucalyptus DNS name for Eucalyptus Token service associated with your Eucalyptus 4.0.2 cloud (e.g. “tokens.future.euca-hasp.cs.prc.eucalyptus-systems.com“)
For EC2Connection.DefaultRegionEndpoint, use the Eucalyptus DNS name for Eucalyptus Compute service associated with your Eucalyptus 4.0.2 cloud (e.g. “compute.future.euca-hasp.cs.prc.eucalyptus-systems.com“)
Access Key ID and Secret Access Key of the ‘user12’ user under Account B.

Running the Script

After downloading the script, and perform the edits to match your Eucalyptus 4.0.2 environment, make sure the executable bit is set on the script, and run it:

# ./describe-instance-script.py

Instance Information:
 --------------------------------------
 Instance ID: i-f97462d3
 Instance State: running
 Public DNS Name: euca-10-104-7-27.future.future.euca-hasp.cs.prc.eucalyptus-systems.com
 Launch Time: 2014-12-03T16:22:54.113Z

Observe how the script has listed the instance under Account A. Confirm this by using the credentials of a user under Account A, and running euca-describe-instances:

# euca-describe-instances --region accountA-user01@
RESERVATION r-370ded0a 290122656840 default
INSTANCE i-f97462d3 emi-bdcec010 euca-10-104-7-27.future.future.euca-hasp.cs.prc.eucalyptus-systems.com euca-172-17-176-24.future.internal running account1-user01 0 m1.medium 2014-12-03T16:22:54.113Z Honest monitoring-disabled 10.104.7.27 172.17.176.24 instance-store hvm sg-525fe580 x86_64

Conclusion

Cross-account access using Eucalyptus EUARE (IAM) roles are very powerful. It allows cloud administrators and account administrators to provide access to limited resources in different accounts without having the hassle of creating/managing users. Please refer to the AWS IAM documentation for further examples and use cases. Enjoy!

More Mind Spew-age from Harold Spencer Jr.

Random Information – sometimes techie, sometimes not..

Menu

boto