Skip to content

More Mind Spew-age from Harold Spencer Jr.

Random Information – sometimes techie, sometimes not..

Menu

  • About The Author
  • Interests
  • Private Cloud Computing
  • Infrastructure
  • System Adminstration
  • Private Cloud Computing

RSS All Things Distributed – Werner Vogels

  • Melbourne gets a Region, a big trip, and a brain mapping startup
  • An album for each year - 2022 version
  • Tech predictions for 2023 and beyond
  • On site in Denmark: Behind the scenes with Veo
  • ¡Bienvenida España! Introducing the new AWS Europe (Spain) Region

RSS Elastician – Mitch Garnaat

  • Back To The Future
  • Looking at Clouds from Both Sides Now
  • Don't reboot me, bro!
  • Mapping Requests to EC2 API Versions
  • Python and AWS Cookbook Available

RSS Amazon Web Services Blog

  • New – AWS CloudTrail Lake Supports Ingesting Activity Events From Non-AWS Sources
  • New – Deployment Pipelines Reference Architecture and Reference Implementations
  • AWS Week in Review – January 30, 2023
  • Now Open — AWS Asia Pacific (Melbourne) Region in Australia
  • AWS Week in Review – January 23, 2023

RSS Amazon Web Services Security Blog

  • Define a custom session duration and terminate active sessions in IAM Identity Center
  • How to set up ongoing replication from your third-party secrets manager to AWS Secrets Manager
  • Reduce risk by implementing HttpOnly cookie authentication in Amazon API Gateway
  • AWS achieves ISO 20000-1:2018 certification for 109 services
  • Visualize AWS WAF logs with an Amazon CloudWatch dashboard

RSS Google Developers Blog

  • Machine Learning Communities: Q4 ‘22 highlights and achievements
  • Google Dev Library Letter: 17th Edition
  • Useful Android projects from Google Dev Library to help you #DevelopwithGoogle
  • Migrating from App Engine Users to Cloud Identity Platform (Module 21)
  • More Voices = More Bazel

RSS Google Cloud Platform Blog

  • What Data Pipeline Architecture should I use?
  • What’s new with Google Cloud
  • BigQuery authorized views permissions via Terraform, avoiding the chicken & egg problem
  • Snap maintains uptime and growth with Mission Critical Services
  • Better together: Looker connector for Looker Studio now generally available

RSS Thinking Out Cloud

  • Ravello Launches the Cloud Application Hypervisor
  • State of the Open Source Cloud
  • Cloud Adoption Patterns - Citrix Synergy 2012 Keynote
  • In the Game of Clouds, You Win Or You Die: CloudStack

Tags

acls ami ansible appscale autoscaling aws aws compatibility aws ec2 boto cloud-init cloud administrator cloud computing cloudformation coreos coreos cluster coreos etcd docker drbd ec2 elb emi emis.eucalyptus.com Euare euca2ools euca2ools 3 eucalyptus eucalyptus 2.0 eucalyptus 3.4 eucalyptus 4.0 eucalyptus 4.0.2 eucalyptus load balancer eustore eutester git haproxy high availability hybrid cloud iaas IAM image management ipython lvm mac os x mdb neo4j openldap paas Personal precise private cloud python boto s3 s3cmd stackato ubuntu cloud image ubuntu cloud images ubuntu raring varnish walrus xcode

Follow me on Twitter

My Tweets

eucalyptus 4.0.x

Eucalyptus 4.0 OSG Bucket ACLs and Object Lifecycle Management using Python Boto and s3cmd

December 14, 2014December 14, 2014 hspencer77acls, aws compatibility, cloud computing, cross-account access, eucalyptus 4.0.x, object lifecycle management, python boto, s3cmd, third party AWS client tools1 Comment

Introduction

My last few blogs have discussed how Eucalyptus 4.0 supports similar Amazon Web Services, such as Elastic Compute Cloud (EC2), Simple Storage Services (S3), Elastic Load Balancing (ELB) and Security Token Services (STS) features. Using third party tools, such as python boto and s3cmd, features such as listing buckets and objects, listing running instances and assuming IAM roles were covered.  This blog entry is yet another example as to how Eucalyptus provides AWS compatible features in a private, on-premise cloud environment.

A couple of little known AWS-like features provided by the Eucalyptus 4.0.x Object Storage Gateway (OSG) that are barely discussed are the following:

  • Bucket and Object ACLs
  • Object Lifecycle Management

Both of these features help cloud users implement cross-account bucket and object access, and define how long an object lives in a bucket.  Keeping consistent with the earlier blog entries and continuing to promote Eucalyptus’s AWS compatibility, these features will be demonstrated using both python boto and s3cmd.

Requirements – Before We Begin

Before we get started, the following is needed:

  • Eucalyptus 4.0.x Cloud with Eucalyptus DNS enabled
  • Two Eucalyptus Euare (IAM) Accounts (this blog uses the ‘eucalyptus’ account and a non-‘eucalyptus’ account)
  • One set of Eucalyptus user credentials
  • IAM access policy applied to the Eucalyptus user to allow access to OSG (S3) API calls.
  • latest version of python boto
  • latest version of s3cmd

Both boto and s3cmd can be installed on the same client machine to make things easier.  Once the tasks above have been completed, we are ready to begin.

Python Boto

As mentioned before, past blog entries have covered how to use boto with Eucalyptus. To help demonstrate bucket/object ACLs and object lifecycle management using boto with Eucalyptus, I created a script – which can be downloaded.  Before using the script, make sure and change the following variables:

  • aws_access_key_id=“<Eucalyptus Access Key ID>”
  • aws_secret_access_key=“<Eucalyptus Secret Key>”
  • host=“<S3_URL – Eucalyptus OSG DNS Name>”

After updating the script, run the script with the –help option to see what arguments can be passed:

$ ./osg-boto-s3.py --help
usage: osg-boto-s3.py [-h] [-g ACCOUNT_ID] [-a ACL_PERM] [-r] [-l LIFECYCLE]
 [-d] [-o BUCKET_OBJECT]
 bucket

Script that sets grantee bucket (and optionally object) ACL and/or Object
Lifecycle on an OSG Bucket; refer to
http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html and
http://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html for
additional information.

positional arguments:
 bucket Eucalyptus OSG Bucket to apply Bucket ACL and/or
 Object Lifecycle

optional arguments:
 -h, --help show this help message and exit

Bucket ACL Arguments:
 Arguments to Define Bucket ACLs

 -g ACCOUNT_ID, --grantee ACCOUNT_ID
 Account ID of Grantee for Defining Bucket ACL
 -a ACL_PERM, --acl ACL_PERM
 Define ACL Permission: READ, WRITE, READ_ACP,
 WRITE_ACP, FULL_CONTROL
 -r, --recursive Option to pass if ACL is to be applied to objects in
 bucket; only applies if bucket already has objects

Object Lifecycle Arguments:
 Arguments to Define Object Lifecycle

 -l LIFECYCLE, --lifecycle LIFECYCLE
 Number of Days for Object Lifecycle Management

Bucket Attribute Display Argument:
 Option to Display Bucket ACLs and Object Lifecycle

 -d, --display Option to display bucket attributes for object
 lifecycle and bucket ACLs

Object Argument:
 Argument to Define Object (file) To Upload to Bucket

 -o BUCKET_OBJECT, --object BUCKET_OBJECT
 Object (file) to be uploaded to given bucket

The script does the following:

  • Accepts a bucket as an argument; if the bucket does not exist, the script will create it.
  • Applies an object lifecycle to the given bucket
  • Applies the desired ACL on bucket and/or objects in the bucket
  • Uploads an object to the given bucket
  • Displays bucket (and optionally object) ACLs and defined object lifecycle

Below are a couple of examples:

Create A Bucket and DEFINE an Object Lifecycle

This example will show how to create a bucket called “test-boto”, and apply an object lifecycle of 10 days.  The user credentials used are from the ‘admin’ user of the ‘eucalyptus’ account:

# ./osg-boto-s3.py --lifecycle 10 test-boto
# ./osg-boto-s3.py --display test-boto

[ Object Lifecycle Configuration for test-boto ]

<?xml version="1.0" ?>
<LifecycleConfiguration>
 <Rule>
 <ID>
 lifecycle-rule-test-boto
 </ID>
 <Prefix/>
 <Status>
 Enabled
 </Status>
 <Expiration>
 <Days>
 10
 </Days>
 </Expiration>
 </Rule>
</LifecycleConfiguration>

[ Bucket ACL for test-boto ]

<?xml version="1.0" ?>
<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
 <Owner>
 <ID>
 181f1e5b54f4a5b52b245124561c25104a5441623a29121d14531a4c1722b295
 </ID>
 <DisplayName>
 eucalyptus
 </DisplayName>
 </Owner>
 <AccessControlList>
 <Grant>
 <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser">
 <ID>
 181f1e5b54f4a5b52b245124561c25104a5441623a29121d14531a4c1722b295
 </ID>
 <DisplayName>
 eucalyptus
 </DisplayName>
 </Grantee>
 <Permission>
 FULL_CONTROL
 </Permission>
 </Grant>
 </AccessControlList>
</AccessControlPolicy>

As you can see, the bucket has been created, and using the ‘–display‘ option, it shows the ACLs and object lifecycle associated with the bucket.

Upload Object and Define ACL for Another Account

In this example, we will upload a file named cfn-ubuntu-docker.json (which is a Cloudformation template that can be used with Eucalyptus Cloudformation Service), and provide a READ ACL for the account ID ‘325271821652‘ – which has the account name of ‘account2‘.

# ./osg-boto-s3.py --object cfn-ubuntu-docker.json --grantee 325271821652 --acl READ --recursive test-boto
# ./osg-boto-s3.py --display test-boto --recursive

[ Object Lifecycle Configuration for test-boto ]

<?xml version="1.0" ?>
<LifecycleConfiguration>
 <Rule>
 <ID>
 lifecycle-rule-test-boto
 </ID>
 <Prefix/>
 <Status>
 Enabled
 </Status>
 <Expiration>
 <Days>
 10
 </Days>
 </Expiration>
 </Rule>
</LifecycleConfiguration>

[ Bucket ACL for test-boto ]

<?xml version="1.0" ?>
<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
 <Owner>
 <ID>
 181f1e5b54f4a5b52b245124561c25104a5441623a29121d14531a4c1722b295
 </ID>
 <DisplayName>
 eucalyptus
 </DisplayName>
 </Owner>
 <AccessControlList>
 <Grant>
 <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser">
 <ID>
 2947352161f1a86121e325043564943471440401a2ac1d20534e40e114b3c244
 </ID>
 <DisplayName>
 account2
 </DisplayName>
 </Grantee>
 <Permission>
 READ
 </Permission>
 </Grant>
 <Grant>
 <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser">
 <ID>
 181f1e5b54f4a5b52b245124561c25104a5441623a29121d14531a4c1722b295
 </ID>
 <DisplayName>
 eucalyptus
 </DisplayName>
 </Grantee>
 <Permission>
 FULL_CONTROL
 </Permission>
 </Grant>
 </AccessControlList>
</AccessControlPolicy>

[ Object ACL for cfn-ubuntu-docker.json ]

<?xml version="1.0" ?>
<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
 <Owner>
 <ID>
 181f1e5b54f4a5b52b245124561c25104a5441623a29121d14531a4c1722b295
 </ID>
 <DisplayName>
 eucalyptus
 </DisplayName>
 </Owner>
 <AccessControlList>
 <Grant>
 <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser">
 <ID>
 2947352161f1a86121e325043564943471440401a2ac1d20534e40e114b3c244
 </ID>
 <DisplayName>
 account2
 </DisplayName>
 </Grantee>
 <Permission>
 READ
 </Permission>
 </Grant>
 <Grant>
 <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser">
 <ID>
 181f1e5b54f4a5b52b245124561c25104a5441623a29121d14531a4c1722b295
 </ID>
 <DisplayName>
 eucalyptus
 </DisplayName>
 </Grantee>
 <Permission>
 FULL_CONTROL
 </Permission>
 </Grant>
 </AccessControlList>
</AccessControlPolicy>

As you can see, the bucket and object ACL have been set to READ access by ‘account2’.  Just as with AWS S3, boto can be used to manage both ACLs and object lifecycles on Eucalyptus 4.0.x.

s3cmd

For cloud users who aren’t savvy with python, s3cmd can also interact with Eucalyptus 4.0.x OSG to manage ACLs and object lifecycles.  To get started, download the latest version of s3cmd from Github.   As mentioned in my previous blog regarding creating a configuration file for s3cmd, run ‘./s3cmd/s3cmd –configure‘ to create the configuration file for s3cmd.  After providing the Access Key ID and Secret Key for the Eucalyptus user, select not to test the credentials.  Open the configuration file (.s3cfg) with your favorite editor, and change the following attributes:

host_base = s3.amazonaws.com

to

host_base = <Eucalyptus OSG DNS Name>:8773

and

host_bucket = %(bucket)s.s3.amazonaws.com

to

host_bucket = %(bucket)s.<Eucalyptus OSG DNS Name>:8773

Once the configuration has been completed, make sure everything is working by listing the buckets owned by the Eucalyptus user account:

# ./s3cmd/s3cmd ls
2014-09-18 02:59 s3://51c700-download-manifests
2014-12-05 20:03 s3://access-logs-325271821652-hasp-euca-lb-435e09fa
2014-12-07 04:23 s3://access-logs-hasp-euca-lb-435e09fa
2014-12-11 03:59 s3://access_logs-hasp-euca-lb_10.104.7.10_d1eaaaef
2014-12-11 04:31 s3://access_logs-hasp-euca-lb_10.104.7.9_315dd646
2014-09-18 02:43 s3://centos-6.5-x86_64-20140917
2014-09-18 02:46 s3://centos-7-x86_64-20140917
2014-11-05 22:05 s3://centos6.4-kernel
2014-11-05 21:54 s3://centos6.4-ramdisk
2014-11-05 22:08 s3://centos6.4-test
2014-09-18 02:52 s3://debian-7-x86_64-20140917

If you get a bucket listing, everything is ready to go.  If not, please re-check your Eucalyptus user credentials, and make sure the user has the appropriate Eucalyptus IAM access rights.

Create a Bucket and Define an Object Lifecycle

To create a bucket with s3cmd, run the following command:

# ./s3cmd/s3cmd mb s3://test-s3cmd
Bucket 's3://test-s3cmd/' created

To apply the object lifecycle to the bucket, create a lifecycle configuration file based upon the AWS Lifecycle Configuration Elements.  For this example, download an example configuration file here:

  • http://bucket-acl-object-lifecycle-blog.s3.amazonaws.com/test-lifecycle-configuration.conf

Apply the configuration policy with s3cmd as follows:

# ./s3cmd/s3cmd setlifecycle test-lifecycle-configuration.conf s3://test-s3cmd

Upload an Object and Define an ACL for Another Account

Using the same Cloudformation template mentioned earlier, upload the file to the ‘test-s3cmd‘ bucket using s3cmd:

# ./s3cmd/s3cmd put cfn-ubuntu-docker.json s3://test-s3cmd/cfn-ubuntu-docker.json
cfn-ubuntu-docker.json -> s3://test-s3cmd/cfn-ubuntu-docker.json [1 of 1]
 2096 of 2096 100% in 2s 919.73 B/s done

As demonstrated in the python boto section, provide the READ ACL to the bucket ‘test-s3cmd‘ for ‘account2‘ (account ID ‘325271821652‘):

# ./s3cmd/s3cmd setacl --acl-grant="read:325271821652" s3://test-s3cmd
s3://test-s3cmd/: ACL updated

To apply the ACL to the ‘cfn-ubuntu-docker.json‘ file in the ‘test-s3cmd‘ bucket, use the same command, except add the bucket object:

# ./s3cmd/s3cmd --acl-grant="read:325271821652" s3://test-s3cmd/cfn-ubuntu-docker.json
s3://test-s3cmd/cfn-ubuntu-docker.json: ACL updated

Display Bucket and Object Attributes

Just as with python boto, the attributes of buckets and objects can be displayed with s3cmd as well.  However, since bucket policies haven’t been implemented yet in Eucalyptus, there will be a warning returned.  A series of warnings will be returned because s3cmd implements the AWS best practice of error retries and exponential backoff.

Use the ‘info‘ flag with s3cmd to display the bucket and object ACLs, and the object lifecycle defined on the bucket.  For example:

# ./s3cmd/s3cmd info s3://test-s3cmd/
s3://test-s3cmd/ (bucket):
 Location: us-east-1
 Expiration Rule: all objects in this bucket will expire in '3650' day(s) after creation
WARNING: Retrying failed request: /?policy
WARNING: 501 (NotImplemented): GET Bucket policy is not implemented
WARNING: Waiting 3 sec...
WARNING: Retrying failed request: /?policy
WARNING: 501 (NotImplemented): GET Bucket policy is not implemented
WARNING: Waiting 6 sec...
WARNING: Retrying failed request: /?policy
WARNING: 501 (NotImplemented): GET Bucket policy is not implemented
WARNING: Waiting 9 sec...
WARNING: Retrying failed request: /?policy
WARNING: 501 (NotImplemented): GET Bucket policy is not implemented
WARNING: Waiting 12 sec...
WARNING: Retrying failed request: /?policy
WARNING: 501 (NotImplemented): GET Bucket policy is not implemented
WARNING: Waiting 15 sec...
 policy: none
 ACL: account2: READ
 ACL: eucalyptus: FULL_CONTROL
# ./s3cmd/s3cmd info s3://test-s3cmd/cfn-ubuntu-docker.json
s3://test-s3cmd/cfn-ubuntu-docker.json (object):
 File size: 2096
 Last mod: Mon, 15 Dec 2014 05:17:03 GMT
 MIME type: application/octet-stream
 MD5 sum: 0472ada0b472c43e781d026343d66157
 SSE: NONE
WARNING: Retrying failed request: /?policy
WARNING: 501 (NotImplemented): GET Bucket policy is not implemented
WARNING: Waiting 3 sec...
WARNING: Retrying failed request: /?policy
WARNING: 501 (NotImplemented): GET Bucket policy is not implemented
WARNING: Waiting 6 sec...
WARNING: Retrying failed request: /?policy
WARNING: 501 (NotImplemented): GET Bucket policy is not implemented
WARNING: Waiting 9 sec...
WARNING: Retrying failed request: /?policy
WARNING: 501 (NotImplemented): GET Bucket policy is not implemented
WARNING: Waiting 12 sec...
WARNING: Retrying failed request: /?policy
WARNING: 501 (NotImplemented): GET Bucket policy is not implemented
WARNING: Waiting 15 sec...
 policy: none
 ACL: account2: READ
 ACL: eucalyptus: FULL_CONTROL

As you can see, the ACLs and object lifecycle can be displayed without a problem using s3cmd.

Closing the Loop

As demonstrated, both python boto and s3cmd can be used with Eucalyptus 4.0.x clouds to manage bucket and object ACLs, and object lifecycles.  It is very important to remember that when using this feature, please make sure that Eucalyptus DNS is enabled on the Eucalyptus 4.0.x cloud.  As mentioned earlier, these features can be used for various use cases – ranging from sharing custom Cloudformation templates with other accounts on the Eucalyptus cloud to controlling how long files stay in a bucket (which is very helpful if an instance is storing logs and/or backup files to a bucket).

Hope you enjoyed this entry.  Please let me know if there are any questions.  As always, feedback is greatly appreciated.

Enjoy!

Eucalyptus 4.0 OSG Bucket ACLs and Object Lifecycle Management using Python Boto and s3cmd
Blog at WordPress.com.
  • Follow Following
    • More Mind Spew-age from Harold Spencer Jr.
    • Already have a WordPress.com account? Log in now.
    • More Mind Spew-age from Harold Spencer Jr.
    • Customize
    • Follow Following
    • Sign up
    • Log in
    • Report this content
    • View site in Reader
    • Manage subscriptions
    • Collapse this bar