Skip to content

More Mind Spew-age from Harold Spencer Jr.

Random Information – sometimes techie, sometimes not..

Menu

  • About The Author
  • Interests
  • Private Cloud Computing
  • Infrastructure
  • System Adminstration
  • Private Cloud Computing

RSS All Things Distributed – Werner Vogels

  • Monoliths are not dinosaurs
  • Can autonomous trucks transform the global supply chain?
  • How AI coding companions will change the way developers work
  • AI doesn't plant trees
  • Demystifying LLMs with Amazon distinguished scientists
  • An introduction to generative AI with Swami Sivasubramanian
  • Is Australia the new epicenter for healthtech startups?
  • Bringing Digital People to life with autonomous animation
  • Melbourne gets a Region, a big trip, and a brain mapping startup
  • An album for each year - 2022 version

RSS Elastician – Mitch Garnaat

  • Back To The Future
  • Looking at Clouds from Both Sides Now
  • Don't reboot me, bro!
  • Mapping Requests to EC2 API Versions
  • Python and AWS Cookbook Available
  • Does Python Scale?
  • Accessing the Eucalyptus Community Cloud with boto
  • Accessing the Internet Archive with boto
  • All Your Website Are Belong to S3
  • It Takes a Village...

RSS Amazon Web Services Blog

  • Announcing the latest AWS Heroes – June 2023
  • A New Set of APIs for Amazon SQS Dead-Letter Queue Redrive
  • AWS Week in Review – Amazon Security Lake Now GA, New Actions on AWS Fault Injection Simulator, and More – June 5, 2023
  • New – AWS DMS Serverless: Automatically Provisions and Scales Capacity for Migration and Data Replication
  • New – Snowball Edge Storage Optimized Devices with More Storage and Bandwidth
  • AWS Week in Review – AWS Wickr, Amazon Redshift, Generative AI, and More – May 29, 2023
  • AWS Week in Review – AWS Documentation Updates, Amazon EventBridge is Faster, and More – May 22, 2023
  • Amazon SageMaker Geospatial Capabilities Now Generally Available with Security Updates and More Use Case Samples
  • New – Simplify the Investigation of AWS Security Findings with Amazon Detective
  • Retiring the AWS Documentation on GitHub

RSS Amazon Web Services Security Blog

  • AWS Security Profile: Matthew Campagna, Senior Principal, Security Engineering, AWS Cryptography
  • 2023 ISO and CSA STAR certificates now available with 8 new services and 1 new Region
  • AWS Security Profile – Cryptography Edition: Valerie Lambert, Senior Software Development Engineer
  • Our commitment to shared cybersecurity goals
  • Updated AWS Ramp-Up Guide available for security, identity, and compliance
  • New eBook: 5 Keys to Secure Enterprise Messaging
  • Announcing the AWS Blueprint for Ransomware Defense
  • Updated whitepaper available: Architecting for PCI DSS Segmentation and Scoping on AWS
  • AWS Security Profile: Ritesh Desai, GM, AWS Secrets Manager
  • Get custom data into Amazon Security Lake through ingesting Azure activity logs

RSS Google Developers Blog

  • Google I/O Extended watch parties & upcoming meetups
  • How Google Enables Experts To Innovate Developer Tools From Food To Music
  • #WeArePlay | Meet Tessa and Saasha from the UK, founders of waste-fighting app Olio
  • AAPI Heritage Month: How Web GDE Vickie Li views the importance of diversity
  • Celebrate Google’s Coding Competitions with a final round of programming fun
  • What’s new in Google Pay
  • How web GDE Erick Wendel forever changed Node.js with the support of the open-source community
  • Using Generative AI for Travel Inspiration and Discovery
  • Jetpack Compose Buttons for Google Pay and Google Wallet
  • How Web GDE Martine Dowden approaches web design from an accessibility perspective

RSS Google Cloud Platform Blog

  • What’s new with Google Cloud
  • Four steps to managing your Cloud Logging costs on a budget
  • How an open data cloud is enabling Airports of Thailand and EVme to reshape the future of travel
  • How to easily migrate your apps to containers — free deep dive and workshop
  • At Google I/O, generative AI gets to work
  • Dialing up the impact of digital natives in the MENA region with Google Cloud
  • Committed use discounts for RHEL and RHEL for SAP now available on Compute Engine
  • Introducing BigQuery Partner Center — a new way to discover and leverage integrated partner solutions
  • Bringing our world-class expertise together under Google Cloud Consulting
  • Accelerate time to value with Google’s Data Cloud for your industry

RSS Thinking Out Cloud

  • Ravello Launches the Cloud Application Hypervisor
  • State of the Open Source Cloud
  • Cloud Adoption Patterns - Citrix Synergy 2012 Keynote
  • In the Game of Clouds, You Win Or You Die: CloudStack
  • More on Bottom-Up Adoption of Cloud Computing and SaaS

Tags

acls ami ansible appscale autoscaling aws aws compatibility aws ec2 boto cloud-init cloud administrator cloud computing cloudformation coreos coreos cluster coreos etcd docker drbd ec2 elb emi emis.eucalyptus.com Euare euca2ools euca2ools 3 eucalyptus eucalyptus 2.0 eucalyptus 3.4 eucalyptus 4.0 eucalyptus 4.0.2 eucalyptus load balancer eustore eutester git haproxy high availability hybrid cloud iaas IAM image management ipython lvm mac os x mdb neo4j openldap paas Personal precise private cloud python boto s3 s3cmd stackato ubuntu cloud image ubuntu cloud images ubuntu raring varnish walrus xcode

Follow me on Twitter

My Tweets

cross-account access

Eucalyptus 4.0 OSG Bucket ACLs and Object Lifecycle Management using Python Boto and s3cmd

December 14, 2014December 14, 2014 hspencer77acls, aws compatibility, cloud computing, cross-account access, eucalyptus 4.0.x, object lifecycle management, python boto, s3cmd, third party AWS client tools1 Comment

Introduction

My last few blogs have discussed how Eucalyptus 4.0 supports similar Amazon Web Services, such as Elastic Compute Cloud (EC2), Simple Storage Services (S3), Elastic Load Balancing (ELB) and Security Token Services (STS) features. Using third party tools, such as python boto and s3cmd, features such as listing buckets and objects, listing running instances and assuming IAM roles were covered.  This blog entry is yet another example as to how Eucalyptus provides AWS compatible features in a private, on-premise cloud environment.

A couple of little known AWS-like features provided by the Eucalyptus 4.0.x Object Storage Gateway (OSG) that are barely discussed are the following:

  • Bucket and Object ACLs
  • Object Lifecycle Management

Both of these features help cloud users implement cross-account bucket and object access, and define how long an object lives in a bucket.  Keeping consistent with the earlier blog entries and continuing to promote Eucalyptus’s AWS compatibility, these features will be demonstrated using both python boto and s3cmd.

Requirements – Before We Begin

Before we get started, the following is needed:

  • Eucalyptus 4.0.x Cloud with Eucalyptus DNS enabled
  • Two Eucalyptus Euare (IAM) Accounts (this blog uses the ‘eucalyptus’ account and a non-‘eucalyptus’ account)
  • One set of Eucalyptus user credentials
  • IAM access policy applied to the Eucalyptus user to allow access to OSG (S3) API calls.
  • latest version of python boto
  • latest version of s3cmd

Both boto and s3cmd can be installed on the same client machine to make things easier.  Once the tasks above have been completed, we are ready to begin.

Python Boto

As mentioned before, past blog entries have covered how to use boto with Eucalyptus. To help demonstrate bucket/object ACLs and object lifecycle management using boto with Eucalyptus, I created a script – which can be downloaded.  Before using the script, make sure and change the following variables:

  • aws_access_key_id=“<Eucalyptus Access Key ID>”
  • aws_secret_access_key=“<Eucalyptus Secret Key>”
  • host=“<S3_URL – Eucalyptus OSG DNS Name>”

After updating the script, run the script with the –help option to see what arguments can be passed:

$ ./osg-boto-s3.py --help
usage: osg-boto-s3.py [-h] [-g ACCOUNT_ID] [-a ACL_PERM] [-r] [-l LIFECYCLE]
 [-d] [-o BUCKET_OBJECT]
 bucket

Script that sets grantee bucket (and optionally object) ACL and/or Object
Lifecycle on an OSG Bucket; refer to
http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html and
http://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html for
additional information.

positional arguments:
 bucket Eucalyptus OSG Bucket to apply Bucket ACL and/or
 Object Lifecycle

optional arguments:
 -h, --help show this help message and exit

Bucket ACL Arguments:
 Arguments to Define Bucket ACLs

 -g ACCOUNT_ID, --grantee ACCOUNT_ID
 Account ID of Grantee for Defining Bucket ACL
 -a ACL_PERM, --acl ACL_PERM
 Define ACL Permission: READ, WRITE, READ_ACP,
 WRITE_ACP, FULL_CONTROL
 -r, --recursive Option to pass if ACL is to be applied to objects in
 bucket; only applies if bucket already has objects

Object Lifecycle Arguments:
 Arguments to Define Object Lifecycle

 -l LIFECYCLE, --lifecycle LIFECYCLE
 Number of Days for Object Lifecycle Management

Bucket Attribute Display Argument:
 Option to Display Bucket ACLs and Object Lifecycle

 -d, --display Option to display bucket attributes for object
 lifecycle and bucket ACLs

Object Argument:
 Argument to Define Object (file) To Upload to Bucket

 -o BUCKET_OBJECT, --object BUCKET_OBJECT
 Object (file) to be uploaded to given bucket

The script does the following:

  • Accepts a bucket as an argument; if the bucket does not exist, the script will create it.
  • Applies an object lifecycle to the given bucket
  • Applies the desired ACL on bucket and/or objects in the bucket
  • Uploads an object to the given bucket
  • Displays bucket (and optionally object) ACLs and defined object lifecycle

Below are a couple of examples:

Create A Bucket and DEFINE an Object Lifecycle

This example will show how to create a bucket called “test-boto”, and apply an object lifecycle of 10 days.  The user credentials used are from the ‘admin’ user of the ‘eucalyptus’ account:

# ./osg-boto-s3.py --lifecycle 10 test-boto
# ./osg-boto-s3.py --display test-boto

[ Object Lifecycle Configuration for test-boto ]

<?xml version="1.0" ?>
<LifecycleConfiguration>
 <Rule>
 <ID>
 lifecycle-rule-test-boto
 </ID>
 <Prefix/>
 <Status>
 Enabled
 </Status>
 <Expiration>
 <Days>
 10
 </Days>
 </Expiration>
 </Rule>
</LifecycleConfiguration>

[ Bucket ACL for test-boto ]

<?xml version="1.0" ?>
<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
 <Owner>
 <ID>
 181f1e5b54f4a5b52b245124561c25104a5441623a29121d14531a4c1722b295
 </ID>
 <DisplayName>
 eucalyptus
 </DisplayName>
 </Owner>
 <AccessControlList>
 <Grant>
 <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser">
 <ID>
 181f1e5b54f4a5b52b245124561c25104a5441623a29121d14531a4c1722b295
 </ID>
 <DisplayName>
 eucalyptus
 </DisplayName>
 </Grantee>
 <Permission>
 FULL_CONTROL
 </Permission>
 </Grant>
 </AccessControlList>
</AccessControlPolicy>

As you can see, the bucket has been created, and using the ‘–display‘ option, it shows the ACLs and object lifecycle associated with the bucket.

Upload Object and Define ACL for Another Account

In this example, we will upload a file named cfn-ubuntu-docker.json (which is a Cloudformation template that can be used with Eucalyptus Cloudformation Service), and provide a READ ACL for the account ID ‘325271821652‘ – which has the account name of ‘account2‘.

# ./osg-boto-s3.py --object cfn-ubuntu-docker.json --grantee 325271821652 --acl READ --recursive test-boto
# ./osg-boto-s3.py --display test-boto --recursive

[ Object Lifecycle Configuration for test-boto ]

<?xml version="1.0" ?>
<LifecycleConfiguration>
 <Rule>
 <ID>
 lifecycle-rule-test-boto
 </ID>
 <Prefix/>
 <Status>
 Enabled
 </Status>
 <Expiration>
 <Days>
 10
 </Days>
 </Expiration>
 </Rule>
</LifecycleConfiguration>

[ Bucket ACL for test-boto ]

<?xml version="1.0" ?>
<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
 <Owner>
 <ID>
 181f1e5b54f4a5b52b245124561c25104a5441623a29121d14531a4c1722b295
 </ID>
 <DisplayName>
 eucalyptus
 </DisplayName>
 </Owner>
 <AccessControlList>
 <Grant>
 <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser">
 <ID>
 2947352161f1a86121e325043564943471440401a2ac1d20534e40e114b3c244
 </ID>
 <DisplayName>
 account2
 </DisplayName>
 </Grantee>
 <Permission>
 READ
 </Permission>
 </Grant>
 <Grant>
 <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser">
 <ID>
 181f1e5b54f4a5b52b245124561c25104a5441623a29121d14531a4c1722b295
 </ID>
 <DisplayName>
 eucalyptus
 </DisplayName>
 </Grantee>
 <Permission>
 FULL_CONTROL
 </Permission>
 </Grant>
 </AccessControlList>
</AccessControlPolicy>

[ Object ACL for cfn-ubuntu-docker.json ]

<?xml version="1.0" ?>
<AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
 <Owner>
 <ID>
 181f1e5b54f4a5b52b245124561c25104a5441623a29121d14531a4c1722b295
 </ID>
 <DisplayName>
 eucalyptus
 </DisplayName>
 </Owner>
 <AccessControlList>
 <Grant>
 <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser">
 <ID>
 2947352161f1a86121e325043564943471440401a2ac1d20534e40e114b3c244
 </ID>
 <DisplayName>
 account2
 </DisplayName>
 </Grantee>
 <Permission>
 READ
 </Permission>
 </Grant>
 <Grant>
 <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser">
 <ID>
 181f1e5b54f4a5b52b245124561c25104a5441623a29121d14531a4c1722b295
 </ID>
 <DisplayName>
 eucalyptus
 </DisplayName>
 </Grantee>
 <Permission>
 FULL_CONTROL
 </Permission>
 </Grant>
 </AccessControlList>
</AccessControlPolicy>

As you can see, the bucket and object ACL have been set to READ access by ‘account2’.  Just as with AWS S3, boto can be used to manage both ACLs and object lifecycles on Eucalyptus 4.0.x.

s3cmd

For cloud users who aren’t savvy with python, s3cmd can also interact with Eucalyptus 4.0.x OSG to manage ACLs and object lifecycles.  To get started, download the latest version of s3cmd from Github.   As mentioned in my previous blog regarding creating a configuration file for s3cmd, run ‘./s3cmd/s3cmd –configure‘ to create the configuration file for s3cmd.  After providing the Access Key ID and Secret Key for the Eucalyptus user, select not to test the credentials.  Open the configuration file (.s3cfg) with your favorite editor, and change the following attributes:

host_base = s3.amazonaws.com

to

host_base = <Eucalyptus OSG DNS Name>:8773

and

host_bucket = %(bucket)s.s3.amazonaws.com

to

host_bucket = %(bucket)s.<Eucalyptus OSG DNS Name>:8773

Once the configuration has been completed, make sure everything is working by listing the buckets owned by the Eucalyptus user account:

# ./s3cmd/s3cmd ls
2014-09-18 02:59 s3://51c700-download-manifests
2014-12-05 20:03 s3://access-logs-325271821652-hasp-euca-lb-435e09fa
2014-12-07 04:23 s3://access-logs-hasp-euca-lb-435e09fa
2014-12-11 03:59 s3://access_logs-hasp-euca-lb_10.104.7.10_d1eaaaef
2014-12-11 04:31 s3://access_logs-hasp-euca-lb_10.104.7.9_315dd646
2014-09-18 02:43 s3://centos-6.5-x86_64-20140917
2014-09-18 02:46 s3://centos-7-x86_64-20140917
2014-11-05 22:05 s3://centos6.4-kernel
2014-11-05 21:54 s3://centos6.4-ramdisk
2014-11-05 22:08 s3://centos6.4-test
2014-09-18 02:52 s3://debian-7-x86_64-20140917

If you get a bucket listing, everything is ready to go.  If not, please re-check your Eucalyptus user credentials, and make sure the user has the appropriate Eucalyptus IAM access rights.

Create a Bucket and Define an Object Lifecycle

To create a bucket with s3cmd, run the following command:

# ./s3cmd/s3cmd mb s3://test-s3cmd
Bucket 's3://test-s3cmd/' created

To apply the object lifecycle to the bucket, create a lifecycle configuration file based upon the AWS Lifecycle Configuration Elements.  For this example, download an example configuration file here:

  • http://bucket-acl-object-lifecycle-blog.s3.amazonaws.com/test-lifecycle-configuration.conf

Apply the configuration policy with s3cmd as follows:

# ./s3cmd/s3cmd setlifecycle test-lifecycle-configuration.conf s3://test-s3cmd

Upload an Object and Define an ACL for Another Account

Using the same Cloudformation template mentioned earlier, upload the file to the ‘test-s3cmd‘ bucket using s3cmd:

# ./s3cmd/s3cmd put cfn-ubuntu-docker.json s3://test-s3cmd/cfn-ubuntu-docker.json
cfn-ubuntu-docker.json -> s3://test-s3cmd/cfn-ubuntu-docker.json [1 of 1]
 2096 of 2096 100% in 2s 919.73 B/s done

As demonstrated in the python boto section, provide the READ ACL to the bucket ‘test-s3cmd‘ for ‘account2‘ (account ID ‘325271821652‘):

# ./s3cmd/s3cmd setacl --acl-grant="read:325271821652" s3://test-s3cmd
s3://test-s3cmd/: ACL updated

To apply the ACL to the ‘cfn-ubuntu-docker.json‘ file in the ‘test-s3cmd‘ bucket, use the same command, except add the bucket object:

# ./s3cmd/s3cmd --acl-grant="read:325271821652" s3://test-s3cmd/cfn-ubuntu-docker.json
s3://test-s3cmd/cfn-ubuntu-docker.json: ACL updated

Display Bucket and Object Attributes

Just as with python boto, the attributes of buckets and objects can be displayed with s3cmd as well.  However, since bucket policies haven’t been implemented yet in Eucalyptus, there will be a warning returned.  A series of warnings will be returned because s3cmd implements the AWS best practice of error retries and exponential backoff.

Use the ‘info‘ flag with s3cmd to display the bucket and object ACLs, and the object lifecycle defined on the bucket.  For example:

# ./s3cmd/s3cmd info s3://test-s3cmd/
s3://test-s3cmd/ (bucket):
 Location: us-east-1
 Expiration Rule: all objects in this bucket will expire in '3650' day(s) after creation
WARNING: Retrying failed request: /?policy
WARNING: 501 (NotImplemented): GET Bucket policy is not implemented
WARNING: Waiting 3 sec...
WARNING: Retrying failed request: /?policy
WARNING: 501 (NotImplemented): GET Bucket policy is not implemented
WARNING: Waiting 6 sec...
WARNING: Retrying failed request: /?policy
WARNING: 501 (NotImplemented): GET Bucket policy is not implemented
WARNING: Waiting 9 sec...
WARNING: Retrying failed request: /?policy
WARNING: 501 (NotImplemented): GET Bucket policy is not implemented
WARNING: Waiting 12 sec...
WARNING: Retrying failed request: /?policy
WARNING: 501 (NotImplemented): GET Bucket policy is not implemented
WARNING: Waiting 15 sec...
 policy: none
 ACL: account2: READ
 ACL: eucalyptus: FULL_CONTROL
# ./s3cmd/s3cmd info s3://test-s3cmd/cfn-ubuntu-docker.json
s3://test-s3cmd/cfn-ubuntu-docker.json (object):
 File size: 2096
 Last mod: Mon, 15 Dec 2014 05:17:03 GMT
 MIME type: application/octet-stream
 MD5 sum: 0472ada0b472c43e781d026343d66157
 SSE: NONE
WARNING: Retrying failed request: /?policy
WARNING: 501 (NotImplemented): GET Bucket policy is not implemented
WARNING: Waiting 3 sec...
WARNING: Retrying failed request: /?policy
WARNING: 501 (NotImplemented): GET Bucket policy is not implemented
WARNING: Waiting 6 sec...
WARNING: Retrying failed request: /?policy
WARNING: 501 (NotImplemented): GET Bucket policy is not implemented
WARNING: Waiting 9 sec...
WARNING: Retrying failed request: /?policy
WARNING: 501 (NotImplemented): GET Bucket policy is not implemented
WARNING: Waiting 12 sec...
WARNING: Retrying failed request: /?policy
WARNING: 501 (NotImplemented): GET Bucket policy is not implemented
WARNING: Waiting 15 sec...
 policy: none
 ACL: account2: READ
 ACL: eucalyptus: FULL_CONTROL

As you can see, the ACLs and object lifecycle can be displayed without a problem using s3cmd.

Closing the Loop

As demonstrated, both python boto and s3cmd can be used with Eucalyptus 4.0.x clouds to manage bucket and object ACLs, and object lifecycles.  It is very important to remember that when using this feature, please make sure that Eucalyptus DNS is enabled on the Eucalyptus 4.0.x cloud.  As mentioned earlier, these features can be used for various use cases – ranging from sharing custom Cloudformation templates with other accounts on the Eucalyptus cloud to controlling how long files stay in a bucket (which is very helpful if an instance is storing logs and/or backup files to a bucket).

Hope you enjoyed this entry.  Please let me know if there are any questions.  As always, feedback is greatly appreciated.

Enjoy!

Eucalyptus 4.0 OSG Bucket ACLs and Object Lifecycle Management using Python Boto and s3cmd
Blog at WordPress.com.
  • Follow Following
    • More Mind Spew-age from Harold Spencer Jr.
    • Already have a WordPress.com account? Log in now.
    • More Mind Spew-age from Harold Spencer Jr.
    • Customize
    • Follow Following
    • Sign up
    • Log in
    • Report this content
    • View site in Reader
    • Manage subscriptions
    • Collapse this bar