Introduction
My last few blogs have discussed how Eucalyptus 4.0 supports similar Amazon Web Services, such as Elastic Compute Cloud (EC2), Simple Storage Services (S3), Elastic Load Balancing (ELB) and Security Token Services (STS) features. Using third party tools, such as python boto and s3cmd, features such as listing buckets and objects, listing running instances and assuming IAM roles were covered. This blog entry is yet another example as to how Eucalyptus provides AWS compatible features in a private, on-premise cloud environment.
A couple of little known AWS-like features provided by the Eucalyptus 4.0.x Object Storage Gateway (OSG) that are barely discussed are the following:
Both of these features help cloud users implement cross-account bucket and object access, and define how long an object lives in a bucket. Keeping consistent with the earlier blog entries and continuing to promote Eucalyptus’s AWS compatibility, these features will be demonstrated using both python boto and s3cmd.
Requirements – Before We Begin
Before we get started, the following is needed:
- Eucalyptus 4.0.x Cloud with Eucalyptus DNS enabled
- Two Eucalyptus Euare (IAM) Accounts (this blog uses the ‘eucalyptus’ account and a non-‘eucalyptus’ account)
- One set of Eucalyptus user credentials
- IAM access policy applied to the Eucalyptus user to allow access to OSG (S3) API calls.
- latest version of python boto
- latest version of s3cmd
Both boto and s3cmd can be installed on the same client machine to make things easier. Once the tasks above have been completed, we are ready to begin.
Python Boto
As mentioned before, past blog entries have covered how to use boto with Eucalyptus. To help demonstrate bucket/object ACLs and object lifecycle management using boto with Eucalyptus, I created a script – which can be downloaded. Before using the script, make sure and change the following variables:
- aws_access_key_id=“<Eucalyptus Access Key ID>”
- aws_secret_access_key=“<Eucalyptus Secret Key>”
- host=“<S3_URL – Eucalyptus OSG DNS Name>”
After updating the script, run the script with the –help option to see what arguments can be passed:
$ ./osg-boto-s3.py --help usage: osg-boto-s3.py [-h] [-g ACCOUNT_ID] [-a ACL_PERM] [-r] [-l LIFECYCLE] [-d] [-o BUCKET_OBJECT] bucket Script that sets grantee bucket (and optionally object) ACL and/or Object Lifecycle on an OSG Bucket; refer to http://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html and http://docs.aws.amazon.com/AmazonS3/latest/dev/object-lifecycle-mgmt.html for additional information. positional arguments: bucket Eucalyptus OSG Bucket to apply Bucket ACL and/or Object Lifecycle optional arguments: -h, --help show this help message and exit Bucket ACL Arguments: Arguments to Define Bucket ACLs -g ACCOUNT_ID, --grantee ACCOUNT_ID Account ID of Grantee for Defining Bucket ACL -a ACL_PERM, --acl ACL_PERM Define ACL Permission: READ, WRITE, READ_ACP, WRITE_ACP, FULL_CONTROL -r, --recursive Option to pass if ACL is to be applied to objects in bucket; only applies if bucket already has objects Object Lifecycle Arguments: Arguments to Define Object Lifecycle -l LIFECYCLE, --lifecycle LIFECYCLE Number of Days for Object Lifecycle Management Bucket Attribute Display Argument: Option to Display Bucket ACLs and Object Lifecycle -d, --display Option to display bucket attributes for object lifecycle and bucket ACLs Object Argument: Argument to Define Object (file) To Upload to Bucket -o BUCKET_OBJECT, --object BUCKET_OBJECT Object (file) to be uploaded to given bucket
The script does the following:
- Accepts a bucket as an argument; if the bucket does not exist, the script will create it.
- Applies an object lifecycle to the given bucket
- Applies the desired ACL on bucket and/or objects in the bucket
- Uploads an object to the given bucket
- Displays bucket (and optionally object) ACLs and defined object lifecycle
Below are a couple of examples:
Create A Bucket and DEFINE an Object Lifecycle
This example will show how to create a bucket called “test-boto”, and apply an object lifecycle of 10 days. The user credentials used are from the ‘admin’ user of the ‘eucalyptus’ account:
# ./osg-boto-s3.py --lifecycle 10 test-boto # ./osg-boto-s3.py --display test-boto [ Object Lifecycle Configuration for test-boto ] <?xml version="1.0" ?> <LifecycleConfiguration> <Rule> <ID> lifecycle-rule-test-boto </ID> <Prefix/> <Status> Enabled </Status> <Expiration> <Days> 10 </Days> </Expiration> </Rule> </LifecycleConfiguration> [ Bucket ACL for test-boto ] <?xml version="1.0" ?> <AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Owner> <ID> 181f1e5b54f4a5b52b245124561c25104a5441623a29121d14531a4c1722b295 </ID> <DisplayName> eucalyptus </DisplayName> </Owner> <AccessControlList> <Grant> <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"> <ID> 181f1e5b54f4a5b52b245124561c25104a5441623a29121d14531a4c1722b295 </ID> <DisplayName> eucalyptus </DisplayName> </Grantee> <Permission> FULL_CONTROL </Permission> </Grant> </AccessControlList> </AccessControlPolicy>
As you can see, the bucket has been created, and using the ‘–display‘ option, it shows the ACLs and object lifecycle associated with the bucket.
Upload Object and Define ACL for Another Account
In this example, we will upload a file named cfn-ubuntu-docker.json (which is a Cloudformation template that can be used with Eucalyptus Cloudformation Service), and provide a READ ACL for the account ID ‘325271821652‘ – which has the account name of ‘account2‘.
# ./osg-boto-s3.py --object cfn-ubuntu-docker.json --grantee 325271821652 --acl READ --recursive test-boto # ./osg-boto-s3.py --display test-boto --recursive [ Object Lifecycle Configuration for test-boto ] <?xml version="1.0" ?> <LifecycleConfiguration> <Rule> <ID> lifecycle-rule-test-boto </ID> <Prefix/> <Status> Enabled </Status> <Expiration> <Days> 10 </Days> </Expiration> </Rule> </LifecycleConfiguration> [ Bucket ACL for test-boto ] <?xml version="1.0" ?> <AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Owner> <ID> 181f1e5b54f4a5b52b245124561c25104a5441623a29121d14531a4c1722b295 </ID> <DisplayName> eucalyptus </DisplayName> </Owner> <AccessControlList> <Grant> <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"> <ID> 2947352161f1a86121e325043564943471440401a2ac1d20534e40e114b3c244 </ID> <DisplayName> account2 </DisplayName> </Grantee> <Permission> READ </Permission> </Grant> <Grant> <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"> <ID> 181f1e5b54f4a5b52b245124561c25104a5441623a29121d14531a4c1722b295 </ID> <DisplayName> eucalyptus </DisplayName> </Grantee> <Permission> FULL_CONTROL </Permission> </Grant> </AccessControlList> </AccessControlPolicy> [ Object ACL for cfn-ubuntu-docker.json ] <?xml version="1.0" ?> <AccessControlPolicy xmlns="http://s3.amazonaws.com/doc/2006-03-01/"> <Owner> <ID> 181f1e5b54f4a5b52b245124561c25104a5441623a29121d14531a4c1722b295 </ID> <DisplayName> eucalyptus </DisplayName> </Owner> <AccessControlList> <Grant> <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"> <ID> 2947352161f1a86121e325043564943471440401a2ac1d20534e40e114b3c244 </ID> <DisplayName> account2 </DisplayName> </Grantee> <Permission> READ </Permission> </Grant> <Grant> <Grantee xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:type="CanonicalUser"> <ID> 181f1e5b54f4a5b52b245124561c25104a5441623a29121d14531a4c1722b295 </ID> <DisplayName> eucalyptus </DisplayName> </Grantee> <Permission> FULL_CONTROL </Permission> </Grant> </AccessControlList> </AccessControlPolicy>
As you can see, the bucket and object ACL have been set to READ access by ‘account2’. Just as with AWS S3, boto can be used to manage both ACLs and object lifecycles on Eucalyptus 4.0.x.
s3cmd
For cloud users who aren’t savvy with python, s3cmd can also interact with Eucalyptus 4.0.x OSG to manage ACLs and object lifecycles. To get started, download the latest version of s3cmd from Github. As mentioned in my previous blog regarding creating a configuration file for s3cmd, run ‘./s3cmd/s3cmd –configure‘ to create the configuration file for s3cmd. After providing the Access Key ID and Secret Key for the Eucalyptus user, select not to test the credentials. Open the configuration file (.s3cfg) with your favorite editor, and change the following attributes:
host_base = s3.amazonaws.com
to
host_base = <Eucalyptus OSG DNS Name>:8773
and
host_bucket = %(bucket)s.s3.amazonaws.com
to
host_bucket = %(bucket)s.<Eucalyptus OSG DNS Name>:8773
Once the configuration has been completed, make sure everything is working by listing the buckets owned by the Eucalyptus user account:
# ./s3cmd/s3cmd ls 2014-09-18 02:59 s3://51c700-download-manifests 2014-12-05 20:03 s3://access-logs-325271821652-hasp-euca-lb-435e09fa 2014-12-07 04:23 s3://access-logs-hasp-euca-lb-435e09fa 2014-12-11 03:59 s3://access_logs-hasp-euca-lb_10.104.7.10_d1eaaaef 2014-12-11 04:31 s3://access_logs-hasp-euca-lb_10.104.7.9_315dd646 2014-09-18 02:43 s3://centos-6.5-x86_64-20140917 2014-09-18 02:46 s3://centos-7-x86_64-20140917 2014-11-05 22:05 s3://centos6.4-kernel 2014-11-05 21:54 s3://centos6.4-ramdisk 2014-11-05 22:08 s3://centos6.4-test 2014-09-18 02:52 s3://debian-7-x86_64-20140917
If you get a bucket listing, everything is ready to go. If not, please re-check your Eucalyptus user credentials, and make sure the user has the appropriate Eucalyptus IAM access rights.
Create a Bucket and Define an Object Lifecycle
To create a bucket with s3cmd, run the following command:
# ./s3cmd/s3cmd mb s3://test-s3cmd Bucket 's3://test-s3cmd/' created
To apply the object lifecycle to the bucket, create a lifecycle configuration file based upon the AWS Lifecycle Configuration Elements. For this example, download an example configuration file here:
Apply the configuration policy with s3cmd as follows:
# ./s3cmd/s3cmd setlifecycle test-lifecycle-configuration.conf s3://test-s3cmd
Upload an Object and Define an ACL for Another Account
Using the same Cloudformation template mentioned earlier, upload the file to the ‘test-s3cmd‘ bucket using s3cmd:
# ./s3cmd/s3cmd put cfn-ubuntu-docker.json s3://test-s3cmd/cfn-ubuntu-docker.json cfn-ubuntu-docker.json -> s3://test-s3cmd/cfn-ubuntu-docker.json [1 of 1] 2096 of 2096 100% in 2s 919.73 B/s done
As demonstrated in the python boto section, provide the READ ACL to the bucket ‘test-s3cmd‘ for ‘account2‘ (account ID ‘325271821652‘):
# ./s3cmd/s3cmd setacl --acl-grant="read:325271821652" s3://test-s3cmd s3://test-s3cmd/: ACL updated
To apply the ACL to the ‘cfn-ubuntu-docker.json‘ file in the ‘test-s3cmd‘ bucket, use the same command, except add the bucket object:
# ./s3cmd/s3cmd --acl-grant="read:325271821652" s3://test-s3cmd/cfn-ubuntu-docker.json s3://test-s3cmd/cfn-ubuntu-docker.json: ACL updated
Display Bucket and Object Attributes
Just as with python boto, the attributes of buckets and objects can be displayed with s3cmd as well. However, since bucket policies haven’t been implemented yet in Eucalyptus, there will be a warning returned. A series of warnings will be returned because s3cmd implements the AWS best practice of error retries and exponential backoff.
Use the ‘info‘ flag with s3cmd to display the bucket and object ACLs, and the object lifecycle defined on the bucket. For example:
# ./s3cmd/s3cmd info s3://test-s3cmd/ s3://test-s3cmd/ (bucket): Location: us-east-1 Expiration Rule: all objects in this bucket will expire in '3650' day(s) after creation WARNING: Retrying failed request: /?policy WARNING: 501 (NotImplemented): GET Bucket policy is not implemented WARNING: Waiting 3 sec... WARNING: Retrying failed request: /?policy WARNING: 501 (NotImplemented): GET Bucket policy is not implemented WARNING: Waiting 6 sec... WARNING: Retrying failed request: /?policy WARNING: 501 (NotImplemented): GET Bucket policy is not implemented WARNING: Waiting 9 sec... WARNING: Retrying failed request: /?policy WARNING: 501 (NotImplemented): GET Bucket policy is not implemented WARNING: Waiting 12 sec... WARNING: Retrying failed request: /?policy WARNING: 501 (NotImplemented): GET Bucket policy is not implemented WARNING: Waiting 15 sec... policy: none ACL: account2: READ ACL: eucalyptus: FULL_CONTROL
# ./s3cmd/s3cmd info s3://test-s3cmd/cfn-ubuntu-docker.json s3://test-s3cmd/cfn-ubuntu-docker.json (object): File size: 2096 Last mod: Mon, 15 Dec 2014 05:17:03 GMT MIME type: application/octet-stream MD5 sum: 0472ada0b472c43e781d026343d66157 SSE: NONE WARNING: Retrying failed request: /?policy WARNING: 501 (NotImplemented): GET Bucket policy is not implemented WARNING: Waiting 3 sec... WARNING: Retrying failed request: /?policy WARNING: 501 (NotImplemented): GET Bucket policy is not implemented WARNING: Waiting 6 sec... WARNING: Retrying failed request: /?policy WARNING: 501 (NotImplemented): GET Bucket policy is not implemented WARNING: Waiting 9 sec... WARNING: Retrying failed request: /?policy WARNING: 501 (NotImplemented): GET Bucket policy is not implemented WARNING: Waiting 12 sec... WARNING: Retrying failed request: /?policy WARNING: 501 (NotImplemented): GET Bucket policy is not implemented WARNING: Waiting 15 sec... policy: none ACL: account2: READ ACL: eucalyptus: FULL_CONTROL
As you can see, the ACLs and object lifecycle can be displayed without a problem using s3cmd.
Closing the Loop
As demonstrated, both python boto and s3cmd can be used with Eucalyptus 4.0.x clouds to manage bucket and object ACLs, and object lifecycles. It is very important to remember that when using this feature, please make sure that Eucalyptus DNS is enabled on the Eucalyptus 4.0.x cloud. As mentioned earlier, these features can be used for various use cases – ranging from sharing custom Cloudformation templates with other accounts on the Eucalyptus cloud to controlling how long files stay in a bucket (which is very helpful if an instance is storing logs and/or backup files to a bucket).
Hope you enjoyed this entry. Please let me know if there are any questions. As always, feedback is greatly appreciated.
Enjoy!
More Mind Spew-age from Harold Spencer Jr. 