Making a Less-Limited USB Stick

hspencer77:

Some nice information about UDF

Originally posted on /dev/zero:

The FAT32 filesystem is the closest thing we have to a universal standard for passing data around, but with the capacity of modern USB flash drives its 4 GB file size limitation has become problematic. exFAT is a popular contender for dealing with that, but the patent issues that surround it make true portability a pipe dream at best.

Enter UDF. As the filesystem of choice for DVDs and Blu-Ray disks, UDF support is ubiquitous. Appropriately-formatted disks are readable by operating systems dating back to the early 2000s. All that remains is figuring out how to format it. In general, there seem to be three important things to keep in mind:

  1. Remove all traces of previous filesystems. Different operating systems use different methods to detect what filesystems a disk contains, so ensure maximum reliability by eliminating potential sources of confusion.
  2. Format the entire disk, not just a partition. OS X…

View original 198 more words

Ubuntu Cloud Images and Eustore: The Eucalyptus Cloud Administrator’s Image Management Dream

Ubuntu provides versatile cloud images that can be utilized on various cloud deployment infrastructures.  Eucalyptus’s euca2ools eustore tool makes it easier for cloud administrator’s to bundle, upload and register images on Eucalyptus clouds.  Using eustore-install-image with Ubuntu Cloud images provides the best of both worlds – solid cloud images that can be easily deployed to any Eucalyptus cloud environment.

Set Up Euca2ools Configuration File

Setting up a euca2ools configuration file makes it easier and more efficient to interact with the euca2ools commands.  For this blog entry, the euca2ools configuration file ~/.euca/euca2ools.ini was set up with the following information:

[global]
default-region = LayinDaSmackDown
[user admin]
key-id = L4836KVYWMJCXT4T6Q6B9
secret-key = XCJ6sZVFVfFMR4DNVIUL7N7e4cgk8ebvEW0ej5dZ
account-id = 441445882805
private-key = /home/hspencer/admin-creds/euca2-admin-c9e4580c-pk.pem
certificate = /home/hspencer/admin-creds/euca2-admin-c9e4580c-cert.pem
[region LayinDaSmackDown]
autoscaling-url = http://10.104.1.216:8773/services/AutoScaling
ec2-url = http://10.104.1.216:8773/services/Eucalyptus
elasticloadbalancing-url = http://10.104.1.216:8773/services/LoadBalancing
iam-url = http://10.104.1.216:8773/services/Euare
monitoring-url = http://10.104.1.216:8773/services/CloudWatch
s3-url = http://10.104.1.216:8773/services/Walrus
eustore-url =http://emis.eucalyptus.com/
certificate = /home/hspencer/admin-creds/cloud-cert.pem

After setting up the euca2ools confirmation file, to utilize this file, run any euca2ools command with the –region option.  For example:

$ euca-describe-images --region admin@

Since kernel, ramdisk and root filesystem images will be bundled, uploaded and registered, the cloud administrator’s credentials were used in the euca2ools configuration file.  For more information about kernel and ramdisk management in Eucalyptus, check out the KB article entitled “Kernel and Ramdisk Management in Eucalyptus”.

Ubuntu Cloud Images

To obtain Ubuntu Cloud images that will be bundled, uploaded and registered with eustore-install-image, download the supported Ubuntu Cloud image release of your choice from the Ubuntu Cloud Images page. Download the file thats ends with either amd64.tar.gz or i386.tar.gz using either curl, wget or any other network transfer tool.

For example, to download the latest Ubuntu 13.10 Saucy Salamander Cloud image, run the following command:

$ wget http://cloud-images.ubuntu.com/saucy/current/saucy-server-cloudimg-amd64.tar.gz

This will download and save the saucy-server-cloudimg-amd64.tar.gz file.  Now its time to use eustore-install-image to bundle, upload and register the image.

Bundle, Upload, Register The Image

As mentioned earlier, eustore-install-image will bundle, upload and register the image.  To do this, use the –tarball option of eustore-install-image with the tar-gzipped file downloaded from Ubuntu Cloud Image page.  The key flag here is the –hypervisor option.  Because Ubuntu Cloud images are crafted to work on multiple hypervisors (e.g. Xen, KVM, VMware, etc.), set the –hypervisor option to “universal”. Here is an example of using these options:

$ eustore-install-image -t saucy-server-cloudimg-amd64.tar.gz -b ubuntu-saucy-server-amd64 --hypervisor universal -s "Ubuntu 13.10 - Saucy Salamander" -p ubuntu-saucy --region admin@ -a x86_64
Preparing to extract image...
Extracting kernel 100% |=========================================================================================| 5.34 MB 158.76 MB/s Time: 0:00:00
Bundling kernel 100% |=========================================================================================| 5.34 MB 27.39 MB/s Time: 0:00:00
-- Uploading kernel image --
saucy-server-cloudimg-amd64-vmlinuz-generic.part.0 100% |=========================================================| 5.28 MB 20.91 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64-vmlinuz-generic.manifest.xml 100% |===================================================| 3.45 kB 26.26 kB/s Time: 0:00:00
Registered kernel image eki-2C54378B
Extracting ramdisk 100% |=========================================================================================| 89.56 kB 45.29 MB/s Time: 0:00:00
Bundling ramdisk 100% |=========================================================================================| 89.56 kB 8.31 MB/s Time: 0:00:00
-- Uploading ramdisk image --
saucy-server-cloudimg-amd64-loader.part.0 100% |==================================================================| 89.38 kB 689.09 kB/s Time: 0:00:00
saucy-server-cloudimg-amd64-loader.manifest.xml 100% |============================================================| 3.36 kB 26.03 kB/s Time: 0:00:00
Registered ramdisk image eri-015435B7
Extracting image 100% |=========================================================================================| 1.38 GB 183.45 MB/s Time: 0:00:08
Bundling image 100% |=========================================================================================| 1.38 GB 34.33 MB/s Time: 0:00:43
-- Uploading machine image --
saucy-server-cloudimg-amd64.img.part.0 ( 1/22) 100% |============================================================| 10.00 MB 42.72 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.1 ( 2/22) 100% |============================================================| 10.00 MB 41.06 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.2 ( 3/22) 100% |============================================================| 10.00 MB 42.43 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.3 ( 4/22) 100% |============================================================| 10.00 MB 44.56 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.4 ( 5/22) 100% |============================================================| 10.00 MB 48.43 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.5 ( 6/22) 100% |============================================================| 10.00 MB 57.00 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.6 ( 7/22) 100% |============================================================| 10.00 MB 48.69 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.7 ( 8/22) 100% |============================================================| 10.00 MB 51.00 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.8 ( 9/22) 100% |============================================================| 10.00 MB 43.92 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.9 (10/22) 100% |============================================================| 10.00 MB 46.26 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.10 (11/22) 100% |============================================================| 10.00 MB 46.27 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.11 (12/22) 100% |============================================================| 10.00 MB 48.74 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.12 (13/22) 100% |============================================================| 10.00 MB 44.48 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.13 (14/22) 100% |============================================================| 10.00 MB 44.20 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.14 (15/22) 100% |============================================================| 10.00 MB 48.89 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.15 (16/22) 100% |============================================================| 10.00 MB 46.45 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.16 (17/22) 100% |============================================================| 10.00 MB 56.84 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.17 (18/22) 100% |============================================================| 10.00 MB 53.62 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.18 (19/22) 100% |============================================================| 10.00 MB 56.75 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.19 (20/22) 100% |============================================================| 10.00 MB 48.67 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.20 (21/22) 100% |============================================================| 10.00 MB 44.25 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.21 (22/22) 100% |============================================================| 3.19 MB 18.21 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.manifest.xml 100% |===============================================================| 6.63 kB 41.32 kB/s Time: 0:00:00
Registered machine image emi-4EFE3A91
-- Done --
Installed new image emi-4EFE3A91

Now that the kernel, ramdisk and root filesystem are bundled, uploaded and registered, the launch permission of each image needs to be changed so that all the users of the Eucalyptus cloud can launch instances from these images.

$ euca-modify-image-attribute -l -a all emi-4EFE3A91 --region admin@
launchPermission emi-4EFE3A91 ADD Group all
$ euca-modify-image-attribute -l -a all eri-015435B7 --region admin@
launchPermission eri-015435B7 ADD Group all
$ euca-modify-image-attribute -l -a all eki-2C54378B --region admin@
launchPermission eki-2C54378B ADD Group all

Thats it!  Now users can launch instances from the EMI, EKI and ERI as below:

$ euca-run-instances -k account1-user01 -t m1.medium emi-4EFE3A91 --region account1-user01@ --user-data-file cloud-init.config

Enjoy!

Step-by-Step Deployment of Docker on Eucalyptus 3.4 for the Cloud Administrator

Docker

Eucalyptus Systems, Inc.

Docker has been in the news lately as one of the hot open-source project promoting linux containers. Some use cases for Docker include the following:

  • Automation of packaging and application deployment
  • Lightweight PaaS environments
  • Automated testing and continuous integration/deployment
  • Deploying and scaling web applications, databases and backend services

The focus of this blog entry is to show how to deploy Docker on Eucalyptus from a cloud administrator’s point-of-view – all in the cloud.  This is a step-by-step guide to create an Docker EMI from an existing Ubuntu Cloud Raring EMI using AWS’s documentation.  This entry will also show how to build euca2ools from source in the Ubuntu Cloud image.

Prerequisites

This entry assumes the following:

After confirming that the prerequisites are met, let’s get started.

Creating an EMI From an Existing EMI

As mentioned earlier, these steps will be based off of  AWS’s documentation on creating an instance store-backed AMI from an existing AMI.  In this example, here is an existing Ubuntu Raring instance thats running on Eucalyptus:

$ euca-describe-instances --region eucalyptus-admin@
RESERVATION r-3E423E33 961915002812 default
INSTANCE i-827E3E88 emi-06663A57 euca-10-104-7-12.eucalyptus.euca-hasp.eucalyptus-systems.com
euca-172-17-118-27.eucalyptus.internal running euca-admin 0 m1.medium 
2013-11-18T22:41:35.694Z LayinDaSmackDown eki-28F338EB eri-51253C0A 
monitoring-disabled 10.104.7.12 172.17.118.27 instance-store

This instance is using the following EMI, EKI and ERI:

$ euca-describe-images emi-06663A57 eki-28F338EB eri-51253C0A --region eucalyptus-admin@
IMAGE eki-28F338EB latest-raring-kernel/raring-server-cloudimg-amd64-vmlinuz-generic.manifest.xml 
441445882805 available public x86_64 kernel instance-store
IMAGE emi-06663A57 latest-raring/raring-server-cloudimg-amd64.img.manifest.xml 441445882805 
available public x86_64 machine eki-28F338EB eri-51253C0A instance-store paravirtualized
IMAGE eri-51253C0A latest-raring-kernel/raring-server-cloudimg-amd64-loader.manifest.xml 441445882805 
available public x86_64 ramdisk instance-store

To start, copy the zip file cloud administrator credentials obtained by the euca_conf command mentioned in the Eucalyptus 3.4 documentation to the running instance:

# scp -i euca-admin.priv admin.zip 
ubuntu@euca-10-104-7-12.eucalyptus.euca-hasp.eucalyptus-systems.com:/tmp/.

Next, install the following packages for the 3.8.0-33 kernel, and packages needed to build euca2ools:

ubuntu@euca-172-17-118-27:~$ sudo apt-get install python-setuptools git python-lxml unzip linux-headers-3.8.0-33-generic linux-image-extra-3.8.0-33-generic

Find the ephemeral storage using the instance metadata service, format, and mount the ephemeral to /mnt/image:

ubuntu@euca-172-17-118-27:~$ curl http://169.254.169.254/latest/meta-data/block-device-mapping/ephemeral
sda2
ubuntu@euca-172-17-118-27:~$ sudo mkdir /mnt/image
ubuntu@euca-172-17-118-27:~$ sudo mkfs.ext4 /dev/vda2
ubuntu@euca-172-17-118-27:~$ sudo mount /dev/vda2 /mnt/image

Download euca2ools from Github:

ubuntu@euca-172-17-118-27:~$ git clone https://github.com/eucalyptus/euca2ools.git

Install euca2ools:

ubuntu@euca-172-17-118-27:~$ cd euca2ools; sudo python setup.py install

Unzip the cloud administrator credentials in /tmp:

ubuntu@euca-172-17-118-27:~$ cd /tmp; unzip admin.zip

Change to the root user, and source the cloud administrator credentials:

ubuntu@euca-172-17-118-27:~$ sudo -s; source /tmp/eucarc

Bundle, upload and register the ramdisk and kernel under /boot:

root@euca-172-17-118-27:~# euca-bundle-image -i /boot/initrd.img-3.8.0-33-generic 
--ramdisk true -r x86_64
root@euca-172-17-118-27:~# euca-upload-bundle -b ubuntu-raring-docker-ramdisk 
-m /var/tmp/bundle-SQrAuT/initrd.img-3.8.0-33-generic.manifest.xml
root@euca-172-17-118-27:~# euca-register -n ubuntu-raring-docker-ramdisk 
ubuntu-raring-docker-ramdisk/initrd.img-3.8.0-33-generic.manifest.xml 
IMAGE eri-6BF033EE
root@euca-172-17-118-27:~# euca-bundle-image -i /boot/vmlinuz-3.8.0-33-generic 
--kernel true -r x86_64
root@euca-172-17-118-27:~# euca-upload-bundle -b ubuntu-raring-docker-kernel 
-m /var/tmp/bundle-31Lnxy/vmlinuz-3.8.0-33-generic.manifest.xml
root@euca-172-17-118-27:~# euca-register -n ubuntu-raring-docker-kernel 
ubuntu-raring-docker-kernel/vmlinuz-3.8.0-33-generic.manifest.xml
IMAGE eki-17093995

Use euca-bundle-vol to bundle the root filesystem. Make sure to exclude /tmp, /mnt/image, and /home/ubuntu. Additionally, make sure and set the size of the image to be 5 GB:

root@euca-172-17-118-27:~# euca-bundle-vol -p ubuntu-raring-docker 
-s 5120 -e /tmp,/root,/mnt/image,/home/ubuntu -d /mnt/image 
--kernel eki-17093995 --ramdisk eri-6BF033EE -r x86_64

Next, upload and register the root filesystem:

root@euca-172-17-118-27:~# euca-upload-bundle -b ubuntu-raring-docker-rootfs 
-m /mnt/image/ubuntu-raring-docker.manifest.xml
root@euca-172-17-118-27:~# euca-register -n ubuntu-raring-docker-rootfs 
ubuntu-raring-docker-rootfs/ubuntu-raring-docker.manifest.xml
IMAGE emi-26403979

We have the new EMI, EKI and ERI for the Docker instance.  Lastly, set the image permissions so that all users on the cloud can use the EMI, EKI and ERI:

root@euca-172-17-118-27:~# euca-modify-image-attribute -l -a all emi-26403979
root@euca-172-17-118-27:~# euca-modify-image-attribute -l -a all eki-17093995
root@euca-172-17-118-27:~# euca-modify-image-attribute -l -a all eri-6BF033EE

Now its time to launch the Docker EMI.

Running the Docker Instance with Cloud-Init

Before launching the EMI, the cloud-init configuration file needs to be created.  This file will be responsible for configuring the instance repositories, downloading and installing Docker.  With your favorite command-line editor, create a file called cloud-init-docker.config, with the following content:

#cloud-config
apt_update: true
apt_upgrade: true
disable_root: true
packages:
 - less
cloud_config_modules:
 - ssh
 - [ apt-update-upgrade, always ]
 - updates-check
 - runcmd
runcmd:
 - [ sh, -xc, "INST_HOSTNAME=`/bin/hostname`; META_IP=`curl -s http://169.254.169.254/latest/meta-data/local-ipv4`; echo ${META_IP} ${INST_HOSTNAME} >> /etc/hosts" ]
 - [ locale-gen, en_US.UTF-8 ]
 - [ sh, -xc, "wget -qO docker-io.gpg https://get.docker.io/gpg" ]
 - [ apt-key, add, docker-io.gpg ]
 - [ sh, -xc, "echo 'deb http://get.docker.io/ubuntu docker main' > /etc/apt/sources.list.d/docker.list" ]
 - [ apt-get, update ]
 - [ apt-get, install, -y, --force-yes, lxc-docker ]
 - [ modprobe, -q, aufs ]

Now, use euca-run-instances to launch the instance:

root@euca-172-17-118-27:~# euca-run-instances -k euca-admin emi-351237D1 
-t m1.medium --user-data-file cloud-init-docker.config

After launching the instance, leave the current instance to get back to end client.

root@euca-172-17-118-27:~# exit
exit
ubuntu@euca-172-17-118-27:~$ exit
logout
Connection to 10.104.7.12 closed.

Once the instance reaches running state, ssh into the instance using the keypair specified (which in this case will be euca-admin.priv), and execute the following Docker command to run an interactive shell session inside a minimal Ubuntu container:

$ euca-describe-instances --region eucalyptus-admin@
RESERVATION r-A1613D7F 961915002812 default
INSTANCE i-AFDB3D4C emi-26403979 euca-10-104-7-13.eucalyptus.euca-hasp.eucalyptus-systems.com 
euca-172-17-118-16.eucalyptus.internal running euca-admin 0 m1.medium 
2013-11-19T01:21:10.880Z LayinDaSmackDown eki-17093995 eri-6BF033EE monitoring-disabled 
10.104.7.13 172.17.118.16 instance-store
# ssh -i euca-admin.priv ubuntu@euca-10-104-7-13.eucalyptus.euca-hasp.eucalyptus-systems.com
Welcome to Ubuntu 13.04 (GNU/Linux 3.8.0-33-generic x86_64)
* Documentation: https://help.ubuntu.com/
System information as of Thu Nov 14 23:18:38 UTC 2013
System load: 0.0 Users logged in: 0
 Usage of /: 21.6% of 4.89GB IP address for eth0: 172.17.184.76
 Memory usage: 4% IP address for lxcbr0: 10.0.3.1
 Swap usage: 0% IP address for docker0: 10.1.42.1
 Processes: 83
Graph this data and manage this system at https://landscape.canonical.com/
Get cloud support with Ubuntu Advantage Cloud Guest:

http://www.ubuntu.com/business/services/cloud

Use Juju to deploy your cloud instances and workloads:

https://juju.ubuntu.com/#cloud-raring

New release '13.10' available.
Run 'do-release-upgrade' to upgrade to it.
Last login: Thu Nov 14 23:08:09 2013 from 10.104.10.6
ubuntu@euca-172-17-184-76:~$ sudo docker run -i -t ubuntu /bin/bash
Unable to find image 'ubuntu' (tag: latest) locally
Pulling repository ubuntu
8dbd9e392a96: Download complete
b750fe79269d: Download complete
27cf78414709: Download complete
root@041d5ddcd6b9:/# (Ctrl-p Ctrl-q to exit out of shell)
ubuntu@euca-172-17-184-76:~$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
041d5ddcd6b9 ubuntu:12.04 /bin/bash 27 seconds ago Up 26 seconds pink_frog

Thats it!  For more information regarding Docker, please refer to the latest Docker documentation.

Enjoy!

IAM Roles and Instance Profiles in Eucalyptus 3.4

IAM Roles in AWS are quite powerful – especially when users need instances to access service APIs to implement complex deployments.  In the past, this could be accomplished by passing access keys and secret keys through the instance user data service, which can be cumbersome and is quite insecure.  With IAM roles, instances can be launched with profiles that allow them to leverage various IAM policies provided by the user to control what service APIs  instances can access in a secure manner.  As part of  constant pursuit for AWS compatibility, one of the new features in Eucalyptus 3.4 is the support of IAM roles and instance profiles (and yes, it works with tools like ec2-api-tools, and libraries like boto, which support accessing IAM roles through the instance meta data service).

This blog entry will demonstrate the following:

  • Set up an Eucalyptus IAM role
  • Create an Eucalyptus instance profile
  • Assign an instance profile when launching an instance
  • Leverage the IAM role from within the instance to access a service API (for this example, it will be the EC2 service API on Eucalyptus)

Prerequisites

To use IAM roles on Eucalyptus, the following is required:

  • A Eucalyptus 3.4 cloud – These packages can be downloaded from the Eucalyptus 3.4 nightly repo.  For additional information regarding downloading nightly builds of Eucalyptus, please refer the Eucalyptus Install Guide (note: anywhere there is a “3.3″ reference, replace with “3.4″)
  • User Credentials – User credentials for an account administrator (admin user), and credentials of a non-admin user of a non-eucalyptus account.
  • Apply an IAM policy for the non-admin user to launch instances, and pass roles to instances launched by that user using euare-useruploadpolicy.  An example policy is below:

    {"Statement": [
     "Effect":"Allow",
     "Action":"iam:PassRole",
     "Resource":"*"
     },
     {
     "Effect":"Allow",
     "Action":"iam:ListInstanceProfiles",
     "Resource":"*"
     },
     {
     "Effect":"Allow",
     "Action":"ec2:*",
     "Resource":"*"
     }]
    }

  • AWS IAM CLI Tools and Euca2ools 3 – The AWS IAM CLI tools are for creating IAM roles and instance profiles; euca2ools for launching instances. There will be one configuration file for the AWS IAM CLI tools that will contain the credentials of the account admin user (for example, account1-admin.config).  Euca2ools will only need the credentials of the non-admin user in the euca2ools.ini file (for example, creating a user section called account1-user01].

Creating  a Eucalyptus IAM Role

Just as in AWS IAM, iam-rolecreate can be used with Eucalyptus IAM to create IAM roles.  To create a IAM role on Eucalyptus, run the following command:

# iam-rolecreate --aws-credential-file account1-admin.config 
--url http://10.104.10.6:8773/services/Euare/ -r ACCT1-EC2-ACTIONS 
-s http://10.104.10.6:8773/services/Eucalyptus
# iam-rolelistbypath --aws-credential-file account1-admin.config
 --url http://10.104.10.6:8773/services/Euare/
arn:aws:iam::735723906303:role/ACCT1-EC2-ACTIONS
IsTruncated: false

This will create a IAM role called ACCT1-EC2-ACTIONS.  Next, we need to add an IAM policy to the role.  As mentioned earlier, the IAM policy will allow the instance to execute an EC2 API call (in this case, ec2-describe-availability-zones).  Use iam-roleuploadpolicy to upload the following IAM policy file:

{
"Statement": [
{
"Sid": "Stmt1381454720306",
"Action": [
"ec2:DescribeAvailabilityZones"
],
"Effect": "Allow",
"Resource": "*"
}
]
}

After the IAM policy file has been created (e.g. ec2-describe-az), upload the policy to the role:

# iam-roleuploadpolicy --aws-credential-file account1-admin.config 
--url http://10.104.10.6:8773/services/Euare/ -p ec2-describe-az 
-f ec2-describe-az -r ACCT1-EC2-ACTIONS
# iam-rolelistpolicies --aws-credential-file account1-admin.config 
--url http://10.104.10.6:8773/services/Euare/ -r ACCT1-EC2-ACTIONS -v
ec2-describe-az
{
 "Statement": [
 {
 "Sid": "Stmt1381454720306",
 "Action": [
 "ec2:DescribeAvailabilityZones"
 ],
 "Effect": "Allow",
 "Resource": "*"
 }
 ]
}
IsTruncated: false

As displayed, the IAM role has been created, and an IAM policy has been added to the role successfully.  Now its time to deal with instance profiles.

Create an Instance Profile and Add a Role to the Profile

Instance profiles are used to pass the IAM role to the instance.  An IAM role can be associated to many instance profiles, but an instance profile can be associated to only one IAM role.  To create an instance profile, use iam-instanceprofilecreate.  Since the IAM role ACCT1-EC2-ACTIONS was previously created, the role can be added as the instance profile is created:

# iam-instanceprofilecreate 
--aws-credential-file account1-admin.config 
--url http://10.104.10.6:8773/services/Euare/ -r ACCT1-EC2-ACTIONS 
-s instance-ec2-actions
# iam-instanceprofilelistbypath --aws-credential-file acct1-user1-aws-iam.config 
--url http://10.104.10.6:8773/services/Euare/
arn:aws:iam::735723906303:instance-profile/instances-ec2-actions
IsTruncated: false

We have successfully created an instance profile and associated an IAM role to it.  All that is left to do is test it out.

Testing out the Instance Profile

Before testing out the instance profile, make sure that the euca2ools.ini file has the correct user and region information for the non-admin user of the account (for this example, the user will be user01).  For information about obtaining the credentials for the user, please refer to the section “Create Credentials” in the Eucalyptus User Guide.

After setting up the euca2ools.ini file, use euca-run-instance to launch an instance with an instance profile.  The image used here is the Ubuntu Raring Cloud Image.  The keypair account1-user01 was created using euca-create-keypair.  To open up SSH access to the instance, use euca-authorize.   Create a cloud-init user data file to enable the multiverse repository.

# cat cloud-init.config
#cloud-config
apt_sources:
 - source: deb $MIRROR $RELEASE multiverse
apt_update: true
apt_upgrade: true
disable_root: true
# euca-run-instances --key account1-user1 emi-C25538DA 
--instance-type m1.large --user-data-file cloud-init.config 
--iam-profile arn:aws:iam::407837561996:instance-profile/instance-ec2-actions 
--region account1-user01@
RESERVATION r-CED1435E 407837561996 default
INSTANCE i-72F244CC emi-C25538DA 0.0.0.0 0.0.0.0 pending account1-user01 0 
m1.large 2013-10-10T22:08:00.589Z Exodus eki-C9083808 eri-39BC3B99 
monitoring-disabled 0.0.0.0 0.0.0.0 instance-store paravirtualized 
arn:aws:iam::407837561996:instance-profile/instance-ec2-actions
....
# euca-describe-instances --region account1-user01@
RESERVATION r-CED1435E 407837561996 default
INSTANCE i-72F244CC emi-C25538DA 10.104.7.22 172.17.190.157 
running account1-user01 0 m1.large 2013-10-10T22:08:00.589Z Exodus eki-C9083808 
eri-39BC3B99 monitoring-disabled 10.104.7.22 172.17.190.157 
instance-store paravirtualized 
arn:aws:iam::407837561996:instance-profile/instance-ec2-actions
TAG instance i-72F244CC euca:node 10.105.10.11

Next, SSH into the instance and confirm the instance profile is accessible by the instance meta-data service.

[root@odc-c-06 ~]# ssh-keygen -R 10.104.7.22
/root/.ssh/known_hosts updated.
Original contents retained as /root/.ssh/known_hosts.old
[root@odc-c-06 ~]# ssh -i euca-admin.priv ubuntu@10.104.7.22
The authenticity of host '10.104.7.22 (10.104.7.22)' can't be established.
RSA key fingerprint is a1:b2:5d:1a:be:e3:cb:0b:58:5f:bd:c1:e2:1f:e3:2d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.104.7.22' (RSA) to the list of known hosts.
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
Welcome to Ubuntu 13.04 (GNU/Linux 3.8.0-31-generic x86_64)
* Documentation: https://help.ubuntu.com/
.....
Get cloud support with Ubuntu Advantage Cloud Guest:

http://www.ubuntu.com/business/services/cloud

Use Juju to deploy your cloud instances and workloads:

https://juju.ubuntu.com/#cloud-raring

0 packages can be updated.
0 updates are security updates.
ubuntu@ip-172-17-190-157:~$ curl http://169.254.169.254/latest/meta-data/
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
hostname
iam/
instance-id
instance-type
kernel-id
local-hostname
local-ipv4
mac
placement/
public-hostname
public-ipv4
public-keys/
ramdisk-id
reservation-id
security-groups
### check for IAM role temporary secuirty credentials ###
ubuntu@ip-172-17-190-157:~$ curl http://169.254.169.254/latest/meta-data/iam/
security-credentials/ACCT1-EC2-ACTIONS
{
 "Code": "Success",
 "LastUpdated": "2013-10-11T18:07:37Z",
 "Type": "AWS-HMAC",
 "AccessKeyId": "AKIYW7FDRV8ZG5HIM91D",
 "SecretAccessKey": "sgVOgLJoc3wXjI5mu7yrYXI3NHtiq18cJuOT7Mwh",
 "Token": "ZXVjYQABQe4E4f2NnIsnvT/5jfpauKh3dClPVwPEoMepqk0lViODSgk4axiQb9rRQyU7Qnhvxb22wO201EoT6Ay/
rg+1i3+2xQLfbkh7kqy4CmqdGM3Q7LNI1dFPSz332E6us5BsSdHpiw3VGLyMLnDAkV8BMi+6lKE5eaJ+hpFI/
KXEVPSNkFMI9R+9bKPIFZvceiBE1w+kAEJC/18uCpZ0kSNy2iFBYcZ+zTwrYTgnsqNYcEIuWzEh4z1WIA==",
 "Expiration": "2013-10-11T19:07:37Z"
}

Install the ec2-api-tools from the Ubuntu Raring multiverse repository.

ubuntu@ip-172-17-190-157:~$ sudo apt-get update
Get:1 http://security.ubuntu.com raring-security Release.gpg [933 B]
Hit http://Exodus.clouds.archive.ubuntu.com raring Release.gpg
......
Ign http://Exodus.clouds.archive.ubuntu.com raring-updates/main Translation-en_US
Ign http://Exodus.clouds.archive.ubuntu.com raring-updates/multiverse Translation-en_US
Ign http://Exodus.clouds.archive.ubuntu.com raring-updates/universe Translation-en_US
Fetched 8,015 kB in 19s (421 kB/s)
Reading package lists... Done
ubuntu@ip-172-17-190-157:~$ sudo apt-get install ec2-api-tools
Reading package lists... Done
The following extra packages will be installed:
 ca-certificates-java default-jre-headless fontconfig-config
 icedtea-7-jre-jamvm java-common libavahi-client3 libavahi-common-data 
libavahi-common3 libcups2 libfontconfig1 libjpeg-turbo8 libjpeg8 liblcms2-2
 libnspr4 libnss3 libnss3-1d openjdk-7-jre-headless openjdk-7-jre-lib 
ttf-dejavu-core tzdata-java
......
Adding debian:TDC_Internet_Root_CA.pem
Adding debian:SecureTrust_CA.pem
done.
Setting up openjdk-7-jre-lib (7u25-2.3.10-1ubuntu0.13.04.2) ...
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
Processing triggers for ca-certificates ...
Updating certificates in /etc/ssl/certs... 0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d....
done.
done.

Finally, run ec2-describe-availability-zones using the –url option to point to the Eucalyptus cloud being used.

ubuntu@ip-172-17-190-157:~$ ec2-describe-availability-zones 
-U http://10.104.10.6:8773/services/Eucalyptus/
AVAILABILITYZONE Legend 10.104.1.185 
arn:euca:eucalyptus:Legend:cluster:IsThisLove/
AVAILABILITYZONE Exodus 10.104.10.22 
arn:euca:eucalyptus:Exodus:cluster:NaturalMystic/

Thats it!  Notice how there wasn’t a need to pass any access key and secret key.  All that information is grabbed from the instance meta-data service.

IAM roles and instance profiles are quite powerful.  Great use cases include enabling CloudWatch metrics, and deploying ELBs on Eucalyptus.

I hope this has been helpful.  As always, any questions/suggestions/ideas/feedback are greatly appreciated.

 

Using Aminator with Eucalyptus

hspencer77:

More Netflix OSS goodness on Eucalyptus…Enjoy!

Originally posted on Testing Clouds at 128bpm:

Introduction to Aminator

The Netflix Cloud Platform has shown how a large scale system can be deployed in a public cloud and maintain an extreme level of performance and reliability. As Adrian Cockcroft has said in the past, Netflix focused on having functional and scalable code rather than worrying immediately about how to make it portable.  At Eucalyptus we have been working over the past year on making sure that as many of the NetflixOSS projects that interact directly with the cloud can be used on Eucalyptus. This level of portability means that anyone with even 1 single linux box can use their system as a test bed for deploying NetflixOSS more broadly on a public cloud. So far at Eucalyptus we have working versions of Asgard, Simian Army, Aminator, and Edda. These tools are cornerstones for app deployment and monitoring with the NetflixOSS stack. In this post I will…

View original 677 more words

Extracting Info From Euca’s Logs

hspencer77:

Great example of how to get performance information from Eucalyptus components.

Originally posted on Testing Clouds at 128bpm:

logstashIntroduction

Throughout my tenure as a Quality Engineer, I have had a love/hate relationship with logs. On one hand, they can be my proof that a problem is occurring and possibly the key to tracking down a fix. On the other hand, they can be an endless stream of seemingly unintelligible information. In debugging a distributed system, such as Eucalyptus, logging can be your only hope in tracing down issues with operations that require the coordination of many components.

Logs are generally presented to users by applications as flat text files that rotate their contents over time in order to bound the amount of space they will take up on the filesystem. Gathering information from these files often involves terminal windows, tail, less, and timestamp correlation. The process of manually aggregating, analyzing and correlating logs can be extremely taxing on the eyes and brain. Having a centralized logging mechanism is…

View original 508 more words

Getting Started with EucaLobo

hspencer77:

Nice tool for AWS/Eucalyptus hybrid utilization.

Originally posted on Testing Clouds at 128bpm:

Initial Setup

In my previous post, I described the story behind EucaLobo, a graphical interface for managing workloads on AWS and Eucalyptus clouds through a <cliche>single pane of glass</cliche>. The tool is built using Javascript and the XUL framework allowing it to be used on Linux, Windows, and Mac for the following APIs:

  • EC2
  • EBS
  • S3
  • IAM
  • CloudWatch
  • AutoScaling
  • Elastic Load Balancing

To get started download the binary for your platform:

Once installation is complete and EucaLobo starts for the first time you will be prompted to enter an endpoint. My esteemed colleague Tony Beckham has created a great intro video showing how to create and edit credentials and endpoints. The default values have been set to the Eucalyptus Community Cloud, a free and easy way to get started using Eucalyptus and clouds in general. This is a great resource for users who want to get a…

View original 653 more words

The Journey to EucaLobo

hspencer77:

EucaLobo released to the wild with Eucalyptus 3.3 GA..

Originally posted on Testing Clouds at 128bpm:

The 3.3.0 feature barrage

As a quality engineer it is always useful to have an at-a-glance view of the state of your system under test. Unfotunately, having reliable graphical tools is not always possible during testing phases as the UI is often trailing the development of core features. During the 3.3.0 release, the Eucalyptus development team added an incredible amount of API calls to its already large catalog of AWS compatible operations:

  • Elastic Load Balancing
  • Autoscaling
  • CloudWatch
  • Resource Tagging
  • Filtering
  • Maintenance Mode
  • Block Device Mappings

As a result of this onslaught of new service functionality from developers the UI and QA teams had their work cut out for them. The UI team had decided early on that they needed to make some architectural changes to the UI code, such as leveraging Backbone.js and Rivets. This meant they would only be able to cover the newly added resource tagging and filtering services…

View original 632 more words

Eucalyptus 3.3.0 in a nutshell

hspencer77:

The new hotness thats arriving with Eucalyptus 3.3..

Originally posted on shaon's Blog:

Eucalyptus 3.3.0, the most exciting Eucalyptus release so far is knocking on the door or perhaps it has been already released when you are reading this post.

Eucalyptus 3.3.0 has couple of most desired Amazon Web Services (AWS) features by the cloud users:

1. Elastic Load Balancing (ELB)

Needless to say, this is an AWS ELB compatible feature which is being introduced in Eucalyptus 3.3.0.

Creating a basic loadbalancer:

1

Register instances with Eucalyptus Elastic Load Balancer,

1

Few other ELB operations,

1

2. CloudWatch

CloudWatch is another AWS-compatible feature which is…

View original 990 more words

Managing a DNS Domain from One Place

hspencer77:

Some DNS slickness..

Originally posted on /dev/zero:

Taking a DNS name and resolving it to the address of a machine is easy to understand and easy to implement if you’re an administrator. Doing a reverse lookup from an address back to a name, however, is more difficult due to the way addresses are divided up. I won’t attempt to describe the details here (I recommend Liu and Albitz’s DNS and BIND for the gory details), but in short, the way this works is by breaking an IP address into its four octets and handling them from there like regular hierarchical names in the special in-addr.arpa zone:

1

This is problematic for two main reasons:

  • You have to change two zones every time you change a DNS name.
  • If you have fewer than 256 addresses, your ISP can’t delegate the appropriate subset of the in-addr.arpa zone to you so you can maintain it yourself. This…

View original 219 more words