AWS EBS-backed AMI to Eucalyptus Walrus-backed EMI

Preface

A few weeks back, I was doing some testing with the guys from AppScale to get an Eucalyptus Machine Image (EMI) to run on Eucalyptus.  The image that was provided to me was an EBS-backed Amazon Machine Image (AMI), using a published EC2 Lucid Ubuntu Cloud image.  This blog entry describes the procedure to convert an EBS-backed AMI to an Walrus-backed EMI.  The goal here is to demonstrate how easy it is to use Ubuntu Cloud images to set up AppScale on both AWS and Eucalyptus as a hybrid cloud use case.  There are many other hybrid cloud use cases that can be done with this setup, but this blog entry will focus on the migration of AMI images to EMI images.

*NOTE* This entry assumes that a user is experienced with both Amazon Web Services and Eucalyptus.  For additional information, please refer to the following resources:

Prerequisites

Before getting started, the following is needed:

*NOTE* Make sure there is an understanding of the IAM policies on AWS and Eucalyptus.  These are key in making sure that the user on both AWS and Eucalyptus can perform all the steps covered in this topic.

Work in AWS…

After setting up the command-line tools for AWS EC2,  and adding in the necessary EC2 and S3 IAM policies, everything is in place to get started with working with the AWS instances and images. *NOTE* To get help with setting up the IAM policies, check out the AWS Policy Generator.    To make sure things look good, I tested out my EC2 access by running ec2-describe-availability-zones:

$ ec2-describe-availability-zones 
AVAILABILITYZONE us-east-1a available us-east-1 
AVAILABILITYZONE us-east-1b available us-east-1 
AVAILABILITYZONE us-east-1c available us-east-1 
AVAILABILITYZONE us-east-1d available us-east-1

After that, I set up a keypair and SSH access for any instance that is launched within the default security group:

$ ec2-create-keypair hspencer-appscale –region ap-northeast-1 > hspencer-appscale.pem

$ ec2-authorize -P tcp -p 22 -s 0.0.0.0/0 default –region ap-northeast-1

With everything looking good, I went ahead and checked out the AMI that I was asked to test.  Below is the AMI that was given to me:

$ ec2-describe-images ami-2e4bf22f --region ap-northeast-1
IMAGE ami-2e4bf22f 839953741869/appscale-lite-1.6.3-testing 839953741869 available public x86_64 machine aki-d409a2d5 ebs paravirtual xen
BLOCKDEVICEMAPPING EBS /dev/sda1 snap-7953a059 8 true standard
BLOCKDEVICEMAPPING EPHEMERAL /dev/sdb ephemeral0

As you can see, the AMI given to me is an EBS-backed image, and it is in a different region (ap-northeast-1).  I could have done all my work in the ap-northeast-1 region, but I wanted to test out region-to-region migration of images on AWS S3 using ec2-migrate-manifest.  In order to access the EBS-backed instance that is launched, I set up a keypair and SSH access for any instance that is launched within the default security group:

$ ec2-create-keypair hspencer-appscale --region ap-northeast-1 > hspencer-appscale.pem
$ ec2-authorize -P tcp -p 22 -s 0.0.0.0/0 default --region ap-northeast-1

Now that I have my image, keypair and security group access,  I am ready to launch an instance, so I can use the ec2-bundle-vol command to create an image of the instance.  To launch the instance, I ran the following:

$ ec2-run-instances -k hspencer-appscale ami-2e4bf22f –region ap-northeast-1

After the instance is up and running, I scp’d my EC2_PRIVATE_KEY and EC2_CERT to the instance using the keypair created (hspencer-appscale.pem).  The instance already had the latest  version of ec2-api-tools and ec2-ami-tools as part of the installation of AppScale.  Similar to the instructions provided by AWS for creating an instance-store backed AMI from an existing AMI, I  used ec2-bundle-vol to bundle a new image and used /mnt/ (which is ephemeral storage) to store the manifest information.

root@ip-10-156-123-126:~# ec2-bundle-vol -u 9xxxxxxx3 -k pk-XXXXXXXXXXXXXXXX.pem -c cert-XXXXXXXXXXXXXXXXX.pem -d /mnt/ -e /mnt/
Please specify a value for arch [x86_64]: x86_64
Copying / into the image file /mnt/image...
Excluding: 
/dev
/sys
/sys/kernel/security
/sys/kernel/debug
/proc
/dev/pts
/dev
/media
/mnt
/proc
/sys
/mnt/
/mnt/image
/mnt/img-mnt
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.00990555 s, 106 MB/s
mke2fs 1.41.11 (14-Mar-2010)
Bundling image file...
Splitting /mnt/image.tar.gz.enc...
Created image.part.000
……………..

Next, I need to inform the manifest to use us-west-1 as the region to store the image, and not ap-northeast-1.  To do this, I used ec2-migrate-manifest.  *NOTE* This tool can only be used in the following regions: EU,US,us-gov-west-1,us-west-1,us-west-2,ap-southeast-1,ap-southeast-2,ap-northeast-1,sa-east-1.

root@ip-10-156-123-126:~# ec2-migrate-manifest -m /mnt/image.manifest.xml -c cert-XXXXXXXXX.pem -k pk-XXXXXXXXXXXX.pem -a XXXXXXXXXX -s XXXXXXXXX --region us-west-1
Backing up manifest...
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
warning: peer certificate won't be verified in this SSL session
Successfully migrated /mnt/image.manifest.xml
It is now suitable for use in us-west-1.

Time to upload the bundle to S3 using ec2-upload-bundle:

root@ip-10-156-123-126:~# ec2-upload-bundle -b appscale-lite-1.6.3-testing -m /mnt/image.manifest.xml -a XXXXXXXXXX -s XXXXXXXXXX --location us-west-1
You are bundling in one region, but uploading to another. If the kernel
or ramdisk associated with this AMI are not in the target region, AMI
registration will fail.
You can use the ec2-migrate-manifest tool to update your manifest file
with a kernel and ramdisk that exist in the target region.
Are you sure you want to continue? [y/N]y
Creating bucket...
Uploading bundled image parts to the S3 bucket appscale-lite-1.6.3-testing ...
Uploaded image.part.000
Uploaded image.part.001
Uploaded image.part.002
Uploaded image.part.003
………….

After the image has been uploaded successfully, all that is left to do is register the image.

root@ip-10-156-123-126:~# export JAVA_HOME=/usr
root@ip-10-156-123-126:~# ec2-register -K pk-XXXXXXXXXXXX.pem -C cert-XXXXXXXXXX.pem --region us-west-1 appscale-lite-1.6.3-testing/image.manifest.xml --name appscale1.6.3-testing
IMAGE ami-705d7c35
$ ec2-describe-images ami-705d7c35 --region us-west-1
IMAGE ami-705d7c35 986451091583/appscale1.6.3-testing 986451091583 available private x86_64 machine aki-9ba0f1de instance-store paravirtual xen
BLOCKDEVICEMAPPING EPHEMERAL /dev/sdb ephemeral0

Work in Eucalyptus…

Now that we have the image registered, we can use ec2-download-bundle and ec2-unbundle to get the machine image to an instance running on Eucalyptus, so that we can bundle, upload and register the image to Eucalyptus.

To start off, I followed the instructions for setting up my command-line environment, and Eucalyptus IAM policies on Eucalyptus – similar to what was done for AWS.

Next, I downloaded the lucid-server-cloudimg-amd64.tar.gz file from the Ubuntu Cloud Images (Lucid) site.  After that, I bundled, uploaded and registered the following images:

  • lucid-server-cloudimg-amd64-loader (ramdisk)
  • lucid-server-cloudimg-amd64-vmlinuz-virtual (kernel)
  • lucid-server-cloudimg-amd64.img (root image)

After bundling, uploading and registering those images, I created a keypair, and SSH access for the instance that is launched within the default security group:

euca-add-keypair hspencer-euca > hspencer-euca.pem
euca-authorize -P tcp -p 22 -s 0.0.0.0/0 default

Now, I run the EMI for the Lucid image that was registered:

euca-run-instance -k hspencer-euca --user-data-file cloud-init.config -t m1.large emi-29433329

I used vm.type m1.large so that I can use the space on ephemeral to store the image that I will pull from AWS.

Once the instance is running, I scp’d my EC2_PRIVATE_KEY and EC2_CERT to the instance using the keypair created (hspencer-euca.pem).  After installing the ec2-ami-tools on the instance, I used ec2-download-bundle to download the bundle to /media/ephemeral0, and ec2-unbundle the image:

# ec2-download-bundle -b appscale-lite-1.6.3-testing -d /media/ephemeral0/ -a XXXXXXXXXXX -s XXXXXXXXXXXX -k pk-XXXXXXXXX.pem --url http://s3-us-west-1.amazonaws.com
# ec2-unbundle -m /media/ephemeral0/image.manifest.xml -s /media/ephemeral0/ -d /media/ephemeral0/ -k pk-XXXXXXXXXX.pem

Now that I have the root image from AWS, I just need to bundle, upload and register the root image to Eucalyptus.  To do so, I scp’d my Eucalyptus user credentials to the instance.  After copying the Eucalyptus credentials to the instance, I ssh’ed into the instance and source the Eucalyptus credentials.

Since I have already bundled the kernel and ramdisk for the Ubuntu Cloud Lucid image before, I just need to upload, bundle and register the image I unbundled from AWS.  To do so, I did the following:

euca-bundle-image -i image  
euca-upload-bundle -b appscale-1.6.3-x86_64 -m /tmp/image.manifest.xml
euca-register -a x86_64 appscale-1.6.3-x86_64/image.manifest.xml

Now the image is ready to be launched on Eucalyptus.

Conclusion

As demonstrated above,  because of the AWS fidelity that Eucalyptus provides, it enables setting up hybrid cloud environments with Eucalyptus and AWS that can be leveraged by applications, like AppScale.

Other examples of AMI to EMI conversions can be found here:

https://github.com/eucalyptus/ami2emi

Enjoy!

 

 

AWS EBS-backed AMI to Eucalyptus Walrus-backed EMI

Another Great Example of AWS Fidelity – Neo4j, Cloud-Init and Eucalyptus

I recently ran across a blog entry entitled Neo4j 1.9.M01 – Self-managed HA.  I found the concept of graph databases storing data really interesting and reached out to the guys at Neo4j to get some insight on how to deploy their HA solution on Eucalyptus.   Amongst the resources that they provided,  they shared this little gem – how to deploy Neo4j on EC2.  In order to run first, you need to know how to walk – so before going down the path of standing up HA Neo4j, I decided to be influenced by the DIY on EC2 article provided by Neo4j and deploy Neo4j on Eucalyptus  – with a little help from Cloud-Init.  The follow-up blog will show how to use the same setup, and deploy an HA Neo4j environment.

The Setup

Eucalyptus

The Eucalyptus cloud I used is configured using Eucalyptus High-Availability.  Its running on CentOS 6.3, running KVM.  Its also running in Managed networking mode, so that we can take advantage of network isolation of the VMs, and the use of security groups  – interacting very much in the same way as its done in the security groups provided in AWS EC2.

Ubuntu Cloud Image – 12.04 LTS Precise Pangolin

The image that we will use is the Ubuntu 12.04 LTS Cloud image.  The reasons for using this image is as follows:

  • Ubuntu cloud images come pre-packaged with cloud-init, which helps with bootstrapping the instance.
  • I wanted to have the solution work on AWS EC2 and Eucalyptus; since Ubuntu cloud images work on both, its a great choice.

Registering the Ubuntu Cloud Image with Eucalyptus

In order for us to get started, we need to get the Ubuntu Cloud image into Eucalyptus so that we can use it for our instance.  To upload, bundle and register the Ubuntu Cloud image, ramdisk and kernel, do the following:

  1. Download current version of  Ubuntu Precise Server AMD64 from the Ubuntu Cloud Image – Precise page, then unpack (ungzip, unarchive) the tar-gzipped file.  
    $ tar -zxvf precise-server-cloudimg-amd64.tar.gz
    x precise-server-cloudimg-amd64.img
    x precise-server-cloudimg-amd64-vmlinuz-virtual
    x precise-server-cloudimg-amd64-loader
    x precise-server-cloudimg-amd64-floppy
    x README.files

  2. Make sure to download and source your Eucalyptus credentials.
  3. We need to bundle, upload, and register precise-server-cloudimg-amd64-loader (ERI), precise-server-cloudimg-amd64-vmlinuz-virtual (EKI), and precise-server-cloudimg-amd64.img (EMI).  For more information regarding this, please refer to the “Image Overview” section of the Eucalyptus 3.1 User Guide.  
    $ euca-bundle-image -i precise-server-cloudimg-amd64-loader --ramdisk true
    $ euca-upload-bundle -b latest-ubuntu-precise -m /tmp/precise-server-cloudimg-amd64-loader.manifest.xml
    $ euca-register -a x86_64 latest-ubuntu-precise/precise-server-cloudimg-amd64-loader.manifest.xml
    $ euca-bundle-image -i precise-server-cloudimg-amd64-vmlinuz-virtual --kernel true
    $ euca-upload-bundle -b latest-ubuntu-precise -m /tmp/precise-server-cloudimg-amd64-vmlinuz-virtual.manifest.xml
    $ euca-register -a x86_64 latest-ubuntu-precise/precise-server-cloudimg-amd64-vmlinuz-virtual.manifest.xml
    $ euca-bundle-image -i precise-server-cloudimg-amd64.img
    $ euca-upload-bundle -b latest-ubuntu-precise -m /tmp/precise-server-cloudimg-amd64.img.manifest.xml
    $ euca-register -a x86_64 latest-ubuntu-precise/precise-server-cloudimg-amd64.img.manifest.xml

After bundling, uploading and registering the ramdisk, kernel and image, the latest-ubuntu-precise bucket in Walrus should have the following images:

$ euca-describe-images | grep latest-ubuntu-precise
IMAGE eki-0F3937E9 latest-ubuntu-precise/precise-server-cloudimg-amd64-vmlinuz-virtual.manifest.xml 345590850920 available public x86_64 kernel instance-store

IMAGE emi-C1613E67 latest-ubuntu-precise/precise-server-cloudimg-amd64.img.manifest.xml 345590850920 available public x86_64 machine instance-store

IMAGE eri-0BE53BFD latest-ubuntu-precise/precise-server-cloudimg-amd64-loader.manifest.xml 345590850920 available public x86_64 ramdisk instance-store

Cloud-init Config File

Now that we have the image ready to go, we need to create a cloud-init config file to pass in using the –user-data-file option that is part of euca-run-instances.  For more examples of different cloud-init files, please refer to the cloud-init-dev/cloud-init repository on bazaar.launchpad.net.  Below is the cloud-init.config file I created for bootstrapping the instance with an install of Neo4j, using ephemeral disk for the application storage, and installing some other packages (i.e. latest euca2ools, mlocate, less, etc.). The script can be also accessed from github as well – under the eucalptus/recipes repo.

#cloud-config
apt_update: true
apt_upgrade: true
disable_root: true
package_reboot_if_required: true
packages:
 - less
 - bind9utils
 - dnsutils
 - mlocate
cloud_config_modules:
 - ssh
 - [ apt-update-upgrade, always ]
 - updates-check
 - runcmd
runcmd:
 - [ sh, -xc, "if [ -b /dev/sda2 ]; then tune2fs -L ephemeral0 /dev/sda2;elif [ -b /dev/vda2 ]; then tune2fs -L ephemeral0 /dev/vda2;elif [ -b /dev/xvda2 ]; then tune2fs -L ephemeral0 /dev/xvda2;fi" ]
 - [ sh, -xc, "mkdir -p /var/lib/neo4j" ]
 - [ sh, -xc, "mount LABEL=ephemeral0 /var/lib/neo4j" ]
 - [ sh, -xc, "if [ -z `ls /var/lib/neo4j/*` ]; then sed --in-place '$ iMETA_HOSTNAME=`curl -s http://169.254.169.254/latest/meta-data/local-hostname`\\nMETA_IP=`curl -s http://169.254.169.254/latest/meta-data/local-ipv4`\\necho ${META_IP}   ${META_HOSTNAME} >> /etc/hosts; hostname ${META_HOSTNAME}; sysctl -w kernel.hostname=${META_HOSTNAME}\\nif [ -d /var/lib/neo4j/ ]; then mount LABEL=ephemeral0 /var/lib/neo4j; service neo4j-service restart; fi' /etc/rc.local; fi" ] 
 - [ sh, -xc, "META_HOSTNAME=`curl -s http://169.254.169.254/latest/meta-data/local-hostname`; META_IP=`curl -s http://169.254.169.254/latest/meta-data/local-ipv4`; echo ${META_IP}   ${META_HOSTNAME} >> /etc/hosts" ]
 - [ sh, -xc, "META_HOSTNAME=`curl -s http://169.254.169.254/latest/meta-data/local-hostname`; hostname ${META_HOSTNAME}; sysctl -w kernel.hostname=${META_HOSTNAME}" ]
 - [ sh, -xc, "wget -O c1240596-eucalyptus-release-key.pub http://www.eucalyptus.com/sites/all/files/c1240596-eucalyptus-release-key.pub" ]
 - [ apt-key, add, c1240596-eucalyptus-release-key.pub ]
 - [ sh, -xc, "echo 'deb http://downloads.eucalyptus.com/software/euca2ools/2.1/ubuntu precise main' > /etc/apt/sources.list.d/euca2ools.list" ]
 - [ sh, -xc, "echo 'deb http://debian.neo4j.org/repo stable/' > /etc/apt/sources.list.d/neo4j.list" ]
 - [ apt-get, update ]
 - [ apt-get, install, -y, --force-yes, euca2ools ]
 - [ apt-get, install, -y, --force-yes, neo4j ]
 - [ sh, -xc, "sed --in-place 's/#org.neo4j.server.webserver.address=0.0.0.0/org.neo4j.server.webserver.address=0.0.0.0/' /etc/neo4j/neo4j-server.properties" ]
 - [ sh, -xc, "service neo4j-service restart" ]
 - [ sh, -xc, "export LANGUAGE=en_US.UTF-8" ]
 - [ sh, -xc, "export LANG=en_US.UTF-8" ]
 - [ sh, -xc, "export LC_ALL=en_US.UTF-8" ]
 - [ locale-gen, en_US.UTF-8 ]
 - [ dpkg-reconfigure, locales ]
 - [ updatedb ]
mounts:
 - [ ephemeral0, /var/lib/neo4j, auto, "defaults,noexec" ]

Now, we are ready to launch the instance.

Putting It All Together

Before launching the instance, we need to set up our keypair and security group that we will use with the instance.

  1. To create a keypair, run euca-create-keypair.  *NOTE* Make sure you change the permissions of the keypair to 0600 after its been created.
    euca-create-keypair  neo4j-user > neo4j-user.priv; chmod 0600 neo4j-user.priv

  2. Next, we need to create a security group for our instance.  To create a security group, use euca-create-group.  To open any ports you  need for the application, use euca-authorize.  The ports we will open up for the Neo4j application are SSH (22), ICMP, HTTP( 7474), and HTTPS (7473).
    • Create security group:
      # euca-create-group neo4j-test -d "Security for Neo4j Instances"

    • Authorize SSH:
      # euca-authorize -P tcp -p 22 -s 0.0.0.0/0 neo4j-test

    • Authorize HTTP:
      # euca-authorize -P tcp -p 7474 -s 0.0.0.0/0 neo4j-test

    • Authorize HTTPS:
      # euca-authorize -P tcp -p 7473 -s 0.0.0.0/0 neo4j-test

    • Authorize ICMP:
      # euca-authorize -P icmp -t -1:-1 -s 0.0.0.0/0 neo4j-test

  3. Finally, we use euca-run-instances to launch the Ubuntu Precise image, and use cloud-init to install Neo4j:
    # euca-run-instances -k neo4j-user --user-data-file cloud-init-neo4j.config emi-C1613E67 --kernel eki-0F3937E9 --ramdisk eri-0BE53BFD --group neo4j-test

To check the status of the instance, use euca-describe-instances.

# euca-describe-instances i-A9EF448C
RESERVATION r-ED8E4699 345590850920 neo4j-test
INSTANCE i-A9EF448C emi-C1613E67 euca-192-168-55-104.wu-tang.euca-hasp.eucalyptus-systems.com 
euca-10-106-69-154.wu-tang.internal running admin 0 m1.small 2012-12-04T03:13:13.869Z 
enter-the-wu eki-0F3937E9 eri-0BE53BFD monitoring-disable 
euca-192-168-55-104.wu-tang.euca-hasp.eucalyptus-systems.com euca-10-106-69-154.wu-tang.internal instance-store

Because I added in the cloud-init config file to do an “apt-get upgrade”, it takes about 5 to 7 minutes until the instance is fully configured and Neo4j is running.  Once you have it running, go to https://<ip-address of instance>:7473.  It will direct you to the web administration page for monitoring and management of the Neo4j instance.  In this example, the URL will be https://euca-192-168-55-104.wu-tang.euca-hasp.eucalyptus-systems.com:7473

Neo4j Monitoring and Management Tool
Neo4j Monitoring and Management Tool

Thats it!  The cool thing about this too, is that you can find an Ubuntu Precise AMI on AWS EC2, use the same cloud-init script, use euca2ools, and follow these instructions to get the same deployment on AWS EC2.

As mentioned before, the follow-up blog will be how to deploy the HA solution of Neo4j on Eucalyptus. Enjoy!

Another Great Example of AWS Fidelity – Neo4j, Cloud-Init and Eucalyptus