In my last blog, I covered three ways Eucalyptus Systems uses the Varnish-Walrus architecture. This blog will cover how eustore takes advantage of this architecture.

Overview

Eustore is an image management tool developed by David Kavanagh. Its primary goal is to automate image bundling, uploading and registration.  The two commands provided by eustore is eustore-describe-images and eustore-install-image.


# eustore-describe-images --help
Usage: eustore-describe-images [options]

lists images from Eucalyptus.com

Options:
-h, --help show this help message and exit
-v, --verbose display more information about images

Standard Options:
-D, --debug Turn on all debugging output
--debugger Enable interactive debugger on error
-U URL, --url=URL Override service URL with value provided
--region=REGION Name of the region to connect to
-I ACCESS_KEY_ID, --access-key-id=ACCESS_KEY_ID
Override access key value
-S SECRET_KEY, --secret-key=SECRET_KEY
Override secret key value
--version Display version string

# eustore-install-image --help
Usage: eustore-install-image [options]

downloads and installs images from Eucalyptus.com

Options:
-h, --help show this help message and exit
-i IMAGE_NAME, --image_name=IMAGE_NAME
name of image to install
-b BUCKET, --bucket=BUCKET
specify the bucket to store the images in
-k KERNEL_TYPE, --kernel_type=KERNEL_TYPE
specify the type you're using [xen|kvm]
-d DIR, --dir=DIR specify a temporary directory for large files
--kernel=KERNEL Override bundled kernel with one already installed
--ramdisk=RAMDISK Override bundled ramdisk with one already installed

Standard Options:
-D, --debug Turn on all debugging output
--debugger Enable interactive debugger on error
-U URL, --url=URL Override service URL with value provided
--region=REGION Name of the region to connect to
-I ACCESS_KEY_ID, --access-key-id=ACCESS_KEY_ID
Override access key value
-S SECRET_KEY, --secret-key=SECRET_KEY
Override secret key value
--version Display version string


Eustore, by default, uses the images located on emis.eucalyptus.com.  Eustore can be configured to use other locations for images.  Eustore utilizes two components for image management:

• JSON configuration file – catalog.json
• Location for tar-gzipped EMIs

The images found on emis.eucalyptus.com and the JSON configuration file associated with those images are all located in unique Walrus buckets. The images shown in this blog are in the starter-emis bucket. The ACLs for these buckets allow for the objects to be publicly accessible. For more information on Walrus ACLs, please reference the section “Access Control List (ACL) Overview” in the AWS S3 Developer’s Guide.

The two commands mentioned above that eustore provides – eustore-describe-images and eustore-install-images – significantly cuts down the number of commands needed to be input by the user. Without using eustore, a user would need to run 3 commands (euca-bundle-image, euca-upload-image, and euca-register) for the kernel, ramdisk, and raw disk image for an EMI (this translates to a total of 9 commands).

The Collaboration

eustore-describe-images

When eustore-describe-images is ran, the following occurs:

1. eustore-describe-image requests information from JSON file (stored in Walrus bucket) from emis.eucalyptus.com (varnishd instance)
2. ^** If JSON file – catalog.json – is not present on emis.eucalyptus.com‘s cache, then JSON file is pulled from Walrus bucket
3. Data from JSON file is returned to eustore-describe-images.

   
# eustore-describe-images
....
centos-x86_64-20120114 centos x86_64 2012.1.14 CentOS 5 1.3GB root, Single Kernel
centos-lg-i386-20110702 centos i386 2011.07.02 CentOS 5 4.5GB root, Hypervisor-Specific Kernels
centos-lg-x86_64-20110702centos x86_64 2011.07.02 CentOS 5 4.5GB root, Hypervisor-Specific Kernels
centos-lg-x86_64-20111228centos x86_64 2011.12.28 CentOS 5 4.5GB root, Single Kernel
centos-lg-x86_64-20120114centos x86_64 2012.1.14 CentOS 5 4.5GB root, Single Kernel
debian-i386-20110702 debian i386 2011.07.02 Debian 6 1.3GB root, Hypervisor-Specific Kernels
debian-x86_64-20110702 debian x86_64 2011.07.02 Debian 6 1.3GB root, Hypervisor-Specific Kernels
debian-x86_64-20120114 debian x86_64 2012.1.14 Debian 6 1.3GB root, Single Kernel
.....


eustore-install-image

eustore-install-image follows the same steps as eustore-describe-images, except it uses the information stored in the JSON file for each EMI.  The following information is present for each EMI:


{
"images": [
{
"name": "centos-x86_64-20120114",
"description": "CentOS 5 1.3GB root, Single Kernel",
"version": "2012.1.14",
"architecture": "x86_64",
"os": "centos",
"url": "starter-emis/euca-centos-2012.1.14-x86_64.tgz",
"date": "20120114150503",
"recipe": "centos-based",
"stamp": "28fc-4826",
"contact": "images@lists.eucalyptus.com"
},
.....
{
"name": "debian-x86_64-20120114",
"description": "Debian 6 1.3GB root, Single Kernel",
"version": "2012.1.14",
"architecture": "x86_64",
"os": "debian",
"url": "starter-emis/euca-debian-2012.1.14-x86_64.tgz",
"date": "20120114152138",
"recipe": "debian-based",
"stamp": "3752-f34a",
"contact": "images@lists.eucalyptus.com"
},
....
]
}

When eustore-install-image -i centos-x86_64-20120114 -b centos_x86-64 is executed, the following occurs:

1. eustore-install-image requests image (which is in a tar-gzipped form) to be downloaded from emis.eucalyptus.com
2. ^** If image is not available (euca-centos-2012.1.14-x86_64.tgz) in varnish cache, then varnishd (emis.eucalyptus.com) will pull image from starter-emis bucket and store it in ephemeral space for caching to handle future requests.
3. Once the tar-gzipped file is downloaded, eustore-install-image will bundle, upload, and register the kernel (EKI), ramdisk (ERI) and image (EMI).

As demonstrated above, eustore definitely makes image management efficient and user-friendly. Stay tuned for upcoming blogs discussing more on how the Varnish-Walrus architecture is utilized.

For any questions, concerns, and/or suggestions, please email images@lists.eucalyptus.com or community@lists.eucalyptus.com. And as always, you can respond with comments here as well.:-)

Enjoy!

^**This step won’t happen if the contents are cached on emis.eucalyptus.com