Deploying CentOS 6.5 Image in Docker on Eucalyptus 3.4

This blog entry is a follow-up of my blog entry entitled “Step-by-Step Deployment of Docker on Eucalyptus 3.4 for the Cloud Administrator“.  In that blog entry, I covered how to deploy an Ubuntu Raring Cloud Image on Eucalyptus, and use that image to deploy Docker.   The really cool thing about Docker is that it provides the ability to deploy different operating systems within one machine – in this case, an instance.   The focus of this entry is to show how to deploy a CentOS 6.5 image in a instance running Docker on Eucalyptus 3.4.

Prerequisites

The prerequisites for this blog is to complete the steps outlined in my previous blog about Docker on Eucalyptus 3.4.  Once those steps are completed by the cloud administrator (i.e. a user associated with the ‘eucalyptus’ account), you can get started  on the steps below.  One thing to note here, to follow these steps, there is no need to be the cloud administrator.  This entry is entirely directed to cloud users.  Since Eucalyptus IAM is similar to AWS IAM, the following EC2 Actions need to be allowed for the cloud user:

Instance Deployment

To get started, we need to launch the Ubuntu Raring EMI that has been provided by the cloud administrator.  In this example, the EMI will be emi-26403979:

$ euca-describe-images emi-26403979 --region account1-user01@
IMAGE emi-26403979 ubuntu-raring-docker-rootfs-v3/ubuntu-raring-docker-v3.manifest.xml
 441445882805 available private x86_64 machine eki-17093995 
eri-6BF033EE instance-store paravirtualized

If the –region option seems confusing, this is due to the fact that I am using the nice configuration file feature in Euca2ools.  Its really helpful when you are using different credentials for different users.

Now that we know the EMI we can use, lets launch the instance.  We will be using the cloud-init config file from my previous Docker blog to configure the instance.  The VM type used here is c1.xlarge.  This is because I wanted to make sure I had 2 CPU, and 2 Gigs of RAM for my instance:

$ euca-run-instances -k account1-user01 -t c1.xlarge 
--user-data-file cloud-init-docker.config emi-26403979 
--region account1-user01@
RESERVATION r-2EE941D4 961915002812 default
INSTANCE i-503642D4 emi-26403979 euca-0-0-0-0.eucalyptus.euca-hasp.eucalyptus-systems.com
 euca-0-0-0-0.eucalyptus.internal pending account1-user01 0 
c1.xlarge 2014-02-14T00:47:15.632Z LayinDaSmackDown eki-17093995 
eri-6BF033EE monitoring-disabled 0.0.0.0 0.0.0.0 instance-store paravirtualized

Make sure the instance gets into the running state:

$ euca-describe-instances --region account1-user01@
RESERVATION r-2EE941D4 961915002812 default
INSTANCE i-503642D4 emi-26403979 euca-10-104-7-10.eucalyptus.euca-hasp.eucalyptus-systems.com
 euca-172-17-112-207.eucalyptus.internal running 
account1-user01 0 c1.xlarge 2014-02-14T00:47:15.632Z 
LayinDaSmackDown eki-17093995 eri-6BF033EE monitoring-disabled 
10.104.7.10 172.17.112.207 instance-store paravirtualized

Now thats in the running state, lets SSH into the instance to make sure its up and running:

$ ssh -i account1-user01.priv ubuntu@euca-10-104-7-10.eucalyptus.euca-hasp.eucalyptus-systems.com
Welcome to Ubuntu 13.04 (GNU/Linux 3.8.0-33-generic x86_64)
* Documentation: https://help.ubuntu.com/
System information as of Fri Feb 14 00:49:52 UTC 2014
System load: 0.21 Users logged in: 0
 Usage of /: 21.8% of 4.80GB IP address for eth0: 172.17.112.207
 Memory usage: 5% IP address for lxcbr0: 10.0.3.1
 Swap usage: 0% IP address for docker0: 10.42.42.1
 Processes: 85
Graph this data and manage this system at https://landscape.canonical.com/
Get cloud support with Ubuntu Advantage Cloud Guest:

http://www.ubuntu.com/business/services/cloud

Use Juju to deploy your cloud instances and workloads:

https://juju.ubuntu.com/#cloud-raring

Your Ubuntu release is not supported anymore.
For upgrade information, please visit:

http://www.ubuntu.com/releaseendoflife

New release '13.10' available.
Run 'do-release-upgrade' to upgrade to it.
Last login: Tue Nov 19 00:48:23 2013 from odc-d-06-07.prc.eucalyptus-systems.com
ubuntu@euca-172-17-112-207:~$

Now its time to prepare the instance to create your own base Docker CentOS 6.5 image.

Preparing the Instance

To prepare the instance to create the base image, install the following packages using apt-get:

ubuntu@euca-172-17-112-207:~$ sudo -s
root@euca-172-17-112-207:~# apt-get install rinse perl rpm \
rpm2cpio libwww-perl liblwp-protocol-https-perl

After the packages finish installing, lets go ahead and mount the ephemeral storage on the instance as /tmp to give Docker extra space for building the base image:

root@euca-172-17-112-207:~# curl http://169.254.169.254/latest/meta-data/block-device-mapping/ephemeral0
sda2
root@euca-172-17-112-207:~# mount /dev/vda2 /tmp
root@euca-172-17-112-207:~# df -ah
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 4.8G 1.1G 3.5G 24% /
proc 0 0 0 - /proc
sysfs 0 0 0 - /sys
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 0 0 0 - /sys/fs/fuse/connections
none 0 0 0 - /sys/kernel/debug
none 0 0 0 - /sys/kernel/security
udev 993M 8.0K 993M 1% /dev
devpts 0 0 0 - /dev/pts
tmpfs 201M 240K 201M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 1002M 0 1002M 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/vda2 9.4G 150M 8.8G 2% /tmp
root@euca-172-17-112-207:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 15G 0 disk
├─vda1 253:1 0 5G 0 part /
├─vda2 253:2 0 9.5G 0 part /tmp
└─vda3 253:3 0 512M 0 part

Next, download the mkimage-rinse.sh script from the Docker repository on Github:

root@euca-172-17-112-207:~#  wget --no-check-certificate https://raw.github.com/dotcloud/docker/master/contrib/mkimage-rinse.sh
root@euca-172-17-112-207:~# chmod 755 mkimage-rinse.sh

Now we are ready to build the CentOS 6.5 Base Image.

Building the CentOS Image

The only thing left to do is build the base image.  Use mkimage-rinse.sh to build the CentOS 6.5 base image:

root@euca-172-17-112-207:~# ./mkimage-rinse.sh ubuntu/centos centos-6

After the installation is complete, test out the base image:

root@euca-172-17-112-207:~# docker run ubuntu/centos:6.5 cat /etc/centos-release
CentOS release 6.5 (Final)

Now we have a CentOS 6.5 base image.  The mkimage-rinse.sh script can also be used to install CentOS 5 base images as well.  Instead of passing centos-6, just pass centos-5.  For example, I have created a CentOS 5 base image in this instance as well.  Below shows the output of the images added to Docker:

root@euca-172-17-112-207:~# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
ubuntu/centos 5.10 06019bdea24b 2 minutes ago 123.8 MB
ubuntu/centos 6.5 cd23a8442c8a 2 hours ago 127.3 MB
root@euca-172-17-112-207:~# docker run ubuntu/centos:5.10 cat /etc/redhat-release
CentOS release 5.10 (Final)

As you can see, Docker helps developers test their software across multiple Linux distributions.  With Eucalyptus (just as in AWS), users can use the ephemeral nature of instances to quickly stand up this test environment in one instance.

Step-by-Step Deployment of Docker on Eucalyptus 3.4 for the Cloud Administrator

Docker

Eucalyptus Systems, Inc.

Docker has been in the news lately as one of the hot open-source project promoting linux containers. Some use cases for Docker include the following:

  • Automation of packaging and application deployment
  • Lightweight PaaS environments
  • Automated testing and continuous integration/deployment
  • Deploying and scaling web applications, databases and backend services

The focus of this blog entry is to show how to deploy Docker on Eucalyptus from a cloud administrator’s point-of-view – all in the cloud.  This is a step-by-step guide to create an Docker EMI from an existing Ubuntu Cloud Raring EMI using AWS’s documentation.  This entry will also show how to build euca2ools from source in the Ubuntu Cloud image.

Prerequisites

This entry assumes the following:

After confirming that the prerequisites are met, let’s get started.

Creating an EMI From an Existing EMI

As mentioned earlier, these steps will be based off of  AWS’s documentation on creating an instance store-backed AMI from an existing AMI.  In this example, here is an existing Ubuntu Raring instance thats running on Eucalyptus:

$ euca-describe-instances --region eucalyptus-admin@
RESERVATION r-3E423E33 961915002812 default
INSTANCE i-827E3E88 emi-06663A57 euca-10-104-7-12.eucalyptus.euca-hasp.eucalyptus-systems.com
euca-172-17-118-27.eucalyptus.internal running euca-admin 0 m1.medium 
2013-11-18T22:41:35.694Z LayinDaSmackDown eki-28F338EB eri-51253C0A 
monitoring-disabled 10.104.7.12 172.17.118.27 instance-store

This instance is using the following EMI, EKI and ERI:

$ euca-describe-images emi-06663A57 eki-28F338EB eri-51253C0A --region eucalyptus-admin@
IMAGE eki-28F338EB latest-raring-kernel/raring-server-cloudimg-amd64-vmlinuz-generic.manifest.xml 
441445882805 available public x86_64 kernel instance-store
IMAGE emi-06663A57 latest-raring/raring-server-cloudimg-amd64.img.manifest.xml 441445882805 
available public x86_64 machine eki-28F338EB eri-51253C0A instance-store paravirtualized
IMAGE eri-51253C0A latest-raring-kernel/raring-server-cloudimg-amd64-loader.manifest.xml 441445882805 
available public x86_64 ramdisk instance-store

To start, copy the zip file cloud administrator credentials obtained by the euca_conf command mentioned in the Eucalyptus 3.4 documentation to the running instance:

# scp -i euca-admin.priv admin.zip 
ubuntu@euca-10-104-7-12.eucalyptus.euca-hasp.eucalyptus-systems.com:/tmp/.

Next, install the following packages for the 3.8.0-33 kernel, and packages needed to build euca2ools:

ubuntu@euca-172-17-118-27:~$ sudo apt-get install python-setuptools git python-lxml unzip linux-headers-3.8.0-33-generic linux-image-extra-3.8.0-33-generic

Find the ephemeral storage using the instance metadata service, format, and mount the ephemeral to /mnt/image:

ubuntu@euca-172-17-118-27:~$ curl http://169.254.169.254/latest/meta-data/block-device-mapping/ephemeral
sda2
ubuntu@euca-172-17-118-27:~$ sudo mkdir /mnt/image
ubuntu@euca-172-17-118-27:~$ sudo mkfs.ext4 /dev/vda2
ubuntu@euca-172-17-118-27:~$ sudo mount /dev/vda2 /mnt/image

Download euca2ools from Github:

ubuntu@euca-172-17-118-27:~$ git clone https://github.com/eucalyptus/euca2ools.git

Install euca2ools:

ubuntu@euca-172-17-118-27:~$ cd euca2ools; sudo python setup.py install

Unzip the cloud administrator credentials in /tmp:

ubuntu@euca-172-17-118-27:~$ cd /tmp; unzip admin.zip

Change to the root user, and source the cloud administrator credentials:

ubuntu@euca-172-17-118-27:~$ sudo -s; source /tmp/eucarc

Bundle, upload and register the ramdisk and kernel under /boot:

root@euca-172-17-118-27:~# euca-bundle-image -i /boot/initrd.img-3.8.0-33-generic 
--ramdisk true -r x86_64
root@euca-172-17-118-27:~# euca-upload-bundle -b ubuntu-raring-docker-ramdisk 
-m /var/tmp/bundle-SQrAuT/initrd.img-3.8.0-33-generic.manifest.xml
root@euca-172-17-118-27:~# euca-register -n ubuntu-raring-docker-ramdisk 
ubuntu-raring-docker-ramdisk/initrd.img-3.8.0-33-generic.manifest.xml 
IMAGE eri-6BF033EE
root@euca-172-17-118-27:~# euca-bundle-image -i /boot/vmlinuz-3.8.0-33-generic 
--kernel true -r x86_64
root@euca-172-17-118-27:~# euca-upload-bundle -b ubuntu-raring-docker-kernel 
-m /var/tmp/bundle-31Lnxy/vmlinuz-3.8.0-33-generic.manifest.xml
root@euca-172-17-118-27:~# euca-register -n ubuntu-raring-docker-kernel 
ubuntu-raring-docker-kernel/vmlinuz-3.8.0-33-generic.manifest.xml
IMAGE eki-17093995

Use euca-bundle-vol to bundle the root filesystem. Make sure to exclude /tmp, /mnt/image, and /home/ubuntu. Additionally, make sure and set the size of the image to be 5 GB:

root@euca-172-17-118-27:~# euca-bundle-vol -p ubuntu-raring-docker 
-s 5120 -e /tmp,/root,/mnt/image,/home/ubuntu -d /mnt/image 
--kernel eki-17093995 --ramdisk eri-6BF033EE -r x86_64

Next, upload and register the root filesystem:

root@euca-172-17-118-27:~# euca-upload-bundle -b ubuntu-raring-docker-rootfs 
-m /mnt/image/ubuntu-raring-docker.manifest.xml
root@euca-172-17-118-27:~# euca-register -n ubuntu-raring-docker-rootfs 
ubuntu-raring-docker-rootfs/ubuntu-raring-docker.manifest.xml
IMAGE emi-26403979

We have the new EMI, EKI and ERI for the Docker instance.  Lastly, set the image permissions so that all users on the cloud can use the EMI, EKI and ERI:

root@euca-172-17-118-27:~# euca-modify-image-attribute -l -a all emi-26403979
root@euca-172-17-118-27:~# euca-modify-image-attribute -l -a all eki-17093995
root@euca-172-17-118-27:~# euca-modify-image-attribute -l -a all eri-6BF033EE

Now its time to launch the Docker EMI.

Running the Docker Instance with Cloud-Init

Before launching the EMI, the cloud-init configuration file needs to be created.  This file will be responsible for configuring the instance repositories, downloading and installing Docker.  With your favorite command-line editor, create a file called cloud-init-docker.config, with the following content:

#cloud-config
apt_update: true
apt_upgrade: true
disable_root: true
packages:
 - less
cloud_config_modules:
 - ssh
 - [ apt-update-upgrade, always ]
 - updates-check
 - runcmd
runcmd:
 - [ sh, -xc, "INST_HOSTNAME=`/bin/hostname`; META_IP=`curl -s http://169.254.169.254/latest/meta-data/local-ipv4`; echo ${META_IP} ${INST_HOSTNAME} >> /etc/hosts" ]
 - [ locale-gen, en_US.UTF-8 ]
 - [ sh, -xc, "wget -qO docker-io.gpg https://get.docker.io/gpg" ]
 - [ apt-key, add, docker-io.gpg ]
 - [ sh, -xc, "echo 'deb http://get.docker.io/ubuntu docker main' > /etc/apt/sources.list.d/docker.list" ]
 - [ apt-get, update ]
 - [ apt-get, install, -y, --force-yes, lxc-docker ]
 - [ modprobe, -q, aufs ]

Now, use euca-run-instances to launch the instance:

root@euca-172-17-118-27:~# euca-run-instances -k euca-admin emi-351237D1 
-t m1.medium --user-data-file cloud-init-docker.config

After launching the instance, leave the current instance to get back to end client.

root@euca-172-17-118-27:~# exit
exit
ubuntu@euca-172-17-118-27:~$ exit
logout
Connection to 10.104.7.12 closed.

Once the instance reaches running state, ssh into the instance using the keypair specified (which in this case will be euca-admin.priv), and execute the following Docker command to run an interactive shell session inside a minimal Ubuntu container:

$ euca-describe-instances --region eucalyptus-admin@
RESERVATION r-A1613D7F 961915002812 default
INSTANCE i-AFDB3D4C emi-26403979 euca-10-104-7-13.eucalyptus.euca-hasp.eucalyptus-systems.com 
euca-172-17-118-16.eucalyptus.internal running euca-admin 0 m1.medium 
2013-11-19T01:21:10.880Z LayinDaSmackDown eki-17093995 eri-6BF033EE monitoring-disabled 
10.104.7.13 172.17.118.16 instance-store
# ssh -i euca-admin.priv ubuntu@euca-10-104-7-13.eucalyptus.euca-hasp.eucalyptus-systems.com
Welcome to Ubuntu 13.04 (GNU/Linux 3.8.0-33-generic x86_64)
* Documentation: https://help.ubuntu.com/
System information as of Thu Nov 14 23:18:38 UTC 2013
System load: 0.0 Users logged in: 0
 Usage of /: 21.6% of 4.89GB IP address for eth0: 172.17.184.76
 Memory usage: 4% IP address for lxcbr0: 10.0.3.1
 Swap usage: 0% IP address for docker0: 10.1.42.1
 Processes: 83
Graph this data and manage this system at https://landscape.canonical.com/
Get cloud support with Ubuntu Advantage Cloud Guest:

http://www.ubuntu.com/business/services/cloud

Use Juju to deploy your cloud instances and workloads:

https://juju.ubuntu.com/#cloud-raring

New release '13.10' available.
Run 'do-release-upgrade' to upgrade to it.
Last login: Thu Nov 14 23:08:09 2013 from 10.104.10.6
ubuntu@euca-172-17-184-76:~$ sudo docker run -i -t ubuntu /bin/bash
Unable to find image 'ubuntu' (tag: latest) locally
Pulling repository ubuntu
8dbd9e392a96: Download complete
b750fe79269d: Download complete
27cf78414709: Download complete
root@041d5ddcd6b9:/# (Ctrl-p Ctrl-q to exit out of shell)
ubuntu@euca-172-17-184-76:~$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
041d5ddcd6b9 ubuntu:12.04 /bin/bash 27 seconds ago Up 26 seconds pink_frog

Thats it!  For more information regarding Docker, please refer to the latest Docker documentation.

Enjoy!

Demo of Eucalyptus hotness: 3.3 milestone 6

hspencer77:

Eucalyptus 3.3 Demo: More AWS Compatibility Goodness..:-)

Originally posted on Greg DeKoenigsberg Speaks:

Our demo day for milestone 6 was yesterday, and it was choice. We’re at feature completeness at this point, and we’re now on final approach for release sometime Soon-ish, as soon as we shake out all the code nasties. We’ve got some good stuff to show off on Vimeo. The basic transcript:

  • 0:00 Eric Choi, Product Mktg Manager, with agenda/housekeeping.
  • 1:25 Tim Cramer, VP of Engineering, sets the table for demos.
  • 3:45 Yours Truly, VP of Community, talks (a little too long) about Eucalyptus compatibility with the AWS Ruby SDK.
  • 16:00 Vic Iglesias, QA Lead, talks about EucaLobo, an amazing fork of ElasticWolf that provides an autoscaling UI for both Eucalyptus and AWS. (Watch this one twice; it’s really the star of the show.)
  • 24:40 Vic Iglesias talks about Micro-QA, the self-contained environment for running automated testing on Eucalyptus installations.
  • 31:30 Chris Grzegorczyk, Co-founder and Chief Architect, shows Asgard…

View original 59 more words

Big Data on the Cloud using Ansible, RHadoop, AppScale, and AWS/Eucalyptus

Background

Big Data has been a hot topic over the last few years.  Big Data on public clouds, such as AWS’s Elastic MapReduce, has been gaining even more popularity as cloud computing becomes more of an industry standard.

R  is an open source project for statistical computing and graphics.  It has been growing in popularity for doing  linear and nonlinear modeling, classical statistical tests, time-series analysis and others, at various Universities and companies.

R Project

R Project

RHadoop was developed by Revolution Analytics to interface with Hadoop.  Revolution Analytics builds analytic software solutions using R.

Revolution Analytics

Revolution Analytics

AppScale is an open source PaaS that implements the Google AppEngine API on IaaS environments.  One of the Google AppEngine APIs that is implemented is AppEngine MapReduce.   The back-end support for this API that AppScale using Cloudera’s Distribution for Apache Hadoop.

AppScale Inc.

AppScale Inc.

Ansible is an open source orchestration software that utilizes SSH for handling configuration management for physical/virtual machines, and machines running in the cloud.

Ansible Works

Ansible Works

Amazon Web Services is a public IaaS that provides infrastructure and application services in the cloud.  Eucalyptus is an open source software solution that provides the AWS APIs for EC2, S3, and IAM for on-premise cloud environments.

Amazon AWS EC2

Amazon AWS EC2

Eucalyptus Systems Inc.

Eucalyptus Systems Inc.

This blog entry will cover how to deploy AppScale (either on AWS or Eucalyptus), then use Ansible to configure each AppScale node with R, and the RHadoop packages in order allow programs written in R to utilize MapReduce in the cloud.

Pre-requisites

To get started, the following is needed on a desktop/laptop computer:

*NOTE:  These variables are used by AppScale Tools version 1.6.9.  Check the AWS and Eucalyptus documentation regarding obtaining user credentials. 

Deployment

AppScale

After installing AppScale Tools and Ansible, the AppScale cluster needs to be deployed.  After defining the AWS/Eucalyptus variables,  initialize the creation of the AppScale cluster configuration file – AppScalefile.

$ ./appscale-tools/bin/appscale init cloud

Edit the AppScalefile, providing information for the keypair, security group, and AppScale AMI/EMI.  The keypair and security group do not need to be pre-created. AppScale will handle this.  The AppScale AMI on AWS (us-east-1) is ami-4e472227.  The Eucalyptus EMI will be unique based upon the Eucalyptus cloud that is being used.  In this example, the AWS AppScale AMI will be used, and the AppScale cluster size will be 3 nodes.  Here is the example AppScalefile:

---
group : 'appscale-rmr'
infrastructure : 'ec2'
instance_type : 'm1.large'
keyname : 'appscale-rmr'
machine : 'ami-4e472227'
max : 3
min : 3
table : 'hypertable'

After editing the AppScalefile, start up the AppScale cluster by running the following command:

$ ./appscale-tools/bin/appscale up

Once the cluster finishes setting up, the status of the cluster can be seen by running the command below:

$ ./appscale-tools/bin/appscale status

R, RHadoop Installation Using Ansible

Now that the cluster is up and running, grab the Ansible playbook for installing R, and RHadoop rmr2 and rhdfs packages onto the AppScale nodes.  The playbook can be downloaded from github using git:

$ git clone https://github.com/hspencer77/ansible-r-appscale-playbook.git

After downloading the playbook, the ansible-r-appscale-playbook/production file needs to be populated with the information of the AppScale cluster.  Grab the cluster  node information by running the following command:

$ ./appscale-tools/bin/appscale status | grep amazon | grep Status | awk '{print $5}' | cut -d ":" -f 1
ec2-50-17-96-162.compute-1.amazonaws.com
ec2-50-19-45-193.compute-1.amazonaws.com
ec2-67-202-23-157.compute-1.amazonaws.com

Add those DNS entries to the ansible-r-appscale-playbook/production file.  After editing, the file will look like the following:

[appscale-nodes]
ec2-50-17-96-162.compute-1.amazonaws.com
ec2-50-19-45-193.compute-1.amazonaws.com
ec2-67-202-23-157.compute-1.amazonaws.com

Now the playbook can be executed.  The playbook requires the SSH private key to the nodes.  This key will be located under the ~/.appscale folder.  In this example, the key file is named appscale-rmr.key.  To execute the playbook, run the following command:

$ ansible-playbook -i r-appscale-deployment/production 
--private-key=~/.appscale/appscale-rmr.key -v r-appscale-deployment/site.yml

Testing Out The Deployment – Wordcount.R

Once the playbook has finished running, the AppScale cluster is now ready to be used.  To test out the setup, SSH into the head node of the AppScale cluster.  To find out the head node of the cluster, execute the following command:

$ ./appscale-tools/bin/appscale status

After discovering the head node, SSH into the head node using the private key located in the ~/.appscale directory:

$ ssh -i ~/.appscale/appscale-rmr.key root@ec2-50-17-96-162.compute-1.amazonaws.com

To test out the R setup on all the nodes, grab the wordcount.R program:

root@appscale-image0:~# tar zxf rmr2_2.0.2.tar.gz rmr2/tests/wordcount.R

In the wordcount.R file, the following lines are present

rmr2:::hdfs.put("/etc/passwd", "/tmp/wordcount-test")
out.hadoop = from.dfs(wordcount("/tmp/wordcount-test", pattern = " +"))

When the wordcount.R program is executed, it will grab the /etc/password file from the head node, copy it to the hdfs filesystem, then run wordcount on /etc/password to look for the pattern ” +”.   NOTE: wordcount.R can be edited to use any file and pattern desired.

Run wordcount.R:

root@appscale-image0:~# R

R version 2.15.3 (2013-03-01) -- "Security Blanket"
Copyright (C) 2013 The R Foundation for Statistical Computing
ISBN 3-900051-07-0
Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY.
You are welcome to redistribute it under certain conditions.
Type 'license()' or 'licence()' for distribution details.

  Natural language support but running in an English locale

R is a collaborative project with many contributors.
Type 'contributors()' for more information and
'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or
'help.start()' for an HTML browser interface to help.
Type 'q()' to quit R.

[Previously saved workspace restored]

> source('rmr2/tests/wordcount.R')
Loading required package: Rcpp
Loading required package: RJSONIO
Loading required package: digest
Loading required package: functional
Loading required package: stringr
Loading required package: plyr
13/04/05 02:33:41 INFO security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing.
13/04/05 02:33:43 INFO security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing.
packageJobJar: [/tmp/RtmprcYtsu/rmr-local-env19811a7afd54, /tmp/RtmprcYtsu/rmr-global-env1981646cf288, /tmp/RtmprcYtsu/rmr-streaming-map198150b6ff60, /tmp/RtmprcYtsu/rmr-streaming-reduce198177b3496f, /tmp/RtmprcYtsu/rmr-streaming-combine19813f7ea210, /var/appscale/hadoop/hadoop-unjar5632722635192578728/] [] /tmp/streamjob8198423737782283790.jar tmpDir=null
13/04/05 02:33:44 WARN snappy.LoadSnappy: Snappy native library is available
13/04/05 02:33:44 INFO util.NativeCodeLoader: Loaded the native-hadoop library
13/04/05 02:33:44 INFO snappy.LoadSnappy: Snappy native library loaded
13/04/05 02:33:44 INFO mapred.FileInputFormat: Total input paths to process : 1
13/04/05 02:33:44 INFO streaming.StreamJob: getLocalDirs(): [/var/appscale/hadoop/mapred/local]
13/04/05 02:33:44 INFO streaming.StreamJob: Running job: job_201304042111_0015
13/04/05 02:33:44 INFO streaming.StreamJob: To kill this job, run:
13/04/05 02:33:44 INFO streaming.StreamJob: /root/appscale/AppDB/hadoop-0.20.2-cdh3u3/bin/hadoop job  -Dmapred.job.tracker=10.77.33.247:9001 -kill job_201304042111_0015
13/04/05 02:33:44 INFO streaming.StreamJob: Tracking URL: http://appscale-image0:50030/jobdetails.jsp?jobid=job_201304042111_0015
13/04/05 02:33:45 INFO streaming.StreamJob:  map 0%  reduce 0%
13/04/05 02:33:51 INFO streaming.StreamJob:  map 50%  reduce 0%
13/04/05 02:33:52 INFO streaming.StreamJob:  map 100%  reduce 0%
13/04/05 02:33:59 INFO streaming.StreamJob:  map 100%  reduce 33%
13/04/05 02:34:02 INFO streaming.StreamJob:  map 100%  reduce 100%
13/04/05 02:34:04 INFO streaming.StreamJob: Job complete: job_201304042111_0015
13/04/05 02:34:04 INFO streaming.StreamJob: Output: /tmp/RtmprcYtsu/file1981524ee1a3
13/04/05 02:34:05 INFO security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing.
13/04/05 02:34:07 INFO security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing.
13/04/05 02:34:08 INFO security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing.
13/04/05 02:34:10 INFO security.UserGroupInformation: JAAS Configuration already set up for Hadoop, not re-installing.
Deleted hdfs://10.77.33.247:9000/tmp/wordcount-test
>quit("yes")

Thats it!  The AppScale cluster is ready for additional R programs that utilize MapReduce.   Enjoy the world of Big Data on public/private IaaS.

What’s new in Ansible 1.1 for AWS and Eucalyptus users?

hspencer77:

Ansible providing more AWS/Eucalyptus support!

Originally posted on Take that to the bank and cash it!:

I thought the Ansible 1.0 development cycle was busy but 1.1 is crammed full of orchestration goodness.  On Tuesday, 1.1 was released and you can read more about it here: http://blog.ansibleworks.com/2013/04/02/ansible-1-1-released/

For those working on AWS and Eucalyptus, 1.1 brings some nice module improvements as well as a new cloudformation and s3 module.  It’s great to see the AWS-related modules becoming so popular so quickly.  Here are some more details about the changes but you can find info in the changelog here: https://github.com/ansible/ansible/blob/devel/CHANGELOG.md

Security group ID support

It’s now possible to specify the security group by its ID.  This is quite typically behaviour in EC2 and Eucalyptus will support this with the pending 3.3 release.  The parameter is optional.

VPC subnet ID

VPC users can now specify a subnet ID associated with their instance.

Instance state wait timeout

In 1.0 there was no way to specify how long to wait…

View original 300 more words

Great Opensource Cloud Management Tool Using SSH – Simple, Powerful and Elegant

hspencer77:

Configuration management through SSH FTW!

Originally posted on Gigaom:

A couple of former Red Hat (s rhat) veterans think there’s an easier way to configure, deploy and manage IT across an organization and founded AnsibleWorks to attack that problem.

Ansible co-founder Said Ziouani.

Ansible co-founder Said Ziouani.

Systems administrators and developers want one tool for deployment, configuration and management — they don’t want to deal with agents and add-ons, said Said Siouani, CEO of Santa Barbara, Calif.-based AnsibleWorks.

No doubt Ansible’s orchestration engine will face off against popular configuration tools like Opscode Chef and Puppet Labs’ Puppet (see disclosure). Siouani characterized Ansible as a more “holistic” solution than what is on the market now in that it focuses on configuration management and actual deployment.

Ansible co-founder Michael DeHaan.

Ansible co-founder Michael DeHaan.

Ansible itself is an open-source project kicked off a year ago by Michael DeHaan, one of the Red Hat vets who also spent time at Puppet Labs (see disclosure) and was also the force behind Cobbler

View original 137 more words

Ansible Playbook for Eucalyptus Deployment == Next Level Configuration Management for the Cloud

hspencer77:

Ansible playbook for Eucalyptus…NICE! Great work Lester!

Originally posted on Take that to the bank and cash it!:

The first cut of the Ansible deployment playbook for deploying Eucalyptus private clouds is ready.  I’ve merged the first “release” into the master branch here: https://github.com/lwade/eucalyptus-playbook. Feedback and contributions are very welcome, please file issues against the project.

This playbook allows a user to deploy a single front-end cloud (i.e. all component on a single system) and as many NC’s as they want.  Although admittedly I’ve only tested with one so far.  I’ve followed, to a certain degree, the best practices described here:  http://ansible.cc/docs/bestpractices.html

Overall I’m pretty happy with it, there are some areas which I’d like to re-write and improve but the groundwork is there.  It was all very fluid to start with and doing something multi-tier has been enjoyable. I’ve also learnt what its like to automate a deployment of Eucalyptus and there are probably a number of things we should improve to make it easier in this…

View original 614 more words

UCSB Sweethearts of Cloud Computing: AppScale and Eucalyptus

hspencer77:

AppScale and Eucalyptus: Great Tag Team for On-Premise Cloud Computing

Originally posted on shaon's blog:

Cloud computing is the use of computing resources (hardware and software) that are delivered as a service over a network (typically the Internet). – Wikipedia

According to Wikipedia currently there are few popular service models exist.

1. Infrastructure as a service (IaaS)
2. Platform as a service (PaaS)
3. Software as a service (SaaS)

So, I have an Eucalyptus cloud, which is great, serves as AWS-like IaaS platform. But now I want PaaS. And right here Appscale comes into play with full compatibility of Google App Engine (GAE) applications. In this post, we will install the popular open source PaaS framework Appscale on Eucalyptus, the AWS compatible open source IaaS platform.

Agenda
0. Introduction
1. Resize Lucid image
2. Install Appscale from source
3. Install Appscale Tool
4. Bundle Appscale image
5. Run Appscale
6. Run an application on Appscale

Eucalyptus
Eucalyptus Cloud platform is open source software for building…

View original 662 more words

Test Drive: Drupal Deployment on Eucalyptus using Stackato, Amazon Route 53 and the Eucalyptus Community Cloud

Recently, I did a blog discussing how to deploy a Jenkins server using Stackato, running on Eucalyptus.  At the end of that blog, I mentioned how the Eucalyptus Community Cloud (ECC) could be used for testing out the Stackato Microcloud image on Eucalyptus.   The previous blog – I felt – was more for DevOps administrators who had access to their own on-premise Eucalyptus clouds.  The inspiration of this blog comes from the blog on ActiveBlog entitled “Deploy & Scale Drupal on Any Cloud with Stackato” to show love to Web Developers, and show the power of Amazon’s Route 53.

Test Drive Pre-Reqs

The prerequisites for this blog are the same that are mentioned in my previous blog regarding using Stackato on Eucalyptus (for the Eucalyptus pre-reqs, make sure the ECC is being used).  In addition to the prerequisites mentioned above, the following is needed:

After the prerequisites have been met, its time to setup the Drupal environment.

Test Drive Engage!

Since the ECC is being used, there is no need to worry about bundling, uploading and registering the Stackato image.  The Stackato image used for this blog is as follows:

IMAGE emi-859B3D5C stackato_v2.6.6/stackato-cloudinit.manifest.xml
150820662310 available public x86_64 machine eki-6FBE3D2D eri-67463B77 instance-store

Next, lets make sure the user has an elastic IP that will be used in AWS Route 53, and a security group to allow proper network traffic to the instance.  Do the following:

  1. Make sure the user credentials are sourced correctly, and euca2ools is installed correctly.
  2. Grab an elastic IP using euca-allocate-address (in this example 173.205.188.105 was allocated):

    # euca-allocate-address
    ADDRESS 173.205.188.105

  3. If the user already doesn’t have a keypair, create a keypair for the user by using euca-create-keypair, and make sure the permission of the file is 0600:  

    # euca-create-keypair hspencer-stackato > hspencer-stackato.priv 
    # chmod 0600 hspencer-stackato.priv

  4. Create a security group for the instance to use:

    # euca-create-group stackato-test -d "Test Security Group for Stackato PaaS"
    GROUP stackato-test Test Security Group for Stackato PaaS

  5. Authorize ping, ssh, http, and https ports:

    # euca-authorize -P icmp -t -1:-1 -s 0.0.0.0/0 stackato-test
    GROUP stackato-test
    PERMISSION stackato-test ALLOWS icmp -1 -1 FROM CIDR 0.0.0.0/0
    
    # euca-authorize -P tcp -p 22 -s 0.0.0.0/0 stackato-test
    GROUP stackato-test
    PERMISSION stackato-test ALLOWS tcp 22 22 FROM CIDR 0.0.0.0/0
    
    # euca-authorize -P tcp -p 80 -s 0.0.0.0/0 stackato-test
    GROUP stackato-test
    PERMISSION stackato-test ALLOWS tcp 80 80 FROM CIDR 0.0.0.0/0
    
    # euca-authorize -P tcp -p 443 -s 0.0.0.0/0 stackato-test
    GROUP stackato-test
    PERMISSION stackato-test ALLOWS tcp 443 443 FROM CIDR 0.0.0.0/0

  6. Now, launch the instance, specifying the keypair name to use, and a VM type.  On the ECC, only m1.xlarge and c1.xlarge meet the requirements of launching the Stackato image:

    # euca-run-instances -k hspencer-stackato -t c1.xlarge emi-859B3D5C -g stackato-test
    RESERVATION r-66EE4030 628376682871 stackato-test
    INSTANCE i-E85843C4 emi-859B3D5C euca-0-0-0-0.eucalyptus.ecc.eucalyptus.com euca-0-0-0-0.eucalyptus.internal
     pending hspencer-stackato 0 c1.xlarge 2013-02-24T19:40:35.516Z partner01 eki-6FBE3D2D
     eri-67463B77 monitoring-disabled 0.0.0.0 0.0.0.0 instance-store

  7. Once the instance gets to a running state, associate the elastic IP that the user owns to the instance:

    # euca-describe-instances
    RESERVATION r-66EE4030 628376682871 stackato-test
    INSTANCE i-E85843C4 emi-859B3D5C euca-173-205-188-106.eucalyptus.ecc.eucalyptus.com
     euca-10-9-190-24.eucalyptus.internal running hspencer-stackato 0 c1.xlarge 
    2013-02-24T19:40:35.516Z partner01 eki-6FBE3D2D eri-67463B77 monitoring-disabled
     173.205.188.10 10.9.190.24 instance-store
    
    # euca-associate-address -i i-E85843C4 173.205.188.105
    ADDRESS 173.205.188.105 i-E85843C4
    
    # euca-describe-instances
    RESERVATION r-66EE4030 628376682871 stackato-test
    INSTANCE i-E85843C4 emi-859B3D5C euca-173-205-188-105.eucalyptus.ecc.eucalyptus.com
     euca-10-9-190-24.eucalyptus.internal running hspencer-stackato 0 c1.xlarge 2013-02-24T19:40:35.516Z
     partner01 eki-6FBE3D2D eri-67463B77 monitoring-disabled 173.205.188.10 10.9.190.24 instance-store

  8. Log into the AWS management console,  select Route 53, and setup the A and CNAME records in your domain as mentioned here under the Stackato Documentation regarding detailed DNS configuration.  In this example, the DNS name associated with the elastic IP 173.205.188.105 is stackato-dev.mindspew-age.com.
  9. Next ssh into the instance, and proceed to follow the steps for setting up the Stackato instance that is mentioned in my previous blog under the section Configuration of the Stackato Instance.  Make sure the DNS name setup in AWS Route 53 is used with “kato rename public-DNS-name” and “kato setup core api.public-DNS-name” configuration steps.
  10. After the instance is configured, just open up the browser and go to the DNS name set up for the Stackato instance in AWS Route 53, as mentioned in the Stackato Documentation regarding configuration via the Management Console.
  11. Once logged into the Stackato Management Console, select “App Store” in the lefthand menu and select “Drupal” to install

    App Store - Drupal Application

    App Store – Drupal Application

     

  12. After Drupal has installed, start the application.  Once it has started successfully, select the URL that shows up in the right-hand menu box.  The Drupal log-in page will appear in your browser

    Drupal Landing Page

    Drupal Landing Page

Thats it!  Now Drupal is ready for any web developer to test out on the ECC.  If there is any questions/comments/suggestions, please feel free to leave comments.  Enjoy!

Jenkins, Stackato, Cloud-Init and Eucalyptus == Potent Combination for an On-Premise Continuous Integration Environment

The Ingredients

Jenkins

An extendable open source continuous integration server.

Stackato

The Enterprise Private PaaS that makes it easy to deploy, manage, and monitor applications on any cloud.

Cloud-init

The Ubuntu package that handles early initialization of a cloud instance. It is installed in the Ubuntu Cloud Images and also in the official Ubuntu images available on EC2.

Eucalyptus

Allows you to build production-ready, AWS-compatible private and hybrid clouds by leveraging your existing virtualized infrastructure to create on-demand cloud resource pools.

What happens when you combine all three of these tools?  A potent combination for continuous integration on a easy-to-configure PaaS and an on-premise, AWS-compatible IaaS.  With this combination, developers can take advantage of easy configuration that Stackato brings to the table, running on top of Eucalyptus – bringing an AWS-like cloud environment into your datacenter.

This blog entry will discuss the steps that I took to get Jenkins installed on a Stackato instance store-backed instance running on Eucalyptus.  But before I get started, I would like to thank the folks from ActiveState for all their guidance.  Their support staff is really top notch, and very helpful.  Check them out in #stackato on freenode.net.  They can also be checked out on Twitter at @ActiveState. Now on to the dirty work…..

The Recipe for Success

The Stackato Microcloud Image and Cloud-Init

To begin, the following is needed:

After downloading the Stackato VM for KVM and unzipping the file, we will need to pull out the root file system, the kernel and ramdisk.  These will be uploaded, bundled and registered as the EMI, EKI, and ERI.  To extract the root filesystem, do the following:

  1. Use parted to locate the root filesystem as follows:    
    # parted stackato-img-kvm-v2.6.6.img
    GNU Parted 2.1
    Using /root/images/Stackato-VM/stackato-img-kvm-v2.6.6.img
    Welcome to GNU Parted! Type 'help' to view a list of commands.
    (parted) U
    Unit? [compact]? b
    (parted) p
    Model: (file)
    Disk /root/images/Stackato-VM/stackato-img-kvm-v2.6.6.img: 10737418240B
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos
    
    Number Start End Size Type File system Flags
    1 1048576B 200278015B 199229440B primary ext3 boot
    3 200278016B 1237319679B 1037041664B primary linux-swap(v1)
    2 1237319680B 10736369663B 9499049984B primary ext4
    
    (parted) quit
  2. In this example, the root filesystem is partition 2.  The value for “Start” and “Size” will need to be used.  Next, run dd to extract the root filesystem:
    dd if=stackato-img-kvm-v2.6.6.img of=stackato-rootfs.img
     bs=1 skip=1237319680 count=9499049984
    
  3. Once it has completed, mount  stackato-rootfs.img to the loopback device:
    mount -o loop stackato-rootfs.img /mnt/
  4. Copy out initrd.img-3.2.0-27-virtual  and vmlinuz-3.2.0-27-virtual from /mnt/boot.
  5. In /mnt/etc/fstab, replace the UUID entry with LABEL.  The LABEL will look simliar to the following:  
    LABEL=cloudimg-rootfs    /               ext4   defaults     1       1
  6. Chroot to /mnt – there may be a need to do a mount -o bind for sys, dev, and proc.
  7. Run “dpkg-reconfigure cloud-init”, and make sure that the EC2 Data Source is selected.
  8. Unmount stackato-rootfs.img (if sys, dev, and proc were mounted, unmount them before unmounting stackato-rootfs.img).  After it has been unmounted, run tune2fs to change the label of the image:
    tune2fs -L cloudimg-rootfs stackato-rootfs.img

After following these steps, the following should be available:

  • initrd.img-3.2.0-27-virtual – to be bundled, uploaded and registered as the ERI
  • vmlinuz-3.2.0-27-virtual – to be bundled, uploaded and registered as the EKI
  • stackato-rootfs.img – to be bundled, uploaded and registered as the EMI

Go through the steps of bundling, uploading and registering the ERI, EKI, and EMI.  For additional information, please refer to the Add an Image section of the Eucalyptus 3.2 User Guide.

Launching the Stackato Image

Now its time to launch the Stackato image on Eucalyptus.  Since cloud-init has the enabled EC2 data source now, when the image is launched, the instance will grab ssh keys, and mount the ephemeral storage.  Also, additional configuration can be passed using the user-data file option.  More information regarding this can be found on Stackato’s documentation in reference to using cloud-init.  Key thing to remember here is that the minimum RAM requirement for the Stackato image is 2 gigs.  Make sure the VM type used for launching the Stackato image has at least 2 gigs of RAM or more.  In this example, the image ID is emi-DAB23A8A.  The ramdisk and kernel are registered as eri-9B453C09 and eki-ADF640B0.   The VM type c1.xlarge is used, which has 4 CPU,  4096 MB of RAM,  and    50 Gigs of disk space.

euca-run-instances -k admin emi-DAB23A8A -t c1.xlarge 
--kernel eki-ADF640B0 --ramdisk eri-9B453C09

Use euca-describe-instances to check to see when the instance reaches a running state:

euca-describe-instances i-100444EF
RESERVATION r-CC69438B 345590850920 default
INSTANCE i-100444EF emi-DAB23A8A euca-192-168-55-104.wu-tang.euca-hasp.eucalyptus-systems.com 
euca-10-106-101-17.wu-tang.internal running admin 0 c1.xlarge 2013-02-23T00:34:07.436Z enter-the-wu eki-ADF640B0 
eri-9B453C09 monitoring-disabled 192.168.55.104 10.106.101.17 instance-store

The key thing for running a Stackato instance is setting up the correct DNS entries.  For more information regarding setting up DNS with regards to a Stackato instance, please read the Detail Configuration section on DNS in the Stackato online documentation.  For this example, instead of using an external DNS service using a tool like nsupdate, to configure the A record and CNAME records, we will use xip.io. xip.io is a magic domain name that provides wildcard DNS for any IP address.  Next, its time to configure the Stackato instance.

Configuration of the Stackato Instance

To configure the Stackato instance, do the following:

  1. SSH into the instance.  

    ssh -i creds/admin.priv 
    stackato@euca-192-168-55-104.wu-tang.euca-hasp.eucalyptus-systems.com
  2. Make note of the ip address associated with eth0 and the netmask using ifconfig.  Also note the gateway IP by using the route command.
  3. Run “kato op static_ip” to configure the static IP address for eth0.  Make sure and add 127.0.0.1 as the first entry as part of the nameservers, and add “containers.” as the first entry under the search domains.
  4. Run “kato rename public DNS name “, where public DNS name includes the public IP of the instance, using xip.io (e.g. 192.168.55.104.xip.io)
  5. Run “kato disable mdns”, then run “sudo reboot” to reboot the instance.
  6. Once the instance has come back up, ssh into the instance, and run the following command  “kato setup core api.public DNS name” where public DNS name is the same value used for the “kato rename” step (e.g.  192.168.55.104.xip.io).
  7. Next, edit /etc/resolv.conf and make sure that the value for the search option is “containers.”, and the first entry for the nameservers is 127.0.0.1.
  8. Finally, run “kato enable –all-but mdns”

Thats it!  Now go to the public DNS name that was used in your favorite browser.  For this example, 192.168.55.104.xip.io was used.  The following landing page should look similar to what you see here in the Stackato documentation regarding accessing the instance through the management console.

Setting Up Jenkins

After setting up the admin account, navigate to the “App Store” on the lefthand menu.  Once selected, navigate to find the Jenkins application:

 

Jenkins Application

Jenkins Application

 

After selecting to install Jenkins, select “Yes” to install.  After the installation takes place,  select “Applications” in the left hand menu.  From there, select the Jenkins application, and select “Start” (its the green arrow to the right of the application window).   Once it has started, you will see the following:

Jenkins Running in Stackato

Jenkins Running in Stackato

 

Now Jenkins is ready to be used.

If anyone wants to test this out on Eucalyptus but doesn’t have access to their own Eucalyptus cloud, fear not, the Eucalyptus Community Cloud has the Stackato image available.  After applying to get access to the Community Cloud, follow the steps above.  The image for Stackato is as follows:

IMAGE emi-859B3D5C stackato_v2.6.6/stackato-cloudinit.manifest.xml 150820662310 available public x86_64 machine eki-6FBE3D2D eri-67463B77 instance-store

And as always, this image and steps can be used on AWS EC2 as well. :-)

Let me know if there are any questions.  Feedback is always welcome.  Enjoy!