Setting Up 3-Factor Authentication (Keypair, Password, Google Authenticator) for Eucalyptus Cloud Instances

Recently, I was logging into my AWS account, where I have multi-factor authentication (MFA) enabled, using the Google Authenticator application on my smart phone.  This inspired me to research how to enable MFA for any Linux distribution.  I ran across the following blog entries:

From there, I figured I would try to create a Eucalyptus EMI that would support three-factor authentication on a Eucalyptus 4.0 cloud.  The trick here was to figure out how to display the Google Authenticator information so users could configure Google Authenticator.  The euca2ools command ‘euca-get-console-output‘ proved to be the perfect mechanism to provide this information to the cloud user.  This blog will show how to configure an Ubuntu Trusty (14.04) Cloud image to support three-factor authentication.

Prerequisites

In order to leverage the steps mentioned in this blog, the following is needed:

Now that the prereqs have been mentioned, lets get started.

Updating the Ubuntu Image

Before we can update the Ubuntu image, let’s download the image:

[root@odc-f-13 ~]# wget http://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img

After the image has been downloaded successfully, the image needs to be converted to a raw format.  Use qemu-img for this conversion:

[root@odc-f-13 ~]# qemu-img convert -O raw trusty-server-cloudimg-amd64-disk1.img trusty-server-cloudimg-amd64-disk1.raw

After converting the image to a raw format, we need to mount it in order to update the image accordingly:

[root@odc-f-13 ~]# losetup /dev/loop0 trusty-server-cloudimg-amd64-disk1.raw
[root@odc-f-13 ~]# kpartx -av /dev/loop0
add map loop0p1 (253:2): 0 4192256 linear /dev/loop0 2048
[root@odc-f-13 ~]# mkdir /mnt/ubuntu
[root@odc-f-13 ~]# mount /dev/mapper/loop0p1 /mnt/ubuntu
[root@odc-f-13 ~]# chroot /mnt/ubuntu

The above command ‘chroot’ allows us to edit the image as if its the current running Linux operating system.  We have to install a couple of packages in the image.  Before we do, use the resolvconf to create the necessary information in /etc/resolv.conf.

root@odc-f-13:/# resolvconf -I

Confirm the settings are correct by running ‘apt-get update’:

root@odc-f-13:/#  apt-get update

Once that command runs successfully, install the PAM module for Google Authenticator and the whois package:

root@odc-f-13:/# apt-get install libpam-google-authenticator whois

After these packages have been installed, run the ‘google-authenticator’ command to see all the available options:

root@odc-f-13:/# google-authenticator --help
google-authenticator [<options>]
 -h, --help Print this message
 -c, --counter-based Set up counter-based (HOTP) verification
 -t, --time-based Set up time-based (TOTP) verification
 -d, --disallow-reuse Disallow reuse of previously used TOTP tokens
 -D, --allow-reuse Allow reuse of previously used TOTP tokens
 -f, --force Write file without first confirming with user
 -l, --label=<label> Override the default label in "otpauth://" URL
 -q, --quiet Quiet mode
 -Q, --qr-mode={NONE,ANSI,UTF8}
 -r, --rate-limit=N Limit logins to N per every M seconds
 -R, --rate-time=M Limit logins to N per every M seconds
 -u, --no-rate-limit Disable rate-limiting
 -s, --secret=<file> Specify a non-standard file location
 -w, --window-size=W Set window of concurrently valid codes
 -W, --minimal-window Disable window of concurrently valid codes

Updating PAM configuration

Next the PAM configuration file /etc/pam.d/common-auth needs to be updated.  Find the following line in that file:

auth[success=1 default=ignore]pam_unix.so nullok_secure

Replace it with the following lines:

auth requisite pam_unix.so nullok_secure
auth requisite pam_google_authenticator.so
auth [success=1 default=ignore] pam_permit.so

Next, we need to update SSHD configuration.

Update SSHD configuration

We need to modify the /etc/ssh/sshd_config file to help make sure the Google Authenticator PAM module works successfully.  Modify/add the following lines to the /etc/ssh/sshd_config file:

ChallengeResponseAuthentication yes
AuthenticationMethods publickey,keyboard-interactive

Updating Cloud-Init Configuration

The next modification involves enabling the ‘ubuntu‘ user to have a password.  By default, the account is locked (i.e. doesn’t have a password assigned) in the cloud-init configuration file.  For this exercise, we will enable it, and assign a password.  Just like the old Ubuntu Cloud images, we will assign the ‘ubuntu‘ user the password ‘ubuntu‘.

Use ‘mkpasswd‘ as mentioned in the cloud-init documentation to create the password for the user:

root@odc-f-13:/# mkpasswd --method=SHA-512
Password:
$6$8/.y8gwYT$dVmtT7jXdBrz0w1ku5mh6HOC.vngjsXpehyeEicJT4kIyhvUMV3p9VGUIDC42Z1mjXdfAaQkINcCfcFe5jEKX/

In the file /etc/cloud/cloud.cfg, find the section ‘default_user‘.  Change the following line from:

lock_passwd: True

to

lock_passwd: False
passwd: $6$8/.y8gwYT$dVmtT7jXdBrz0w1ku5mh6HOC.vngjsXpehyeEicJT4kIyhvUMV3p9VGUIDC42Z1mjXdfAaQkINcCfcFe5jEKX/

The value for the ‘passwd‘ option is the output from the mkpasswd command executed earlier.

Updating /etc/rc.local

The final update to the image is to add some bash code to the /etc/rc.local file.   The reason for this update is so the information for configuring Google Authenticator with the instance can be presented to the user through the output of ‘euca-get-console-output‘.  Add the following code to the /etc/rc.local file above the ‘exit 0‘ line:

if [ ! -f /home/ubuntu/.google_authenticator ]; then
 /bin/su ubuntu -c "google-authenticator -t -d -f -r 3 -R 30 -w 4" > /root/google-auth.txt
 echo "############################################################"
 echo "Google Authenticator Information:"
 echo "############################################################"
 cat /root/google-auth.txt
 echo "############################################################"
fi

Thats it!  Now we need to bundle, upload and register the image.

Bundle, Upload and Register the Image

Since we are using an HVM image, we don’t have to worry about the kernel and ramdisk.  We can just bundle, upload and register the image.  To do so, use the euca-install-image command.  Before we do that, we need to exit out of the chroot environment and unmount the image:

root@odc-f-13:/# exit
[root@odc-f-13 ~]# umount /mnt/ubuntu
[root@odc-f-13 ~]# kpartx -dv /dev/loop0
del devmap : loop0p1
[root@odc-f-13 ~]# losetup -d /dev/loop0

After unmounting the image, bundle, upload and register the image with the euca-install-image command:

[root@odc-f-13 ~]# euca-install-image -b ubuntu-trusty-server-google-auth-x86_64-hvm -i trusty-server-cloudimg-amd64-disk1.raw --virtualization-type hvm -n trusty-server-google-auth -r x86_64
/var/tmp/bundle-Q8yit1/trusty-server-cloudimg-amd64-disk1.raw.manifest.xml 100% |===============| 7.38 kB 3.13 kB/s Time: 0:00:02
IMAGE emi-FF439CBA

After the image is registered, launch the instance with a keypair that has been created using the ‘euca-create-keypair‘ command:

[root@odc-f-13 ~]# euca-run-instances -k account1-user01 -t m1.medium emi-FF439CBA
RESERVATION r-B79E6A59 408396244283 default
INSTANCE i-48D98090 emi-FF439CBA pending account1-user01 0 m1.medium 2014-07-21T20:23:10.285Z ViciousLiesAndDangerousRumors monitoring-disabled 0.0.0.0 0.0.0.0 instance-store hvm sg-A5133B59

Once the instance has reached the ‘running’ state, use ‘euca-get-console-ouptut’ to grab the Google Authenticator information:

[root@odc-f-13 ~]# euca-describe-instances i-48D98090
RESERVATION r-B79E6A59 408396244283 default
INSTANCE i-48D98090 emi-FF439CBA euca-10-104-6-237.bigboi.acme.eucalyptus-systems.com euca-172-18-238-157.bigboi.internal running account1-user01 0 m1.medium 2014-07-21T20:23:10.285Z ViciousLiesAndDangerousRumors monitoring-disabled 10.104.6.237 172.18.238.157 instance-store hvm sg-A5133B59
[root@odc-f-13 ~]# euca-get-console-output i-48D98090
.......
############################################################
Google Authenticator Information:
############################################################
https://www.google.com/chart?chs=200x200&chld=M|0&cht=qr&chl=otpauth://totp/ubuntu@euca-172-18-238-157%3Fsecret%3D2MGKGDZTFLVE5LCX
Your new secret key is: 2MGKGDZTFLVE5LCX
Your verification code is 275414
Your emergency scratch codes are:
 59078604
 17425999
 89676696
 65201554
 14740079
############################################################
.....

Now we are ready to test access to the instance.

Testing Access to the Instance

To test access to the instance, make sure the Google Authenticator application is installed on your smart phone/hand-held device.  Next, copy the URL seen in the output (e.g. https://www.google.com/chart?chs=200×200&chld=M|0&cht=qr&chl=otpauth://totp/ubuntu@euca-172-18-238-157%3Fsecret%3D2MGKGDZTFLVE5LCX) from ‘euca-get-console-output’, and past it into a browser:

OTPAUTH URL for Google Authenticator
OTPAUTH URL for Google Authenticator

Use the ‘Google Authenticator’ application on your smart phone/hand-held device, and scan the QR Code:

Google Authenticator Application
Google Authenticator Application

 

Google Authenticator Application - Set Up Account
Google Authenticator Application – Set Up Account

 

After selecting the ‘Set up account‘ option, select ‘Scan a barcode‘, hold your smartphone/hand-held device to the screen where your browser is showing the QR code, and scan:

Google Authenticator Application - Scan Barcode
Google Authenticator Application – Scan Barcode

 

After scanning the QR code, you should see the account get added, and the verification codes begin to populate for the account:

Verification Code For Instance
Verification Code For Instance

 

Finally, SSH into the instance using the following:

  • the private key of the keypair used when launching the instance with euca-run-instances
  • the password ‘ubuntu
  • the verification code displayed in Google Authenticator for the new account added

With the information above, the SSH authentication should look similar to the following:

[root@odc-f-13 ~]# ssh -i account1-user01/account1-user01.priv ubuntu@euca-10-104-6-237.bigboi.acme.eucalyptus-systems.com
The authenticity of host 'euca-10-104-6-237.bigboi.acme.eucalyptus-systems.com (10.104.6.237)' can't be established.
RSA key fingerprint is c9:37:18:66:e3:ee:66:d2:8a:ac:a4:21:a6:84:92:08.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'euca-10-104-6-237.bigboi.acme.eucalyptus-systems.com,10.104.6.237' (RSA) to the list of known hosts.
Authenticated with partial success.
Password:
Verification code:
Welcome to Ubuntu 14.04 LTS (GNU/Linux 3.13.0-32-generic x86_64)

* Documentation: https://help.ubuntu.com/

System information as of Mon Jul 21 13:23:48 UTC 2014

System load: 0.0 Memory usage: 5% Processes: 68
 Usage of /: 56.1% of 1.32GB Swap usage: 0% Users logged in: 0

Graph this data and manage this system at:
 https://landscape.canonical.com/

Get cloud support with Ubuntu Advantage Cloud Guest:
 http://www.ubuntu.com/business/services/cloud

0 packages can be updated.
0 updates are security updates.

The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.

Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.

ubuntu@euca-172-18-238-157:~$

Three-factor authentication has been successfully configured for the Ubuntu cloud image.  If cloud administrators would like to use different authentication for the instance user, I suggest investigating how to set up PAM LDAP authentication, where SSH public keys are stored in OpenLDAP.  In order to do this, the Ubuntu image  would have to be updated to work.  I would check out the ‘sss_ssh_authorizedkeys‘ command, and the pam-script module to potentially help get this working.

Enjoy!

Setting Up 3-Factor Authentication (Keypair, Password, Google Authenticator) for Eucalyptus Cloud Instances

Step-by-Step Deployment of Docker on Eucalyptus 3.4 for the Cloud Administrator

Docker

Eucalyptus Systems, Inc.

Docker has been in the news lately as one of the hot open-source project promoting linux containers. Some use cases for Docker include the following:

  • Automation of packaging and application deployment
  • Lightweight PaaS environments
  • Automated testing and continuous integration/deployment
  • Deploying and scaling web applications, databases and backend services

The focus of this blog entry is to show how to deploy Docker on Eucalyptus from a cloud administrator’s point-of-view – all in the cloud.  This is a step-by-step guide to create an Docker EMI from an existing Ubuntu Cloud Raring EMI using AWS’s documentation.  This entry will also show how to build euca2ools from source in the Ubuntu Cloud image.

Prerequisites

This entry assumes the following:

After confirming that the prerequisites are met, let’s get started.

Creating an EMI From an Existing EMI

As mentioned earlier, these steps will be based off of  AWS’s documentation on creating an instance store-backed AMI from an existing AMI.  In this example, here is an existing Ubuntu Raring instance thats running on Eucalyptus:

$ euca-describe-instances --region eucalyptus-admin@
RESERVATION r-3E423E33 961915002812 default
INSTANCE i-827E3E88 emi-06663A57 euca-10-104-7-12.eucalyptus.euca-hasp.eucalyptus-systems.com
euca-172-17-118-27.eucalyptus.internal running euca-admin 0 m1.medium 
2013-11-18T22:41:35.694Z LayinDaSmackDown eki-28F338EB eri-51253C0A 
monitoring-disabled 10.104.7.12 172.17.118.27 instance-store

This instance is using the following EMI, EKI and ERI:

$ euca-describe-images emi-06663A57 eki-28F338EB eri-51253C0A --region eucalyptus-admin@
IMAGE eki-28F338EB latest-raring-kernel/raring-server-cloudimg-amd64-vmlinuz-generic.manifest.xml 
441445882805 available public x86_64 kernel instance-store
IMAGE emi-06663A57 latest-raring/raring-server-cloudimg-amd64.img.manifest.xml 441445882805 
available public x86_64 machine eki-28F338EB eri-51253C0A instance-store paravirtualized
IMAGE eri-51253C0A latest-raring-kernel/raring-server-cloudimg-amd64-loader.manifest.xml 441445882805 
available public x86_64 ramdisk instance-store

To start, copy the zip file cloud administrator credentials obtained by the euca_conf command mentioned in the Eucalyptus 3.4 documentation to the running instance:

# scp -i euca-admin.priv admin.zip 
ubuntu@euca-10-104-7-12.eucalyptus.euca-hasp.eucalyptus-systems.com:/tmp/.

Next, install the following packages for the 3.8.0-33 kernel, and packages needed to build euca2ools:

ubuntu@euca-172-17-118-27:~$ sudo apt-get install python-setuptools git python-lxml unzip linux-headers-3.8.0-33-generic linux-image-extra-3.8.0-33-generic

Find the ephemeral storage using the instance metadata service, format, and mount the ephemeral to /mnt/image:

ubuntu@euca-172-17-118-27:~$ curl http://169.254.169.254/latest/meta-data/block-device-mapping/ephemeral
sda2
ubuntu@euca-172-17-118-27:~$ sudo mkdir /mnt/image
ubuntu@euca-172-17-118-27:~$ sudo mkfs.ext4 /dev/vda2
ubuntu@euca-172-17-118-27:~$ sudo mount /dev/vda2 /mnt/image

Download euca2ools from Github:

ubuntu@euca-172-17-118-27:~$ git clone https://github.com/eucalyptus/euca2ools.git

Install euca2ools:

ubuntu@euca-172-17-118-27:~$ cd euca2ools; sudo python setup.py install

Unzip the cloud administrator credentials in /tmp:

ubuntu@euca-172-17-118-27:~$ cd /tmp; unzip admin.zip

Change to the root user, and source the cloud administrator credentials:

ubuntu@euca-172-17-118-27:~$ sudo -s; source /tmp/eucarc

Bundle, upload and register the ramdisk and kernel under /boot:

root@euca-172-17-118-27:~# euca-bundle-image -i /boot/initrd.img-3.8.0-33-generic 
--ramdisk true -r x86_64
root@euca-172-17-118-27:~# euca-upload-bundle -b ubuntu-raring-docker-ramdisk 
-m /var/tmp/bundle-SQrAuT/initrd.img-3.8.0-33-generic.manifest.xml
root@euca-172-17-118-27:~# euca-register -n ubuntu-raring-docker-ramdisk 
ubuntu-raring-docker-ramdisk/initrd.img-3.8.0-33-generic.manifest.xml 
IMAGE eri-6BF033EE
root@euca-172-17-118-27:~# euca-bundle-image -i /boot/vmlinuz-3.8.0-33-generic 
--kernel true -r x86_64
root@euca-172-17-118-27:~# euca-upload-bundle -b ubuntu-raring-docker-kernel 
-m /var/tmp/bundle-31Lnxy/vmlinuz-3.8.0-33-generic.manifest.xml
root@euca-172-17-118-27:~# euca-register -n ubuntu-raring-docker-kernel 
ubuntu-raring-docker-kernel/vmlinuz-3.8.0-33-generic.manifest.xml
IMAGE eki-17093995

Use euca-bundle-vol to bundle the root filesystem. Make sure to exclude /tmp, /mnt/image, and /home/ubuntu. Additionally, make sure and set the size of the image to be 5 GB:

root@euca-172-17-118-27:~# euca-bundle-vol -p ubuntu-raring-docker 
-s 5120 -e /tmp,/root,/mnt/image,/home/ubuntu -d /mnt/image 
--kernel eki-17093995 --ramdisk eri-6BF033EE -r x86_64

Next, upload and register the root filesystem:

root@euca-172-17-118-27:~# euca-upload-bundle -b ubuntu-raring-docker-rootfs 
-m /mnt/image/ubuntu-raring-docker.manifest.xml
root@euca-172-17-118-27:~# euca-register -n ubuntu-raring-docker-rootfs 
ubuntu-raring-docker-rootfs/ubuntu-raring-docker.manifest.xml
IMAGE emi-26403979

We have the new EMI, EKI and ERI for the Docker instance.  Lastly, set the image permissions so that all users on the cloud can use the EMI, EKI and ERI:

root@euca-172-17-118-27:~# euca-modify-image-attribute -l -a all emi-26403979
root@euca-172-17-118-27:~# euca-modify-image-attribute -l -a all eki-17093995
root@euca-172-17-118-27:~# euca-modify-image-attribute -l -a all eri-6BF033EE

Now its time to launch the Docker EMI.

Running the Docker Instance with Cloud-Init

Before launching the EMI, the cloud-init configuration file needs to be created.  This file will be responsible for configuring the instance repositories, downloading and installing Docker.  With your favorite command-line editor, create a file called cloud-init-docker.config, with the following content:

#cloud-config
apt_update: true
apt_upgrade: true
disable_root: true
packages:
 - less
cloud_config_modules:
 - ssh
 - [ apt-update-upgrade, always ]
 - updates-check
 - runcmd
runcmd:
 - [ sh, -xc, "INST_HOSTNAME=`/bin/hostname`; META_IP=`curl -s http://169.254.169.254/latest/meta-data/local-ipv4`; echo ${META_IP} ${INST_HOSTNAME} >> /etc/hosts" ]
 - [ locale-gen, en_US.UTF-8 ]
 - [ sh, -xc, "wget -qO docker-io.gpg https://get.docker.io/gpg" ]
 - [ apt-key, add, docker-io.gpg ]
 - [ sh, -xc, "echo 'deb http://get.docker.io/ubuntu docker main' > /etc/apt/sources.list.d/docker.list" ]
 - [ apt-get, update ]
 - [ apt-get, install, -y, --force-yes, lxc-docker ]
 - [ modprobe, -q, aufs ]

Now, use euca-run-instances to launch the instance:

root@euca-172-17-118-27:~# euca-run-instances -k euca-admin emi-351237D1 
-t m1.medium --user-data-file cloud-init-docker.config

After launching the instance, leave the current instance to get back to end client.

root@euca-172-17-118-27:~# exit
exit
ubuntu@euca-172-17-118-27:~$ exit
logout
Connection to 10.104.7.12 closed.

Once the instance reaches running state, ssh into the instance using the keypair specified (which in this case will be euca-admin.priv), and execute the following Docker command to run an interactive shell session inside a minimal Ubuntu container:

$ euca-describe-instances --region eucalyptus-admin@
RESERVATION r-A1613D7F 961915002812 default
INSTANCE i-AFDB3D4C emi-26403979 euca-10-104-7-13.eucalyptus.euca-hasp.eucalyptus-systems.com 
euca-172-17-118-16.eucalyptus.internal running euca-admin 0 m1.medium 
2013-11-19T01:21:10.880Z LayinDaSmackDown eki-17093995 eri-6BF033EE monitoring-disabled 
10.104.7.13 172.17.118.16 instance-store
# ssh -i euca-admin.priv ubuntu@euca-10-104-7-13.eucalyptus.euca-hasp.eucalyptus-systems.com
Welcome to Ubuntu 13.04 (GNU/Linux 3.8.0-33-generic x86_64)
* Documentation: https://help.ubuntu.com/
System information as of Thu Nov 14 23:18:38 UTC 2013
System load: 0.0 Users logged in: 0
 Usage of /: 21.6% of 4.89GB IP address for eth0: 172.17.184.76
 Memory usage: 4% IP address for lxcbr0: 10.0.3.1
 Swap usage: 0% IP address for docker0: 10.1.42.1
 Processes: 83
Graph this data and manage this system at https://landscape.canonical.com/
Get cloud support with Ubuntu Advantage Cloud Guest:
 http://www.ubuntu.com/business/services/cloud
Use Juju to deploy your cloud instances and workloads:
 https://juju.ubuntu.com/#cloud-raring
New release '13.10' available.
Run 'do-release-upgrade' to upgrade to it.
Last login: Thu Nov 14 23:08:09 2013 from 10.104.10.6
ubuntu@euca-172-17-184-76:~$ sudo docker run -i -t ubuntu /bin/bash
Unable to find image 'ubuntu' (tag: latest) locally
Pulling repository ubuntu
8dbd9e392a96: Download complete
b750fe79269d: Download complete
27cf78414709: Download complete
root@041d5ddcd6b9:/# (Ctrl-p Ctrl-q to exit out of shell)
ubuntu@euca-172-17-184-76:~$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
041d5ddcd6b9 ubuntu:12.04 /bin/bash 27 seconds ago Up 26 seconds pink_frog

Thats it!  For more information regarding Docker, please refer to the latest Docker documentation.

Enjoy!

Step-by-Step Deployment of Docker on Eucalyptus 3.4 for the Cloud Administrator

Jenkins, Stackato, Cloud-Init and Eucalyptus == Potent Combination for an On-Premise Continuous Integration Environment

The Ingredients

Jenkins

An extendable open source continuous integration server.

Stackato

The Enterprise Private PaaS that makes it easy to deploy, manage, and monitor applications on any cloud.

Cloud-init

The Ubuntu package that handles early initialization of a cloud instance. It is installed in the Ubuntu Cloud Images and also in the official Ubuntu images available on EC2.

Eucalyptus

Allows you to build production-ready, AWS-compatible private and hybrid clouds by leveraging your existing virtualized infrastructure to create on-demand cloud resource pools.

What happens when you combine all three of these tools?  A potent combination for continuous integration on a easy-to-configure PaaS and an on-premise, AWS-compatible IaaS.  With this combination, developers can take advantage of easy configuration that Stackato brings to the table, running on top of Eucalyptus – bringing an AWS-like cloud environment into your datacenter.

This blog entry will discuss the steps that I took to get Jenkins installed on a Stackato instance store-backed instance running on Eucalyptus.  But before I get started, I would like to thank the folks from ActiveState for all their guidance.  Their support staff is really top notch, and very helpful.  Check them out in #stackato on freenode.net.  They can also be checked out on Twitter at @ActiveState. Now on to the dirty work…..

The Recipe for Success

The Stackato Microcloud Image and Cloud-Init

To begin, the following is needed:

After downloading the Stackato VM for KVM and unzipping the file, we will need to pull out the root file system, the kernel and ramdisk.  These will be uploaded, bundled and registered as the EMI, EKI, and ERI.  To extract the root filesystem, do the following:

  1. Use parted to locate the root filesystem as follows:    
    # parted stackato-img-kvm-v2.6.6.img
    GNU Parted 2.1
    Using /root/images/Stackato-VM/stackato-img-kvm-v2.6.6.img
    Welcome to GNU Parted! Type 'help' to view a list of commands.
    (parted) U
    Unit? [compact]? b
    (parted) p
    Model: (file)
    Disk /root/images/Stackato-VM/stackato-img-kvm-v2.6.6.img: 10737418240B
    Sector size (logical/physical): 512B/512B
    Partition Table: msdos
    
    Number Start End Size Type File system Flags
    1 1048576B 200278015B 199229440B primary ext3 boot
    3 200278016B 1237319679B 1037041664B primary linux-swap(v1)
    2 1237319680B 10736369663B 9499049984B primary ext4
    
    (parted) quit
  2. In this example, the root filesystem is partition 2.  The value for “Start” and “Size” will need to be used.  Next, run dd to extract the root filesystem:
    dd if=stackato-img-kvm-v2.6.6.img of=stackato-rootfs.img
     bs=1 skip=1237319680 count=9499049984
    
  3. Once it has completed, mount  stackato-rootfs.img to the loopback device:
    mount -o loop stackato-rootfs.img /mnt/
  4. Copy out initrd.img-3.2.0-27-virtual  and vmlinuz-3.2.0-27-virtual from /mnt/boot.
  5. In /mnt/etc/fstab, replace the UUID entry with LABEL.  The LABEL will look simliar to the following:  
    LABEL=cloudimg-rootfs    /               ext4   defaults     1       1
  6. Chroot to /mnt – there may be a need to do a mount -o bind for sys, dev, and proc.
  7. Run “dpkg-reconfigure cloud-init”, and make sure that the EC2 Data Source is selected.
  8. Unmount stackato-rootfs.img (if sys, dev, and proc were mounted, unmount them before unmounting stackato-rootfs.img).  After it has been unmounted, run tune2fs to change the label of the image:
    tune2fs -L cloudimg-rootfs stackato-rootfs.img

After following these steps, the following should be available:

  • initrd.img-3.2.0-27-virtual – to be bundled, uploaded and registered as the ERI
  • vmlinuz-3.2.0-27-virtual – to be bundled, uploaded and registered as the EKI
  • stackato-rootfs.img – to be bundled, uploaded and registered as the EMI

Go through the steps of bundling, uploading and registering the ERI, EKI, and EMI.  For additional information, please refer to the Add an Image section of the Eucalyptus 3.2 User Guide.

Launching the Stackato Image

Now its time to launch the Stackato image on Eucalyptus.  Since cloud-init has the enabled EC2 data source now, when the image is launched, the instance will grab ssh keys, and mount the ephemeral storage.  Also, additional configuration can be passed using the user-data file option.  More information regarding this can be found on Stackato’s documentation in reference to using cloud-init.  Key thing to remember here is that the minimum RAM requirement for the Stackato image is 2 gigs.  Make sure the VM type used for launching the Stackato image has at least 2 gigs of RAM or more.  In this example, the image ID is emi-DAB23A8A.  The ramdisk and kernel are registered as eri-9B453C09 and eki-ADF640B0.   The VM type c1.xlarge is used, which has 4 CPU,  4096 MB of RAM,  and    50 Gigs of disk space.

euca-run-instances -k admin emi-DAB23A8A -t c1.xlarge 
--kernel eki-ADF640B0 --ramdisk eri-9B453C09

Use euca-describe-instances to check to see when the instance reaches a running state:

euca-describe-instances i-100444EF
RESERVATION r-CC69438B 345590850920 default
INSTANCE i-100444EF emi-DAB23A8A euca-192-168-55-104.wu-tang.euca-hasp.eucalyptus-systems.com 
euca-10-106-101-17.wu-tang.internal running admin 0 c1.xlarge 2013-02-23T00:34:07.436Z enter-the-wu eki-ADF640B0 
eri-9B453C09 monitoring-disabled 192.168.55.104 10.106.101.17 instance-store

The key thing for running a Stackato instance is setting up the correct DNS entries.  For more information regarding setting up DNS with regards to a Stackato instance, please read the Detail Configuration section on DNS in the Stackato online documentation.  For this example, instead of using an external DNS service using a tool like nsupdate, to configure the A record and CNAME records, we will use xip.io. xip.io is a magic domain name that provides wildcard DNS for any IP address.  Next, its time to configure the Stackato instance.

Configuration of the Stackato Instance

To configure the Stackato instance, do the following:

  1. SSH into the instance.  

    ssh -i creds/admin.priv 
    stackato@euca-192-168-55-104.wu-tang.euca-hasp.eucalyptus-systems.com
  2. Make note of the ip address associated with eth0 and the netmask using ifconfig.  Also note the gateway IP by using the route command.
  3. Run “kato op static_ip” to configure the static IP address for eth0.  Make sure and add 127.0.0.1 as the first entry as part of the nameservers, and add “containers.” as the first entry under the search domains.
  4. Run “kato rename public DNS name “, where public DNS name includes the public IP of the instance, using xip.io (e.g. 192.168.55.104.xip.io)
  5. Run “kato disable mdns”, then run “sudo reboot” to reboot the instance.
  6. Once the instance has come back up, ssh into the instance, and run the following command  “kato setup core api.public DNS name” where public DNS name is the same value used for the “kato rename” step (e.g.  192.168.55.104.xip.io).
  7. Next, edit /etc/resolv.conf and make sure that the value for the search option is “containers.”, and the first entry for the nameservers is 127.0.0.1.
  8. Finally, run “kato enable –all-but mdns”

Thats it!  Now go to the public DNS name that was used in your favorite browser.  For this example, 192.168.55.104.xip.io was used.  The following landing page should look similar to what you see here in the Stackato documentation regarding accessing the instance through the management console.

Setting Up Jenkins

After setting up the admin account, navigate to the “App Store” on the lefthand menu.  Once selected, navigate to find the Jenkins application:

 

Jenkins Application
Jenkins Application

 

After selecting to install Jenkins, select “Yes” to install.  After the installation takes place,  select “Applications” in the left hand menu.  From there, select the Jenkins application, and select “Start” (its the green arrow to the right of the application window).   Once it has started, you will see the following:

Jenkins Running in Stackato
Jenkins Running in Stackato

 

Now Jenkins is ready to be used.

If anyone wants to test this out on Eucalyptus but doesn’t have access to their own Eucalyptus cloud, fear not, the Eucalyptus Community Cloud has the Stackato image available.  After applying to get access to the Community Cloud, follow the steps above.  The image for Stackato is as follows:

IMAGE emi-859B3D5C stackato_v2.6.6/stackato-cloudinit.manifest.xml 150820662310 available public x86_64 machine eki-6FBE3D2D eri-67463B77 instance-store

And as always, this image and steps can be used on AWS EC2 as well. 🙂

Let me know if there are any questions.  Feedback is always welcome.  Enjoy!

Jenkins, Stackato, Cloud-Init and Eucalyptus == Potent Combination for an On-Premise Continuous Integration Environment

Another Great Example of AWS Fidelity – Neo4j, Cloud-Init and Eucalyptus

I recently ran across a blog entry entitled Neo4j 1.9.M01 – Self-managed HA.  I found the concept of graph databases storing data really interesting and reached out to the guys at Neo4j to get some insight on how to deploy their HA solution on Eucalyptus.   Amongst the resources that they provided,  they shared this little gem – how to deploy Neo4j on EC2.  In order to run first, you need to know how to walk – so before going down the path of standing up HA Neo4j, I decided to be influenced by the DIY on EC2 article provided by Neo4j and deploy Neo4j on Eucalyptus  – with a little help from Cloud-Init.  The follow-up blog will show how to use the same setup, and deploy an HA Neo4j environment.

The Setup

Eucalyptus

The Eucalyptus cloud I used is configured using Eucalyptus High-Availability.  Its running on CentOS 6.3, running KVM.  Its also running in Managed networking mode, so that we can take advantage of network isolation of the VMs, and the use of security groups  – interacting very much in the same way as its done in the security groups provided in AWS EC2.

Ubuntu Cloud Image – 12.04 LTS Precise Pangolin

The image that we will use is the Ubuntu 12.04 LTS Cloud image.  The reasons for using this image is as follows:

  • Ubuntu cloud images come pre-packaged with cloud-init, which helps with bootstrapping the instance.
  • I wanted to have the solution work on AWS EC2 and Eucalyptus; since Ubuntu cloud images work on both, its a great choice.

Registering the Ubuntu Cloud Image with Eucalyptus

In order for us to get started, we need to get the Ubuntu Cloud image into Eucalyptus so that we can use it for our instance.  To upload, bundle and register the Ubuntu Cloud image, ramdisk and kernel, do the following:

  1. Download current version of  Ubuntu Precise Server AMD64 from the Ubuntu Cloud Image – Precise page, then unpack (ungzip, unarchive) the tar-gzipped file.  

    $ tar -zxvf precise-server-cloudimg-amd64.tar.gz
    x precise-server-cloudimg-amd64.img
    x precise-server-cloudimg-amd64-vmlinuz-virtual
    x precise-server-cloudimg-amd64-loader
    x precise-server-cloudimg-amd64-floppy
    x README.files

  2. Make sure to download and source your Eucalyptus credentials.
  3. We need to bundle, upload, and register precise-server-cloudimg-amd64-loader (ERI), precise-server-cloudimg-amd64-vmlinuz-virtual (EKI), and precise-server-cloudimg-amd64.img (EMI).  For more information regarding this, please refer to the “Image Overview” section of the Eucalyptus 3.1 User Guide.  

    $ euca-bundle-image -i precise-server-cloudimg-amd64-loader --ramdisk true
    $ euca-upload-bundle -b latest-ubuntu-precise -m /tmp/precise-server-cloudimg-amd64-loader.manifest.xml
    $ euca-register -a x86_64 latest-ubuntu-precise/precise-server-cloudimg-amd64-loader.manifest.xml
    $ euca-bundle-image -i precise-server-cloudimg-amd64-vmlinuz-virtual --kernel true
    $ euca-upload-bundle -b latest-ubuntu-precise -m /tmp/precise-server-cloudimg-amd64-vmlinuz-virtual.manifest.xml
    $ euca-register -a x86_64 latest-ubuntu-precise/precise-server-cloudimg-amd64-vmlinuz-virtual.manifest.xml
    $ euca-bundle-image -i precise-server-cloudimg-amd64.img
    $ euca-upload-bundle -b latest-ubuntu-precise -m /tmp/precise-server-cloudimg-amd64.img.manifest.xml
    $ euca-register -a x86_64 latest-ubuntu-precise/precise-server-cloudimg-amd64.img.manifest.xml

After bundling, uploading and registering the ramdisk, kernel and image, the latest-ubuntu-precise bucket in Walrus should have the following images:

$ euca-describe-images | grep latest-ubuntu-precise
IMAGE eki-0F3937E9 latest-ubuntu-precise/precise-server-cloudimg-amd64-vmlinuz-virtual.manifest.xml 345590850920 available public x86_64 kernel instance-store

IMAGE emi-C1613E67 latest-ubuntu-precise/precise-server-cloudimg-amd64.img.manifest.xml 345590850920 available public x86_64 machine instance-store

IMAGE eri-0BE53BFD latest-ubuntu-precise/precise-server-cloudimg-amd64-loader.manifest.xml 345590850920 available public x86_64 ramdisk instance-store

Cloud-init Config File

Now that we have the image ready to go, we need to create a cloud-init config file to pass in using the –user-data-file option that is part of euca-run-instances.  For more examples of different cloud-init files, please refer to the cloud-init-dev/cloud-init repository on bazaar.launchpad.net.  Below is the cloud-init.config file I created for bootstrapping the instance with an install of Neo4j, using ephemeral disk for the application storage, and installing some other packages (i.e. latest euca2ools, mlocate, less, etc.). The script can be also accessed from github as well – under the eucalptus/recipes repo.

#cloud-config
apt_update: true
apt_upgrade: true
disable_root: true
package_reboot_if_required: true
packages:
 - less
 - bind9utils
 - dnsutils
 - mlocate
cloud_config_modules:
 - ssh
 - [ apt-update-upgrade, always ]
 - updates-check
 - runcmd
runcmd:
 - [ sh, -xc, "if [ -b /dev/sda2 ]; then tune2fs -L ephemeral0 /dev/sda2;elif [ -b /dev/vda2 ]; then tune2fs -L ephemeral0 /dev/vda2;elif [ -b /dev/xvda2 ]; then tune2fs -L ephemeral0 /dev/xvda2;fi" ]
 - [ sh, -xc, "mkdir -p /var/lib/neo4j" ]
 - [ sh, -xc, "mount LABEL=ephemeral0 /var/lib/neo4j" ]
 - [ sh, -xc, "if [ -z `ls /var/lib/neo4j/*` ]; then sed --in-place '$ iMETA_HOSTNAME=`curl -s http://169.254.169.254/latest/meta-data/local-hostname`\\nMETA_IP=`curl -s http://169.254.169.254/latest/meta-data/local-ipv4`\\necho ${META_IP}   ${META_HOSTNAME} >> /etc/hosts; hostname ${META_HOSTNAME}; sysctl -w kernel.hostname=${META_HOSTNAME}\\nif [ -d /var/lib/neo4j/ ]; then mount LABEL=ephemeral0 /var/lib/neo4j; service neo4j-service restart; fi' /etc/rc.local; fi" ] 
 - [ sh, -xc, "META_HOSTNAME=`curl -s http://169.254.169.254/latest/meta-data/local-hostname`; META_IP=`curl -s http://169.254.169.254/latest/meta-data/local-ipv4`; echo ${META_IP}   ${META_HOSTNAME} >> /etc/hosts" ]
 - [ sh, -xc, "META_HOSTNAME=`curl -s http://169.254.169.254/latest/meta-data/local-hostname`; hostname ${META_HOSTNAME}; sysctl -w kernel.hostname=${META_HOSTNAME}" ]
 - [ sh, -xc, "wget -O c1240596-eucalyptus-release-key.pub http://www.eucalyptus.com/sites/all/files/c1240596-eucalyptus-release-key.pub" ]
 - [ apt-key, add, c1240596-eucalyptus-release-key.pub ]
 - [ sh, -xc, "echo 'deb http://downloads.eucalyptus.com/software/euca2ools/2.1/ubuntu precise main' > /etc/apt/sources.list.d/euca2ools.list" ]
 - [ sh, -xc, "echo 'deb http://debian.neo4j.org/repo stable/' > /etc/apt/sources.list.d/neo4j.list" ]
 - [ apt-get, update ]
 - [ apt-get, install, -y, --force-yes, euca2ools ]
 - [ apt-get, install, -y, --force-yes, neo4j ]
 - [ sh, -xc, "sed --in-place 's/#org.neo4j.server.webserver.address=0.0.0.0/org.neo4j.server.webserver.address=0.0.0.0/' /etc/neo4j/neo4j-server.properties" ]
 - [ sh, -xc, "service neo4j-service restart" ]
 - [ sh, -xc, "export LANGUAGE=en_US.UTF-8" ]
 - [ sh, -xc, "export LANG=en_US.UTF-8" ]
 - [ sh, -xc, "export LC_ALL=en_US.UTF-8" ]
 - [ locale-gen, en_US.UTF-8 ]
 - [ dpkg-reconfigure, locales ]
 - [ updatedb ]
mounts:
 - [ ephemeral0, /var/lib/neo4j, auto, "defaults,noexec" ]

Now, we are ready to launch the instance.

Putting It All Together

Before launching the instance, we need to set up our keypair and security group that we will use with the instance.

  1. To create a keypair, run euca-create-keypair.  *NOTE* Make sure you change the permissions of the keypair to 0600 after its been created.

    euca-create-keypair  neo4j-user > neo4j-user.priv; chmod 0600 neo4j-user.priv

  2. Next, we need to create a security group for our instance.  To create a security group, use euca-create-group.  To open any ports you  need for the application, use euca-authorize.  The ports we will open up for the Neo4j application are SSH (22), ICMP, HTTP( 7474), and HTTPS (7473).
    • Create security group:

      # euca-create-group neo4j-test -d "Security for Neo4j Instances"

    • Authorize SSH:

      # euca-authorize -P tcp -p 22 -s 0.0.0.0/0 neo4j-test

    • Authorize HTTP:

      # euca-authorize -P tcp -p 7474 -s 0.0.0.0/0 neo4j-test

    • Authorize HTTPS:

      # euca-authorize -P tcp -p 7473 -s 0.0.0.0/0 neo4j-test

    • Authorize ICMP:

      # euca-authorize -P icmp -t -1:-1 -s 0.0.0.0/0 neo4j-test

  3. Finally, we use euca-run-instances to launch the Ubuntu Precise image, and use cloud-init to install Neo4j:

    # euca-run-instances -k neo4j-user --user-data-file cloud-init-neo4j.config emi-C1613E67 --kernel eki-0F3937E9 --ramdisk eri-0BE53BFD --group neo4j-test

To check the status of the instance, use euca-describe-instances.

# euca-describe-instances i-A9EF448C
RESERVATION r-ED8E4699 345590850920 neo4j-test
INSTANCE i-A9EF448C emi-C1613E67 euca-192-168-55-104.wu-tang.euca-hasp.eucalyptus-systems.com 
euca-10-106-69-154.wu-tang.internal running admin 0 m1.small 2012-12-04T03:13:13.869Z 
enter-the-wu eki-0F3937E9 eri-0BE53BFD monitoring-disable 
euca-192-168-55-104.wu-tang.euca-hasp.eucalyptus-systems.com euca-10-106-69-154.wu-tang.internal instance-store

Because I added in the cloud-init config file to do an “apt-get upgrade”, it takes about 5 to 7 minutes until the instance is fully configured and Neo4j is running.  Once you have it running, go to https://<ip-address of instance>:7473.  It will direct you to the web administration page for monitoring and management of the Neo4j instance.  In this example, the URL will be https://euca-192-168-55-104.wu-tang.euca-hasp.eucalyptus-systems.com:7473

Neo4j Monitoring and Management Tool
Neo4j Monitoring and Management Tool

Thats it!  The cool thing about this too, is that you can find an Ubuntu Precise AMI on AWS EC2, use the same cloud-init script, use euca2ools, and follow these instructions to get the same deployment on AWS EC2.

As mentioned before, the follow-up blog will be how to deploy the HA solution of Neo4j on Eucalyptus. Enjoy!

Another Great Example of AWS Fidelity – Neo4j, Cloud-Init and Eucalyptus