Growing and Securing Instance Root Filesystems With Overlayroot on Eucalyptus

Background

I frequently check out Dustin Kirkland’s blog to get ideas about how to secure instances running on various cloud infrastructures.  Recently, I stumbled across one of his blog entries, where he discussed a package called ‘overlayroot‘, which is part of the cloud-initramfs-tools package for Ubuntu.  The really interesting feature I liked about this package is the ability to encrypt – using dmcrypt –  the root filesystem of the instance.  What this means that if the Node Controller (NC) that hosts the instance happens to become compromised, the information on the device associated with the instance can’t be accessed.  In addition, overlayroot allows the root filesystem of the instance to be ‘laid’ on top of another device that is bigger in size.  The advantage here is that there isn’t a need to re-create an image with a larger root filesystem.

Prerequisites

The key prerequisites for this blog were mentioned in my previous blog, which discusses how to bundle, upload and register a CoreOS EMI.  In addition, to these prerequisites, the following EC2 actions are needed for the Eucalyptus IAM policy:

Overlayroot is available with any Ubuntu Cloud image from 12.10 (Quantal Quetzal) to the most recent at the time of this blog, 14.10 (Utopic Unicorn).  For this blog, Ubuntu 14.04 LTS (Trusty Tahr) will be used.

Setting Up the EMI

As mentioned in the overlayroot blog entry, the configuration of overlayroot is located in /etc/overlayroot.conf.  This can be configured before this Ubuntu Cloud image is bundled, uploaded and registered as an EMI, or after.  For brevity, overlayroot will be configured after the Ubuntu Cloud image has been bundled, uploaded and registered.

To get started, download the Ubuntu Cloud image:

# wget http://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img

Next, convert the image from qcow2 to raw:

# qemu-img convert -O raw trusty-server-cloudimg-amd64-disk1.img trusty-server-cloudimg-amd64-disk1.raw

Bundle, upload and register the image as an instance store-backed HVM EMI:

# euca-bundle-and-upload-image -i trusty-server-cloudimg-amd64-disk1.raw -b trusty-server-hvm -r x86_64
# euca-register -n trusty-server-hvm trusty-server-hvm/trusty-server-cloudimg-amd64-disk1.raw.manifest.xml --virtualization-type hvm 
IMAGE emi-348861E4

After creating a keypair using euca-create-keypair, and authorizing SSH access using euca-authorize, launch an instance from the EMI:

# euca-run-instances -k account1-user01 -t m1.medium emi-348861E4
RESERVATION r-BE317660 408396244283 default
INSTANCE i-41DA6A90 emi-348861E4 pending account1-user01 0 m1.medium 2014-06-15T22:47:03.552Z ViciousLiesAndDangerousRumors monitoring-disabled 0.0.0.0 0.0.0.0 instance-store hvm sg-A5133B59

Once the instance has reached a ‘running’ state, SSH into the instance:

# euca-describe-instances i-41DA6A90 
RESERVATION r-BE317660 408396244283 default
INSTANCE i-41DA6A90 emi-348861E4 euca-10-104-6-235.bigboi.acme.eucalyptus-systems.com euca-172-18-238-170.bigboi.internal running account1-user01 0 m1.medium 2014-06-15T22:47:03.552Z ViciousLiesAndDangerousRumors monitoring-disabled 10.104.6.235 172.18.238.170 instance-store hvm sg-A5133B59
# ssh -i account1-user01/account1-user01.priv ubuntu@euca-10-104-6-235.bigboi.acme.eucalyptus-systems.com

Encrypted Root Filesystem

Once inside the instance, since the EMI is an instance store-backed HVM EMI, the ephemeral disk is /dev/vdb:

ubuntu@euca-172-18-238-170:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 2.2G 0 disk
└─vda1 253:1 0 2.2G 0 part /
vdb 253:16 0 7.8G 0 disk

Edit the ‘overlayroot’ option in the /etc/overlayroot.conf to set /dev/vdb to used to encrypt the root filesystem:

ubuntu@euca-172-18-238-170:~$ sudo vi /etc/overlayroot.conf
......
overlayroot=”crypt:dev=/dev/vdb”

After editing /etc/overlayroot.conf, reboot the instance:

ubuntu@euca-172-18-238-170:~$ sudo reboot

After the instance has been rebooted, SSH back into the instance and observe the new layout of the filesystem:

ubuntu@euca-172-18-238-170:~$ mount
overlayroot on / type overlayfs (rw,lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/cgroup type tmpfs (rw)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755)
/dev/vda1 on /media/root-ro type ext4 (ro)
/dev/mapper/secure on /media/root-rw type ext4 (rw,relatime,data=ordered)
none on /sys/fs/pstore type pstore (rw)
systemd on /sys/fs/cgroup/systemd type cgroup (rw,noexec,nosuid,nodev,none,name=systemd)

To verify the encrypted filesystem, use cryptsetup:

ubuntu@euca-172-18-238-170:~$ sudo cryptsetup luksDump /dev/vdb
LUKS header information for /dev/vdb
Version: 1
Cipher name: aes
Cipher mode: xts-plain64
Hash spec: sha1
Payload offset: 4096
MK bits: 256
MK digest: 77 70 97 81 9e d6 56 4b c5 79 60 92 23 02 18 80 9c eb 40 c8
MK salt: ed d1 69 1e d7 49 a6 a6 91 fb 0f 44 3a a2 0b b2
 2a 56 fb 82 77 3f 70 9c 70 ff c2 15 37 09 10 2c
MK iterations: 33250
UUID: 164026af-f6fa-4dd4-8042-5cd24b8c00c9
Key Slot 0: ENABLED
 Iterations: 132641
 Salt: 5e 7f 03 33 d9 af e8 a9 37 fb 5c 4e 10 db c5 38
 f7 9f 49 12 d8 c4 43 8c cb 79 4a 25 da 04 c9 73
 Key material offset: 8
 AF stripes: 4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

Expand and Encrypt the Root Filesystem

If the cloud user wants to increase the size of the root filesystem and encrypt it, but use a larger size than available for ephemeral storage, the user can create a volume, attach it to the instance, then configure overlayroot to use that device.  For example:

# euca-create-volume -s 20 -z ViciousLiesAndDangerousRumors
# euca-describe-volumes vol-450936D7
VOLUME vol-450936D7 20 ViciousLiesAndDangerousRumors available 2014-06-16T19:25:36.936Z standard
# euca-attach-volume -i i-41DA6A90 -d /dev/sdf vol-450936D7
ATTACHMENT vol-450936D7 i-41DA6A90 /dev/sdf attaching 2014-06-16T19:30:29.605Z
# euca-describe-instances i-41DA6A90
RESERVATION r-BE317660 408396244283 default
INSTANCE i-41DA6A90 emi-348861E4 euca-10-104-6-235.bigboi.acme.eucalyptus-systems.com euca-172-18-238-170.bigboi.internal running account1-user01 0 m1.medium 2014-06-15T22:47:03.552Z ViciousLiesAndDangerousRumors monitoring-disabled 10.104.6.235 172.18.238.170 instance-store hvm sg-A5133B59
BLOCKDEVICE /dev/sdf vol-450936D7 2014-06-16T19:30:29.619Z false

Follow the steps above regarding editing the /etc/overlayroot.conf, except use /dev/vdc instead of /dev/vdb:

# ssh -i account1-user01/account1-user01.priv ubuntu@euca-10-104-6-235.bigboi.acme.eucalyptus-systems.com
ubuntu@euca-172-18-238-170:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 2.2G 0 disk
└─vda1 253:1 0 2.2G 0 part /
vdb 253:16 0 7.8G 0 disk
vdc 253:32 0 20G 0 disk
ubuntu@euca-172-18-238-170:~$ sudo vi /etc/overlayroot.conf
.....
overlayroot=”crypt:dev=/dev/vdc”

Reboot the instance.  After the reboot completes, SSH back into the instance and noticed the root filesystem increased in size, and encrypted:

ubuntu@euca-172-18-238-170:~$ sudo reboot
# ssh -i account1-user01/account1-user01.priv ubuntu@euca-10-104-6-235.bigboi.acme.eucalyptus-systems.com
ubuntu@euca-172-18-238-170:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 2.2G 0 disk
└─vda1 253:1 0 2.2G 0 part /media/root-ro
vdb 253:16 0 7.8G 0 disk
vdc 253:32 0 20G 0 disk
└─secure (dm-0) 252:0 0 20G 0 crypt /media/root-rw
ubuntu@euca-172-18-238-170:~$ df -ah
Filesystem Size Used Avail Use% Mounted on
overlayroot 20G 46M 19G 1% /
proc 0 0 0 - /proc
sysfs 0 0 0 - /sys
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 0 0 0 - /sys/fs/fuse/connections
none 0 0 0 - /sys/kernel/debug
none 0 0 0 - /sys/kernel/security
udev 493M 8.0K 493M 1% /dev
devpts 0 0 0 - /dev/pts
tmpfs 100M 328K 100M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 498M 0 498M 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/vda1 2.2G 754M 1.3G 37% /media/root-ro
/dev/mapper/secure 20G 46M 19G 1% /media/root-rw
none 0 0 0 - /sys/fs/pstore
systemd 0 0 0 - /sys/fs/cgroup/systemd

Bonus – Configuring Overlayroot Before Bundle, Uploading and Registering the EMI

As mention before,  overlayroot can be configured before the image has been bundled, uploaded and registered as an instance store-backed HVM EMI.  After downloading the Ubuntu Trusty cloud image, use losetup and kpartx to aid in mounting the image in order to configure overlayroot:

# qemu-img convert -O raw trusty-server-cloudimg-amd64-disk1.img trusty-server-cloudimg-amd64-disk1.raw
# losetup /dev/loop0 trusty-server-cloudimg-amd64-disk1.raw
# kpartx -av /dev/loop0
add map loop0p1 (253:2): 0 4192256 linear /dev/loop0 2048
# mkdir /mnt/ubuntu
# mount /dev/mapper/loop0p1 /mnt/ubuntu
# chroot /mnt/ubuntu
root@odc-f-13:/# vi /etc/overlayroot.conf
(added the following)
overlayroot=”crypt:dev=/dev/vdb”
root@odc-f-13:/# exit
exit
# umount /mnt/ubuntu
# kpartx -dv /dev/loop0
# losetup -d /dev/loop0

After unmounting the image, just bundle, upload and register the image as an instance store-backed HVM EMI.  When an instance is launched from the EMI, the root filesystem will automatically be encrypted.

Conclusion

Using overlayroot gives additional instance security for the cloud user.  For more information about the additional configuration options of overlayroot, please refer to the overlayroot.conf file in the cloud-initramfs-tools repository on Launchpad.

Enjoy!

Growing and Securing Instance Root Filesystems With Overlayroot on Eucalyptus

CoreOS CloudInit Config for Docker Storage Management

CoreOS is a Linux distribution that allows easy deployment of Docker environments.  With CoreOS, users have the ability to deploy clustered Docker environments,  or deploy zero downtime applications.  Recently, I have blogged about how to deploy and use Docker on Eucalyptus cloud environments. This blog will focus on how to leverage cloud-init configuration with a CoreOS EMI to manage instance storage that will be used by Docker containers on Eucalyptus 4.0.  The same cloud-init configuration file can be used  on AWS with CoreOS AMIs, which is yet another example of how Eucalyptus has continued to maintain its focus on being the best on-premise AWS compatible cloud environment.

Prerequisites

Since Eucalyptus Identity and Access Management (IAM) is very similar to AWS’s IAM, at a minimum – the following Elastic Compute Cloud (EC2) actions need to be allowed:

In order to bundle, upload and register the CoreOS image, use the following AWS S3 policy (which can be generated using AWS Policy Generator):

{
  "Statement": [
    {
      "Sid": "Stmt1402675433766",
      "Action": "s3:*",
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

For more information about how to use Eucalyptus IAM, please refer to the Eucalyptus 4.0 Administrator documentation regarding access concepts and policy overview.

In addition to the correct IAM policy being applied to the user, here are the other prerequisites that need to be met:

Once these prerequisites are met, the Eucalyptus user will be able to implement the topic for this blog.

CoreOS CloudInit Config for Docker Storage Management

As mentioned in the CoreOS documentation regarding how to use CoreOS with Eucalyptus, the user needs to do the following:

  • Download the CoreOS image
  • Decompress the CoreOS image
  • Bundle, upload and register the image

For example:

# wget -q http://beta.release.core-os.net/amd64-usr/current/coreos_production_openstack_image.img.bz2

# bunzip2 coreos_production_openstack_image.img.bz2

# qemu-img convert -O raw coreos_production_openstack_image.img coreos_production_openstack_image.raw
# euca-bundle-and-upload-image -i coreos_production_openstack_image.raw -b coreos-production-beta -r x86_64
# euca-register -n coreos-production coreos-production-beta/coreos_production_openstack_image.raw.manifest.xml --virtualization-type hvm
IMAGE emi-98868F66

After the image is registered, create a security group and authorize port 22 for SSH access to the CoreOS instance:

# euca-create-group coreos-testing -d "Security Group for CoreOS Cluster"
GROUP sg-C8E3B168 coreos-testing Security Group for CoreOS Cluster
# euca-authorize -P tcp -p ssh coreos-testing
GROUP coreos-testing
PERMISSION coreos-testing ALLOWS tcp 22 22 FROM CIDR 0.0.0.0/0

Next, create a keypair that will be used to access the CoreOS instance:

# euca-create-keypair coreos > coreos.priv
# chmod 0600 coreos.priv

Now, we are need to create the cloud-init configuration file.  CoreOS implements a subset of cloud-init config spec with coreos-cloudinit.  The cloud-init config below will do the following:

  1. wipe the the ephemeral device – /dev/vdb (since the CoreOS EMI is an instance store-backed HVM image, ephemeral device will be /dev/vdb)
  2. format the ephemeral device with BTRFS filesystem
  3. mount /dev/vdb to /var/lib/docker (which is the location for images used by the Docker containers)

Create a cloud-init.config file with the following information:

#cloud-config
coreos:
 units:
 - name: format-ephemeral.service
 command: start
 content: |
 [Unit]
 Description=Formats the ephemeral drive
 [Service]
 Type=oneshot
 RemainAfterExit=yes
 ExecStart=/usr/sbin/wipefs -f /dev/vdb
 ExecStart=/usr/sbin/mkfs.btrfs -f /dev/vdb
 - name: var-lib-docker.mount
 command: start
 content: |
 [Unit]
 Description=Mount ephemeral to /var/lib/docker
 Requires=format-ephemeral.service
 Before=docker.service
 [Mount]
 What=/dev/vdb
 Where=/var/lib/docker
 Type=btrfs

Use euca-describe-instance-types to select the desired instance type for the CoreOS instance (in this example, c1.medium will be used).

# euca-describe-instance-types 
INSTANCETYPE Name CPUs Memory (MiB) Disk (GiB)
INSTANCETYPE t1.micro 1 256 5
INSTANCETYPE m1.small 1 512 10
INSTANCETYPE m1.medium 1 1024 10
INSTANCETYPE c1.xlarge 2 2048 10
INSTANCETYPE m1.large 2 1024 15
INSTANCETYPE c1.medium 1 1024 20
INSTANCETYPE m1.xlarge 2 1024 30
INSTANCETYPE m2.2xlarge 2 4096 30
INSTANCETYPE m3.2xlarge 4 4096 30
INSTANCETYPE m2.xlarge 2 2048 40
INSTANCETYPE m3.xlarge 2 2048 50
INSTANCETYPE cc1.4xlarge 8 3072 60
INSTANCETYPE m2.4xlarge 8 4096 60
INSTANCETYPE hi1.4xlarge 8 6144 120
INSTANCETYPE cc2.8xlarge 16 6144 120
INSTANCETYPE cg1.4xlarge 16 12288 200
INSTANCETYPE cr1.8xlarge 16 16384 240
INSTANCETYPE hs1.8xlarge 48 119808 24000

Use euca-run-instances to launch the CoreOS image as an instance, passing the cloud-init.config file using the –user-data-file option:

# euca-run-instances -k coreos -t c1.medium emi-98868F66 --user-data-file cloud-init-docker-storage.config
RESERVATION r-FC799274 408396244283 default
INSTANCE i-AF303D5D emi-98868F66 pending coreos 0 c1.medium 2014-06-12T13:38:31.008Z ViciousLiesAndDangerousRumors monitoring-disabled 0.0.0.0 0.0.0.0 instance-store hvm sg-A5133B59

Once the instance reaches the ‘running’ state, SSH into the instance to see the ephemeral storage mounted and formatted correctly:

# euca-describe-instances i-AF303D5D --region account1-user01@
RESERVATION r-FC799274 408396244283 default
INSTANCE i-AF303D5D emi-98868F66 euca-10-104-6-236.bigboi.acme.eucalyptus-systems.com euca-172-18-238-171.bigboi.internal running coreos 0 c1.medium 2014-06-12T13:38:31.008Z ViciousLiesAndDangerousRumors monitoring-disabled 10.104.6.236 172.18.238.17 instance-store hvm sg-A5133B59
# ssh -i coreos.priv core@euca-10-104-6-236.bigboi.acme.eucalyptus-systems.com
CoreOS (beta)
core@localhost ~ $ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 254:0 0 8.3G 0 disk
|-vda1 254:1 0 128M 0 part
|-vda2 254:2 0 64M 0 part
|-vda3 254:3 0 1G 0 part
|-vda4 254:4 0 1G 0 part /usr
|-vda6 254:6 0 128M 0 part /usr/share/oem
`-vda9 254:9 0 6G 0 part /
vdb 254:16 0 11.7G 0 disk /var/lib/docker
core@localhost ~ $ mount
.......
/dev/vda6 on /usr/share/oem type ext4 (rw,nodev,relatime,commit=600,data=ordered
/dev/vdb on /var/lib/docker type btrfs (rw,relatime,space_cache)

The instance is now ready for docker containers to be created.  For some docker container examples, check out the CoreOS documentation and the Docker documentation.

Enjoy!

CoreOS CloudInit Config for Docker Storage Management

Yet Another AWS Compatibility Example Using Vagrant-AWS Plugin and Eucalyptus 3.4 to Deploy Docker

Something that Eucalyptus has been constant about from the beginning is its stance on being the best open source, on-premise AWS-compatible cloud on the market.   This blog entry is just another example of demonstrating this compatibility.

My most recent blog entries have been centered around Docker and how to deploy it on Eucalyptus.  This entry will show how a user can take Docker’s own documentation on deploying Docker using Vagrant with AWS – but with Eucalyptus.   Before getting started, there are some prerequisites that need to be in place.

Prerequisites for Eucalyptus Cloud

In order to get started, the Eucalyptus cloud needs to have an Ubuntu Raring Cloud Image bundled, uploaded and registered before the steps below can be followed.  The previous blog entries below will help here:

In addition to the Ubuntu Raring Cloud EMI being available, the user also needs to have the following:

After these prerequisites have been met, the user is ready to set up Vagrant to interact with Eucalyptus.

Setting up Vagrant Environment

To start out, we need to set up the Vagrant environment.  The steps below will get you going:

  1. Install Vagrant from http://www.vagrantup.com/. (optional – package manager can be used here instead)
  2. Install the vagrant aws plugin:  

    vagrant plugin install vagrant-aws

After Vagrant and the vagrant aws plugin have been successfully installed, all that is left is to create a Vagrantfile to provide information to Vagrant as to how to interact with Eucalyptus.  Since the vagrant aws plugin is being used, and Eucalyptus is compatible with AWS, the configuration will be very similar to AWS.

I provided a Vagrantfile on Github to help get users up to speed quicker.  To check out the Vagrantfile, just clone the repository from Github:

$ git clone https://github.com/hspencer77/eucalyptus-docker-raring.git

After checking out the file, change directory to eucalyptus-docker-raring, and edit the following variables to match your user information for the Eucalyptus cloud that will run the Docker instance:

AWS_ENDPOINT = "<EC2_URL for Eucalyptus Cloud>"
AWS_AMI = "<ID for Ubuntu Raring EMI>"
AWS_ACCESS_KEY = "<Access Key ID>"
AWS_SECRET_KEY = "<Secret Key>"
AWS_KEYPAIR_NAME = "<Key Pair name>"
AWS_INSTANCE_TYPE = '<VM type>'
SSH_PRIVKEY_PATH = "<The path to the private key for the named keypair, for example ~/.ssh/docker.pem>"

Once the Vagrantfile is populated with the correct information, we are now ready to launch the Docker instance.

Launch the Docker Instance Using Vagrant

From here, Vagrant makes this very straight-forward.  There are only two steps to launch the Docker instance.

  1. Run the following command to launch the instance:

     vagrant up --provider=aws
  2. After Vagrant finishes deploying the instance, SSH into the instance:

    vagrant ssh

Thats it!  Once you are SSHed into the instance, to run Docker, execute the following command:

ubuntu@euca-172-17-120-212:~$ sudo docker

You have successfully launched a Docker instance on Eucalyptus using Vagrant.  Since Eucalyptus works with the vagrant aws plugin, the same Vagrantfile can be used against AWS (of course, the values for the variables above will change).  This is a perfect dev/test to production setup whether Eucalyptus is being used for dev/test and AWS being used for production (and vise versa).

Yet Another AWS Compatibility Example Using Vagrant-AWS Plugin and Eucalyptus 3.4 to Deploy Docker

Deploying CentOS 6.5 Image in Docker on Eucalyptus 3.4

This blog entry is a follow-up of my blog entry entitled “Step-by-Step Deployment of Docker on Eucalyptus 3.4 for the Cloud Administrator“.  In that blog entry, I covered how to deploy an Ubuntu Raring Cloud Image on Eucalyptus, and use that image to deploy Docker.   The really cool thing about Docker is that it provides the ability to deploy different operating systems within one machine – in this case, an instance.   The focus of this entry is to show how to deploy a CentOS 6.5 image in a instance running Docker on Eucalyptus 3.4.

Prerequisites

The prerequisites for this blog is to complete the steps outlined in my previous blog about Docker on Eucalyptus 3.4.  Once those steps are completed by the cloud administrator (i.e. a user associated with the ‘eucalyptus’ account), you can get started  on the steps below.  One thing to note here, to follow these steps, there is no need to be the cloud administrator.  This entry is entirely directed to cloud users.  Since Eucalyptus IAM is similar to AWS IAM, the following EC2 Actions need to be allowed for the cloud user:

Instance Deployment

To get started, we need to launch the Ubuntu Raring EMI that has been provided by the cloud administrator.  In this example, the EMI will be emi-26403979:

$ euca-describe-images emi-26403979 --region account1-user01@
IMAGE emi-26403979 ubuntu-raring-docker-rootfs-v3/ubuntu-raring-docker-v3.manifest.xml
 441445882805 available private x86_64 machine eki-17093995 
eri-6BF033EE instance-store paravirtualized

If the –region option seems confusing, this is due to the fact that I am using the nice configuration file feature in Euca2ools.  Its really helpful when you are using different credentials for different users.

Now that we know the EMI we can use, lets launch the instance.  We will be using the cloud-init config file from my previous Docker blog to configure the instance.  The VM type used here is c1.xlarge.  This is because I wanted to make sure I had 2 CPU, and 2 Gigs of RAM for my instance:

$ euca-run-instances -k account1-user01 -t c1.xlarge 
--user-data-file cloud-init-docker.config emi-26403979 
--region account1-user01@
RESERVATION r-2EE941D4 961915002812 default
INSTANCE i-503642D4 emi-26403979 euca-0-0-0-0.eucalyptus.euca-hasp.eucalyptus-systems.com
 euca-0-0-0-0.eucalyptus.internal pending account1-user01 0 
c1.xlarge 2014-02-14T00:47:15.632Z LayinDaSmackDown eki-17093995 
eri-6BF033EE monitoring-disabled 0.0.0.0 0.0.0.0 instance-store paravirtualized

Make sure the instance gets into the running state:

$ euca-describe-instances --region account1-user01@
RESERVATION r-2EE941D4 961915002812 default
INSTANCE i-503642D4 emi-26403979 euca-10-104-7-10.eucalyptus.euca-hasp.eucalyptus-systems.com
 euca-172-17-112-207.eucalyptus.internal running 
account1-user01 0 c1.xlarge 2014-02-14T00:47:15.632Z 
LayinDaSmackDown eki-17093995 eri-6BF033EE monitoring-disabled 
10.104.7.10 172.17.112.207 instance-store paravirtualized

Now thats in the running state, lets SSH into the instance to make sure its up and running:

$ ssh -i account1-user01.priv ubuntu@euca-10-104-7-10.eucalyptus.euca-hasp.eucalyptus-systems.com
Welcome to Ubuntu 13.04 (GNU/Linux 3.8.0-33-generic x86_64)
* Documentation: https://help.ubuntu.com/
System information as of Fri Feb 14 00:49:52 UTC 2014
System load: 0.21 Users logged in: 0
 Usage of /: 21.8% of 4.80GB IP address for eth0: 172.17.112.207
 Memory usage: 5% IP address for lxcbr0: 10.0.3.1
 Swap usage: 0% IP address for docker0: 10.42.42.1
 Processes: 85
Graph this data and manage this system at https://landscape.canonical.com/
Get cloud support with Ubuntu Advantage Cloud Guest:
 http://www.ubuntu.com/business/services/cloud
Use Juju to deploy your cloud instances and workloads:
 https://juju.ubuntu.com/#cloud-raring
Your Ubuntu release is not supported anymore.
For upgrade information, please visit:
http://www.ubuntu.com/releaseendoflife
New release '13.10' available.
Run 'do-release-upgrade' to upgrade to it.
Last login: Tue Nov 19 00:48:23 2013 from odc-d-06-07.prc.eucalyptus-systems.com
ubuntu@euca-172-17-112-207:~$

Now its time to prepare the instance to create your own base Docker CentOS 6.5 image.

Preparing the Instance

To prepare the instance to create the base image, install the following packages using apt-get:

ubuntu@euca-172-17-112-207:~$ sudo -s
root@euca-172-17-112-207:~# apt-get install rinse perl rpm \
rpm2cpio libwww-perl liblwp-protocol-https-perl

After the packages finish installing, lets go ahead and mount the ephemeral storage on the instance as /tmp to give Docker extra space for building the base image:

root@euca-172-17-112-207:~# curl http://169.254.169.254/latest/meta-data/block-device-mapping/ephemeral0
sda2
root@euca-172-17-112-207:~# mount /dev/vda2 /tmp
root@euca-172-17-112-207:~# df -ah
Filesystem Size Used Avail Use% Mounted on
/dev/vda1 4.8G 1.1G 3.5G 24% /
proc 0 0 0 - /proc
sysfs 0 0 0 - /sys
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 0 0 0 - /sys/fs/fuse/connections
none 0 0 0 - /sys/kernel/debug
none 0 0 0 - /sys/kernel/security
udev 993M 8.0K 993M 1% /dev
devpts 0 0 0 - /dev/pts
tmpfs 201M 240K 201M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 1002M 0 1002M 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/vda2 9.4G 150M 8.8G 2% /tmp
root@euca-172-17-112-207:~# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 15G 0 disk
├─vda1 253:1 0 5G 0 part /
├─vda2 253:2 0 9.5G 0 part /tmp
└─vda3 253:3 0 512M 0 part

Next, download the mkimage-rinse.sh script from the Docker repository on Github:

root@euca-172-17-112-207:~#  wget --no-check-certificate https://raw.github.com/dotcloud/docker/master/contrib/mkimage-rinse.sh
root@euca-172-17-112-207:~# chmod 755 mkimage-rinse.sh

Now we are ready to build the CentOS 6.5 Base Image.

Building the CentOS Image

The only thing left to do is build the base image.  Use mkimage-rinse.sh to build the CentOS 6.5 base image:

root@euca-172-17-112-207:~# ./mkimage-rinse.sh ubuntu/centos centos-6

After the installation is complete, test out the base image:

root@euca-172-17-112-207:~# docker run ubuntu/centos:6.5 cat /etc/centos-release
CentOS release 6.5 (Final)

Now we have a CentOS 6.5 base image.  The mkimage-rinse.sh script can also be used to install CentOS 5 base images as well.  Instead of passing centos-6, just pass centos-5.  For example, I have created a CentOS 5 base image in this instance as well.  Below shows the output of the images added to Docker:

root@euca-172-17-112-207:~# docker images
REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE
ubuntu/centos 5.10 06019bdea24b 2 minutes ago 123.8 MB
ubuntu/centos 6.5 cd23a8442c8a 2 hours ago 127.3 MB
root@euca-172-17-112-207:~# docker run ubuntu/centos:5.10 cat /etc/redhat-release
CentOS release 5.10 (Final)

As you can see, Docker helps developers test their software across multiple Linux distributions.  With Eucalyptus (just as in AWS), users can use the ephemeral nature of instances to quickly stand up this test environment in one instance.

Deploying CentOS 6.5 Image in Docker on Eucalyptus 3.4

12 Steps To EBS-Backed EMI Bliss on Eucalyptus

In previous posts, I shared how to use Ubuntu Cloud Images and eustore with Eucalyptus and AWS.  This blog entry will focus on how to use these assets to create EBS-backed EMIs in 12 steps.   These steps can be used on AWS as well, but instead of creating an instance store-backed AMI first, Ubuntu has already provided AMIs that can be used as the building block instance on AWS.  Let’s get started.

Prerequisites

On Eucalyptus and AWS, it is required the user has the appropriate IAM policy in order to perform these steps.  The policy should contain the following EC2 Actions at a minimum:

  • RunInstances
  • AttachVolume
  • AuthorizeSecurityGroupEgress
  • AuthorizeSecurityGroupIngress
  • CreateKeyPair
  • CreateSnapshot
  • CreateVolume
  • DescribeImages
  • DescribeInstances
  • DescribeInstanceStatus
  • DescribeSnapshots
  • DetachVolume
  • RegisterImage

In addition, the user needs an access key ID and secret key.  For more information, check out the following resources:

This entry also assumes Eucalyptus euca2ools are installed on the client machine.

The 12 Steps

Although the Ubuntu Cloud Image used in this entry is Ubuntu Precise (12.04) LTS, any of of the maintained Ubuntu Cloud images can be used.

  1. Use wget to download tar-gzipped precise-server-cloudimg:
    $ wget http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64.tar.gz
  2. After setting the EC2_ACCESS_KEY, EC2_SECRET_KEY, and EC2_URL, use eustore-install-image to an instance stored-backed EMI:
    $ eustore-install-image -t precise-server-cloudimg-amd64.tar.gz \
    -b ubuntu-latest-precise-x86_64 --hypervisor universal \
    -s "Ubuntu Cloud Image - Precise Pangolin - 12.04 LTS"
  3. Create a keypair using euca-create-keypair, then use euca-run-instances to launch an instance from the EMI returned from eustore-install-image. For example:
    $ euca-run-instances -t m1.medium \
    -k account1-user01 emi-5C8C3909
  4. Use euca-create-volume to create a volume based upon the size of how big you want the root filesystem to be.  The availability zone (-z option) will be based on if you are using Eucalyptus or AWS:
    $ euca-create-volume -s 6 \
    -z LayinDaSmackDown
  5. Using euca-attach-volume, attach the resulting volume to the running instance. For example:
    $ euca-attach-volume -d /dev/vdd \
    -i i-839E3FB0 vol-B5863B3B
  6. Use euca-authorize to open SSH access to the instance, SSH into the instance, then use wget to download the Ubuntu Precise Cloud Image (qcow2 format):
    $ ssh -i account1-user01.priv ubuntu@euca-10-104-7-10.eucalyptus.euca-hasp.eucalyptus-systems.com
    # sudo -s
    # wget http://cloud-images.ubuntu.com/precise/current/precise-server-cloudimg-amd64-disk1.img
  7. Install qemu-utils:
    # apt-get install -y qemu-utils
  8. Use qemu-img to convert image from qcow2 to raw:
    # qemu-img convert \
    -O raw precise-server-cloudimg-amd64-disk1.img precise-server-cloudimg-amd64-disk1-raw.img
  9. dd raw image to block device where volume is attached (use dmesg to figure that out easily):
    # dmesg | tail
    [ 7026.943212] virtio-pci 0000:00:05.0: using default PCI settings
    [ 7026.943249] pci 0000:00:07.0: no hotplug settings from platform
    [ 7026.943251] pci 0000:00:07.0: using default PCI settings
    [ 7026.945964] virtio-pci 0000:00:07.0: enabling device (0000 -> 0003)
    [ 7026.955143] virtio-pci 0000:00:07.0: PCI INT A -> Link[LNKC] -> GSI 10 (level, high) -> IRQ 10
    [ 7026.955180] virtio-pci 0000:00:07.0: setting latency timer to 64
    [ 7026.955429] virtio-pci 0000:00:07.0: irq 45 for MSI/MSI-X
    [ 7026.955456] virtio-pci 0000:00:07.0: irq 46 for MSI/MSI-X
    [ 7026.986990] vdb: unknown partition table
    [10447.093426] virtio-pci 0000:00:07.0: PCI INT A disabled
    # dd if=/mnt/precise-server-cloudimg-amd64-disk1-raw.img of=/dev/vdb bs=1M
  10. Log out the instance, and use euca-detach-volume to detach the volume:
    $ euca-detach-volume vol-B5863B3B
  11. Use euca-create-snapshot to create a snapshot of the volume:
    $ euca-create-snapshot vol-B5863B3B
  12. Use euca-register to register the resulting snapshot to create the EBS-backed EMI:
    $ euca-register --name ebs-precise-x86_64-sda \
    --snapshot snap-EFDB40A1 --root-device-name /dev/sda

Thats it!  You have successfully created an EBS-backed EMI/AMI.  As mentioned earlier, these steps can be used on AWS just as well (just skip steps 1 & 2, and use one of the Ubuntu Cloud Images in the AWS region of your choice).  Enjoy!

12 Steps To EBS-Backed EMI Bliss on Eucalyptus

Ubuntu Cloud Images and Eustore: The Eucalyptus Cloud Administrator’s Image Management Dream

Ubuntu provides versatile cloud images that can be utilized on various cloud deployment infrastructures.  Eucalyptus’s euca2ools eustore tool makes it easier for cloud administrator’s to bundle, upload and register images on Eucalyptus clouds.  Using eustore-install-image with Ubuntu Cloud images provides the best of both worlds – solid cloud images that can be easily deployed to any Eucalyptus cloud environment.

Set Up Euca2ools Configuration File

Setting up a euca2ools configuration file makes it easier and more efficient to interact with the euca2ools commands.  For this blog entry, the euca2ools configuration file ~/.euca/euca2ools.ini was set up with the following information:

[global]
default-region = LayinDaSmackDown
[user admin]
key-id = L4836KVYWMJCXT4T6Q6B9
secret-key = XCJ6sZVFVfFMR4DNVIUL7N7e4cgk8ebvEW0ej5dZ
account-id = 441445882805
private-key = /home/hspencer/admin-creds/euca2-admin-c9e4580c-pk.pem
certificate = /home/hspencer/admin-creds/euca2-admin-c9e4580c-cert.pem
[region LayinDaSmackDown]
autoscaling-url = http://10.104.1.216:8773/services/AutoScaling
ec2-url = http://10.104.1.216:8773/services/Eucalyptus
elasticloadbalancing-url = http://10.104.1.216:8773/services/LoadBalancing
iam-url = http://10.104.1.216:8773/services/Euare
monitoring-url = http://10.104.1.216:8773/services/CloudWatch
s3-url = http://10.104.1.216:8773/services/Walrus
eustore-url =http://emis.eucalyptus.com/
certificate = /home/hspencer/admin-creds/cloud-cert.pem

After setting up the euca2ools confirmation file, to utilize this file, run any euca2ools command with the –region option.  For example:

$ euca-describe-images --region admin@

Since kernel, ramdisk and root filesystem images will be bundled, uploaded and registered, the cloud administrator’s credentials were used in the euca2ools configuration file.  For more information about kernel and ramdisk management in Eucalyptus, check out the KB article entitled “Kernel and Ramdisk Management in Eucalyptus”.

Ubuntu Cloud Images

To obtain Ubuntu Cloud images that will be bundled, uploaded and registered with eustore-install-image, download the supported Ubuntu Cloud image release of your choice from the Ubuntu Cloud Images page. Download the file thats ends with either amd64.tar.gz or i386.tar.gz using either curl, wget or any other network transfer tool.

For example, to download the latest Ubuntu 13.10 Saucy Salamander Cloud image, run the following command:

$ wget http://cloud-images.ubuntu.com/saucy/current/saucy-server-cloudimg-amd64.tar.gz

This will download and save the saucy-server-cloudimg-amd64.tar.gz file.  Now its time to use eustore-install-image to bundle, upload and register the image.

Bundle, Upload, Register The Image

As mentioned earlier, eustore-install-image will bundle, upload and register the image.  To do this, use the –tarball option of eustore-install-image with the tar-gzipped file downloaded from Ubuntu Cloud Image page.  The key flag here is the –hypervisor option.  Because Ubuntu Cloud images are crafted to work on multiple hypervisors (e.g. Xen, KVM, VMware, etc.), set the –hypervisor option to “universal”. Here is an example of using these options:

$ eustore-install-image -t saucy-server-cloudimg-amd64.tar.gz -b ubuntu-saucy-server-amd64 --hypervisor universal -s "Ubuntu 13.10 - Saucy Salamander" -p ubuntu-saucy --region admin@ -a x86_64
Preparing to extract image...
Extracting kernel 100% |=========================================================================================| 5.34 MB 158.76 MB/s Time: 0:00:00
Bundling kernel 100% |=========================================================================================| 5.34 MB 27.39 MB/s Time: 0:00:00
-- Uploading kernel image --
saucy-server-cloudimg-amd64-vmlinuz-generic.part.0 100% |=========================================================| 5.28 MB 20.91 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64-vmlinuz-generic.manifest.xml 100% |===================================================| 3.45 kB 26.26 kB/s Time: 0:00:00
Registered kernel image eki-2C54378B
Extracting ramdisk 100% |=========================================================================================| 89.56 kB 45.29 MB/s Time: 0:00:00
Bundling ramdisk 100% |=========================================================================================| 89.56 kB 8.31 MB/s Time: 0:00:00
-- Uploading ramdisk image --
saucy-server-cloudimg-amd64-loader.part.0 100% |==================================================================| 89.38 kB 689.09 kB/s Time: 0:00:00
saucy-server-cloudimg-amd64-loader.manifest.xml 100% |============================================================| 3.36 kB 26.03 kB/s Time: 0:00:00
Registered ramdisk image eri-015435B7
Extracting image 100% |=========================================================================================| 1.38 GB 183.45 MB/s Time: 0:00:08
Bundling image 100% |=========================================================================================| 1.38 GB 34.33 MB/s Time: 0:00:43
-- Uploading machine image --
saucy-server-cloudimg-amd64.img.part.0 ( 1/22) 100% |============================================================| 10.00 MB 42.72 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.1 ( 2/22) 100% |============================================================| 10.00 MB 41.06 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.2 ( 3/22) 100% |============================================================| 10.00 MB 42.43 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.3 ( 4/22) 100% |============================================================| 10.00 MB 44.56 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.4 ( 5/22) 100% |============================================================| 10.00 MB 48.43 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.5 ( 6/22) 100% |============================================================| 10.00 MB 57.00 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.6 ( 7/22) 100% |============================================================| 10.00 MB 48.69 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.7 ( 8/22) 100% |============================================================| 10.00 MB 51.00 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.8 ( 9/22) 100% |============================================================| 10.00 MB 43.92 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.9 (10/22) 100% |============================================================| 10.00 MB 46.26 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.10 (11/22) 100% |============================================================| 10.00 MB 46.27 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.11 (12/22) 100% |============================================================| 10.00 MB 48.74 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.12 (13/22) 100% |============================================================| 10.00 MB 44.48 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.13 (14/22) 100% |============================================================| 10.00 MB 44.20 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.14 (15/22) 100% |============================================================| 10.00 MB 48.89 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.15 (16/22) 100% |============================================================| 10.00 MB 46.45 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.16 (17/22) 100% |============================================================| 10.00 MB 56.84 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.17 (18/22) 100% |============================================================| 10.00 MB 53.62 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.18 (19/22) 100% |============================================================| 10.00 MB 56.75 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.19 (20/22) 100% |============================================================| 10.00 MB 48.67 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.20 (21/22) 100% |============================================================| 10.00 MB 44.25 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.part.21 (22/22) 100% |============================================================| 3.19 MB 18.21 MB/s Time: 0:00:00
saucy-server-cloudimg-amd64.img.manifest.xml 100% |===============================================================| 6.63 kB 41.32 kB/s Time: 0:00:00
Registered machine image emi-4EFE3A91
-- Done --
Installed new image emi-4EFE3A91

Now that the kernel, ramdisk and root filesystem are bundled, uploaded and registered, the launch permission of each image needs to be changed so that all the users of the Eucalyptus cloud can launch instances from these images.

$ euca-modify-image-attribute -l -a all emi-4EFE3A91 --region admin@
launchPermission emi-4EFE3A91 ADD Group all
$ euca-modify-image-attribute -l -a all eri-015435B7 --region admin@
launchPermission eri-015435B7 ADD Group all
$ euca-modify-image-attribute -l -a all eki-2C54378B --region admin@
launchPermission eki-2C54378B ADD Group all

Thats it!  Now users can launch instances from the EMI, EKI and ERI as below:

$ euca-run-instances -k account1-user01 -t m1.medium emi-4EFE3A91 --region account1-user01@ --user-data-file cloud-init.config

Enjoy!

Ubuntu Cloud Images and Eustore: The Eucalyptus Cloud Administrator’s Image Management Dream

Step-by-Step Deployment of Docker on Eucalyptus 3.4 for the Cloud Administrator

Docker

Eucalyptus Systems, Inc.

Docker has been in the news lately as one of the hot open-source project promoting linux containers. Some use cases for Docker include the following:

  • Automation of packaging and application deployment
  • Lightweight PaaS environments
  • Automated testing and continuous integration/deployment
  • Deploying and scaling web applications, databases and backend services

The focus of this blog entry is to show how to deploy Docker on Eucalyptus from a cloud administrator’s point-of-view – all in the cloud.  This is a step-by-step guide to create an Docker EMI from an existing Ubuntu Cloud Raring EMI using AWS’s documentation.  This entry will also show how to build euca2ools from source in the Ubuntu Cloud image.

Prerequisites

This entry assumes the following:

After confirming that the prerequisites are met, let’s get started.

Creating an EMI From an Existing EMI

As mentioned earlier, these steps will be based off of  AWS’s documentation on creating an instance store-backed AMI from an existing AMI.  In this example, here is an existing Ubuntu Raring instance thats running on Eucalyptus:

$ euca-describe-instances --region eucalyptus-admin@
RESERVATION r-3E423E33 961915002812 default
INSTANCE i-827E3E88 emi-06663A57 euca-10-104-7-12.eucalyptus.euca-hasp.eucalyptus-systems.com
euca-172-17-118-27.eucalyptus.internal running euca-admin 0 m1.medium 
2013-11-18T22:41:35.694Z LayinDaSmackDown eki-28F338EB eri-51253C0A 
monitoring-disabled 10.104.7.12 172.17.118.27 instance-store

This instance is using the following EMI, EKI and ERI:

$ euca-describe-images emi-06663A57 eki-28F338EB eri-51253C0A --region eucalyptus-admin@
IMAGE eki-28F338EB latest-raring-kernel/raring-server-cloudimg-amd64-vmlinuz-generic.manifest.xml 
441445882805 available public x86_64 kernel instance-store
IMAGE emi-06663A57 latest-raring/raring-server-cloudimg-amd64.img.manifest.xml 441445882805 
available public x86_64 machine eki-28F338EB eri-51253C0A instance-store paravirtualized
IMAGE eri-51253C0A latest-raring-kernel/raring-server-cloudimg-amd64-loader.manifest.xml 441445882805 
available public x86_64 ramdisk instance-store

To start, copy the zip file cloud administrator credentials obtained by the euca_conf command mentioned in the Eucalyptus 3.4 documentation to the running instance:

# scp -i euca-admin.priv admin.zip 
ubuntu@euca-10-104-7-12.eucalyptus.euca-hasp.eucalyptus-systems.com:/tmp/.

Next, install the following packages for the 3.8.0-33 kernel, and packages needed to build euca2ools:

ubuntu@euca-172-17-118-27:~$ sudo apt-get install python-setuptools git python-lxml unzip linux-headers-3.8.0-33-generic linux-image-extra-3.8.0-33-generic

Find the ephemeral storage using the instance metadata service, format, and mount the ephemeral to /mnt/image:

ubuntu@euca-172-17-118-27:~$ curl http://169.254.169.254/latest/meta-data/block-device-mapping/ephemeral
sda2
ubuntu@euca-172-17-118-27:~$ sudo mkdir /mnt/image
ubuntu@euca-172-17-118-27:~$ sudo mkfs.ext4 /dev/vda2
ubuntu@euca-172-17-118-27:~$ sudo mount /dev/vda2 /mnt/image

Download euca2ools from Github:

ubuntu@euca-172-17-118-27:~$ git clone https://github.com/eucalyptus/euca2ools.git

Install euca2ools:

ubuntu@euca-172-17-118-27:~$ cd euca2ools; sudo python setup.py install

Unzip the cloud administrator credentials in /tmp:

ubuntu@euca-172-17-118-27:~$ cd /tmp; unzip admin.zip

Change to the root user, and source the cloud administrator credentials:

ubuntu@euca-172-17-118-27:~$ sudo -s; source /tmp/eucarc

Bundle, upload and register the ramdisk and kernel under /boot:

root@euca-172-17-118-27:~# euca-bundle-image -i /boot/initrd.img-3.8.0-33-generic 
--ramdisk true -r x86_64
root@euca-172-17-118-27:~# euca-upload-bundle -b ubuntu-raring-docker-ramdisk 
-m /var/tmp/bundle-SQrAuT/initrd.img-3.8.0-33-generic.manifest.xml
root@euca-172-17-118-27:~# euca-register -n ubuntu-raring-docker-ramdisk 
ubuntu-raring-docker-ramdisk/initrd.img-3.8.0-33-generic.manifest.xml 
IMAGE eri-6BF033EE
root@euca-172-17-118-27:~# euca-bundle-image -i /boot/vmlinuz-3.8.0-33-generic 
--kernel true -r x86_64
root@euca-172-17-118-27:~# euca-upload-bundle -b ubuntu-raring-docker-kernel 
-m /var/tmp/bundle-31Lnxy/vmlinuz-3.8.0-33-generic.manifest.xml
root@euca-172-17-118-27:~# euca-register -n ubuntu-raring-docker-kernel 
ubuntu-raring-docker-kernel/vmlinuz-3.8.0-33-generic.manifest.xml
IMAGE eki-17093995

Use euca-bundle-vol to bundle the root filesystem. Make sure to exclude /tmp, /mnt/image, and /home/ubuntu. Additionally, make sure and set the size of the image to be 5 GB:

root@euca-172-17-118-27:~# euca-bundle-vol -p ubuntu-raring-docker 
-s 5120 -e /tmp,/root,/mnt/image,/home/ubuntu -d /mnt/image 
--kernel eki-17093995 --ramdisk eri-6BF033EE -r x86_64

Next, upload and register the root filesystem:

root@euca-172-17-118-27:~# euca-upload-bundle -b ubuntu-raring-docker-rootfs 
-m /mnt/image/ubuntu-raring-docker.manifest.xml
root@euca-172-17-118-27:~# euca-register -n ubuntu-raring-docker-rootfs 
ubuntu-raring-docker-rootfs/ubuntu-raring-docker.manifest.xml
IMAGE emi-26403979

We have the new EMI, EKI and ERI for the Docker instance.  Lastly, set the image permissions so that all users on the cloud can use the EMI, EKI and ERI:

root@euca-172-17-118-27:~# euca-modify-image-attribute -l -a all emi-26403979
root@euca-172-17-118-27:~# euca-modify-image-attribute -l -a all eki-17093995
root@euca-172-17-118-27:~# euca-modify-image-attribute -l -a all eri-6BF033EE

Now its time to launch the Docker EMI.

Running the Docker Instance with Cloud-Init

Before launching the EMI, the cloud-init configuration file needs to be created.  This file will be responsible for configuring the instance repositories, downloading and installing Docker.  With your favorite command-line editor, create a file called cloud-init-docker.config, with the following content:

#cloud-config
apt_update: true
apt_upgrade: true
disable_root: true
packages:
 - less
cloud_config_modules:
 - ssh
 - [ apt-update-upgrade, always ]
 - updates-check
 - runcmd
runcmd:
 - [ sh, -xc, "INST_HOSTNAME=`/bin/hostname`; META_IP=`curl -s http://169.254.169.254/latest/meta-data/local-ipv4`; echo ${META_IP} ${INST_HOSTNAME} >> /etc/hosts" ]
 - [ locale-gen, en_US.UTF-8 ]
 - [ sh, -xc, "wget -qO docker-io.gpg https://get.docker.io/gpg" ]
 - [ apt-key, add, docker-io.gpg ]
 - [ sh, -xc, "echo 'deb http://get.docker.io/ubuntu docker main' > /etc/apt/sources.list.d/docker.list" ]
 - [ apt-get, update ]
 - [ apt-get, install, -y, --force-yes, lxc-docker ]
 - [ modprobe, -q, aufs ]

Now, use euca-run-instances to launch the instance:

root@euca-172-17-118-27:~# euca-run-instances -k euca-admin emi-351237D1 
-t m1.medium --user-data-file cloud-init-docker.config

After launching the instance, leave the current instance to get back to end client.

root@euca-172-17-118-27:~# exit
exit
ubuntu@euca-172-17-118-27:~$ exit
logout
Connection to 10.104.7.12 closed.

Once the instance reaches running state, ssh into the instance using the keypair specified (which in this case will be euca-admin.priv), and execute the following Docker command to run an interactive shell session inside a minimal Ubuntu container:

$ euca-describe-instances --region eucalyptus-admin@
RESERVATION r-A1613D7F 961915002812 default
INSTANCE i-AFDB3D4C emi-26403979 euca-10-104-7-13.eucalyptus.euca-hasp.eucalyptus-systems.com 
euca-172-17-118-16.eucalyptus.internal running euca-admin 0 m1.medium 
2013-11-19T01:21:10.880Z LayinDaSmackDown eki-17093995 eri-6BF033EE monitoring-disabled 
10.104.7.13 172.17.118.16 instance-store
# ssh -i euca-admin.priv ubuntu@euca-10-104-7-13.eucalyptus.euca-hasp.eucalyptus-systems.com
Welcome to Ubuntu 13.04 (GNU/Linux 3.8.0-33-generic x86_64)
* Documentation: https://help.ubuntu.com/
System information as of Thu Nov 14 23:18:38 UTC 2013
System load: 0.0 Users logged in: 0
 Usage of /: 21.6% of 4.89GB IP address for eth0: 172.17.184.76
 Memory usage: 4% IP address for lxcbr0: 10.0.3.1
 Swap usage: 0% IP address for docker0: 10.1.42.1
 Processes: 83
Graph this data and manage this system at https://landscape.canonical.com/
Get cloud support with Ubuntu Advantage Cloud Guest:
 http://www.ubuntu.com/business/services/cloud
Use Juju to deploy your cloud instances and workloads:
 https://juju.ubuntu.com/#cloud-raring
New release '13.10' available.
Run 'do-release-upgrade' to upgrade to it.
Last login: Thu Nov 14 23:08:09 2013 from 10.104.10.6
ubuntu@euca-172-17-184-76:~$ sudo docker run -i -t ubuntu /bin/bash
Unable to find image 'ubuntu' (tag: latest) locally
Pulling repository ubuntu
8dbd9e392a96: Download complete
b750fe79269d: Download complete
27cf78414709: Download complete
root@041d5ddcd6b9:/# (Ctrl-p Ctrl-q to exit out of shell)
ubuntu@euca-172-17-184-76:~$ sudo docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
041d5ddcd6b9 ubuntu:12.04 /bin/bash 27 seconds ago Up 26 seconds pink_frog

Thats it!  For more information regarding Docker, please refer to the latest Docker documentation.

Enjoy!

Step-by-Step Deployment of Docker on Eucalyptus 3.4 for the Cloud Administrator

IAM Roles and Instance Profiles in Eucalyptus 3.4

IAM Roles in AWS are quite powerful – especially when users need instances to access service APIs to implement complex deployments.  In the past, this could be accomplished by passing access keys and secret keys through the instance user data service, which can be cumbersome and is quite insecure.  With IAM roles, instances can be launched with profiles that allow them to leverage various IAM policies provided by the user to control what service APIs  instances can access in a secure manner.  As part of  constant pursuit for AWS compatibility, one of the new features in Eucalyptus 3.4 is the support of IAM roles and instance profiles (and yes, it works with tools like ec2-api-tools, and libraries like boto, which support accessing IAM roles through the instance meta data service).

This blog entry will demonstrate the following:

  • Set up an Eucalyptus IAM role
  • Create an Eucalyptus instance profile
  • Assign an instance profile when launching an instance
  • Leverage the IAM role from within the instance to access a service API (for this example, it will be the EC2 service API on Eucalyptus)

Prerequisites

To use IAM roles on Eucalyptus, the following is required:

  • A Eucalyptus 3.4 cloud – These packages can be downloaded from the Eucalyptus 3.4 nightly repo.  For additional information regarding downloading nightly builds of Eucalyptus, please refer the Eucalyptus Install Guide (note: anywhere there is a “3.3” reference, replace with “3.4”)
  • User Credentials – User credentials for an account administrator (admin user), and credentials of a non-admin user of a non-eucalyptus account.
  • Apply an IAM policy for the non-admin user to launch instances, and pass roles to instances launched by that user using euare-useruploadpolicy.  An example policy is below:

    {"Statement": [
     "Effect":"Allow",
     "Action":"iam:PassRole",
     "Resource":"*"
     },
     {
     "Effect":"Allow",
     "Action":"iam:ListInstanceProfiles",
     "Resource":"*"
     },
     {
     "Effect":"Allow",
     "Action":"ec2:*",
     "Resource":"*"
     }]
    }

  • AWS IAM CLI Tools and Euca2ools 3 – The AWS IAM CLI tools are for creating IAM roles and instance profiles; euca2ools for launching instances. There will be one configuration file for the AWS IAM CLI tools that will contain the credentials of the account admin user (for example, account1-admin.config).  Euca2ools will only need the credentials of the non-admin user in the euca2ools.ini file (for example, creating a user section called account1-user01].

Creating  a Eucalyptus IAM Role

Just as in AWS IAM, iam-rolecreate can be used with Eucalyptus IAM to create IAM roles.  To create a IAM role on Eucalyptus, run the following command:

# iam-rolecreate --aws-credential-file account1-admin.config 
--url http://10.104.10.6:8773/services/Euare/ -r ACCT1-EC2-ACTIONS 
-s http://10.104.10.6:8773/services/Eucalyptus
# iam-rolelistbypath --aws-credential-file account1-admin.config
 --url http://10.104.10.6:8773/services/Euare/
arn:aws:iam::735723906303:role/ACCT1-EC2-ACTIONS
IsTruncated: false

This will create a IAM role called ACCT1-EC2-ACTIONS.  Next, we need to add an IAM policy to the role.  As mentioned earlier, the IAM policy will allow the instance to execute an EC2 API call (in this case, ec2-describe-availability-zones).  Use iam-roleuploadpolicy to upload the following IAM policy file:

{
"Statement": [
{
"Sid": "Stmt1381454720306",
"Action": [
"ec2:DescribeAvailabilityZones"
],
"Effect": "Allow",
"Resource": "*"
}
]
}

After the IAM policy file has been created (e.g. ec2-describe-az), upload the policy to the role:

# iam-roleuploadpolicy --aws-credential-file account1-admin.config 
--url http://10.104.10.6:8773/services/Euare/ -p ec2-describe-az 
-f ec2-describe-az -r ACCT1-EC2-ACTIONS
# iam-rolelistpolicies --aws-credential-file account1-admin.config 
--url http://10.104.10.6:8773/services/Euare/ -r ACCT1-EC2-ACTIONS -v
ec2-describe-az
{
 "Statement": [
 {
 "Sid": "Stmt1381454720306",
 "Action": [
 "ec2:DescribeAvailabilityZones"
 ],
 "Effect": "Allow",
 "Resource": "*"
 }
 ]
}
IsTruncated: false

As displayed, the IAM role has been created, and an IAM policy has been added to the role successfully.  Now its time to deal with instance profiles.

Create an Instance Profile and Add a Role to the Profile

Instance profiles are used to pass the IAM role to the instance.  An IAM role can be associated to many instance profiles, but an instance profile can be associated to only one IAM role.  To create an instance profile, use iam-instanceprofilecreate.  Since the IAM role ACCT1-EC2-ACTIONS was previously created, the role can be added as the instance profile is created:

# iam-instanceprofilecreate 
--aws-credential-file account1-admin.config 
--url http://10.104.10.6:8773/services/Euare/ -r ACCT1-EC2-ACTIONS 
-s instance-ec2-actions
# iam-instanceprofilelistbypath --aws-credential-file acct1-user1-aws-iam.config 
--url http://10.104.10.6:8773/services/Euare/
arn:aws:iam::735723906303:instance-profile/instances-ec2-actions
IsTruncated: false

We have successfully created an instance profile and associated an IAM role to it.  All that is left to do is test it out.

Testing out the Instance Profile

Before testing out the instance profile, make sure that the euca2ools.ini file has the correct user and region information for the non-admin user of the account (for this example, the user will be user01).  For information about obtaining the credentials for the user, please refer to the section “Create Credentials” in the Eucalyptus User Guide.

After setting up the euca2ools.ini file, use euca-run-instance to launch an instance with an instance profile.  The image used here is the Ubuntu Raring Cloud Image.  The keypair account1-user01 was created using euca-create-keypair.  To open up SSH access to the instance, use euca-authorize.   Create a cloud-init user data file to enable the multiverse repository.

# cat cloud-init.config
#cloud-config
apt_sources:
 - source: deb $MIRROR $RELEASE multiverse
apt_update: true
apt_upgrade: true
disable_root: true
# euca-run-instances --key account1-user1 emi-C25538DA 
--instance-type m1.large --user-data-file cloud-init.config 
--iam-profile arn:aws:iam::407837561996:instance-profile/instance-ec2-actions 
--region account1-user01@
RESERVATION r-CED1435E 407837561996 default
INSTANCE i-72F244CC emi-C25538DA 0.0.0.0 0.0.0.0 pending account1-user01 0 
m1.large 2013-10-10T22:08:00.589Z Exodus eki-C9083808 eri-39BC3B99 
monitoring-disabled 0.0.0.0 0.0.0.0 instance-store paravirtualized 
arn:aws:iam::407837561996:instance-profile/instance-ec2-actions
....
# euca-describe-instances --region account1-user01@
RESERVATION r-CED1435E 407837561996 default
INSTANCE i-72F244CC emi-C25538DA 10.104.7.22 172.17.190.157 
running account1-user01 0 m1.large 2013-10-10T22:08:00.589Z Exodus eki-C9083808 
eri-39BC3B99 monitoring-disabled 10.104.7.22 172.17.190.157 
instance-store paravirtualized 
arn:aws:iam::407837561996:instance-profile/instance-ec2-actions
TAG instance i-72F244CC euca:node 10.105.10.11

Next, SSH into the instance and confirm the instance profile is accessible by the instance meta-data service.

[root@odc-c-06 ~]# ssh-keygen -R 10.104.7.22
/root/.ssh/known_hosts updated.
Original contents retained as /root/.ssh/known_hosts.old
[root@odc-c-06 ~]# ssh -i euca-admin.priv ubuntu@10.104.7.22
The authenticity of host '10.104.7.22 (10.104.7.22)' can't be established.
RSA key fingerprint is a1:b2:5d:1a:be:e3:cb:0b:58:5f:bd:c1:e2:1f:e3:2d.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '10.104.7.22' (RSA) to the list of known hosts.
The programs included with the Ubuntu system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Ubuntu comes with ABSOLUTELY NO WARRANTY, to the extent permitted by
applicable law.
Welcome to Ubuntu 13.04 (GNU/Linux 3.8.0-31-generic x86_64)
* Documentation: https://help.ubuntu.com/
.....
Get cloud support with Ubuntu Advantage Cloud Guest:
 http://www.ubuntu.com/business/services/cloud
Use Juju to deploy your cloud instances and workloads:
 https://juju.ubuntu.com/#cloud-raring
0 packages can be updated.
0 updates are security updates.
ubuntu@ip-172-17-190-157:~$ curl http://169.254.169.254/latest/meta-data/
ami-id
ami-launch-index
ami-manifest-path
block-device-mapping/
hostname
iam/
instance-id
instance-type
kernel-id
local-hostname
local-ipv4
mac
placement/
public-hostname
public-ipv4
public-keys/
ramdisk-id
reservation-id
security-groups
### check for IAM role temporary secuirty credentials ###
ubuntu@ip-172-17-190-157:~$ curl http://169.254.169.254/latest/meta-data/iam/
security-credentials/ACCT1-EC2-ACTIONS
{
 "Code": "Success",
 "LastUpdated": "2013-10-11T18:07:37Z",
 "Type": "AWS-HMAC",
 "AccessKeyId": "AKIYW7FDRV8ZG5HIM91D",
 "SecretAccessKey": "sgVOgLJoc3wXjI5mu7yrYXI3NHtiq18cJuOT7Mwh",
 "Token": "ZXVjYQABQe4E4f2NnIsnvT/5jfpauKh3dClPVwPEoMepqk0lViODSgk4axiQb9rRQyU7Qnhvxb22wO201EoT6Ay/
rg+1i3+2xQLfbkh7kqy4CmqdGM3Q7LNI1dFPSz332E6us5BsSdHpiw3VGLyMLnDAkV8BMi+6lKE5eaJ+hpFI/
KXEVPSNkFMI9R+9bKPIFZvceiBE1w+kAEJC/18uCpZ0kSNy2iFBYcZ+zTwrYTgnsqNYcEIuWzEh4z1WIA==",
 "Expiration": "2013-10-11T19:07:37Z"
}

Install the ec2-api-tools from the Ubuntu Raring multiverse repository.

ubuntu@ip-172-17-190-157:~$ sudo apt-get update
Get:1 http://security.ubuntu.com raring-security Release.gpg [933 B]
Hit http://Exodus.clouds.archive.ubuntu.com raring Release.gpg
......
Ign http://Exodus.clouds.archive.ubuntu.com raring-updates/main Translation-en_US
Ign http://Exodus.clouds.archive.ubuntu.com raring-updates/multiverse Translation-en_US
Ign http://Exodus.clouds.archive.ubuntu.com raring-updates/universe Translation-en_US
Fetched 8,015 kB in 19s (421 kB/s)
Reading package lists... Done
ubuntu@ip-172-17-190-157:~$ sudo apt-get install ec2-api-tools
Reading package lists... Done
The following extra packages will be installed:
 ca-certificates-java default-jre-headless fontconfig-config
 icedtea-7-jre-jamvm java-common libavahi-client3 libavahi-common-data 
libavahi-common3 libcups2 libfontconfig1 libjpeg-turbo8 libjpeg8 liblcms2-2
 libnspr4 libnss3 libnss3-1d openjdk-7-jre-headless openjdk-7-jre-lib 
ttf-dejavu-core tzdata-java
......
Adding debian:TDC_Internet_Root_CA.pem
Adding debian:SecureTrust_CA.pem
done.
Setting up openjdk-7-jre-lib (7u25-2.3.10-1ubuntu0.13.04.2) ...
Processing triggers for libc-bin ...
ldconfig deferred processing now taking place
Processing triggers for ca-certificates ...
Updating certificates in /etc/ssl/certs... 0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d....
done.
done.

Finally, run ec2-describe-availability-zones using the –url option to point to the Eucalyptus cloud being used.

ubuntu@ip-172-17-190-157:~$ ec2-describe-availability-zones 
-U http://10.104.10.6:8773/services/Eucalyptus/
AVAILABILITYZONE Legend 10.104.1.185 
arn:euca:eucalyptus:Legend:cluster:IsThisLove/
AVAILABILITYZONE Exodus 10.104.10.22 
arn:euca:eucalyptus:Exodus:cluster:NaturalMystic/

Thats it!  Notice how there wasn’t a need to pass any access key and secret key.  All that information is grabbed from the instance meta-data service.

IAM roles and instance profiles are quite powerful.  Great use cases include enabling CloudWatch metrics, and deploying ELBs on Eucalyptus.

I hope this has been helpful.  As always, any questions/suggestions/ideas/feedback are greatly appreciated.

 

IAM Roles and Instance Profiles in Eucalyptus 3.4

Bind DNS + OpenLDAP MDB == Dynamic Domain and Fully Delegated Sub-Domain Configuration of DNS

This blog post was driven by the need to make it easier to test Eucalyptus DNS in a lab environment.   The goal was to have a scriptable way to add/delete fully delegated sub-domains without having to reload/restart DNS when Eucalyptus clouds were being deployed/destroyed.   This was tested on an CentOS 6 instance running in a Eucalyptus 3.3 HA Cloud.

Prerequisites

This entry will not cover setting up Eucalyptus HA, creating a Eucalyptus user, using eustore to register the image, and/or opening up ports in security groups.  Its assumed the reader understands these concepts.  The focus will be configuring and deploying Bind9 and OpenLDAP.

In addition to using a CentOS 6.4 image, the following is needed:

  • ports open for DNS (tcp and udp 53)
  • port open for OpenLDAP (tcp 389)
  • port open for SSH (tcp 22)

Now that the prereqs have been covered, lets jump into setting up the environment.

Base Software Installation

There are a series of packages needed to install OpenLDAP (since we are building from source), and Bind9.  Once the instance is launched and running, SSH into the instance, and run the following commands:

# sudo yum -y upgrade
# sudo yum install -y git cyrus-sasl gcc glibc-devel libtool-ltdl \
db4-devel openssl-devel unixODBC-devel libtool-ltdl-devel libtool \
cyrus-sasl-devel cyrus-sasl-gssapi cyrus-sasl-lib cyrus-sasl-md5 \
make bind-dyndb-ldap

All the packages except for bind-dyndb-ldap are needed to build OpenLDAP from source.  The reason we are building OpenLDAP from source is to take advantage of their powerful backend – MDB.  Check out my previous blogs on this topic from this listing.

The key package for Bind9 DNS to communicate to OpenLDAP as a backend is bind-dyndb-ldap.  This plug-in is used by the FreeIPA Identity/Policy Management application to help leverage 389 Directory Servers (which is based off OpenLDAP) for storing domain name information.

OpenLDAP Installation and Configuration

Since all the base packages are installed, we can now grab and install the latest source version of OpenLDAP.  While still being logged into the instance, run the following command:

# git clone git://git.openldap.org/openldap.git ~/openldap

Since we are working with an instance based on the CentOS 6 image on eustore, we will use the ephemeral store (which is mounted under /media/ephemeral0) for the location of our OpenLDAP installation.  Create a directory for installing OpenLDAP on the ephemeral store by running the command below:

# mkdir /media/ephemeral0/openldap

Next, configure OpenLDAP:

# cd ~/openldap
# ./configure --prefix=/media/ephemeral0/openldap --enable-debug=yes \
--enable-syslog --enable-dynamic --enable-slapd --enable-dynacl \
--enable-spasswd --enable-modules --enable-rlookups --enable-mdb \
--enable-monitor --enable-overlays --with-cyrus-sasl --with-threads \
--with-tls=openssl CC="gcc" LDFLAGS="-L/usr/lib64/sasl2" CPPFLAGS="-I/usr/include/sasl"

After thats completed successfully, compile and install OpenLDAP:

# make depend
# make
# sudo make install

After the installation is complete, create the openldap user that will be responsible for running OpenLDAP:

# sudo useradd -m -U -c "OpenLDAP User" -s /bin/bash openldap
# sudo passwd -l openldap

Since the bind-dyndb-ldap package was installed earlier, copy the schema to where OpenLDAP stores its schemas, so that it can be added to the OpenLDAP configuration:

# sudo cp /usr/share/doc/bind-dyndb-ldap-2.3/schema \
/media/ephemeral0/openldap/etc/openldap/schema/bind-dyndb-ldap.schema

Next, create the LDAP password for the cn=admin,cn=config user.  This user is responsible for managing the configuration of the OpenLDAP server using OLC:

# /media/ephemeral0/openldap/sbin/slappasswd -h {SSHA}

Modify the slapd.conf file located under /media/ephemeral0/openldap/etc/openldap/, to set up the base configuration structure (cn=config) for OpenLDAP.  When completed, it should look like the following:

#######################################################################
# Config database definitions
#######################################################################
pidfile /media/ephemeral0/openldap/var/run/slapd.pid
argsfile /media/ephemeral0/openldap/var/run/slapd.args
database config
rootdn cn=admin,cn=config
rootpw {SSHA}xxxxxxxxxxxxxxxxxxxxxxxx - (password created for cn=admin,cn=config user)
# Schemas, in order
include /media/ephemeral0/openldap/etc/openldap/schema/core.schema
include /media/ephemeral0/openldap/etc/openldap/schema/cosine.schema
include /media/ephemeral0/openldap/etc/openldap/schema/inetorgperson.schema
include /media/ephemeral0/openldap/etc/openldap/schema/collective.schema
include /media/ephemeral0/openldap/etc/openldap/schema/corba.schema
include /media/ephemeral0/openldap/etc/openldap/schema/duaconf.schema
include /media/ephemeral0/openldap/etc/openldap/schema/dyngroup.schema
include /media/ephemeral0/openldap/etc/openldap/schema/misc.schema
include /media/ephemeral0/openldap/etc/openldap/schema/nis.schema
include /media/ephemeral0/openldap/etc/openldap/schema/openldap.schema
include /media/ephemeral0/openldap/etc/openldap/schema/ppolicy.schema
include /media/ephemeral0/openldap/etc/openldap/schema/bind-dyndb-ldap.schema

Create the slapd.d directory under /media/ephemeral0/openldap/etc/openldap/.  This will contain all the directory information:

# sudo chown -R openldap:openldap /media/ephemeral0/openldap/* 
# su - openldap -c "mkdir /media/ephemeral0/openldap/etc/openldap/slapd.d"

Populate the slapd.d directory with the base configuration by running the following command:

su - openldap -c "/media/ephemeral0/openldap/sbin/slaptest \
-f /media/ephemeral0/openldap/etc/openldap/slapd.conf \
-F /media/ephemeral0/openldap/etc/openldap/slapd.d"
config file testing succeeded

For this example, we will be setting up the directory to use dc=eucalyptus,dc=com as the LDAP base.  Create the cn=Directory Manager,dc=eucalyptus,dc=com LDAP password:

# /media/ephemeral0/openldap/sbin/slappasswd -h {SSHA}

Create an LDIF that will define the configuration of the DB associated with the information regarding the DNS entries.  For this example, the LDIF will be called directory-layout.ldif.  It should look like the following:

#######################################################################
# MDB database definitions
#######################################################################
#
dn: olcDatabase=mdb,cn=config
changetype: add
objectClass: olcDatabaseConfig
objectClass: olcMdbConfig
olcDatabase: mdb
olcSuffix: dc=eucalyptus,dc=com
olcRootDN: cn=Directory Manager,dc=eucalyptus,dc=com
olcRootPW: {SSHA}xxxx - (password of cn=Directory Manager,dc=eucalyptus,dc=com user)
olcDbDirectory: /media/ephemeral0/openldap/var/openldap-data/dns
olcDbIndex: objectClass eq
olcAccess: to attrs=userPassword by dn="cn=Directory Manager,dc=eucalyptus,dc=com"
 write by anonymous auth by self write by * none
olcAccess: to attrs=shadowLastChange by self write by * read
olcAccess: to dn.base="" by * read
olcAccess: to * by dn="cn=Directory Manager,dc=eucalyptus,dc=com" write by * read
olcDbMaxReaders: 0
olcDbMode: 0600
olcDbSearchStack: 16
olcDbMaxSize: 4294967296
olcAddContentAcl: FALSE
olcLastMod: TRUE
olcMaxDerefDepth: 15
olcReadOnly: FALSE
olcSyncUseSubentry: FALSE
olcMonitoring: TRUE
olcDbNoSync: FALSE
olcDbEnvFlags: writemap
olcDbEnvFlags: nometasync

Make sure and create the directory where the DB information will be stored:

# su - openldap -c "mkdir /media/ephemeral0/openldap/var/openldap-data/dns"

Start up the OpenLDAP directory:

# sudo /media/ephemeral0/openldap/libexec/slapd -h "ldap:/// ldapi:///" \
-u openldap -g openldap

After OpenLDAP has been started successfully,  upload the directory-layout.ldif as the cn=admin,cn=config user:

# /media/ephemeral0/openldap/bin/ldapadd -D cn=admin,cn=config -W \
-f directory-layout.ldif
Enter LDAP Password:
adding new entry "olcDatabase=mdb,cn=config"

To allow search access to the directory, create an LDIF called frontend.ldif, that contains the following:

dn: olcDatabase={-1}frontend,cn=config
changetype: modify
replace: olcAccess
olcAccess: to dn.base="" by * read
olcAccess: to dn.base="cn=Subschema" by * read
olcAccess: to * by self write by users read by anonymous auth

Upload the LDIF using the ldapmodify command:

# /media/ephemeral0/openldap/bin/ldapmodify -D cn=admin,cn=config -W -f frontend.ldif
Enter LDAP Password:
modifying entry "olcDatabase={-1}frontend,cn=config"

To check the results of these changes, use ldapsearch:

# /media/ephemeral0/openldap/bin/ldapsearch -D cn=admin,cn=config -W -b cn=config
Enter LDAP Password:
(ldapsearch results...)

After confirming that the base configurations have been stored, create an LDIF called dns-domain.ldif that lays out the directory structure for the database.  As seen previously in the directory-layout.ldif, the base is dc=eucalyptus,dc=com. The dns-domain.ldif file should look like the following:

dn: dc=eucalyptus,dc=com
objectClass: top
objectClass: dcObject
objectclass: organization
o: Eucalyptus Systems Inc - QA DNS Domain
dc: eucalyptus
description: Test LDAP+DNS Setup
 
dn: ou=dns,dc=eucalyptus,dc=com
objectClass: organizationalUnit
ou: dns

After creating the dns-domain.ldif file, upload the file using the cn=Directory Manager,dc=eucalyptus,dc=com user:

# /media/ephemeral0/openldap/bin/ldapadd -H ldap://localhost \
-D "cn=Directory Manager,dc=eucalyptus,dc=com" -W -f dns-domain.ldif
Enter LDAP Password:
adding new entry "dc=eucalyptus,dc=com"
adding new entry "ou=dns,dc=eucalyptus,dc=com"

To allow the cn=Directory Manager,dc=eucalyptus,dc=com user to see what updates are being done to the Directory, enable the Access Log overlay.  To enable this option, create an LDIF file called access-log.ldif.  The contents should look like the following:

dn: olcDatabase={2}mdb,cn=config
objectClass: olcDatabaseConfig
objectClass: olcMdbConfig
olcDatabase: {2}mdb
olcDbDirectory: /media/ephemeral0/openldap/var/openldap-data/access
olcSuffix: cn=log
olcDbIndex: reqStart eq
olcDbMaxSize: 1073741824
olcDbMode: 0600
olcAccess: {1}to * by dn="cn=Directory Manager,dc=eucalyptus,dc=com" read

dn: olcOverlay={1}accesslog,olcDatabase={3}mdb,cn=config
objectClass: olcOverlayConfig
objectClass: olcAccessLogConfig
olcOverlay: {1}accesslog
olcAccessLogDB: cn=log
olcAccessLogOps: all
olcAccessLogPurge: 7+00:00 1+00:00
olcAccessLogSuccess: TRUE
olcAccessLogOld: (objectclass=idnsRecord)

After creating the access-log.ldif, create the directory for storing the access database:

# su - openldap -c "mkdir /media/ephemeral0/openldap/var/openldap-data/access"

Upload the LDIF using the cn=admin,cn=config user:

# /media/ephemeral0/openldap/bin/ldapadd -D cn=admin,cn=config -W -f access-log.ldif

Now that OpenLDAP is ready to go, let’s work on configuring Bind9 DNS.

Bind9 DNS Configuration

Since bind-dyndb-ldap has a dependency package of bind,  named has already been installed on the instance.  The only thing left to do is edit /etc/named.conf so that we are able to use the dynamic ldap backend module.  Edit /etc/named.conf so that it looks like the following:

options {
 listen-on port 53 { <private IP address of the instance>; };
 listen-on-v6 port 53 { ::1; };
 directory "/var/named";
 dump-file "/var/named/data/cache_dump.db";
 statistics-file "/var/named/data/named_stats.txt";
 memstatistics-file "/var/named/data/named_mem_stats.txt";
 recursion yes;

 dnssec-enable yes;
 dnssec-validation yes;
 dnssec-lookaside auto;

/* Path to ISC DLV key */
 bindkeys-file "/etc/named.iscdlv.key";
managed-keys-directory "/var/named/dynamic";
 allow-recursion { any; };
};

dynamic-db "qa_dns_test" {
 library "ldap.so";
 arg "uri ldap://localhost";
 arg "base ou=dns,dc=eucalyptus, dc=com";
 arg "auth_method none";
 arg "cache_ttl 10";
 arg "zone_refresh 1";
 arg "dyn_update yes";
};

logging {
 channel default_debug {
 file "data/named.run";
 severity debug;
 print-time yes;
 };
};

zone "." IN {
 type hint;
 file "named.ca";
};

include "/etc/named.rfc1912.zones";
include "/etc/named.root.key";

As seen above, the dynamic-db section is the configuration for connecting to the OpenLDAP server.  For more advanced configurations, please reference the README in the bind-dyndb-ldap repository on git.fedorahosted.org.

Now we are ready to start named (the bind DNS server).  Before starting the server, make sure and create the rndc key, then start named:

# rndc-confgen -a -r /dev/urandom
# service named start

You have successfully created an Bind9 DNS + OpenLDAP deployment.  Let’s run a quick test.

Test the Deployment

To test the deployment, I created an LDIF called test-cloud.ldif.  The configuration sets up a domain called eucalyptus-systems.com.  It also creates a sub-domain that will be forwarding requests for euca-hasp.eucalyptus-systems.com to the CLCs of the Eucalyptus HA deployment that has been set up.  The contents of the file are as follows:

dn: idnsName=eucalyptus-systems.com,ou=dns,dc=eucalyptus,dc=com
objectClass: top
objectClass: idnsZone
objectClass: idnsRecord
idnsName: eucalyptus-systems.com
idnsUpdatePolicy: grant EUCALYPTUS-SYSTEMS.COM krb5-self * A;
idnsZoneActive: TRUE
idnsSOAmName: server.eucalyptus-systems.com
idnsSOArName: root.server.eucalyptus-systems.com
idnsAllowQuery: any;
idnsAllowDynUpdate: TRUE
idnsSOAserial: 1
idnsSOArefresh: 10800
idnsSOAretry: 900
idnsSOAexpire: 604800
idnsSOAminimum: 86400
NSRecord: ns
ARecord: 192.168.55.103 - (the public IP of the instance)

dn: idnsName=ns,idnsName=eucalyptus-systems.com,ou=dns,dc=eucalyptus,dc=com
objectClass: idnsRecord
objectClass: top
idnsName: ns
aRecord: 192.168.55.103

dn: idnsName=server,idnsName=eucalyptus-systems.com,ou=dns,dc=eucalyptus,dc=com
objectClass: idnsRecord
objectClass: top
idnsName: server
CNAMERecord: eucalyptus-systems.com.

dn: idnsname=39.168.192.in-addr.arpa.,ou=dns,dc=eucalyptus,dc=com
objectClass: idnszone
objectClass: idnsrecord
objectClass: top
idnsName: 39.168.192.in-addr.arpa.
idnsSOAmName: server.eucalyptus-systems.com
idnsSOArName: root.server.eucalyptus-systems.com
idnsSOAserial: 1350039556
idnsSOArefresh: 10800
idnsSOAretry: 900
idnsSOAexpire: 604800
idnsSOAminimum: 86400
idnsZoneActive: TRUE
idnsAllowDynUpdate: TRUE
idnsAllowQuery: any;
idnsAllowTransfer: none;
idnsUpdatePolicy: grant EUCALYPTUS-SYSTEMS.COM krb5-subdomain 39.168.192.in-addr.arpa. PTR;
nSRecord: server.eucalyptus-systems.com.

dn: idnsName=_ldap._tcp,idnsName=eucalyptus-systems.com,ou=dns,dc=eucalyptus,dc=com
objectClass: idnsRecord
objectClass: top
idnsName: _ldap._tcp
SRVRecord: 0 100 389 server

dn: idnsName=_ntp._udp,idnsName=eucalyptus-systems.com,ou=dns,dc=eucalyptus,dc=com
objectClass: idnsRecord
objectClass: top
idnsName: _ntp._udp
SRVRecord: 0 100 123 server

# The DNS entries for the CLCs of the cloud - viking-01 and viking-02

dn: idnsName=viking-02,idnsName=eucalyptus-systems.com,ou=dns,dc=eucalyptus,dc=com
objectClass: idnsRecord
objectClass: top
idnsName: viking-02
aRecord: 192.168.39.102

dn: idnsname=102,idnsname=39.168.192.in-addr.arpa.,ou=dns,dc=eucalyptus,dc=com
objectClass: idnsrecord
objectClass: top
idnsName: 102
pTRRecord: viking-02.eucalyptus-systems.com.

dn: idnsName=viking-01,idnsName=eucalyptus-systems.com,ou=dns,dc=eucalyptus,dc=com
objectClass: idnsRecord
objectClass: top
idnsName: viking-01
aRecord: 192.168.39.101

dn: idnsname=101,idnsname=39.168.192.in-addr.arpa.,ou=dns,dc=eucalyptus,dc=com
objectClass: idnsrecord
objectClass: top
idnsName: 101
pTRRecord: viking-01.eucalyptus-systems.com.

# The delegated zone - euca-hasp.eucalyptus-systems.com

dn: idnsName=euca-hasp,idnsName=eucalyptus-systems.com,ou=dns,dc=eucalyptus,dc=com
objectClass: top
objectClass: idnsRecord
objectClass: idnsZone
idnsForwardPolicy: first
idnsAllowDynUpdate: FALSE
idnsZoneActive: TRUE
idnsAllowQuery: any;
idnsForwarders: 192.168.39.101
idnsForwarders: 192.168.39.102
idnsName: euca-hasp
idnsSOAmName: server.eucalyptus-systems.com
idnsSOArName: root.server.eucalyptus-systems.com
idnsSOAretry: 15
idnsSOAserial: 1
idnsSOArefresh: 80
idnsSOAexpire: 120
idnsSOAminimum: 30
nSRecord: viking-01.eucalyptus-systems.com
nSRecord: viking-02.eucalyptus-systems.com

Since this was intended for a lab environment where sub-domains (and possibly domains) would be added/deleted on a regular basis, the SOA records were not set to the RFC 1912 standards defined for production DNS use.

After creating this LDIF,  it was uploaded to the LDAP server as the cn=Directory Manager,dc=eucalyptus,dc=com user:

# /media/ephemeral0/openldap/bin/ldapadd -H ldap://localhost \
-D "cn=Directory Manager,dc=eucalyptus,dc=com" -W -f test-cloud.ldif
Enter LDAP Password:
adding new entry "idnsName=eucalyptus-systems.com,ou=dns,dc=eucalyptus,dc=com"
adding new entry "idnsName=ns,idnsName=eucalyptus-systems.com,ou=dns,dc=eucalyptus,dc=com"
adding new entry "idnsName=server,idnsName=eucalyptus-systems.com,ou=dns,dc=eucalyptus,dc=com"
adding new entry "idnsname=39.168.192.in-addr.arpa.,ou=dns,dc=eucalyptus,dc=com"
adding new entry "idnsName=_ldap._tcp,idnsName=eucalyptus-systems.com,ou=dns,dc=eucalyptus,dc=com"
adding new entry "idnsName=_ntp._udp,idnsName=eucalyptus-systems.com,ou=dns,dc=eucalyptus,dc=com"
adding new entry "idnsName=viking-02,idnsName=eucalyptus-systems.com,ou=dns,dc=eucalyptus,dc=com"
adding new entry "idnsname=102,idnsname=39.168.192.in-addr.arpa.,ou=dns,dc=eucalyptus,dc=com"
adding new entry "idnsName=viking-01,idnsName=eucalyptus-systems.com,ou=dns,dc=eucalyptus,dc=com"
adding new entry "idnsname=101,idnsname=39.168.192.in-addr.arpa.,ou=dns,dc=eucalyptus,dc=com"
adding new entry "idnsName=euca-hasp,idnsName=eucalyptus-systems.com,ou=dns,dc=eucalyptus,dc=com"

To test out the setup, tests were ran against the public IP address of the instance to resolve for the various configurations:

# nslookup viking-01.eucalyptus-systems.com 192.168.55.103
Server: 192.168.55.103
Address: 192.168.55.103#53

Name: viking-01.eucalyptus-systems.com
Address: 192.168.39.101

# nslookup 192.168.39.101 192.168.55.103
Server: 192.168.55.103
Address: 192.168.55.103#53

101.39.168.192.in-addr.arpa name = viking-01.eucalyptus-systems.com.

# nslookup eucalyptus.euca-hasp.eucalyptus-systems.com 192.168.55.103
Server: 192.168.55.103
Address: 192.168.55.103#53

Non-authoritative answer:
Name: eucalyptus.euca-hasp.eucalyptus-systems.com
Address: 192.168.39.102

# nslookup walrus.euca-hasp.eucalyptus-systems.com 192.168.55.103
Server: 192.168.55.103
Address: 192.168.55.103#53

Non-authoritative answer:
Name: walrus.euca-hasp.eucalyptus-systems.com
Address: 192.168.39.101

As see above, not only did the resolution come back correct for the machines under the eucalyptus-systems.com domain, but it also forwarded the requests for the hosts under euca-hasp.eucalyptus-systems.com correctly, and returned the correct response.

To delete the set up, an LDIF called delete-test-cloud.ldif from the test-cloud.ldif as follows:

# tac test-cloud.ldif | grep dn: > delete-test-cloud.ldif

Open up the delete-test-cloud.ldif and add the following lines between each dn:

changetype: delete
(empty line)

Now, use ldapmodify as the cn=Directory Manager,dc=eucalyptus,dc=com user to delete the entries:

# /media/ephemeral0/openldap/bin/ldapmodify -H ldap://localhost -D "cn=Directory Manager,dc=eucalyptus,dc=com" -W -f delete-test-cloud.ldif
Enter LDAP Password:
deleting entry "idnsName=euca-hasp,idnsName=eucalyptus-systems.com,ou=dns,dc=eucalyptus,dc=com"
deleting entry "idnsname=101,idnsname=39.168.192.in-addr.arpa.,ou=dns,dc=eucalyptus,dc=com"
deleting entry "idnsName=viking-01,idnsName=eucalyptus-systems.com,ou=dns,dc=eucalyptus,dc=com"
deleting entry "idnsname=102,idnsname=39.168.192.in-addr.arpa.,ou=dns,dc=eucalyptus,dc=com"
deleting entry "idnsName=viking-02,idnsName=eucalyptus-systems.com,ou=dns,dc=eucalyptus,dc=com"
deleting entry "idnsName=_ntp._udp,idnsName=eucalyptus-systems.com,ou=dns,dc=eucalyptus,dc=com"
deleting entry "idnsName=_ldap._tcp,idnsName=eucalyptus-systems.com,ou=dns,dc=eucalyptus,dc=com"
deleting entry "idnsname=39.168.192.in-addr.arpa.,ou=dns,dc=eucalyptus,dc=com"
deleting entry "idnsName=server,idnsName=eucalyptus-systems.com,ou=dns,dc=eucalyptus,dc=com"
deleting entry "idnsName=ns,idnsName=eucalyptus-systems.com,ou=dns,dc=eucalyptus,dc=com"
deleting entry "idnsName=eucalyptus-systems.com,ou=dns,dc=eucalyptus,dc=com"

To confirm, do a lookup against one of the entries to see if it still exists:

# nslookup viking-01.eucalyptus-systems.com 192.168.55.103
Server: 192.168.55.103
Address: 192.168.55.103#53

** server can't find viking-01.eucalyptus-systems.com: NXDOMAIN

There you have it!  A successful Bind9 DNS + OpenLDAP deployment is ready to be used.

Enjoy!  And as always, questions/suggestions/comments are always welcome.

Bind DNS + OpenLDAP MDB == Dynamic Domain and Fully Delegated Sub-Domain Configuration of DNS