Background

I frequently check out Dustin Kirkland’s blog to get ideas about how to secure instances running on various cloud infrastructures.  Recently, I stumbled across one of his blog entries, where he discussed a package called ‘overlayroot‘, which is part of the cloud-initramfs-tools package for Ubuntu.  The really interesting feature I liked about this package is the ability to encrypt – using dmcrypt –  the root filesystem of the instance.  What this means that if the Node Controller (NC) that hosts the instance happens to become compromised, the information on the device associated with the instance can’t be accessed.  In addition, overlayroot allows the root filesystem of the instance to be ‘laid’ on top of another device that is bigger in size.  The advantage here is that there isn’t a need to re-create an image with a larger root filesystem.

Prerequisites

The key prerequisites for this blog were mentioned in my previous blog, which discusses how to bundle, upload and register a CoreOS EMI.  In addition, to these prerequisites, the following EC2 actions are needed for the Eucalyptus IAM policy:

• AttachVolume
• CreateVolume

Overlayroot is available with any Ubuntu Cloud image from 12.10 (Quantal Quetzal) to the most recent at the time of this blog, 14.10 (Utopic Unicorn).  For this blog, Ubuntu 14.04 LTS (Trusty Tahr) will be used.

Setting Up the EMI

As mentioned in the overlayroot blog entry, the configuration of overlayroot is located in /etc/overlayroot.conf.  This can be configured before this Ubuntu Cloud image is bundled, uploaded and registered as an EMI, or after.  For brevity, overlayroot will be configured after the Ubuntu Cloud image has been bundled, uploaded and registered.

To get started, download the Ubuntu Cloud image:

# wget http://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img

Next, convert the image from qcow2 to raw:

# qemu-img convert -O raw trusty-server-cloudimg-amd64-disk1.img trusty-server-cloudimg-amd64-disk1.raw

Bundle, upload and register the image as an instance store-backed HVM EMI:

# euca-bundle-and-upload-image -i trusty-server-cloudimg-amd64-disk1.raw -b trusty-server-hvm -r x86_64
# euca-register -n trusty-server-hvm trusty-server-hvm/trusty-server-cloudimg-amd64-disk1.raw.manifest.xml --virtualization-type hvm 
IMAGE emi-348861E4

After creating a keypair using euca-create-keypair, and authorizing SSH access using euca-authorize, launch an instance from the EMI:

# euca-run-instances -k account1-user01 -t m1.medium emi-348861E4
RESERVATION r-BE317660 408396244283 default
INSTANCE i-41DA6A90 emi-348861E4 pending account1-user01 0 m1.medium 2014-06-15T22:47:03.552Z ViciousLiesAndDangerousRumors monitoring-disabled 0.0.0.0 0.0.0.0 instance-store hvm sg-A5133B59

Once the instance has reached a ‘running’ state, SSH into the instance:

# euca-describe-instances i-41DA6A90 
RESERVATION r-BE317660 408396244283 default
INSTANCE i-41DA6A90 emi-348861E4 euca-10-104-6-235.bigboi.acme.eucalyptus-systems.com euca-172-18-238-170.bigboi.internal running account1-user01 0 m1.medium 2014-06-15T22:47:03.552Z ViciousLiesAndDangerousRumors monitoring-disabled 10.104.6.235 172.18.238.170 instance-store hvm sg-A5133B59

# ssh -i account1-user01/account1-user01.priv ubuntu@euca-10-104-6-235.bigboi.acme.eucalyptus-systems.com

Encrypted Root Filesystem

Once inside the instance, since the EMI is an instance store-backed HVM EMI, the ephemeral disk is /dev/vdb:

ubuntu@euca-172-18-238-170:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 2.2G 0 disk
└─vda1 253:1 0 2.2G 0 part /
vdb 253:16 0 7.8G 0 disk

Edit the ‘overlayroot’ option in the /etc/overlayroot.conf to set /dev/vdb to used to encrypt the root filesystem:

ubuntu@euca-172-18-238-170:~$ sudo vi /etc/overlayroot.conf
......
overlayroot=”crypt:dev=/dev/vdb”

After editing /etc/overlayroot.conf, reboot the instance:

ubuntu@euca-172-18-238-170:~$ sudo reboot

After the instance has been rebooted, SSH back into the instance and observe the new layout of the filesystem:

ubuntu@euca-172-18-238-170:~$ mount
overlayroot on / type overlayfs (rw,lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay)
proc on /proc type proc (rw,noexec,nosuid,nodev)
sysfs on /sys type sysfs (rw,noexec,nosuid,nodev)
none on /sys/fs/cgroup type tmpfs (rw)
none on /sys/fs/fuse/connections type fusectl (rw)
none on /sys/kernel/debug type debugfs (rw)
none on /sys/kernel/security type securityfs (rw)
udev on /dev type devtmpfs (rw,mode=0755)
devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620)
tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755)
none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880)
none on /run/shm type tmpfs (rw,nosuid,nodev)
none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755)
/dev/vda1 on /media/root-ro type ext4 (ro)
/dev/mapper/secure on /media/root-rw type ext4 (rw,relatime,data=ordered)
none on /sys/fs/pstore type pstore (rw)
systemd on /sys/fs/cgroup/systemd type cgroup (rw,noexec,nosuid,nodev,none,name=systemd)

To verify the encrypted filesystem, use cryptsetup:

ubuntu@euca-172-18-238-170:~$ sudo cryptsetup luksDump /dev/vdb
LUKS header information for /dev/vdb

Version: 1
Cipher name: aes
Cipher mode: xts-plain64
Hash spec: sha1
Payload offset: 4096
MK bits: 256
MK digest: 77 70 97 81 9e d6 56 4b c5 79 60 92 23 02 18 80 9c eb 40 c8
MK salt: ed d1 69 1e d7 49 a6 a6 91 fb 0f 44 3a a2 0b b2
 2a 56 fb 82 77 3f 70 9c 70 ff c2 15 37 09 10 2c
MK iterations: 33250
UUID: 164026af-f6fa-4dd4-8042-5cd24b8c00c9

Key Slot 0: ENABLED
 Iterations: 132641
 Salt: 5e 7f 03 33 d9 af e8 a9 37 fb 5c 4e 10 db c5 38
 f7 9f 49 12 d8 c4 43 8c cb 79 4a 25 da 04 c9 73
 Key material offset: 8
 AF stripes: 4000
Key Slot 1: DISABLED
Key Slot 2: DISABLED
Key Slot 3: DISABLED
Key Slot 4: DISABLED
Key Slot 5: DISABLED
Key Slot 6: DISABLED
Key Slot 7: DISABLED

Expand and Encrypt the Root Filesystem

If the cloud user wants to increase the size of the root filesystem and encrypt it, but use a larger size than available for ephemeral storage, the user can create a volume, attach it to the instance, then configure overlayroot to use that device.  For example:

# euca-create-volume -s 20 -z ViciousLiesAndDangerousRumors

# euca-describe-volumes vol-450936D7
VOLUME vol-450936D7 20 ViciousLiesAndDangerousRumors available 2014-06-16T19:25:36.936Z standard

# euca-attach-volume -i i-41DA6A90 -d /dev/sdf vol-450936D7
ATTACHMENT vol-450936D7 i-41DA6A90 /dev/sdf attaching 2014-06-16T19:30:29.605Z

# euca-describe-instances i-41DA6A90
RESERVATION r-BE317660 408396244283 default
INSTANCE i-41DA6A90 emi-348861E4 euca-10-104-6-235.bigboi.acme.eucalyptus-systems.com euca-172-18-238-170.bigboi.internal running account1-user01 0 m1.medium 2014-06-15T22:47:03.552Z ViciousLiesAndDangerousRumors monitoring-disabled 10.104.6.235 172.18.238.170 instance-store hvm sg-A5133B59
BLOCKDEVICE /dev/sdf vol-450936D7 2014-06-16T19:30:29.619Z false

Follow the steps above regarding editing the /etc/overlayroot.conf, except use /dev/vdc instead of /dev/vdb:

# ssh -i account1-user01/account1-user01.priv ubuntu@euca-10-104-6-235.bigboi.acme.eucalyptus-systems.com

ubuntu@euca-172-18-238-170:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 2.2G 0 disk
└─vda1 253:1 0 2.2G 0 part /
vdb 253:16 0 7.8G 0 disk
vdc 253:32 0 20G 0 disk

ubuntu@euca-172-18-238-170:~$ sudo vi /etc/overlayroot.conf
.....
overlayroot=”crypt:dev=/dev/vdc”

Reboot the instance.  After the reboot completes, SSH back into the instance and noticed the root filesystem increased in size, and encrypted:

ubuntu@euca-172-18-238-170:~$ sudo reboot

# ssh -i account1-user01/account1-user01.priv ubuntu@euca-10-104-6-235.bigboi.acme.eucalyptus-systems.com

ubuntu@euca-172-18-238-170:~$ lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda 253:0 0 2.2G 0 disk
└─vda1 253:1 0 2.2G 0 part /media/root-ro
vdb 253:16 0 7.8G 0 disk
vdc 253:32 0 20G 0 disk
└─secure (dm-0) 252:0 0 20G 0 crypt /media/root-rw

ubuntu@euca-172-18-238-170:~$ df -ah
Filesystem Size Used Avail Use% Mounted on
overlayroot 20G 46M 19G 1% /
proc 0 0 0 - /proc
sysfs 0 0 0 - /sys
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 0 0 0 - /sys/fs/fuse/connections
none 0 0 0 - /sys/kernel/debug
none 0 0 0 - /sys/kernel/security
udev 493M 8.0K 493M 1% /dev
devpts 0 0 0 - /dev/pts
tmpfs 100M 328K 100M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 498M 0 498M 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/vda1 2.2G 754M 1.3G 37% /media/root-ro
/dev/mapper/secure 20G 46M 19G 1% /media/root-rw
none 0 0 0 - /sys/fs/pstore
systemd 0 0 0 - /sys/fs/cgroup/systemd

Bonus – Configuring Overlayroot Before Bundle, Uploading and Registering the EMI

As mention before,  overlayroot can be configured before the image has been bundled, uploaded and registered as an instance store-backed HVM EMI.  After downloading the Ubuntu Trusty cloud image, use losetup and kpartx to aid in mounting the image in order to configure overlayroot:

# qemu-img convert -O raw trusty-server-cloudimg-amd64-disk1.img trusty-server-cloudimg-amd64-disk1.raw

# losetup /dev/loop0 trusty-server-cloudimg-amd64-disk1.raw

# kpartx -av /dev/loop0
add map loop0p1 (253:2): 0 4192256 linear /dev/loop0 2048

# mkdir /mnt/ubuntu

# mount /dev/mapper/loop0p1 /mnt/ubuntu

# chroot /mnt/ubuntu

root@odc-f-13:/# vi /etc/overlayroot.conf

(added the following)

overlayroot=”crypt:dev=/dev/vdb”

root@odc-f-13:/# exit
exit

# umount /mnt/ubuntu

# kpartx -dv /dev/loop0

# losetup -d /dev/loop0

After unmounting the image, just bundle, upload and register the image as an instance store-backed HVM EMI.  When an instance is launched from the EMI, the root filesystem will automatically be encrypted.

Conclusion

Using overlayroot gives additional instance security for the cloud user.  For more information about the additional configuration options of overlayroot, please refer to the overlayroot.conf file in the cloud-initramfs-tools repository on Launchpad.

Enjoy!