Background
I frequently check out Dustin Kirkland’s blog to get ideas about how to secure instances running on various cloud infrastructures. Recently, I stumbled across one of his blog entries, where he discussed a package called ‘overlayroot‘, which is part of the cloud-initramfs-tools package for Ubuntu. The really interesting feature I liked about this package is the ability to encrypt – using dmcrypt – the root filesystem of the instance. What this means that if the Node Controller (NC) that hosts the instance happens to become compromised, the information on the device associated with the instance can’t be accessed. In addition, overlayroot allows the root filesystem of the instance to be ‘laid’ on top of another device that is bigger in size. The advantage here is that there isn’t a need to re-create an image with a larger root filesystem.
Prerequisites
The key prerequisites for this blog were mentioned in my previous blog, which discusses how to bundle, upload and register a CoreOS EMI. In addition, to these prerequisites, the following EC2 actions are needed for the Eucalyptus IAM policy:
Overlayroot is available with any Ubuntu Cloud image from 12.10 (Quantal Quetzal) to the most recent at the time of this blog, 14.10 (Utopic Unicorn). For this blog, Ubuntu 14.04 LTS (Trusty Tahr) will be used.
Setting Up the EMI
As mentioned in the overlayroot blog entry, the configuration of overlayroot is located in /etc/overlayroot.conf. This can be configured before this Ubuntu Cloud image is bundled, uploaded and registered as an EMI, or after. For brevity, overlayroot will be configured after the Ubuntu Cloud image has been bundled, uploaded and registered.
To get started, download the Ubuntu Cloud image:
# wget http://cloud-images.ubuntu.com/trusty/current/trusty-server-cloudimg-amd64-disk1.img
Next, convert the image from qcow2 to raw:
# qemu-img convert -O raw trusty-server-cloudimg-amd64-disk1.img trusty-server-cloudimg-amd64-disk1.raw
Bundle, upload and register the image as an instance store-backed HVM EMI:
# euca-bundle-and-upload-image -i trusty-server-cloudimg-amd64-disk1.raw -b trusty-server-hvm -r x86_64 # euca-register -n trusty-server-hvm trusty-server-hvm/trusty-server-cloudimg-amd64-disk1.raw.manifest.xml --virtualization-type hvm IMAGE emi-348861E4
After creating a keypair using euca-create-keypair, and authorizing SSH access using euca-authorize, launch an instance from the EMI:
# euca-run-instances -k account1-user01 -t m1.medium emi-348861E4 RESERVATION r-BE317660 408396244283 default INSTANCE i-41DA6A90 emi-348861E4 pending account1-user01 0 m1.medium 2014-06-15T22:47:03.552Z ViciousLiesAndDangerousRumors monitoring-disabled 0.0.0.0 0.0.0.0 instance-store hvm sg-A5133B59
Once the instance has reached a ‘running’ state, SSH into the instance:
# euca-describe-instances i-41DA6A90 RESERVATION r-BE317660 408396244283 default INSTANCE i-41DA6A90 emi-348861E4 euca-10-104-6-235.bigboi.acme.eucalyptus-systems.com euca-172-18-238-170.bigboi.internal running account1-user01 0 m1.medium 2014-06-15T22:47:03.552Z ViciousLiesAndDangerousRumors monitoring-disabled 10.104.6.235 172.18.238.170 instance-store hvm sg-A5133B59
# ssh -i account1-user01/account1-user01.priv ubuntu@euca-10-104-6-235.bigboi.acme.eucalyptus-systems.com
Encrypted Root Filesystem
Once inside the instance, since the EMI is an instance store-backed HVM EMI, the ephemeral disk is /dev/vdb:
ubuntu@euca-172-18-238-170:~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 253:0 0 2.2G 0 disk └─vda1 253:1 0 2.2G 0 part / vdb 253:16 0 7.8G 0 disk
Edit the ‘overlayroot’ option in the /etc/overlayroot.conf to set /dev/vdb to used to encrypt the root filesystem:
ubuntu@euca-172-18-238-170:~$ sudo vi /etc/overlayroot.conf ...... overlayroot=”crypt:dev=/dev/vdb”
After editing /etc/overlayroot.conf, reboot the instance:
ubuntu@euca-172-18-238-170:~$ sudo reboot
After the instance has been rebooted, SSH back into the instance and observe the new layout of the filesystem:
ubuntu@euca-172-18-238-170:~$ mount overlayroot on / type overlayfs (rw,lowerdir=/media/root-ro/,upperdir=/media/root-rw/overlay) proc on /proc type proc (rw,noexec,nosuid,nodev) sysfs on /sys type sysfs (rw,noexec,nosuid,nodev) none on /sys/fs/cgroup type tmpfs (rw) none on /sys/fs/fuse/connections type fusectl (rw) none on /sys/kernel/debug type debugfs (rw) none on /sys/kernel/security type securityfs (rw) udev on /dev type devtmpfs (rw,mode=0755) devpts on /dev/pts type devpts (rw,noexec,nosuid,gid=5,mode=0620) tmpfs on /run type tmpfs (rw,noexec,nosuid,size=10%,mode=0755) none on /run/lock type tmpfs (rw,noexec,nosuid,nodev,size=5242880) none on /run/shm type tmpfs (rw,nosuid,nodev) none on /run/user type tmpfs (rw,noexec,nosuid,nodev,size=104857600,mode=0755) /dev/vda1 on /media/root-ro type ext4 (ro) /dev/mapper/secure on /media/root-rw type ext4 (rw,relatime,data=ordered) none on /sys/fs/pstore type pstore (rw) systemd on /sys/fs/cgroup/systemd type cgroup (rw,noexec,nosuid,nodev,none,name=systemd)
To verify the encrypted filesystem, use cryptsetup:
ubuntu@euca-172-18-238-170:~$ sudo cryptsetup luksDump /dev/vdb LUKS header information for /dev/vdb
Version: 1 Cipher name: aes Cipher mode: xts-plain64 Hash spec: sha1 Payload offset: 4096 MK bits: 256 MK digest: 77 70 97 81 9e d6 56 4b c5 79 60 92 23 02 18 80 9c eb 40 c8 MK salt: ed d1 69 1e d7 49 a6 a6 91 fb 0f 44 3a a2 0b b2 2a 56 fb 82 77 3f 70 9c 70 ff c2 15 37 09 10 2c MK iterations: 33250 UUID: 164026af-f6fa-4dd4-8042-5cd24b8c00c9
Key Slot 0: ENABLED Iterations: 132641 Salt: 5e 7f 03 33 d9 af e8 a9 37 fb 5c 4e 10 db c5 38 f7 9f 49 12 d8 c4 43 8c cb 79 4a 25 da 04 c9 73 Key material offset: 8 AF stripes: 4000 Key Slot 1: DISABLED Key Slot 2: DISABLED Key Slot 3: DISABLED Key Slot 4: DISABLED Key Slot 5: DISABLED Key Slot 6: DISABLED Key Slot 7: DISABLED
Expand and Encrypt the Root Filesystem
If the cloud user wants to increase the size of the root filesystem and encrypt it, but use a larger size than available for ephemeral storage, the user can create a volume, attach it to the instance, then configure overlayroot to use that device. For example:
# euca-create-volume -s 20 -z ViciousLiesAndDangerousRumors
# euca-describe-volumes vol-450936D7 VOLUME vol-450936D7 20 ViciousLiesAndDangerousRumors available 2014-06-16T19:25:36.936Z standard
# euca-attach-volume -i i-41DA6A90 -d /dev/sdf vol-450936D7 ATTACHMENT vol-450936D7 i-41DA6A90 /dev/sdf attaching 2014-06-16T19:30:29.605Z
# euca-describe-instances i-41DA6A90 RESERVATION r-BE317660 408396244283 default INSTANCE i-41DA6A90 emi-348861E4 euca-10-104-6-235.bigboi.acme.eucalyptus-systems.com euca-172-18-238-170.bigboi.internal running account1-user01 0 m1.medium 2014-06-15T22:47:03.552Z ViciousLiesAndDangerousRumors monitoring-disabled 10.104.6.235 172.18.238.170 instance-store hvm sg-A5133B59 BLOCKDEVICE /dev/sdf vol-450936D7 2014-06-16T19:30:29.619Z false
Follow the steps above regarding editing the /etc/overlayroot.conf, except use /dev/vdc instead of /dev/vdb:
# ssh -i account1-user01/account1-user01.priv ubuntu@euca-10-104-6-235.bigboi.acme.eucalyptus-systems.com
ubuntu@euca-172-18-238-170:~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 253:0 0 2.2G 0 disk └─vda1 253:1 0 2.2G 0 part / vdb 253:16 0 7.8G 0 disk vdc 253:32 0 20G 0 disk
ubuntu@euca-172-18-238-170:~$ sudo vi /etc/overlayroot.conf ..... overlayroot=”crypt:dev=/dev/vdc”
Reboot the instance. After the reboot completes, SSH back into the instance and noticed the root filesystem increased in size, and encrypted:
ubuntu@euca-172-18-238-170:~$ sudo reboot
# ssh -i account1-user01/account1-user01.priv ubuntu@euca-10-104-6-235.bigboi.acme.eucalyptus-systems.com
ubuntu@euca-172-18-238-170:~$ lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 253:0 0 2.2G 0 disk └─vda1 253:1 0 2.2G 0 part /media/root-ro vdb 253:16 0 7.8G 0 disk vdc 253:32 0 20G 0 disk └─secure (dm-0) 252:0 0 20G 0 crypt /media/root-rw
ubuntu@euca-172-18-238-170:~$ df -ah Filesystem Size Used Avail Use% Mounted on overlayroot 20G 46M 19G 1% / proc 0 0 0 - /proc sysfs 0 0 0 - /sys none 4.0K 0 4.0K 0% /sys/fs/cgroup none 0 0 0 - /sys/fs/fuse/connections none 0 0 0 - /sys/kernel/debug none 0 0 0 - /sys/kernel/security udev 493M 8.0K 493M 1% /dev devpts 0 0 0 - /dev/pts tmpfs 100M 328K 100M 1% /run none 5.0M 0 5.0M 0% /run/lock none 498M 0 498M 0% /run/shm none 100M 0 100M 0% /run/user /dev/vda1 2.2G 754M 1.3G 37% /media/root-ro /dev/mapper/secure 20G 46M 19G 1% /media/root-rw none 0 0 0 - /sys/fs/pstore systemd 0 0 0 - /sys/fs/cgroup/systemd
Bonus – Configuring Overlayroot Before Bundle, Uploading and Registering the EMI
As mention before, overlayroot can be configured before the image has been bundled, uploaded and registered as an instance store-backed HVM EMI. After downloading the Ubuntu Trusty cloud image, use losetup and kpartx to aid in mounting the image in order to configure overlayroot:
# qemu-img convert -O raw trusty-server-cloudimg-amd64-disk1.img trusty-server-cloudimg-amd64-disk1.raw
# losetup /dev/loop0 trusty-server-cloudimg-amd64-disk1.raw
# kpartx -av /dev/loop0 add map loop0p1 (253:2): 0 4192256 linear /dev/loop0 2048
# mkdir /mnt/ubuntu
# mount /dev/mapper/loop0p1 /mnt/ubuntu
# chroot /mnt/ubuntu
root@odc-f-13:/# vi /etc/overlayroot.conf
(added the following)
overlayroot=”crypt:dev=/dev/vdb”
root@odc-f-13:/# exit exit
# umount /mnt/ubuntu
# kpartx -dv /dev/loop0
# losetup -d /dev/loop0
After unmounting the image, just bundle, upload and register the image as an instance store-backed HVM EMI. When an instance is launched from the EMI, the root filesystem will automatically be encrypted.
Conclusion
Using overlayroot gives additional instance security for the cloud user. For more information about the additional configuration options of overlayroot, please refer to the overlayroot.conf file in the cloud-initramfs-tools repository on Launchpad.
Enjoy!
More Mind Spew-age from Harold Spencer Jr. 