ZFS is a filesystem designed by Sun Microsystems that focuses on data integrity. What makes this such an attractive filesystem to use in the cloud is that a cloud user can easily do the following:
- set up an LVM + RAID filesystem for storing large amounts of data (e.g. database information)
- expand the filesystem by adding more storage (i.e. EBS volumes)
- backup the filesystem without taking the filesystem offline/unmounting
- restore the filesystem
This blog entry will focus on how a cloud user can create their own Eucalyptus Machine Image (EMI) that has ZFS support. The CentOS 6.5 EMI on the Eucalyptus Machine Image Catalog will be used as the base image.
Before Starting…
Before following the steps in this blog, make sure the following is in place:
- A fully deployed Eucalyptus 4.0.2 Cloud with Eucalyptus DNS enabled
- Latest version of euca2ools installed
- Cloud Administrator (admin user under ‘eucalyptus’ account) access or a cloud user that has at a minimum the following IAM access policy:
Once these requirements have been met, everything should be ready to go.
Set Up Base Image/Instance
To begin, follow the ‘Quick Start’ instructions mentioned on the Eucalyptus Machine Image Catalog page. This will install all the images provided by the catalog. When the process has finished, list the CentOS 6.5 EMI. For example:
# euca-describe-images emi-bdcec010 IMAGE emi-bdcec010 centos-6.5-x86_64-20140917/centos.raw.manifest.xml 094999295155 available public x86_64 machine instance-store hvm
Once the CentOS 6.5 EMI has been listed, launch an instance from the EMI. For example:
# euca-run-instances -k account2-user11 -t m1.medium emi-bdcec010 RESERVATION r-a22f0201 325271821652 default INSTANCE i-b9fccf9f emi-bdcec010 pending account2-user11 0 m1.medium 2014-12-03T22:52:41.522Z Honest monitoring-disabled 0.0.0.0 0.0.0.0 instance-store hvm sg-6ef9907f x86_64
# euca-describe-instances i-b9fccf9f RESERVATION r-a22f0201 325271821652 default INSTANCE i-b9fccf9f emi-bdcec010 euca-10-104-7-15.future.future.euca-hasp.cs.prc.eucalyptus-systems.com euca-172-17-248-178.future.internal running account2-user11 0 m1.medium 2014-12-03T22:52:41.522Z Honest monitoring-disabled 10.104.7.15 172.17.248.178 instance-store hvm sg-6ef9907f x86_64
Once the instance is running, its ready to be customized.
Adding ZFS Support to the Instance
Now that the instance is running, SSH into the instance so the following ZFS repository can be added:
[root@odc-f-13 ~]# ssh -i account2-user11.priv root@euca-10-104-7-15.future.future.euca-hasp.cs.prc.eucalyptus-systems.com [root@euca-172-17-248-178 ~]# yum localinstall --nogpgcheck https://download.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm [root@euca-172-17-248-178 ~]# yum localinstall --nogpgcheck http://archive.zfsonlinux.org/epel/zfs-release.el6.noarch.rpm [root@euca-172-17-248-178 ~]# yum upgrade -y [root@euca-172-17-248-178 ~]# yum install kernel-devel zfs -y
After all the packages have been installed, reboot the instance:
[root@euca-172-17-248-178 ~]# reboot
Preparing the Instance For EMI Creation
After rebooting the instance, SSH back into the instance and prepare the instance for EMI creation. First, load the zfs module:
[root@odc-f-13 ~]# ssh -i account2-user11.priv root@euca-10-104-7-15.future.future.euca-hasp.cs.prc.eucalyptus-systems.com [root@euca-172-17-248-178 ~]# modprobe zfs [root@euca-172-17-248-178 ~]# lsmod | grep zfs zfs 1195522 0 zcommon 46278 1 zfs znvpair 80974 2 zfs,zcommon zavl 6925 1 zfs zunicode 323159 1 zfs spl 266655 5 zfs,zcommon,znvpair,zavl,zunicode
After confirming that the ZFS module is loaded, clear the network udev rules, and confirm PERSISTENT_DHCLIENT is set to “yes” in the /etc/sysconfig/network-scripts/ifcfg-eth0 file:
[root@euca-172-17-248-178 ~]# echo "" > /etc/udev/rules.d/70-persistent-net.rules [root@euca-172-17-248-178 ~]# echo "" > /lib/udev/rules.d/75-persistent-net-generator.rules [root@euca-172-17-248-178 ~]# echo "PERSISTENT_DHCLIENT=yes" >> /etc/sysconfig/network-scripts/ifcfg-eth0
Confirm that the instance has been upgraded to CentOS 6.6, then exit the instance.
[root@euca-172-17-248-178 ~]# cat /etc/redhat-release CentOS release 6.6 (Final) [root@euca-172-17-248-178 ~]# exit
Create the CentOS 6.6 EMI with ZFS Support
The instance is now ready to be bundled. Bundle the instance using the euca-bundle-instance command. This command is used to bundle Windows instances, however Eucalyptus extended this command to work with Linux instances as well. Use euca-describe-bundle-tasks to monitor the bundling status:
[root@odc-f-13 ~]# euca-bundle-instance --bucket centos6.6-zfs --prefix centos6.6-zfs i-b9fccf9f BUNDLE bun-b9fccf9f i-b9fccf9f centos6.6-zfs centos6.6-zfs 2014-12-03T23:54:51.644Z 2014-12-03T23:54:51.644Z pending 0 centos6.6-zfs/centos6.6-zfs.manifest.xml .. [root@odc-f-13 ~]# euca-describe-bundle-tasks BUNDLE bun-b9fccf9f i-b9fccf9f centos6.6-zfs centos6.6-zfs 2014-12-03T23:54:51.644Z 2014-12-03T23:57:37.517Z complete 0 centos6.6-zfs/centos6.6-zfs.manifest.xml
Once the bundle task completes, register the instance store-backed HVM image using the euca-register command:
[root@odc-f-13 ~]# euca-register -a x86_64 -n centos6.6-zfs centos6.6-zfs/centos6.6-zfs.manifest.xml --virtualization-type hvm IMAGE emi-5e63f02c
The custom image has been registered. Now lets test it out.
ZFS Test
To test the image out, we will do the following:
- Launch an instance from the new EMI
- Create 5 volumes and attach them to the instance
- Create a ZFS storage pool and dataset
To launch the instance, use the euca-run-instances command. To create the 5 EBS volumes, use euca-create-volume command. After the volumes are created, use euca-attach-volume to attach the volumes to the instance. Once the volumes are attached, the output of euca-describe-instances should look similar to the following:
# euca-describe-instances i-0cd3b6b8 RESERVATION r-cf7c5c73 325271821652 default INSTANCE i-0cd3b6b8 emi-5e63f02c euca-10-104-7-3.future.future.euca-hasp.cs.prc.eucalyptus-systems.com euca-172-17-248-184.future.internal running account2-user11 0 m1.medium 2014-12-04T00:16:52.887Z Honest monitoring-disabled 10.104.7.3 172.17.248.184 instance-store hvm sg-6ef9907f x86_64 BLOCKDEVICE /dev/sdd vol-a23cfb1f 2014-12-04T01:45:59.730Z false BLOCKDEVICE /dev/sdh vol-a27b75a5 2014-12-04T01:47:31.162Z false BLOCKDEVICE /dev/sdf vol-2a971204 2014-12-04T01:46:54.575Z false BLOCKDEVICE /dev/sdg vol-b33e9890 2014-12-04T01:47:13.346Z false BLOCKDEVICE /dev/sde vol-dcc8b6ac 2014-12-04T01:46:15.011Z false
SSH into the instance and check what block devices are associated with the EBS volumes using the lsblk command:
# ssh -i account2-user11.priv root@euca-10-104-7-3.future.future.euca-hasp.cs.prc.eucalyptus-systems.com [root@euca-172-17-248-184 ~]# lsblk NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT vda 252:0 0 4.9G 0 disk ├─vda1 252:1 0 500M 0 part /boot └─vda2 252:2 0 4.4G 0 part ├─VolGroup-lv_root (dm-0) 253:0 0 3.9G 0 lvm / └─VolGroup-lv_swap (dm-1) 253:1 0 500M 0 lvm [SWAP] vdb 252:16 0 5.1G 0 disk vdc 252:32 0 5G 0 disk vdd 252:48 0 5G 0 disk vde 252:64 0 5G 0 disk vdf 252:80 0 5G 0 disk vdg 252:96 0 5G 0 disk
The EBS volumes are /dev/vdc, /dev/vdd, /dev/vde, /dev/vdf, and /dev/vdg. Use these devices to create the ZFS storage pool by using the zpool command:
[root@euca-172-17-248-184 ~]# zpool create -f app-pool vdc vdd vde vdf vdg [root@euca-172-17-248-184 ~]# zpool status pool: app-pool state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM app-pool ONLINE 0 0 0 vdc1 ONLINE 0 0 0 vdd1 ONLINE 0 0 0 vde1 ONLINE 0 0 0 vdf1 ONLINE 0 0 0 vdg1 ONLINE 0 0 0 errors: No known data errors
Next, we need to create a ZFS dataset. For this example, this instance will end up being a MySQL server, so we will create a dataset for storing the MySQL data.
[root@euca-172-17-248-184 ~]# zfs create app-pool/mysql [root@euca-172-17-248-184 ~]# zfs list NAME USED AVAIL REFER MOUNTPOINT app-pool 152K 24.5G 30K /app-pool app-pool/mysql 30K 24.5G 30K /app-pool/mysql
The mount point of the dataset can be adjusted by setting the mountpoint option:
[root@euca-172-17-248-184 ~]# zfs set mountpoint=/opt/mysql app-pool/mysql [root@euca-172-17-248-184 ~]# zfs list NAME USED AVAIL REFER MOUNTPOINT app-pool 162K 24.5G 31K /app-pool app-pool/mysql 30K 24.5G 30K /opt/mysql
Thats it! Notice how this only required 2 commands to set up a LVM + RAID filesystem, compared to around 7 commands using mdadm, pvcreate, vgcreate, mkfs, mkdir and mount. The instance is now ready to utilize the ZFS filesystem for the MySQL server.
Online Backup Example to OSG Bucket using s3cmd
As mentioned earlier, a slick feature of using ZFS is being able to perform backups online. This section will show the following:
- Setup and configure s3cmd
- Create a ZFS snapshot, and use ZFS send with s3cmd to place the snapshot on an OSG bucket
To get started, in the instance, install the following packages:
[root@euca-172-17-248-184 ~]# yum install -y git python-dateutil.noarch xz
Next, clone the s3tools/s3cmd repository from Github:
[root@euca-172-17-248-184 ~]# git clone https://github.com/s3tools/s3cmd.git
If the instance was launched with an instance profile that assumes a role with OSG (S3) API access, s3cmd will pick up the temporary credentials and token through the Eucalyptus instance metadata service, as if the instance was launched on AWS EC2. This wasn’t the case here, so we need to provide the Access Key ID and Secret Key manually:
[root@euca-172-17-248-184 ~]# ./s3cmd/s3cmd --configure Enter new values or accept defaults in brackets with Enter. Refer to user manual for detailed description of all options. Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables. Access Key: AKIRAGCHAGFE6IIX9BYF Secret Key: GMdrL97AqcybhfyyxOpNmVUnBtiMenag3ju82L7L Encryption password is used to protect your files from reading by unauthorized persons while in transfer to S3 Encryption password: Path to GPG program [/usr/bin/gpg]: When using secure HTTPS protocol all communication with Amazon S3 servers is protected from 3rd party eavesdropping. This method is slower than plain HTTP and can't be used if you're behind a proxy Use HTTPS protocol [No]: On some networks all internet access must go through a HTTP proxy. Try setting it here if you can't connect to S3 directly HTTP Proxy server name: New settings: Access Key: AKIRAGCHAGFE6IIX9BYF Secret Key: GMdrL97AqcybhfyyxOpNmVUnBtiMenag3ju82L7L Encryption password: Path to GPG program: /usr/bin/gpg Use HTTPS protocol: False HTTP Proxy server name: HTTP Proxy server port: 0 Test access with supplied credentials? [Y/n] n Save settings? [y/N] y Configuration saved to '/root/.s3cfg'
Edit the .s3cfg file to make sure to point to the OSG on your Eucalyptus 4.0.2 cloud. For example, change the following:
host_base = s3.amazonaws.com
to
host_base = objectstorage.future.euca-hasp.cs.prc.eucalyptus-systems.com:8773
and
host_bucket = %(bucket)s.s3.amazonaws.com
to
host_bucket = %(bucket)s.objectstorage.future.euca-hasp.cs.prc.eucalyptus-systems.com:8773
Confirm that s3cmd is configured correctly. For example:
[root@euca-172-17-248-184 ~]# ./s3cmd/s3cmd ls 2014-11-05 21:45 s3://centos-images 2014-12-03 23:54 s3://centos6.6-zfs 2014-10-08 01:50 s3://instance-profile-testing 2014-12-01 22:27 s3://mongodb-snapshots 2014-10-10 20:01 s3://new-ubuntu-bundled-image 2014-09-17 18:31 s3://s3cmd-testing 2014-09-30 01:58 s3://ubuntu-bundled-vol 2014-10-22 14:47 s3://ubuntu-docker-template 2014-10-08 13:39 s3://ubuntu-images 2014-10-02 01:42 s3://ubuntu-trusty-imported-20141001 2014-10-30 18:25 s3://ubuntu-trusty-imported-20141030 2014-10-29 02:18 s3://ubuntu-trusty-server-10282014 2014-10-01 00:28 s3://wrong-s3-url-test
To perform a ZFS snapshot of the app-pool/mysql dataset, do the following:
[root@euca-172-17-248-184 ~]# zfs snapshot app-pool/mysql@wednesday [root@euca-172-17-248-184 ~]# zfs list -t snapshot NAME USED AVAIL REFER MOUNTPOINT app-pool/mysql@wednesday 0 - 30K -
After creating a bucket for the backup, send the ZFS snapshot to the bucket:
[root@euca-172-17-248-184 ~]# ./s3cmd/s3cmd mb s3://mysql-backups [root@euca-172-17-248-184 ~]# zfs send app-pool/mysql@wednesday | xz | ./s3cmd/s3cmd put - s3://mysql-backups/mysql-backup-wednesday.img.xz <stdin> -> s3://mysql-backups/mysql-backup-wednesday.img.xz [part 1, 1440B] 1440 of 1440 100% in 2s 561.67 B/s done
To confirm if the snapshot is located in the bucket, use s3cmd:
[root@euca-172-17-248-184 ~]# ./s3cmd/s3cmd ls s3://mysql-backups 2014-12-04 02:22 1440 s3://mysql-backups/mysql-backup-wednesday.img.xz
Thats all folks. We have successfully created a CentOS 6.6 EMI with ZFS support. For more information regarding ZFS (and inspirations for this blog), check out the following resources:
More Mind Spew-age from Harold Spencer Jr. 