Customizing Eucalyptus Load Balancer for Eucalyptus 4.0


For the Elastic Load Balancing service, Eucalyptus utilizes an HAProxy instance.  The load balancer image contains the following version of haproxy (as of Eucalyptus 4.0.0):

# /usr/sbin/haproxy -vv
HA-Proxy version 1.5-dev21-6b07bf7 +2013/12/17
Copyright 2000-2013 Willy Tarreau <>

Build options :
 TARGET = linux2628
 CPU = generic
 CC = gcc
 CFLAGS = -O2 -g -fno-strict-aliasing

Default settings :
 maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built without zlib support (USE_ZLIB not set)
Compression algorithms supported : identity
Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 7.8 2008-09-05
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
 epoll : pref=300, test result OK
 poll : pref=200, test result OK
 select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

By default, the following HAProxy configuration options are used by the Eucalyptus Load Balancer image (defined by the Eucalyptus load-balancer-servo application, which is the controlling mechanism for Eucalyptus load balancing):

 maxconn 100000
 ulimit-n 655360
 pidfile /var/run/

#drop privileges after port binding
 user servo
 group servo

 contimeout 1000
 clitimeout 10000
 srvtimeout 10000
 option http-server-close # affects KA on/off

Given what backend applications will be used with the Eucalyptus Load Balancer, these settings may not be sufficient.

The goal of this entry is to demonstrate how to customize the Eucalyptus Load Balancer image configuration to handle various backend applications that will be used with the load balancer.


In order to customize the Eucalyptus Load Balancer image, the credentials of the cloud administrator (Eucalyptus IAM credentials of a user under the ‘eucalyptus’ account) must be used.   These credentials are needed to do the following:

Since the cloud administrator credentials will be used, there will be no need to define any Eucalyptus IAM policies.

To mount, and modify the Eucalyptus Load Balancer image, the following Linux tools are needed:

The examples in this blog were all done on CentOS 6.5 machine where the eucalyptus-load-balancer-image package has been installed.  This package contains the ‘euca-install-load-balancer’ command.

Obtaining the Eucalyptus Elastic Load Balancer Image

There are couple of ways to obtain the Eucalyptus Load Balancer image:

This blog entry will use the eucalyptus-load-balancer-image RPM, and update the image according.  To get started, create a directory (in this example ‘eucalyptus-lb’), and  download the latest eucalyptus load balancer image RPM from

[root@odc-f-13 ~]# mkdir eucalyptus-lb; cd eucalyptus-lb
[root@odc-f-13 eucalyptus-lb]# wget

Once the RPM package has been downloaded, unpack the RPM:

[root@odc-f-13 eucalyptus-lb]# rpm2cpio eucalyptus-load-balancer-image-1.1.0-0.212.el6.x86_64.rpm | cpio --extract --make-directories --preserve-modification-time --verbose
559369 blocks

After unpacking the RPM, change directory to ~/usr/share/eucalyptus-load-balancer-image, and decompress the eucalyptus-load-balancer-image-1.1.0-212.tgz file to obtain the Eucalyptus Load Balancer image:

[root@odc-f-13 eucalyptus-lb]# cd usr/share/eucalyptus-load-balancer-image
[root@odc-f-13 eucalyptus-load-balancer-image]# tar -xzvf eucalyptus-load-balancer-image-1.1.0-212.tgz

Now that the image is available, we can modify it accordingly.

Modifying the Eucalyptus Load Balancer Image

To modify the Eucalyptus Load Balancer image, the image needs to be mounted to a loopback device, as demonstrated below:

[root@odc-f-13 eucalyptus-load-balancer-image]# mkdir /mnt/centos
[root@odc-f-13 eucalyptus-load-balancer-image]# losetup /dev/loop0 eucalyptus-load-balancer-image.img
[root@odc-f-13 eucalyptus-load-balancer-image]# kpartx -av /dev/loop0
add map loop0p1 (253:2): 0 3145728 linear /dev/loop0 2048
[root@odc-f-13 eucalyptus-load-balancer-image]# mount /dev/mapper/loop0p1 /mnt/centos
[root@odc-f-13 eucalyptus-load-balancer-image]# chroot /mnt/centos
[root@odc-f-13 /]#

We will modify and add the following HAProxy options under the ‘default’ section in the /etc/load-balancer-servo/haproxy_template.conf file.  For information about these options, please refer to the HAProxy 1.5 documentation:

  • replace ‘srvtimeout‘ with ‘timeout server‘ since ‘srvtimeout‘ is deprecated, and set the value to ‘2m
  • replace ‘clitimeout‘ with ‘timeout client‘ since ‘clitimeout‘ is deprecated, and set the value to ‘2m
  • replace ‘contimeout ‘ with ‘timeout connect‘ since ‘contimeout‘ is deprecated, and set the value to ‘5s
  • add ‘timeout http-keep-alive‘ with the value of  ‘10s
  • add ‘timeout queue‘ with the value of ‘1m
  • add ‘timeout check‘ with the value of ‘5s
  • add ‘retries‘ with the value of ‘3
  • add the following options to not log null connections, and to enable session redistribution in case of failure:
    • option dontlognull
    • option redispatch

The  /etc/load-balancer-servo/haproxy_template.conf should look similar to the following after all the desired attributes are added:

[root@odc-f-13 /]# cat /etc/load-balancer-servo/haproxy_template.conf
 maxconn 100000
 ulimit-n 655360
 pidfile /var/run/

#drop privileges after port binding
 user servo
 group servo

 timeout connect 5s
 timeout client 2m
 timeout server 2m
 timeout http-keep-alive 10s
 timeout queue 1m
 timeout check 5s
 retries 3
 option dontlognull
 option redispatch
 option http-server-close # affects KA on/off

(Note:  Depending upon what edits are being done to the HAProxy configuration settings, there may also be a need to edit the /etc/sysctl.conf file to help get the desired behavior from the Eucalyptus Load Balancer.  For example, the following sysctl properties can be edited to increase/decrease TCP timeouts:

  • net.ipv4.tcp_keepalive_time
  • net.ipv4.tcp_keepalive_intvl
  • net.ipv4.tcp_keepalive_probes

For more information about editing sysctl values, the documentation from RedHat can be referenced.)

Once all edits are completed, confirm that the configuration file is correct, exit out of the chroot environment and unmount the image:

[root@odc-f-13 /]# /usr/sbin/haproxy -c -f /etc/load-balancer-servo/haproxy_template.conf
Configuration file has no error but will not start (no listener) => exit(2).

[root@odc-f-13 eucalyptus-load-balancer-image]# umount /mnt/centos
[root@odc-f-13 eucalyptus-load-balancer-image]# kpartx -dv /dev/loop0
del devmap : loop0p1
[root@odc-f-13 eucalyptus-load-balancer-image]# losetup -d /dev/loop0

Installing the New Eucalyptus Load Balancer Image

After the image has been unmounted, create a new tar-gzipped file that contains the modified Eucalyptus Load Balancer image:

[root@odc-f-13 eucalyptus-load-balancer-image]# tar -zcvf eucalyptus-load-balancer-image-updated.tgz eucalyptus-load-balancer-image.img

Next, make sure the cloud administrator credentials are sourced and check the cloud properties for the Eucalyptus Load Balancer service:

[root@odc-f-13 eucalyptus-load-balancer-image]# cd
[root@odc-f-13 ~]# source eucarc
[root@odc-f-13 ~]# euca-describe-properties | grep load
PROPERTY authentication.credential_download_host_match {}
PROPERTY loadbalancing.loadbalancer_app_cookie_duration 24
PROPERTY loadbalancing.loadbalancer_dns_subdomain elb
PROPERTY loadbalancing.loadbalancer_emi emi-F0D5828C
PROPERTY loadbalancing.loadbalancer_instance_type m1.medium
PROPERTY loadbalancing.loadbalancer_num_vm 1
PROPERTY loadbalancing.loadbalancer_restricted_ports 22
PROPERTY loadbalancing.loadbalancer_vm_keyname euca-elb
PROPERTY loadbalancing.loadbalancer_vm_ntp_server

Check to see what load balancer images are enabled:

[root@odc-f-13 ~]# euca-install-load-balancer --list
Currently Installed Load Balancer Bundles:

Version 1
emi-FA373789 (loadbalancer_v1/eucalyptus-load-balancer-image-1.0.4-164.img.manifest.xml)
 Installed on 2014-05-20 at 07:12:18 PDT

Version 2 (enabled)
emi-F0D5828C (loadbalancer-v2/eucalyptus-load-balancer-image.img.manifest.xml)
 Installed on 2014-05-28 at 11:10:03 PDT

To install the modified load balancer image, use the ‘euca-install-load-balancer‘ command:

[root@odc-f-13 ~]# euca-install-load-balancer -t ~/eucalyptus-lb/usr/share/eucalyptus-load-balancer-image/eucalyptus-load-balancer-image-updated.tgz
Decompressing tarball: eucalyptus-lb/usr/share/eucalyptus-load-balancer-image/eucalyptus-load-balancer-image-updated.tgz
Bundling and uploading image to bucket: loadbalancer-v3
Registering image manifest: loadbalancer-v3/eucalyptus-load-balancer-image.img.manifest.xml
Registered image: emi-BCAD86BE
PROPERTY loadbalancing.loadbalancer_emi emi-BCAD86BE was emi-F0D5828C

Confirm that the new EMI is enabled:

[root@odc-f-13 ~]# euca-install-load-balancer --list
Currently Installed Load Balancer Bundles:

Version 1
emi-FA373789 (loadbalancer_v1/eucalyptus-load-balancer-image-1.0.4-164.img.manifest.xml)
 Installed on 2014-05-20 at 07:12:18 PDT

Version 2
emi-F0D5828C (loadbalancer-v2/eucalyptus-load-balancer-image.img.manifest.xml)
 Installed on 2014-05-28 at 11:10:03 PDT

Version 3 (enabled)
emi-BCAD86BE (loadbalancer-v3/eucalyptus-load-balancer-image-updated.img.manifest.xml)
 Installed on 2014-07-06 at 18:02:20 PDT

Confirming the Updated Load Balancer Configuration

To confirm the changes, make sure the cloud property loadbalancing.loadbalancer_vm_keyname‘ has a defined value for debugging purposes, then create a Eucalyptus Elastic Load Balancer:

[root@odc-f-13 ~]# eulb-create-lb TestLoadBalancer -z ViciousLiesAndDangerousRumors -l "lb-port=80, protocol=HTTP, instance-port=80, instance-protocol=HTTP"

Confirm that the load balancer instance is running (only the cloud administrator can see the load balancing instance IDs), and authorize port 22 (SSH) to the instance:

[root@odc-f-13 ~]# euca-describe-instances
RESERVATION r-C27D6F37 944786667073 euca-internal-408396244283-TestLoadBalancer
INSTANCE i-A0DBA47D emi-BCAD86BE euca-172-18-246-58.bigboi.internal running euca-elb 0 m1.medium 2014-07-07T01:08:32.400Z ViciousLiesAndDangerousRumors monitoring-enabled instance-store hvm bce54dd2-a9af-4587-9421-457e22dda5ff_ViciousLiesAndDangerousR_1 sg-4F469747 arn:aws:iam::944786667073:instance-profile/internal/loadbalancer/loadbalancer-vm-408396244283-TestLoadBalancer
TAG instance i-A0DBA47D Name loadbalancer-resources
TAG instance i-A0DBA47D aws:autoscaling:groupName asg-euca-internal-elb-408396244283-TestLoadBalancer
TAG instance i-A0DBA47D euca:node
[root@odc-f-13 ~]# euca-authorize -P tcp -p ssh euca-internal-408396244283-TestLoadBalancer
GROUP euca-internal-408396244283-TestLoadBalancer
PERMISSION euca-internal-408396244283-TestLoadBalancer ALLOWS tcp 22 22 FROM CIDR

After SSHing into the load balancer instance, confirm that the /var/lib/load-balancer-servo/euca_haproxy.conf file has the updated changes:

[root@odc-f-13 ~]# ssh -i euca-elb.priv
The authenticity of host ' (' can't be established.
RSA key fingerprint is e3:9b:80:e2:f3:12:a3:0b:f0:5c:7c:6b:bc:d8:9d:77.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '' (RSA) to the list of known hosts.
Warning: the RSA host key for '' differs from the key for the IP address ''
Offending key for IP in /root/.ssh/known_hosts:9
Are you sure you want to continue connecting (yes/no)? yes
[root@euca-172-18-246-58 ~]# ps aux | grep haprox
servo 1027 0.0 0.2 58648 3020 ? Ss 01:09 0:00 /usr/sbin/haproxy -f /var/lib/load-balancer-servo/euca_haproxy.conf -p /var/run/load-balancer-servo/ -V -C /var/lib/load-balancer-servo -D
[root@euca-172-18-246-58 ~]# cat /var/lib/load-balancer-servo/euca_haproxy.conf
 maxconn 100000
 ulimit-n 655360
 pidfile /var/run/
 #drop privileges after port binding
 user servo
 group servo

 timeout connect 5s
 timeout client 2m
 timeout server 2m
 timeout http-keep-alive 10s
 timeout queue 1m
 timeout check 5s
 retries 3
 option dontlognull
 option redispatch
 option http-server-close # affects KA on/off

frontend http-80
 # lb-TestLoadBalancer
 mode http
 option forwardfor except
 log /var/lib/load-balancer-servo/haproxy.sock local2 info
 log-format httplog\ %f\ %b\ %s\ %ST\ %ts\ %Tq\ %Tw\ %Tc\ %Tr\ %Tt
 default_backend backend-http-80

backend backend-http-80
 mode http
 balance roundrobin

The ‘default‘ section should contain all the modifications made to the /etc/load-balancer-servo/haproxy_template.conf file.   The Eucalyptus Load Balancer will now utilize the updated changes needed to address the desired performance with the various backend applications that will be used with the load balancer.

Step-by-Step Deployment of Docker on Eucalyptus 3.4 for the Cloud Administrator


Eucalyptus Systems, Inc.

Docker has been in the news lately as one of the hot open-source project promoting linux containers. Some use cases for Docker include the following:

  • Automation of packaging and application deployment
  • Lightweight PaaS environments
  • Automated testing and continuous integration/deployment
  • Deploying and scaling web applications, databases and backend services

The focus of this blog entry is to show how to deploy Docker on Eucalyptus from a cloud administrator’s point-of-view – all in the cloud.  This is a step-by-step guide to create an Docker EMI from an existing Ubuntu Cloud Raring EMI using AWS’s documentation.  This entry will also show how to build euca2ools from source in the Ubuntu Cloud image.


This entry assumes the following:

After confirming that the prerequisites are met, let’s get started.

Creating an EMI From an Existing EMI

As mentioned earlier, these steps will be based off of  AWS’s documentation on creating an instance store-backed AMI from an existing AMI.  In this example, here is an existing Ubuntu Raring instance thats running on Eucalyptus:

$ euca-describe-instances --region eucalyptus-admin@
RESERVATION r-3E423E33 961915002812 default
INSTANCE i-827E3E88 emi-06663A57
euca-172-17-118-27.eucalyptus.internal running euca-admin 0 m1.medium 
2013-11-18T22:41:35.694Z LayinDaSmackDown eki-28F338EB eri-51253C0A 
monitoring-disabled instance-store

This instance is using the following EMI, EKI and ERI:

$ euca-describe-images emi-06663A57 eki-28F338EB eri-51253C0A --region eucalyptus-admin@
IMAGE eki-28F338EB latest-raring-kernel/raring-server-cloudimg-amd64-vmlinuz-generic.manifest.xml 
441445882805 available public x86_64 kernel instance-store
IMAGE emi-06663A57 latest-raring/raring-server-cloudimg-amd64.img.manifest.xml 441445882805 
available public x86_64 machine eki-28F338EB eri-51253C0A instance-store paravirtualized
IMAGE eri-51253C0A latest-raring-kernel/raring-server-cloudimg-amd64-loader.manifest.xml 441445882805 
available public x86_64 ramdisk instance-store

To start, copy the zip file cloud administrator credentials obtained by the euca_conf command mentioned in the Eucalyptus 3.4 documentation to the running instance:

# scp -i euca-admin.priv

Next, install the following packages for the 3.8.0-33 kernel, and packages needed to build euca2ools:

ubuntu@euca-172-17-118-27:~$ sudo apt-get install python-setuptools git python-lxml unzip linux-headers-3.8.0-33-generic linux-image-extra-3.8.0-33-generic

Find the ephemeral storage using the instance metadata service, format, and mount the ephemeral to /mnt/image:

ubuntu@euca-172-17-118-27:~$ curl
ubuntu@euca-172-17-118-27:~$ sudo mkdir /mnt/image
ubuntu@euca-172-17-118-27:~$ sudo mkfs.ext4 /dev/vda2
ubuntu@euca-172-17-118-27:~$ sudo mount /dev/vda2 /mnt/image

Download euca2ools from Github:

ubuntu@euca-172-17-118-27:~$ git clone

Install euca2ools:

ubuntu@euca-172-17-118-27:~$ cd euca2ools; sudo python install

Unzip the cloud administrator credentials in /tmp:

ubuntu@euca-172-17-118-27:~$ cd /tmp; unzip

Change to the root user, and source the cloud administrator credentials:

ubuntu@euca-172-17-118-27:~$ sudo -s; source /tmp/eucarc

Bundle, upload and register the ramdisk and kernel under /boot:

root@euca-172-17-118-27:~# euca-bundle-image -i /boot/initrd.img-3.8.0-33-generic 
--ramdisk true -r x86_64
root@euca-172-17-118-27:~# euca-upload-bundle -b ubuntu-raring-docker-ramdisk 
-m /var/tmp/bundle-SQrAuT/initrd.img-3.8.0-33-generic.manifest.xml
root@euca-172-17-118-27:~# euca-register -n ubuntu-raring-docker-ramdisk 
IMAGE eri-6BF033EE
root@euca-172-17-118-27:~# euca-bundle-image -i /boot/vmlinuz-3.8.0-33-generic 
--kernel true -r x86_64
root@euca-172-17-118-27:~# euca-upload-bundle -b ubuntu-raring-docker-kernel 
-m /var/tmp/bundle-31Lnxy/vmlinuz-3.8.0-33-generic.manifest.xml
root@euca-172-17-118-27:~# euca-register -n ubuntu-raring-docker-kernel 
IMAGE eki-17093995

Use euca-bundle-vol to bundle the root filesystem. Make sure to exclude /tmp, /mnt/image, and /home/ubuntu. Additionally, make sure and set the size of the image to be 5 GB:

root@euca-172-17-118-27:~# euca-bundle-vol -p ubuntu-raring-docker 
-s 5120 -e /tmp,/root,/mnt/image,/home/ubuntu -d /mnt/image 
--kernel eki-17093995 --ramdisk eri-6BF033EE -r x86_64

Next, upload and register the root filesystem:

root@euca-172-17-118-27:~# euca-upload-bundle -b ubuntu-raring-docker-rootfs 
-m /mnt/image/ubuntu-raring-docker.manifest.xml
root@euca-172-17-118-27:~# euca-register -n ubuntu-raring-docker-rootfs 
IMAGE emi-26403979

We have the new EMI, EKI and ERI for the Docker instance.  Lastly, set the image permissions so that all users on the cloud can use the EMI, EKI and ERI:

root@euca-172-17-118-27:~# euca-modify-image-attribute -l -a all emi-26403979
root@euca-172-17-118-27:~# euca-modify-image-attribute -l -a all eki-17093995
root@euca-172-17-118-27:~# euca-modify-image-attribute -l -a all eri-6BF033EE

Now its time to launch the Docker EMI.

Running the Docker Instance with Cloud-Init

Before launching the EMI, the cloud-init configuration file needs to be created.  This file will be responsible for configuring the instance repositories, downloading and installing Docker.  With your favorite command-line editor, create a file called cloud-init-docker.config, with the following content:

apt_update: true
apt_upgrade: true
disable_root: true
 - less
 - ssh
 - [ apt-update-upgrade, always ]
 - updates-check
 - runcmd
 - [ sh, -xc, "INST_HOSTNAME=`/bin/hostname`; META_IP=`curl -s`; echo ${META_IP} ${INST_HOSTNAME} >> /etc/hosts" ]
 - [ locale-gen, en_US.UTF-8 ]
 - [ sh, -xc, "wget -qO docker-io.gpg" ]
 - [ apt-key, add, docker-io.gpg ]
 - [ sh, -xc, "echo 'deb docker main' > /etc/apt/sources.list.d/docker.list" ]
 - [ apt-get, update ]
 - [ apt-get, install, -y, --force-yes, lxc-docker ]
 - [ modprobe, -q, aufs ]

Now, use euca-run-instances to launch the instance:

root@euca-172-17-118-27:~# euca-run-instances -k euca-admin emi-351237D1 
-t m1.medium --user-data-file cloud-init-docker.config

After launching the instance, leave the current instance to get back to end client.

root@euca-172-17-118-27:~# exit
ubuntu@euca-172-17-118-27:~$ exit
Connection to closed.

Once the instance reaches running state, ssh into the instance using the keypair specified (which in this case will be euca-admin.priv), and execute the following Docker command to run an interactive shell session inside a minimal Ubuntu container:

$ euca-describe-instances --region eucalyptus-admin@
RESERVATION r-A1613D7F 961915002812 default
INSTANCE i-AFDB3D4C emi-26403979 
euca-172-17-118-16.eucalyptus.internal running euca-admin 0 m1.medium 
2013-11-19T01:21:10.880Z LayinDaSmackDown eki-17093995 eri-6BF033EE monitoring-disabled instance-store
# ssh -i euca-admin.priv
Welcome to Ubuntu 13.04 (GNU/Linux 3.8.0-33-generic x86_64)
* Documentation:
System information as of Thu Nov 14 23:18:38 UTC 2013
System load: 0.0 Users logged in: 0
 Usage of /: 21.6% of 4.89GB IP address for eth0:
 Memory usage: 4% IP address for lxcbr0:
 Swap usage: 0% IP address for docker0:
 Processes: 83
Graph this data and manage this system at
Get cloud support with Ubuntu Advantage Cloud Guest:

Use Juju to deploy your cloud instances and workloads:

New release '13.10' available.
Run 'do-release-upgrade' to upgrade to it.
Last login: Thu Nov 14 23:08:09 2013 from
ubuntu@euca-172-17-184-76:~$ sudo docker run -i -t ubuntu /bin/bash
Unable to find image 'ubuntu' (tag: latest) locally
Pulling repository ubuntu
8dbd9e392a96: Download complete
b750fe79269d: Download complete
27cf78414709: Download complete
root@041d5ddcd6b9:/# (Ctrl-p Ctrl-q to exit out of shell)
ubuntu@euca-172-17-184-76:~$ sudo docker ps
041d5ddcd6b9 ubuntu:12.04 /bin/bash 27 seconds ago Up 26 seconds pink_frog

Thats it!  For more information regarding Docker, please refer to the latest Docker documentation.