Background

For the Elastic Load Balancing service, Eucalyptus utilizes an HAProxy instance.  The load balancer image contains the following version of haproxy (as of Eucalyptus 4.0.0):

# /usr/sbin/haproxy -vv
HA-Proxy version 1.5-dev21-6b07bf7 +2013/12/17
Copyright 2000-2013 Willy Tarreau <w@1wt.eu>

Build options :
 TARGET = linux2628
 CPU = generic
 CC = gcc
 CFLAGS = -O2 -g -fno-strict-aliasing
 OPTIONS = USE_LINUX_TPROXY=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1

Default settings :
 maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200

Encrypted password support via crypt(3): yes
Built without zlib support (USE_ZLIB not set)
Compression algorithms supported : identity
Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013
OpenSSL library supports TLS extensions : yes
OpenSSL library supports SNI : yes
OpenSSL library supports prefer-server-ciphers : yes
Built with PCRE version : 7.8 2008-09-05
PCRE library supports JIT : no (USE_PCRE_JIT not set)
Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND

Available polling systems :
 epoll : pref=300, test result OK
 poll : pref=200, test result OK
 select : pref=150, test result OK
Total: 3 (3 usable), will use epoll.

By default, the following HAProxy configuration options are used by the Eucalyptus Load Balancer image (defined by the Eucalyptus load-balancer-servo application, which is the controlling mechanism for Eucalyptus load balancing):

#template
global
 maxconn 100000
 ulimit-n 655360
 pidfile /var/run/haproxy.pid

#drop privileges after port binding
 user servo
 group servo

defaults
 contimeout 1000
 clitimeout 10000
 srvtimeout 10000
 option http-server-close # affects KA on/off

Given what backend applications will be used with the Eucalyptus Load Balancer, these settings may not be sufficient.

The goal of this entry is to demonstrate how to customize the Eucalyptus Load Balancer image configuration to handle various backend applications that will be used with the load balancer.

Prerequisites

In order to customize the Eucalyptus Load Balancer image, the credentials of the cloud administrator (Eucalyptus IAM credentials of a user under the ‘eucalyptus’ account) must be used.   These credentials are needed to do the following:

• Modify the loadbalancing.loadbalancer_emi cloud property
• Use the ‘euca-install-load-balancer‘ command to install the modified load balancer image

Since the cloud administrator credentials will be used, there will be no need to define any Eucalyptus IAM policies.

To mount, and modify the Eucalyptus Load Balancer image, the following Linux tools are needed:

• rpm2cpio
• cpio
• tar
• losetup
• kpartx
• mount
• chroot

The examples in this blog were all done on CentOS 6.5 machine where the eucalyptus-load-balancer-image package has been installed.  This package contains the ‘euca-install-load-balancer’ command.

Obtaining the Eucalyptus Elastic Load Balancer Image

There are couple of ways to obtain the Eucalyptus Load Balancer image:

• Use the eucalyptus/load-balancer-image repository on Github
• Download the eucalyptus-load-balancer-image RPM from downloads.eucalyptus.com

This blog entry will use the eucalyptus-load-balancer-image RPM, and update the image according.  To get started, create a directory (in this example ‘eucalyptus-lb’), and  download the latest eucalyptus load balancer image RPM from downloads.eucalyptus.com:

[root@odc-f-13 ~]# mkdir eucalyptus-lb; cd eucalyptus-lb
[root@odc-f-13 eucalyptus-lb]# wget http://downloads.eucalyptus.com/software/eucalyptus/4.0/centos/6/x86_64/eucalyptus-load-balancer-image-1.1.0-0.212.el6.x86_64.rpm

Once the RPM package has been downloaded, unpack the RPM:

[root@odc-f-13 eucalyptus-lb]# rpm2cpio eucalyptus-load-balancer-image-1.1.0-0.212.el6.x86_64.rpm | cpio --extract --make-directories --preserve-modification-time --verbose
./usr/bin/euca-install-load-balancer
./usr/share/doc/eucalyptus-load-balancer-image-1.1.0
./usr/share/doc/eucalyptus-load-balancer-image-1.1.0/IMAGE-LICENSE
./usr/share/doc/eucalyptus-load-balancer-image-1.1.0/eucalyptus-load-balancer-image.ks
./usr/share/eucalyptus-load-balancer-image
./usr/share/eucalyptus-load-balancer-image/eucalyptus-load-balancer-image-1.1.0-212.tgz
559369 blocks

After unpacking the RPM, change directory to ~/usr/share/eucalyptus-load-balancer-image, and decompress the eucalyptus-load-balancer-image-1.1.0-212.tgz file to obtain the Eucalyptus Load Balancer image:

[root@odc-f-13 eucalyptus-lb]# cd usr/share/eucalyptus-load-balancer-image
[root@odc-f-13 eucalyptus-load-balancer-image]# tar -xzvf eucalyptus-load-balancer-image-1.1.0-212.tgz
eucalyptus-load-balancer-image.img

Now that the image is available, we can modify it accordingly.

Modifying the Eucalyptus Load Balancer Image

To modify the Eucalyptus Load Balancer image, the image needs to be mounted to a loopback device, as demonstrated below:

[root@odc-f-13 eucalyptus-load-balancer-image]# mkdir /mnt/centos
[root@odc-f-13 eucalyptus-load-balancer-image]# losetup /dev/loop0 eucalyptus-load-balancer-image.img
[root@odc-f-13 eucalyptus-load-balancer-image]# kpartx -av /dev/loop0
add map loop0p1 (253:2): 0 3145728 linear /dev/loop0 2048
[root@odc-f-13 eucalyptus-load-balancer-image]# mount /dev/mapper/loop0p1 /mnt/centos
[root@odc-f-13 eucalyptus-load-balancer-image]# chroot /mnt/centos
[root@odc-f-13 /]#

We will modify and add the following HAProxy options under the ‘default’ section in the /etc/load-balancer-servo/haproxy_template.conf file.  For information about these options, please refer to the HAProxy 1.5 documentation:

• replace ‘srvtimeout‘ with ‘timeout server‘ since ‘srvtimeout‘ is deprecated, and set the value to ‘2m‘
• replace ‘clitimeout‘ with ‘timeout client‘ since ‘clitimeout‘ is deprecated, and set the value to ‘2m‘
• replace ‘contimeout ‘ with ‘timeout connect‘ since ‘contimeout‘ is deprecated, and set the value to ‘5s‘
• add ‘timeout http-keep-alive‘ with the value of  ‘10s‘
• add ‘timeout queue‘ with the value of ‘1m‘
• add ‘timeout check‘ with the value of ‘5s‘
• add ‘retries‘ with the value of ‘3‘
• add the following options to not log null connections, and to enable session redistribution in case of failure:
  • ‘option dontlognull‘
  • ‘option redispatch‘

The  /etc/load-balancer-servo/haproxy_template.conf should look similar to the following after all the desired attributes are added:

[root@odc-f-13 /]# cat /etc/load-balancer-servo/haproxy_template.conf
#template
global
 maxconn 100000
 ulimit-n 655360
 pidfile /var/run/haproxy.pid

#drop privileges after port binding
 user servo
 group servo

defaults
 timeout connect 5s
 timeout client 2m
 timeout server 2m
 timeout http-keep-alive 10s
 timeout queue 1m
 timeout check 5s
 retries 3
 option dontlognull
 option redispatch
 option http-server-close # affects KA on/off

(Note:  Depending upon what edits are being done to the HAProxy configuration settings, there may also be a need to edit the /etc/sysctl.conf file to help get the desired behavior from the Eucalyptus Load Balancer.  For example, the following sysctl properties can be edited to increase/decrease TCP timeouts:

• net.ipv4.tcp_keepalive_time
• net.ipv4.tcp_keepalive_intvl
• net.ipv4.tcp_keepalive_probes

For more information about editing sysctl values, the documentation from RedHat can be referenced.)

Once all edits are completed, confirm that the configuration file is correct, exit out of the chroot environment and unmount the image:

[root@odc-f-13 /]# /usr/sbin/haproxy -c -f /etc/load-balancer-servo/haproxy_template.conf
Configuration file has no error but will not start (no listener) => exit(2).

[root@odc-f-13 eucalyptus-load-balancer-image]# umount /mnt/centos
[root@odc-f-13 eucalyptus-load-balancer-image]# kpartx -dv /dev/loop0
del devmap : loop0p1
[root@odc-f-13 eucalyptus-load-balancer-image]# losetup -d /dev/loop0

Installing the New Eucalyptus Load Balancer Image

After the image has been unmounted, create a new tar-gzipped file that contains the modified Eucalyptus Load Balancer image:

[root@odc-f-13 eucalyptus-load-balancer-image]# tar -zcvf eucalyptus-load-balancer-image-updated.tgz eucalyptus-load-balancer-image.img

Next, make sure the cloud administrator credentials are sourced and check the cloud properties for the Eucalyptus Load Balancer service:

[root@odc-f-13 eucalyptus-load-balancer-image]# cd
[root@odc-f-13 ~]# source eucarc
[root@odc-f-13 ~]# euca-describe-properties | grep load
PROPERTY ViciousLiesAndDangerousRumors.storage.maxconcurrentsnapshotuploads 3
PROPERTY ViciousLiesAndDangerousRumors.storage.snapshotuploadtimeoutinhours 48
PROPERTY authentication.credential_download_host_match {}
PROPERTY loadbalancing.loadbalancer_app_cookie_duration 24
PROPERTY loadbalancing.loadbalancer_dns_subdomain elb
PROPERTY loadbalancing.loadbalancer_emi emi-F0D5828C
PROPERTY loadbalancing.loadbalancer_instance_type m1.medium
PROPERTY loadbalancing.loadbalancer_num_vm 1
PROPERTY loadbalancing.loadbalancer_restricted_ports 22
PROPERTY loadbalancing.loadbalancer_vm_keyname euca-elb
PROPERTY loadbalancing.loadbalancer_vm_ntp_server pool.ntp.org

Check to see what load balancer images are enabled:

[root@odc-f-13 ~]# euca-install-load-balancer --list
Currently Installed Load Balancer Bundles:

Version 1
emi-FA373789 (loadbalancer_v1/eucalyptus-load-balancer-image-1.0.4-164.img.manifest.xml)
 Installed on 2014-05-20 at 07:12:18 PDT

Version 2 (enabled)
emi-F0D5828C (loadbalancer-v2/eucalyptus-load-balancer-image.img.manifest.xml)
 Installed on 2014-05-28 at 11:10:03 PDT

To install the modified load balancer image, use the ‘euca-install-load-balancer‘ command:

[root@odc-f-13 ~]# euca-install-load-balancer -t ~/eucalyptus-lb/usr/share/eucalyptus-load-balancer-image/eucalyptus-load-balancer-image-updated.tgz
Decompressing tarball: eucalyptus-lb/usr/share/eucalyptus-load-balancer-image/eucalyptus-load-balancer-image-updated.tgz
Bundling and uploading image to bucket: loadbalancer-v3
Registering image manifest: loadbalancer-v3/eucalyptus-load-balancer-image.img.manifest.xml
Registered image: emi-BCAD86BE
PROPERTY loadbalancing.loadbalancer_emi emi-BCAD86BE was emi-F0D5828C

Confirm that the new EMI is enabled:

[root@odc-f-13 ~]# euca-install-load-balancer --list
Currently Installed Load Balancer Bundles:

Version 1
emi-FA373789 (loadbalancer_v1/eucalyptus-load-balancer-image-1.0.4-164.img.manifest.xml)
 Installed on 2014-05-20 at 07:12:18 PDT

Version 2
emi-F0D5828C (loadbalancer-v2/eucalyptus-load-balancer-image.img.manifest.xml)
 Installed on 2014-05-28 at 11:10:03 PDT

Version 3 (enabled)
emi-BCAD86BE (loadbalancer-v3/eucalyptus-load-balancer-image-updated.img.manifest.xml)
 Installed on 2014-07-06 at 18:02:20 PDT

Confirming the Updated Load Balancer Configuration

To confirm the changes, make sure the cloud property ‘loadbalancing.loadbalancer_vm_keyname‘ has a defined value for debugging purposes, then create a Eucalyptus Elastic Load Balancer:

[root@odc-f-13 ~]# eulb-create-lb TestLoadBalancer -z ViciousLiesAndDangerousRumors -l "lb-port=80, protocol=HTTP, instance-port=80, instance-protocol=HTTP"
DNS_NAME TestLoadBalancer-408396244283.elb.acme.eucalyptus-systems.com

Confirm that the load balancer instance is running (only the cloud administrator can see the load balancing instance IDs), and authorize port 22 (SSH) to the instance:

[root@odc-f-13 ~]# euca-describe-instances
RESERVATION r-C27D6F37 944786667073 euca-internal-408396244283-TestLoadBalancer
INSTANCE i-A0DBA47D emi-BCAD86BE euca-10-104-6-233.bigboi.acme.eucalyptus-systems.com euca-172-18-246-58.bigboi.internal running euca-elb 0 m1.medium 2014-07-07T01:08:32.400Z ViciousLiesAndDangerousRumors monitoring-enabled 10.104.6.233 172.18.246.58 instance-store hvm bce54dd2-a9af-4587-9421-457e22dda5ff_ViciousLiesAndDangerousR_1 sg-4F469747 arn:aws:iam::944786667073:instance-profile/internal/loadbalancer/loadbalancer-vm-408396244283-TestLoadBalancer
TAG instance i-A0DBA47D Name loadbalancer-resources
TAG instance i-A0DBA47D aws:autoscaling:groupName asg-euca-internal-elb-408396244283-TestLoadBalancer
TAG instance i-A0DBA47D euca:node 10.105.10.7

[root@odc-f-13 ~]# euca-authorize -P tcp -p ssh euca-internal-408396244283-TestLoadBalancer
GROUP euca-internal-408396244283-TestLoadBalancer
PERMISSION euca-internal-408396244283-TestLoadBalancer ALLOWS tcp 22 22 FROM CIDR 0.0.0.0/0

After SSHing into the load balancer instance, confirm that the /var/lib/load-balancer-servo/euca_haproxy.conf file has the updated changes:

[root@odc-f-13 ~]# ssh -i euca-elb.priv root@euca-10-104-6-233.bigboi.acme.eucalyptus-systems.com
The authenticity of host 'euca-10-104-6-233.bigboi.acme.eucalyptus-systems.com (10.104.6.233)' can't be established.
RSA key fingerprint is e3:9b:80:e2:f3:12:a3:0b:f0:5c:7c:6b:bc:d8:9d:77.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added 'euca-10-104-6-233.bigboi.acme.eucalyptus-systems.com' (RSA) to the list of known hosts.
Warning: the RSA host key for 'euca-10-104-6-233.bigboi.acme.eucalyptus-systems.com' differs from the key for the IP address '10.104.6.233'
Offending key for IP in /root/.ssh/known_hosts:9
Are you sure you want to continue connecting (yes/no)? yes
[root@euca-172-18-246-58 ~]# ps aux | grep haprox
servo 1027 0.0 0.2 58648 3020 ? Ss 01:09 0:00 /usr/sbin/haproxy -f /var/lib/load-balancer-servo/euca_haproxy.conf -p /var/run/load-balancer-servo/haproxy.pid -V -C /var/lib/load-balancer-servo -D

[root@euca-172-18-246-58 ~]# cat /var/lib/load-balancer-servo/euca_haproxy.conf
global
 maxconn 100000
 ulimit-n 655360
 pidfile /var/run/haproxy.pid
 #drop privileges after port binding
 user servo
 group servo

defaults
 timeout connect 5s
 timeout client 2m
 timeout server 2m
 timeout http-keep-alive 10s
 timeout queue 1m
 timeout check 5s
 retries 3
 option dontlognull
 option redispatch
 option http-server-close # affects KA on/off

frontend http-80
 # lb-TestLoadBalancer
 mode http
 option forwardfor except 127.0.0.1
 bind 0.0.0.0:80
 log /var/lib/load-balancer-servo/haproxy.sock local2 info
 log-format httplog\ %f\ %b\ %s\ %ST\ %ts\ %Tq\ %Tw\ %Tc\ %Tr\ %Tt
 default_backend backend-http-80

backend backend-http-80
 mode http
 balance roundrobin

The ‘default‘ section should contain all the modifications made to the /etc/load-balancer-servo/haproxy_template.conf file.   The Eucalyptus Load Balancer will now utilize the updated changes needed to address the desired performance with the various backend applications that will be used with the load balancer.