Background
For the Elastic Load Balancing service, Eucalyptus utilizes an HAProxy instance. The load balancer image contains the following version of haproxy (as of Eucalyptus 4.0.0):
# /usr/sbin/haproxy -vv HA-Proxy version 1.5-dev21-6b07bf7 +2013/12/17 Copyright 2000-2013 Willy Tarreau <w@1wt.eu> Build options : TARGET = linux2628 CPU = generic CC = gcc CFLAGS = -O2 -g -fno-strict-aliasing OPTIONS = USE_LINUX_TPROXY=1 USE_REGPARM=1 USE_OPENSSL=1 USE_PCRE=1 Default settings : maxconn = 2000, bufsize = 16384, maxrewrite = 8192, maxpollevents = 200 Encrypted password support via crypt(3): yes Built without zlib support (USE_ZLIB not set) Compression algorithms supported : identity Built with OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 Running on OpenSSL version : OpenSSL 1.0.1e-fips 11 Feb 2013 OpenSSL library supports TLS extensions : yes OpenSSL library supports SNI : yes OpenSSL library supports prefer-server-ciphers : yes Built with PCRE version : 7.8 2008-09-05 PCRE library supports JIT : no (USE_PCRE_JIT not set) Built with transparent proxy support using: IP_TRANSPARENT IPV6_TRANSPARENT IP_FREEBIND Available polling systems : epoll : pref=300, test result OK poll : pref=200, test result OK select : pref=150, test result OK Total: 3 (3 usable), will use epoll.
By default, the following HAProxy configuration options are used by the Eucalyptus Load Balancer image (defined by the Eucalyptus load-balancer-servo application, which is the controlling mechanism for Eucalyptus load balancing):
#template global maxconn 100000 ulimit-n 655360 pidfile /var/run/haproxy.pid #drop privileges after port binding user servo group servo defaults contimeout 1000 clitimeout 10000 srvtimeout 10000 option http-server-close # affects KA on/off
Given what backend applications will be used with the Eucalyptus Load Balancer, these settings may not be sufficient.
The goal of this entry is to demonstrate how to customize the Eucalyptus Load Balancer image configuration to handle various backend applications that will be used with the load balancer.
Prerequisites
In order to customize the Eucalyptus Load Balancer image, the credentials of the cloud administrator (Eucalyptus IAM credentials of a user under the ‘eucalyptus’ account) must be used. These credentials are needed to do the following:
- Modify the loadbalancing.loadbalancer_emi cloud property
- Use the ‘euca-install-load-balancer‘ command to install the modified load balancer image
Since the cloud administrator credentials will be used, there will be no need to define any Eucalyptus IAM policies.
To mount, and modify the Eucalyptus Load Balancer image, the following Linux tools are needed:
The examples in this blog were all done on CentOS 6.5 machine where the eucalyptus-load-balancer-image package has been installed. This package contains the ‘euca-install-load-balancer’ command.
Obtaining the Eucalyptus Elastic Load Balancer Image
There are couple of ways to obtain the Eucalyptus Load Balancer image:
- Use the eucalyptus/load-balancer-image repository on Github
- Download the eucalyptus-load-balancer-image RPM from downloads.eucalyptus.com
This blog entry will use the eucalyptus-load-balancer-image RPM, and update the image according. To get started, create a directory (in this example ‘eucalyptus-lb’), and download the latest eucalyptus load balancer image RPM from downloads.eucalyptus.com:
[root@odc-f-13 ~]# mkdir eucalyptus-lb; cd eucalyptus-lb [root@odc-f-13 eucalyptus-lb]# wget http://downloads.eucalyptus.com/software/eucalyptus/4.0/centos/6/x86_64/eucalyptus-load-balancer-image-1.1.0-0.212.el6.x86_64.rpm
Once the RPM package has been downloaded, unpack the RPM:
[root@odc-f-13 eucalyptus-lb]# rpm2cpio eucalyptus-load-balancer-image-1.1.0-0.212.el6.x86_64.rpm | cpio --extract --make-directories --preserve-modification-time --verbose ./usr/bin/euca-install-load-balancer ./usr/share/doc/eucalyptus-load-balancer-image-1.1.0 ./usr/share/doc/eucalyptus-load-balancer-image-1.1.0/IMAGE-LICENSE ./usr/share/doc/eucalyptus-load-balancer-image-1.1.0/eucalyptus-load-balancer-image.ks ./usr/share/eucalyptus-load-balancer-image ./usr/share/eucalyptus-load-balancer-image/eucalyptus-load-balancer-image-1.1.0-212.tgz 559369 blocks
After unpacking the RPM, change directory to ~/usr/share/eucalyptus-load-balancer-image, and decompress the eucalyptus-load-balancer-image-1.1.0-212.tgz file to obtain the Eucalyptus Load Balancer image:
[root@odc-f-13 eucalyptus-lb]# cd usr/share/eucalyptus-load-balancer-image [root@odc-f-13 eucalyptus-load-balancer-image]# tar -xzvf eucalyptus-load-balancer-image-1.1.0-212.tgz eucalyptus-load-balancer-image.img
Now that the image is available, we can modify it accordingly.
Modifying the Eucalyptus Load Balancer Image
To modify the Eucalyptus Load Balancer image, the image needs to be mounted to a loopback device, as demonstrated below:
[root@odc-f-13 eucalyptus-load-balancer-image]# mkdir /mnt/centos [root@odc-f-13 eucalyptus-load-balancer-image]# losetup /dev/loop0 eucalyptus-load-balancer-image.img [root@odc-f-13 eucalyptus-load-balancer-image]# kpartx -av /dev/loop0 add map loop0p1 (253:2): 0 3145728 linear /dev/loop0 2048 [root@odc-f-13 eucalyptus-load-balancer-image]# mount /dev/mapper/loop0p1 /mnt/centos [root@odc-f-13 eucalyptus-load-balancer-image]# chroot /mnt/centos [root@odc-f-13 /]#
We will modify and add the following HAProxy options under the ‘default’ section in the /etc/load-balancer-servo/haproxy_template.conf file. For information about these options, please refer to the HAProxy 1.5 documentation:
- replace ‘srvtimeout‘ with ‘timeout server‘ since ‘srvtimeout‘ is deprecated, and set the value to ‘2m‘
- replace ‘clitimeout‘ with ‘timeout client‘ since ‘clitimeout‘ is deprecated, and set the value to ‘2m‘
- replace ‘contimeout ‘ with ‘timeout connect‘ since ‘contimeout‘ is deprecated, and set the value to ‘5s‘
- add ‘timeout http-keep-alive‘ with the value of ‘10s‘
- add ‘timeout queue‘ with the value of ‘1m‘
- add ‘timeout check‘ with the value of ‘5s‘
- add ‘retries‘ with the value of ‘3‘
- add the following options to not log null connections, and to enable session redistribution in case of failure:
- ‘option dontlognull‘
- ‘option redispatch‘
The /etc/load-balancer-servo/haproxy_template.conf should look similar to the following after all the desired attributes are added:
[root@odc-f-13 /]# cat /etc/load-balancer-servo/haproxy_template.conf #template global maxconn 100000 ulimit-n 655360 pidfile /var/run/haproxy.pid #drop privileges after port binding user servo group servo defaults timeout connect 5s timeout client 2m timeout server 2m timeout http-keep-alive 10s timeout queue 1m timeout check 5s retries 3 option dontlognull option redispatch option http-server-close # affects KA on/off
(Note: Depending upon what edits are being done to the HAProxy configuration settings, there may also be a need to edit the /etc/sysctl.conf file to help get the desired behavior from the Eucalyptus Load Balancer. For example, the following sysctl properties can be edited to increase/decrease TCP timeouts:
- net.ipv4.tcp_keepalive_time
- net.ipv4.tcp_keepalive_intvl
- net.ipv4.tcp_keepalive_probes
For more information about editing sysctl values, the documentation from RedHat can be referenced.)
Once all edits are completed, confirm that the configuration file is correct, exit out of the chroot environment and unmount the image:
[root@odc-f-13 /]# /usr/sbin/haproxy -c -f /etc/load-balancer-servo/haproxy_template.conf Configuration file has no error but will not start (no listener) => exit(2). [root@odc-f-13 eucalyptus-load-balancer-image]# umount /mnt/centos [root@odc-f-13 eucalyptus-load-balancer-image]# kpartx -dv /dev/loop0 del devmap : loop0p1 [root@odc-f-13 eucalyptus-load-balancer-image]# losetup -d /dev/loop0
Installing the New Eucalyptus Load Balancer Image
After the image has been unmounted, create a new tar-gzipped file that contains the modified Eucalyptus Load Balancer image:
[root@odc-f-13 eucalyptus-load-balancer-image]# tar -zcvf eucalyptus-load-balancer-image-updated.tgz eucalyptus-load-balancer-image.img
Next, make sure the cloud administrator credentials are sourced and check the cloud properties for the Eucalyptus Load Balancer service:
[root@odc-f-13 eucalyptus-load-balancer-image]# cd
[root@odc-f-13 ~]# source eucarc
[root@odc-f-13 ~]# euca-describe-properties | grep load
PROPERTY ViciousLiesAndDangerousRumors.storage.maxconcurrentsnapshotuploads 3
PROPERTY ViciousLiesAndDangerousRumors.storage.snapshotuploadtimeoutinhours 48
PROPERTY authentication.credential_download_host_match {}
PROPERTY loadbalancing.loadbalancer_app_cookie_duration 24
PROPERTY loadbalancing.loadbalancer_dns_subdomain elb
PROPERTY loadbalancing.loadbalancer_emi emi-F0D5828C
PROPERTY loadbalancing.loadbalancer_instance_type m1.medium
PROPERTY loadbalancing.loadbalancer_num_vm 1
PROPERTY loadbalancing.loadbalancer_restricted_ports 22
PROPERTY loadbalancing.loadbalancer_vm_keyname euca-elb
PROPERTY loadbalancing.loadbalancer_vm_ntp_server pool.ntp.org
Check to see what load balancer images are enabled:
[root@odc-f-13 ~]# euca-install-load-balancer --list Currently Installed Load Balancer Bundles: Version 1 emi-FA373789 (loadbalancer_v1/eucalyptus-load-balancer-image-1.0.4-164.img.manifest.xml) Installed on 2014-05-20 at 07:12:18 PDT Version 2 (enabled) emi-F0D5828C (loadbalancer-v2/eucalyptus-load-balancer-image.img.manifest.xml) Installed on 2014-05-28 at 11:10:03 PDT
To install the modified load balancer image, use the ‘euca-install-load-balancer‘ command:
[root@odc-f-13 ~]# euca-install-load-balancer -t ~/eucalyptus-lb/usr/share/eucalyptus-load-balancer-image/eucalyptus-load-balancer-image-updated.tgz Decompressing tarball: eucalyptus-lb/usr/share/eucalyptus-load-balancer-image/eucalyptus-load-balancer-image-updated.tgz Bundling and uploading image to bucket: loadbalancer-v3 Registering image manifest: loadbalancer-v3/eucalyptus-load-balancer-image.img.manifest.xml Registered image: emi-BCAD86BE PROPERTY loadbalancing.loadbalancer_emi emi-BCAD86BE was emi-F0D5828C
Confirm that the new EMI is enabled:
[root@odc-f-13 ~]# euca-install-load-balancer --list Currently Installed Load Balancer Bundles: Version 1 emi-FA373789 (loadbalancer_v1/eucalyptus-load-balancer-image-1.0.4-164.img.manifest.xml) Installed on 2014-05-20 at 07:12:18 PDT Version 2 emi-F0D5828C (loadbalancer-v2/eucalyptus-load-balancer-image.img.manifest.xml) Installed on 2014-05-28 at 11:10:03 PDT Version 3 (enabled) emi-BCAD86BE (loadbalancer-v3/eucalyptus-load-balancer-image-updated.img.manifest.xml) Installed on 2014-07-06 at 18:02:20 PDT
Confirming the Updated Load Balancer Configuration
To confirm the changes, make sure the cloud property ‘loadbalancing.loadbalancer_vm_keyname‘ has a defined value for debugging purposes, then create a Eucalyptus Elastic Load Balancer:
[root@odc-f-13 ~]# eulb-create-lb TestLoadBalancer -z ViciousLiesAndDangerousRumors -l "lb-port=80, protocol=HTTP, instance-port=80, instance-protocol=HTTP" DNS_NAME TestLoadBalancer-408396244283.elb.acme.eucalyptus-systems.com
Confirm that the load balancer instance is running (only the cloud administrator can see the load balancing instance IDs), and authorize port 22 (SSH) to the instance:
[root@odc-f-13 ~]# euca-describe-instances RESERVATION r-C27D6F37 944786667073 euca-internal-408396244283-TestLoadBalancer INSTANCE i-A0DBA47D emi-BCAD86BE euca-10-104-6-233.bigboi.acme.eucalyptus-systems.com euca-172-18-246-58.bigboi.internal running euca-elb 0 m1.medium 2014-07-07T01:08:32.400Z ViciousLiesAndDangerousRumors monitoring-enabled 10.104.6.233 172.18.246.58 instance-store hvm bce54dd2-a9af-4587-9421-457e22dda5ff_ViciousLiesAndDangerousR_1 sg-4F469747 arn:aws:iam::944786667073:instance-profile/internal/loadbalancer/loadbalancer-vm-408396244283-TestLoadBalancer TAG instance i-A0DBA47D Name loadbalancer-resources TAG instance i-A0DBA47D aws:autoscaling:groupName asg-euca-internal-elb-408396244283-TestLoadBalancer TAG instance i-A0DBA47D euca:node 10.105.10.7
[root@odc-f-13 ~]# euca-authorize -P tcp -p ssh euca-internal-408396244283-TestLoadBalancer GROUP euca-internal-408396244283-TestLoadBalancer PERMISSION euca-internal-408396244283-TestLoadBalancer ALLOWS tcp 22 22 FROM CIDR 0.0.0.0/0
After SSHing into the load balancer instance, confirm that the /var/lib/load-balancer-servo/euca_haproxy.conf file has the updated changes:
[root@odc-f-13 ~]# ssh -i euca-elb.priv root@euca-10-104-6-233.bigboi.acme.eucalyptus-systems.com The authenticity of host 'euca-10-104-6-233.bigboi.acme.eucalyptus-systems.com (10.104.6.233)' can't be established. RSA key fingerprint is e3:9b:80:e2:f3:12:a3:0b:f0:5c:7c:6b:bc:d8:9d:77. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'euca-10-104-6-233.bigboi.acme.eucalyptus-systems.com' (RSA) to the list of known hosts. Warning: the RSA host key for 'euca-10-104-6-233.bigboi.acme.eucalyptus-systems.com' differs from the key for the IP address '10.104.6.233' Offending key for IP in /root/.ssh/known_hosts:9 Are you sure you want to continue connecting (yes/no)? yes [root@euca-172-18-246-58 ~]# ps aux | grep haprox servo 1027 0.0 0.2 58648 3020 ? Ss 01:09 0:00 /usr/sbin/haproxy -f /var/lib/load-balancer-servo/euca_haproxy.conf -p /var/run/load-balancer-servo/haproxy.pid -V -C /var/lib/load-balancer-servo -D
[root@euca-172-18-246-58 ~]# cat /var/lib/load-balancer-servo/euca_haproxy.conf global maxconn 100000 ulimit-n 655360 pidfile /var/run/haproxy.pid #drop privileges after port binding user servo group servo defaults timeout connect 5s timeout client 2m timeout server 2m timeout http-keep-alive 10s timeout queue 1m timeout check 5s retries 3 option dontlognull option redispatch option http-server-close # affects KA on/off frontend http-80 # lb-TestLoadBalancer mode http option forwardfor except 127.0.0.1 bind 0.0.0.0:80 log /var/lib/load-balancer-servo/haproxy.sock local2 info log-format httplog\ %f\ %b\ %s\ %ST\ %ts\ %Tq\ %Tw\ %Tc\ %Tr\ %Tt default_backend backend-http-80 backend backend-http-80 mode http balance roundrobin
The ‘default‘ section should contain all the modifications made to the /etc/load-balancer-servo/haproxy_template.conf file. The Eucalyptus Load Balancer will now utilize the updated changes needed to address the desired performance with the various backend applications that will be used with the load balancer.
More Mind Spew-age from Harold Spencer Jr. 