Background

From the cloud user’s perspective, the Eucalyptus Load Balancer is a “black box“.  The only interaction cloud user’s have with the Eucalyptus Load Balancer is through the eulb-* commands in euca2ools or the AWS Elastic Load Balancing API tools.   In Eucalyptus 3.4 and greater, the cloud administrator (any user under the ‘eucalyptus’ account) has the ability to access the instance that implements the load balancing solution used by the Eucalyptus Load Balancing service.  This access can be used to help troubleshoot the Eucalyptus Load Balancer if there are any issues reported by the cloud user.

The Eucalyptus Load Balancer utilizes HAProxy to implement the load balancing solution.  HAProxy has a cool feature to enable the ability to display a statistics page for the HAProxy application.  Enabling this feature to the Eucalyptus Load Balancer can help cloud administrators obtain valuable information from the load balancer in the following areas:

• Network traffic to the backend instances registered with the load balancer
• Network traffic to the load balancer
• Triaging any Eucalyptus Load Balancer behavior associated with Eucalyptus CloudWatch alarms

Before getting into the details, I would like to thank Nathan Evans for his entry entitled “Cultural learnings of HA-Proxy, for make benefit…“, which helped influence this blog entry.   Now on to the fun stuff….

Prerequisites

The prerequisites for this blog entry are pretty straight forward – just read my previous entry entitled “Customizing Eucalyptus Load Balancer for Eucalyptus 4.0“.  To enable the web UI stats page, we will just add information to the /etc/load-balancer-servo/haproxy_template.conf file in the load balancer image.

In addition, the cloud administrator credentials will be needed, along with euca2ools 3.1 installed.

Enabling the HAProxy Web Statistics Page

After downloading and mounting the Eucalyptus Load Balancer image (as mentioned in my previous blog entry), to enable the HAProxy web statistics page, update the /etc/load-balancer-servo/haproxy_template.conf to look like the following:

[root@odc-f-13 /]# cat etc/load-balancer-servo/haproxy_template.conf
#template
global
 maxconn 100000
 ulimit-n 655360
 pidfile /var/run/haproxy.pid

#drop privileges after port binding
 user servo
 group servo

defaults
 timeout connect 5s
 timeout client 2m
 timeout server 2m
 timeout http-keep-alive 10s
 timeout queue 1m
 timeout check 5s
 retries 3
 option dontlognull
 option redispatch
 option http-server-close # affects KA on/off

 userlist UsersFor_HAProxyStatistics
  group admin users admin
  user admin insecure-password pwd*4admin
  user stats insecure-password pwd*4stats

listen HAProxy-Statistics *:81
 mode http
 stats enable
 stats uri /haproxy?stats
 stats refresh 60s
 stats show-node
 stats show-legends
 acl AuthOkay_ReadOnly http_auth(UsersFor_HAProxyStatistics)
 acl AuthOkay_Admin http_auth_group(UsersFor_HAProxyStatistics) admin
 stats http-request auth realm HAProxy-Statistics unless AuthOkay_ReadOnly
 stats admin if AuthOkay_Admin

For more information regarding these options, please refer to the HAProxy 1.5 documentation.  The key options here are as follows:

• The port defined in the ‘listen’ section – listen HAProxy-Statistics *:81
• The username and passwords defined in the ‘userlist‘ subsection under the ‘defaults’ section.
• The URI defined in the ‘listen’ section – stats uri /haproxy?stats

After making these changes, confirm that there aren’t any configuration file errors:

[root@odc-f-13 /]# /usr/sbin/haproxy -c -f etc/load-balancer-servo/haproxy_template.conf
 Configuration file is valid

Next, unmount the image, and tar-gzip the image:

[root@odc-f-13 eucalyptus-load-balancer-image]# umount /mnt/centos
[root@odc-f-13 eucalyptus-load-balancer-image]# kpartx -dv /dev/loop0
del devmap : loop0p1
[root@odc-f-13 eucalyptus-load-balancer-image]# losetup -d /dev/loop0
[root@odc-f-13 eucalyptus-load-balancer-image]# tar -zcvf eucalyptus-load-balancer-image-monitored.tgz eucalyptus-load-balancer-image.img
eucalyptus-load-balancer-image.img

Use euca-install-load-balancer to upload the new image:

[root@odc-f-13 eucalyptus-load-balancer-image]# cd
[root@odc-f-13 ~]# euca-install-load-balancer --list
Currently Installed Load Balancer Bundles:

Version 2 (enabled)
emi-F0D5828C (loadbalancer-v2/eucalyptus-load-balancer-image.img.manifest.xml)
 Installed on 2014-05-28 at 11:10:03 PDT

[root@odc-f-13 ~]# euca-install-load-balancer -t eucalyptus-lb/usr/share/eucalyptus-load-balancer-image/eucalyptus-load-balancer-image-monitored.tgz
Decompressing tarball: eucalyptus-lb/usr/share/eucalyptus-load-balancer-image/eucalyptus-load-balancer-image-monitored.tgz
Bundling and uploading image to bucket: loadbalancer-v3
Registering image manifest: loadbalancer-v3/eucalyptus-load-balancer-image.img.manifest.xml
Registered image: emi-DB150EC0
PROPERTY loadbalancing.loadbalancer_emi emi-DB150EC0 was emi-F0D5828C

Load Balancing Support is Enabled
[root@odc-f-13 ~]# euca-install-load-balancer --list
Currently Installed Load Balancer Bundles:

Version 2
emi-F0D5828C (loadbalancer-v2/eucalyptus-load-balancer-image.img.manifest.xml)
 Installed on 2014-05-28 at 11:10:03 PDT

Version 3 (enabled)
emi-DB150EC0 (loadbalancer-v3/eucalyptus-load-balancer-image.img.manifest.xml)
 Installed on 2014-07-08 at 18:38:29 PDT

Testing the Eucalyptus Load Balancer Statistics Page

To view the HAProxy statistics page, create a Eucalyptus Load Balancer instance by using eulb-create-lb:

[root@odc-f-13 ~]# eulb-create-lb TestLoadBalancer -z ViciousLiesAndDangerousRumors -l "lb-port=80, protocol=HTTP, instance-port=80, instance-protocol=HTTP"
DNS_NAME TestLoadBalancer-408396244283.elb.acme.eucalyptus-systems.com

[root@odc-f-13 ~]# euca-describe-instances
RESERVATION r-06DF089F 944786667073 euca-internal-408396244283-TestLoadBalancer
INSTANCE i-3DA342C2 emi-DB150EC0 euca-10-104-6-233.bigboi.acme.eucalyptus-systems.com euca-172-18-229-187.bigboi.internal running euca-elb 0 m1.medium 2014-07-09T01:45:11.753Z ViciousLiesAndDangerousRumors monitoring-enabled 10.104.6.233 172.18.229.187 instance-store hvm 8ba248ae-dbeb-41ce-97df-fb13b91a337b_ViciousLiesAndDangerousR_1 sg-3EA4ADEC arn:aws:iam::944786667073:instance-profile/internal/loadbalancer/loadbalancer-vm-408396244283-TestLoadBalancer
TAG instance i-3DA342C2 Name loadbalancer-resources
TAG instance i-3DA342C2 aws:autoscaling:groupName asg-euca-internal-elb-408396244283-TestLoadBalancer
TAG instance i-3DA342C2 euca:node 10.105.1.188

Since the web statistics page is configured to display on port 81, use euca-authorize to allow access to that port in the load balancer’s security group.  I recommend limiting access to the port for security reasons.  In the example below, access is limited to only the client 192.168.30.25:

[root@odc-f-13 ~]# euca-authorize -P tcp -p 81 -s 192.168.30.25/32 euca-internal-408396244283-TestLoadBalancer
 GROUP euca-internal-408396244283-TestLoadBalancer
 PERMISSION euca-internal-408396244283-TestLoadBalancer ALLOWS tcp 81 81 FROM CIDR 192.168.30.25/32

Finally, use a browser on the authorized client to view the statistics page on the load balancer.  In this example, the URL – http://testloadbalancer-408396244283.elb.acme.eucalyptus-systems.com:81/haproxy?stats – will be used.  Use the username and password credentials that were added to to the HAProxy configuration file to view the page.  It should look similar to the screenshot below:

 

Thats it!  For any load balancer thats launched on the Eucalyptus 4.0 cloud, the cloud administrator will be able to display statistics of the load balancer.  This is also something that the cloud administrator can provide to cloud users as a service.  By leveraging restrictions placed in security groups of the load balancer, cloud administrators can limit access to the statistics page based upon the source IP addresses of the cloud users’ client machine(s).

Enjoy!