Background
From the cloud user’s perspective, the Eucalyptus Load Balancer is a “black box“. The only interaction cloud user’s have with the Eucalyptus Load Balancer is through the eulb-* commands in euca2ools or the AWS Elastic Load Balancing API tools. In Eucalyptus 3.4 and greater, the cloud administrator (any user under the ‘eucalyptus’ account) has the ability to access the instance that implements the load balancing solution used by the Eucalyptus Load Balancing service. This access can be used to help troubleshoot the Eucalyptus Load Balancer if there are any issues reported by the cloud user.
The Eucalyptus Load Balancer utilizes HAProxy to implement the load balancing solution. HAProxy has a cool feature to enable the ability to display a statistics page for the HAProxy application. Enabling this feature to the Eucalyptus Load Balancer can help cloud administrators obtain valuable information from the load balancer in the following areas:
- Network traffic to the backend instances registered with the load balancer
- Network traffic to the load balancer
- Triaging any Eucalyptus Load Balancer behavior associated with Eucalyptus CloudWatch alarms
Before getting into the details, I would like to thank Nathan Evans for his entry entitled “Cultural learnings of HA-Proxy, for make benefit…“, which helped influence this blog entry. Now on to the fun stuff….
Prerequisites
The prerequisites for this blog entry are pretty straight forward – just read my previous entry entitled “Customizing Eucalyptus Load Balancer for Eucalyptus 4.0“. To enable the web UI stats page, we will just add information to the /etc/load-balancer-servo/haproxy_template.conf file in the load balancer image.
In addition, the cloud administrator credentials will be needed, along with euca2ools 3.1 installed.
Enabling the HAProxy Web Statistics Page
After downloading and mounting the Eucalyptus Load Balancer image (as mentioned in my previous blog entry), to enable the HAProxy web statistics page, update the /etc/load-balancer-servo/haproxy_template.conf to look like the following:
[root@odc-f-13 /]# cat etc/load-balancer-servo/haproxy_template.conf #template global maxconn 100000 ulimit-n 655360 pidfile /var/run/haproxy.pid #drop privileges after port binding user servo group servo defaults timeout connect 5s timeout client 2m timeout server 2m timeout http-keep-alive 10s timeout queue 1m timeout check 5s retries 3 option dontlognull option redispatch option http-server-close # affects KA on/off userlist UsersFor_HAProxyStatistics group admin users admin user admin insecure-password pwd*4admin user stats insecure-password pwd*4stats listen HAProxy-Statistics *:81 mode http stats enable stats uri /haproxy?stats stats refresh 60s stats show-node stats show-legends acl AuthOkay_ReadOnly http_auth(UsersFor_HAProxyStatistics) acl AuthOkay_Admin http_auth_group(UsersFor_HAProxyStatistics) admin stats http-request auth realm HAProxy-Statistics unless AuthOkay_ReadOnly stats admin if AuthOkay_Admin
For more information regarding these options, please refer to the HAProxy 1.5 documentation. The key options here are as follows:
- The port defined in the ‘listen’ section – listen HAProxy-Statistics *:81
- The username and passwords defined in the ‘userlist‘ subsection under the ‘defaults’ section.
- The URI defined in the ‘listen’ section – stats uri /haproxy?stats
After making these changes, confirm that there aren’t any configuration file errors:
[root@odc-f-13 /]# /usr/sbin/haproxy -c -f etc/load-balancer-servo/haproxy_template.conf Configuration file is valid
Next, unmount the image, and tar-gzip the image:
[root@odc-f-13 eucalyptus-load-balancer-image]# umount /mnt/centos [root@odc-f-13 eucalyptus-load-balancer-image]# kpartx -dv /dev/loop0 del devmap : loop0p1 [root@odc-f-13 eucalyptus-load-balancer-image]# losetup -d /dev/loop0 [root@odc-f-13 eucalyptus-load-balancer-image]# tar -zcvf eucalyptus-load-balancer-image-monitored.tgz eucalyptus-load-balancer-image.img eucalyptus-load-balancer-image.img Use euca-install-load-balancer to upload the new image: [root@odc-f-13 eucalyptus-load-balancer-image]# cd [root@odc-f-13 ~]# euca-install-load-balancer --list Currently Installed Load Balancer Bundles: Version 2 (enabled) emi-F0D5828C (loadbalancer-v2/eucalyptus-load-balancer-image.img.manifest.xml) Installed on 2014-05-28 at 11:10:03 PDT [root@odc-f-13 ~]# euca-install-load-balancer -t eucalyptus-lb/usr/share/eucalyptus-load-balancer-image/eucalyptus-load-balancer-image-monitored.tgz Decompressing tarball: eucalyptus-lb/usr/share/eucalyptus-load-balancer-image/eucalyptus-load-balancer-image-monitored.tgz Bundling and uploading image to bucket: loadbalancer-v3 Registering image manifest: loadbalancer-v3/eucalyptus-load-balancer-image.img.manifest.xml Registered image: emi-DB150EC0 PROPERTY loadbalancing.loadbalancer_emi emi-DB150EC0 was emi-F0D5828C Load Balancing Support is Enabled [root@odc-f-13 ~]# euca-install-load-balancer --list Currently Installed Load Balancer Bundles: Version 2 emi-F0D5828C (loadbalancer-v2/eucalyptus-load-balancer-image.img.manifest.xml) Installed on 2014-05-28 at 11:10:03 PDT Version 3 (enabled) emi-DB150EC0 (loadbalancer-v3/eucalyptus-load-balancer-image.img.manifest.xml) Installed on 2014-07-08 at 18:38:29 PDT
Testing the Eucalyptus Load Balancer Statistics Page
To view the HAProxy statistics page, create a Eucalyptus Load Balancer instance by using eulb-create-lb:
[root@odc-f-13 ~]# eulb-create-lb TestLoadBalancer -z ViciousLiesAndDangerousRumors -l "lb-port=80, protocol=HTTP, instance-port=80, instance-protocol=HTTP" DNS_NAME TestLoadBalancer-408396244283.elb.acme.eucalyptus-systems.com [root@odc-f-13 ~]# euca-describe-instances RESERVATION r-06DF089F 944786667073 euca-internal-408396244283-TestLoadBalancer INSTANCE i-3DA342C2 emi-DB150EC0 euca-10-104-6-233.bigboi.acme.eucalyptus-systems.com euca-172-18-229-187.bigboi.internal running euca-elb 0 m1.medium 2014-07-09T01:45:11.753Z ViciousLiesAndDangerousRumors monitoring-enabled 10.104.6.233 172.18.229.187 instance-store hvm 8ba248ae-dbeb-41ce-97df-fb13b91a337b_ViciousLiesAndDangerousR_1 sg-3EA4ADEC arn:aws:iam::944786667073:instance-profile/internal/loadbalancer/loadbalancer-vm-408396244283-TestLoadBalancer TAG instance i-3DA342C2 Name loadbalancer-resources TAG instance i-3DA342C2 aws:autoscaling:groupName asg-euca-internal-elb-408396244283-TestLoadBalancer TAG instance i-3DA342C2 euca:node 10.105.1.188
Since the web statistics page is configured to display on port 81, use euca-authorize to allow access to that port in the load balancer’s security group. I recommend limiting access to the port for security reasons. In the example below, access is limited to only the client 192.168.30.25:
[root@odc-f-13 ~]# euca-authorize -P tcp -p 81 -s 192.168.30.25/32 euca-internal-408396244283-TestLoadBalancer GROUP euca-internal-408396244283-TestLoadBalancer PERMISSION euca-internal-408396244283-TestLoadBalancer ALLOWS tcp 81 81 FROM CIDR 192.168.30.25/32
Finally, use a browser on the authorized client to view the statistics page on the load balancer. In this example, the URL – http://testloadbalancer-408396244283.elb.acme.eucalyptus-systems.com:81/haproxy?stats – will be used. Use the username and password credentials that were added to to the HAProxy configuration file to view the page. It should look similar to the screenshot below:
Thats it! For any load balancer thats launched on the Eucalyptus 4.0 cloud, the cloud administrator will be able to display statistics of the load balancer. This is also something that the cloud administrator can provide to cloud users as a service. By leveraging restrictions placed in security groups of the load balancer, cloud administrators can limit access to the statistics page based upon the source IP addresses of the cloud users’ client machine(s).
Enjoy!
More Mind Spew-age from Harold Spencer Jr. 