More Mind Spew-age from Harold Spencer Jr.

As a follow-up to my last Neo4j, AWS/Eucalyptus blog, this entry demonstrates another great example of AWS/Eucalyptus fidelity by using Ansible to deploy a Neo4j High Available cluster.

Neo4j – Graph Database

Amazon AWS EC2

Eucalyptus Systems Inc.

Pre-requisites

In order to use this Ansible playbook on AWS/Eucalyptus, the following is needed:

• An AWS or Eucalyptus account, with a user’s access key and secret access key.
• EC2 IAM Policy to allow launching of instances, and authorize ports in security group
• Ubuntu Cloud Image (Precise 12.04)
• EC2 API Client Tools
• git repository tools

Before deploying the cluster, a security group needs to be created that the cluster will use.  The security group must allow the following:

• port 22 (SSH)
• all instances part of the security group allowed to community with each other (ports 0 - 65535)

To create the security group and authorize the ports, make sure the user’s access key, secret access key, and EC2 URL are noted, and do the following:

1. Create the security group

   ec2-create-group --aws-access-key <EC2_ACCESS_KEY> 
   --aws-secret-key <EC2_SECRET_KEY> 
   --url <EC2_URL> -g neo4j-cluster -d "Neo4j HA Cluster"

2. Authorize port for SSH in neo4j-cluster security group

   ec2-authorize 
   --aws-access-key <EC2_ACCESS_KEY> 
   --aws-secret-key <EC2_SECRET_KEY> 
   --url <EC2_URL> -P tcp -p 22 -s 0.0.0.0/0 neo4j-cluster

3. Authorize all port communication between cluster members 

   ec2-authorize 
   --aws-access-key <EC2_ACCESS_KEY> --aws-secret-key <EC2_SECRET_KEY> 
   --url <EC2_URL> -P tcp -o neo4j-cluster -p -1 neo4j-cluster

After completing these steps, use

ec2-describe-group

to view the security group:

ec2-describe-group --aws-access-key <EC2_ACCESS_KEY> 
--aws-secret-key <EC2_SECRET_KEY> --url <EC2_URL> neo4j-cluster

GROUP sg-1cbc5777 986451091583 neo4j-cluster Neo4j HA Cluster
PERMISSION 986451091583 neo4j-cluster ALLOWS tcp 0 65535 FROM 
USER 986451091583 NAME neo4j-cluster ID sg-1cbc5777 ingress
PERMISSION 986451091583 neo4j-cluster 
ALLOWS tcp 22 22 FROM CIDR 0.0.0.0/0 ingress

Neo4j HA Cluster Deployment

Once the security group is created with the correct ports authorized, the cluster can be deployed.  To deploy the cluster, do the following:

1. Obtain Ansible from git and setup the environment by following the instructions mentioned here - http://ansible.cc/docs/gettingstarted.html#getting-ansible
2. Obtain the Ansible Playbook for Neo4j HA Cluster using git

   git clone https://github.com/hspencer77/ansible-neo4j-cluster.git

3. Change directory into ansible-neo4j-cluster  

   cd ansible-neo4j-cluster

4. Set up /etc/ansible/hosts with the following information:

   [local]
   127.0.0.1

5. Populate vars/ec2-config with either Eucalyptus/AWS information. vars/ec2-config contains the following variables:

   keypair: <EC2/Eucalyptus Keypair>
   ec2_access_key: <EC2_ACCESS_KEY>
   ec2_secret_key: <EC2_SECRET_KEY>
   ec2_url: <EC2_URL>
   instance_type: m1.small
   security_group: <AWS/Eucalyptus Security Group>
   image: <AMI/EMI>

6. Execute the following command:

   ansible-playbook neo4j-cluster.yml \
    --private-key=<AWS/Eucalyptus Private Key file> --extra-vars "node_count=3"

7. After the playbook finishes, there will be an URL provided to access the cluster – similar to the example below:

   TASK: [Display HAProxy URL] *********************
   changed: [23.22.248.75] => {"changed": true, "cmd": 
   "echo \"HAProxy URL for Neo4j -
    http://ec2-23-22-248-75.compute-1.amazonaws.com/webadmin/#/info/org.neo4j/High%20Availability/\" ",
    "delta": "0:00:00.006835", "end": "2013-03-30 19:54:31.104320", 
   "rc": 0, "start": "2013-03-30 19:54:31.097485", "stderr": "", 
   "stdout": 
   "HAProxy URL for Neo4j - 
   http://ec2-23-22-248-75.compute-1.amazonaws.com/webadmin/#/info/org.neo4j/High%20Availability/"}

   To view the status of cluster in the browser, open up http://ec2-23-22-248-75.compute-1.amazonaws.com/webadmin/#/info/org.neo4j/High%20Availability/.

8. To get the status of the cluster, use curl:

   curl -H "Content-Type:application/json" -d '["org.neo4j:*"]' 
   
   http://ec2-23-22-248-75.compute-1.amazonaws.com/db/manage/server/jmx/query

Thats it!  A Neo4j HA cluster with an HA Proxy server serving as an endpoint is available to be used.   If a bigger cluster is desired, just change the

node_count

value.   For additional information regarding this playbook, and how it handles the cluster membership, please refer to the following URL - https://github.com/hspencer77/ansible-neo4j-cluster/blob/master/README.md.

Hope you enjoy!  As always, questions/comments/suggestions are always welcome.