This post is a tutorial on how to deploy Drizzle on an EC2 instance using chef and the Opscode Chef platform. The tutorial is specifically targetted at Ubuntu platforms. In particular, the procedures outlined here have only been tested on Ubuntu 10.04. It is expected however that the instructions here should apply on other Ubuntu versions with minimal modifications needed.

The Opscode Platform

In this article, we'll use the Opscode platform since it provides an easy way for anyone to get started with chef. If you are a new user, proceed to sign up for a new account. Once you are signed up, the next step is to create a new organization. For this article, I'm going to create an organization named drizzle-test. Once your organization is created, you should see the organization in your list of organizations when you click on the Organizations link at the top right of the opscode console. My view looks like (you should be able to click on the image to see a larger version):

Configure AWS

An assumption made in this article is that you have an AWS account. If you don't, signing up is relatively straightforward.

There are a few items that need to be configured for EC2 that we need to do to make our lives easier before starting with chef. Amazon blocks all incoming traffic to EC2 instances by default. SSH is used by chef to access and bootstrap a newly created instance. We want to allow SSH traffic to our EC2 instances and for this article, I want to permit traffic to the drizzle port (default drizzle port is 4427) as well. This is accomplished using the AWS console. We need to configure Security Groups. You can either create a new security group and modify the default security group. For this article, I'll create a new security group named drizzle and add the appropriate rules. After creating the group and adding the rules, the security group details should look like:

I'll also create a new key pair in the AWS console specifically for this article. I'm going to give this key pair the name drizzle. After creating the key pair, I copy the downloaded private key to my SSH folder and update permissions on the key:

mv ~/Downloads/drizzle.pem ~/.ssh/
chmod 600 ~/.ssh/drizzle.pem

Install chef

To install chef on Ubuntu is quite straightforward. Opscode maintains an APT repository which I simply need to add to my sources list. In the file /etc/apt/sources.list.d/opscode.list, add (and replace lucid with whatever release you are running):

deb lucid main

Next, I need to add the GPG key:

wget -qO - | sudo apt-key add -
sudo apt-get update

To install chef, its as simple as installing the chef package:

sudo apt-get install chef

When prompted for the server URL during this package installation, you can leave it blank. We will be configuring this later. You can also stop and disable the chef-client service now if you wish since we will only be using the knife utility in this article. Finally, verify the version you have installed:

knife -v

For this article, the output of the above command needs to be a least 0.9.14

Other packages required for this article are rubygems and git:

sudo apt-get install rubygems git

Once rubygems is installed, there a few gems required for interacting with EC2:

sudo gem install net-ssh net-ssh-multi fog highline

Configure chef

We are now all set to get started. The first thing to do is create a chef repository on your workstation. In this article, I will use git for this:

git clone drizzle-chef-repo

Create a .chef directory within this repository. This directory contains all the configuration files for just this repository:

mkdir -p ~/drizzle-chef-repo/.chef

Next, we need to download keys and knife configuration files from the Opscode platform that will be used for interacting with Opscode platform. Keys are needed for both your user and organization on the Opscode Platform. To retrieve your user key (if you did not download it when signing up), click on your username through the console and you will a 'get private key' link on your account page:

After Downloading this key, I need to place it in the configuration directory for the chef repository I am using here:

mv ~/Downloads/posulliv.pem ~/drizzle-chef-repo/.chef

For your organization, click on the 'Regenerate validation key' link and 'Generate knife config' link from the organizations over page as mentioned in the first section of this article. After clicking those 2 links, you will have 2 files: 1) drizzle-test-validator.pem and 2) knife.rb. Move these 2 files into the configuration directory for the chef repository being used for this article:

mv ~/Downloads/drizzle-test-validator.pem ~/drizzle-chef-repo/.chef
mv ~/Downloads/knife.rb ~/drizzle-chef-repo/.chef

From now on, whenever you are in the drizzle-chef-repo directory, the knife utility will connect to the Opscode Platform. To verify this, lets list out the current clients:

posulliv@curragh:~/drizzle-chef-repo$ knife client list

We need to tell knife about our AWS credentials. This is done by adding the following 2 lines to your knife.rb file in the ~/drizzle-chef-repo/.chef directory:

knife[:aws_access_key_id]     = "Your AWS Access Key"
knife[:aws_secret_access_key] = "Your AWS Secret Access Key"

After adding these credentials I should now be able to list all the EC2 instances associated with my AWS account:

posulliv@curragh:~/drizzle-chef-repo$ knife ec2 server list
Instance ID      Public IP        Private IP       Flavor           Image            Security Groups  State          
i-5e1ce433    m1.large         ami-879f70ee     AkibanWeb        running        
i-1bcb4f77   t1.micro         ami-548c783d     AkibanWeb        running        
i-d6fa10b9     m1.large         ami-548c783d     AkibanQA         running        
i-98db31f7   m1.large         ami-548c783d     AkibanQA         running        
i-1e16fc71   m1.large         ami-548c783d     AkibanQA         running        

Drizzle Cookbook

chef should now be configured to work with your AWS account. The next step is to decide on what roles or recipes you want to apply to an instance you create. Since this article is on drizzle, I'll show how to bootstrap an EC2 instance with drizzle. I have developed a simple drizzle cookbook in a fork of Opscode's official cookbook repository that can be retrieved with git:

cd ~/drizzle-chef-repo
rm -rf cookbooks
git clone git://

I have opened a pull request for this fork to get merged into Opscode's official repository. Hopefully, it will get merged in soon.

Now we want to upload cookbooks to our chef server. The only cookbook I will upload in this article is the Drizzle cookbook:

cd ~/drizzle-chef-repo
knife cookbook upload drizzle
It is simple to list the cookbooks that have been uploaded so far to your chef server:
posulliv@curragh:~/drizzle-chef-repo$ knife cookbook list

Create and Verify EC2 Instance

We are now ready to create an EC2 instance and have it bootstrap itself and install the drizzle GA release! You will see a spew of output when you issue the command below (feel free to use any AMI image or flavor you wish, I just picked one):

knife ec2 server create "recipe[drizzle]" \
--image ami-2d4aa444 \
--flavor m1.small \
--groups drizzle \
--ssh-key drizzle \
--identity-file ~/.ssh/drizzle.pem \
--ssh-user ubuntu

To verify the server is created, first we check in the server list output from EC2:

posulliv@curragh:~/drizzle-chef-repo$ knife ec2 server list
Instance ID      Public IP        Private IP       Flavor           Image            Security Groups  State          
i-5e1ce433    m1.large         ami-879f70ee     AkibanWeb        running        
i-1bcb4f77   t1.micro         ami-548c783d     AkibanWeb        running        
i-d6fa10b9     m1.large         ami-548c783d     AkibanQA         running        
i-98db31f7   m1.large         ami-548c783d     AkibanQA         running        
i-1e16fc71   m1.large         ami-548c783d     AkibanQA         running        
i-c03b5caf    m1.small         ami-2d4aa444     drizzle          running        

We should also verify that it is listed as a node:

posulliv@curragh:~/drizzle-chef-repo$ knife node list

Finally, if I log onto the EC2 instance I should be able to connect to drizzle:

posulliv@curragh:~$ ssh -i ~/.ssh/drizzle.pem ubuntu@
Linux ip-10-116-210-131 2.6.32-305-ec2 #9-Ubuntu SMP Thu Apr 15 04:14:01 UTC 2010 i686 GNU/Linux
Ubuntu 10.04 LTS

Welcome to Ubuntu!
 * Documentation:

  System information as of Mon Apr 11 23:01:28 UTC 2011

  System load: 0.36             Memory usage: 13%   Processes:       55
  Usage of /:  8.6% of 9.92GB   Swap usage:   0%    Users logged in: 0

  Graph this data and manage this system at
At the moment, only the core of the system is installed. To tune the 
system to your needs, you can choose to install one or more          
predefined collections of software by running the following          
   sudo tasksel --section server                                     

A newer build of the Ubuntu lucid server image is available.
It is named 'release' and has build serial '20110201.1'.
Last login: Mon Apr 11 22:27:04 2011 from
ubuntu@ip-10-116-210-131:~$ drizzle
Welcome to the Drizzle client..  Commands end with ; or \g.
Your Drizzle connection id is 9
Connection protocol: mysql
Server version: 2011.03.13 Ubuntu

Type 'help;' or '\h' for help. Type '\c' to clear the buffer.



Hopefully, this tutorial proves useful. I hope to work more on the Drizzle cookbook in the near future and add support for the various plugin types present in Drizzle.

blog comments powered by Disqus


07 April 2011