Introduction to OpsWorks Workshop

by Oz Akan
howto
, ,

Objective

We will create a local Chef development environment and a demo cookbook. We will use this cookbook to create an OpsWorks stack and will also take the advantage of OpsWorks data bags to gather information about the environment.

Create Chef Development Instance

Create an EC2 key pair named chefuser and download pem file chefuser.pem. This is used in by chef to login to EC2 instances.

Create another EC2 key pair, or use existing one, to login to Chef development instance. (Mine happens to be named awsplus.pem.)

Create an instance to use as Chef development instance. In this workshop, we use the latest Amazon AMI (ami-6869aa05). m4.large or a smaller instance type would be enough. Use awsplus as the key.

Login to the instance.

$ echo "ServerAliveInterval 50" > ~/.ssh/config
$ chmod 644 ~/.ssh/config
$ sudo yum update -y
$ sudo yum install tree -y #optional
$ sudo yum install emacs -y #optional

Create an IAM user that can create EC2 instances and enter it’s credentials into ~/.aws/credentials on chef instance.

[default]
aws_access_key_id = AKIAJGBFBGOJJJJMKYXQ
aws_secret_access_key = EnlAfEjw4S4JWfX9ABCAAAAAAAAAABC

Create a security group that will allow inbound connections to TCP port 80. We will use this security group in .kitchen.yml configuration file. (In the example below there are two security groups. One of them allows ssh in, other one allows all connections to my IP address so I can try TCP port 80)

Configure emacs (http://cloudway.io/post/emacs-tips/)

Logout, login.

Configure Chef Instance

Install ChefDK

You can use screen to keep ssh session

$ screen -S OZ

Find the latest chefdk package on https://downloads.chef.io/chef-dk/ and download it.

$ wget https://packages.chef.io/stable/el/6/chefdk-0.16.28-1.el6.x86_64.rpm

Install the chefdk.

$ sudo rpm -iUvh chefdk-0.16.28-1.el6.x86_64.rpm

Create demo cookbook

$ mkdir -p ~/chef/cookbooks; cd ~/chef/cookbooks
$ chef generate cookbook demo
$ cd demo

Configure Kitchen

Modify ~/cookbooks/demo/.kitchen.yml to look like below.

---
driver:
  name: ec2
  aws_ssh_key_id: chefuser
  region: us-east-1
  availability_zone: us-east-1c
  require_chef_omnibus: true
  instance_type: t2.small
  security_group_ids: ["sg-2385c744", "sg-2e48f255"]

transport:
  ssh_key: /home/ec2-user/.ssh/chefuser.pem
  connection_timeout: 10
  connection_retries: 5
  username: ec2-user

provisioner:
  name: chef_zero

platforms:
  - name: amazon
    driver:
      image_id: ami-6869aa05
      #image_id: ami-f5f41398 - older
    transport:
      username: ec2-user

suites:
  - name: Web1
    data_bags_path: data_bags
    run_list:
      - recipe[demo::default]
  - name: Web2
    data_bags_path: data_bags
    run_list:
      - recipe[demo::default]

We will modify this file later on.

Copy chefuser.pem from your computer to Chef development instance, and change permissions.

$ chmod 600 /home/ec2-user/.ssh/chefuser.pem

Try kitchen list

$ kitchen list
Instance          Driver  Provisioner  Verifier  Transport  Last Action
Web1-amazon  Ec2     ChefZero     Busser    Ssh        <Not Created>
Web2-amazon  Ec2     ChefZero     Busser    Ssh        <Not Created>

Create A Simple Recipe with A Template

Create a template

$ cd cookbooks
$ chef generate template demo test.txt
Recipe: code_generator::template
  * directory[./demo/templates/default] action create
    - create new directory ./demo/templates/default
  * template[./demo/templates/test.txt.erb] action create
    - create new file ./demo/templates/test.txt.erb
    - update content in file ./demo/templates/test.txt.erb from none to e3b0c4
    (diff output suppressed by config)

Content of ~/demo/templates/test.txt.erb

This is a test file created by Chef.

Create a recipe

$ cd cookbooks
$ chef generate recipe demo init
$ cd cookbooks/demo/recipes

Modify init.rb to look like below.

#
# Cookbook Name:: demo
# Recipe:: init
#
# Copyright (c) 2016 The Authors, All Rights Reserved.

template '/tmp/test.txt' do
  source 'test.txt.erb'
  owner 'root'
  group 'root'
  mode '0766'
end

Include init recipe in the default recipe. Edit demo/recipes/default.rb to look like below.

include_recipe 'demo::init'

Create the instance using kitchen.

$ kitchen converge Web1
...
...
Running handlers:
Running handlers complete
Chef Client finished, 1/4 resources updated in 05 seconds
Finished converging <Web1-amazon> (0m21.95s).

Login to newly created instance and check test.txt file.

$ kitchen login Web1-amazon
$ cat /tmp/test.txt
This is a test file created by Chef.

Configure Web Servers

Create a User for Web Service

Under cookbooks folder run the command below to create service_user recipe.

$ chef generate recipe demo service_user

As all recipes for demo cookbook, this recipe is created under cookbooks/demo/recipes/ folder.

Add the line below to cookbooks/demo/recipes/service_user.rb.

group 'service_admin'

user 'service_admin' do
  group 'service_admin'
  system true
  shell '/bin/bash'
end

Create custom attributes file to define our own attributes instead of prividing them in the recipes.

$ chef generate attribute demo default

Edit ./demo/attributes/default.rb to have the code below.

default['demo']['user'] = 'service_admin'
default['demo']['group'] = 'service_admin'

Modify service_user recipe to use custom attributes.

group node['demo']['group']

user node['demo']['user'] do
  group node['demo']['group']
  system true
  shell '/bin/bash'
end

Include service_user recipe in the default recipe. Edit demo/recipes/default.rb to look like below.

include_recipe 'demo::init'
include_recipe 'demo::service_user'

Run the cookbook again

$ kitchen converge Web1-amazon

Login and check if service_admin user and groups are created.

$ kitchen login Web1-amazon
$ grep service_admin /etc/passwd
service_admin:x:498:501::/home/service_admin:/bin/bash
$ grep service_admin /etc/group
service_admin:x:501:

Configure SELinux

To configure SELinux we will use a community cookbook. You can find community cookbooks at https://supermarket.chef.io/. If you know the name of the cookbook, you can use knife command as shown below.

$ knife cookbook site show selinux|grep latest_version
latest_version:       https://supermarket.chef.io/api/v1/cookbooks/selinux/versions/0.9.0

To use a community cookbook, we have to include it in the cookbooks/demo/metadata.rb file. Append the line below to metadata.rb

depends 'selinux', '~> 0.9.0'

Now we can refer to selinux cookbook and the resources it provides in from our demo cookbook. selinux cookbook is relatively easy to use and we will only call a recipe to disable selinux on the linux instance.

Append the code below to default recipe located at cookbooks/demo/recipes/default.rb

include_recipe 'selinux::permissive'

Run the cookbook and you will see selinux recipe is applied.

$ kitchen converge
...
   Recipe: selinux::_common
     * yum_package[libselinux-utils] action install (up to date)
     * directory[/etc/selinux] action create (up to date)
   Recipe: selinux::permissive
     * selinux_state[SELinux Permissive] action permissive (up to date)
   Recipe: demo::service_user
...

Configure Apache

To create web_server recipe, under cookbooks folder run the command below.

$ chef generate recipe demo web_server

Modify web_server recipe to look like below.

#
# Cookbook Name:: demo
# Recipe:: web_server
#
# Copyright (c) 2016 The Authors, All Rights Reserved.

package 'httpd' do
  action :install
end

directory '/var/www/html' do
  recursive true
end

template '/etc/httpd/conf.d/hello.conf' do
  source 'hello.conf.erb'
  mode '0644'
  owner node['demo']['user']
  group node['demo']['group']
  notifies :restart, 'service[httpd]'
end

template '/var/www/html/hello.html' do
  source 'hello.html.erb'
  mode '0644'
  owner node['demo']['user']
  group node['demo']['group']
end

service 'httpd' do
  action [:enable, :start]
end

We will create two templates.

  • hello.conf: apache virtual host configuration
  • hello.html: to be used as index page

Create hello.conf template that is referred in the web_server recipe.

$ chef generate template demo hello.conf

Put the code below into hello.conf.erb.

<VirtualHost *:80>
    ServerAdmin webmaster@localhost

    DocumentRoot /var/www/html

    DirectoryIndex hello.html

    <Directory />
        Options FollowSymLinks
        AllowOverride None
    </Directory>

    <Directory /var/www/html>
        Options Indexes FollowSymLinks MultiViews
        AllowOverride None
        Order allow,deny
        allow from all
    </Directory>

    ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
    <Directory "/usr/lib/cgi-bin">
        AllowOverride None
        Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
        Order allow,deny
        Allow from all
    </Directory>

    LogLevel warn
</VirtualHost>

As you can notice, we change the DirectoryIndex to hello.html from default index.html.

Create hello.html template that is referred in the web_server recipe.

$ chef generate template demo hello.html

Put the code below into hello.html.erb

<html>
    Hello AWS World!
</html>

Finally add the recipe to default recipe.

cookbooks/demo/recipes/default.rb must look like below.

include_recipe 'demo::init'
include_recipe 'selinux::permissive'
include_recipe 'demo::service_user'
include_recipe 'demo::web_server'

Run the cookbook

$ kitchen converge Web1-amazon

Find the public IP address of the instance and try on a browser (You need a security group that will let you access TCP port 80).

Use Variables in The Template

We will modify hello.html template in order to use data bags, node attributes.

Kitchen lets you pass a data_bag_path. Lets create a data bag.

$ mkdir -p data_bags/demo_bag
$ touch data_bags/demo_bag/config.json

Put the code below into conifg.json

{
    "name": "demo application"
}

Now we have a data bag named demo_bag with an item named config. It has a key named name with a value of demo application.

Let’s modify web_server recipe to use the data bag.

config = data_bag_item('demo_bag', 'config')

package 'httpd' do
  action :install
end

directory '/var/www/html' do
  recursive true
end

template '/etc/httpd/conf.d/hello.conf' do
  source 'hello.conf.erb'
  mode '0644'
  owner node['demo']['user']
  group node['demo']['group']
  notifies :restart, 'service[httpd]'
end

template '/var/www/html/hello.html' do
  source 'hello.html.erb'
  mode '0644'
  owner node['demo']['user']
  group node['demo']['group']
  variables(:name => config['name'])
end

service 'httpd' do
  action [:enable, :start]
end

We used config item from demo_bag data bag and used name key.

Modify cookbooks/demo/templates/hello.html.erb template so it will look like below.

<html>
  Hello AWS World! <br>
  Application: <%= @name %> (-> comes from data bag) <br>
  This is <%= node['hostname'] %> (-> comes from node attributes) <br>
</html>

Modify suites section in .kitchen.yml so it will look like below.

suites:
  - name: Web1
    data_bags_path: data_bags
    run_list:
      - recipe[demo::default]
  - name: Web2
    data_bags_path: data_bags
    run_list:
      - recipe[demo::default]

Run the command below.

$ kitchen converge

This will update Web1-amazon instance and also will create the second instance identical to first one.

Moving the Cookbook to OpsWorks

Preparation for OpsWorks Stack

OpsWorks accepts data bag data in a custom JSON. Details here

Below is the JSON we will pass to OpsWorks.

{
  "opsworks": {
    "data_bags": {
      "demo_bag": {
        "config": {
          "name": "demo application"
        }
      }
    }
  }
}

Create a package of the cookbook and upload to your S3 bucket. (oz-opsworks is the name of the bucket in this demo)

$ berks package demo-cookbook.tar.gz; aws s3 cp demo-cookbook.tar.gz s3://oz-opsworks/demo-cookbook.tar.gz

Create elb demo-elb with port 80 and health check on /hello.html.

Create the Stack

Go to OpsWork console and click on Chef 12 Stack. Give a name (demo) to your stack. Choose the region, VPC as you want. Choose Linux and latest Amazon AMI, the SSH key you want to use.

As seen below check yes on Use custom Chef cookbooks and provide S3 information. This is where OpsWorks will look for the cookbook.

In the Advanced Section make the selections as seen before. Remember the JSON we mentioned earlier, paste that into Custom JSON.

Click Add Stack.

At this point we have a stack and we will create a Layer for Web Service.

Create the Layer

On the following page, click Add a Layer. Give a long and a short name to the layer and click Add Layer.

On the next page, click on Network.

Choose the ELB you created earlier and click Save.

Now we have associated an ELB with our layer and each instance we will create under this layer will automatically join to ELB.

Click recipes and enter the recipe names and save. It will look like below.

Use OpsWorks Data Bags

Before going forward and creating instances, we will modify our demo cookbook in order to use aws_opsworks_instance data bag which provides information about the instances in the stack.(More here)

We will make changes to web_server recipe and hello.html template in order to find all EC2 instances in the layer using OpsWorks aws_opsworks_instance data bag. These data bags are automatically populated by OpsWorks and are especially useful in dynamic environments where you add/remove instances based on the load.

Edit web_server.rb recipe so it will look like below.

#
# Recipe:: web_server
#
# Copyright (c) 2016 The Authors, All Rights Reserved.

config = data_bag_item('demo_bag', 'config')
instances_data_bag = data_bag('aws_opsworks_instance')

package 'httpd' do
  action :install
end

directory '/var/www/html' do
  recursive true
end

template '/etc/httpd/conf.d/hello.conf' do
  source 'hello.conf.erb'
  mode '0644'
  owner node['demo']['user']
  group node['demo']['group']
  notifies :restart, 'service[httpd]'
end

template '/var/www/html/hello.html' do
  source 'hello.html.erb'
  mode '0644'
  owner node['demo']['user']
  group node['demo']['group']
  variables({
              :name => config['name'],
              :instances => instances_data_bag
            })
end

service 'httpd' do
  action [:enable, :start]
end

Create update recipe.

$ cd cookbooks
$ chef generate recipe demo update
$ cd cookbooks/demo/recipes

Add the code below to update.rb. It is actually a section of web_server recipe. We could include update recipe in web_server recipe in order not to repeat the code. For the sake of this example, we are just copying it.

config = data_bag_item('demo_bag', 'config')
instances_data_bag = data_bag('aws_opsworks_instance')

template '/var/www/html/hello.html' do
  source 'hello.html.erb'
  mode '0644'
  owner node['demo']['user']
  group node['demo']['group']
  variables({
              :name => config['name'],
              :instances => instances_data_bag
            })
end

Edit hello.html.erb template so it will look like below.

<html>
  Hello AWS World! <br>
  Application: <%= @name %> (-> comes from data bag) <br>
  This is <%= node['hostname'] %> (-> comes from node attributes) <br>
  <b>All Instances:</b> (-> comes from OpsWorks data bag)<br>
  <% @instances.each do |instance| %>
  <%= "#{instance} <br>" %>
  <% end %>
</html>

Package and upload the cookbook to S3.

$ berks package demo-cookbook.tar.gz; aws s3 cp demo-cookbook.tar.gz s3://oz-opsworks/demo-cookbook.tar.gz

Add Instances

Click Instances from the left navigation, click Add Instance and choose size and subnet for the instance. You can use the proposed name or enter a new one.

Click + Instance to add another one. Instances sizes can be different if you like.

Once you see two instances there, click Start All Instances.

After several minutes, both instances will come online. It will take longer than a plain EC2 instance boot as OpsWorks will run Chef recipes on these instances. The more you do with the Chef recipes, the longer it will take for instances to boot up.

Check the Web Site

Click Layer from the left navigation bar. As seen in the screenshot below, you will find the public DNS record for the ELB. Click the link.

Check ELB public URL on the browser. Your http request will go through ELB and hit one of the web servers OpsWorks provisioned using the demo cookbook.

First request hits Web2

When I reload the page, this time request goes to Web1

If we add another instance, for example Web3, OpsWorks would automatically run configure lifecycle event on the existing instances. This event would call update recipe, which would then update hello.html template and include all the instances in the layer using aws_opsworks_instance data bag. This mechanism helps creating dynamic recipes that can take environment changes into account.

Going a step further, you can check the state of the instance and print it in the hello.html only if it is in running state. The recipe right now, just checks the instances in the layer and prints them even if they are not running.

Summary

In this workshop, we created a cookbook that uses attributes, data bags and a community cookbook as we probably do most of the time in a chef environment while developing cookbooks. While we could use this cookbook without any modification, to get most out of OpsWorks we used OpsWorks data bag named aws_opsworks_instance to find out the instances in the stack. Using lifecycle events, instances in the environment automatically changed their configuration (hello.html) when an instance is added to the layer.

comments powered by Disqus