Tarkblog The personal writings of Caleb and Annie Tennis

Brew at the Zoo

Tickets for the 6th Annual Brew at the Zoo went on sale this morning! I got mine this morning. Bought tickets for my parents, too.

Stoked!

Checking in on your chef nodes

If you are running a chef server with multiple client nodes, you probably have these nodes checking in periodically via chef-client - either in daemon mode or via a cron job.

If you’re like me, from time you time you may turn off chef-client on a node while you’re testing something out. Or maybe you are using EC2 and you have to spin nodes down and up periodically. Either way, it would be nice to have a place you can go and take a quick look at when your nodes have last checked in to the chef server, so you can get an idea if something has gone wrong with one of them.

The beauty of this is that it’s incredibly easy. All of the info you need is stored right in the couchdb backend. Let me show you how I did it.

Compact those #chef #couchdb databases

When you have tons of chef clients connecting to your chef server, the couch file can get really big. It needs to be compacted every so often. You can do it a couple of different ways.

1. Use futon



2. Use a cron job, like 37signals:

cron "compact chef couchDB" do
  command "curl -X POST http://localhost:5984/chef/_compact >> /var/log/cron.log 2>&1"
  hour "5"
  minute "0"
end

3. Just use chef itself (my preference, as it’s one less cronjob I have to keep tabs on)

http_request "compact chef couchDB" do
  action :post
  url "http://localhost:5984/chef/_compact"
  only_if { File.size("/var/lib/couchdb/chef.couch") > 100_000_000 }
end

Since chef runs periodically anyway, I don’t have any need to add it to cron.

There’s probably a more DRY way of doing #3 above, but it works for now.

Delete chef file backups by timestamp

So, chef keeps around backups of files it manipulates on the system (5 by default), and while this is modifyable, I like having 5 in most cases. However, some of the files I have don’t change much. In fact, by the time I was to accumulate 5 changes, it may be a period of months or years.

So, I whipped up a little bit of code to delete backup files after a certain period of time..7 days by default.

Just throw this into one of your cookbook libraries:

require 'chef/provider/file'
 
class Chef
  class Provider
    class File
 
      alias old_backup backup
 
      def delete_old_backups(secs = 604800)
        time = Time.now - secs
        savetime = time.strftime("%Y%m%d%H%M%S")
 
        Dir["#{@new_resource.path}.chef-*"].sort { |a,b| b < => a }.each do |backup_file|
          timestamp = /\.chef-(.+)/.match(backup_file)[1]
          if(timestamp.to_i < savetime.to_i)
            Chef::Log.info("Removing timestamped backup of #{@new_resource} at #{backup_file}")
            FileUtils.rm(backup_file)            
          end
        end
 
      end
 
      def backup(file=nil)
        delete_old_backups
        old_backup(file)
      end
 
    end
  end
end

Currently it will only perform the deletion on a file change, but I presume it’d be easy to make it delete during a converge whether the file changed or not (think overriding action_create).

Hosts updating with chef searches

For intra-ec2 instance communication, we used to keep track of internal IP addresses via a bit of a hack that would update the hosts file. This keeps us from having to use DNS for internal communications.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

#!/usr/bin/env ruby
 
%w(optparse rubygems EC2 resolv pp).each { |l| require l }
 
options = {}
parser = OptionParser.new do |p|
  p.banner = "Usage: hosts [options]"
  p.on("-a", "--access-key USER", "The user's AWS access key ID.") do |aki|
    options[:access_key_id] = aki
  end
  p.on("-s",
       "--secret-key PASSWORD",
       "The user's AWS secret access key.") do |sak|
    options[:secret_access_key] = sak
  end
  p.on_tail("-h", "--help", "Show this message") {
    puts(p)
    exit
  }
  p.parse!(ARGV) rescue puts(p)
end
if options.key?(:access_key_id) and options.key?(:secret_access_key)
  puts "127.0.0.1 localhost"
  EC2::Base.new(options).describe_instances.reservationSet.item.each do |r|
    r.instancesSet.item.each do |i|
      if i.instanceState.name =~ /running/
        puts(Resolv::DNS.new.getaddress(i.privateDnsName).to_s +
             " #{i.keyName}.ec2 #{i.keyName}")
      end
    end
  end
else
  puts(parser)
  exit(1)
end

Running this periodically via cron like this:

0 * * * * /usr/local/sbin/hosts -a ACCESS_KEY -s SECRET_KEY

Produced an /etc/hosts like this:

$ cat /etc/hosts

127.0.0.1 localhost
10.252.135.144 larry.ec2 larry
10.250.206.242 moe.ec2 moe

The script required us to keep the AWS account information on disk, and also only worked for one AWS account, whereas you may have more than one.

With chef’s built in search indices, however, doing this now is much easier.

First, I set up a search directly within a recipe:

cluster_hosts = 
  search(:node, "cluster_name:#{node[:cluster][:name]}", ['ec2_local_ipv4','hostname'])
 
template "/etc/hosts" do
  source "hosts.erb"
  variables(
    :cluster_hosts => cluster_hosts || []
  )
  backup false
end

with a hosts.erb of:

127.0.0.1 localhost
 
# Cluster Hosts
< % @cluster_hosts.each do |host| %>
  < %= "#{host['ec2_local_ipv4']} #{host['hostname']} #{host['hostname']}.ec2" %>
< % end %>

And we get the same results. The cluster name stuff is a little bit of sugar specific to our setup, but still I think this gives the general idea.