Automate .ssh/config for Chef Using Knife

Mar 29th, 2012

First off let me state that I copied this plugin and made it better and am sharing with you the end result.

To use this copy and paste the contents to a file in .chef/plugins/knife called sshgen.rb for example.

## Knife plugin to generate an OpenSSH config file from a Chef search
# From Harvest. www.getharvest.com
# Modified for EC2 usage by Scott Likens
#
# Source: https://github.com/harvesthq/knife-plugins
#
# See http://wiki.opscode.com/display/chef/Knife+Plugins
# See http://www.openbsd.org/cgi-bin/man.cgi?query=ssh_config&sektion=5
#
## Install
# Place in .chef/plugins/knife/sshgen.rb
#
## Usage
# $ knife sshgen -p to print a config stanza to the screen
# $ knife sshgen -w to append to your current ~/.ssh/config file
#
## Suggestion
# use with bash-completion, by installing bash-completion and adding this to ~/.bash_profile:
# complete -W "$(echo `cat ~/.ssh/config | grep Hostname | uniq | awk '{print $2}'`;)" ssh
#
## Credit
# Hints taken from: https://github.com/danielsdeleo/knife-plugins/blob/master/deploy.rb
# Hints taken from: https://github.com/javan/whenever

module MakeSSHConfigFile
  class Sshgen < Chef::Knife

    banner "knife sshgen -w OR -p"

    option :ssh_config_file,
    :long => "--config-file",
    :short => "-c",
    :description => "The name of your SSH config file",
    :default => "#{ENV['HOME']}/.ssh/config"

    option :print,
    :long => "--print",
    :short => "-p",
    :description => "Print the ssh config entry here in this screen",
    :default => true

    option :write,
    :long => "--write",
    :short => "-w",
    :description => "Write (append) the config to local SSH config file",
    :default => false

    deps do
      require 'chef/search/query'
      require 'chef/knife/ssh'
    end

    def run

      unless config[:print] || config[:write]
        ui.error "You must specify either --print or --write."
        exit 1
      end

      nodes = find_all_nodes
      ssh_config = build_ssh_config(nodes)

      if config[:write]

        unless File.exists? config[:ssh_config_file]
          ui.msg "WARNING: #{config[:ssh_config_file]} non-existent, but I'll try anyway"
        end

        new_ssh_config = read_ssh_config

        if new_ssh_config.index(comment_open) && new_ssh_config.index(comment_close) #we have an existing config in between our markers, update it
          final_ssh_config = new_ssh_config.gsub(Regexp.new("#{comment_open}.+#{comment_close}", Regexp::MULTILINE), "#{comment_open}\n#{ssh_config.chomp}\n#{comment_close}")
        else # no existing markers, we'll append them
          final_ssh_config = "#{new_ssh_config}\n#{comment_open}\n#{ssh_config}#{comment_close}"
        end

        write_ssh_config(final_ssh_config)
      elsif config[:print]
        ui.msg "#{ssh_config}"
      end
    end

    def find_all_nodes
      query = "name:*"
      searcher = Chef::Search::Query.new
      rows, _start, _total = searcher.search(:node, query)
      if rows.empty?
        ui.error "No nodes matched the query: #{query}"
        exit 1
      end

      rows
    end

    def build_ssh_config(my_nodes)
      ssh_config_block = ""
      my_nodes.each do |n|
        if n.has_key?("ec2")
          ssh_config_block << "Host #{n.name} \n"
          ssh_config_block << "\t Hostname #{n[:ec2][:public_hostname]} \n"
          ssh_config_block << "\t User ubuntu \n"
          else
          ui.info "#{n} lacks an ec2 key"
        end
      end

      ssh_config_block
    end

    def read_ssh_config
      file = File.open(config[:ssh_config_file], "r+")
      ssh_config_contents = ""
      while(!file.eof?)
        ssh_config_contents << file.readline
      end
      file.close()

      ssh_config_contents
    end

    def write_ssh_config(contents)
      begin
        File.open(config[:ssh_config_file], 'w') {|f| f.write(contents) }
      rescue
        ui.error "ERROR: Writing #{config[:ssh_config_file]} failed."
      else
        ui.msg "Appended to: #{config[:ssh_config_file]}"
      end
    end

    def comment_open
      "###STARTSSHGEN"
    end

    def comment_close
      "###ENDSSHGEN"
    end


  end
end

A Process of Making a Custom HVM Virtualized AMI for EC2

Jan 7th, 2012

Forewarning: This post is about a process; it is not meant to give you a complete working basis to create a hvm virtualization AMI. It is here to give you the information and or tools to build your own.

Required Reading Material.

hvmloader
pv-on-hvm
Note: Usage of a vanilla kernel later then 2.6.37+ will have the pv-on-hvm drivers. Pick a kernel later then that at your descretion.

Requirements.

Amazon AWS Account
Amazon API Tools Installed
Amazon SOAP Keys Created
Time; patience and a willingness to fail.

Step 1

booting an instance.

ec2-run-instances ami-0da96764 -k ssh_key -t cc1.4xlarge -H
Type  ReservationID   Owner   Groups  Platform
RESERVATION   r-f5743394  000000000000    default
INSTANCE  i-de453ebc  ami-0da96764            pending ssh_key 0       cc1.4xlarge 2012-01-07T23:01:54+0000    us-east-1c              monitoring-disabled                 ebs                 hvm xen     sg-7baf4812 default

Take note of the instance-id and the zone the instance is booted in.
additionally Amazon limits us to an 8G root volume; so create an 8gig volume in the zone specifieid.

create your EBS Root volume

ec2-create-volume -z us-east-1c -s 8
VOLUME    vol-03aff26e    8       us-east-1c  creating    2012-01-07T23:03:25+0000

attach the volume to the instance as /dev/sdp

ec2-attach-volume vol-03aff26e -d /dev/sdp -i i-de453ebc
ATTACHMENT    vol-03aff26e    i-de453ebc  /dev/sdp    attaching   2012-01-07T23:05:26+0000

SSH to the instance in question as ec2-user@$hostname

obtain root priviledges and install screen

sudo -s
yum install screen
screen -S ami

install the amazon api and ami tools

mkdir -p /mnt/work && cd /mnt/work
wget http://s3.amazonaws.com/ec2-downloads/ec2-ami-tools.zip
unzip ec2-ami-tools.zip
wget http://s3.amazonaws.com/ec2-downloads/ec2-api-tools.zip
unzip ec2-api-tools.zip*

Place your X.509 keys in /mnt/work/.ec2
Note: the version numbers do change; please update to reflect this.

configure your environment

export EC2_BASE="/mnt/work"
export EC2_AMIHOME="${EC2_BASE}/ec2-ami-tools-1.4.0.5"
export EC2_HOME="${EC2_BASE}/ec2-api-tools-1.5.2.3"
export PATH=${EC2_AMIHOME}/bin:${EC2_HOME}/bin:${PATH}
export EC2_CERT="/mnt/work/.ec2/cert-example.pem"
export EC2_PRIVATE_KEY="/mnt/work/.ec2/pk-example.pem"
export EC2_SECRET_KEY="secret_key"
export EC2_ACCESS_KEY="access_key"
export SOURCE_AMI_NAME="example_ami_name"
export SOURCE_AMI_MANIFEST="${SOURCE_AMI_NAME}.manifest.xml"
export EC2_AMIS="${EC2_BASE}/amis"
export EBS_DEV="/dev/xvdp1"

download your s3 ami locally

ln -sf $EC2_AMIHOME/lib/ec2 $EC2_HOME/lib/ec2
temp_dir=$(mktemp -d)
mkdir -p ${EC2_AMIS}
cd ${EC2_AMIS}
ec2-download-bundle -b example_bucket -a ${EC2_ACCESS_KEY} -s ${EC2_SECRET_KEY} -k ${EC2_CERT} -m ${SOURCE_AMI_MANIFEST}

unbundle your s3 ami and mount it

ec2-unbundle -k ${EC2_PRIVATE_KEY} -m ${SOURCE_AMI_MANIFEST} -d ${temp_dir}
cd ${EC2_BASE}
export EC2_AMI_IMAGE="${temp_dir}/${SOURCE_AMI_NAME}"
export EC2_SOURCE_MOUNT=$(mktemp -d)
export EC2_DEST_MOUNT=$(mktemp -d)
mount -o loop ${EC2_AMI_IMAGE} ${EC2_SOURCE_MOUNT}

Please send suggestions on how to improve this to /dev/null.

partition out your ebs volume

fdisk /dev/xvdp
n
p
1
<enter>
<enter>
a
1
w
q

format the EBS volume and mount it. No filesystem checks desired.

blockdev --rereadpt /dev/xvdp
mkfs -t ext4 -L rootfs ${EBS_DEV}
tune2fs -i 0 -c 0 ${EBS_DEV}
mount ${EBS_DEV} ${EC2_DEST_MOUNT}

copy your local S3 AMI to the EBS volume

rsync -aHAS ${EC2_SOURCE_MOUNT}/ ${EC2_DEST_MOUNT}/

chroot into your EBS root and download the linux kernel

cp /boot/config* ${EC2_DEST_MOUNT}
mount -o bind /dev ${EC2_DEST_MOUNT}/dev
chroot ${EC2_DEST_MOUNT}
cd /usr/src && curl http://www.kernel.org/pub/linux/kernel/v2.6/linux-2.6.39.4.tar.bz2 | tar jxf -
ln -s /usr/src/linux-2.6.39.4 linux

configure your kernel

cd /usr/src/linux
cp /config* .config
make bzImage -j16
make modules -j16
make modules_install
cp arch/x86/boot/bzImage /boot/vmlinuz-hvm

Configure /boot/grub/menu.lst similar to how you see provided by amazon; but replace the kernel and initrd with your own.
Use your favorite package manager to install grub. The following should work provided no ephemeral disks are attached.

install grub on the disk

grub
root (hd1,0)
setup (hd1)
quit

Exit out of your chroot and unmount everything

unmount your ebs root

umount ${EC2_DEST_MOUNT}/dev
umount ${EC2_DEST_MOUNT}

Now locally create a snapshot of your EBS root in case something goes wrong and you want to re-attach it; fix it; re-register.

snapshot your ebs-root

ec2-create-snapshot vol-03aff26e
SNAPSHOT  snap-9239e6f6   vol-03aff26e    pending 2012-01-07T23:37:08+0000        621517282122    8

stop your instance while it’s snapshotting

ec2-stop-instances i-de453ebc
INSTANCE  i-de453ebc  running stopping

Verify the instance is stopped

list your instance stopping

ec2-describe-instances i-de453ebc -H
Type  ReservationID   Owner   Groups  Platform
RESERVATION   r-f5743394  621517282122    default
INSTANCE  i-de453ebc  ami-0da96764            stopped ssh_key 0       cc1.4xlarge 2012-01-07T23:01:54+0000    us-east-1c              monitoring-disabled                 ebs                 hvm xen     sg-7baf4812 default
BLOCKDEVICE   /dev/sda1   vol-a9acf1c4    2012-01-07T23:39:34.000Z    
BLOCKDEVICE   /dev/sdp    vol-03aff26e    2012-01-07T23:39:34.000Z    

detach the old root volume provided by amazon and your ebs root volume you made

ec2-detach-volume vol-a9acf1c4
ATTACHMENT    vol-a9acf1c4    i-de453ebc  /dev/sda1   detaching   2012-01-07T23:39:34+0000
ec2-detach-volume vol-03aff26e
ATTACHMENT    vol-03aff26e    i-de453ebc  /dev/sdp    detaching   2012-01-07T23:39:34+0000

attach volume

ec2-attach-volume vol-03aff26e -i i-de453ebc -d /dev/sda1
ATTACHMENT    vol-03aff26e    i-de453ebc  /dev/sda1   attaching   2012-01-07T23:42:38+0000

create your AMI

ec2-create-image i-de453ebc -n "fail"
IMAGE ami-ed9d4a84

Once it’s done preparing the AMI; boot it and hope for success. If you have a problem review the Console log and research. I suggest reviewing Amazon’s Kernel configuration and Ubuntu’s Kernel configuration for the HVM instance types if you have any problems; most of all good luck!

Learn to Respect the Problem

Jul 24th, 2011

There is a time in your life when you start learning to respect the problem. Of course you can fix a problem easily by doing x. But if you spend a little more time you can fix x,y and z and in a way that you feel proud of.

Yes I am likely twisting the definition for my own gains.

Getting Chef 0.8 Bootstrapped and Up in Screen

Dec 14th, 2009

Simple post, just to detail if you want .8 and don’t want to use bootstrap or runit or init and you just want it working.

Get the base system up

sed -i 's/universe/multiverse universe/' /etc/apt/sources.list
apt-get update
env DEBIAN_FRONTEND=noninteractive apt-get upgrade -y
echo 'sun-java6-jdk shared/accepted-sun-dlj-v1-1 boolean true' | sudo debconf-set-selections
apt-get install -y ruby ruby1.8-dev libopenssl-ruby1.8 rdoc ri irb build-essential zlib1g-dev libxml2-dev wget ssl-cert git-core couchdb sun-java6-jdk sun-java6-jre

Setup gem to not annoy me

echo "gem: --no-rdoc --no-ri" >> ~/.gemrc

Now install rubygems,

cd /tmp
wget http://rubyforge.org/frs/download.php/60718/rubygems-1.3.5.tgz
tar zxf rubygems-1.3.5.tgz
cd rubygems-1.3.5
sudo ruby setup.rb --no-ri --no-rdoc
sudo ln -sfv /usr/bin/gem1.8 /usr/bin/gem

Install RabbitMQ

wget http://www.rabbitmq.com/releases/rabbitmq-server/v1.7.0/rabbitmq-server_1.7.0-1_all.deb
dpkg -i rabbitmq-server_1.7.0-1_all.deb

Make sure you have rake/etc installed

gem install rake cucumber rspec libxml-ruby jeweler thin unicorn merb-core merb-slices

now setup RabbitMQ so that chef can talk to it

rabbitmqctl add_vhost /chef
rabbitmqctl add_user chef testing
rabbitmqctl set_permissions -p /chef chef ".*" ".*" ".*"

Grab Chef .8 from master and start work

mkdir git;git clone git://github.com/opscode/chef.git

Install chef from master

rake install

You may catch a few missing dependencies, gem install them if I missed it (but I think this is right)
Now you need to install the chef-solr bins (the gemspec isn’t fixed to do this yet)

cd git/chef/chef-solr/bin;cp * /usr/bin

Grab mixlib-authentication from git

git clone git://github.com/opscode/mixlib-authentication.git

Install mixlib-authentication

cd mixlib-authentication && rake install

Grab mixlib-log from git

git clone git://github.com/opscode/mixlib-log

Install that also

cd mixlib-log && rake install

Now fire up screen and start it up
In not the perfect order, but.
Start chef-solr

chef-solr

Start chef-solr-indexer

chef-solr-indexer

Grab some configs that should work,

mkdir -p /etc/chef
cd /etc/chef;wget http://likens.us/server.rb  
cd /etc/chef;wget http://likens.us/client.rb

Start chef-server

chef-server

Now configure knife to work with the webui key and you should be solid.

knife configure

Run chef-client over localhost and it should register and work hopefully
Enjoy
Thanks to Bryan Helmkamp for updates to this post.

Why You Should Never Change API on a Minor Release

Nov 2nd, 2009

In this case, my pain is Chef; as 0.7.0 came out we were rocking with metadata.rb, as I wasn’t paying attention and reading release documentation properly suddenly that doesn’t do squat, and everything is in metadata.json. I’m sure there was something in one of the release notes that I omitted, however it brings up a valid point that Opscode has broken many times (I am only mentioning these 2) in the .7 release cycle

They have repeatedly broken the basic idea of not making major changes in a minor release, such as this, or when they re-wrote the bootstrap system; and then neutered the old system so you had to use the old bootstrap tarball (which is the only reason I am not up in arms).

… However, if you want to do things like this, you should release a new major version and allow people to get used to the major changes. There are always growing pains with anything new, different, … however the past has taught us that you should not make stupid major changes to a structure without a major release.

Opscode has made this much smoother since the original post of this message and they have delivered a much better product then this post originally ranted about.

Blog Archives

because we all share ...

undefined clue 'adversity' for nil:nil (NameError)