Infrastructure Testing


#1

Hi all,

I’ve been beating myself up over the last few days trying to figure out the best way of testing our build automation. I’ve played with cucumber-chef, minitest_handler, rspec-chef and more. All of them were either broken in implementation or theory. Here are my thoughts:

Cucumber-chef

I fought for 2 days with the gem trying to get it to work, and after multiple gem incompatibilities I thought I had it sorted. Until I got to the part where it builds an EC2 instance, which then failed miserably as well. In the end I dumped it because if it’s this hard to get running initially, how can I trust it to continue to work in the future without another 2+ days worth of headaches. In theory this was probably what I want, but over-complicated and brittle.

Rspec-chef

I’ve possibly misunderstood the concept behind this one, but it looks like it’s testing that I’ve written the correct pieces in my recipes rather than my infrastructure has turned out correctly after a run.

Minitest-Handler

There are 2 variants on this idea, 1 is strictly for chef-solo and the other is for chef-client. I don’t run chef-solo so I opted to try out the cookbook_file based method that was recently discussed on the mailing list (http://lists.opscode.com/sympa/arc/chef/2012-03/msg00000.html). I’m struggling to understand why you’d follow this method. Here’s why: A cookbook is generic and can be passed around between users/environments, for example the ‘generic-users’ cookbook. To add a files/default/tests/minitest/moo_test.rb to this cookbook and upload it, means I’ve just made that test specific to my environment but included it within a generic cookbook. It’s completely possible that I’ve not grasped the methodology on this, but it seems fundamentally wrong because the 2 should be separate.

What I’m looking for is something as simple as this workflow, and if it comes to light that nobody else has done this and I need to write my won then I’m happy to do so:

1 - I write my infrastructure test, describing what change I am about to implement
2 - I run my test suite
a - a vagrant/ec2/heroku/whatever instance is started
b - chef-client is run on that server as I would expect in production
c - my test suite looks to see if changes I want are in place
d - it fails
3 - I write my cookbook changes to implement the small change I want
4 - is our test instance running?
a - no, GOTO 2a
b - yes, GOTO 2b
5 - did it pass?
a - yes, GOTO 1
b - no, GOTO 3

I would expect my test suite to simply describe what should and should not be in place, running, or open. Have I completely missed the boat here and this already exists in a nice simple structure, or are we at the stage in Chef specific DevOps where we need to write something that fulfils this?

To be clear, I would expect such a tool to have a generic ‘setup’ step that isn’t automatic and needs a little work. Something like ‘start a vagrant instance with the role X’ or similar. I would also expect to have to write my own ‘teardown’.

Again, I must stress that I feel like this si such a simple thing that needs to be in place that I must have missed some vital information I need to get cracking. Apologies if it comes across as whiney, I’m just frustrated and need to know if I embark on such a project, would I be re-inventing a wheel I missed somewhere.

Cheers
Jamie van Dyke


#2

On Thu, Mar 1, 2012 at 12:13 PM, Jamie van Dyke jamie@fearoffish.com wrote:

Again, I must stress that I feel like this si such a simple thing that needs
to be in place that I must have missed some vital information I need to get
cracking. Apologies if it comes across as whiney, I’m just frustrated and
need to know if I embark on such a project, would I be re-inventing a wheel
I missed somewhere.

The problem here is that not everyone is solving the same problem and
we have different preferences about tools and language. TMTOWTDI.

Robert Berger started another mailing list for this kind of discussion
recently, but I’m not compelled to have the conversation on a
different list myself:
https://groups.google.com/forum/?fromgroups#!forum/chef-testing

Also, it is one of those things that most everyone thinks they should
do, but we don’t have to do it, so we spend more time thinking about
it and talking about it than implementing it.

Minitest-Handler

There are 2 variants on this idea, 1 is strictly for chef-solo and the other
is for chef-client. I don’t run chef-solo so I opted to try out the
cookbook_file based method that was recently discussed on the mailing list
(http://lists.opscode.com/sympa/arc/chef/2012-03/msg00000.html). I’m
struggling to understand why you’d follow this method. Here’s why: A
cookbook is generic and can be passed around between users/environments, for
example the ‘generic-users’ cookbook. To add a
files/default/tests/minitest/moo_test.rb to this cookbook and upload it,
means I’ve just made that test specific to my environment but included it
within a generic cookbook. It’s completely possible that I’ve not grasped
the methodology on this, but it seems fundamentally wrong because the 2
should be separate.

Calavera’s minitest-handler is the really same for chef-solo or
chef-server. In my example we use the chef_handler because lets us
install it to a node by simply using the run list rather than having
to edit the node configuration.

Opscode does a lot of heavy lifting trying to test a cookbook after
merging a community contribution to ensure the cookbook still does
what it is supposed to do. This gets more complex as the cookbook
becomes more complex, as well as when it supports multiple
distributions and operating systems. If the apache2 cookbook contained
integration tests for many use cases on many distributions we could
easily spin up cloud instances or virtual machines to verify that
these tests pass. It has proven harder to choose a framework for
integration testing than unit testing however, where we have already
landed solidly on rspec.

While the apache2 cookbook makes a more straight forward example, the
generic-users cookbook can still have quality tests written for it
that are not dependent on the user because the cookbook is data
driven. In fact, the minitest-handler solution is a particularly
strong choice here because we still have access to data in the chef
run, and can create a data driven test. For all of the users in the
data bag that should have been created, check on the system that they
actually made it there.

Bryan


#3

I had similar thoughts about these methods. I really hoped I could make
cucumber-chef work but we don’t use EC2 here.

So I wrote a vagrant cookbook which provides a vagrant resource to specify
what kind of OS you want to use, what recipes/roles you want to run, and
any additional configuration. I wrote it to support running chef-solo or
chef-client, and it can even create an embedded chef-server to connect your
instance to or you can specify a remote chef-server instance.

I also created an rspec cookbook which would allow you to write rspec tests
inside your recipes.

The main idea is to create separate cookbooks or recipes which test the
cookbook using vagrant/virtualbox to create a virtual host to install the
cookbooks and then using RSpec to test the functionality during the chef
run.

This project is still quite new, I don’t actually have any real tests
written at the moment but here is a sample idea. We also haven’t made this
project/cookbooks available publicly but I am hoping to shortly.

vagrant-test::default
{

vagrant “CENT55-32” do
action :up
recipes [“vagrant-test::beforeTest”, “tomcat”, “vagrant-test::afterTest”]end

}

vagrant-test::beforeTest

{

require ‘net/http’

rspec “Tomcat” do
it “should not be running” do
lambda {Net::HTTP.get_response(URI.parse(‘http://localhost:8080’))}.should
raise_error
endend

}

vagrant-test::afterTest

{

require ‘net/http’

rspec “Tomcat” do
it “should connect successfully” do
puts "Waiting for tomcat to start"
sleep 10
Net::HTTP.get_response(URI.parse(‘http://localhost:8080’)).is_a?(Net::HTTPSuccess).should
be(true)
endend

}

On Thu, Mar 1, 2012 at 11:13 AM, Jamie van Dyke jamie@fearoffish.comwrote:

Hi all,

I’ve been beating myself up over the last few days trying to figure out
the best way of testing our build automation. I’ve played with
cucumber-chef, minitest_handler, rspec-chef and more. All of them were
either broken in implementation or theory. Here are my thoughts:

Cucumber-chef

I fought for 2 days with the gem trying to get it to work, and after
multiple gem incompatibilities I thought I had it sorted. Until I got to
the part where it builds an EC2 instance, which then failed miserably as
well. In the end I dumped it because if it’s this hard to get running
initially, how can I trust it to continue to work in the future without
another 2+ days worth of headaches. In theory this was probably what I
want, but over-complicated and brittle.

Rspec-chef

I’ve possibly misunderstood the concept behind this one, but it looks like
it’s testing that I’ve written the correct pieces in my recipes rather than
my infrastructure has turned out correctly after a run.

Minitest-Handler

There are 2 variants on this idea, 1 is strictly for chef-solo and the
other is for chef-client. I don’t run chef-solo so I opted to try out the
cookbook_file based method that was recently discussed on the mailing list (
http://lists.opscode.com/sympa/arc/chef/2012-03/msg00000.html). I’m
struggling to understand why you’d follow this method. Here’s why: A
cookbook is generic and can be passed around between users/environments,
for example the ‘generic-users’ cookbook. To add a
files/default/tests/minitest/moo_test.rb to this cookbook and upload it,
means I’ve just made that test specific to my environment but included it
within a generic cookbook. It’s completely possible that I’ve not grasped
the methodology on this, but it seems fundamentally wrong because the 2
should be separate.

What I’m looking for is something as simple as this workflow, and if it
comes to light that nobody else has done this and I need to write my won
then I’m happy to do so:

1 - I write my infrastructure test, describing what change I am about to
implement
2 - I run my test suite
a - a vagrant/ec2/heroku/whatever instance is started
b - chef-client is run on that server as I would expect in production
c - my test suite looks to see if changes I want are in place
d - it fails
3 - I write my cookbook changes to implement the small change I want
4 - is our test instance running?
a - no, GOTO 2a
b - yes, GOTO 2b
5 - did it pass?
a - yes, GOTO 1
b - no, GOTO 3

I would expect my test suite to simply describe what should and should not
be in place, running, or open. Have I completely missed the boat here and
this already exists in a nice simple structure, or are we at the stage in
Chef specific DevOps where we need to write something that fulfils this?

To be clear, I would expect such a tool to have a generic ‘setup’ step
that isn’t automatic and needs a little work. Something like ‘start a
vagrant instance with the role X’ or similar. I would also expect to have
to write my own ‘teardown’.

Again, I must stress that I feel like this si such a simple thing that
needs to be in place that I must have missed some vital information I need
to get cracking. Apologies if it comes across as whiney, I’m just
frustrated and need to know if I embark on such a project, would I be
re-inventing a wheel I missed somewhere.

Cheers
Jamie van Dyke


-Bryan


#4

@Jamie, that looks cool among the experimental chef testings attempt at all.

This one too looks promising


@millisami
~ Sachin Sagar Rai
Ruby on Rails Developer
http://tfm.com.np
http://nepalonrails.tumblr.com
Sent with Sparrow (http://www.sparrowmailapp.com/?sig)

On Friday, March 2, 2012 at 9:57 PM, Jamie van Dyke wrote:

I decided to do something simple for now, and work on it as I go. That way I have a better idea of requirements and how I will actually use it. My thinking was simple, I want to boot something, ssh into it, run tests against the structure, and shut it down. I don’t want to boot/terminate it every run though, so I separate that piece.

So here’s what I put together this morning, I installed rspec, and created the following folder/file structure:

spec
├── cookbooks
│ ├── build_essentials_spec.rb
│ └── generic_users_spec.rb
├── spec_helper.rb
└── support
├── matchers
└── models
└── host.rb

Here are the files:

spec_helper.rb

require 'bundler’
begin
Bundler.setup
rescue Bundler::BundlerError => e
$stderr.puts e.message
$stderr.puts "Run bundle install to install missing gems"
exit e.status_code
end

Requires supporting ruby files with custom matchers and macros, etc,

in spec/support/ and its subdirectories.

Dir.glob("./spec/support/**/*.rb").each {|f| require f}

RSpec.configure do |config|
config.before(:all) do
# Note: I use vagrant with chef-client provisioning, this is what it uses to run chef-client
host.run_chef(“chef-client -c /tmp/vagrant-chef-1/client.rb -j /tmp/vagrant-chef-1/dna.json”)
end

config.after(:all) do
host.close
end
end

start an instance to test

run chef

test the structure looks as we want it

This represents the host we’re running tests against

def host

I set up this hostname using vagrant ssh-config

@host ||= Host.new ‘cc.local’, ‘vagrant’, :port => '2222’
end

host.rb

require ‘net/ssh’

class Host
def initialize(host, user, options={})
@ssh = Net::SSH.start(host, user, options)
end

def close
@ssh.close
end

def run_chef(chef_command=“sudo chef-client”)
@ssh.exec! chef_command
end

def installed_packages
@packages ||= @ssh.exec! ‘dpkg --get-selections’
@packages.split("\n").collect {|pkg| pkg.split(" ")[0] }
end

def users
@users ||= @ssh.exec! ‘cat /etc/passwd’
@users.split("\n").collect {|u| u.split(":")[0] }
end
end

Vagrantfile

Vagrant::Config.run do |config|
config.vm.define :cc do |host|
host.vm.box = "cc.dev"
host.vm.forward_port 80, 8080
host.vm.forward_port 22, 2222
host.vm.network :hostonly, “33.33.33.220”

host.vm.provision :chef_client do |chef|
  # chef.log_level              = :debug
  chef.node_name            = "cc.dev"
  chef.chef_server_url        = "https://api.opscode.com/organizations/#{ENV['ORG']}"
  chef.environment            = "develop"
  chef.validation_client_name = ENV['validation_key_name']
  chef.validation_key_path    = ENV['my_key_location']
   
  chef.add_role 'base'
end

end
end

Rakefile

task :go => [“vagrant:up”, “spec”]
task :setup => [“vagrant:up”]
task :teardown => [“vagrant:destroy”, “knife:client_delete”, “knife:node_delete”]

task :spec do
sh 'bundle exec rspec spec/cookbooks/*.rb’
end

namespace :vagrant do
task :up do
safe_task do
sh 'bundle exec vagrant up cc’
end
end

task :destroy do
safe_task do
sh 'bundle exec vagrant destroy’
end
end

task :provision do
safe_task do
sh 'bundle exec vagrant provision’
end
end
end

namespace :knife do
task :client_delete do
safe_task do
sh 'bundle exec knife client delete cc.dev -y’
end
end

task :node_delete do
safe_task do
sh 'bundle exec knife node delete cc.dev -y’
end
end
end

def safe_task(&block)
begin
yield
rescue
puts "Command failed. Next!"
end
end

Gemfile

source :rubygems

gem ‘vagrant’, ‘= 0.9.7’

stop a clash between net-ssh dependencies (vagrant requires a higher version)

gem “chef”,
:git => “https://github.com/opscode/chef.git”,
:ref => “ba4d58f4223”

gem ‘rspec’, ‘= 2.8.0’

Obviously this is the start of something, not a finished idea. It’s just to show whoever wants to try this out, how easy it actually is. The Host class is the start of something that needs more thinking, or might just end up being replaced with something else.

Thoughts?

Cheers
Jamie van Dyke

On Mar 1, 2012, at 5:28 PM, Bryan Baugher wrote:

I had similar thoughts about these methods. I really hoped I could make cucumber-chef work but we don’t use EC2 here.

So I wrote a vagrant cookbook which provides a vagrant resource to specify what kind of OS you want to use, what recipes/roles you want to run, and any additional configuration. I wrote it to support running chef-solo or chef-client, and it can even create an embedded chef-server to connect your instance to or you can specify a remote chef-server instance.

I also created an rspec cookbook which would allow you to write rspec tests inside your recipes.

The main idea is to create separate cookbooks or recipes which test the cookbook using vagrant/virtualbox to create a virtual host to install the cookbooks and then using RSpec to test the functionality during the chef run.

This project is still quite new, I don’t actually have any real tests written at the moment but here is a sample idea. We also haven’t made this project/cookbooks available publicly but I am hoping to shortly.

vagrant-test::default
{

vagrant “CENT55-32” do action :up recipes [“vagrant-test::beforeTest”, “tomcat”, “vagrant-test::afterTest”] end

}

vagrant-test::beforeTest
{
require ‘net/http’ rspec “Tomcat” do it “should not be running” do lambda {Net::HTTP.get_response(URI.parse(‘http://localhost:8080 (http://localhost:8080/)’))}.should raise_error end end

}

vagrant-test::afterTest
{
require ‘net/http’ rspec “Tomcat” do it “should connect successfully” do puts “Waiting for tomcat to start” sleep 10 Net::HTTP.get_response(URI.parse(‘http://localhost:8080 (http://localhost:8080/)’)).is_a?(Net::HTTPSuccess).should be(true) end end

}

On Thu, Mar 1, 2012 at 11:13 AM, Jamie van Dyke <jamie@fearoffish.com (mailto:jamie@fearoffish.com)> wrote:

Hi all,

I’ve been beating myself up over the last few days trying to figure out the best way of testing our build automation. I’ve played with cucumber-chef, minitest_handler, rspec-chef and more. All of them were either broken in implementation or theory. Here are my thoughts:

Cucumber-chef

I fought for 2 days with the gem trying to get it to work, and after multiple gem incompatibilities I thought I had it sorted. Until I got to the part where it builds an EC2 instance, which then failed miserably as well. In the end I dumped it because if it’s this hard to get running initially, how can I trust it to continue to work in the future without another 2+ days worth of headaches. In theory this was probably what I want, but over-complicated and brittle.

Rspec-chef

I’ve possibly misunderstood the concept behind this one, but it looks like it’s testing that I’ve written the correct pieces in my recipes rather than my infrastructure has turned out correctly after a run.

Minitest-Handler

There are 2 variants on this idea, 1 is strictly for chef-solo and the other is for chef-client. I don’t run chef-solo so I opted to try out the cookbook_file based method that was recently discussed on the mailing list (http://lists.opscode.com/sympa/arc/chef/2012-03/msg00000.html). I’m struggling to understand why you’d follow this method. Here’s why: A cookbook is generic and can be passed around between users/environments, for example the ‘generic-users’ cookbook. To add a files/default/tests/minitest/moo_test.rb to this cookbook and upload it, means I’ve just made that test specific to my environment but included it within a generic cookbook. It’s completely possible that I’ve not grasped the methodology on this, but it seems fundamentally wrong because the 2 should be separate.

What I’m looking for is something as simple as this workflow, and if it comes to light that nobody else has done this and I need to write my won then I’m happy to do so:

1 - I write my infrastructure test, describing what change I am about to implement
2 - I run my test suite
a - a vagrant/ec2/heroku/whatever instance is started
b - chef-client is run on that server as I would expect in production
c - my test suite looks to see if changes I want are in place
d - it fails
3 - I write my cookbook changes to implement the small change I want
4 - is our test instance running?
a - no, GOTO 2a
b - yes, GOTO 2b
5 - did it pass?
a - yes, GOTO 1
b - no, GOTO 3

I would expect my test suite to simply describe what should and should not be in place, running, or open. Have I completely missed the boat here and this already exists in a nice simple structure, or are we at the stage in Chef specific DevOps where we need to write something that fulfils this?

To be clear, I would expect such a tool to have a generic ‘setup’ step that isn’t automatic and needs a little work. Something like ‘start a vagrant instance with the role X’ or similar. I would also expect to have to write my own ‘teardown’.

Again, I must stress that I feel like this si such a simple thing that needs to be in place that I must have missed some vital information I need to get cracking. Apologies if it comes across as whiney, I’m just frustrated and need to know if I embark on such a project, would I be re-inventing a wheel I missed somewhere.

Cheers
Jamie van Dyke


-Bryan


#5

Several folks got in a conversation off of Twitter and ended up starting
http://groups.google.com/group/chef-testing to discuss this topic.

Here’s a combination of the tools that we’ve found and my rough thoughts on
how they might fit together:
http://groups.google.com/group/chef-testing/browse_thread/thread/523fde7ce4f9eaf2?hl=en

The conversation started with how we’d like to incorporate these tools
together with Ironfan to have some truly end-to-end workflow for testing
all the way from recipes to a full cluster. The Infochimps guys have been
doing a lot of stuff on Ironfan’s new release, so my position has been
"wait and see" what they come up with.

The tool that’s been working best for me is a combination of
cucumber-nagios and toft (https://github.com/exceedhl/toft). It uses LXC
containers to quickly fire up instances, apply the current repository to it
as a chef-solo recipe, and use cucumber to check it. It does have a Vagrant
setup that I haven’t tried.

Toft still needs a bit of work, though, to be a graceful solution. Since it
uses chef-solo, anything with a search will fail. I’m sure that can be
mocked and incorporated as part of the harness. The whole thing could be
easily extended so that the cucumber features run against EC2 nodes, etc.
I’m thinking of extending Ironfan to have a Toft cloud service, pending the
cooldown of that project.

Any thoughts on the subject would be greatly appreciated in the google
group.

On Thu, Mar 1, 2012 at 12:13 PM, Jamie van Dyke jamie@fearoffish.comwrote:

Hi all,

I’ve been beating myself up over the last few days trying to figure out
the best way of testing our build automation. I’ve played with
cucumber-chef, minitest_handler, rspec-chef and more. All of them were
either broken in implementation or theory. Here are my thoughts:

Cucumber-chef

I fought for 2 days with the gem trying to get it to work, and after
multiple gem incompatibilities I thought I had it sorted. Until I got to
the part where it builds an EC2 instance, which then failed miserably as
well. In the end I dumped it because if it’s this hard to get running
initially, how can I trust it to continue to work in the future without
another 2+ days worth of headaches. In theory this was probably what I
want, but over-complicated and brittle.

Rspec-chef

I’ve possibly misunderstood the concept behind this one, but it looks like
it’s testing that I’ve written the correct pieces in my recipes rather than
my infrastructure has turned out correctly after a run.

Minitest-Handler

There are 2 variants on this idea, 1 is strictly for chef-solo and the
other is for chef-client. I don’t run chef-solo so I opted to try out the
cookbook_file based method that was recently discussed on the mailing list (
http://lists.opscode.com/sympa/arc/chef/2012-03/msg00000.html). I’m
struggling to understand why you’d follow this method. Here’s why: A
cookbook is generic and can be passed around between users/environments,
for example the ‘generic-users’ cookbook. To add a
files/default/tests/minitest/moo_test.rb to this cookbook and upload it,
means I’ve just made that test specific to my environment but included it
within a generic cookbook. It’s completely possible that I’ve not grasped
the methodology on this, but it seems fundamentally wrong because the 2
should be separate.

What I’m looking for is something as simple as this workflow, and if it
comes to light that nobody else has done this and I need to write my won
then I’m happy to do so:

1 - I write my infrastructure test, describing what change I am about to
implement
2 - I run my test suite
a - a vagrant/ec2/heroku/whatever instance is started
b - chef-client is run on that server as I would expect in production
c - my test suite looks to see if changes I want are in place
d - it fails
3 - I write my cookbook changes to implement the small change I want
4 - is our test instance running?
a - no, GOTO 2a
b - yes, GOTO 2b
5 - did it pass?
a - yes, GOTO 1
b - no, GOTO 3

I would expect my test suite to simply describe what should and should not
be in place, running, or open. Have I completely missed the boat here and
this already exists in a nice simple structure, or are we at the stage in
Chef specific DevOps where we need to write something that fulfils this?

To be clear, I would expect such a tool to have a generic ‘setup’ step
that isn’t automatic and needs a little work. Something like ‘start a
vagrant instance with the role X’ or similar. I would also expect to have
to write my own ‘teardown’.

Again, I must stress that I feel like this si such a simple thing that
needs to be in place that I must have missed some vital information I need
to get cracking. Apologies if it comes across as whiney, I’m just
frustrated and need to know if I embark on such a project, would I be
re-inventing a wheel I missed somewhere.

Cheers
Jamie van Dyke


Jason Wagner
jason@tensorwrench.com


#6

I would recommend looking at cucumber-nagios http://auxesis.github.com/cucumber-nagios. It’s just one piece of the puzzle but it helped me.

Alex

On Thursday, March 1, 2012 at 9:28 AM, Bryan Baugher wrote:

I had similar thoughts about these methods. I really hoped I could make cucumber-chef work but we don’t use EC2 here.

So I wrote a vagrant cookbook which provides a vagrant resource to specify what kind of OS you want to use, what recipes/roles you want to run, and any additional configuration. I wrote it to support running chef-solo or chef-client, and it can even create an embedded chef-server to connect your instance to or you can specify a remote chef-server instance.

I also created an rspec cookbook which would allow you to write rspec tests inside your recipes.

The main idea is to create separate cookbooks or recipes which test the cookbook using vagrant/virtualbox to create a virtual host to install the cookbooks and then using RSpec to test the functionality during the chef run.

This project is still quite new, I don’t actually have any real tests written at the moment but here is a sample idea. We also haven’t made this project/cookbooks available publicly but I am hoping to shortly.

vagrant-test::default
{

vagrant “CENT55-32” do action :up recipes [“vagrant-test::beforeTest”, “tomcat”, “vagrant-test::afterTest”] end

}

vagrant-test::beforeTest
{
require ‘net/http’ rspec “Tomcat” do it “should not be running” do lambda {Net::HTTP.get_response(URI.parse(‘http://localhost:8080’))}.should raise_error end end

}

vagrant-test::afterTest
{
require ‘net/http’ rspec “Tomcat” do it “should connect successfully” do puts “Waiting for tomcat to start” sleep 10 Net::HTTP.get_response(URI.parse(‘http://localhost:8080’)).is_a?(Net::HTTPSuccess).should be(true) end end

}

On Thu, Mar 1, 2012 at 11:13 AM, Jamie van Dyke <jamie@fearoffish.com (mailto:jamie@fearoffish.com)> wrote:

Hi all,

I’ve been beating myself up over the last few days trying to figure out the best way of testing our build automation. I’ve played with cucumber-chef, minitest_handler, rspec-chef and more. All of them were either broken in implementation or theory. Here are my thoughts:

Cucumber-chef

I fought for 2 days with the gem trying to get it to work, and after multiple gem incompatibilities I thought I had it sorted. Until I got to the part where it builds an EC2 instance, which then failed miserably as well. In the end I dumped it because if it’s this hard to get running initially, how can I trust it to continue to work in the future without another 2+ days worth of headaches. In theory this was probably what I want, but over-complicated and brittle.

Rspec-chef

I’ve possibly misunderstood the concept behind this one, but it looks like it’s testing that I’ve written the correct pieces in my recipes rather than my infrastructure has turned out correctly after a run.

Minitest-Handler

There are 2 variants on this idea, 1 is strictly for chef-solo and the other is for chef-client. I don’t run chef-solo so I opted to try out the cookbook_file based method that was recently discussed on the mailing list (http://lists.opscode.com/sympa/arc/chef/2012-03/msg00000.html). I’m struggling to understand why you’d follow this method. Here’s why: A cookbook is generic and can be passed around between users/environments, for example the ‘generic-users’ cookbook. To add a files/default/tests/minitest/moo_test.rb to this cookbook and upload it, means I’ve just made that test specific to my environment but included it within a generic cookbook. It’s completely possible that I’ve not grasped the methodology on this, but it seems fundamentally wrong because the 2 should be separate.

What I’m looking for is something as simple as this workflow, and if it comes to light that nobody else has done this and I need to write my won then I’m happy to do so:

1 - I write my infrastructure test, describing what change I am about to implement
2 - I run my test suite
a - a vagrant/ec2/heroku/whatever instance is started
b - chef-client is run on that server as I would expect in production
c - my test suite looks to see if changes I want are in place
d - it fails
3 - I write my cookbook changes to implement the small change I want
4 - is our test instance running?
a - no, GOTO 2a
b - yes, GOTO 2b
5 - did it pass?
a - yes, GOTO 1
b - no, GOTO 3

I would expect my test suite to simply describe what should and should not be in place, running, or open. Have I completely missed the boat here and this already exists in a nice simple structure, or are we at the stage in Chef specific DevOps where we need to write something that fulfils this?

To be clear, I would expect such a tool to have a generic ‘setup’ step that isn’t automatic and needs a little work. Something like ‘start a vagrant instance with the role X’ or similar. I would also expect to have to write my own ‘teardown’.

Again, I must stress that I feel like this si such a simple thing that needs to be in place that I must have missed some vital information I need to get cracking. Apologies if it comes across as whiney, I’m just frustrated and need to know if I embark on such a project, would I be re-inventing a wheel I missed somewhere.

Cheers
Jamie van Dyke


-Bryan


#7

I decided to do something simple for now, and work on it as I go. That way I have a better idea of requirements and how I will actually use it. My thinking was simple, I want to boot something, ssh into it, run tests against the structure, and shut it down. I don’t want to boot/terminate it every run though, so I separate that piece.

So here’s what I put together this morning, I installed rspec, and created the following folder/file structure:

spec
├── cookbooks
│ ├── build_essentials_spec.rb
│ └── generic_users_spec.rb
├── spec_helper.rb
└── support
├── matchers
└── models
└── host.rb

Here are the files:

spec_helper.rb

require 'bundler’
begin
Bundler.setup
rescue Bundler::BundlerError => e
$stderr.puts e.message
$stderr.puts "Run bundle install to install missing gems"
exit e.status_code
end

Requires supporting ruby files with custom matchers and macros, etc,

in spec/support/ and its subdirectories.

Dir.glob("./spec/support/**/*.rb").each {|f| require f}

RSpec.configure do |config|
config.before(:all) do
# Note: I use vagrant with chef-client provisioning, this is what it uses to run chef-client
host.run_chef(“chef-client -c /tmp/vagrant-chef-1/client.rb -j /tmp/vagrant-chef-1/dna.json”)
end

config.after(:all) do
host.close
end
end

start an instance to test

run chef

test the structure looks as we want it

This represents the host we’re running tests against

def host

I set up this hostname using vagrant ssh-config

@host ||= Host.new ‘cc.local’, ‘vagrant’, :port => '2222’
end

host.rb

require ‘net/ssh’

class Host
def initialize(host, user, options={})
@ssh = Net::SSH.start(host, user, options)
end

def close
@ssh.close
end

def run_chef(chef_command=“sudo chef-client”)
@ssh.exec! chef_command
end

def installed_packages
@packages ||= @ssh.exec! ‘dpkg --get-selections’
@packages.split("\n").collect {|pkg| pkg.split(" ")[0] }
end

def users
@users ||= @ssh.exec! ‘cat /etc/passwd’
@users.split("\n").collect {|u| u.split(":")[0] }
end
end

Vagrantfile

Vagrant::Config.run do |config|
config.vm.define :cc do |host|
host.vm.box = "cc.dev"
host.vm.forward_port 80, 8080
host.vm.forward_port 22, 2222
host.vm.network :hostonly, “33.33.33.220”

host.vm.provision :chef_client do |chef|
  # chef.log_level              = :debug
  chef.node_name            = "cc.dev"
  chef.chef_server_url        = "https://api.opscode.com/organizations/#{ENV['ORG']}"
  chef.environment            = "develop"
  chef.validation_client_name = ENV['validation_key_name']
  chef.validation_key_path    = ENV['my_key_location']
  
  chef.add_role 'base'
end

end
end

Rakefile

task :go => [“vagrant:up”, “spec”]
task :setup => [“vagrant:up”]
task :teardown => [“vagrant:destroy”, “knife:client_delete”, “knife:node_delete”]

task :spec do
sh 'bundle exec rspec spec/cookbooks/*.rb’
end

namespace :vagrant do
task :up do
safe_task do
sh 'bundle exec vagrant up cc’
end
end

task :destroy do
safe_task do
sh 'bundle exec vagrant destroy’
end
end

task :provision do
safe_task do
sh 'bundle exec vagrant provision’
end
end
end

namespace :knife do
task :client_delete do
safe_task do
sh 'bundle exec knife client delete cc.dev -y’
end
end

task :node_delete do
safe_task do
sh 'bundle exec knife node delete cc.dev -y’
end
end
end

def safe_task(&block)
begin
yield
rescue
puts "Command failed. Next!"
end
end

Gemfile

source :rubygems

gem ‘vagrant’, ‘= 0.9.7’

stop a clash between net-ssh dependencies (vagrant requires a higher version)

gem “chef”,
:git => “https://github.com/opscode/chef.git”,
:ref => “ba4d58f4223”

gem ‘rspec’, ‘= 2.8.0’

Obviously this is the start of something, not a finished idea. It’s just to show whoever wants to try this out, how easy it actually is. The Host class is the start of something that needs more thinking, or might just end up being replaced with something else.

Thoughts?

Cheers
Jamie van Dyke

On Mar 1, 2012, at 5:28 PM, Bryan Baugher wrote:

I had similar thoughts about these methods. I really hoped I could make cucumber-chef work but we don’t use EC2 here.

So I wrote a vagrant cookbook which provides a vagrant resource to specify what kind of OS you want to use, what recipes/roles you want to run, and any additional configuration. I wrote it to support running chef-solo or chef-client, and it can even create an embedded chef-server to connect your instance to or you can specify a remote chef-server instance.

I also created an rspec cookbook which would allow you to write rspec tests inside your recipes.

The main idea is to create separate cookbooks or recipes which test the cookbook using vagrant/virtualbox to create a virtual host to install the cookbooks and then using RSpec to test the functionality during the chef run.

This project is still quite new, I don’t actually have any real tests written at the moment but here is a sample idea. We also haven’t made this project/cookbooks available publicly but I am hoping to shortly.

vagrant-test::default
{

vagrant “CENT55-32” do
action :up
recipes [“vagrant-test::beforeTest”, “tomcat”, “vagrant-test::afterTest”]
end

}

vagrant-test::beforeTest
{
require ‘net/http’

rspec “Tomcat” do
it “should not be running” do
lambda {Net::HTTP.get_response(URI.parse(‘http://localhost:8080’))}.should raise_error
end
end
}

vagrant-test::afterTest
{
require ‘net/http’

rspec “Tomcat” do
it “should connect successfully” do
puts "Waiting for tomcat to start"
sleep 10
Net::HTTP.get_response(URI.parse(‘http://localhost:8080’)).is_a?(Net::HTTPSuccess).should be(true)
end
end
}

On Thu, Mar 1, 2012 at 11:13 AM, Jamie van Dyke jamie@fearoffish.com wrote:
Hi all,

I’ve been beating myself up over the last few days trying to figure out the best way of testing our build automation. I’ve played with cucumber-chef, minitest_handler, rspec-chef and more. All of them were either broken in implementation or theory. Here are my thoughts:

Cucumber-chef

I fought for 2 days with the gem trying to get it to work, and after multiple gem incompatibilities I thought I had it sorted. Until I got to the part where it builds an EC2 instance, which then failed miserably as well. In the end I dumped it because if it’s this hard to get running initially, how can I trust it to continue to work in the future without another 2+ days worth of headaches. In theory this was probably what I want, but over-complicated and brittle.

Rspec-chef

I’ve possibly misunderstood the concept behind this one, but it looks like it’s testing that I’ve written the correct pieces in my recipes rather than my infrastructure has turned out correctly after a run.

Minitest-Handler

There are 2 variants on this idea, 1 is strictly for chef-solo and the other is for chef-client. I don’t run chef-solo so I opted to try out the cookbook_file based method that was recently discussed on the mailing list (http://lists.opscode.com/sympa/arc/chef/2012-03/msg00000.html). I’m struggling to understand why you’d follow this method. Here’s why: A cookbook is generic and can be passed around between users/environments, for example the ‘generic-users’ cookbook. To add a files/default/tests/minitest/moo_test.rb to this cookbook and upload it, means I’ve just made that test specific to my environment but included it within a generic cookbook. It’s completely possible that I’ve not grasped the methodology on this, but it seems fundamentally wrong because the 2 should be separate.

What I’m looking for is something as simple as this workflow, and if it comes to light that nobody else has done this and I need to write my won then I’m happy to do so:

1 - I write my infrastructure test, describing what change I am about to implement
2 - I run my test suite
a - a vagrant/ec2/heroku/whatever instance is started
b - chef-client is run on that server as I would expect in production
c - my test suite looks to see if changes I want are in place
d - it fails
3 - I write my cookbook changes to implement the small change I want
4 - is our test instance running?
a - no, GOTO 2a
b - yes, GOTO 2b
5 - did it pass?
a - yes, GOTO 1
b - no, GOTO 3

I would expect my test suite to simply describe what should and should not be in place, running, or open. Have I completely missed the boat here and this already exists in a nice simple structure, or are we at the stage in Chef specific DevOps where we need to write something that fulfils this?

To be clear, I would expect such a tool to have a generic ‘setup’ step that isn’t automatic and needs a little work. Something like ‘start a vagrant instance with the role X’ or similar. I would also expect to have to write my own ‘teardown’.

Again, I must stress that I feel like this si such a simple thing that needs to be in place that I must have missed some vital information I need to get cracking. Apologies if it comes across as whiney, I’m just frustrated and need to know if I embark on such a project, would I be re-inventing a wheel I missed somewhere.

Cheers
Jamie van Dyke


-Bryan