Serverspec vs Mintest Feedback

Chef Infra (archive)
1

TLDR - I’m interesting in knowing why folks are moving away from Minitest,
toward ServerSpec.

I’ve been noticing a lot of cookbooks recently have been migrating their
testings from minitest to ServerSpec. I’ve got nothing against ServerSpec,
but I’m curious why/where this trend is coming from.

One of the things I have always liked about minitest handler is that the
tests “follow” the cookbook. As an example we run the handler in our dev
and qa environments which we converge hourly, so as a by product we are
testing the integration of that cookbook in more places than just a
developers local workstation, and we are doing it on an hourly basis. As I
understand this is might be possible with ServerSpec, however its not as
simple, and comes with the lose of the node object data?

Maybe my question would be better asked in the context of an example…
Lets assume I am consuming a community cookbook which sets up some
application that has an HTTP port. There is an attribute that controls what
that HTTP port is.
In ServerSpec land, I think you need to hard code the HTTP port to test?
In Minitest, I have access to the node object, so my test is not hard coded
to a port, but instead uses the value of the node attribute.

For the sake of the discussion, lets say I run that application on port 78
in dev, 79 in qa, and 80 in production. While using minitests I’m able to
assert the application is working properly, because the minitest test would
be reading the node attribute. However from what I’ve seen with ServerSpec,
I don’t have that same ability to adapt the test to use the right port,
without manual intervention?

Have I totally missed the boat here on ServerSpec capabilities?

Thanks for any feedback/experience/war stories.

2

On Sat, Jan 3, 2015 at 12:00 PM, David Petzel davidpetzel@gmail.com wrote:

TLDR - I'm interesting in knowing why folks are moving away from Minitest,
toward ServerSpec.

I think your statement:

" As I understand this is might be possible with ServerSpec, however
its not as simple, and comes with the lose of the node object data?"

is the crux of the reason why people are moving away from Minitest to
Serverspec -- because folks want the acceptance tests to not be
contaminated in any way by the mechanism by which it got there. They
see it as a /benefit/ that node state is not shared between the two.

  • Julian

--
[ Julian C. Dunn jdunn@aquezada.com * Sorry, I'm ]
[ WWW: Julian Dunn's Blog - Commentary on media, technology, and everything in between. * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]

3

and you can always construct the node object as

Chef::Config.from_file '/etc/chef/clinet.rb'
node = Chef::Node.load(Chef::Config[:node_name])

..but as julian mentioned, chef is not a test harness system, but
rspec+serverspec is. Also, serverspec provides ssh and various other mode
of transport (specinfra subproject),

cheers
ranjib

On Mon, Jan 5, 2015 at 12:31 AM, Julian C. Dunn jdunn@aquezada.com wrote:

On Sat, Jan 3, 2015 at 12:00 PM, David Petzel davidpetzel@gmail.com
wrote:

TLDR - I'm interesting in knowing why folks are moving away from
Minitest,
toward ServerSpec.

I think your statement:

" As I understand this is might be possible with ServerSpec, however
its not as simple, and comes with the lose of the node object data?"

is the crux of the reason why people are moving away from Minitest to
Serverspec -- because folks want the acceptance tests to not be
contaminated in any way by the mechanism by which it got there. They
see it as a /benefit/ that node state is not shared between the two.

  • Julian

--
[ Julian C. Dunn jdunn@aquezada.com * Sorry, I'm ]
[ WWW: Julian Dunn's Blog - Commentary on media, technology, and everything in between. * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]

4

Hi guys,
What I need is to change nodes run_list basing on their tags and roles and names etc.
For example I have multiple nodes with different roles etc.
And I would like to assign some particular recipe to nodes which met some condition:
knife exec -E “nodes.find(:name => ‘isec’) { |node| node.run_list << “recipe[foo]”; n.save }”

It will work for nodes which have isec in its names.

But what if I would like to use query something like:
knife exec -E “nodes.find(:name => ‘isec’ AND :tags => ‘tag_name’ AND :role => ‘role_name’) { |node| node.run_list << “recipe[foo]”; n.save }”

It fails. As it seems that “nodes.find” method only can take 1 statement. And right after first statement it is looking for ‘)’ instead of AND or other things… Or at least I have not found other usage.
Can you please help me with this issue?
Many thanks in advance.
Regards, Taras.

5

Ohai!

On Sat, Jan 3, 2015 at 1:00 PM, David Petzel davidpetzel@gmail.com wrote:

TLDR - I'm interesting in knowing why folks are moving away from Minitest,
toward ServerSpec.

As Julian said, there's an element of not wanting to "contaminate" the
tests. However, there's a couple other reasons.

I've been noticing a lot of cookbooks recently have been migrating their
testings from minitest to ServerSpec. I've got nothing against ServerSpec,
but I'm curious why/where this trend is coming from.

Back in the before times, that is, before test kitchen and the notion of
bussers, the best (only?) way of reasonably testing nodes post
convergence was through the minitest chef handler. The early version of
test kitchen (pre 1.0) did run those, but the rewritten version, with the
bussers made it more flexible.

I think Serverspec is popular because Rspec is popular. It's not a value
judgment, just more widely used - Rspec is what CHEF uses for chef itself,
Ruby and Ruby on Rails programmers are familiar with Rspec. While minitest
is in the Ruby stdlib (and I like it more than Rspec for that reason), it's
simply not widespread. Also, Serverspec does an "outside in" kind of test
approach. I have more confidence that the tests are doing the right thing.
I can defer configurable options to attributes, and then set those
attributes on a per-suite basis.

Maybe my question would be better asked in the context of an example...

Lets assume I am consuming a community cookbook which sets up some
application that has an HTTP port. There is an attribute that controls what
that HTTP port is.
In ServerSpec land, I think you need to hard code the HTTP port to test?
In Minitest, I have access to the node object, so my test is not hard
coded to a port, but instead uses the value of the node attribute.

For the sake of the discussion, lets say I run that application on port 78
in dev, 79 in qa, and 80 in production. While using minitests I'm able to
assert the application is working properly, because the minitest test would
be reading the node attribute. However from what I've seen with ServerSpec,
I don't have that same ability to adapt the test to use the right port,
without manual intervention?

This is why I like, and use test kitchen. I can set up two suites.

suites:

  • name: default
    run_list:
    • recipe[webapp]
      attributes:
  • name: port78
    run_list:
    • recipe[webapp]
      attributes:
      www:
      port: 78

Then I can have tests for each suite, one that checks that my web app
responds on port 80 (presuming its the default), and responds on port 78. I
don't actually care about the different ports in different environments, I
want to test that the non-default port does the right thing. Adding more
suites that only change this attribute just tests that Chef works, which
we've already proved in the first case. In fact, if the port is an
attribute, I don't think the default suite is really really necessary.

Cheers,
Joshua

6

On Monday, January 5, 2015 at 4:55 AM, klum_tz@ukr.net wrote:

Hi guys,

What I need is to change nodes run_list basing on their tags and roles and names etc.

For example I have multiple nodes with different roles etc.

And I would like to assign some particular recipe to nodes which met some condition:

knife exec -E "nodes.find(:name => 'isec') { |node| node.run_list << "recipe[foo]"; n.save }"

It will work for nodes which have isec in its names.

But what if I would like to use query something like:

knife exec -E "nodes.find(:name => 'isec' AND :tags => 'tag_name' AND :role => 'role_name') { |node| node.run_list << "recipe[foo]"; n.save }"
You’re mixing ruby syntax and Solr query syntax here. If you want all the conditions to be AND’ed together, then you can pass a regular ruby Hash object, e.g., :name => “NAME_QUERY”, :tags => “TAG_QUERY”, …
If you instead need to do something more complicated, you can pass a string to nodes.find using the same syntax as you would with knife search.

For reference, the code that generates the query from the nodes.find call is here: https://github.com/opscode/chef/blob/005f75e158ce702bb28fd6199e9654b3d1115d3b/lib/chef/shell/model_wrapper.rb#L86-L92

HTH,

--
Daniel DeLeo

7

I always thought of minitest-handler and test-kitchen / serverspec as
being two orthogonal things:

  • minitest-handler for "smoke tests" that run from the inside
    everytime the system converges (e.g. check if processes are up and
    running, etc.)
  • kitchenci / serverspec for "integration tests" that still run from
    the inside but not necessarily after each run (i.e. on demand via
    kitchen test)

Actually, now that I'm writing it... the kitchenci tests actually run
inside the node, right?

What are the preferred frameworks for acceptance-level testing from the outside?

I had played with cucumber-nagios [0] once, but it doesn't seem
widespread in the chef community. Anyone having experience with
leibniz [1] btw?

Cheers,
Torben

[0] cucumber-nagios
[1] GitHub - Atalanta/leibniz: Leibniz is an integration testing framework for Chef

On Mon, Jan 5, 2015 at 4:48 PM, Joshua Timberman joshua@chef.io wrote:

Ohai!

On Sat, Jan 3, 2015 at 1:00 PM, David Petzel davidpetzel@gmail.com wrote:

TLDR - I'm interesting in knowing why folks are moving away from Minitest,
toward ServerSpec.

As Julian said, there's an element of not wanting to "contaminate" the
tests. However, there's a couple other reasons.

I've been noticing a lot of cookbooks recently have been migrating their
testings from minitest to ServerSpec. I've got nothing against ServerSpec,
but I'm curious why/where this trend is coming from.

Back in the before times, that is, before test kitchen and the notion of
bussers, the best (only?) way of reasonably testing nodes post convergence
was through the minitest chef handler. The early version of test kitchen
(pre 1.0) did run those, but the rewritten version, with the bussers made it
more flexible.

I think Serverspec is popular because Rspec is popular. It's not a value
judgment, just more widely used - Rspec is what CHEF uses for chef itself,
Ruby and Ruby on Rails programmers are familiar with Rspec. While minitest
is in the Ruby stdlib (and I like it more than Rspec for that reason), it's
simply not widespread. Also, Serverspec does an "outside in" kind of test
approach. I have more confidence that the tests are doing the right thing. I
can defer configurable options to attributes, and then set those attributes
on a per-suite basis.

Maybe my question would be better asked in the context of an example...
Lets assume I am consuming a community cookbook which sets up some
application that has an HTTP port. There is an attribute that controls what
that HTTP port is.
In ServerSpec land, I think you need to hard code the HTTP port to test?
In Minitest, I have access to the node object, so my test is not hard
coded to a port, but instead uses the value of the node attribute.

For the sake of the discussion, lets say I run that application on port 78
in dev, 79 in qa, and 80 in production. While using minitests I'm able to
assert the application is working properly, because the minitest test would
be reading the node attribute. However from what I've seen with ServerSpec,
I don't have that same ability to adapt the test to use the right port,
without manual intervention?

This is why I like, and use test kitchen. I can set up two suites.

suites:

  • name: default
    run_list:
    • recipe[webapp]
      attributes:
  • name: port78
    run_list:
    • recipe[webapp]
      attributes:
      www:
      port: 78

Then I can have tests for each suite, one that checks that my web app
responds on port 80 (presuming its the default), and responds on port 78. I
don't actually care about the different ports in different environments, I
want to test that the non-default port does the right thing. Adding more
suites that only change this attribute just tests that Chef works, which
we've already proved in the first case. In fact, if the port is an
attribute, I don't think the default suite is really really necessary.

Cheers,
Joshua

8

On Jan 5, 2015, at 4:17 PM, Torben Knerr mail@tknerr.de wrote:

I always thought of minitest-handler and test-kitchen / serverspec as
being two orthogonal things:

  • minitest-handler for "smoke tests" that run from the inside
    everytime the system converges (e.g. check if processes are up and
    running, etc.)

In it hasn't been mentioned that I've seen, check out https://github.com/opscode/chef-rfc/blob/master/rfc035-audit-mode.md for a description of the next evolution of this idea.

--Noah

9

Thanks Noah!

audit-mode sounds super useful!

On Tue, Jan 6, 2015 at 1:24 AM, Noah Kantrowitz noah@coderanger.net wrote:

On Jan 5, 2015, at 4:17 PM, Torben Knerr mail@tknerr.de wrote:

I always thought of minitest-handler and test-kitchen / serverspec as
being two orthogonal things:

  • minitest-handler for "smoke tests" that run from the inside
    everytime the system converges (e.g. check if processes are up and
    running, etc.)

In it hasn't been mentioned that I've seen, check out https://github.com/opscode/chef-rfc/blob/master/rfc035-audit-mode.md for a description of the next evolution of this idea.

--Noah

10

Thanks for all the feedback on this. I do have a better understanding now,
but it still feels like we “loosing something” in this transition. Maybe
I’m in the minority, but I like the fact that a cookbook author writes
minitests, and I can have those executed every single converge, not just
when someone decides to kick off a test-kitchen run.

On our team we are certainly making use of test-kitchen and bussers for
development and CI testing, but I can’t shake the feeling those are not
replacements for a validation test running on every converge.

In any case, thanks again for taking the time give me your perspectives.