My Wishlist: Better Chef-Solo Support

Chef Infra (archive)
1

Ohai Chefs,

are you cooking solo?

I’m trying to figure out a sane way of working with Chef-solo, thereby came
across lots of useful projects aiming to make the Chef-solo experience
better, e.g.:

They add features which are currently missing from Chef solo, e.g.

  • data_bags (also encrypted data_bags)
  • search (within node and data_bags)
  • environments support

There is also a proof of concept
https://gist.github.com/tmatilai/4559333which integrates knife-solo
with the knife bootstrap command by passing a
--solo option, which lets you transparently use knife-ec2,
knife-rackspace, etc. (name any plugin which uses knife bootstrap) with
Chef solo as well.

It seems to me that there is much demand for better Chef-solo integration
in the community, but unfortunately this is not there yet and you have to
cherry-pick specific patches from the approaches above if you want a better
Chef solo experience.

I’d really love to see these approaches being integrated into core Chef and
have a single and consistent way of working with it.

Is there anything that prevents this from happening or is it just that
nobody had done it yet?

Cheers, Torben

2

+1

I use vagrant with chef-solo for testing and I cannot say I would highly
recommend it.

Vagrant + chef-solo-search + Berkshelf + isn't too bad, but still, if I
finde a 'node.save' I then need to hack up a recipe.

On Wed, Feb 20, 2013 at 5:29 AM, Torben Knerr ukio@gmx.de wrote:

Ohai Chefs,

are you cooking solo?

I'm trying to figure out a sane way of working with Chef-solo, thereby
came across lots of useful projects aiming to make the Chef-solo experience
better, e.g.:

They add features which are currently missing from Chef solo, e.g.

  • data_bags (also encrypted data_bags)
  • search (within node and data_bags)
  • environments support

There is also a proof of concepthttps://gist.github.com/tmatilai/4559333which integrates knife-solo with the knife bootstrap command by passing a
--solo option, which lets you transparently use knife-ec2,
knife-rackspace, etc. (name any plugin which uses knife bootstrap) with
Chef solo as well.

It seems to me that there is much demand for better Chef-solo integration
in the community, but unfortunately this is not there yet and you have to
cherry-pick specific patches from the approaches above if you want a better
Chef solo experience.

I'd really love to see these approaches being integrated into core Chef
and have a single and consistent way of working with it.

Is there anything that prevents this from happening or is it just that
nobody had done it yet?

Cheers, Torben

3

I also use vagrant with chef-solo for testing, and while it's not perfect I would definitely say that it is far better than testing with chef server.

That said, I would definitely love to see some improvements to chef-solo, such as being able to bootstrap a server with it, support for search, etc.

--
Larry Wright

On Wednesday, February 20, 2013 at 8:27 AM, Brian Demers wrote:

+1

I use vagrant with chef-solo for testing and I cannot say I would highly recommend it.

Vagrant + chef-solo-search + Berkshelf + isn't too bad, but still, if I finde a 'node.save' I then need to hack up a recipe.

On Wed, Feb 20, 2013 at 5:29 AM, Torben Knerr <ukio@gmx.de (mailto:ukio@gmx.de)> wrote:

Ohai Chefs,

are you cooking solo?

I'm trying to figure out a sane way of working with Chef-solo, thereby came across lots of useful projects aiming to make the Chef-solo experience better, e.g.:

They add features which are currently missing from Chef solo, e.g.

  • data_bags (also encrypted data_bags)
  • search (within node and data_bags)
  • environments support

There is also a proof of concept (POC to integrate knife-solo to `knife bootstrap` · GitHub) which integrates knife-solo with the knife bootstrap command by passing a --solo option, which lets you transparently use knife-ec2, knife-rackspace, etc. (name any plugin which uses knife bootstrap) with Chef solo as well.

It seems to me that there is much demand for better Chef-solo integration in the community, but unfortunately this is not there yet and you have to cherry-pick specific patches from the approaches above if you want a better Chef solo experience.

I'd really love to see these approaches being integrated into core Chef and have a single and consistent way of working with it.

Is there anything that prevents this from happening or is it just that nobody had done it yet?

Cheers, Torben

4

Maybe knife-solo can help some? My goal for the project is to help get
better parity with chef-client. Granted there's a fair bit of code between
here and there.

-Mat

On Wed, Feb 20, 2013 at 9:27 AM, Brian Demers brian.demers@gmail.comwrote:

+1

I use vagrant with chef-solo for testing and I cannot say I would highly
recommend it.

Vagrant + chef-solo-search + Berkshelf + isn't too bad, but still, if I
finde a 'node.save' I then need to hack up a recipe.

On Wed, Feb 20, 2013 at 5:29 AM, Torben Knerr ukio@gmx.de wrote:

Ohai Chefs,

are you cooking solo?

I'm trying to figure out a sane way of working with Chef-solo, thereby
came across lots of useful projects aiming to make the Chef-solo experience
better, e.g.:

They add features which are currently missing from Chef solo, e.g.

  • data_bags (also encrypted data_bags)
  • search (within node and data_bags)
  • environments support

There is also a proof of concepthttps://gist.github.com/tmatilai/4559333which integrates knife-solo with the knife bootstrap command by passing a
--solo option, which lets you transparently use knife-ec2,
knife-rackspace, etc. (name any plugin which uses knife bootstrap) with
Chef solo as well.

It seems to me that there is much demand for better Chef-solo integration
in the community, but unfortunately this is not there yet and you have to
cherry-pick specific patches from the approaches above if you want a better
Chef solo experience.

I'd really love to see these approaches being integrated into core Chef
and have a single and consistent way of working with it.

Is there anything that prevents this from happening or is it just that
nobody had done it yet?

Cheers, Torben

5

I've always been curious about people using chef-solo in cluster
environments...

Why not just use a chef-server instead of moving mountains to to
replicate the functionality it provides?

I'm honestly curious about the use cases.

-s

On Wed, Feb 20, 2013 at 5:29 AM, Torben Knerr ukio@gmx.de wrote:

Ohai Chefs,

are you cooking solo?

I'm trying to figure out a sane way of working with Chef-solo, thereby came
across lots of useful projects aiming to make the Chef-solo experience
better, e.g.:

They add features which are currently missing from Chef solo, e.g.

  • data_bags (also encrypted data_bags)
  • search (within node and data_bags)
  • environments support

There is also a proof of concept which integrates knife-solo with the knife bootstrap command by passing a --solo option, which lets you
transparently use knife-ec2, knife-rackspace, etc. (name any plugin which
uses knife bootstrap) with Chef solo as well.

It seems to me that there is much demand for better Chef-solo integration in
the community, but unfortunately this is not there yet and you have to
cherry-pick specific patches from the approaches above if you want a better
Chef solo experience.

I'd really love to see these approaches being integrated into core Chef and
have a single and consistent way of working with it.

Is there anything that prevents this from happening or is it just that
nobody had done it yet?

Cheers, Torben

6

My actual use cases where I prefer Chef solo:

  • one-time setup of infrastructure (e.g. setting up project infrastructure
    for our development teams as a one-time service - no service included)
  • test environments for testing single cookbooks / recipes
  • simple environments, maybe clustered with a few servers (but no
    elasticity!)

In these cases I prefer chef solo because it's less complex both in terms
of workflow and infrastructure.

What I'd ideally like to have is the choice of doing so with as much
chef-client parity as possible. Right now I'm using

  • knife-solo for deploying servers with chef-solo where I only have plain
    ssh access
  • mccloud for deploying ec2 instances with chef solo
  • vagrant for local testing with chef solo

Sadly, each of these provide different levels of parity with chef-client,
e.g. knife-solo adds search via
edelight/chef-solo-searchhttps://github.com/edelight/chef-solo-search,
but this is not supported by the other approaches. I guess all of them
support data bags. Not sure if anyone of them already supports environments
under chef-solo https://github.com/opscode/chef/pull/359/commits...

If chef solo already provided that level of parity, it would be a) much
easier to switch to chef server and vice versa, b) easier to write / reuse
recipes that just work with chef solo and c) existing tools leveraging
knife bootstrap (e.g. knife-ec2) would just work as well.

Plus, If I could always use knife bootstrap rather than different
commandlines for knife-solo, mccloud and vagrant this would greatly
harmonize my workflow.

On Wed, Feb 20, 2013 at 6:11 PM, Sean OMeara someara@gmail.com wrote:

I've always been curious about people using chef-solo in cluster
environments...

Why not just use a chef-server instead of moving mountains to to
replicate the functionality it provides?

I'm honestly curious about the use cases.

-s

On Wed, Feb 20, 2013 at 5:29 AM, Torben Knerr ukio@gmx.de wrote:

Ohai Chefs,

are you cooking solo?

I'm trying to figure out a sane way of working with Chef-solo, thereby
came
across lots of useful projects aiming to make the Chef-solo experience
better, e.g.:

They add features which are currently missing from Chef solo, e.g.

  • data_bags (also encrypted data_bags)
  • search (within node and data_bags)
  • environments support

There is also a proof of concept which integrates knife-solo with the
knife bootstrap command by passing a --solo option, which lets you
transparently use knife-ec2, knife-rackspace, etc. (name any plugin which
uses knife bootstrap) with Chef solo as well.

It seems to me that there is much demand for better Chef-solo
integration in
the community, but unfortunately this is not there yet and you have to
cherry-pick specific patches from the approaches above if you want a
better
Chef solo experience.

I'd really love to see these approaches being integrated into core Chef
and
have a single and consistent way of working with it.

Is there anything that prevents this from happening or is it just that
nobody had done it yet?

Cheers, Torben

7

On Wed, Feb 20, 2013 at 1:02 PM, Torben Knerr ukio@gmx.de wrote:

Sadly, each of these provide different levels of parity with chef-client,
e.g. knife-solo adds search via edelight/chef-solo-searchhttps://github.com/edelight/chef-solo-search,
but this is not supported by the other approaches. I guess all of them
support data bags. Not sure if anyone of them already supports environments
under chef-solo https://github.com/opscode/chef/pull/359/commits...

I had been planning some work thinking that chef-solo-search would work in
a vagrant environment by just adding the cookbook - does it not work? I
haven't tested yet, but if someone else has already been down this route it
might save me from going down a dead end.

--
Elliot Murphy
Pat Deegan PhD & Associates, LLC

8

While we're generally using Chef Server for the rest of the infrastructure,
we do use Solo to build ephemeral Jenkins node images for EC2, which I can
then snapshot into an AMI. Example: developers want "phantomjs" on a
Jenkins box to do JS unit testing, so I add it to Chef and kick a solo run
on a bare node.

It does drive me crazy when cookbook authors require Chef Server
functionality to accomplish basic tasks. For instance, we've forked &
modified the Jenkins cookbook significantly because of its assumption that
you can query for the node with the Jenkins master role to retrieve the SSH
key.

  • Julian

On Wed, Feb 20, 2013 at 12:11 PM, Sean OMeara someara@gmail.com wrote:

I've always been curious about people using chef-solo in cluster
environments...

Why not just use a chef-server instead of moving mountains to to
replicate the functionality it provides?

I'm honestly curious about the use cases.

-s

--
[ Julian C. Dunn jdunn@aquezada.com * Sorry, I'm ]
[ WWW: Julian Dunn's Blog - Commentary on media, technology, and everything in between. * only Web 1.0 ]
[ gopher://sdf.org/1/users/keymaker/ * compliant! ]
[ PGP: 91B3 7A9D 683C 7C16 715F 442C 6065 D533 FDC2 05B9 ]

9

On Feb 20, 2013, at 1:47 PM, Julian C. Dunn jdunn@aquezada.com wrote:

It does drive me crazy when cookbook authors require Chef Server functionality to accomplish basic tasks. For instance, we've forked & modified the Jenkins cookbook significantly because of its assumption that you can query for the node with the Jenkins master role to retrieve the SSH key.

Opscode is taking over maintenance of this cookbook and I will be making sure things operate as expected in a solo-based environment…mainly because we do development in Vagrant/Berkshelf and testing in Test Kitchen. The code will now live at: GitHub - sous-chefs/jenkins: Development repository for the jenkins cookbook

Until the new 1.0.0 version of the cookbook is released I believe you can get around your issue [1] by explicitly setting the following attributes:

node.set['jenkins']['server']['pubkey']

--
Seth Chisamore
Software Development Engineer, Opscode, Inc.
IRC, Skype, Twitter, Github: schisamo

[1] - https://github.com/heavywater/chef-jenkins/blob/develop/recipes/node_ssh.rb#L24

10

On Feb 20, 2013, at 1:32 PM, Elliot Murphy elliot.murphy@patdeegan.com wrote:

On Wed, Feb 20, 2013 at 1:02 PM, Torben Knerr ukio@gmx.de wrote:
Sadly, each of these provide different levels of parity with chef-client, e.g. knife-solo adds search via edelight/chef-solo-search, but this is not supported by the other approaches. I guess all of them support data bags. Not sure if anyone of them already supports environments under chef-solo...

I had been planning some work thinking that chef-solo-search would work in a vagrant environment by just adding the cookbook - does it not work? I haven't tested yet, but if someone else has already been down this route it might save me from going down a dead end.

Have you taken a look at Fletcher Nichol's searchef [0] (and it's associated cookbook [1])? It allows you to stub out the search request at the HTTP request level:

search for nodes

stub_search(:node, "roles:web_node").to_return([
node_stub("web1.example.com"),
node_stub("web2.example.com")
])

search in data bag

stub_search(:users, 'groups:admin').to_return([
{
"id" => "adam",
"comment" => "Adam Administrator",
"groups" => [
"admin"
],
"ssh_keys" => ,
"shell" => "/bin/bash"
}
])

stub_search(:users, 'groups:admin').to_return([
data_bag_item("users", "adam")
])

We have been using this a bit internally when running our cookbooks in 'development' mode, i.e. in Vagrant.

--
Seth Chisamore
Software Development Engineer, Opscode, Inc.
IRC, Skype, Twitter, Github: schisamo

[0] - GitHub - fnichol/searchef: Stub Chef Search!
[1] - GitHub - fnichol/chef-searchef: Chef cookbook to use Searchef (Stub Chef Search!).

11

On Wed, Feb 20, 2013 at 12:11 PM, Sean OMeara someara@gmail.com wrote:

I've always been curious about people using chef-solo in cluster
environments...

Why not just use a chef-server instead of moving mountains to to
replicate the functionality it provides?

I'm honestly curious about the use cases.

Because up until erchef, Chef Server was a bit of a pain :wink: I say this
as someone who had no problems setting it up. Yes there were various
solo-based cookbooks but the number of moving parts was hard to
swallow and manage.

Additionally, Chef server now becomes another dependency to your
environment and must be treated as such from an HA perspective. Not
everyone was/is able to use hosted chef.

-s

On Wed, Feb 20, 2013 at 5:29 AM, Torben Knerr ukio@gmx.de wrote:

Ohai Chefs,

are you cooking solo?

I'm trying to figure out a sane way of working with Chef-solo, thereby came
across lots of useful projects aiming to make the Chef-solo experience
better, e.g.:

They add features which are currently missing from Chef solo, e.g.

  • data_bags (also encrypted data_bags)
  • search (within node and data_bags)
  • environments support

There is also a proof of concept which integrates knife-solo with the knife bootstrap command by passing a --solo option, which lets you
transparently use knife-ec2, knife-rackspace, etc. (name any plugin which
uses knife bootstrap) with Chef solo as well.

It seems to me that there is much demand for better Chef-solo integration in
the community, but unfortunately this is not there yet and you have to
cherry-pick specific patches from the approaches above if you want a better
Chef solo experience.

I'd really love to see these approaches being integrated into core Chef and
have a single and consistent way of working with it.

Is there anything that prevents this from happening or is it just that
nobody had done it yet?

Cheers, Torben

12

On Wednesday, 20 February, 2013 at 1:03 PM, Seth Chisamore wrote:

On Feb 20, 2013, at 1:32 PM, Elliot Murphy <elliot.murphy@patdeegan.com (mailto:elliot.murphy@patdeegan.com)> wrote:

On Wed, Feb 20, 2013 at 1:02 PM, Torben Knerr <ukio@gmx.de (mailto:ukio@gmx.de)> wrote:
Sadly, each of these provide different levels of parity with chef-client, e.g. knife-solo adds search via edelight/chef-solo-search, but this is not supported by the other approaches. I guess all of them support data bags. Not sure if anyone of them already supports environments under chef-solo...

I had been planning some work thinking that chef-solo-search would work in a vagrant environment by just adding the cookbook - does it not work? I haven't tested yet, but if someone else has already been down this route it might save me from going down a dead end.

Have you taken a look at Fletcher Nichol's searchef [0] (and it's associated cookbook [1])? It allows you to stub out the search request at the HTTP request level:

search for nodes

stub_search(:node, "roles:web_node").to_return([
node_stub("web1.example.com (http://web1.example.com)"),
node_stub("web2.example.com (http://web2.example.com)")
])

search in data bag

stub_search(:users, 'groups:admin').to_return([
{
"id" => "adam",
"comment" => "Adam Administrator",
"groups" => [
"admin"
],
"ssh_keys" => ,
"shell" => "/bin/bash"
}
])

stub_search(:users, 'groups:admin').to_return([
data_bag_item("users", "adam")
])

We have been using this a bit internally when running our cookbooks in 'development' mode, i.e. in Vagrant.

--
Seth Chisamore
Software Development Engineer, Opscode, Inc.
IRC, Skype, Twitter, Github: schisamo

[0] - GitHub - fnichol/searchef: Stub Chef Search!
[1] - GitHub - fnichol/chef-searchef: Chef cookbook to use Searchef (Stub Chef Search!).

Thanks Seth!

Searchef is still reasonably young (started last Tuesday) but was written to help with testing cookbooks that depend heavily on search, but are being exercised in solo mode. Think stubbing (dumb responses) rather than emulating search. Now that it's in the wild, I'll push the cookbook to the community site :slight_smile:

Fletcher.

p.s. While stubbing out search responses might be clever, please also consider adding solo mode detection with Chef::Config[:solo] if the cookbook is meant for public consumption. Sometimes looking in a node attribute for your data is a great alternative for solo. After all, don't forget FC003 [0] :slight_smile:

[0] - http://acrmp.github.com/foodcritic/#FC003

13

This stubbed search thing is sick - I had a similar idea as the
extension point to bolting in more advanced discovery mechanisms
(intercept search/databag lookup, translate to Super Discovery Thing)

Very cool -- will play with this more. Thanks Fletcher!

--AJ

On 21 February 2013 10:44, Fletcher Nichol fnichol@nichol.ca wrote:

On Wednesday, 20 February, 2013 at 1:03 PM, Seth Chisamore wrote:

On Feb 20, 2013, at 1:32 PM, Elliot Murphy <elliot.murphy@patdeegan.com (mailto:elliot.murphy@patdeegan.com)> wrote:

On Wed, Feb 20, 2013 at 1:02 PM, Torben Knerr <ukio@gmx.de (mailto:ukio@gmx.de)> wrote:
Sadly, each of these provide different levels of parity with chef-client, e.g. knife-solo adds search via edelight/chef-solo-search, but this is not supported by the other approaches. I guess all of them support data bags. Not sure if anyone of them already supports environments under chef-solo...

I had been planning some work thinking that chef-solo-search would work in a vagrant environment by just adding the cookbook - does it not work? I haven't tested yet, but if someone else has already been down this route it might save me from going down a dead end.

Have you taken a look at Fletcher Nichol's searchef [0] (and it's associated cookbook [1])? It allows you to stub out the search request at the HTTP request level:

search for nodes

stub_search(:node, "roles:web_node").to_return([
node_stub("web1.example.com (http://web1.example.com)"),
node_stub("web2.example.com (http://web2.example.com)")
])

search in data bag

stub_search(:users, 'groups:admin').to_return([
{
"id" => "adam",
"comment" => "Adam Administrator",
"groups" => [
"admin"
],
"ssh_keys" => ,
"shell" => "/bin/bash"
}
])

stub_search(:users, 'groups:admin').to_return([
data_bag_item("users", "adam")
])

We have been using this a bit internally when running our cookbooks in 'development' mode, i.e. in Vagrant.

--
Seth Chisamore
Software Development Engineer, Opscode, Inc.
IRC, Skype, Twitter, Github: schisamo

[0] - GitHub - fnichol/searchef: Stub Chef Search!
[1] - GitHub - fnichol/chef-searchef: Chef cookbook to use Searchef (Stub Chef Search!).

Thanks Seth!

Searchef is still reasonably young (started last Tuesday) but was written to help with testing cookbooks that depend heavily on search, but are being exercised in solo mode. Think stubbing (dumb responses) rather than emulating search. Now that it's in the wild, I'll push the cookbook to the community site :slight_smile:

Fletcher.

p.s. While stubbing out search responses might be clever, please also consider adding solo mode detection with Chef::Config[:solo] if the cookbook is meant for public consumption. Sometimes looking in a node attribute for your data is a great alternative for solo. After all, don't forget FC003 [0] :slight_smile:

[0] - http://acrmp.github.com/foodcritic/#FC003