# Chef Server vs. Not Guidance

**URL:** <https://discourse.chef.io/t/chef-server-vs-not-guidance/6829>\
**Category:** Chef Infra (archive)\
**Created:** [August 4, 2015, 7:24pm UTC](https://discourse.chef.io/t/chef-server-vs-not-guidance/6829 "2015-08-04T19:24:11Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kelly\_Brownsberger](https://avatars.discourse-cdn.com/v4/letter/k/cab0a1/32.png) [@Kelly\_Brownsberger](https://discourse.chef.io/u/Kelly_Brownsberger)\
**Post date:** [August 4, 2015, 7:24pm UTC](https://discourse.chef.io/t/chef-server-vs-not-guidance/6829/1 "2015-08-04T19:24:11Z")

</div>

My team has been developing recipes with Chef for several months now and  
we’ve made significant progress in automating our infrastructure, but  
we’re now getting to a point where we need to scale this up and make it  
a first class part of our CI pipeline where many teams are maintaining  
their own cookbooks sitting side by side the software code that’s  
running on this infrastructure. Up until now we’ve been focused  
entirely on recipe development using chef-client and Vagrant for  
spinning up nodes and running the cookbooks via Vagrant’s built in  
provisioners. As you’d expect, things are starting to become unwieldy  
with managing attributes and common shared cookbooks, etc. We’ve  
started looking at Berkshelf for managing dependencies and environments  
and roles for putting some structure to our attributes.

Here’s where it gets interesting… to put it bluntly, there are several  
within my team that aren’t bought into the idea of Chef Server. Our  
entire enterprise orbits around our version control system. And we  
already have a large investment in Artifactory which is a home for all  
of the technical artifacts for all of our products across several  
platforms and it works quite well as a general purpose distribution hub.  
As we’ve started playing with the chef concepts of environments and  
roles while running chef-client in local mode, I think I see the value  
in these abstractions, however… I’m starting to get the sense that  
using these outside the context of a Chef Server is going to be a very  
bumpy road.

Right now I’m able to get our cookbook using environments and roles and  
have chef-client find them by pointed to a directory on disk where these  
json files list under version control. This works, but the next use  
case we’re looking to support is running these outside the context of  
Vagrant and the current thought is using --recipe-url to point to a  
tarball that we published to artifactory from our CI builds in Jenkins  
(by way of Berks package). This use case would be for running these  
cookbooks are production machines that do not have connectivity to our  
version control system and will not be running in the context of  
Vagrant. As best I can tell, this will not work as I do not believe  
there is any way to include environments and roles json files in this  
tarbar and have chef-client recognize it via --recipe-url. Can someone  
confirm my suspicion?

Questions

1. are roles and environments even recommended at all? I’m finding  
folks out there that really feel strongly they are an anti-pattern  
[1][2], though I’m not sure I agree with them.
2. I keep hearing about how there are loads of people out there using  
chef without chef-server and doing so happily. are there folks using  
environments and roles?
3. similar to environments and roles, how are nodes managed in a chef-server-  
less world? I see that you can define these in json and even knife  
these into a chef server, but when I do a chef-client run in local  
mode, the nodes/ directory is written to with data from that run. In  
other words files in nodes/ doesn’t seem to be something that’s  
considered a “source” of anything, but rather a runtime output dir.
4. I’m torn with chef-server because it feels like something like this  
shouldn’t be necessary, but the deeper I get into these concepts  
more it feels like we’re going against the grain not fully embracing  
chef-server. Can anyone point me to some detailed information on  
what the full development production lifecycle would look liek with  
a chef-server oriented world? What about the same in a chef-server-  
less world?

Sorry for the long rambling email, but was hoping some kind soul could  
give me some guidance on how to proceed - every direction we look right  
now feel less than ideal.

[1][http://dougireton.com/blog/2013/02/16/chef-cookbook-anti-patterns/](http://dougireton.com/blog/2013/02/16/chef-cookbook-anti-patterns/)  
[2][http://devopsanywhere.blogspot.com/2012/11/how-to-write-reusable-chef-cookbooks.html](http://devopsanywhere.blogspot.com/2012/11/how-to-write-reusable-chef-cookbooks.html)

---

<div class="post-metadata">

**Author:** ![kallistec](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/kallistec/32/23_2.png) [@kallistec](https://discourse.chef.io/u/kallistec)\
**Post date:** [August 5, 2015, 4:01am UTC](https://discourse.chef.io/t/chef-server-vs-not-guidance/6829/2 "2015-08-05T04:01:39Z")

</div>

On Tuesday, August 4, 2015 at 12:24 PM, Kelly Brownsberger wrote:

> My team has been developing recipes with Chef for several months now and we've made significant progress in automating our infrastructure, but we're now getting to a point where we need to scale this up and make it a first class part of our CI pipeline where many teams are maintaining their own cookbooks sitting side by side the software code that's running on this infrastructure. Up until now we've been focused entirely on recipe development using chef-client and Vagrant for spinning up nodes and running the cookbooks via Vagrant's built in provisioners. As you'd expect, things are starting to become unwieldy with managing attributes and common shared cookbooks, etc. We've started looking at Berkshelf for managing dependencies and environments and roles for putting some structure to our attributes.
> 
> Here's where it gets interesting... to put it bluntly, there are several within my team that aren't bought into the idea of Chef Server. Our entire enterprise orbits around our version control system. And we already have a large investment in Artifactory which is a home for all of the technical artifacts for all of our products across several platforms and it works quite well as a general purpose distribution hub. As we've started playing with the chef concepts of environments and roles while running chef-client in local mode, I think I see the value in these abstractions, however.... I'm starting to get the sense that using these outside the context of a Chef Server is going to be a very bumpy road.

If you really want to avoid using a Chef Server, you can just run local mode in production, using a local mode formatted repo as your distribution artifact.

> Right now I'm able to get our cookbook using environments and roles and have chef-client find them by pointed to a directory on disk where these json files list under version control. This works, but the next use case we're looking to support is running these outside the context of Vagrant and the current thought is using --recipe-url to point to a tarball that we published to artifactory from our CI builds in Jenkins (by way of Berks package). This use case would be for running these cookbooks are production machines that do not have connectivity to our version control system and will not be running in the context of Vagrant. As best I can tell, this will not work as I do not believe there is any way to include environments and roles json files in this tarbar and have chef-client recognize it via --recipe-url. Can someone confirm my suspicion?

Not 100% sure, but I think you’d basically need to make your own script to replace berks package. But this shouldn’t be hard.

> Questions
> 
> 1. are roles and environments even recommended at all? I'm finding folks out there that really feel strongly they are an anti-pattern [1][2], though I'm not sure I agree with them.

The big issue most folks have with roles is that the Chef Server doesn’t have any way to version them, so a change to a role would go live on all systems that use it at once. If you solve for this in the way you do your distribution, they’re fine.

Environments mostly just exist to provide version pinning for cookbooks, but they do have attributes you could use if you want. But you could also just use roles for that.

> 1. I keep hearing about how there are loads of people out there using chef without chef-server and doing so happily. are there folks using environments and roles?
> 2. similar to environments and roles, how are nodes managed in a chef-server-less world? I see that you can define these in json and even knife these into a chef server, but when I do a chef-client run in local mode, the nodes/ directory is written to with data from that run. In other words files in nodes/ doesn't seem to be something that's considered a "source" of anything, but rather a runtime output dir.

The node data there will be loaded the next time Chef runs, which gives you persistence for ‘normal’ level attributes. It’s also possible to create other nodes there so that search works (if you use a 3rd party cookbook that hardcodes search, for example).

> 1. I'm torn with chef-server because it feels like something like this shouldn't be necessary, but the deeper I get into these concepts more it feels like we're going against the grain not fully embracing chef-server. Can anyone point me to some detailed information on what the full development production lifecycle would look liek with a chef-server oriented world? What about the same in a chef-server-less world?

Since you already have Ci in the mix, you’d have Ci push your cookbooks to the server. The tricky bit is making sure each set of nodes gets the correct cookbook versions and you do not overwrite any existing cookbooks. There’s several ways to do this; you can either put the responsibility on developers to update cookbook versions when submitting changes, or you can have Ci automatically bump the version number. In either case, you can then generate an environment JSON file and then upload that to pin the cookbooks.

Alternatively, you could look at policyfiles, which solve the issues with roles being unversioned and also with cookbook version conflicts. You can read more about them here: [https://github.com/chef/chef-dk/blob/master/POLICYFILE\_README.md](https://github.com/chef/chef-dk/blob/master/POLICYFILE_README.md)

> Sorry for the long rambling email, but was hoping some kind soul could give me some guidance on how to proceed - every direction we look right now feel less than ideal.
> 
> [1] [http://dougireton.com/blog/2013/02/16/chef-cookbook-anti-patterns/](http://dougireton.com/blog/2013/02/16/chef-cookbook-anti-patterns/)  
> [2][devopsanywhere: How to Write Reusable Chef Cookbooks, Gangnam Style](http://devopsanywhere.blogspot.com/2012/11/how-to-write-reusable-chef-cookbooks.html)

HTH, I would like to have gone into more depth but it’s getting to be late here. Feel free to follow up with more questions.

--  
Daniel DeLeo

---

<div class="post-metadata">

**Author:** ![tknerr](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/tknerr/32/21_2.png) [@tknerr](https://discourse.chef.io/u/tknerr)\
**Post date:** [August 5, 2015, 6:15am UTC](https://discourse.chef.io/t/chef-server-vs-not-guidance/6829/3 "2015-08-05T06:15:11Z")

</div>

Hi Kelly,

on 2) + 3):

If you are happily using Vagrant for local development / CI already, you  
could also use it for describing your production setups in a multi-vm  
Vagrantfile and use the specific providers (eg vsphere, aws, rackspace  
etc). This works quite well as long as your infrastructure is static and  
you can describe it in terms of individual nodes, i.e. no autoscaling etc.

If you want to go a more cheffish route the chef-provisioning might be  
interesting for you.

I believe with either approach you can still use roles and environments.  
But not sure whether chef-provisioning works in solo / zero mode.

HTH, Torben  
Am 04.08.2015 21:24 schrieb "Kelly Brownsberger" [me@kellybrownsberger.com](mailto:me@kellybrownsberger.com):

> My team has been developing recipes with Chef for several months now and  
> we've made significant progress in automating our infrastructure, but we're  
> now getting to a point where we need to scale this up and make it a first  
> class part of our CI pipeline where many teams are maintaining their own  
> cookbooks sitting side by side the software code that's running on this  
> infrastructure. Up until now we've been focused entirely on recipe  
> development using chef-client and Vagrant for spinning up nodes and running  
> the cookbooks via Vagrant's built in provisioners. As you'd expect, things  
> are starting to become unwieldy with managing attributes and common shared  
> cookbooks, etc. We've started looking at Berkshelf for managing  
> dependencies and environments and roles for putting some structure to our  
> attributes.
> 
> Here's where it gets interesting... to put it bluntly, there are several  
> within my team that aren't bought into the idea of Chef Server. Our entire  
> enterprise orbits around our version control system. And we already have a  
> large investment in Artifactory which is a home for all of the technical  
> artifacts for all of our products across several platforms and it works  
> quite well as a general purpose distribution hub. As we've started playing  
> with the chef concepts of environments and roles while running chef-client  
> in local mode, I think I see the value in these abstractions, however....  
> I'm starting to get the sense that using these outside the context of a  
> Chef Server is going to be a very bumpy road.
> 
> Right now I'm able to get our cookbook using environments and roles and  
> have chef-client find them by pointed to a directory on disk where these  
> json files list under version control. This works, but the next use case  
> we're looking to support is running these outside the context of Vagrant  
> and the current thought is using --recipe-url to point to a tarball that we  
> published to artifactory from our CI builds in Jenkins (by way of Berks  
> package). This use case would be for running these cookbooks are  
> production machines that do not have connectivity to our version control  
> system and will not be running in the context of Vagrant. As best I can  
> tell, this will not work as I do not believe there is any way to include  
> environments and roles json files in this tarbar and have chef-client  
> recognize it via --recipe-url. Can someone confirm my suspicion?
> 
> Questions
> 
> 1. are roles and environments even recommended at all? I'm finding folks  
> out there that really feel strongly they are an anti-pattern [1][2], though  
> I'm not sure I agree with them.
> 2. I keep hearing about how there are loads of people out there using chef  
> without chef-server and doing so happily. are there folks using  
> environments and roles?
> 3. similar to environments and roles, how are nodes managed in a  
> chef-server-less world? I see that you can define these in json and even  
> knife these into a chef server, but when I do a chef-client run in local  
> mode, the nodes/ directory is written to with data from that run. In other  
> words files in nodes/ doesn't seem to be something that's considered a  
> "source" of anything, but rather a runtime output dir.
> 4. I'm torn with chef-server because it feels like something like this  
> shouldn't be necessary, but the deeper I get into these concepts more it  
> feels like we're going against the grain not fully embracing chef-server.  
> Can anyone point me to some detailed information on what the full  
> development production lifecycle would look liek with a chef-server  
> oriented world? What about the same in a chef-server-less world?
> 
> Sorry for the long rambling email, but was hoping some kind soul could  
> give me some guidance on how to proceed - every direction we look right now  
> feel less than ideal.
> 
> [1] [http://dougireton.com/blog/2013/02/16/chef-cookbook-anti-patterns/](http://dougireton.com/blog/2013/02/16/chef-cookbook-anti-patterns/)  
> [2]  
> [devopsanywhere: How to Write Reusable Chef Cookbooks, Gangnam Style](http://devopsanywhere.blogspot.com/2012/11/how-to-write-reusable-chef-cookbooks.html)

---

<div class="post-metadata">

**Author:** ![Kelly\_Brownsberger](https://avatars.discourse-cdn.com/v4/letter/k/cab0a1/32.png) [@Kelly\_Brownsberger](https://discourse.chef.io/u/Kelly_Brownsberger)\
**Post date:** [August 5, 2015, 11:54am UTC](https://discourse.chef.io/t/chef-server-vs-not-guidance/6829/4 "2015-08-05T11:54:42Z")

</div>

Thanks Torben and Daniel - this was very helpful. I appreciate you  
taking the time to respond.

Your comment about generating the environment json is interest... I've  
been considering doing that. I was surprised to find that all cookbooks  
and their versions have to be specified at the enviornment json level in  
order to have versioning. But after thinking more about it, I guess  
that makes sense... however it presents a maintenance problem. We  
already have a pretty tight CI pipeline with strict versioning  
methodologies in place. It doesn't make sense for us to maintain this  
list in an environment json file when our Jenkins server already know  
this information.

I'm still a bit confused on Nodes though. Here is a use case we're  
faced with now: We're trying to automate our CI infrastructure - i.e.  
the build out of a Jenkins Master and various flavors of Jenkins Agent  
machines. Part of the master setup is configuring it to manage one or  
more agent machines. We would very much like to have a single source of  
truth for this information and not maintain it in multiple files. At a  
conceptual level it seems to need to have some way from the recipes  
responsible for building out the master machine be able to query for all  
chef nodes that are of a certain role (or some means for knowing which  
machine have or will be provisioned as an agent). How would one  
approach this? In our current chef-server-less world it's not clear to  
me how you would define these node in json files and then make them  
available for querying at chef-client runtime. Does embracing chef-  
server make this problem any easier?

On Wed, Aug 5, 2015, at 02:15 AM, Torben Knerr wrote:

> Hi Kelly,

> on 2) + 3):

> If you are happily using Vagrant for local development / CI already,  
> you could also use it for describing your production setups in a multi-  
> vm Vagrantfile and use the specific providers (eg vsphere, aws,  
> rackspace etc). This works quite well as long as your infrastructure  
> is static and you can describe it in terms of individual nodes, i.e.  
> no autoscaling etc.

> If you want to go a more cheffish route the chef-provisioning might be  
> interesting for you.

> I believe with either approach you can still use roles and  
> environments. But not sure whether chef-provisioning works in solo /  
> zero mode.

> HTH, Torben

> Am 04.08.2015 21:24 schrieb "Kelly Brownsberger"  
> [me@kellybrownsberger.com](mailto:me@kellybrownsberger.com):
> 
> > \_\_  
> > My team has been developing recipes with Chef for several months now  
> > and we've made significant progress in automating our infrastructure,  
> > but we're now getting to a point where we need to scale this up and  
> > make it a first class part of our CI pipeline where many teams are  
> > maintaining their own cookbooks sitting side by side the software  
> > code that's running on this infrastructure. Up until now we've been  
> > focused entirely on recipe development using chef-client and Vagrant  
> > for spinning up nodes and running the cookbooks via Vagrant's built  
> > in provisioners. As you'd expect, things are starting to become  
> > unwieldy with managing attributes and common shared cookbooks, etc.  
> > We've started looking at Berkshelf for managing dependencies and  
> > environments and roles for putting some structure to our attributes.
> > 
> > Here's where it gets interesting... to put it bluntly, there are  
> > several within my team that aren't bought into the idea of Chef  
> > Server. Our entire enterprise orbits around our version control  
> > system. And we already have a large investment in Artifactory which  
> > is a home for all of the technical artifacts for all of our products  
> > across several platforms and it works quite well as a general purpose  
> > distribution hub. As we've started playing with the chef concepts of  
> > environments and roles while running chef-client in local mode, I  
> > think I see the value in these abstractions, however.... I'm starting  
> > to get the sense that using these outside the context of a Chef  
> > Server is going to be a very bumpy road.
> > 
> > Right now I'm able to get our cookbook using environments and roles  
> > and have chef-client find them by pointed to a directory on disk  
> > where these json files list under version control. This works, but  
> > the next use case we're looking to support is running these outside  
> > the context of Vagrant and the current thought is using --recipe-url  
> > to point to a tarball that we published to artifactory from our CI  
> > builds in Jenkins (by way of Berks package). This use case would be  
> > for running these cookbooks are production machines that do not have  
> > connectivity to our version control system and will not be running in  
> > the context of Vagrant. As best I can tell, this will not work as I  
> > do not believe there is any way to include environments and roles  
> > json files in this tarbar and have chef-client recognize it via --recipe-  
> > url. Can someone confirm my suspicion?
> > 
> > Questions
> > 
> > 1. are roles and environments even recommended at all? I'm finding  
> > folks out there that really feel strongly they are an anti-pattern  
> > [1][2], though I'm not sure I agree with them.
> > 2. I keep hearing about how there are loads of people out there using  
> > chef without chef-server and doing so happily. are there folks  
> > using environments and roles?
> > 3. similar to environments and roles, how are nodes managed in a chef-server-  
> > less world? I see that you can define these in json and even  
> > knife these into a chef server, but when I do a chef-client run in  
> > local mode, the nodes/ directory is written to with data from that  
> > run. In other words files in nodes/ doesn't seem to be something  
> > that's considered a "source" of anything, but rather a runtime  
> > output dir.
> > 4. I'm torn with chef-server because it feels like something like  
> > this shouldn't be necessary, but the deeper I get into these  
> > concepts more it feels like we're going against the grain not  
> > fully embracing chef-server. Can anyone point me to some detailed  
> > information on what the full development production lifecycle  
> > would look liek with a chef-server oriented world? What about the  
> > same in a chef-server-less world?
> > 
> > Sorry for the long rambling email, but was hoping some kind soul  
> > could give me some guidance on how to proceed - every direction we  
> > look right now feel less than ideal.
> > 
> > [1][http://dougireton.com/blog/2013/02/16/chef-cookbook-anti-patterns/](http://dougireton.com/blog/2013/02/16/chef-cookbook-anti-patterns/)  
> > [2][devopsanywhere: How to Write Reusable Chef Cookbooks, Gangnam Style](http://devopsanywhere.blogspot.com/2012/11/how-to-write-reusable-chef-cookbooks.html)

---

<div class="post-metadata">

**Author:** ![andi\_abes](https://avatars.discourse-cdn.com/v4/letter/a/dc4da7/32.png) [@andi\_abes](https://discourse.chef.io/u/andi_abes)\
**Post date:** [August 5, 2015, 12:18pm UTC](https://discourse.chef.io/t/chef-server-vs-not-guidance/6829/5 "2015-08-05T12:18:59Z")

</div>

When we evaluated switching away from Chef server (haven't yet), we hit the  
same question - how do you do search?  
Our answer was to avoid search, since it leads to eventual converge  
behaviour - your search results on the Jenkins server will only return full  
results returned some time after the full set of agents have converged....

The folks at airbnb have written up their approach [1].  
As it relates ti search, it boils down to - if you have to use it, store  
the relevant inventory information somewhere that's built to so just that.

[1] [http://nerds.airbnb.com/making-breakfast-chef-airbnb/](http://nerds.airbnb.com/making-breakfast-chef-airbnb/)  
On Aug 5, 2015 7:55 AM, "Kelly Brownsberger" [me@kellybrownsberger.com](mailto:me@kellybrownsberger.com)  
wrote:

> Thanks Torben and Daniel - this was very helpful. I appreciate you taking  
> the time to respond.
> 
> Your comment about generating the environment json is interest... I've  
> been considering doing that. I was surprised to find that all cookbooks  
> and their versions have to be specified at the enviornment json level in  
> order to have versioning. But after thinking more about it, I guess that  
> makes sense... however it presents a maintenance problem. We already have  
> a pretty tight CI pipeline with strict versioning methodologies in place.  
> It doesn't make sense for us to maintain this list in an environment json  
> file when our Jenkins server already know this information.
> 
> I'm still a bit confused on Nodes though. Here is a use case we're faced  
> with now: We're trying to automate our CI infrastructure - i.e. the build  
> out of a Jenkins Master and various flavors of Jenkins Agent machines.  
> Part of the master setup is configuring it to manage one or more agent  
> machines. We would very much like to have a single source of truth for  
> this information and not maintain it in multiple files. At a conceptual  
> level it seems to need to have some way from the recipes responsible for  
> building out the master machine be able to query for all chef nodes that  
> are of a certain role (or some means for knowing which machine have or will  
> be provisioned as an agent). How would one approach this? In our current  
> chef-server-less world it's not clear to me how you would define these node  
> in json files and then make them available for querying at chef-client  
> runtime. Does embracing chef-server make this problem any easier?
> 
> On Wed, Aug 5, 2015, at 02:15 AM, Torben Knerr wrote:
> 
> Hi Kelly,
> 
> on 2) + 3):
> 
> If you are happily using Vagrant for local development / CI already, you  
> could also use it for describing your production setups in a multi-vm  
> Vagrantfile and use the specific providers (eg vsphere, aws, rackspace  
> etc). This works quite well as long as your infrastructure is static and  
> you can describe it in terms of individual nodes, i.e. no autoscaling etc.
> 
> If you want to go a more cheffish route the chef-provisioning might be  
> interesting for you.
> 
> I believe with either approach you can still use roles and environments.  
> But not sure whether chef-provisioning works in solo / zero mode.
> 
> HTH, Torben  
> Am 04.08.2015 21:24 schrieb "Kelly Brownsberger" \<[me@kellybrownsberger.com](mailto:me@kellybrownsberger.com)
> 
> > :
> 
> My team has been developing recipes with Chef for several months now and  
> we've made significant progress in automating our infrastructure, but we're  
> now getting to a point where we need to scale this up and make it a first  
> class part of our CI pipeline where many teams are maintaining their own  
> cookbooks sitting side by side the software code that's running on this  
> infrastructure. Up until now we've been focused entirely on recipe  
> development using chef-client and Vagrant for spinning up nodes and running  
> the cookbooks via Vagrant's built in provisioners. As you'd expect, things  
> are starting to become unwieldy with managing attributes and common shared  
> cookbooks, etc. We've started looking at Berkshelf for managing  
> dependencies and environments and roles for putting some structure to our  
> attributes.
> 
> Here's where it gets interesting... to put it bluntly, there are several  
> within my team that aren't bought into the idea of Chef Server. Our entire  
> enterprise orbits around our version control system. And we already have a  
> large investment in Artifactory which is a home for all of the technical  
> artifacts for all of our products across several platforms and it works  
> quite well as a general purpose distribution hub. As we've started playing  
> with the chef concepts of environments and roles while running chef-client  
> in local mode, I think I see the value in these abstractions, however....  
> I'm starting to get the sense that using these outside the context of a  
> Chef Server is going to be a very bumpy road.
> 
> Right now I'm able to get our cookbook using environments and roles and  
> have chef-client find them by pointed to a directory on disk where these  
> json files list under version control. This works, but the next use case  
> we're looking to support is running these outside the context of Vagrant  
> and the current thought is using --recipe-url to point to a tarball that we  
> published to artifactory from our CI builds in Jenkins (by way of Berks  
> package). This use case would be for running these cookbooks are  
> production machines that do not have connectivity to our version control  
> system and will not be running in the context of Vagrant. As best I can  
> tell, this will not work as I do not believe there is any way to include  
> environments and roles json files in this tarbar and have chef-client  
> recognize it via --recipe-url. Can someone confirm my suspicion?
> 
> Questions
> 
> 1. are roles and environments even recommended at all? I'm finding folks  
> out there that really feel strongly they are an anti-pattern [1][2], though  
> I'm not sure I agree with them.
> 2. I keep hearing about how there are loads of people out there using chef  
> without chef-server and doing so happily. are there folks using  
> environments and roles?
> 3. similar to environments and roles, how are nodes managed in a  
> chef-server-less world? I see that you can define these in json and even  
> knife these into a chef server, but when I do a chef-client run in local  
> mode, the nodes/ directory is written to with data from that run. In other  
> words files in nodes/ doesn't seem to be something that's considered a  
> "source" of anything, but rather a runtime output dir.
> 4. I'm torn with chef-server because it feels like something like this  
> shouldn't be necessary, but the deeper I get into these concepts more it  
> feels like we're going against the grain not fully embracing chef-server.  
> Can anyone point me to some detailed information on what the full  
> development production lifecycle would look liek with a chef-server  
> oriented world? What about the same in a chef-server-less world?
> 
> Sorry for the long rambling email, but was hoping some kind soul could  
> give me some guidance on how to proceed - every direction we look right now  
> feel less than ideal.
> 
> [1] [http://dougireton.com/blog/2013/02/16/chef-cookbook-anti-patterns/](http://dougireton.com/blog/2013/02/16/chef-cookbook-anti-patterns/)  
> [2]  
> [devopsanywhere: How to Write Reusable Chef Cookbooks, Gangnam Style](http://devopsanywhere.blogspot.com/2012/11/how-to-write-reusable-chef-cookbooks.html)

---

<div class="post-metadata">

**Author:** ![Marco\_Betti](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/marco_betti/32/105_2.png) [@Marco\_Betti](https://discourse.chef.io/u/Marco_Betti)\
**Post date:** [August 5, 2015, 12:46pm UTC](https://discourse.chef.io/t/chef-server-vs-not-guidance/6829/6 "2015-08-05T12:46:39Z")

</div>

Hi,  
In our company we use chef with local mode, without chef server, but using  
berkshelf and environment cookbook pattern.  
We have a single repo per cookbook approach and our "top level" cookbook,  
as Torben calls it, is the environment cookbook. We have an environment  
cookbook per each product we use to provide services with (we are an  
in-house IT service provider for public administration). Within each  
environment cookbook we version everything, environments, databags, nodes  
and so on. We use tags on nodes and search based on tags to provide  
auto-riconfiguration if we _manually_ (because no chef server exists) add a  
node, for example for jboss nodes balanced by apache. Finally we developed  
some custom script to build each environment cookbook as payload of a  
sibgle stell script that can then be pushed/run on any node of that product  
infrastructure, we call this custom pattern we invented "chef zero package  
pattern" 🙂 because that package is built by our CI server.  
It would not be suitable for large infrastructures, I wouldn't recommend  
for real web-scale IT, but it is not our use case, we have many little  
product and having to manually manage nodes within environment cookbook is  
not a big problem.  
We never wrote o published anything about that because it does not seem to  
me the right chef approach, but chef is an awesome framework and allows you  
to do such an horrible thing also 😃  
Sorry for this very quick answer if you think it is useful and want some  
other info, feel free to ask.

Cheers!  
Marco  
Il giorno 04/ago/2015 21:24, "Kelly Brownsberger" [me@kellybrownsberger.com](mailto:me@kellybrownsberger.com)  
ha scritto:

> My team has been developing recipes with Chef for several months now and  
> we've made significant progress in automating our infrastructure, but we're  
> now getting to a point where we need to scale this up and make it a first  
> class part of our CI pipeline where many teams are maintaining their own  
> cookbooks sitting side by side the software code that's running on this  
> infrastructure. Up until now we've been focused entirely on recipe  
> development using chef-client and Vagrant for spinning up nodes and running  
> the cookbooks via Vagrant's built in provisioners. As you'd expect, things  
> are starting to become unwieldy with managing attributes and common shared  
> cookbooks, etc. We've started looking at Berkshelf for managing  
> dependencies and environments and roles for putting some structure to our  
> attributes.
> 
> Here's where it gets interesting... to put it bluntly, there are several  
> within my team that aren't bought into the idea of Chef Server. Our entire  
> enterprise orbits around our version control system. And we already have a  
> large investment in Artifactory which is a home for all of the technical  
> artifacts for all of our products across several platforms and it works  
> quite well as a general purpose distribution hub. As we've started playing  
> with the chef concepts of environments and roles while running chef-client  
> in local mode, I think I see the value in these abstractions, however....  
> I'm starting to get the sense that using these outside the context of a  
> Chef Server is going to be a very bumpy road.
> 
> Right now I'm able to get our cookbook using environments and roles and  
> have chef-client find them by pointed to a directory on disk where these  
> json files list under version control. This works, but the next use case  
> we're looking to support is running these outside the context of Vagrant  
> and the current thought is using --recipe-url to point to a tarball that we  
> published to artifactory from our CI builds in Jenkins (by way of Berks  
> package). This use case would be for running these cookbooks are  
> production machines that do not have connectivity to our version control  
> system and will not be running in the context of Vagrant. As best I can  
> tell, this will not work as I do not believe there is any way to include  
> environments and roles json files in this tarbar and have chef-client  
> recognize it via --recipe-url. Can someone confirm my suspicion?
> 
> Questions
> 
> 1. are roles and environments even recommended at all? I'm finding folks  
> out there that really feel strongly they are an anti-pattern [1][2], though  
> I'm not sure I agree with them.
> 2. I keep hearing about how there are loads of people out there using chef  
> without chef-server and doing so happily. are there folks using  
> environments and roles?
> 3. similar to environments and roles, how are nodes managed in a  
> chef-server-less world? I see that you can define these in json and even  
> knife these into a chef server, but when I do a chef-client run in local  
> mode, the nodes/ directory is written to with data from that run. In other  
> words files in nodes/ doesn't seem to be something that's considered a  
> "source" of anything, but rather a runtime output dir.
> 4. I'm torn with chef-server because it feels like something like this  
> shouldn't be necessary, but the deeper I get into these concepts more it  
> feels like we're going against the grain not fully embracing chef-server.  
> Can anyone point me to some detailed information on what the full  
> development production lifecycle would look liek with a chef-server  
> oriented world? What about the same in a chef-server-less world?
> 
> Sorry for the long rambling email, but was hoping some kind soul could  
> give me some guidance on how to proceed - every direction we look right now  
> feel less than ideal.
> 
> [1] [http://dougireton.com/blog/2013/02/16/chef-cookbook-anti-patterns/](http://dougireton.com/blog/2013/02/16/chef-cookbook-anti-patterns/)  
> [2]  
> [devopsanywhere: How to Write Reusable Chef Cookbooks, Gangnam Style](http://devopsanywhere.blogspot.com/2012/11/how-to-write-reusable-chef-cookbooks.html)

---

<div class="post-metadata">

**Author:** ![kallistec](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/kallistec/32/23_2.png) [@kallistec](https://discourse.chef.io/u/kallistec)\
**Post date:** [August 5, 2015, 3:47pm UTC](https://discourse.chef.io/t/chef-server-vs-not-guidance/6829/7 "2015-08-05T15:47:17Z")

</div>

On Wednesday, August 5, 2015 at 4:54 AM, Kelly Brownsberger wrote:

> Thanks Torben and Daniel - this was very helpful. I appreciate you taking the time to respond.
> 
> Your comment about generating the environment json is interest... I've been considering doing that. I was surprised to find that all cookbooks and their versions have to be specified at the enviornment json level in order to have versioning. But after thinking more about it, I guess that makes sense... however it presents a maintenance problem. We already have a pretty tight CI pipeline with strict versioning methodologies in place. It doesn't make sense for us to maintain this list in an environment json file when our Jenkins server already know this information.

You wouldn’t really be maintaining it, so much as producing a build artifact based on the information in the Jenkins server. You’d just have a script that takes the info from Jenkins and converts it to the format that the Chef Server understands, and you’d have Jenkins run this script as part of your build. As long as you only ever move data from Jenkins to the Chef Server, you will just have one source of truth and a well defined process for syncing that data. It does make using environment attributes more difficult, but that can be worked around in a number of ways depending on your needs/priorities. Just think of it as being like the step in your current workflow where you run `berks package` — hypothetically, you could unpack the tarball, and make some edits, but you just don’t do that.

> I'm still a bit confused on Nodes though. Here is a use case we're faced with now: We're trying to automate our CI infrastructure - i.e. the build out of a Jenkins Master and various flavors of Jenkins Agent machines. Part of the master setup is configuring it to manage one or more agent machines. We would very much like to have a single source of truth for this information and not maintain it in multiple files. At a conceptual level it seems to need to have some way from the recipes responsible for building out the master machine be able to query for all chef nodes that are of a certain role (or some means for knowing which machine have or will be provisioned as an agent). How would one approach this? In our current chef-server-less world it's not clear to me how you would define these node in json files and then make them available for querying at chef-client runtime. Does embracing chef-server make this problem any easier?  
> This is what the search feature of the Chef Server was created to do (well, one use case, anyway).

If you want to stay serverless, then you would maintain this data in some other system that you can query programmatically, the simplest being a JSON/YAML file you maintain by hand, with more effort you could write your own web service to do it. Alternatively, you could look into using a service discovery tool (e.g., etcd) and then integrating it with chef-client.

I know of one implementation which is built on top of a web service (which provides a bunch of other features) where they create minimal node JSON for every system in the cluster so that search mostly works for basic use cases, though it doesn’t have the full range of attributes available to search so it’s not 100% the same as the chef search you’d get with a Chef Server. And again, they already have a node database to start from.

As a final note, your workflow sounds like it could be an excellent fit for Policyfiles. They’re kinda new, but getting to “reasonably complete” very quickly. This is a draft of a blog post that I’m planning to put on the Chef blog once ChefDK 0.7 comes out, which should give you a better idea of how they work: [Policyfile Guided Tour · GitHub](https://gist.github.com/danielsdeleo/9112b6c7932452bdb7e8) Relevant to this conversation is that all the version locking type stuff you’d have to script yourself would be done automatically by the `chef` tool during the development process, so you’d just confirm that it works in Ci and then publish it.

--  
Daniel DeLeo

---

<div class="post-metadata">

**Author:** ![Chip](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/chip/32/15_2.png) [@Chip](https://discourse.chef.io/u/Chip)\
**Post date:** [August 5, 2015, 6:58pm UTC](https://discourse.chef.io/t/chef-server-vs-not-guidance/6829/8 "2015-08-05T18:58:52Z")

</div>

Usual disclaimer: I work for CHEF. YMMV. etc.

The Chef Server is a publishing platform. It contains APIs for publishing  
and consuming particular data structures, and tools for securing and  
managing that published data. It also contains an artifact server for  
cookbooks.

Humans and clients all publish data. Humans tend to publish policies and  
code, clients tend to publish data about the nodes in the infrastructure.  
All actors tend to consume the data set in most models.

The most useful thing about this platform, to me, is that if you set the  
clients to run on a scheduled basis, then every node in your infrastructure  
becomes an autonomous actor. Each of those autonomous actors becomes the  
SOA for its own configuration, because the client manages its own node  
object, and humans just modify policy by publishing run\_lists on individual  
nodes or via roles, and publish updated cookbooks. Essentially the  
infrastructure starts to manage itself, and reacts to change that flows in  
via publishes to the Chef Server.

This enables some powerful, highly scalable workflows that just aren’t  
possible when you’re manually firing off ad-hoc config runs on nodes. But that  
said: Some people are never comfortable with this model, because it removes  
the central observer/controller from the equation. No longer do I weird REAL  
ULTIMATE POWER! No longer does my workstation reach out, poke hosts, and  
tell them to “Dance, monkey, dance!” Instead I publish policy & code, and  
trust that the nodes in my infrastructure will configure themselves  
accordingly, within the allowed timeframe. As a sysadmin who had to wrap  
his head around Chef, it took me a long time to grok and embrace this  
model, because it was so very different from the way that I was used to  
interacting with the world. At first it felt like giving up control, but as  
my experience grew I realized I was gaining a lot more than I ever gave up.

(I’m intentionally leaving aside the "roles/environments good or bad?"  
conversation here, as I don’t see it as critical to the Chef Server  
workflow, and you can go either way really.)

What a Chef Server is:

- A publishing platform that gives you a defined API for distributing  
policy and code to chef clients.
- An easy way to distribute change through your distributed system, when  
coupled with daemonized chef-client runs.
- A searchable point-in-time database of information about your  
infrastructure.

What a Chef Server isn’t:

- A centralized controller for Chef. Out of the box, it’s just a searchable  
publishing platform for point-in-time data about your infrastructure, with  
no history.
- A source code repository: It is an artifact repository. Source code lives  
in SCM.
- A canonical source of truth: Your truth lives in source control, and on  
the configuration of the nodes themselves. Humans report that truth to the  
Chef Server from SCM, and nodes report that truth to the Chef Server from  
Chef Client.

Hope that helps you with your decision!

–Charles