Application builds/ci involving Chef

Chef Infra (archive)
1 
Is anyone here doing application builds (either yourself or for
devs) that involve any of the following:




-Local (workstation) testing of a developed app with Chef housing
configuration files


-Application bundling with ant, maven, groovy, scripts, whatever
that <b>also includes</b> parallel check-ins to Chef code?


-CI testing of an application that builds an artifact but also has
configurations in Chef?




My current client is moving away from keeping any application
information in Chef at all but the problem is still an interesting
one to me.  If I tell a dev team to put their config files into chef
templates, how do I help them combine build triggers for application
code as well as configuration code? Part of the trick I'm learning
is to get them to think of Chef as part of their code base at all
and not just provisioning scripts maintained by their release
engineer. Also, how do you ensure that dependent config/code changes
are built together and not deployed in such a way as to break the
app? I'm not terribly concerned about production at the moment, my
main interest is around CI testing, although ultimately that would
be the goal.




If you are doing anything like this, I would love to hear about.
Currently my team has Jenkins-driven workflow CI around our key Chef
repositories and some common application components but it's not
something we've rolled out to dev teams and it's not triggered by
java/app code check-ins, but rather by us checking in Chef code.
It's decoupled at the moment.




TL;DR I'm looking for ways to make it easy for dev teams to regard
Chef as a natural part of their development cycle as opposed to a
weird third arm.






Sascha

–

  Sascha Bates <i><b>/</b></i><i><b></b></i><i><b><a class="moz-txt-link-abbreviated" href="mailto:sbates@brattyredhead.com">sbates@brattyredhead.com</a></b></i><i><b><i><b>/</b></i> blog.brattyredhead.com </b></i><i><b><i><b>/</b></i>
      @sascha_d

2

Is anyone here doing application builds (either yourself or for devs) that

involve any of the following:

-Local (workstation) testing of a developed app with Chef housing
configuration files

For local development with chef managing configs, you could try using
vagrant + chef

-Application bundling with ant, maven, groovy, scripts, whatever that also
includes parallel check-ins to Chef code?
-CI testing of an application that builds an artifact but also has
configurations in Chef?

I have used Jenkins + chef. You could also add capistrano in the mix for
deployment.

Thanks
Gourav

3

On Jan 26, 2013, at 7:59 PM, Gourav Shah gs@initcron.org wrote:

Is anyone here doing application builds (either yourself or for devs) that involve any of the following:

-Local (workstation) testing of a developed app with Chef housing configuration files

For local development with chef managing configs, you could try using vagrant + chef

Soon...

-Application bundling with ant, maven, groovy, scripts, whatever that also includes parallel check-ins to Chef code?
-CI testing of an application that builds an artifact but also has configurations in Chef?
I have used Jenkins + chef. You could also add capistrano in the mix for deployment.

I am building out our Jenkins testing of cookbooks as we speak. We use berkshelf to install to vendor cookbooks with dependencies, then run RSpec / Chefspec to test them and Foodcritic to ensure they meet our standards.

Our script looks like this (assuming a job named "cookbook-[COOKBOOK NAME]", i.e. "cookbook-users")

COOKBOOK=${JOB_NAME#cookbook-}

cd $WORKSPACE
set -x
bundle install --without development --deployment
bundle exec berks install --path vendor/cookbooks
bundle exec rspec --format CI::Reporter::RSpec
bundle exec foodcritic $WORKSPACE/vendor/cookbooks

By including ci-reporter in my gem list I can get rspec to generate JUnit style reports and use those to define health of the jobs.

Thanks
Gourav

--
Jeffrey Hulten
Principal Consultant at Automated Labs
jeffh@automatedlabs.com 206-853-5216
Skype: jeffhulten

4

Thanks. I'm in an environment where we
use jenkins and maven for a lot of this at the moment. There's
also some Vagrant/vbox in the mix as well as Eclipse/STS. I'm not
looking for pointers so much as being interested in what others
have found to be successful.

  One of the problems I see is that Chef code and Application code
  are decoupled: different repos, different teams managing
  collections of cookbooks, diverse environments. Dev teams don't
  think of Chef as part of the code base and I believe that this has
  led to some dysfunctional attitudes and behaviors. At the moment
  I'm noodling on methodology for introducing a tool like Chef that
  decouples configuration code from application logic.  




  So I guess I'm maybe not looking for deployment solutions so much
  as wondering what people are doing to get their chef and app code
  working together and how it's working out for them.

Sascha Bates
| sascha.bates@gmail.com | 612 850 0444 | sascha.bates@skype |
sascha_bates@yahoo |
On 1/26/13 9:59 PM, Gourav Shah wrote:

Is anyone here doing application builds (either yourself or for devs) that involve any of the following: 
        -Local (workstation) testing of a developed app with Chef
        housing configuration files

      For local development with chef managing configs, you could
      try using vagrant + chef  <blockquote class="gmail_quote" style="margin:0 0 0
      .8ex;border-left:1px #ccc solid;padding-left:1ex">-Application bundling
        with ant, maven, groovy, scripts, whatever that <b>also
          includes</b> parallel check-ins to Chef code?


        -CI testing of an application that builds an artifact but
        also has configurations in Chef?
I have used Jenkins + chef. You could also add capistrano in the mix for deployment.

ThanksGourav

5

I have the same problems on my team, and I think for me the best thing I
can aim to do over the next few months is to get my app deployment scripts
to function in chef solo on a developer's machine. Maybe even get the local
installer team to use it instead of the complicated combination of ant
scripts and installanywhere.

If they can see how deploying their own app is a one command process, as
opposed to the 50+ commands they have in their wiki now, then they may use
it.
And if they use it, they are much more likely to adopt it as part of their
code.

Unfortunately its all in ruby, and we are a java shop...
Thanks. I'm in an environment where we use jenkins and maven for a lot of
this at the moment. There's also some Vagrant/vbox in the mix as well as
Eclipse/STS. I'm not looking for pointers so much as being interested in
what others have found to be successful.

One of the problems I see is that Chef code and Application code are
decoupled: different repos, different teams managing collections of
cookbooks, diverse environments. Dev teams don't think of Chef as part of
the code base and I believe that this has led to some dysfunctional
attitudes and behaviors. At the moment I'm noodling on methodology for
introducing a tool like Chef that decouples configuration code from
application logic.

So I guess I'm maybe not looking for deployment solutions so much as
wondering what people are doing to get their chef and app code working
together and how it's working out for them.

Sascha Bates | sascha.bates@gmail.com | 612 850 0444 | sascha.bates@skype |
sascha_bates@yahoo |
On 1/26/13 9:59 PM, Gourav Shah wrote:

Is anyone here doing application builds (either yourself or for devs) that

involve any of the following:

-Local (workstation) testing of a developed app with Chef housing
configuration files

For local development with chef managing configs, you could try using
vagrant + chef

-Application bundling with ant, maven, groovy, scripts, whatever that also
includes parallel check-ins to Chef code?
-CI testing of an application that builds an artifact but also has
configurations in Chef?

I have used Jenkins + chef. You could also add capistrano in the mix for
deployment.

Thanks
Gourav

6

One idea that EJ (your cohort on the podcast had) that I thought was
brilliantly simple is to have the applications ship ERB templates for any
conf files by default and let chef do its template magic with a local file
instead of one in the repo.

The disconnect is always going to exist unless your dev team is willing to
work with the chef repo. Frankly it's pretty unfair to ask that out id the
gate and keeping the two in sync is painful.

The first motivation I had for the (sadly languishing) Noah was this exact
use case.

On Jan 27, 2013 1:14 AM, "Sascha Bates" sascha.bates@gmail.com wrote:

Thanks. I'm in an environment where we use jenkins and maven for a lot of
this at the moment. There's also some Vagrant/vbox in the mix as well as
Eclipse/STS. I'm not looking for pointers so much as being interested in
what others have found to be successful.

One of the problems I see is that Chef code and Application code are
decoupled: different repos, different teams managing collections of
cookbooks, diverse environments. Dev teams don't think of Chef as part of
the code base and I believe that this has led to some dysfunctional
attitudes and behaviors. At the moment I'm noodling on methodology for
introducing a tool like Chef that decouples configuration code from
application logic.

So I guess I'm maybe not looking for deployment solutions so much as
wondering what people are doing to get their chef and app code working
together and how it's working out for them.

Sascha Bates | sascha.bates@gmail.com | 612 850 0444 | sascha.bates@skype| sascha_bates@yahoo|
On 1/26/13 9:59 PM, Gourav Shah wrote:

Is anyone here doing application builds (either yourself or for devs)
that involve any of the following:

-Local (workstation) testing of a developed app with Chef housing
configuration files

For local development with chef managing configs, you could try using
vagrant + chef

-Application bundling with ant, maven, groovy, scripts, whatever that
also includes parallel check-ins to Chef code?
-CI testing of an application that builds an artifact but also has
configurations in Chef?

I have used Jenkins + chef. You could also add capistrano in the mix
for deployment.

Thanks
Gourav

7

On Sun, Jan 27, 2013 at 1:14 AM, Sascha Bates sascha.bates@gmail.comwrote:

Thanks. I'm in an environment where we use jenkins and maven for a lot
of this at the moment. There's also some Vagrant/vbox in the mix as well as
Eclipse/STS. I'm not looking for pointers so much as being interested in
what others have found to be successful.

One of the problems I see is that Chef code and Application code are
decoupled: different repos, different teams managing collections of
cookbooks, diverse environments. Dev teams don't think of Chef as part of
the code base and I believe that this has led to some dysfunctional
attitudes and behaviors. At the moment I'm noodling on methodology for
introducing a tool like Chef that decouples configuration code from
application logic.

This is actually a pain point for us as well, and really the only way we
were able to solve it was thru policy, in short we just made sure Ops was
notified of any changes to configuration files managed by Chef.

So I guess I'm maybe not looking for deployment solutions so much as
wondering what people are doing to get their chef and app code working
together and how it's working out for them.

Right now we are not testing the cookbooks in isolation, but we do use
them as part of our functional tests suite for the core products. In these
case Jenkins spins up a slave for the actual test, and then that mgmnt
slave then spins up 2 vanilla EC2 instances ( Driver and Catcher ). We
then have a small fog script which does the "bootstrapping" on the
instances. It installs Chef via omnibus, installs our yum repo, and then
drops the first-run.json. It then triggers the chef run on the node. The
Jenkins slave server then controls the "driver" instance which then
exercises about 500 scenarios on the actual runtime instance. So far this
is working well for us, but there certainly is room for improvement :slight_smile:

-John

Sascha Bates | sascha.bates@gmail.com | 612 850 0444 | sascha.bates@skype| sascha_bates@yahoo|
On 1/26/13 9:59 PM, Gourav Shah wrote:

Is anyone here doing application builds (either yourself or for devs)

that involve any of the following:

-Local (workstation) testing of a developed app with Chef housing
configuration files

For local development with chef managing configs, you could try using
vagrant + chef

-Application bundling with ant, maven, groovy, scripts, whatever that also
includes parallel check-ins to Chef code?
-CI testing of an application that builds an artifact but also has
configurations in Chef?

I have used Jenkins + chef. You could also add capistrano in the mix for
deployment.

Thanks
Gourav

8

Interesting that you say that about
Noah. My current client is looking at a service management model
to deal with configuration files, eliminating Chef entirely from
the mix. I have also discussed with a dev team shipping in config
files from artifact repository and we did it with overrideable
attributes, allowing someone else to use either a default template
in the library cookbook or supply their own in another cookbook or
drop a template on the server from elsewhere. I guess my
preference would be to eventually have the templates parametrized
so that anyone could use the same template.

  But then the question arises, how do you deal with changing data? 
  For this we have been using a combination of attributes and data
  bags. Chef solo would be a viable option for workstation testing,
  but they'd still have to check these things in and have build
  scripts that dealt with getting them into the Chef server/org.  We
  have a lot of this now and it's just not connected.




  Maybe this is something that would actually work if you could get
  in front of the development and build process and hand a dev team
  a working CI process that involved Chef.  Lately I only get
  involved when it's too late to do anything besides panic.

Sascha Bates
| sascha.bates@gmail.com | 612 850 0444 | sascha.bates@skype |
sascha_bates@yahoo |
On 1/27/13 9:14 AM, John E. Vincent (lusis) wrote:

One idea that EJ (your cohort on the podcast had) that I thought was brilliantly simple is to have the applications ship ERB templates for any conf files by default and let chef do its template magic with a local file instead of one in the repo.

The disconnect is always going to exist unless your dev team is willing to work with the chef repo. Frankly it's pretty unfair to ask that out id the gate and keeping the two in sync is painful.

The first motivation I had for the (sadly languishing) Noah was this exact use case. 

    On Jan 27, 2013 1:14 AM, "Sascha Bates" &lt;<a moz-do-not-send="true" href="mailto:sascha.bates@gmail.com">sascha.bates@gmail.com</a>&gt;
    wrote:


    &gt;


    &gt; Thanks. I'm in an environment where we use jenkins and
    maven for a lot of this at the moment. There's also some
    Vagrant/vbox in the mix as well as Eclipse/STS. I'm not looking
    for pointers so much as being interested in what others have
    found to be successful. 


    &gt;


    &gt; One of the problems I see is that Chef code and Application
    code are decoupled: different repos, different teams managing
    collections of cookbooks, diverse environments. Dev teams don't
    think of Chef as part of the code base and I believe that this
    has led to some dysfunctional attitudes and behaviors. At the
    moment I'm noodling on methodology for introducing a tool like
    Chef that decouples configuration code from application logic. 
    


    &gt;


    &gt; So I guess I'm maybe not looking for deployment solutions
    so much as wondering what people are doing to get their chef and
    app code working together and how it's working out for them.


    &gt;


    &gt;


    &gt;


    &gt;


    &gt; Sascha Bates | <a moz-do-not-send="true" href="mailto:sascha.bates@gmail.com">sascha.bates@gmail.com</a>
    | 612 850 0444 | sascha.bates@skype | sascha_bates@yahoo |


    &gt; On 1/26/13 9:59 PM, Gourav Shah wrote:


    &gt;&gt;


    &gt;&gt;


    &gt;&gt;


    &gt;&gt;&gt; Is anyone here doing application builds (either
    yourself or for devs) that involve any of the following:


    &gt;&gt;&gt;


    &gt;&gt;&gt; -Local (workstation) testing of a developed app
    with Chef housing configuration files


    &gt;&gt;


    &gt;&gt;


    &gt;&gt; For local development with chef managing configs, you
    could try using vagrant + chef 


    &gt;&gt;  


    &gt;&gt;&gt;


    &gt;&gt;&gt; -Application bundling with ant, maven, groovy,
    scripts, whatever that also includes parallel check-ins to Chef
    code?


    &gt;&gt;&gt; -CI testing of an application that builds an
    artifact but also has configurations in Chef?


    &gt;&gt;


    &gt;&gt; I have used Jenkins + chef.  You could also add
    capistrano in the mix for deployment. 


    &gt;&gt;


    &gt;&gt; Thanks


    &gt;&gt; Gourav


    &gt;


    &gt;

9

Even though my team is doing both: developing application features as well as Chef cookbooks we see the disconnect. We've two separate repositories and developers do not work on Chef configured Vagrant boxes all the time. They only use Vagrant when they're changing cookbooks.
Additionally we've split configuration values between our Rails app (in environments) and Chef.
The only place where app and infrastructure mingle is our staging environment so far.

We're planning to setup a CI using Chef to create the VMs to run the tests. But we're not there yet :frowning:

Interesting that you say that about Noah. My current client is looking at a service management model to deal with configuration files, eliminating Chef entirely from the mix. I have also discussed with a dev team shipping in config files from artifact repository and we did it with overrideable attributes, allowing someone else to use either a default template in the library cookbook or supply their own in another cookbook or drop a template on the server from elsewhere. I guess my preference would be to eventually have the templates parametrized so that anyone could use the same template.

But then the question arises, how do you deal with changing data? For this we have been using a combination of attributes and data bags. Chef solo would be a viable option for workstation testing, but they'd still have to check these things in and have build scripts that dealt with getting them into the Chef server/org. We have a lot of this now and it's just not connected.

Maybe this is something that would actually work if you could get in front of the development and build process and hand a dev team a working CI process that involved Chef. Lately I only get involved when it's too late to do anything besides panic.

Sascha Bates | sascha.bates@gmail.com | 612 850 0444 | sascha.bates@skype | sascha_bates@yahoo |
On 1/27/13 9:14 AM, John E. Vincent (lusis) wrote:

One idea that EJ (your cohort on the podcast had) that I thought was brilliantly simple is to have the applications ship ERB templates for any conf files by default and let chef do its template magic with a local file instead of one in the repo.

The disconnect is always going to exist unless your dev team is willing to work with the chef repo. Frankly it's pretty unfair to ask that out id the gate and keeping the two in sync is painful.

The first motivation I had for the (sadly languishing) Noah was this exact use case.

On Jan 27, 2013 1:14 AM, "Sascha Bates" sascha.bates@gmail.com wrote:

Thanks. I'm in an environment where we use jenkins and maven for a lot of this at the moment. There's also some Vagrant/vbox in the mix as well as Eclipse/STS. I'm not looking for pointers so much as being interested in what others have found to be successful.

One of the problems I see is that Chef code and Application code are decoupled: different repos, different teams managing collections of cookbooks, diverse environments. Dev teams don't think of Chef as part of the code base and I believe that this has led to some dysfunctional attitudes and behaviors. At the moment I'm noodling on methodology for introducing a tool like Chef that decouples configuration code from application logic.

So I guess I'm maybe not looking for deployment solutions so much as wondering what people are doing to get their chef and app code working together and how it's working out for them.

Sascha Bates | sascha.bates@gmail.com | 612 850 0444 | sascha.bates@skype | sascha_bates@yahoo |
On 1/26/13 9:59 PM, Gourav Shah wrote:

Is anyone here doing application builds (either yourself or for devs) that involve any of the following:

-Local (workstation) testing of a developed app with Chef housing configuration files

For local development with chef managing configs, you could try using vagrant + chef

-Application bundling with ant, maven, groovy, scripts, whatever that also includes parallel check-ins to Chef code?
-CI testing of an application that builds an artifact but also has configurations in Chef?

I have used Jenkins + chef. You could also add capistrano in the mix for deployment.

Thanks
Gourav

10

Hi,

Our newer applications are all JEEish and thus have relatively low
variability. We developed a “configuration template” in json for each
application. As most of our cookbooks use an attribute driven design (See
[1]) we actually could just merge parts of the json template into the node
data. We just needed to make sure that the data conforms with the cookbooks
attribute_driven schema. ie We could generate a json blob something like

json ={
‘config’ =>
{
‘max_memory’ => ‘${CM:MAX_MEMORY|1548}’,

'max_perm_size' => 192,
'port' => 80,
'admin_port' => 8085,
'max_stack_size' => 128,
'username' => '${CM:ADMIN_USERNAME}',
'password' => '${CM:ADMIN_PASS}'

},
‘deployables’ =>
{
‘somapp’ =>
{
‘url’ => ‘http://repo.example.com/somapp-{CM:VERSION}.war', 'context_root' => '/somapp' } }, 'extra_libraries' => { 'mydatabasedriver' => 'http://repo.example.com/mydatabasedriver-1.2.3.jar' }, 'jdbc_connection_pools' => { 'SomeappSQL' => { 'datasourceclassname' => 'net.sourceforge.jtds.jdbcx.JtdsDataSource', 'restype' => 'javax.sql.DataSource', 'isconnectvalidatereq' => 'true', 'validationmethod' => 'auto-commit', 'ping' => 'true', 'description' => 'SomeappSQL Connection Pool', 'properties' => { 'Instance' => 'Instance1', 'ServerName' => 'db.example.com', 'User' => '{CM:DB_WRITER_USER}’,
‘Password’ => ‘${CM:DB_WRITER_PASSWORD}’

       'PortNumber' => '1433',
       'DatabaseName' => 'SOMEAPP'
    },
    'resources' =>
    {
      'jdbc/SomeappDS' =>
        {'description' => 'SomeappSQL Connection Resource'}
    }
},
'custom_resources' =>
{
  'MyServiceURL' => 'http://${CM:SOA_HOSTNAME}/MyService'

}}.to_json

You will notice that there is variables that need to be replaced as part of
the template expansion and this is done via various mechanisms. Sometimes
these values are extracted from data bags, sometimes derived etc.

These templates also declare other resources that the application either
needs or provides. i.e. It defines the set of queues the queue server is
meant to manage, the set of checks that our monitoring infrastructure
should run etc. These are all deep merged into the required nodes as part
of the release.

The application developers are responsible for writing the template, while
the operations are responsible for validating and deploying. The developers
are responsible for deploying their binaries and configuration templates
into the same location (aka nexus) and chef picks it up from there. Locally
devs run using vagrant, and then the changes are rolled out through
different environments.

So what I wrote above is partially a lie. The above describes the model we
are moving towards.

At the moment there is no clear distinction between dev and ops so
everyone has to modify chef artefacts to do a release. Currently our
configuration template is not in nexus but is instead namespaced entries in
a data bag that needs to be uploaded as part of a release. Currently some
applications still need custom chef scripts, if they do exceptional things
(like have a data directory, set up in a particular way in a particular
location on host). We have also not yet figured out any reasonable approach
to pairing the version of the configuration template with the version of
the associated binary artefacts. We also have limited value interpolation.

However, said that we like what we have got now and plan to move further
down this path in the future.

[1]
http://realityforge.org/code/2012/11/12/reusable-cookbooks-revisited.html

–
Cheers,

Peter Donald

11

I have to say that it's really therapeutic for me to hear that others
are struggling with similar issues. I think that the idea of
introducing CI earlier in the process of CM as the community matures
will be a big win as far as managing workstreams.

Sascha Bates | sascha.bates@gmail.com | 612 850 0444 |
sascha.bates@skype | sascha_bates@yahoo |
On 1/28/13 3:16 AM, Matthias Marschall wrote:

Even though my team is doing both: developing application features as well as Chef cookbooks we see the disconnect. We've two separate repositories and developers do not work on Chef configured Vagrant boxes all the time. They only use Vagrant when they're changing cookbooks.
Additionally we've split configuration values between our Rails app (in environments) and Chef.
The only place where app and infrastructure mingle is our staging environment so far.

We're planning to setup a CI using Chef to create the VMs to run the tests. But we're not there yet :frowning:

Interesting that you say that about Noah. My current client is looking at a service management model to deal with configuration files, eliminating Chef entirely from the mix. I have also discussed with a dev team shipping in config files from artifact repository and we did it with overrideable attributes, allowing someone else to use either a default template in the library cookbook or supply their own in another cookbook or drop a template on the server from elsewhere. I guess my preference would be to eventually have the templates parametrized so that anyone could use the same template.

But then the question arises, how do you deal with changing data? For this we have been using a combination of attributes and data bags. Chef solo would be a viable option for workstation testing, but they'd still have to check these things in and have build scripts that dealt with getting them into the Chef server/org. We have a lot of this now and it's just not connected.

Maybe this is something that would actually work if you could get in front of the development and build process and hand a dev team a working CI process that involved Chef. Lately I only get involved when it's too late to do anything besides panic.

Sascha Bates | sascha.bates@gmail.com | 612 850 0444 | sascha.bates@skype | sascha_bates@yahoo |
On 1/27/13 9:14 AM, John E. Vincent (lusis) wrote:

One idea that EJ (your cohort on the podcast had) that I thought was brilliantly simple is to have the applications ship ERB templates for any conf files by default and let chef do its template magic with a local file instead of one in the repo.

The disconnect is always going to exist unless your dev team is willing to work with the chef repo. Frankly it's pretty unfair to ask that out id the gate and keeping the two in sync is painful.

The first motivation I had for the (sadly languishing) Noah was this exact use case.

On Jan 27, 2013 1:14 AM, "Sascha Bates" sascha.bates@gmail.com wrote:

Thanks. I'm in an environment where we use jenkins and maven for a lot of this at the moment. There's also some Vagrant/vbox in the mix as well as Eclipse/STS. I'm not looking for pointers so much as being interested in what others have found to be successful.

One of the problems I see is that Chef code and Application code are decoupled: different repos, different teams managing collections of cookbooks, diverse environments. Dev teams don't think of Chef as part of the code base and I believe that this has led to some dysfunctional attitudes and behaviors. At the moment I'm noodling on methodology for introducing a tool like Chef that decouples configuration code from application logic.

So I guess I'm maybe not looking for deployment solutions so much as wondering what people are doing to get their chef and app code working together and how it's working out for them.

Sascha Bates | sascha.bates@gmail.com | 612 850 0444 | sascha.bates@skype | sascha_bates@yahoo |
On 1/26/13 9:59 PM, Gourav Shah wrote:

Is anyone here doing application builds (either yourself or for devs) that involve any of the following:

-Local (workstation) testing of a developed app with Chef housing configuration files

For local development with chef managing configs, you could try using vagrant + chef

-Application bundling with ant, maven, groovy, scripts, whatever that also includes parallel check-ins to Chef code?
-CI testing of an application that builds an artifact but also has configurations in Chef?
I have used Jenkins + chef. You could also add capistrano in the mix for deployment.

Thanks
Gourav

12

I’ve been embedding the cookbooks/roles/data bags that are directly related to an application in the application repository itself, and using CI to update development environments (or organizations) at build time, then promoting through with asset revisions.

Adam

From: Sascha Bates <sascha.bates@gmail.commailto:sascha.bates@gmail.com>
Reply-To: "chef@lists.opscode.commailto:chef@lists.opscode.com" <chef@lists.opscode.commailto:chef@lists.opscode.com>
Date: Saturday, January 26, 2013 10:14 PM
To: "chef@lists.opscode.commailto:chef@lists.opscode.com" <chef@lists.opscode.commailto:chef@lists.opscode.com>
Cc: Gourav Shah <gs@initcron.orgmailto:gs@initcron.org>
Subject: [chef] Re: Re: application builds/ci involving Chef

Thanks. I’m in an environment where we use jenkins and maven for a lot of this at the moment. There’s also some Vagrant/vbox in the mix as well as Eclipse/STS. I’m not looking for pointers so much as being interested in what others have found to be successful.

One of the problems I see is that Chef code and Application code are decoupled: different repos, different teams managing collections of cookbooks, diverse environments. Dev teams don’t think of Chef as part of the code base and I believe that this has led to some dysfunctional attitudes and behaviors. At the moment I’m noodling on methodology for introducing a tool like Chef that decouples configuration code from application logic.

So I guess I’m maybe not looking for deployment solutions so much as wondering what people are doing to get their chef and app code working together and how it’s working out for them.

Sascha Bates | sascha.bates@gmail.commailto:sascha.bates@gmail.com | 612 850 0444 | sascha.bates@skype | sascha_bates@yahoo |
On 1/26/13 9:59 PM, Gourav Shah wrote:

Is anyone here doing application builds (either yourself or for devs) that involve any of the following:

-Local (workstation) testing of a developed app with Chef housing configuration files

For local development with chef managing configs, you could try using vagrant + chef

-Application bundling with ant, maven, groovy, scripts, whatever that also includes parallel check-ins to Chef code?
-CI testing of an application that builds an artifact but also has configurations in Chef?
I have used Jenkins + chef. You could also add capistrano in the mix for deployment.

Thanks
Gourav

13

Adam,

can you provide some more info on how you do this? do you use custom
git hooks + ruby script to update the data bag values ? Also, by
"promoting through with asset revisions" or you referring to promoting
cookbooks through environments or something else?

On Mon, Jan 28, 2013 at 5:59 PM, Adam Jacob adam@opscode.com wrote:

I've been embedding the cookbooks/roles/data bags that are directly related
to an application in the application repository itself, and using CI to
update development environments (or organizations) at build time, then
promoting through with asset revisions.

Adam

From: Sascha Bates sascha.bates@gmail.com
Reply-To: "chef@lists.opscode.com" chef@lists.opscode.com
Date: Saturday, January 26, 2013 10:14 PM
To: "chef@lists.opscode.com" chef@lists.opscode.com
Cc: Gourav Shah gs@initcron.org
Subject: [chef] Re: Re: application builds/ci involving Chef

Thanks. I'm in an environment where we use jenkins and maven for a lot of
this at the moment. There's also some Vagrant/vbox in the mix as well as
Eclipse/STS. I'm not looking for pointers so much as being interested in
what others have found to be successful.

One of the problems I see is that Chef code and Application code are
decoupled: different repos, different teams managing collections of
cookbooks, diverse environments. Dev teams don't think of Chef as part of
the code base and I believe that this has led to some dysfunctional
attitudes and behaviors. At the moment I'm noodling on methodology for
introducing a tool like Chef that decouples configuration code from
application logic.

So I guess I'm maybe not looking for deployment solutions so much as
wondering what people are doing to get their chef and app code working
together and how it's working out for them.

Sascha Bates | sascha.bates@gmail.com | 612 850 0444 | sascha.bates@skype |
sascha_bates@yahoo |
On 1/26/13 9:59 PM, Gourav Shah wrote:

Is anyone here doing application builds (either yourself or for devs) that
involve any of the following:

-Local (workstation) testing of a developed app with Chef housing
configuration files

For local development with chef managing configs, you could try using
vagrant + chef

-Application bundling with ant, maven, groovy, scripts, whatever that also
includes parallel check-ins to Chef code?
-CI testing of an application that builds an artifact but also has
configurations in Chef?

I have used Jenkins + chef. You could also add capistrano in the mix for
deployment.

Thanks
Gourav

14

Essentially it's:

  • Gated trunk style repositories with code review by Gerrit
  • Hypothetical-master-merge testing for proposed code
  • Cookbook/data bag/role/etc changes get automatically uploaded and the
    development environment gets equality pinned
  • Development environments promote to a single integration environment

If a repository in Gerrit has a chef-repo directory in it, the above takes
place.

We also have been doing application development and deployment in a
similar workflow, that includes tracking build information in a data bag
per application, that maps to a desired application version in an
environment default attribute. This comes together in the deployment
recipes to allow you to select the right deployment assets for an
environment.

Deployment beyond integration happens by taking the environment cookbook
pins and the application revisions from the previous environment and
setting the next environment to those values, then kicking off chef runs.

Best,
Adam

On 1/28/13 11:44 AM, "Bryan Berry" bryan.berry@gmail.com wrote:

Adam,

can you provide some more info on how you do this? do you use custom
git hooks + ruby script to update the data bag values ? Also, by
"promoting through with asset revisions" or you referring to promoting
cookbooks through environments or something else?

On Mon, Jan 28, 2013 at 5:59 PM, Adam Jacob adam@opscode.com wrote:

I've been embedding the cookbooks/roles/data bags that are directly
related
to an application in the application repository itself, and using CI to
update development environments (or organizations) at build time, then
promoting through with asset revisions.

Adam

From: Sascha Bates sascha.bates@gmail.com
Reply-To: "chef@lists.opscode.com" chef@lists.opscode.com
Date: Saturday, January 26, 2013 10:14 PM
To: "chef@lists.opscode.com" chef@lists.opscode.com
Cc: Gourav Shah gs@initcron.org
Subject: [chef] Re: Re: application builds/ci involving Chef

Thanks. I'm in an environment where we use jenkins and maven for a lot
of
this at the moment. There's also some Vagrant/vbox in the mix as well as
Eclipse/STS. I'm not looking for pointers so much as being interested in
what others have found to be successful.

One of the problems I see is that Chef code and Application code are
decoupled: different repos, different teams managing collections of
cookbooks, diverse environments. Dev teams don't think of Chef as part
of
the code base and I believe that this has led to some dysfunctional
attitudes and behaviors. At the moment I'm noodling on methodology for
introducing a tool like Chef that decouples configuration code from
application logic.

So I guess I'm maybe not looking for deployment solutions so much as
wondering what people are doing to get their chef and app code working
together and how it's working out for them.

Sascha Bates | sascha.bates@gmail.com | 612 850 0444 |
sascha.bates@skype |
sascha_bates@yahoo |
On 1/26/13 9:59 PM, Gourav Shah wrote:

Is anyone here doing application builds (either yourself or for devs)
that
involve any of the following:

-Local (workstation) testing of a developed app with Chef housing
configuration files

For local development with chef managing configs, you could try using
vagrant + chef

-Application bundling with ant, maven, groovy, scripts, whatever that
also
includes parallel check-ins to Chef code?
-CI testing of an application that builds an artifact but also has
configurations in Chef?

I have used Jenkins + chef. You could also add capistrano in the mix
for
deployment.

Thanks
Gourav

15

tks Adam, this is quite helpful

On Mon, Jan 28, 2013 at 9:02 PM, Adam Jacob adam@opscode.com wrote:

Essentially it's:

  • Gated trunk style repositories with code review by Gerrit
  • Hypothetical-master-merge testing for proposed code
  • Cookbook/data bag/role/etc changes get automatically uploaded and the
    development environment gets equality pinned
  • Development environments promote to a single integration environment

If a repository in Gerrit has a chef-repo directory in it, the above takes
place.

We also have been doing application development and deployment in a
similar workflow, that includes tracking build information in a data bag
per application, that maps to a desired application version in an
environment default attribute. This comes together in the deployment
recipes to allow you to select the right deployment assets for an
environment.

Deployment beyond integration happens by taking the environment cookbook
pins and the application revisions from the previous environment and
setting the next environment to those values, then kicking off chef runs.

Best,
Adam

On 1/28/13 11:44 AM, "Bryan Berry" bryan.berry@gmail.com wrote:

Adam,

can you provide some more info on how you do this? do you use custom
git hooks + ruby script to update the data bag values ? Also, by
"promoting through with asset revisions" or you referring to promoting
cookbooks through environments or something else?

On Mon, Jan 28, 2013 at 5:59 PM, Adam Jacob adam@opscode.com wrote:

I've been embedding the cookbooks/roles/data bags that are directly
related
to an application in the application repository itself, and using CI to
update development environments (or organizations) at build time, then
promoting through with asset revisions.

Adam

From: Sascha Bates sascha.bates@gmail.com
Reply-To: "chef@lists.opscode.com" chef@lists.opscode.com
Date: Saturday, January 26, 2013 10:14 PM
To: "chef@lists.opscode.com" chef@lists.opscode.com
Cc: Gourav Shah gs@initcron.org
Subject: [chef] Re: Re: application builds/ci involving Chef

Thanks. I'm in an environment where we use jenkins and maven for a lot
of
this at the moment. There's also some Vagrant/vbox in the mix as well as
Eclipse/STS. I'm not looking for pointers so much as being interested in
what others have found to be successful.

One of the problems I see is that Chef code and Application code are
decoupled: different repos, different teams managing collections of
cookbooks, diverse environments. Dev teams don't think of Chef as part
of
the code base and I believe that this has led to some dysfunctional
attitudes and behaviors. At the moment I'm noodling on methodology for
introducing a tool like Chef that decouples configuration code from
application logic.

So I guess I'm maybe not looking for deployment solutions so much as
wondering what people are doing to get their chef and app code working
together and how it's working out for them.

Sascha Bates | sascha.bates@gmail.com | 612 850 0444 |
sascha.bates@skype |
sascha_bates@yahoo |
On 1/26/13 9:59 PM, Gourav Shah wrote:

Is anyone here doing application builds (either yourself or for devs)
that
involve any of the following:

-Local (workstation) testing of a developed app with Chef housing
configuration files

For local development with chef managing configs, you could try using
vagrant + chef

-Application bundling with ant, maven, groovy, scripts, whatever that
also
includes parallel check-ins to Chef code?
-CI testing of an application that builds an artifact but also has
configurations in Chef?

I have used Jenkins + chef. You could also add capistrano in the mix
for
deployment.

Thanks
Gourav