Best Practice :: Performing App Deployments with Chef

All,

I’m reaching out to you from a project that is harnessing Chef to automate
deployment of Application artifacts that change regularly, such as EARs, WARs,
DB Schemas etc. This is definitely something that Chef can do however I feel
that the approach my team has come to is a poor one. I’m looking for a ‘best
practice’ solution to solving this particular problem with Chef. Let me explain
where my team stands right now…

Someone wants to deploy a new version of HelloWorld EAR into our Dev
environment. Presently he does this by kicking off a job in Jenkins which
simply exposes the target host and the application artifact to deploy.

Under the covers Jenkins SSH’s onto the remote server and invokes the
’chef-client’ command on that host, passing some JSON that specifies where the
recipe should get the EAR file from — basically the version of the EAR since
the path is consistent.

So this works. Chef goes ahead and deploys the new artifact. Awesome. However
it seems really hacky. I don’t like that this approach means that Environment
config on the Chef Server will always of date with our Environment JSON in
source control.

I feel that the correct approach is to use Jenkins to update the JSON for the
environment you want to deploy to in source control, then updates Chef-Server
using knife. Finally Jenkins calls ‘chef-client’ on the box and it will work as
above.

The real value that I see from this second approach is that if you commit each
change to your environment’s JSON to source control, you have traceability. The
other benefit is that you could build a totally new Chef Server (e.g. in a
second cloud) and because your source control is up to date with your
environment, you should get an identical environment built up by Chef. Finally,
everything is aligned: if you need to check what version of your app is
deployed to your dev environment you could check in source control, on Chef
server or on Jenkins and it will all be aligned.

What are your thoughts on these two approaches? I would like to see what the
Chef community has done to solve this problem before so I can align my project
with best practice in this particular area. Need your support :slight_smile:

Many Thanks,
Luke

Hey Luke,

Thanks for bringing this up, this is a common problem that I personally
have seen time and time again needing to be solved in the chef community. I
can at least shed some light on how my teams have accomplished this. My
team performs deploys at any time, sometimes we can have up to 20-30
deploys a day to any environment (dev, staging, qa, production). We first
started out using the environment files as the first indication of the
build. Essentially we would all update an attribute override for
application-version in the environment. We have about 50 different portions
that go into our whole application so we can build a single portion an
deploy that portion as needed. The problem with this was, our developers
needed to push as well and eventually we ended up having to put information
overrides into the environment file as well that included things like
production application passwords or keys that would fill in
certain configuration files per environment. These attributes we did not
want everyone to be able to see or have access to.

Our second approach was to use jenkins to edit the json file. We attempted
a series of regex/perl replacements for strings, this sucked and i will
leave it at that :stuck_out_tongue:

Eventually, we got to where we recently landed within the last couple of
weeks. I ended up writing tools for our team, I am contemplating open
sourcing it if I can see enough people who would like to use it. But
essentially I wrote a web based (django app) that allows users to edit
portions of the environment files based on their permissions level. This
does sanity checking to ensure that its modified properly, when the edit is
done, it pushes to git to track and backup the modification, and it also
makes an API call to the local chef server to actually process the change.
The tool also allows the developers and admins to launch deploys per
server/environment based on their permissions levels as well.

I hope that helps but essentially yes, you should be tracking everything in
git. How you want to edit the json files seems to be the #1 disconnect so
far of managing chef. I have written this tool and I know some others who
have done similar. If interested I could hack together an open source
version of my tool you can fit into your environment.

Jonathan

On Fri, May 31, 2013 at 7:28 AM, Luke luke.a.egging@accenture.com wrote:

All,

I’m reaching out to you from a project that is harnessing Chef to automate
deployment of Application artifacts that change regularly, such as EARs,
WARs,
DB Schemas etc. This is definitely something that Chef can do however I
feel
that the approach my team has come to is a poor one. I'm looking for a
'best
practice' solution to solving this particular problem with Chef. Let me
explain
where my team stands right now...

Someone wants to deploy a new version of HelloWorld EAR into our Dev
environment. Presently he does this by kicking off a job in Jenkins which
simply exposes the target host and the application artifact to deploy.

Under the covers Jenkins SSH's onto the remote server and invokes the
'chef-client' command on that host, passing some JSON that specifies where
the
recipe should get the EAR file from --- basically the version of the EAR
since
the path is consistent.

So this works. Chef goes ahead and deploys the new artifact. Awesome.
However
it seems really hacky. I don't like that this approach means that
Environment
config on the Chef Server will always of date with our Environment JSON in
source control.

I feel that the correct approach is to use Jenkins to update the JSON for
the
environment you want to deploy to in source control, then updates
Chef-Server
using knife. Finally Jenkins calls 'chef-client' on the box and it will
work as
above.

The real value that I see from this second approach is that if you commit
each
change to your environment's JSON to source control, you have
traceability. The
other benefit is that you could build a totally new Chef Server (e.g. in a
second cloud) and because your source control is up to date with your
environment, you should get an identical environment built up by Chef.
Finally,
everything is aligned: if you need to check what version of your app is
deployed to your dev environment you could check in source control, on Chef
server or on Jenkins and it will all be aligned.

What are your thoughts on these two approaches? I would like to see what
the
Chef community has done to solve this problem before so I can align my
project
with best practice in this particular area. Need your support :slight_smile:

Many Thanks,
Luke

On Friday, May 31, 2013 at 7:56 AM, Jonathan Mickle wrote:

Hey Luke,

Thanks for bringing this up, this is a common problem that I personally have seen time and time again needing to be solved in the chef community. I can at least shed some light on how my teams have accomplished this. My team performs deploys at any time, sometimes we can have up to 20-30 deploys a day to any environment (dev, staging, qa, production). We first started out using the environment files as the first indication of the build. Essentially we would all update an attribute override for application-version in the environment. We have about 50 different portions that go into our whole application so we can build a single portion an deploy that portion as needed. The problem with this was, our developers needed to push as well and eventually we ended up having to put information overrides into the environment file as well that included things like production application passwords or keys that would fill in certain configuration files per environment. These attributes we did no
t want everyone to be able to see or have access to.

Our second approach was to use jenkins to edit the json file. We attempted a series of regex/perl replacements for strings, this sucked and i will leave it at that :stuck_out_tongue:

Eventually, we got to where we recently landed within the last couple of weeks. I ended up writing tools for our team, I am contemplating open sourcing it if I can see enough people who would like to use it. But essentially I wrote a web based (django app) that allows users to edit portions of the environment files based on their permissions level. This does sanity checking to ensure that its modified properly, when the edit is done, it pushes to git to track and backup the modification, and it also makes an API call to the local chef server to actually process the change. The tool also allows the developers and admins to launch deploys per server/environment based on their permissions levels as well.

I hope that helps but essentially yes, you should be tracking everything in git. How you want to edit the json files seems to be the #1 disconnect so far of managing chef. I have written this tool and I know some others who have done similar. If interested I could hack together an open source version of my tool you can fit into your environment.

Jonathan
Why not use data bags? You can have either one big data bag of app versions, or one per app, and have data bag items named after the environment, which makes it easy to look them up in your recipe code. You can then just let Ci own these data bags and not have to worry about partial edits (as long as you make things granular enough). If git is the only workable change tracking solution for you, you could probably put these data bags in their own repo that only Ci commits to, so you won't have to worry about merge conflicts either.

--
Daniel DeLeo

Daniel,

Data bags are doable as well and I have used and seen them used in the
past. For us, environment files were easiest for everyone to pick up. Also
we keep our most secure items in the data bags so that they are not in the
environment files where developers have access to in the github repo.

On Fri, May 31, 2013 at 8:52 AM, Daniel DeLeo dan@kallistec.com wrote:

On Friday, May 31, 2013 at 7:56 AM, Jonathan Mickle wrote:

Hey Luke,

Thanks for bringing this up, this is a common problem that I personally
have seen time and time again needing to be solved in the chef community. I
can at least shed some light on how my teams have accomplished this. My
team performs deploys at any time, sometimes we can have up to 20-30
deploys a day to any environment (dev, staging, qa, production). We first
started out using the environment files as the first indication of the
build. Essentially we would all update an attribute override for
application-version in the environment. We have about 50 different portions
that go into our whole application so we can build a single portion an
deploy that portion as needed. The problem with this was, our developers
needed to push as well and eventually we ended up having to put information
overrides into the environment file as well that included things like
production application passwords or keys that would fill in
certain configuration files per environment. These attributes we did not
want everyone to be able to see or have access to.

Our second approach was to use jenkins to edit the json file. We attempted
a series of regex/perl replacements for strings, this sucked and i will
leave it at that :stuck_out_tongue:

Eventually, we got to where we recently landed within the last couple of
weeks. I ended up writing tools for our team, I am contemplating open
sourcing it if I can see enough people who would like to use it. But
essentially I wrote a web based (django app) that allows users to edit
portions of the environment files based on their permissions level. This
does sanity checking to ensure that its modified properly, when the edit is
done, it pushes to git to track and backup the modification, and it also
makes an API call to the local chef server to actually process the change.
The tool also allows the developers and admins to launch deploys per
server/environment based on their permissions levels as well.

I hope that helps but essentially yes, you should be tracking everything
in git. How you want to edit the json files seems to be the #1 disconnect
so far of managing chef. I have written this tool and I know some others
who have done similar. If interested I could hack together an open source
version of my tool you can fit into your environment.

Jonathan

Why not use data bags? You can have either one big data bag of app
versions, or one per app, and have data bag items named after the
environment, which makes it easy to look them up in your recipe code. You
can then just let Ci own these data bags and not have to worry about
partial edits (as long as you make things granular enough). If git is the
only workable change tracking solution for you, you could probably put
these data bags in their own repo that only Ci commits to, so you won't
have to worry about merge conflicts either.

--
Daniel DeLeo

I've had to solve this problem a few times now at work for different teams
with different philosophies, so I've had the chance to try a few different
approaches.

The one that seems to have worked the best for everyone concerned is this:

  • Build server uploads build artifacts to an artifact repository
    (Artifactory or Nexus, for us).
  • The artifact repository app gets to decide what artifact's appropriate
    for what environment.
  • Chef client runs query the artifact repository (possibly multiple repos)
    and use the newest accessible version.

I don't much like the idea of a development server that's ssh'ing directly
into nodes and running chef-client on them (though I guess if you're
limiting that user to only be able to "sudo chef-client" that could be okay
from a security perspective). The security guys here didn't like it
either, so we set up a RunDeck server in the data center environment and
have the developer-controlled CI servers triggering pre-scoped RunDeck jobs
to do the Chef runs (i.e., "trigger the RunDeck job that runs Chef on all
the Tomcat servers that run my app in the development environment").

Works pretty well so far.

On Fri, May 31, 2013 at 10:34 AM, Jonathan Mickle jmickle05@gmail.comwrote:

Daniel,

Data bags are doable as well and I have used and seen them used in the
past. For us, environment files were easiest for everyone to pick up. Also
we keep our most secure items in the data bags so that they are not in the
environment files where developers have access to in the github repo.

On Fri, May 31, 2013 at 8:52 AM, Daniel DeLeo dan@kallistec.com wrote:

On Friday, May 31, 2013 at 7:56 AM, Jonathan Mickle wrote:

Hey Luke,

Thanks for bringing this up, this is a common problem that I personally
have seen time and time again needing to be solved in the chef community. I
can at least shed some light on how my teams have accomplished this. My
team performs deploys at any time, sometimes we can have up to 20-30
deploys a day to any environment (dev, staging, qa, production). We first
started out using the environment files as the first indication of the
build. Essentially we would all update an attribute override for
application-version in the environment. We have about 50 different portions
that go into our whole application so we can build a single portion an
deploy that portion as needed. The problem with this was, our developers
needed to push as well and eventually we ended up having to put information
overrides into the environment file as well that included things like
production application passwords or keys that would fill in
certain configuration files per environment. These attributes we did not
want everyone to be able to see or have access to.

Our second approach was to use jenkins to edit the json file. We
attempted a series of regex/perl replacements for strings, this sucked and
i will leave it at that :stuck_out_tongue:

Eventually, we got to where we recently landed within the last couple of
weeks. I ended up writing tools for our team, I am contemplating open
sourcing it if I can see enough people who would like to use it. But
essentially I wrote a web based (django app) that allows users to edit
portions of the environment files based on their permissions level. This
does sanity checking to ensure that its modified properly, when the edit is
done, it pushes to git to track and backup the modification, and it also
makes an API call to the local chef server to actually process the change.
The tool also allows the developers and admins to launch deploys per
server/environment based on their permissions levels as well.

I hope that helps but essentially yes, you should be tracking everything
in git. How you want to edit the json files seems to be the #1 disconnect
so far of managing chef. I have written this tool and I know some others
who have done similar. If interested I could hack together an open source
version of my tool you can fit into your environment.

Jonathan

Why not use data bags? You can have either one big data bag of app
versions, or one per app, and have data bag items named after the
environment, which makes it easy to look them up in your recipe code. You
can then just let Ci own these data bags and not have to worry about
partial edits (as long as you make things granular enough). If git is the
only workable change tracking solution for you, you could probably put
these data bags in their own repo that only Ci commits to, so you won't
have to worry about merge conflicts either.

--
Daniel DeLeo