Is Chef a good fit for Java deployments?

Chef Infra (archive)
1

My company is looking at a way to automate Java deployments, and manage
environments. There are tools from the Java community that will automate
deployments, but they won’t do configuration management more broadly, and none
of them are terribly well established.

Our environments are relatively diverse, but to start with, we’d like to manage
a collection of apps running on Tomcat servers. We build with Maven and deploy
CI snapshots and releases to an internal Nexus repository. In an ideal world,
I’d like to be able to do two things:

  1. When a new release occurs, tell Chef that test servers should receive that
    release by incrementing the version number. Then have Chef download the
    binaries from Nexus (which I’m sure it can do with its Maven resource), and
    pass them to Tomcat. It seems to me this should be pretty easy.

  2. On development servers, have Chef continuously check the snapshots on Nexus
    and perform the same installation procedure when an update occurs. This also
    seems like it’s probably achievable.

What worries me is that Chef doesn’t seem to be very widely used for Java
environments, and I’m wondering if there’s a reason. Am I going down the wrong
road, or does it seem like this is a good fit?

For background, Chef will also need to configure agents for Logstash and Nagios
(which I know it can do), and update an XML file that defines Logback logging
configuration for Tomcat.

2

What we do is package the java tomcat app (war) in an rpm along with tomcat bundled in the same rpm.

On a CI build, the app is pushed into a Nexus repository which then exposes an Yum repository of these rpms.

Then we have chef do what it does best: install that application as a package resource with action :update or controlled via an attr for a specific version in production. If you use Nexus Pro, they have staging repos so you could promote these builds through nexus and have your various environments simply do an update of the rpm.

This of course assumes you are using a red hat based system. But if not, I would highly recommend the packaging of the app in the manner I described in the package format you require.

Once the package is deployed, your cookbook knows how to configure the appropriate files to make the tomcat app run.

On 2013-05-31, at 12:15 PM, jamesged@gmail.com wrote:

My company is looking at a way to automate Java deployments, and manage
environments. There are tools from the Java community that will automate
deployments, but they won't do configuration management more broadly, and none
of them are terribly well established.

Our environments are relatively diverse, but to start with, we'd like to manage
a collection of apps running on Tomcat servers. We build with Maven and deploy
CI snapshots and releases to an internal Nexus repository. In an ideal world,
I'd like to be able to do two things:

  1. When a new release occurs, tell Chef that test servers should receive that
    release by incrementing the version number. Then have Chef download the
    binaries from Nexus (which I'm sure it can do with its Maven resource), and
    pass them to Tomcat. It seems to me this should be pretty easy.

  2. On development servers, have Chef continuously check the snapshots on Nexus
    and perform the same installation procedure when an update occurs. This also
    seems like it's probably achievable.

What worries me is that Chef doesn't seem to be very widely used for Java
environments, and I'm wondering if there's a reason. Am I going down the wrong
road, or does it seem like this is a good fit?

For background, Chef will also need to configure agents for Logstash and Nagios
(which I know it can do), and update an XML file that defines Logback logging
configuration for Tomcat.

3

I’ve used chef in a java shop before. We had hudson build the war files since it simply understands maven and can process pom files easily. The build artifacts are uploaded into Nexus. From there, you could also have hudson or jenkins update a data bag or a node attribute that specifies the version of the artifact you need chef-client to deploy. If you have chef-client running as a daemon, they’ll pick up the change on the next chef run and deploy your app. The recipe could be written in a way that uses the remote_file resource if it knew the nexus url to download the artifact from – the main difference on a new artifact is the version. You could also develop your own nexus LWRP to achieve this same result and be more flexible. If you don’t have chef-client running as a daemon, you’ll need some orchestration tool that kicks off chef-client on-demand. Again, hudson and jenkins could execute the knife ssh commands for you and achieve this, but you should consider the orchestration tool that best suits your needs.

The one downside to this approach: works well for one organization where you don’t have to manage the knife client keys and configuration on the orchestration side, but for larger implementations where you’re dealing with many chef organizations, you’ll need to figure out how to manage the knife client keys and configurations for each organization.

–Dang Nguyen

On 5/31/13 9:15 AM, "jamesged@gmail.commailto:jamesged@gmail.com" <jamesged@gmail.commailto:jamesged@gmail.com> wrote:

My company is looking at a way to automate Java deployments, and manage
environments. There are tools from the Java community that will automate
deployments, but they won’t do configuration management more broadly, and none
of them are terribly well established.

Our environments are relatively diverse, but to start with, we’d like to manage
a collection of apps running on Tomcat servers. We build with Maven and deploy
CI snapshots and releases to an internal Nexus repository. In an ideal world,
I’d like to be able to do two things:

  1. When a new release occurs, tell Chef that test servers should receive that
    release by incrementing the version number. Then have Chef download the
    binaries from Nexus (which I’m sure it can do with its Maven resource), and
    pass them to Tomcat. It seems to me this should be pretty easy.

  2. On development servers, have Chef continuously check the snapshots on Nexus
    and perform the same installation procedure when an update occurs. This also
    seems like it’s probably achievable.

What worries me is that Chef doesn’t seem to be very widely used for Java
environments, and I’m wondering if there’s a reason. Am I going down the wrong
road, or does it seem like this is a good fit?

For background, Chef will also need to configure agents for Logstash and Nagios
(which I know it can do), and update an XML file that defines Logback logging
configuration for Tomcat.

4

You're describing exactly what we do at Simply Measured. Hard version pins manually updated for prod, continuous delivery of snapshots for staging, artifacts in nexus.

We use an ancient version of the nexus repo cookbook: GitHub - RiotGamesCookbooks/nexus-cookbook, and are considering reworking our home grown artifact deploys to use GitHub - RiotGamesCookbooks/artifact-cookbook: Provides your cookbooks with the Artifact Deploy LWRP.

BTW, I'm pretty sure that there's a significant number of chef shops with non-trivial java environments. I think they're just less ostentatious than rubyists like me.

Joseph Holsten

On 2013-05-31, at 09:15, jamesged@gmail.com wrote:

My company is looking at a way to automate Java deployments, and manage
environments. There are tools from the Java community that will automate
deployments, but they won't do configuration management more broadly, and none
of them are terribly well established.

Our environments are relatively diverse, but to start with, we'd like to manage
a collection of apps running on Tomcat servers. We build with Maven and deploy
CI snapshots and releases to an internal Nexus repository. In an ideal world,
I'd like to be able to do two things:

  1. When a new release occurs, tell Chef that test servers should receive that
    release by incrementing the version number. Then have Chef download the
    binaries from Nexus (which I'm sure it can do with its Maven resource), and
    pass them to Tomcat. It seems to me this should be pretty easy.

  2. On development servers, have Chef continuously check the snapshots on Nexus
    and perform the same installation procedure when an update occurs. This also
    seems like it's probably achievable.

What worries me is that Chef doesn't seem to be very widely used for Java
environments, and I'm wondering if there's a reason. Am I going down the wrong
road, or does it seem like this is a good fit?

For background, Chef will also need to configure agents for Logstash and Nagios
(which I know it can do), and update an XML file that defines Logback logging
configuration for Tomcat.

5

We did the same for one of larger j2ee deployments. We had a notion of blue
green deployment where staging env will always deploy the latest but prod
env would go through gated blue green stages, i.e only on subset of tomcats
will perform the deployment, & they'll go out of the load balancer during
blue phase. Initially we had an lwrp that would exploit tomcat7's manager
api for hot deployment, later we changed the implementation to do actual
war cut/paste on disk and bounce the tomcat service (there was memery leaks
in the manager api). We also had some additional settings to manage the
hierarchical caching stuff.
On May 31, 2013 10:41 AM, "Joseph Holsten" joseph@josephholsten.com wrote:

You're describing exactly what we do at Simply Measured. Hard version pins
manually updated for prod, continuous delivery of snapshots for staging,
artifacts in nexus.

We use an ancient version of the nexus repo cookbook:
GitHub - RiotGamesCookbooks/nexus-cookbook, and are considering
reworking our home grown artifact deploys to use
GitHub - RiotGamesCookbooks/artifact-cookbook: Provides your cookbooks with the Artifact Deploy LWRP.

BTW, I'm pretty sure that there's a significant number of chef shops with
non-trivial java environments. I think they're just less ostentatious than
rubyists like me.

Joseph Holsten

On 2013-05-31, at 09:15, jamesged@gmail.com wrote:

My company is looking at a way to automate Java deployments, and manage
environments. There are tools from the Java community that will automate
deployments, but they won't do configuration management more broadly,
and none
of them are terribly well established.

Our environments are relatively diverse, but to start with, we'd like to
manage
a collection of apps running on Tomcat servers. We build with Maven and
deploy
CI snapshots and releases to an internal Nexus repository. In an ideal
world,
I'd like to be able to do two things:

  1. When a new release occurs, tell Chef that test servers should receive
    that
    release by incrementing the version number. Then have Chef download the
    binaries from Nexus (which I'm sure it can do with its Maven resource),
    and
    pass them to Tomcat. It seems to me this should be pretty easy.

  2. On development servers, have Chef continuously check the snapshots on
    Nexus
    and perform the same installation procedure when an update occurs. This
    also
    seems like it's probably achievable.

What worries me is that Chef doesn't seem to be very widely used for Java
environments, and I'm wondering if there's a reason. Am I going down the
wrong
road, or does it seem like this is a good fit?

For background, Chef will also need to configure agents for Logstash and
Nagios
(which I know it can do), and update an XML file that defines Logback
logging
configuration for Tomcat.

6

Hi,

We automate a large collection of java services that are hosted on
glassfish, tomcat or are command line applications. The artifacts are
stored in nexus and the version of the artifacts deployed in each
environment are stored as data bag items. We have 7 different environments
and the promotion of an artifacts between environments is either
manual(click a button to go) or automatic depending on the delivery
pipeline. We use Jenkins to do the deploy which mostly involves updating a
version in a data bag item and running chef client across a selection of
nodes. The release may involve multiple phases (I.e down application on
nodes in reverse order, migrate data, up nodes) but ultimately we have a
common approach to automating release that will rollback on failure.

One thing we don't do is use snapshot versions. Every artifact has a
specific version that can be traced to a specific version in git.

It is true that not s many talk about automating java but there is some of
us about. You may also run into different problems - long chef converge
times due to persistent services. But other than that it is all good :slight_smile:

On Saturday, June 1, 2013, wrote:

My company is looking at a way to automate Java deployments, and manage
environments. There are tools from the Java community that will automate
deployments, but they won't do configuration management more broadly, and
none
of them are terribly well established.

Our environments are relatively diverse, but to start with, we'd like to
manage
a collection of apps running on Tomcat servers. We build with Maven and
deploy
CI snapshots and releases to an internal Nexus repository. In an ideal
world,
I'd like to be able to do two things:

  1. When a new release occurs, tell Chef that test servers should receive
    that
    release by incrementing the version number. Then have Chef download the
    binaries from Nexus (which I'm sure it can do with its Maven resource), and
    pass them to Tomcat. It seems to me this should be pretty easy.

  2. On development servers, have Chef continuously check the snapshots on
    Nexus
    and perform the same installation procedure when an update occurs. This
    also
    seems like it's probably achievable.

What worries me is that Chef doesn't seem to be very widely used for Java
environments, and I'm wondering if there's a reason. Am I going down the
wrong
road, or does it seem like this is a good fit?

For background, Chef will also need to configure agents for Logstash and
Nagios
(which I know it can do), and update an XML file that defines Logback
logging
configuration for Tomcat.

--
Cheers,

Peter Donald