AWS OpsWorks custom

Chef Infra (archive)
1

I'm using AWS OpsWorks for a project. I think most of my cookbooks will not
require modification, and I can just use the ones from


I expect there will be some of those I will need to modify and perhaps will
need to create some new ones.
(For example, I would like to use the django recipe, but want to substitute
uwsgi for gunicorn.)

I'm a bit confused on how to structure this. I believe I need to point my
OpsWorks stack to a github repo that contains the cookbooks. That's done in
the stack configuration:
"Use custom Chef Cookbooks"
OpsWorks requires the repo to be structured like this:
http://docs.aws.amazon.com/opsworks/latest/userguide/workingcookbook-installingcustom-repo.html

I've seen some mentions of using git submodel to incorporate the required repos
in a "parent" repo. I've also read that that practice is deprecated. What
is the best practice?

I've looked into Bershelf and Librarian-chef, but don't see an easy way to end
up with the required directory structure while still being able to add new
cookbooks and modify the existing ones. I'm trying to avoid just making a copy
of the cookbooks. I think I also need to make sure dependencies can be
satisfied.

Can anyone give me some tips for a sane OpsWorks/Chef workflow?

2

Hi,

We use a variant of the braid tool to do something similar to this.
Braid will merge in a remote git repository into a local git
repository and allow you to periodically update from the source repo.
The original author stopped looking after the product so we use a
variant you can install via “gem install realityforge-braid”

So a sample workflow may be

mkdir mycookbooks cd mycookbooks
git init Initialized empty Git repository in .../mycookbooks/.git/ vi README
git add README git commit -m “My First commit”
[master (root-commit) 407698e] My First commit
0 files changed
create mode 100644 README
braid add https://github.com/opscode-cookbooks/sql_server.git sql_server Braid: Adding mirror of 'https://github.com/opscode-cookbooks/sql_server.git'. Braid: Setup: Creating remote for 'sql_server'. Braid: Added mirror at 'd9702f8'. ls -a
. … .braids .git README sql_server

And then you can import as many cookbooks as you want, potentially
upgrade them at a later point via

$ braid update sql_server/
Braid: Updating mirror ‘sql_server’.
Braid: Mirror ‘sql_server’ is already up to date.

You can also edit the local cookbook and commit changes. It is easy to
view these changes via

$ braid diff sql_server/
diff --git a/README.md b/README.md
index 34e8973…a9e6356 100644
— a/README.md
+++ b/README.md
@@ -3,6 +3,8 @@ Description

Installs and configures Microsoft SQL Server 2008 R2 server and
client. By default the Express edition is installed, but the
sql_server::server recipe supports installation of other editions
(see Usage below).

+Look at me - I am local changes
+
Requirements

You can see the list of mirrors via something like

$ braid list

Braid: Listing all mirrors.

sql_server (Locally Modified)

HTH

–
Cheers,

Peter Donald

3

On May 23, 2013, at 5:42 PM, liamk@numenet.com wrote:

I'm using AWS OpsWorks for a project. I think most of my cookbooks will not
require modification, and I can just use the ones from
http://github.com/opscode-cookbooks/
I expect there will be some of those I will need to modify and perhaps will
need to create some new ones.
(For example, I would like to use the django recipe, but want to substitute
uwsgi for gunicorn.)

I'm afraid OpsWorks is on such an old version of Chef that you may find that few community cookbooks work on it as is. There is no good solution for this that I'm aware of, except for Amazon moving to a somewhat more recent version of Chef.

--Noah

4

Word is that Amazon is working on moving to Chef 11... someone in IRC
posted a link to an AWS engineer confirming that the other day.

--
~~ StormeRider ~~

"Every world needs its heroes [...] They inspire us to be better than we
are. And they protect from the darkness that's just around the corner."

(from Smallville Season 6x1: "Zod")

On why I hate the phrase "that's so lame"... http://bit.ly/Ps3uSS

On Thu, May 23, 2013 at 6:25 PM, Noah Kantrowitz noah@coderanger.netwrote:

On May 23, 2013, at 5:42 PM, liamk@numenet.com wrote:

I'm using AWS OpsWorks for a project. I think most of my cookbooks will
not
require modification, and I can just use the ones from
http://github.com/opscode-cookbooks/
I expect there will be some of those I will need to modify and perhaps
will
need to create some new ones.
(For example, I would like to use the django recipe, but want to
substitute
uwsgi for gunicorn.)

I'm afraid OpsWorks is on such an old version of Chef that you may find
that few community cookbooks work on it as is. There is no good solution
for this that I'm aware of, except for Amazon moving to a somewhat more
recent version of Chef.

--Noah

5

Thanks, Peter! That looks like what was looking for.

Is the mirror a mirror of the master branch, or can you specify
different branches?
The repo is not a fork, I gather.

I may give this a try, although someone else made a comment that
OpsWorks version of Chef is old and does not support many of the opscode
community cookbooks.
So, I may have to come up with a different plan.

Liam

On 05/23/2013 06:20 PM, Peter Donald wrote:

Hi,

We use a variant of the braid tool to do something similar to this.
Braid will merge in a remote git repository into a local git
repository and allow you to periodically update from the source repo.
The original author stopped looking after the product so we use a
variant you can install via "gem install realityforge-braid"

So a sample workflow may be

$ mkdir mycookbooks
$ cd mycookbooks
$ git init
Initialized empty Git repository in .../mycookbooks/.git/
$ vi README
$ git add README
$ git commit -m "My First commit"
[master (root-commit) 407698e] My First commit
0 files changed
create mode 100644 README
$ braid add GitHub - sous-chefs/sql_server: Development repository for the sql_server cookbook sql_server
Braid: Adding mirror of 'GitHub - sous-chefs/sql_server: Development repository for the sql_server cookbook'.
Braid: Setup: Creating remote for 'sql_server'.
Braid: Added mirror at 'd9702f8'.
$ ls -a
. .. .braids .git README sql_server

And then you can import as many cookbooks as you want, potentially
upgrade them at a later point via

$ braid update sql_server/
Braid: Updating mirror 'sql_server'.
Braid: Mirror 'sql_server' is already up to date.

You can also edit the local cookbook and commit changes. It is easy to
view these changes via

$ braid diff sql_server/
diff --git a/README.md b/README.md
index 34e8973..a9e6356 100644
--- a/README.md
+++ b/README.md
@@ -3,6 +3,8 @@ Description

Installs and configures Microsoft SQL Server 2008 R2 server and
client. By default the Express edition is installed, but the
sql_server::server recipe supports installation of other editions
(see Usage below).

+Look at me - I am local changes
+
Requirements

You can see the list of mirrors via something like

$ braid list

Braid: Listing all mirrors.

sql_server (Locally Modified)

HTH

--
Liam Kirsher
PGP: http://liam.numenet.com/pgp/

6

Noah,

Thanks for mentioning that!
Is there any way to know which cookbooks work with OpsWorks and which
don't? These are the ones I think I need:

cookbook 'nginx'
cookbook 'rabbitmq'
cookbook 'mongodb'
cookbook 'mercurial'
cookbook 'newrelic'
cookbook 'newrelic_monitoring'
cookbook 'django'
cookbook 'application'
cookbook 'application_nginx'
cookbook 'application_python'
cookbook 'uwsgi'

(Along with other cookbooks they require, I guess.)

Thanks again.

On 05/23/2013 06:25 PM, Noah Kantrowitz wrote:

On May 23, 2013, at 5:42 PM, liamk@numenet.com wrote:

I'm using AWS OpsWorks for a project. I think most of my cookbooks will not
require modification, and I can just use the ones from
http://github.com/opscode-cookbooks/
I expect there will be some of those I will need to modify and perhaps will
need to create some new ones.
(For example, I would like to use the django recipe, but want to substitute
uwsgi for gunicorn.)
I'm afraid OpsWorks is on such an old version of Chef that you may find that few community cookbooks work on it as is. There is no good solution for this that I'm aware of, except for Amazon moving to a somewhat more recent version of Chef.

--Noah

--
Liam Kirsher
PGP: http://liam.numenet.com/pgp/

7

Okay, I see the requirements are listed on the cookbook page.

On 05/24/2013 01:11 PM, Liam Kirsher wrote:

Noah,

Thanks for mentioning that!
Is there any way to know which cookbooks work with OpsWorks and which
don't? These are the ones I think I need:

cookbook 'nginx'
cookbook 'rabbitmq'
cookbook 'mongodb'
cookbook 'mercurial'
cookbook 'newrelic'
cookbook 'newrelic_monitoring'
cookbook 'django'
cookbook 'application'
cookbook 'application_nginx'
cookbook 'application_python'
cookbook 'uwsgi'
(Along with other cookbooks they require, I guess.)

Thanks again.

On 05/23/2013 06:25 PM, Noah Kantrowitz wrote:

On May 23, 2013, at 5:42 PM, liamk@numenet.com wrote:

I'm using AWS OpsWorks for a project. I think most of my cookbooks will not
require modification, and I can just use the ones from
http://github.com/opscode-cookbooks/
I expect there will be some of those I will need to modify and perhaps will
need to create some new ones.
(For example, I would like to use the django recipe, but want to substitute
uwsgi for gunicorn.)
I'm afraid OpsWorks is on such an old version of Chef that you may find that few community cookbooks work on it as is. There is no good solution for this that I'm aware of, except for Amazon moving to a somewhat more recent version of Chef.

--Noah

--
Liam Kirsher
PGP: http://liam.numenet.com/pgp/

8

I don't know your requirements, but I do have to ask; is there a specific
reason you're looking at using OpsWorks instead of running Hosted Chef or
an Open Source Chef server? Until AWS upgrades OpsWorks to Chef 11, I think
it's a bit crippled, personally.

--
~~ StormeRider ~~

"Every world needs its heroes [...] They inspire us to be better than we
are. And they protect from the darkness that's just around the corner."

(from Smallville Season 6x1: "Zod")

On why I hate the phrase "that's so lame"... http://bit.ly/Ps3uSS

On Fri, May 24, 2013 at 1:11 PM, Liam Kirsher liamk@numenet.com wrote:

Noah,

Thanks for mentioning that!
Is there any way to know which cookbooks work with OpsWorks and which
don't? These are the ones I think I need:

cookbook 'nginx'
cookbook 'rabbitmq'
cookbook 'mongodb'
cookbook 'mercurial'
cookbook 'newrelic'
cookbook 'newrelic_monitoring'
cookbook 'django'
cookbook 'application'
cookbook 'application_nginx'
cookbook 'application_python'
cookbook 'uwsgi'

(Along with other cookbooks they require, I guess.)

Thanks again.

On 05/23/2013 06:25 PM, Noah Kantrowitz wrote:

On May 23, 2013, at 5:42 PM, liamk@numenet.com wrote:

I'm using AWS OpsWorks for a project. I think most of my cookbooks
will not
require modification, and I can just use the ones from
http://github.com/opscode-cookbooks/
I expect there will be some of those I will need to modify and perhaps
will
need to create some new ones.
(For example, I would like to use the django recipe, but want to
substitute
uwsgi for gunicorn.)
I'm afraid OpsWorks is on such an old version of Chef that you may find
that few community cookbooks work on it as is. There is no good solution
for this that I'm aware of, except for Amazon moving to a somewhat more
recent version of Chef.

--Noah

--
Liam Kirsher
PGP: http://liam.numenet.com/pgp/

9

There's no specific reason -- I'm just trying to find the quickest way
to setup deployment infrastructure. OpsWorks looked like it would be
easy to set up, and integrated with all the other AWS services.
However, now that I'm aware of the chef version issue... not so much.
So, I'm setting up a Hosted Chef account now.
I'm sure I'll have more questions, though. Thanks for you help!

On May 20th AWS mentioned they were upgrading chef to version 11, but
there was no mention of /when/ that would be accomplished.

On 05/24/2013 01:33 PM, Morgan Blackthorne wrote:

I don't know your requirements, but I do have to ask; is there a
specific reason you're looking at using OpsWorks instead of running
Hosted Chef or an Open Source Chef server? Until AWS upgrades OpsWorks
to Chef 11, I think it's a bit crippled, personally.

--
~~ StormeRider ~~

"Every world needs its heroes [...] They inspire us to be better than
we are. And they protect from the darkness that's just around the corner."

(from Smallville Season 6x1: "Zod")

On why I hate the phrase "that's so lame"... http://bit.ly/Ps3uSS

On Fri, May 24, 2013 at 1:11 PM, Liam Kirsher <liamk@numenet.com
mailto:liamk@numenet.com> wrote:

Noah,

Thanks for mentioning that!
Is there any way to know which cookbooks work with OpsWorks and which
don't?  These are the ones I think I need:
> cookbook 'nginx'
> cookbook 'rabbitmq'
> cookbook 'mongodb'
> cookbook 'mercurial'
> cookbook 'newrelic'
> cookbook 'newrelic_monitoring'
> cookbook 'django'
> cookbook 'application'
> cookbook 'application_nginx'
> cookbook 'application_python'
> cookbook 'uwsgi'

(Along with other cookbooks they require, I guess.)

Thanks again.


On 05/23/2013 06:25 PM, Noah Kantrowitz wrote:
> On May 23, 2013, at 5:42 PM, <liamk@numenet.com
<mailto:liamk@numenet.com>> wrote:
>
>> I'm using AWS OpsWorks for a project.  I think most of my
cookbooks will not
>> require modification, and I can just use the ones from
>> http://github.com/opscode-cookbooks/
>> I expect there will be some of those I will need to modify and
perhaps will
>> need to create some new ones.
>> (For example, I would like to use the django recipe, but want
to substitute
>> uwsgi for gunicorn.)
> I'm afraid OpsWorks is on such an old version of Chef that you
may find that few community cookbooks work on it as is. There is
no good solution for this that I'm aware of, except for Amazon
moving to a somewhat more recent version of Chef.
>
> --Noah
>

--
Liam Kirsher
PGP: http://liam.numenet.com/pgp/

--
Liam Kirsher
PGP: http://liam.numenet.com/pgp/

10

nod In theory, OpsWorks is great. In practice... losing the ability to
reliably use the vast amount of community cookbooks is a pretty strong
negative.

--
~~ StormeRider ~~

"Every world needs its heroes [...] They inspire us to be better than we
are. And they protect from the darkness that's just around the corner."

(from Smallville Season 6x1: "Zod")

On why I hate the phrase "that's so lame"... http://bit.ly/Ps3uSS

On Fri, May 24, 2013 at 1:52 PM, Liam Kirsher liamk@numenet.com wrote:

There's no specific reason -- I'm just trying to find the quickest way to
setup deployment infrastructure. OpsWorks looked like it would be easy to
set up, and integrated with all the other AWS services.
However, now that I'm aware of the chef version issue... not so much.
So, I'm setting up a Hosted Chef account now.
I'm sure I'll have more questions, though. Thanks for you help!

On May 20th AWS mentioned they were upgrading chef to version 11, but
there was no mention of when that would be accomplished.

On 05/24/2013 01:33 PM, Morgan Blackthorne wrote:

I don't know your requirements, but I do have to ask; is there a specific
reason you're looking at using OpsWorks instead of running Hosted Chef or
an Open Source Chef server? Until AWS upgrades OpsWorks to Chef 11, I think
it's a bit crippled, personally.

--
~~ StormeRider ~~

"Every world needs its heroes [...] They inspire us to be better than we
are. And they protect from the darkness that's just around the corner."

(from Smallville Season 6x1: "Zod")

On why I hate the phrase "that's so lame"... http://bit.ly/Ps3uSS

On Fri, May 24, 2013 at 1:11 PM, Liam Kirsher liamk@numenet.com wrote:

Noah,

Thanks for mentioning that!
Is there any way to know which cookbooks work with OpsWorks and which
don't? These are the ones I think I need:

cookbook 'nginx'
cookbook 'rabbitmq'
cookbook 'mongodb'
cookbook 'mercurial'
cookbook 'newrelic'
cookbook 'newrelic_monitoring'
cookbook 'django'
cookbook 'application'
cookbook 'application_nginx'
cookbook 'application_python'
cookbook 'uwsgi'

(Along with other cookbooks they require, I guess.)

Thanks again.

On 05/23/2013 06:25 PM, Noah Kantrowitz wrote:

On May 23, 2013, at 5:42 PM, liamk@numenet.com wrote:

I'm using AWS OpsWorks for a project. I think most of my cookbooks
will not
require modification, and I can just use the ones from
http://github.com/opscode-cookbooks/
I expect there will be some of those I will need to modify and perhaps
will
need to create some new ones.
(For example, I would like to use the django recipe, but want to
substitute
uwsgi for gunicorn.)
I'm afraid OpsWorks is on such an old version of Chef that you may find
that few community cookbooks work on it as is. There is no good solution
for this that I'm aware of, except for Amazon moving to a somewhat more
recent version of Chef.

--Noah

--
Liam Kirsher
PGP: http://liam.numenet.com/pgp/

--
Liam Kirsher
PGP: http://liam.numenet.com/pgp/

11

I'm going to throw in my 2c here...

On Fri, May 24, 2013 at 9:33 PM, Morgan Blackthorne stormerider@gmail.com
wrote:

I don't know your requirements, but I do have to ask; is there a specific
reason you're looking at using OpsWorks instead of running Hosted Chef or
an Open Source Chef server? Until AWS upgrades OpsWorks to Chef 11, I think
it's a bit crippled, personally.

Even as someone who has mostly used Chef in a client/server environment, I
don't think it's fair to describe OpsWorks as crippled. It's old, sure,
but it's useful today - though I think Amazon are being incredibly
disingenuous in claiming that you can go ahead and use the community
cookbooks with it.

I looked at it for a project recently and decided that the older Chef
version would be an irritation, but not a deal-breaker.

Why not Hosted Chef? For one thing, it can be painfully slow in some
regions. My experience in ap-southeast-1 last year was not especially
positive. For another, it's not free. Though I think the pricing is
reasonable, cost sensitivities in bootstrapped projects make it a hard sell
(particularly when the quality of service is less that awesome).
Why not Open Source Chef? Not necessarily any cheaper than Hosted (if
you're running it on AWS), and it's something else to manage - that's a
small increment in many environments, but a big deal when you're standing
up a small environment.

Why would I consider OpsWorks instead of Chef Solo? There's a bunch of
plumbing involved in running chef-solo securely that I would have happily
avoided, and the "lifecycle events" in OpsWorks sounded like they'd be
useful enough to replace search for the parts I really cared about. No
(encrypted) data bags is a bit sad, but wouldn't have been an
insurmountable problem.

So, then, why didn't I use it? The dealbreaker for me was spotty support
for provisioning and managing other AWS services. I was also somewhat
uncomfortable with the level of control I appeared to be trading for
convenience, and wasn't confident that I'd be able to easily shift back in
the other direction. I ended up using CloudFormation with Chef Solo. It
has involved more plumbing than I'd like, but there's no deal-breakers (for
my use case) and I'm confident that I can easily upgrade to chef-client in
the future.

I can imagine using OpsWorks for other projects, but I'm not bothered by
the prospect of backporting community cookbooks (or implementing my own).
The cost of doing so isn't always significant, and the benefits are worth
something. That being said, folks who are fairly new to Chef might want to
wait until they AWS upgrade to version 11. The community largely moved on
from 0.9 a long time ago, and it's not a good place to start.

Zac

12

On Fri, May 24, 2013 at 1:42 AM, liamk@numenet.com wrote:

I'm a bit confused on how to structure this. I believe I need to point my
OpsWorks stack to a github repo that contains the cookbooks. That's done
in
the stack configuration:
"Use custom Chef Cookbooks"
OpsWorks requires the repo to be structured like this:

Cookbook Repositories - AWS OpsWorks

I've seen some mentions of using git submodel to incorporate the required
repos
in a "parent" repo. I've also read that that practice is deprecated. What
is the best practice?

I've looked into Bershelf and Librarian-chef, but don't see an easy way to
end
up with the required directory structure while still being able to add new
cookbooks and modify the existing ones. I'm trying to avoid just making a
copy
of the cookbooks. I think I also need to make sure dependencies can be
satisfied.

If I was doing this, I'd use Berkshelf. I would create a Berksfile to
declare the cookbooks I was using. The process to deploy updated cookbooks
would be:

  1. run "berks install" to copy them into a local directory
  2. created a tarball of the directory
  3. upload the tarball to S3
    This can be trivially scripted.

Zac