Extending Chef Deploy for Object Storage Services


#1

Ohai devs!

I’m finally had a chance to play around with deploy resource and
application cookbooks. While the deploy resource is pretty complicated, it
works great and has an amazing amount of flexibility.

I have a requirement to deploy application code from tarballs living in an
object storage service (e.g. S3) – this so we don’t depend on github
being available during an autoscaling event. I have to support all kinds
of app stacks, including rails, tomcat and php.

I just wanted to get some expert eyes on this idea to see if there is any
value in it and if this approach is the right direction.

Here are some more details about what I am thinking…

https://github.com/caryp-contrib/proposals/tree/chef-deploy-oss

Questions:

  • Does it make sense to extend the deploy and scm resources to support
    an ObjectStorage SCM resource and providers?
  • Does the way of matching archive filenames and the naming conventions
    make sense?
  • Other thoughts?

Thanks!

@kindsol (aka caryp)


#2

the ark resource shares some of the features. might be easier to extend
that one,

On Wed, May 8, 2013 at 5:22 PM, Sol Kindle kindsol2@gmail.com wrote:

Ohai devs!

I’m finally had a chance to play around with deploy resource and
application cookbooks. While the deploy resource is pretty complicated, it
works great and has an amazing amount of flexibility.

I have a requirement to deploy application code from tarballs living in an
object storage service (e.g. S3) – this so we don’t depend on github
being available during an autoscaling event. I have to support all kinds
of app stacks, including rails, tomcat and php.

I just wanted to get some expert eyes on this idea to see if there is any
value in it and if this approach is the right direction.

Here are some more details about what I am thinking…

https://github.com/caryp-contrib/proposals/tree/chef-deploy-oss

Questions:

  • Does it make sense to extend the deploy and scm resources to support
    an ObjectStorage SCM resource and providers?
  • Does the way of matching archive filenames and the naming
    conventions make sense?
  • Other thoughts?

Thanks!

@kindsol (aka caryp)


#3

Thanks for the reply!

I agree that the ‘ark’ resource shares some similar features. There are also some potential features in ark that would be nice to “borrow” for this proposed SCM provider.

Whereas, the main purpose for ark is to do the “fetch-unpack-configure-build-install dance” for building/installing binaries, the purpose of this ObjectSCM provider is to fetch web application code in a way that we can leverage the all the migration, callback and rollback functionality offered by the existing Deploy Resource[1].

I have updated the overview and implementation sections in an attempt to be more clear about that. I have also renamed the proposed provider to “Chef::Provider::ScmObject”. Please see:

Cheers!

  • CaryP

[1] - http://docs.opscode.com/resource_deploy.html

On May 8, 2013, at 6:13 PM, Ranjib Dey wrote:

the ark resource shares some of the features. might be easier to extend that one,

On Wed, May 8, 2013 at 5:22 PM, Sol Kindle kindsol2@gmail.com wrote:
Ohai devs!

I’m finally had a chance to play around with deploy resource and application cookbooks. While the deploy resource is pretty complicated, it works great and has an amazing amount of flexibility.

I have a requirement to deploy application code from tarballs living in an object storage service (e.g. S3) – this so we don’t depend on github being available during an autoscaling event. I have to support all kinds of app stacks, including rails, tomcat and php.

I just wanted to get some expert eyes on this idea to see if there is any value in it and if this approach is the right direction.

Here are some more details about what I am thinking…

https://github.com/caryp-contrib/proposals/tree/chef-deploy-oss

Questions:
Does it make sense to extend the deploy and scm resources to support an ObjectStorage SCM resource and providers?
Does the way of matching archive filenames and the naming conventions make sense?
Other thoughts?
Thanks!

@kindsol (aka caryp)


#4

I totally support your effort, I’ve noticed quite a few people asking for this. I’d especially to make the application cookbooks work with non-scm artifacts (we’re leaning toward maven, gem and deb packs).

One piece of advice: call your work something other than deploy until you’ve got it awesome. I also would avoid the term scm, most people don’t think of artifact repos as scm repos. And getting bogged down in naming discussions is useful.

We most need an awesome resource that supports deploy’s (and the application cb’s) event callbacks but lets us use any repo. If we get it awesome by the feature freeze in chef 12, I’ll happily fight to get it in core. But let’s get it awesome first.

On 2013-05-09, at 11:54, kindsol kindsol2@gmail.com wrote:

Thanks for the reply!

I agree that the ‘ark’ resource shares some similar features. There are also some potential features in ark that would be nice to “borrow” for this proposed SCM provider.

Whereas, the main purpose for ark is to do the “fetch-unpack-configure-build-install dance” for building/installing binaries, the purpose of this ObjectSCM provider is to fetch web application code in a way that we can leverage the all the migration, callback and rollback functionality offered by the existing Deploy Resource[1].

I have updated the overview and implementation sections in an attempt to be more clear about that. I have also renamed the proposed provider to “Chef::Provider::ScmObject”. Please see:

https://github.com/caryp-contrib/proposals/tree/chef-deploy-oss

Cheers!

  • CaryP

[1] - http://docs.opscode.com/resource_deploy.html

On May 8, 2013, at 6:13 PM, Ranjib Dey wrote:

the ark resource shares some of the features. might be easier to extend that one,

On Wed, May 8, 2013 at 5:22 PM, Sol Kindle kindsol2@gmail.com wrote:
Ohai devs!

I’m finally had a chance to play around with deploy resource and application cookbooks. While the deploy resource is pretty complicated, it works great and has an amazing amount of flexibility.

I have a requirement to deploy application code from tarballs living in an object storage service (e.g. S3) – this so we don’t depend on github being available during an autoscaling event. I have to support all kinds of app stacks, including rails, tomcat and php.

I just wanted to get some expert eyes on this idea to see if there is any value in it and if this approach is the right direction.

Here are some more details about what I am thinking…

https://github.com/caryp-contrib/proposals/tree/chef-deploy-oss

Questions:
• Does it make sense to extend the deploy and scm resources to support an ObjectStorage SCM resource and providers?
• Does the way of matching archive filenames and the naming conventions make sense?
• Other thoughts?
Thanks!

@kindsol (aka caryp)


#5

I agree w/ Cary that this is outside the scope of ark, which doesn’t
have any of the cool callback stuff that deploy does. Until today I
had never looked at the deploy resource :stuck_out_tongue:

I notice in the doc that you have support for accessing Azure’s object
storage. As far as I am aware fog doesn’t support Azure. Is there an
alternate open-source library or extension to fog for this?

I recommend starting this off as an independent ruby gem and then
pushing the code upstream into chef proper. Starting as a ruby gem
will make the code easier to integrate into chef proper later as you
can follow the structure and conventions of the chef gem.

I would be happy to call it “cloud_file” or raindrop :slight_smile: i am also
quite interested in contributing to this as I have similar
requirements. I would call it oss as well, that means open-source
software to me :slight_smile:

sorry I missed you at chefconf caryp!

On Thu, May 9, 2013 at 9:15 PM, Joseph Holsten joseph@josephholsten.com wrote:

I totally support your effort, I’ve noticed quite a few people asking for this. I’d especially to make the application cookbooks work with non-scm artifacts (we’re leaning toward maven, gem and deb packs).

One piece of advice: call your work something other than deploy until you’ve got it awesome. I also would avoid the term scm, most people don’t think of artifact repos as scm repos. And getting bogged down in naming discussions is useful.

We most need an awesome resource that supports deploy’s (and the application cb’s) event callbacks but lets us use any repo. If we get it awesome by the feature freeze in chef 12, I’ll happily fight to get it in core. But let’s get it awesome first.

On 2013-05-09, at 11:54, kindsol kindsol2@gmail.com wrote:

Thanks for the reply!

I agree that the ‘ark’ resource shares some similar features. There are also some potential features in ark that would be nice to “borrow” for this proposed SCM provider.

Whereas, the main purpose for ark is to do the “fetch-unpack-configure-build-install dance” for building/installing binaries, the purpose of this ObjectSCM provider is to fetch web application code in a way that we can leverage the all the migration, callback and rollback functionality offered by the existing Deploy Resource[1].

I have updated the overview and implementation sections in an attempt to be more clear about that. I have also renamed the proposed provider to “Chef::Provider::ScmObject”. Please see:

https://github.com/caryp-contrib/proposals/tree/chef-deploy-oss

Cheers!

  • CaryP

[1] - http://docs.opscode.com/resource_deploy.html

On May 8, 2013, at 6:13 PM, Ranjib Dey wrote:

the ark resource shares some of the features. might be easier to extend that one,

On Wed, May 8, 2013 at 5:22 PM, Sol Kindle kindsol2@gmail.com wrote:
Ohai devs!

I’m finally had a chance to play around with deploy resource and application cookbooks. While the deploy resource is pretty complicated, it works great and has an amazing amount of flexibility.

I have a requirement to deploy application code from tarballs living in an object storage service (e.g. S3) – this so we don’t depend on github being available during an autoscaling event. I have to support all kinds of app stacks, including rails, tomcat and php.

I just wanted to get some expert eyes on this idea to see if there is any value in it and if this approach is the right direction.

Here are some more details about what I am thinking…

https://github.com/caryp-contrib/proposals/tree/chef-deploy-oss

Questions:
• Does it make sense to extend the deploy and scm resources to support an ObjectStorage SCM resource and providers?
• Does the way of matching archive filenames and the naming conventions make sense?
• Other thoughts?
Thanks!

@kindsol (aka caryp)


#6

In fact, we are already doing this by passing signed S3 links to
Chef::Provider::RemoteFile::Deploy sub resource of deploy. We played around
with an implementation of a S3 file sub resource but found that the deploy
resource doesn’t allow passing username/password
(access_key_id/secret_access_key) to sub resources. support for this needs
to be added to deploy because svn may need this as well (i think there’s a
ticket somewhere).

Another thing is support for artifactory/nexus repos - this is also a
frequent scenario.

On Fri, May 10, 2013 at 11:56 AM, Bryan Berry bryan.berry@gmail.com wrote:

I agree w/ Cary that this is outside the scope of ark, which doesn’t
have any of the cool callback stuff that deploy does. Until today I
had never looked at the deploy resource :stuck_out_tongue:

I notice in the doc that you have support for accessing Azure’s object
storage. As far as I am aware fog doesn’t support Azure. Is there an
alternate open-source library or extension to fog for this?

I recommend starting this off as an independent ruby gem and then
pushing the code upstream into chef proper. Starting as a ruby gem
will make the code easier to integrate into chef proper later as you
can follow the structure and conventions of the chef gem.

I would be happy to call it “cloud_file” or raindrop :slight_smile: i am also
quite interested in contributing to this as I have similar
requirements. I would call it oss as well, that means open-source
software to me :slight_smile:

sorry I missed you at chefconf caryp!

On Thu, May 9, 2013 at 9:15 PM, Joseph Holsten joseph@josephholsten.com
wrote:

I totally support your effort, I’ve noticed quite a few people asking
for this. I’d especially to make the application cookbooks work with
non-scm artifacts (we’re leaning toward maven, gem and deb packs).

One piece of advice: call your work something other than deploy until
you’ve got it awesome. I also would avoid the term scm, most people don’t
think of artifact repos as scm repos. And getting bogged down in naming
discussions is useful.

We most need an awesome resource that supports deploy’s (and the
application cb’s) event callbacks but lets us use any repo. If we get it
awesome by the feature freeze in chef 12, I’ll happily fight to get it in
core. But let’s get it awesome first.

On 2013-05-09, at 11:54, kindsol kindsol2@gmail.com wrote:

Thanks for the reply!

I agree that the ‘ark’ resource shares some similar features. There
are also some potential features in ark that would be nice to “borrow” for
this proposed SCM provider.

Whereas, the main purpose for ark is to do the
"fetch-unpack-configure-build-install dance" for building/installing
binaries, the purpose of this ObjectSCM provider is to fetch web
application code in a way that we can leverage the all the migration,
callback and rollback functionality offered by the existing Deploy
Resource[1].

I have updated the overview and implementation sections in an attempt
to be more clear about that. I have also renamed the proposed provider to
"Chef::Provider::ScmObject". Please see:

https://github.com/caryp-contrib/proposals/tree/chef-deploy-oss

Cheers!

  • CaryP

[1] - http://docs.opscode.com/resource_deploy.html

On May 8, 2013, at 6:13 PM, Ranjib Dey wrote:

the ark resource shares some of the features. might be easier to
extend that one,

On Wed, May 8, 2013 at 5:22 PM, Sol Kindle kindsol2@gmail.com wrote:
Ohai devs!

I’m finally had a chance to play around with deploy resource and
application cookbooks. While the deploy resource is pretty complicated, it
works great and has an amazing amount of flexibility.

I have a requirement to deploy application code from tarballs living
in an object storage service (e.g. S3) – this so we don’t depend on
github being available during an autoscaling event. I have to support all
kinds of app stacks, including rails, tomcat and php.

I just wanted to get some expert eyes on this idea to see if there is
any value in it and if this approach is the right direction.

Here are some more details about what I am thinking…

https://github.com/caryp-contrib/proposals/tree/chef-deploy-oss

Questions:
• Does it make sense to extend the deploy and scm resources to
support an ObjectStorage SCM resource and providers?

 • Does the way of matching archive filenames and the naming

conventions make sense?

 • Other thoughts?

Thanks!

@kindsol (aka caryp)