Cross-posting this from https://coderanger.net/state-of-the-onion-may-26/ as this is likely of interest to y’all.
It has now been almost six months since I started on my adventures in application
deployment and I want to update everybody on how things are progressing. First
off, a huge “thank you” to my Kickstarter backers and Bloomberg for supporting
this development time. It truly would not have been possible without community
support.
Halite
The biggest new development so far has been
Halite. While not specifically related to
application deployment, it has greatly improved my development workflow for
cookbooks. You can read more in the Halite documentation, the short version is
that Halite allows converting normal Ruby gems in to Chef cookbooks. This is
handy for cookbooks that are otherwise 99% library code anyway as I can use the
normal gem workflow tools more effectively, and then convert to a cookbook for
release/upload. Halite 1.0 is available on Rubygems.org and should be ready for
general use if you find the workflow helpful. I wouldn’t go so far as to call it
a “best practice” and normal cookbooks are still perfectly fine if you prefer
them. All my Halite’d cookbooks are released in cookbook form to the
Supermarket, so you can use them without needing
to care about any of this.
If you want to try out prerelease versions of my cookbooks you will have to
do a little extra work to get Berkshelf to recognize them. Halite includes a
new source location for Berkshelf, gem: 'name'
. To use a prerelease version of
a Halite-based cookbook/gem you first need to install the gem itself, generally
through a Gemfile:
gem 'poise-application', github: 'poise/application'
Once the gem is installed you can use it in your Berksfile:
source 'https://supermarket.chef.io/'
extension 'halite'
cookbook 'application', gem: 'poise-application'
Anything that integrates with Berkshelf (Test Kitchen, Chefspec, etc) will now
build the prerelease cookbook from the gem code as needed.
Poise
Poise has seen big internal improvements and
restructuring around Halite and some other workflow tools I’ve written, but from
a user perspective it is still mostly the same. A few new helpers have been
added including Fused mode where action implementations can be written directly
in the resource class, and Inversion which adds dependency injection/inversion
to the Chef resource and provider model.
Poise 2.0 has been released to both Rubygems.org and the Supermarket.
Application Cookbooks
The core application
cookbook has been rebuilt from scratch to be more
flexible and powerful. It no longer uses the deploy
resource internally and
many of the old bugs are gone thanks to switching to Poise instead of the early,
ad-hoc versions of the helpers I wrote all those years ago. It will still be
plugin-based; but rather than a model based around the deploy
resource
callbacks, plugins are now nested sub-resources and can define their own internal
structure as needed. This also means that the explicit deployment phases are
gone, instead relying on the ordering of resources in your code.
Two of the plugin cookbooks are relatively complete. application_git
provides
support for deploying application code from a git repository, including all the
deploy key handling the old application
resource handled internally.
application_ruby
has been converted over to the new structure with resources
for bundle_install
, rackup
, thin
, unicorn
, and rails
. Passenger
support in the old cookbook was minimal at best and is going to be put off to
its own application_passenger
cookbook to be written in the future.
An example of a full Rails deployment and configuration using the new cookbooks:
include_recipe 'build-essential'
package %w{ruby-dev zlib1g-dev libsqlite3-dev}
application '/opt/test_rails' do
git 'https://github.com/poise/test_rails.git'
bundle_install do
deployment true
without %w{development test}
end
rails do
database 'sqlite3:///db.sqlite3'
secret_token 'd78fe08df56c9'
end
unicorn do
port 8080
end
end
Next on the list is to rebuild the application_python
cookbook, and then
application_javascript
and application_java
after that.
Poise Service Cookbook
As part of the application deployment work I found a need to have a generic
service configuration API for Chef. In the above example, I need to install
unicorn as a service, but from the application_ruby
cookbook I have no way to
know which service framework the end-user wants to do that with. Traditionally
we’ve worked around this by ignoring the end-user and hard-wiring things to use
whatever system is generally dominant in that community, Runit for Ruby and
Supervisord for Python.
poise-service
uses the new Inversion
helper in Poise to allow the library cookbook code to stay generic while the
end-user can configure which underlying service framework to use and how to
set it up. It currently has support for SysVInit, Upstart, Systemd, and Runit.
Support for Supervisord will be added when I get to that point in the
application_python
rebuild.
An example of using poise-service
to create an init script for Apache:
poise_service_user 'www-data'
poise_service 'apache2' do
command '/usr/sbin/apache2 -f /etc/apache2/apache2.conf -DFOREGROUND'
stop_signal 'WINCH'
reload_signal 'USR1'
end
You can find more information about poise-service
in
the documentation.
poise-service
and poise-service-runit
1.0 have been released to both
Rubygems.org and the Supermarket.
Poise Ruby Cookbook
poise-ruby
uses the same dependency
inversion approach as poise-service
but for installing Ruby runtimes. It
currently supports installing from system packages and from ruby-build
(the
underlying build tool used by both rbenv and chruby, and rvm soon). This is
still prerelease but should be ready for external testing and will hopefully
be released shortly. If you have ever struggled with RVM or Rbenv in Chef
recipes I would greatly value your feedback on the design and functionality.
Incoming Cookbooks
As part of the Python and Javascript revamps I plan to work on equivalents to
poise-ruby
for each of those. Java would be nice too, but is enough of a mess
that I don’t currently plan to rework the java
cookbook itself. Bloomberg is
also going to support basic application deployment plugins for Go and Erlang, as
well as exploratory work on integrating service discovery frameworks like
ZooKeeper and Consul in to Chef for situations where Chef Search is not
available or usable.
Secrets Management
Totally unrelated to the above but I wanted to mention that there have been
several interesting new entries in to the field of
secrets management and while all are still very new they are
worth following in the hopes they will be proven safe enough for production use.
These include Sneaker,
KeyWhiz, and Vault.
As part of improving application deployment I would like to see deep integration
between these systems and Chef, but I don’t currently have time scheduled to
work on this.
Questions?
If anyone has any questions on any of these cookbooks or my future plans for
them please don’t hesitate to get in contact with me. You can reach me at
noah@coderanger.net or any of the methods on my contact page.
–Noah