Open source Chef Server API: big changes in internal code is coming?


#1

Hello,

We were planning to :

  • modify the base source code of the Open Source Chef REST API
  • use the internal source code of the Open Source Chef REST API in a Rack
    middleware

But then, we have read that Opscode plan to do big changes in the code bases of
Chef Server (see below the references), for exemple, switching from Ruby to
Erlang.

So we are afraid that we will have to redo our work based on the actual chef
source code in some months.

Could you please give more information about the plan changes in code base ?
for example:
will the “chef-server-api\app\controllers” source code be totaly changed ?
if we write ruby code based on open source code of Chef, until when will it be
usable ?

We are now thinking that we should only program based on REST API through HTTP
( initially we wanted to bypass the HTTP and authentication layer).

Thanks in advance for your feedback.

Best regards,
Christophe

References:



In short, Chef is converging the code bases of their three products (hosted,
private and open). The primary change on this will moving from CouchBD to a SQL
based DB and moving away the API calls away from Merb/Ruby to Erlang. They are
also improving search so that we can make more fine-tuned requests that perform
better and return less extraneous data.

http://wiki.opscode.com/display/chef/Opscode+Chef+Short-Term+Roadmap+and+Performance+Improvements

"
Rob Hirschfeld’s Notes

Chef <3 Erlang

Scaling
moving to more Erlang on REST APIs, made a 2x in memory use difference
Erlang bits have realy steady performance
"

http://wiki.opscode.com/display/chef/Closing+Session+-+Decisions%2C+Follow-Ups%2C+and+Next+Steps
"
Erlang Chef Server - Chris Brown

Write docs on operating the erlang server

"


"
On the server side, we can expect a lot of performance improvements as
chef-server moves from ruby + couchdb to erlang + mysql. This may happen sooner
than one might normally expect as Opscode has already converted part of Hosted
Chef to Erlang and MySQL, it just hasn’t open-sourced that code yet.
"


#2

On May 11, 2012, at 6:42 AM, cl.subscription@gmail.com wrote:

Hello,

We were planning to :

  • modify the base source code of the Open Source Chef REST API
  • use the internal source code of the Open Source Chef REST API in a Rack
    middleware

But then, we have read that Opscode plan to do big changes in the code bases of
Chef Server (see below the references), for exemple, switching from Ruby to
Erlang.

So we are afraid that we will have to redo our work based on the actual chef
source code in some months.

Could you please give more information about the plan changes in code base ?
for example:
will the “chef-server-api\app\controllers” source code be totaly changed ?
if we write ruby code based on open source code of Chef, until when will it be
usable ?

We are now thinking that we should only program based on REST API through HTTP
( initially we wanted to bypass the HTTP and authentication layer).

The backend implementation will be entirely changed (somewhat unavoidable given that it is moving to a new language), but at least at first there will be no backwards incompatible changes to the API protocol/format itself. If you want something that is going to work more permanently, them yes it would be best to grab one of the API client libraries (Spice, PyChef, jclouds-chef, etc) and interact with the server through that.

–Noah


#3

Hello,

Thanks for your answer.

So as far as I understand, if we modify the source code of the REST API or
implementing a Rack middleware, we will have to rewrite everything in some
times, but if we work at the HTTP/REST/JSON level, the new implementation
should stay compatible with our work.
is my understanding correct please ?

in how much time approximatively the shift to the new backend
implementation is planned to be released please ?

Our goal is to “place” some custom code between the client and the server.
We were going to work at two level:

  • modifying the source code of the REST API
  • intercepting the HTTP request to the REST API in a Rack Middleware

What would be the best way to achieve that in a way that our code won’t
need to be changed when the new backend implementation will be release
please ?

Thanks in advance for your feedback.

Best regards,
Christophe

On Sat, May 12, 2012 at 12:41 AM, Noah Kantrowitz noah@coderanger.netwrote:

On May 11, 2012, at 6:42 AM, cl.subscription@gmail.com wrote:

Hello,

We were planning to :

  • modify the base source code of the Open Source Chef REST API
  • use the internal source code of the Open Source Chef REST API in a Rack
    middleware

But then, we have read that Opscode plan to do big changes in the code
bases of
Chef Server (see below the references), for exemple, switching from Ruby
to
Erlang.

So we are afraid that we will have to redo our work based on the actual
chef
source code in some months.

Could you please give more information about the plan changes in code
base ?
for example:
will the “chef-server-api\app\controllers” source code be totaly changed
?
if we write ruby code based on open source code of Chef, until when will
it be
usable ?

We are now thinking that we should only program based on REST API
through HTTP
( initially we wanted to bypass the HTTP and authentication layer).

The backend implementation will be entirely changed (somewhat unavoidable
given that it is moving to a new language), but at least at first there
will be no backwards incompatible changes to the API protocol/format
itself. If you want something that is going to work more permanently, them
yes it would be best to grab one of the API client libraries (Spice,
PyChef, jclouds-chef, etc) and interact with the server through that.

–Noah


#4

On Sunday, May 13, 2012 at 10:15 PM, Christophe L wrote:

Hello,

Thanks for your answer.

So as far as I understand, if we modify the source code of the REST API or implementing a Rack middleware, we will have to rewrite everything in some times, but if we work at the HTTP/REST/JSON level, the new implementation should stay compatible with our work.
is my understanding correct please ?

in how much time approximatively the shift to the new backend implementation is planned to be released please ?

Our goal is to “place” some custom code between the client and the server. We were going to work at two level:

  • modifying the source code of the REST API
  • intercepting the HTTP request to the REST API in a Rack Middleware

What would be the best way to achieve that in a way that our code won’t need to be changed when the new backend implementation will be release please ?

Thanks in advance for your feedback.

Best regards,
Christophe

What are you guys trying to do? There may be a variety of ways to achieve it.


Dan DeLeo


#5

Hello,

We are trying to add a security layer between the client and the server.

For the moment, the rule is simple: a non-admin client is allowed to see
only the information related to it-self.

The rules will be as follows:

  1. the endpoints /clients , /cookbooks , /nodes , /roles , /environments ,
    /search, /cookbooks/COOKBOOK_NAME on GET method will be forbidden for all
    non-admin client, so they can’t retrieve information about the whole
    infrastructure and settings related to other servers

  2. the endpoints /clients/CLIENT_NAME, /nodes/NODE_NAME and
    /nodes/NODE_NAME/cookbooks on GET method will be allowed only if
    CLIENT_NAME and NODE_NAME match the UserId found in the HTTP header

  3. the endpoints /roles/ROLE_NAME, /environments/ENVIRONMENT_NAME and
    /cookbooks/COOKBOOK_NAME/COOKBOOK_VERSION will be allowed only if the node
    (defined by the userId in the HTTP header) is respectively assigned to the
    role ROLE_NAME, in the environement ENVIRONMENT_NAME and has the cookbook
    COOKBOOK_NAME/COOKBOOK_VERSION in its cookbook list.
    Those information will be retrieved using knife.

For the moment, we are thinking to go with an Apache Handler written in
Perl and doing some regulare expression on URL based on the UserId
available in the HTTP Header and some system calls to knife when extra
information are necessary.
The Apache handler refusing to process the HTTP Request when the security
rule are not respected.
This should work since in our infrastructure the Chef API is accessible to
a VHost proxy on Apache.

Could you please let us know your feelings / your advices about this
solution ?

Another question would be: will there be an ACL system (even a simple one)
in the new backend implementation ?

Best regards,
Christophe

On Mon, May 14, 2012 at 8:14 PM, Daniel DeLeo dan@kallistec.com wrote:

On Sunday, May 13, 2012 at 10:15 PM, Christophe L wrote:

Hello,

Thanks for your answer.

So as far as I understand, if we modify the source code of the REST API
or implementing a Rack middleware, we will have to rewrite everything in
some times, but if we work at the HTTP/REST/JSON level, the new
implementation should stay compatible with our work.
is my understanding correct please ?

in how much time approximatively the shift to the new backend
implementation is planned to be released please ?

Our goal is to “place” some custom code between the client and the
server. We were going to work at two level:

  • modifying the source code of the REST API
  • intercepting the HTTP request to the REST API in a Rack Middleware

What would be the best way to achieve that in a way that our code won’t
need to be changed when the new backend implementation will be release
please ?

Thanks in advance for your feedback.

Best regards,
Christophe

What are you guys trying to do? There may be a variety of ways to achieve
it.


Dan DeLeo


#6

On May 15, 2012, at 10:39 PM, Christophe L wrote:

Hello,

We are trying to add a security layer between the client and the server.

For the moment, the rule is simple: a non-admin client is allowed to see only the information related to it-self.

The rules will be as follows:

  1. the endpoints /clients , /cookbooks , /nodes , /roles , /environments , /search, /cookbooks/COOKBOOK_NAME on GET method will be forbidden for all non-admin client, so they can’t retrieve information about the whole infrastructure and settings related to other servers

  2. the endpoints /clients/CLIENT_NAME, /nodes/NODE_NAME and /nodes/NODE_NAME/cookbooks on GET method will be allowed only if CLIENT_NAME and NODE_NAME match the UserId found in the HTTP header

  3. the endpoints /roles/ROLE_NAME, /environments/ENVIRONMENT_NAME and /cookbooks/COOKBOOK_NAME/COOKBOOK_VERSION will be allowed only if the node (defined by the userId in the HTTP header) is respectively assigned to the role ROLE_NAME, in the environement ENVIRONMENT_NAME and has the cookbook COOKBOOK_NAME/COOKBOOK_VERSION in its cookbook list.
    Those information will be retrieved using knife.

For the moment, we are thinking to go with an Apache Handler written in Perl and doing some regulare expression on URL based on the UserId available in the HTTP Header and some system calls to knife when extra information are necessary.
The Apache handler refusing to process the HTTP Request when the security rule are not respected.
This should work since in our infrastructure the Chef API is accessible to a VHost proxy on Apache.

Could you please let us know your feelings / your advices about this solution ?

Another question would be: will there be an ACL system (even a simple one) in the new backend implementation ?

I’ve answered similar questions on here before so I’ll keep this brief. Security isolation and/or multitenancy isn’t something that can be retrofitted on to the open source codebase unless you want to effectively rewrite it. Yes, this is subpar and we would like to see it changed, but currently our (opscode’s) backend dev cycles are being directed at getting Erchef up to snuff instead of improving the Ruby implementation massively. If this is truly a business requirement for you, Private Chef is a multitenant Chef server you can run on your own hardware. Beyond that the only way to get multiple effective chef servers is to run multiple chef-server-api daemons (and webui, expander). You can at least share some backend components, but the unit of security isolation in the open source codebase for now is the server instance. If you don’t mind starting from scratch but aren’t big on Ruby, you could take my Commis project (github.com/coderanger/commis) and gut a lot of implementation and/or add lots of ACLs and security checks. Hope this helps.

–Noah Kantrowitz