Chef deregistration via an external client


#1

Hi,

So I’m working on implementing a Chef deregistration mechanism for
nodes/clients in AWS autoscaling groups. Because of the lack of guarantees
around soft termination in AWS, I have an external node that’s intended to
handle ASG notifications (delivered via SNS to an SQS queue, it’s nothing
new).

The problem I’m running into is centered around authentication. In Chef
Manage (currently using Hosted Chef), I’ve created a client ‘dereg’,
arranged storage for its private key, so on and so forth. That’s fine. I’ve
since encountered an issue with Chef permissioning, in that it doesn’t
appear that I can say “this client is able to read and delete other
clients”; I can manually grant permissions through Chef Manage for 'dereg’
to be able to modify/delete other nodes and clients, but I can’t seem to
make this a blanket grant for that client. I can do that with a user, but
for obvious reasons I neither want to go create another Hosted Chef account
(and find an email address for it) nor increase my attack surface by
enabling destructive access via a password-based account.

Any advice on implementing this sanely and securely?

Thanks,
Ed


#2

I’m not 100% clear on your use case, but if you:

  1. Give dereg permissions to read and delete in the /containers/clients ACL
    (the
  2. Give dereg permissions to read and delete in each and every client under
    /clients/*

Doing #1 doesn’t automatically get you #1. If you need a tool to edit the
acls, knife edit can probably help: knife edit acls/containers/clients and
knife edit acls/clients/* will let you edit the individual permissions.

–John

On Mon, Feb 23, 2015 at 1:21 PM, Ed Ropple eropple@leaf.me wrote:

Hi,

So I’m working on implementing a Chef deregistration mechanism for
nodes/clients in AWS autoscaling groups. Because of the lack of guarantees
around soft termination in AWS, I have an external node that’s intended to
handle ASG notifications (delivered via SNS to an SQS queue, it’s nothing
new).

The problem I’m running into is centered around authentication. In Chef
Manage (currently using Hosted Chef), I’ve created a client ‘dereg’,
arranged storage for its private key, so on and so forth. That’s fine. I’ve
since encountered an issue with Chef permissioning, in that it doesn’t
appear that I can say “this client is able to read and delete other
clients”; I can manually grant permissions through Chef Manage for 'dereg’
to be able to modify/delete other nodes and clients, but I can’t seem to
make this a blanket grant for that client. I can do that with a user, but
for obvious reasons I neither want to go create another Hosted Chef account
(and find an email address for it) nor increase my attack surface by
enabling destructive access via a password-based account.

Any advice on implementing this sanely and securely?

Thanks,
Ed


#3

Hi,

My use case is pretty straightforward: I have an auto-scaling group
creating and destroying instances and I want to ensure that they are always
properly removed from Chef. I cannot rely on the auto-scaled instances to
de-register themselves because AWS does not in all cases allow an instance
to run any init.d or upstart jobs on system termination. But an
auto-scaling group can send an SNS notification (chaining to an SQS queue),
and so I’m trying–mostly failing at the moment–to have a "watchdog"
machine, equipped with Chef credentials, that can clean up the instances
(knife node delete and knife client delete) recorded as terminated in
this queue. I don’t much like this strategy for cleanup, but it seems like
my only option; suggestions, of course, are greatly welcomed.

The acls/containers path is a new one on me, but looks promising. I can’t
find much documentation for it; Google keeps feeding me dead or
domain-switched links to the Chef site and making navigation frustrating.
Am I correct in intuiting that these are the base permissions for all
objects of the given class, such that if I add dereg to .delete.actors in
acls/containers/clients.json that that will be honored across all newly
instantiated clients?

Thanks for your help–I greatly appreciate it.

-Ed

On Mon, Feb 23, 2015 at 6:57 PM, John Keiser jkeiser@chef.io wrote:

I’m not 100% clear on your use case, but if you:

  1. Give dereg permissions to read and delete in the /containers/clients
    ACL (the
  2. Give dereg permissions to read and delete in each and every client
    under /clients/*

Doing #1 doesn’t automatically get you #1. If you need a tool to edit the
acls, knife edit can probably help: knife edit acls/containers/clients and
knife edit acls/clients/* will let you edit the individual permissions.

–John

On Mon, Feb 23, 2015 at 1:21 PM, Ed Ropple eropple@leaf.me wrote:

Hi,

So I’m working on implementing a Chef deregistration mechanism for
nodes/clients in AWS autoscaling groups. Because of the lack of guarantees
around soft termination in AWS, I have an external node that’s intended to
handle ASG notifications (delivered via SNS to an SQS queue, it’s nothing
new).

The problem I’m running into is centered around authentication. In Chef
Manage (currently using Hosted Chef), I’ve created a client ‘dereg’,
arranged storage for its private key, so on and so forth. That’s fine. I’ve
since encountered an issue with Chef permissioning, in that it doesn’t
appear that I can say “this client is able to read and delete other
clients”; I can manually grant permissions through Chef Manage for 'dereg’
to be able to modify/delete other nodes and clients, but I can’t seem to
make this a blanket grant for that client. I can do that with a user, but
for obvious reasons I neither want to go create another Hosted Chef account
(and find an email address for it) nor increase my attack surface by
enabling destructive access via a password-based account.

Any advice on implementing this sanely and securely?

Thanks,
Ed


#4

Could you maybe tail the autoscale logs instead, and have a more HA-oriented system (a Jenkins cluster maybe) run the deletes?

Disclaimer: I haven’t used AWS in a bit.

On Feb 24, 2015, at 8:01 AM, Ed Ropple <eropple@leaf.memailto:eropple@leaf.me> wrote:

Hi,

My use case is pretty straightforward: I have an auto-scaling group creating and destroying instances and I want to ensure that they are always properly removed from Chef. I cannot rely on the auto-scaled instances to de-register themselves because AWS does not in all cases allow an instance to run any init.d or upstart jobs on system termination. But an auto-scaling group can send an SNS notification (chaining to an SQS queue), and so I’m trying–mostly failing at the moment–to have a “watchdog” machine, equipped with Chef credentials, that can clean up the instances (knife node delete and knife client delete) recorded as terminated in this queue. I don’t much like this strategy for cleanup, but it seems like my only option; suggestions, of course, are greatly welcomed.

The acls/containers path is a new one on me, but looks promising. I can’t find much documentation for it; Google keeps feeding me dead or domain-switched links to the Chef site and making navigation frustrating. Am I correct in intuiting that these are the base permissions for all objects of the given class, such that if I add dereg to .delete.actors in acls/containers/clients.json that that will be honored across all newly instantiated clients?

Thanks for your help–I greatly appreciate it.

-Ed

On Mon, Feb 23, 2015 at 6:57 PM, John Keiser <jkeiser@chef.iomailto:jkeiser@chef.io> wrote:
I’m not 100% clear on your use case, but if you:

  1. Give dereg permissions to read and delete in the /containers/clients ACL (the
  2. Give dereg permissions to read and delete in each and every client under /clients/*

Doing #1 doesn’t automatically get you #1. If you need a tool to edit the acls, knife edit can probably help: knife edit acls/containers/clients and knife edit acls/clients/* will let you edit the individual permissions.

–John

On Mon, Feb 23, 2015 at 1:21 PM, Ed Ropple <eropple@leaf.memailto:eropple@leaf.me> wrote:
Hi,

So I’m working on implementing a Chef deregistration mechanism for nodes/clients in AWS autoscaling groups. Because of the lack of guarantees around soft termination in AWS, I have an external node that’s intended to handle ASG notifications (delivered via SNS to an SQS queue, it’s nothing new).

The problem I’m running into is centered around authentication. In Chef Manage (currently using Hosted Chef), I’ve created a client ‘dereg’, arranged storage for its private key, so on and so forth. That’s fine. I’ve since encountered an issue with Chef permissioning, in that it doesn’t appear that I can say “this client is able to read and delete other clients”; I can manually grant permissions through Chef Manage for ‘dereg’ to be able to modify/delete other nodes and clients, but I can’t seem to make this a blanket grant for that client. I can do that with a user, but for obvious reasons I neither want to go create another Hosted Chef account (and find an email address for it) nor increase my attack surface by enabling destructive access via a password-based account.

Any advice on implementing this sanely and securely?

Thanks,
Ed


#5

John put me on the right track with editing the contents of
acls/containers - if there’s doc on it I couldn’t find it, but I was able
to get my Chef setup to do the Right Thing and enable a client to delete
nodes and clients all proper.

The infrastructure around ASG watching and management, including the Chef
deregistration task, is open-sourced at https://github.com/eropple/asger .

Thanks!
-Ed

On Wed, Feb 25, 2015 at 1:29 PM, Gottesman, Eric <eric.gottesman@wbgames.com

wrote:

Could you maybe tail the autoscale logs instead, and have a more
HA-oriented system (a Jenkins cluster maybe) run the deletes?

Disclaimer: I haven’t used AWS in a bit.

On Feb 24, 2015, at 8:01 AM, Ed Ropple eropple@leaf.me wrote:

Hi,

My use case is pretty straightforward: I have an auto-scaling group
creating and destroying instances and I want to ensure that they are always
properly removed from Chef. I cannot rely on the auto-scaled instances to
de-register themselves because AWS does not in all cases allow an instance
to run any init.d or upstart jobs on system termination. But an
auto-scaling group can send an SNS notification (chaining to an SQS queue),
and so I’m trying–mostly failing at the moment–to have a "watchdog"
machine, equipped with Chef credentials, that can clean up the instances
(knife node delete and knife client delete) recorded as terminated in
this queue. I don’t much like this strategy for cleanup, but it seems like
my only option; suggestions, of course, are greatly welcomed.

The acls/containers path is a new one on me, but looks promising. I
can’t find much documentation for it; Google keeps feeding me dead or
domain-switched links to the Chef site and making navigation frustrating.
Am I correct in intuiting that these are the base permissions for all
objects of the given class, such that if I add dereg to .delete.actors in
acls/containers/clients.json that that will be honored across all newly
instantiated clients?

Thanks for your help–I greatly appreciate it.

-Ed

On Mon, Feb 23, 2015 at 6:57 PM, John Keiser jkeiser@chef.io wrote:

I’m not 100% clear on your use case, but if you:

  1. Give dereg permissions to read and delete in the /containers/clients
    ACL (the
  2. Give dereg permissions to read and delete in each and every client
    under /clients/*

Doing #1 doesn’t automatically get you #1. If you need a tool to edit
the acls, knife edit can probably help: knife edit acls/containers/clients
and knife edit acls/clients/* will let you edit the individual permissions.

–John

On Mon, Feb 23, 2015 at 1:21 PM, Ed Ropple eropple@leaf.me wrote:

Hi,

So I’m working on implementing a Chef deregistration mechanism for
nodes/clients in AWS autoscaling groups. Because of the lack of guarantees
around soft termination in AWS, I have an external node that’s intended to
handle ASG notifications (delivered via SNS to an SQS queue, it’s nothing
new).

The problem I’m running into is centered around authentication. In Chef
Manage (currently using Hosted Chef), I’ve created a client ‘dereg’,
arranged storage for its private key, so on and so forth. That’s fine. I’ve
since encountered an issue with Chef permissioning, in that it doesn’t
appear that I can say “this client is able to read and delete other
clients”; I can manually grant permissions through Chef Manage for 'dereg’
to be able to modify/delete other nodes and clients, but I can’t seem to
make this a blanket grant for that client. I can do that with a user, but
for obvious reasons I neither want to go create another Hosted Chef account
(and find an email address for it) nor increase my attack surface by
enabling destructive access via a password-based account.

Any advice on implementing this sanely and securely?

Thanks,
Ed


#6

You have it right, containers is the default permissions for new objects :slight_smile:
It also doubles as permission to view the list of nodes and create new ones.
On Feb 24, 2015 8:01 AM, “Ed Ropple” eropple@leaf.me wrote:

Hi,

My use case is pretty straightforward: I have an auto-scaling group
creating and destroying instances and I want to ensure that they are always
properly removed from Chef. I cannot rely on the auto-scaled instances to
de-register themselves because AWS does not in all cases allow an instance
to run any init.d or upstart jobs on system termination. But an
auto-scaling group can send an SNS notification (chaining to an SQS queue),
and so I’m trying–mostly failing at the moment–to have a "watchdog"
machine, equipped with Chef credentials, that can clean up the instances
(knife node delete and knife client delete) recorded as terminated in
this queue. I don’t much like this strategy for cleanup, but it seems like
my only option; suggestions, of course, are greatly welcomed.

The acls/containers path is a new one on me, but looks promising. I
can’t find much documentation for it; Google keeps feeding me dead or
domain-switched links to the Chef site and making navigation frustrating.
Am I correct in intuiting that these are the base permissions for all
objects of the given class, such that if I add dereg to .delete.actors in
acls/containers/clients.json that that will be honored across all newly
instantiated clients?

Thanks for your help–I greatly appreciate it.

-Ed

On Mon, Feb 23, 2015 at 6:57 PM, John Keiser jkeiser@chef.io wrote:

I’m not 100% clear on your use case, but if you:

  1. Give dereg permissions to read and delete in the /containers/clients
    ACL (the
  2. Give dereg permissions to read and delete in each and every client
    under /clients/*

Doing #1 doesn’t automatically get you #1. If you need a tool to edit
the acls, knife edit can probably help: knife edit acls/containers/clients
and knife edit acls/clients/* will let you edit the individual permissions.

–John

On Mon, Feb 23, 2015 at 1:21 PM, Ed Ropple eropple@leaf.me wrote:

Hi,

So I’m working on implementing a Chef deregistration mechanism for
nodes/clients in AWS autoscaling groups. Because of the lack of guarantees
around soft termination in AWS, I have an external node that’s intended to
handle ASG notifications (delivered via SNS to an SQS queue, it’s nothing
new).

The problem I’m running into is centered around authentication. In Chef
Manage (currently using Hosted Chef), I’ve created a client ‘dereg’,
arranged storage for its private key, so on and so forth. That’s fine. I’ve
since encountered an issue with Chef permissioning, in that it doesn’t
appear that I can say “this client is able to read and delete other
clients”; I can manually grant permissions through Chef Manage for 'dereg’
to be able to modify/delete other nodes and clients, but I can’t seem to
make this a blanket grant for that client. I can do that with a user, but
for obvious reasons I neither want to go create another Hosted Chef account
(and find an email address for it) nor increase my attack surface by
enabling destructive access via a password-based account.

Any advice on implementing this sanely and securely?

Thanks,
Ed