Programmatic creation/deletion of resources

Chef Infra (archive)
1

Hello,

I’m putting together a cookbook to manage nxlog (https://git.widgit.com/widgit-cookbooks/nxlog) which manages routing log data from one place to another. I’ve created some LWRPs for log sources and destinations. (nxlog_source, nxlog_destination)

I’d like to add a mechanism to automatically create log sources and destinations from node attributes. For example:
node[‘nxlog’][‘sources’] = [ { some_data }, { different_data } ]

I figured this could be easier in some cases than modifying existing recipes or creating new ones.

I’m not sure about the best way to go about this… The things I’m currently stuck on are:

  • How do I programatically create a new resource? i.e.

node[‘nxlog’][‘sources’].each { |source| create_source(source) }
– I’m not sure how I would implement create_source

  • How do I enumerate currently-extant resources so that I can delete ones that have been removed? (or… is this even possible?)

for example: If I’ve removed an element from the ‘sources’ array how can I ensure that the resource’s config file is cleaned up on the node?

Hope this makes sense… Thanks,
Simon

–
Simon Detheridge - CTO, Widgit Software
26 Queen Street, Cubbington, CV32 7NA - Tel: +44 (0)1926 333680

2

On Thursday, May 7, 2015 at 6:17 AM, Simon Detheridge wrote:

Hello,

I'm putting together a cookbook to manage nxlog (https://git.widgit.com/widgit-cookbooks/nxlog) which manages routing log data from one place to another. I've created some LWRPs for log sources and destinations. (nxlog_source, nxlog_destination)

I'd like to add a mechanism to automatically create log sources and destinations from node attributes. For example:
node['nxlog']['sources'] = [ { some_data }, { different_data } ]

I figured this could be easier in some cases than modifying existing recipes or creating new ones.

I'm not sure about the best way to go about this... The things I'm currently stuck on are:

  • How do I programatically create a new resource? i.e.

node['nxlog']['sources'].each { |source| create_source(source) }
-- I'm not sure how I would implement create_source

What do you imagine create_source does? Creates a resource with the resource attributes filled out from the node attributes? That can be accomplished by writing a ruby module with a method, like extend Chef Recipe DSL · GitHub

  • How do I enumerate currently-extant resources so that I can delete ones that have been removed? (or... is this even possible?)

for example: If I've removed an element from the 'sources' array how can I ensure that the resource's config file is cleaned up on the node?
What thing on the system represents the stuff that exists? Are these individual files (sounds like yes)? As long as there aren’t unmanaged files in that particular directory, you can use Ruby’s Dir.glob to enumerate the existing files and compare to the list of things you want, and remove the extra elements.

Hope this makes sense... Thanks,
Simon

--
Simon Detheridge - CTO, Widgit Software
26 Queen Street, Cubbington, CV32 7NA - Tel: +44 (0)1926 333680

HTH,

--
Daniel DeLeo

3

----- On 7 May, 2015, at 16:23, Daniel DeLeo dan@kallistec.com wrote:

What do you imagine create_source does? Creates a resource with the resource
attributes filled out from the node attributes? That can be accomplished by
writing a ruby module with a method, like
extend Chef Recipe DSL · GitHub

Thanks, that is very much what I'm looking for.

Follow-up: My resource has a fairly large number of potential attributes. Is there a way I can set them in a less DSL-like way? For example:

nxlog_source "name" do
{"hash": "with", "a": "bunch", "of": "pairs"}.each_pair do |k,v|
set_attribute k, v
end
end

as opposed to

resource_attr1 node_attrs["resource_attr1"]
...
resource_attr50 node_attrs["resource_attr50"]

This way I can not repeat myself and avoid having to maintain a file with another list of my resource attributes...

  • How do I enumerate currently-extant resources so that I can delete ones that
    have been removed? (or... is this even possible?)

for example: If I've removed an element from the 'sources' array how can I
ensure that the resource's config file is cleaned up on the node?

What thing on the system represents the stuff that exists? Are these individual
files (sounds like yes)? As long as there aren’t unmanaged files in that
particular directory, you can use Ruby’s Dir.glob to enumerate the existing
files and compare to the list of things you want, and remove the extra
elements.

Yeah, it's files in a conf.d. I think the issue that I'm grappling with is that let's say someone has used node attributes to create a source, but elsewhere someone has used a lwrp to create a different source in another recipe... How can I ensure I don't clean up the source that's been defined via the lwrp instead of the node attributes?

Thanks,

--
Simon Detheridge - CTO, Widgit Software
26 Queen Street, Cubbington, CV32 7NA - Tel: +44 (0)1926 333680

4

On Thursday, May 7, 2015 at 8:58 AM, Simon Detheridge wrote:

----- On 7 May, 2015, at 16:23, Daniel DeLeo dan@kallistec.com (mailto:dan@kallistec.com) wrote:

What do you imagine create_source does? Creates a resource with the resource
attributes filled out from the node attributes? That can be accomplished by
writing a ruby module with a method, like
extend Chef Recipe DSL · GitHub

Thanks, that is very much what I'm looking for.

Follow-up: My resource has a fairly large number of potential attributes. Is there a way I can set them in a less DSL-like way? For example:

nxlog_source "name" do
{"hash": "with", "a": "bunch", "of": "pairs"}.each_pair do |k,v|
set_attribute k, v
end
end

as opposed to

resource_attr1 node_attrs["resource_attr1"]
...
resource_attr50 node_attrs["resource_attr50"]

This way I can not repeat myself and avoid having to maintain a file with another list of my resource attributes...
You can use Ruby’s #public_send method to call methods that you don’t know the name of ahead of time. (There is also #send, but that can call private methods, so you shouldn’t use it unless you really have to).

  • How do I enumerate currently-extant resources so that I can delete ones that
    have been removed? (or... is this even possible?)

for example: If I've removed an element from the 'sources' array how can I
ensure that the resource's config file is cleaned up on the node?

What thing on the system represents the stuff that exists? Are these individual
files (sounds like yes)? As long as there aren’t unmanaged files in that
particular directory, you can use Ruby’s Dir.glob to enumerate the existing
files and compare to the list of things you want, and remove the extra
elements.

Yeah, it's files in a conf.d. I think the issue that I'm grappling with is that let's say someone has used node attributes to create a source, but elsewhere someone has used a lwrp to create a different source in another recipe... How can I ensure I don't clean up the source that's been defined via the lwrp instead of the node attributes?
You can’t do it perfectly in all cases, but if all changes to files in that dir occur via core chef resources (file, template, etc.) then you can inspect the resource collection to find orphaned files. The “zap” cookbook does this, you can maybe use it directly or read the code for inspiration: GitHub - nvwls/zap: Library cookbook for garbage collecting chef controlled resource sets.

Thanks,

--
Simon Detheridge - CTO, Widgit Software
26 Queen Street, Cubbington, CV32 7NA - Tel: +44 (0)1926 333680

--
Daniel DeLeo

5

Thanks, that is very much what I'm looking for.

Follow-up: My resource has a fairly large number of potential attributes. Is
there a way I can set them in a less DSL-like way? For example:

nxlog_source "name" do
{"hash": "with", "a": "bunch", "of": "pairs"}.each_pair do |k,v|
set_attribute k, v
end
end

as opposed to

resource_attr1 node_attrs["resource_attr1"]
...
resource_attr50 node_attrs["resource_attr50"]

This way I can not repeat myself and avoid having to maintain a file with
another list of my resource attributes...

You can use Ruby’s #public_send method to call methods that you don’t know the
name of ahead of time. (There is also #send, but that can call private methods,
so you shouldn’t use it unless you really have to).

Class: Object (Ruby 2.1.1)

That's great, thanks.

  • How do I enumerate currently-extant resources so that I can delete ones that
    have been removed? (or... is this even possible?)

for example: If I've removed an element from the 'sources' array how can I
ensure that the resource's config file is cleaned up on the node?

What thing on the system represents the stuff that exists? Are these individual
files (sounds like yes)? As long as there aren’t unmanaged files in that
particular directory, you can use Ruby’s Dir.glob to enumerate the existing
files and compare to the list of things you want, and remove the extra
elements.

Yeah, it's files in a conf.d. I think the issue that I'm grappling with is that
let's say someone has used node attributes to create a source, but elsewhere
someone has used a lwrp to create a different source in another recipe... How
can I ensure I don't clean up the source that's been defined via the lwrp
instead of the node attributes?

You can’t do it perfectly in all cases, but if all changes to files in that dir
occur via core chef resources (file, template, etc.) then you can inspect the
resource collection to find orphaned files. The “zap” cookbook does this, you
can maybe use it directly or read the code for inspiration:
GitHub - nvwls/zap: Library cookbook for garbage collecting chef controlled resource sets.

Perfect. Yes, it uses standard Chef templates which would become orphaned when the attributes are removed from the node. I'll take a look at 'zap'.

Thanks for all your help.

--
Simon Detheridge - CTO, Widgit Software
26 Queen Street, Cubbington, CV32 7NA - Tel: +44 (0)1926 333680