Managing an ec2 instance and security group together with chef?

[Peter_Norton]

Hello,

I’m new to chef, but I’m trying it out as a way to manage amazon ec2
instances, and I’ve found some places that I think chef can help manage,
but which it can’t completely manage yet.

My first challenge comes from the fact that it seems like security groups
are inflexible, and can be limiting.

Specifically a use case that would be nice is to be able to create a new
system, and have it attached to a security group after it’s been allocated to
a service. Since services require ports to be opened, this would mean
changing security groups. However, since ec2 doesn’t allow for security
groups to change after a reservation is created, one way to get this
behavior would be to have a security group created per-host.

I think I see how to do this with knife, so I think I can add the necessary
knife ec2 command to create a group, and maybe tie the same function
into creating an instance to make my use case easier (1 command instead
of 2).

However, the second part that makes this useful is to have the host’s
security group be patterned off of an existing group, or off a configuration
that chef maintains. Either way, the important point is that changes to
that template reflected in all groups that are patterned off of it.

One issue with this is that I don’t see clearly how chef could be used to
manage a security group in the way I envision it should work. Here’s what I
imagine:

1. Knife creates a new security group using a pseudo-random name - let’s say
   ’sg-0xDEADBEEF’ for this example.
2. The instance’s reservation is created with two security groups:
   default,sg-0xDEADBEEF
3. The instance is created, and chef is bootstrapped
4. The host is created with an role called “available”. The "available"
   role includes the default security group’s settings, so these are cloned
   into sg-0xDEADBEEF’s settings and now it looks the same as default.
5. The server is needed an hour later, so it’s prepared for a deployment by
   being assigned the roles “http,https,tomcat-2121” which each include the
   security groups needed as well as the packages. The installation of apache,
   along with laying out conf files for port 80, port 443, and port 2121, the
   tomcat instance, java, etc. get pumped out, but each role also depends on an
   associated security group, let’s say sg-http, sg-https, and sg-tomcat-2121,
   which allow port 80, port 443, and port 2121 respectively.

If later on the tomcat-2121 also needs port 2122 for some reason then
port 2122 would be added to the recipe, and sg-0xDEADBEEF would have
that port added as part of the regular chef operation.

So… is it possible to do this with chef? If someone can point me to any
examples of similar situations (e.g. a configuration that is related to a host,
managed along with that host, but is not part of the host) that I
could use as a
starting point, I’d be quite willing to point my zero knowledge of ruby at this
problem, and use fog to make this happen.

Thanks,

-Peter

[Aaron_Abramson]

I would do it this way:

1. Launch EC2 servers with 3 security groups, default, chef, and unique (unique being unique for that server)

• Remember, your server has to allow ssh access from where you're working, and I prefer to keep the 'default' SG as amazon internal only, thus the 'chef' SG.

2. Make sure server has ec2 tools installed, and can update security group settings.

3. Update all cookbooks which install/configure services to open necessary ports in the 'unique' security group.

Now when you install your http server of choice, part of the server setup is to open up the ports used in the security group. You can store the 'unique' security group name as a node attribute.

I would suggest creating an EC2-controls cookbook, and make "sg-add" and "sg-del" LWRP's for use by your other cookbooks. That way you simply need to tell the apache cookbook:

sg-add "apache ports" do
group node[:ec2][:unique_group]
tcp_ports [ "80", "443" ]
udp_ports nil
end

This keeps it idempotent, only the packages/services which are installed get added to your security group. And you don't have to dink around with SG templates or anything else. Let the security group be built/configured 'on-the-fly'.

On Sep 16, 2011, at 2:22 PM, pn+chef-list@knewton.com wrote:

Hello,

I'm new to chef, but I'm trying it out as a way to manage amazon ec2
instances, and I've found some places that I think chef can help manage,
but which it can't completely manage yet.

My first challenge comes from the fact that it seems like security groups
are inflexible, and can be limiting.

Specifically a use case that would be nice is to be able to create a new
system, and have it attached to a security group after it's been allocated to
a service. Since services require ports to be opened, this would mean
changing security groups. However, since ec2 doesn't allow for security
groups to change after a reservation is created, one way to get this
behavior would be to have a security group created per-host.

I think I see how to do this with knife, so I think I can add the necessary
knife ec2 command to create a group, and maybe tie the same function
into creating an instance to make my use case easier (1 command instead
of 2).

However, the second part that makes this useful is to have the host's
security group be patterned off of an existing group, or off a configuration
that chef maintains. Either way, the important point is that changes to
that template reflected in all groups that are patterned off of it.

One issue with this is that I don't see clearly how chef could be used to
manage a security group in the way I envision it should work. Here's what I
imagine:

1. Knife creates a new security group using a pseudo-random name - let's say
   'sg-0xDEADBEEF' for this example.
2. The instance's reservation is created with two security groups:
   default,sg-0xDEADBEEF
3. The instance is created, and chef is bootstrapped
4. The host is created with an role called "available". The "available"
   role includes the default security group's settings, so these are cloned
   into sg-0xDEADBEEF's settings and now it looks the same as default.
5. The server is needed an hour later, so it's prepared for a deployment by
   being assigned the roles "http,https,tomcat-2121" which each include the
   security groups needed as well as the packages. The installation of apache,
   along with laying out conf files for port 80, port 443, and port 2121, the
   tomcat instance, java, etc. get pumped out, but each role also depends on an
   associated security group, let's say sg-http, sg-https, and sg-tomcat-2121,
   which allow port 80, port 443, and port 2121 respectively.

If later on the tomcat-2121 also needs port 2122 for some reason then
port 2122 would be added to the recipe, and sg-0xDEADBEEF would have
that port added as part of the regular chef operation.

So... is it possible to do this with chef? If someone can point me to any
examples of similar situations (e.g. a configuration that is related to a host,
managed along with that host, but is not part of the host) that I
could use as a
starting point, I'd be quite willing to point my zero knowledge of ruby at this
problem, and use fog to make this happen.

Thanks,

-Peter

[Peter_Norton]

Questions in-line

On Fri, Sep 16, 2011 at 4:02 PM, Aaron Abramson aabramson@wi-figuys.com wrote:

I would do it this way:

1. Launch EC2 servers with 3 security groups, default, chef, and unique (unique being unique for that server)

• Remember, your server has to allow ssh access from where you're working, and I prefer to keep the 'default' SG as amazon internal only, thus the 'chef' SG.

Good point. I was using default as a shorthand for "the things
everything needs", but it should be "default,common,sg-0xDEADBEEF"
where "default" should be everything off, and "common" should include
anything that all servers need (ssh being the most likely, or ssh on
an alternate port for example).

2. Make sure server has ec2 tools installed, and can update security group settings.

Ec2 tools? Would knife (with modules to handle security group
manipulations) be sufficient? I'm pretty sure the AMIs we use include
the tools, but they're slooooow.

3. Update all cookbooks which install/configure services to open necessary ports in the 'unique' security group.

So once we decide on our group manipulation tool, use that tool in the
recipes? OK.

Now when you install your http server of choice, part of the server setup is to open up the ports used in the security group. You can store the 'unique' security group name as a node attribute.

I would suggest creating an EC2-controls cookbook, and make "sg-add" and "sg-del" LWRP's for use by your other cookbooks. That way you simply need to tell the apache cookbook:

sg-add "apache ports" do
group node[:ec2][:unique_group]
tcp_ports [ "80", "443" ]
udp_ports nil
end

This keeps it idempotent, only the packages/services which are installed get added to your security group. And you don't have to dink around with SG templates or anything else. Let the security group be built/configured 'on-the-fly'.

Thanks for the input, I think that clarifies that this is possible.
One thing concerns me, though, and that is that this style of using
EC2 does require ec2 api keys be installed on every host, and that
means that if any host is compromised then the entire cloud can be
removed. If this could be managed centrally then that reduces the
proliferation of keys. Is there a method to proxy this sort of
meta-server management on either a second host, or on the chef server?

-Peter

On Sep 16, 2011, at 2:22 PM, pn+chef-list@knewton.com wrote:

Hello,

I'm new to chef, but I'm trying it out as a way to manage amazon ec2
instances, and I've found some places that I think chef can help manage,
but which it can't completely manage yet.

My first challenge comes from the fact that it seems like security groups
are inflexible, and can be limiting.

Specifically a use case that would be nice is to be able to create a new
system, and have it attached to a security group after it's been allocated to
a service. Since services require ports to be opened, this would mean
changing security groups. However, since ec2 doesn't allow for security
groups to change after a reservation is created, one way to get this
behavior would be to have a security group created per-host.

I think I see how to do this with knife, so I think I can add the necessary
knife ec2 command to create a group, and maybe tie the same function
into creating an instance to make my use case easier (1 command instead
of 2).

However, the second part that makes this useful is to have the host's
security group be patterned off of an existing group, or off a configuration
that chef maintains. Either way, the important point is that changes to
that template reflected in all groups that are patterned off of it.

One issue with this is that I don't see clearly how chef could be used to
manage a security group in the way I envision it should work. Here's what I
imagine:

1. Knife creates a new security group using a pseudo-random name - let's say
   'sg-0xDEADBEEF' for this example.
2. The instance's reservation is created with two security groups:
   default,sg-0xDEADBEEF
3. The instance is created, and chef is bootstrapped
4. The host is created with an role called "available". The "available"
   role includes the default security group's settings, so these are cloned
   into sg-0xDEADBEEF's settings and now it looks the same as default.
5. The server is needed an hour later, so it's prepared for a deployment by
   being assigned the roles "http,https,tomcat-2121" which each include the
   security groups needed as well as the packages. The installation of apache,
   along with laying out conf files for port 80, port 443, and port 2121, the
   tomcat instance, java, etc. get pumped out, but each role also depends on an
   associated security group, let's say sg-http, sg-https, and sg-tomcat-2121,
   which allow port 80, port 443, and port 2121 respectively.

If later on the tomcat-2121 also needs port 2122 for some reason then
port 2122 would be added to the recipe, and sg-0xDEADBEEF would have
that port added as part of the regular chef operation.

So... is it possible to do this with chef? If someone can point me to any
examples of similar situations (e.g. a configuration that is related to a host,
managed along with that host, but is not part of the host) that I
could use as a
starting point, I'd be quite willing to point my zero knowledge of ruby at this
problem, and use fog to make this happen.

Thanks,

-Peter

[Aaron_Abramson]

Ec2 tools? Would knife (with modules to handle security group
manipulations) be sufficient? I'm pretty sure the AMIs we use include
the tools, but they're slooooow.

You would need the commandline tools to be installed on the instance itself. Knife is being run from your local machine, so your knife ec2 commands are run locally, and can't be called when chef-client is run on the remote instance.

When chef-client runs on the particular instance, the recipes need to execute the ec2 command line tools there. Unless you're installing knife, and all of the requisite tools and key files on each instance, having a cookbook run a knife command is going to be inefficient.

If you build your "contoller" cookbook around the standard ec2 api java tools (or a particular set of tools like the python based ones), then simply have that cookbook set up the tools, set your api credentials, and be a dependency of every other cookbook, or part of your base role.

On Sep 16, 2011, at 3:22 PM, Peter Norton wrote:

Questions in-line

On Fri, Sep 16, 2011 at 4:02 PM, Aaron Abramson aabramson@wi-figuys.com wrote:

I would do it this way:

1. Launch EC2 servers with 3 security groups, default, chef, and unique (unique being unique for that server)

• Remember, your server has to allow ssh access from where you're working, and I prefer to keep the 'default' SG as amazon internal only, thus the 'chef' SG.

Good point. I was using default as a shorthand for "the things
everything needs", but it should be "default,common,sg-0xDEADBEEF"
where "default" should be everything off, and "common" should include
anything that all servers need (ssh being the most likely, or ssh on
an alternate port for example).

2. Make sure server has ec2 tools installed, and can update security group settings.

Ec2 tools? Would knife (with modules to handle security group
manipulations) be sufficient? I'm pretty sure the AMIs we use include
the tools, but they're slooooow.

3. Update all cookbooks which install/configure services to open necessary ports in the 'unique' security group.

So once we decide on our group manipulation tool, use that tool in the
recipes? OK.

Now when you install your http server of choice, part of the server setup is to open up the ports used in the security group. You can store the 'unique' security group name as a node attribute.

I would suggest creating an EC2-controls cookbook, and make "sg-add" and "sg-del" LWRP's for use by your other cookbooks. That way you simply need to tell the apache cookbook:

sg-add "apache ports" do
group node[:ec2][:unique_group]
tcp_ports [ "80", "443" ]
udp_ports nil
end

This keeps it idempotent, only the packages/services which are installed get added to your security group. And you don't have to dink around with SG templates or anything else. Let the security group be built/configured 'on-the-fly'.

Thanks for the input, I think that clarifies that this is possible.
One thing concerns me, though, and that is that this style of using
EC2 does require ec2 api keys be installed on every host, and that
means that if any host is compromised then the entire cloud can be
removed. If this could be managed centrally then that reduces the
proliferation of keys. Is there a method to proxy this sort of
meta-server management on either a second host, or on the chef server?

-Peter

On Sep 16, 2011, at 2:22 PM, pn+chef-list@knewton.com wrote:

Hello,

I'm new to chef, but I'm trying it out as a way to manage amazon ec2
instances, and I've found some places that I think chef can help manage,
but which it can't completely manage yet.

My first challenge comes from the fact that it seems like security groups
are inflexible, and can be limiting.

Specifically a use case that would be nice is to be able to create a new
system, and have it attached to a security group after it's been allocated to
a service. Since services require ports to be opened, this would mean
changing security groups. However, since ec2 doesn't allow for security
groups to change after a reservation is created, one way to get this
behavior would be to have a security group created per-host.

I think I see how to do this with knife, so I think I can add the necessary
knife ec2 command to create a group, and maybe tie the same function
into creating an instance to make my use case easier (1 command instead
of 2).

However, the second part that makes this useful is to have the host's
security group be patterned off of an existing group, or off a configuration
that chef maintains. Either way, the important point is that changes to
that template reflected in all groups that are patterned off of it.

One issue with this is that I don't see clearly how chef could be used to
manage a security group in the way I envision it should work. Here's what I
imagine:

1. Knife creates a new security group using a pseudo-random name - let's say
   'sg-0xDEADBEEF' for this example.
2. The instance's reservation is created with two security groups:
   default,sg-0xDEADBEEF
3. The instance is created, and chef is bootstrapped
4. The host is created with an role called "available". The "available"
   role includes the default security group's settings, so these are cloned
   into sg-0xDEADBEEF's settings and now it looks the same as default.
5. The server is needed an hour later, so it's prepared for a deployment by
   being assigned the roles "http,https,tomcat-2121" which each include the
   security groups needed as well as the packages. The installation of apache,
   along with laying out conf files for port 80, port 443, and port 2121, the
   tomcat instance, java, etc. get pumped out, but each role also depends on an
   associated security group, let's say sg-http, sg-https, and sg-tomcat-2121,
   which allow port 80, port 443, and port 2121 respectively.

If later on the tomcat-2121 also needs port 2122 for some reason then
port 2122 would be added to the recipe, and sg-0xDEADBEEF would have
that port added as part of the regular chef operation.

So... is it possible to do this with chef? If someone can point me to any
examples of similar situations (e.g. a configuration that is related to a host,
managed along with that host, but is not part of the host) that I
could use as a
starting point, I'd be quite willing to point my zero knowledge of ruby at this
problem, and use fog to make this happen.

Thanks,

-Peter

[Aaron_Abramson]

Thanks for the input, I think that clarifies that this is possible.
One thing concerns me, though, and that is that this style of using
EC2 does require ec2 api keys be installed on every host, and that
means that if any host is compromised then the entire cloud can be
removed. If this could be managed centrally then that reduces the
proliferation of keys. Is there a method to proxy this sort of
meta-server management on either a second host, or on the chef server?

I forgot to address this part...

Write your controller cookbook to run the ec2 commandline tools giving the api-credentials at the commandline, and not through an environment variable. Then store your api-keys in a data bag, and use them in the cookbook when running your script resource that executes the ec2 tools.

Use amazon's IAM to create a user that ONLY has access to what you need, and store THAT api key/secret id in the databag.

On Sep 16, 2011, at 3:22 PM, Peter Norton wrote:

Questions in-line

On Fri, Sep 16, 2011 at 4:02 PM, Aaron Abramson aabramson@wi-figuys.com wrote:

I would do it this way:

1. Launch EC2 servers with 3 security groups, default, chef, and unique (unique being unique for that server)

• Remember, your server has to allow ssh access from where you're working, and I prefer to keep the 'default' SG as amazon internal only, thus the 'chef' SG.

Good point. I was using default as a shorthand for "the things
everything needs", but it should be "default,common,sg-0xDEADBEEF"
where "default" should be everything off, and "common" should include
anything that all servers need (ssh being the most likely, or ssh on
an alternate port for example).

2. Make sure server has ec2 tools installed, and can update security group settings.

Ec2 tools? Would knife (with modules to handle security group
manipulations) be sufficient? I'm pretty sure the AMIs we use include
the tools, but they're slooooow.

3. Update all cookbooks which install/configure services to open necessary ports in the 'unique' security group.

So once we decide on our group manipulation tool, use that tool in the
recipes? OK.

Now when you install your http server of choice, part of the server setup is to open up the ports used in the security group. You can store the 'unique' security group name as a node attribute.

I would suggest creating an EC2-controls cookbook, and make "sg-add" and "sg-del" LWRP's for use by your other cookbooks. That way you simply need to tell the apache cookbook:

sg-add "apache ports" do
group node[:ec2][:unique_group]
tcp_ports [ "80", "443" ]
udp_ports nil
end

This keeps it idempotent, only the packages/services which are installed get added to your security group. And you don't have to dink around with SG templates or anything else. Let the security group be built/configured 'on-the-fly'.

Thanks for the input, I think that clarifies that this is possible.
One thing concerns me, though, and that is that this style of using
EC2 does require ec2 api keys be installed on every host, and that
means that if any host is compromised then the entire cloud can be
removed. If this could be managed centrally then that reduces the
proliferation of keys. Is there a method to proxy this sort of
meta-server management on either a second host, or on the chef server?

-Peter

On Sep 16, 2011, at 2:22 PM, pn+chef-list@knewton.com wrote:

Hello,

I'm new to chef, but I'm trying it out as a way to manage amazon ec2
instances, and I've found some places that I think chef can help manage,
but which it can't completely manage yet.

My first challenge comes from the fact that it seems like security groups
are inflexible, and can be limiting.

Specifically a use case that would be nice is to be able to create a new
system, and have it attached to a security group after it's been allocated to
a service. Since services require ports to be opened, this would mean
changing security groups. However, since ec2 doesn't allow for security
groups to change after a reservation is created, one way to get this
behavior would be to have a security group created per-host.

I think I see how to do this with knife, so I think I can add the necessary
knife ec2 command to create a group, and maybe tie the same function
into creating an instance to make my use case easier (1 command instead
of 2).

However, the second part that makes this useful is to have the host's
security group be patterned off of an existing group, or off a configuration
that chef maintains. Either way, the important point is that changes to
that template reflected in all groups that are patterned off of it.

One issue with this is that I don't see clearly how chef could be used to
manage a security group in the way I envision it should work. Here's what I
imagine:

1. Knife creates a new security group using a pseudo-random name - let's say
   'sg-0xDEADBEEF' for this example.
2. The instance's reservation is created with two security groups:
   default,sg-0xDEADBEEF
3. The instance is created, and chef is bootstrapped
4. The host is created with an role called "available". The "available"
   role includes the default security group's settings, so these are cloned
   into sg-0xDEADBEEF's settings and now it looks the same as default.
5. The server is needed an hour later, so it's prepared for a deployment by
   being assigned the roles "http,https,tomcat-2121" which each include the
   security groups needed as well as the packages. The installation of apache,
   along with laying out conf files for port 80, port 443, and port 2121, the
   tomcat instance, java, etc. get pumped out, but each role also depends on an
   associated security group, let's say sg-http, sg-https, and sg-tomcat-2121,
   which allow port 80, port 443, and port 2121 respectively.

If later on the tomcat-2121 also needs port 2122 for some reason then
port 2122 would be added to the recipe, and sg-0xDEADBEEF would have
that port added as part of the regular chef operation.

So... is it possible to do this with chef? If someone can point me to any
examples of similar situations (e.g. a configuration that is related to a host,
managed along with that host, but is not part of the host) that I
could use as a
starting point, I'd be quite willing to point my zero knowledge of ruby at this
problem, and use fog to make this happen.

Thanks,

-Peter

[Peter_Norton]

OK, so my api key proliferation issue may be better solved by creating
an IAM user for each host with permission to manipulate just the
instance's unique group. It looks like python's boto library supports
IAM already, so the tool to do the mapping of the correct IAM user to
the host+group should be pretty simple.

-Peter

On Fri, Sep 16, 2011 at 4:34 PM, Aaron Abramson aabramson@wi-figuys.com wrote:

Ec2 tools? Would knife (with modules to handle security group
manipulations) be sufficient? I'm pretty sure the AMIs we use include
the tools, but they're slooooow.

You would need the commandline tools to be installed on the instance itself. Knife is being run from your local machine, so your knife ec2 commands are run locally, and can't be called when chef-client is run on the remote instance.

When chef-client runs on the particular instance, the recipes need to execute the ec2 command line tools there. Unless you're installing knife, and all of the requisite tools and key files on each instance, having a cookbook run a knife command is going to be inefficient.

If you build your "contoller" cookbook around the standard ec2 api java tools (or a particular set of tools like the python based ones), then simply have that cookbook set up the tools, set your api credentials, and be a dependency of every other cookbook, or part of your base role.

On Sep 16, 2011, at 3:22 PM, Peter Norton wrote:

Questions in-line

On Fri, Sep 16, 2011 at 4:02 PM, Aaron Abramson aabramson@wi-figuys.com wrote:

I would do it this way:

1. Launch EC2 servers with 3 security groups, default, chef, and unique (unique being unique for that server)

• Remember, your server has to allow ssh access from where you're working, and I prefer to keep the 'default' SG as amazon internal only, thus the 'chef' SG.

Good point. I was using default as a shorthand for "the things
everything needs", but it should be "default,common,sg-0xDEADBEEF"
where "default" should be everything off, and "common" should include
anything that all servers need (ssh being the most likely, or ssh on
an alternate port for example).

2. Make sure server has ec2 tools installed, and can update security group settings.

Ec2 tools? Would knife (with modules to handle security group
manipulations) be sufficient? I'm pretty sure the AMIs we use include
the tools, but they're slooooow.

3. Update all cookbooks which install/configure services to open necessary ports in the 'unique' security group.

So once we decide on our group manipulation tool, use that tool in the
recipes? OK.

Now when you install your http server of choice, part of the server setup is to open up the ports used in the security group. You can store the 'unique' security group name as a node attribute.

I would suggest creating an EC2-controls cookbook, and make "sg-add" and "sg-del" LWRP's for use by your other cookbooks. That way you simply need to tell the apache cookbook:

sg-add "apache ports" do
group node[:ec2][:unique_group]
tcp_ports [ "80", "443" ]
udp_ports nil
end

This keeps it idempotent, only the packages/services which are installed get added to your security group. And you don't have to dink around with SG templates or anything else. Let the security group be built/configured 'on-the-fly'.

Thanks for the input, I think that clarifies that this is possible.
One thing concerns me, though, and that is that this style of using
EC2 does require ec2 api keys be installed on every host, and that
means that if any host is compromised then the entire cloud can be
removed. If this could be managed centrally then that reduces the
proliferation of keys. Is there a method to proxy this sort of
meta-server management on either a second host, or on the chef server?

-Peter

On Sep 16, 2011, at 2:22 PM, pn+chef-list@knewton.com wrote:

Hello,

I'm new to chef, but I'm trying it out as a way to manage amazon ec2
instances, and I've found some places that I think chef can help manage,
but which it can't completely manage yet.

My first challenge comes from the fact that it seems like security groups
are inflexible, and can be limiting.

Specifically a use case that would be nice is to be able to create a new
system, and have it attached to a security group after it's been allocated to
a service. Since services require ports to be opened, this would mean
changing security groups. However, since ec2 doesn't allow for security
groups to change after a reservation is created, one way to get this
behavior would be to have a security group created per-host.

I think I see how to do this with knife, so I think I can add the necessary
knife ec2 command to create a group, and maybe tie the same function
into creating an instance to make my use case easier (1 command instead
of 2).

However, the second part that makes this useful is to have the host's
security group be patterned off of an existing group, or off a configuration
that chef maintains. Either way, the important point is that changes to
that template reflected in all groups that are patterned off of it.

One issue with this is that I don't see clearly how chef could be used to
manage a security group in the way I envision it should work. Here's what I
imagine:

1. Knife creates a new security group using a pseudo-random name - let's say
   'sg-0xDEADBEEF' for this example.
2. The instance's reservation is created with two security groups:
   default,sg-0xDEADBEEF
3. The instance is created, and chef is bootstrapped
4. The host is created with an role called "available". The "available"
   role includes the default security group's settings, so these are cloned
   into sg-0xDEADBEEF's settings and now it looks the same as default.
5. The server is needed an hour later, so it's prepared for a deployment by
   being assigned the roles "http,https,tomcat-2121" which each include the
   security groups needed as well as the packages. The installation of apache,
   along with laying out conf files for port 80, port 443, and port 2121, the
   tomcat instance, java, etc. get pumped out, but each role also depends on an
   associated security group, let's say sg-http, sg-https, and sg-tomcat-2121,
   which allow port 80, port 443, and port 2121 respectively.

If later on the tomcat-2121 also needs port 2122 for some reason then
port 2122 would be added to the recipe, and sg-0xDEADBEEF would have
that port added as part of the regular chef operation.

So... is it possible to do this with chef? If someone can point me to any
examples of similar situations (e.g. a configuration that is related to a host,
managed along with that host, but is not part of the host) that I
could use as a
starting point, I'd be quite willing to point my zero knowledge of ruby at this
problem, and use fog to make this happen.

Thanks,

-Peter

[Peter_Norton]

Here's the problem:

Use amazon's IAM to create a user that ONLY has access to what you need, and store THAT api key/secret id in the databag.

IAM can't resolve to the granularity of only granting access to a
user/group to operate on a specifc security group. It appears that as
long as that's the case, the right thing to do is to create the
security group, but stash the configuration template for the group
into revision control and have chef take care of applying
configuration to all instances' security groups from a central
location when changes need to be made.

-Peter

On Fri, Sep 16, 2011 at 4:37 PM, Aaron Abramson aabramson@wi-figuys.com wrote:

Thanks for the input, I think that clarifies that this is possible.
One thing concerns me, though, and that is that this style of using
EC2 does require ec2 api keys be installed on every host, and that
means that if any host is compromised then the entire cloud can be
removed. If this could be managed centrally then that reduces the
proliferation of keys. Is there a method to proxy this sort of
meta-server management on either a second host, or on the chef server?

I forgot to address this part...

Write your controller cookbook to run the ec2 commandline tools giving the api-credentials at the commandline, and not through an environment variable. Then store your api-keys in a data bag, and use them in the cookbook when running your script resource that executes the ec2 tools.

Use amazon's IAM to create a user that ONLY has access to what you need, and store THAT api key/secret id in the databag.

On Sep 16, 2011, at 3:22 PM, Peter Norton wrote:

Questions in-line

On Fri, Sep 16, 2011 at 4:02 PM, Aaron Abramson aabramson@wi-figuys.com wrote:

I would do it this way:

1. Launch EC2 servers with 3 security groups, default, chef, and unique (unique being unique for that server)

• Remember, your server has to allow ssh access from where you're working, and I prefer to keep the 'default' SG as amazon internal only, thus the 'chef' SG.

Good point. I was using default as a shorthand for "the things
everything needs", but it should be "default,common,sg-0xDEADBEEF"
where "default" should be everything off, and "common" should include
anything that all servers need (ssh being the most likely, or ssh on
an alternate port for example).

2. Make sure server has ec2 tools installed, and can update security group settings.

Ec2 tools? Would knife (with modules to handle security group
manipulations) be sufficient? I'm pretty sure the AMIs we use include
the tools, but they're slooooow.

3. Update all cookbooks which install/configure services to open necessary ports in the 'unique' security group.

So once we decide on our group manipulation tool, use that tool in the
recipes? OK.

Now when you install your http server of choice, part of the server setup is to open up the ports used in the security group. You can store the 'unique' security group name as a node attribute.

I would suggest creating an EC2-controls cookbook, and make "sg-add" and "sg-del" LWRP's for use by your other cookbooks. That way you simply need to tell the apache cookbook:

sg-add "apache ports" do
group node[:ec2][:unique_group]
tcp_ports [ "80", "443" ]
udp_ports nil
end

This keeps it idempotent, only the packages/services which are installed get added to your security group. And you don't have to dink around with SG templates or anything else. Let the security group be built/configured 'on-the-fly'.

Thanks for the input, I think that clarifies that this is possible.
One thing concerns me, though, and that is that this style of using
EC2 does require ec2 api keys be installed on every host, and that
means that if any host is compromised then the entire cloud can be
removed. If this could be managed centrally then that reduces the
proliferation of keys. Is there a method to proxy this sort of
meta-server management on either a second host, or on the chef server?

-Peter

On Sep 16, 2011, at 2:22 PM, pn+chef-list@knewton.com wrote:

Hello,

I'm new to chef, but I'm trying it out as a way to manage amazon ec2
instances, and I've found some places that I think chef can help manage,
but which it can't completely manage yet.

My first challenge comes from the fact that it seems like security groups
are inflexible, and can be limiting.

Specifically a use case that would be nice is to be able to create a new
system, and have it attached to a security group after it's been allocated to
a service. Since services require ports to be opened, this would mean
changing security groups. However, since ec2 doesn't allow for security
groups to change after a reservation is created, one way to get this
behavior would be to have a security group created per-host.

I think I see how to do this with knife, so I think I can add the necessary
knife ec2 command to create a group, and maybe tie the same function
into creating an instance to make my use case easier (1 command instead
of 2).

However, the second part that makes this useful is to have the host's
security group be patterned off of an existing group, or off a configuration
that chef maintains. Either way, the important point is that changes to
that template reflected in all groups that are patterned off of it.

One issue with this is that I don't see clearly how chef could be used to
manage a security group in the way I envision it should work. Here's what I
imagine:

1. Knife creates a new security group using a pseudo-random name - let's say
   'sg-0xDEADBEEF' for this example.
2. The instance's reservation is created with two security groups:
   default,sg-0xDEADBEEF
3. The instance is created, and chef is bootstrapped
4. The host is created with an role called "available". The "available"
   role includes the default security group's settings, so these are cloned
   into sg-0xDEADBEEF's settings and now it looks the same as default.
5. The server is needed an hour later, so it's prepared for a deployment by
   being assigned the roles "http,https,tomcat-2121" which each include the
   security groups needed as well as the packages. The installation of apache,
   along with laying out conf files for port 80, port 443, and port 2121, the
   tomcat instance, java, etc. get pumped out, but each role also depends on an
   associated security group, let's say sg-http, sg-https, and sg-tomcat-2121,
   which allow port 80, port 443, and port 2121 respectively.

If later on the tomcat-2121 also needs port 2122 for some reason then
port 2122 would be added to the recipe, and sg-0xDEADBEEF would have
that port added as part of the regular chef operation.

So... is it possible to do this with chef? If someone can point me to any
examples of similar situations (e.g. a configuration that is related to a host,
managed along with that host, but is not part of the host) that I
could use as a
starting point, I'd be quite willing to point my zero knowledge of ruby at this
problem, and use fog to make this happen.

Thanks,

-Peter

[James]

Just want to point out that instead of working around all of these issues,
you could simply use something other than security groups. This approach
would be vendor agnostic, and allow you to transition off of EC2 if you ever
need to.

James

On Fri, Sep 16, 2011 at 3:02 PM, Peter Norton pn+chef-list@knewton.comwrote:

Here's the problem:

Use amazon's IAM to create a user that ONLY has access to what you need,
and store THAT api key/secret id in the databag.

IAM can't resolve to the granularity of only granting access to a
user/group to operate on a specifc security group. It appears that as
long as that's the case, the right thing to do is to create the
security group, but stash the configuration template for the group
into revision control and have chef take care of applying
configuration to all instances' security groups from a central
location when changes need to be made.

-Peter

On Fri, Sep 16, 2011 at 4:37 PM, Aaron Abramson aabramson@wi-figuys.com
wrote:

Thanks for the input, I think that clarifies that this is possible.
One thing concerns me, though, and that is that this style of using
EC2 does require ec2 api keys be installed on every host, and that
means that if any host is compromised then the entire cloud can be
removed. If this could be managed centrally then that reduces the
proliferation of keys. Is there a method to proxy this sort of
meta-server management on either a second host, or on the chef server?

I forgot to address this part...

Write your controller cookbook to run the ec2 commandline tools giving
the api-credentials at the commandline, and not through an environment
variable. Then store your api-keys in a data bag, and use them in the
cookbook when running your script resource that executes the ec2 tools.

Use amazon's IAM to create a user that ONLY has access to what you need,
and store THAT api key/secret id in the databag.

On Sep 16, 2011, at 3:22 PM, Peter Norton wrote:

Questions in-line

On Fri, Sep 16, 2011 at 4:02 PM, Aaron Abramson <
aabramson@wi-figuys.com> wrote:

I would do it this way:

1. Launch EC2 servers with 3 security groups, default, chef, and unique
   (unique being unique for that server)

• Remember, your server has to allow ssh access from where you're
  working, and I prefer to keep the 'default' SG as amazon internal only, thus
  the 'chef' SG.

Good point. I was using default as a shorthand for "the things
everything needs", but it should be "default,common,sg-0xDEADBEEF"
where "default" should be everything off, and "common" should include
anything that all servers need (ssh being the most likely, or ssh on
an alternate port for example).

2. Make sure server has ec2 tools installed, and can update security
   group settings.

Ec2 tools? Would knife (with modules to handle security group
manipulations) be sufficient? I'm pretty sure the AMIs we use include
the tools, but they're slooooow.

3. Update all cookbooks which install/configure services to open
   necessary ports in the 'unique' security group.

So once we decide on our group manipulation tool, use that tool in the
recipes? OK.

Now when you install your http server of choice, part of the server
setup is to open up the ports used in the security group. You can store
the 'unique' security group name as a node attribute.

I would suggest creating an EC2-controls cookbook, and make "sg-add"
and "sg-del" LWRP's for use by your other cookbooks. That way you simply
need to tell the apache cookbook:

sg-add "apache ports" do
group node[:ec2][:unique_group]
tcp_ports [ "80", "443" ]
udp_ports nil
end

This keeps it idempotent, only the packages/services which are
installed get added to your security group. And you don't have to dink
around with SG templates or anything else. Let the security group be
built/configured 'on-the-fly'.

Thanks for the input, I think that clarifies that this is possible.
One thing concerns me, though, and that is that this style of using
EC2 does require ec2 api keys be installed on every host, and that
means that if any host is compromised then the entire cloud can be
removed. If this could be managed centrally then that reduces the
proliferation of keys. Is there a method to proxy this sort of
meta-server management on either a second host, or on the chef server?

-Peter

On Sep 16, 2011, at 2:22 PM, pn+chef-list@knewton.com wrote:

Hello,

I'm new to chef, but I'm trying it out as a way to manage amazon ec2
instances, and I've found some places that I think chef can help
manage,
but which it can't completely manage yet.

My first challenge comes from the fact that it seems like security
groups
are inflexible, and can be limiting.

Specifically a use case that would be nice is to be able to create a
new
system, and have it attached to a security group after it's been
allocated to
a service. Since services require ports to be opened, this would mean
changing security groups. However, since ec2 doesn't allow for
security
groups to change after a reservation is created, one way to get this
behavior would be to have a security group created per-host.

I think I see how to do this with knife, so I think I can add the
necessary
knife ec2 command to create a group, and maybe tie the same function
into creating an instance to make my use case easier (1 command
instead
of 2).

However, the second part that makes this useful is to have the host's
security group be patterned off of an existing group, or off a
configuration
that chef maintains. Either way, the important point is that changes
to
that template reflected in all groups that are patterned off of it.

One issue with this is that I don't see clearly how chef could be used
to
manage a security group in the way I envision it should work. Here's
what I
imagine:

1. Knife creates a new security group using a pseudo-random name -
   let's say
   'sg-0xDEADBEEF' for this example.
2. The instance's reservation is created with two security groups:
   default,sg-0xDEADBEEF
3. The instance is created, and chef is bootstrapped
4. The host is created with an role called "available". The
   "available"
   role includes the default security group's settings, so these are
   cloned
   into sg-0xDEADBEEF's settings and now it looks the same as default.
5. The server is needed an hour later, so it's prepared for a
   deployment by
   being assigned the roles "http,https,tomcat-2121" which each include
   the
   security groups needed as well as the packages. The installation of
   apache,
   along with laying out conf files for port 80, port 443, and port 2121,
   the
   tomcat instance, java, etc. get pumped out, but each role also depends
   on an
   associated security group, let's say sg-http, sg-https, and
   sg-tomcat-2121,
   which allow port 80, port 443, and port 2121 respectively.

If later on the tomcat-2121 also needs port 2122 for some reason then
port 2122 would be added to the recipe, and sg-0xDEADBEEF would have
that port added as part of the regular chef operation.

So... is it possible to do this with chef? If someone can point me to
any
examples of similar situations (e.g. a configuration that is related
to a host,
managed along with that host, but is not part of the host) that I
could use as a
starting point, I'd be quite willing to point my zero knowledge of
ruby at this
problem, and use fog to make this happen.

Thanks,

-Peter

[John_E_Vincent_lusis]

On Fri, Sep 16, 2011 at 8:20 PM, James js@aegisco.com wrote:

Just want to point out that instead of working around all of these issues,
you could simply use something other than security groups. This approach
would be vendor agnostic, and allow you to transition off of EC2 if you ever
need to.

James

I was considering doing an EC2 security-group LWRP till I heard
rumblings of a firewall cookbook that managed not only iptables but
security groups and other forms of "firewall"ing.

For what it's worth, all this talk of jclouds and boto and shelling
out is confusing. Amazon has a perfectly good ruby AWS SDK. Fog
exists. You'll get +9000x greater flexibility doing it in an LWRP or
as a knife plugin than trying to add yet another tool to the mix.

Am I missing something here?

[Geoff_Meakin]

Can anyone tell me why this doesnt work?

somenode:~# shef -z
loading configuration: /etc/chef/client.rb
Session type: client
Loading… (etc. etc.)
done.

Ohai2u root@node!
chef > node.automatic_attrs[‘network’]
=> {“interfaces”=>{}}
chef > node.automatic_attrs[‘ipaddress’]
=> nil

Yet it clearly has interfaces:

somenode:~# /sbin/ifconfig
eth0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet addr:xxx.xx.x.xx Bcast:xxx.xx.x.xxx Mask:255.255.255.128
(etc.)

Thanks

–

Geoff

[Bryan_McLellan]

On Mon, Sep 19, 2011 at 4:40 PM, Geoff Meakin Acid
geoffmeakin@aciddevelopments.co.uk wrote:

Yet it clearly has interfaces:

somenode:~# /sbin/ifconfig
eth0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet addr:xxx.xx.x.xx Bcast:xxx.xx.x.xxx Mask:255.255.255.128
(etc.)

I'd start with running "ohai network" and see if it returns data. If
not, run "ohai network -l debug" and look for errors.

[Geoff_Meakin]

Thanks, yeah there is certainly an error -

[Mon, 19 Sep 2011 22:11:21 +0100] DEBUG: Loading plugin linux::network
[Mon, 19 Sep 2011 22:11:21 +0100] DEBUG: Plugin linux::network threw exception #<NameError: uninitialized constant Ohai::Mixin::Command::StringIO>

Not sure what that means, but I'll snoop around.. (chef 0.10.4)

On 19 Sep 2011, at 21:56, Bryan McLellan wrote:

On Mon, Sep 19, 2011 at 4:40 PM, Geoff Meakin Acid
geoffmeakin@aciddevelopments.co.uk wrote:

Yet it clearly has interfaces:

somenode:~# /sbin/ifconfig
eth0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet addr:xxx.xx.x.xx Bcast:xxx.xx.x.xxx Mask:255.255.255.128
(etc.)

I'd start with running "ohai network" and see if it returns data. If
not, run "ohai network -l debug" and look for errors.

[Geoff_Meakin]

Hmm very strange - following some old bug reports, I edited this file:

gems/ohai-0.6.4/lib/ohai/mixin/command.rb
And required 'stringio' at the top which fixed the problem.

This seems like a bug to me? Or I've got an old version of ohai.

Incidentally I see from googling that this bug has been fixed and regressed a few times… maybe it has regressed again?

--
Geoff

Thanks, yeah there is certainly an error -

[Mon, 19 Sep 2011 22:11:21 +0100] DEBUG: Loading plugin linux::network
[Mon, 19 Sep 2011 22:11:21 +0100] DEBUG: Plugin linux::network threw exception #<NameError: uninitialized constant Ohai::Mixin::Command::StringIO>

Not sure what that means, but I'll snoop around.. (chef 0.10.4)

On 19 Sep 2011, at 21:56, Bryan McLellan wrote:

On Mon, Sep 19, 2011 at 4:40 PM, Geoff Meakin Acid
geoffmeakin@aciddevelopments.co.uk wrote:

Yet it clearly has interfaces:

somenode:~# /sbin/ifconfig
eth0 Link encap:Ethernet HWaddr xx:xx:xx:xx:xx:xx
inet addr:xxx.xx.x.xx Bcast:xxx.xx.x.xxx Mask:255.255.255.128
(etc.)

I'd start with running "ohai network" and see if it returns data. If
not, run "ohai network -l debug" and look for errors.

[Steven_Danna]

On Mon, Sep 19, 2011 at 2:24 PM, Geoff Meakin Acid
geoffmeakin@aciddevelopments.co.uk wrote:

Hmm very strange - following some old bug reports, I edited this file:

gems/ohai-0.6.4/lib/ohai/mixin/command.rb
And required 'stringio' at the top which fixed the problem.

This seems like a bug to me? Or I've got an old version of ohai.

Incidentally I see from googling that this bug has been fixed and regressed a few times… maybe it has regressed again?

A fix for this is already committed to master and should be in the
next release of Ohai: http://tickets.opscode.com/browse/OHAI-291

Cheers,

Steven