So, I was at one of the chef sessions at AWS Re:Invent yesterday,
apparently there’s a cookbook for creating security groups. Giant gaping
security issues aside, when I tried to use it, it fails building nokigiri.
I don’t think that’s ever compiled successfully.
Is there an alternative?
Doug
On Friday, November 14, 2014 at 9:43 AM, Douglas Garstang wrote:
So, I was at one of the chef sessions at AWS Re:Invent yesterday, apparently there's a cookbook for creating security groups. Giant gaping security issues aside, when I tried to use it, it fails building nokigiri. I don't think that's ever compiled successfully.
Is there an alternative?
Doug
Ruby has an XML library in the standard lib called REXML. It’s obviously much slower, but on the client side I think most use cases can tolerate the performance hit, but most tooling authors just use nokogiri for whatever reason. You could file an issue asking the authors to investigate using REXML instead (though this may not be possible if they’re relying on an upstream AWS library).
As for installing nokogiri, there are options for the installer to use local libxml2 packages rather than vendoring its own, these should be described in the nokogiri documentation. You might have to take over the nokogiri installation step to get this to work, depending on how this cookbook installs it.
--
Daniel DeLeo
For those of us not at the conference, can you say what the name of the cookbook you’re trying to use is?
FYI there was a recent change to the Nokogiri gem that added another external package dependency (specifically libghc-zlib-dev on Ubuntu). I’d suggest filing an issue with the cookbook author letting them know that this problem exists.
On November 14, 2014 at 12:50:23 PM, Daniel DeLeo (dan@kallistec.com) wrote:
On Friday, November 14, 2014 at 9:43 AM, Douglas Garstang wrote:
So, I was at one of the chef sessions at AWS Re:Invent yesterday, apparently there's a cookbook for creating security groups. Giant gaping security issues aside, when I tried to use it, it fails building nokigiri. I don't think that's ever compiled successfully.
Is there an alternative?
Doug
Ruby has an XML library in the standard lib called REXML. It’s obviously much slower, but on the client side I think most use cases can tolerate the performance hit, but most tooling authors just use nokogiri for whatever reason. You could file an issue asking the authors to investigate using REXML instead (though this may not be possible if they’re relying on an upstream AWS library).
As for installing nokogiri, there are options for the installer to use local libxml2 packages rather than vendoring its own, these should be described in the nokogiri documentation. You might have to take over the nokogiri installation step to get this to work, depending on how this cookbook installs it.
--
Daniel DeLeo
Eric,
They didn't specifically say at the conference which one they where
referring to, but this seems like the likely candidate:
I just took a look at the mkmf log file, and it seems to be failing here:
/tmp/ccGwgqii.o: In function main': /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/nokogiri-1.6.4.1/ext/nokogiri/conftest.c:8: undefined reference to libiconv_open'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/nokogiri-1.6.4.1/ext/nokogiri/conftest.c:9:
undefined reference to `libiconv'
collect2: ld returned 1 exit status
Usually this would mean a library is missing (how I'd get the package
installed first from chef would be a challenge), but I can't seem to find
any library called libiconv on CentOS 6.5. It looks like you can compile
nokogirl with --without-iconv, but I don't know how I could wrap the
aws_security cookbook to make it do that. 
Doug.
On Fri, Nov 14, 2014 at 10:27 AM, Eric Herot eric.opscode@herot.com wrote:
For those of us not at the conference, can you say what the name of the
cookbook you’re trying to use is?FYI there was a recent change to the Nokogiri gem that added another
external package dependency (specifically libghc-zlib-dev on Ubuntu). I’d
suggest filing an issue with the cookbook author letting them know that
this problem exists.On November 14, 2014 at 12:50:23 PM, Daniel DeLeo (dan@kallistec.com)
wrote:On Friday, November 14, 2014 at 9:43 AM, Douglas Garstang wrote:
So, I was at one of the chef sessions at AWS Re:Invent yesterday,
apparently there's a cookbook for creating security groups. Giant gaping
security issues aside, when I tried to use it, it fails building nokigiri.
I don't think that's ever compiled successfully.Is there an alternative?
Doug
Ruby has an XML library in the standard lib called REXML. It’s obviously
much slower, but on the client side I think most use cases can tolerate the
performance hit, but most tooling authors just use nokogiri for whatever
reason. You could file an issue asking the authors to investigate using
REXML instead (though this may not be possible if they’re relying on an
upstream AWS library).As for installing nokogiri, there are options for the installer to use
local libxml2 packages rather than vendoring its own, these should be
described in the nokogiri documentation. You might have to take over the
nokogiri installation step to get this to work, depending on how this
cookbook installs it.--
Daniel DeLeo
--
Regards,
Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627
Handling Nokogiri dependencies seems like a likely candidate for a new
community cookbook? I've run into this problem repeatedly, on
different platforms, any time there's a reliance on nokogiri somewhere
up the chain.
Doug, try fulfilling the deps for your platform based on the link
above and you'll proceed past the libiconv errors.
George Miranda — Partner Engineering
512.481.2876 – gmiranda@getchef.com – Linkedin Twitter
CHEF
TM
getchef.com Blog Facebook Twitter Youtube
Learn more about CHEF at re:Invent
Chef is an exclusive partner at the AWS Pop-Up Loft in SF
On Sun, Nov 16, 2014 at 9:31 AM, Douglas Garstang
doug.garstang@gmail.com wrote:
Eric,
They didn't specifically say at the conference which one they where
referring to, but this seems like the likely candidate:GitHub - searchspring/aws-security: Chef library cookbook for managing AWS Security Groups
I just took a look at the mkmf log file, and it seems to be failing here:
/tmp/ccGwgqii.o: In function
main': /opt/chef/embedded/lib/ruby/gems/1.9.1/gems/nokogiri-1.6.4.1/ext/nokogiri/conftest.c:8: undefined reference tolibiconv_open'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/nokogiri-1.6.4.1/ext/nokogiri/conftest.c:9:
undefined reference to `libiconv'
collect2: ld returned 1 exit statusUsually this would mean a library is missing (how I'd get the package
installed first from chef would be a challenge), but I can't seem to find
any library called libiconv on CentOS 6.5. It looks like you can compile
nokogirl with --without-iconv, but I don't know how I could wrap the
aws_security cookbook to make it do that.Doug.
On Fri, Nov 14, 2014 at 10:27 AM, Eric Herot eric.opscode@herot.com wrote:
For those of us not at the conference, can you say what the name of the
cookbook you’re trying to use is?FYI there was a recent change to the Nokogiri gem that added another
external package dependency (specifically libghc-zlib-dev on Ubuntu). I’d
suggest filing an issue with the cookbook author letting them know that this
problem exists.On November 14, 2014 at 12:50:23 PM, Daniel DeLeo (dan@kallistec.com)
wrote:On Friday, November 14, 2014 at 9:43 AM, Douglas Garstang wrote:
So, I was at one of the chef sessions at AWS Re:Invent yesterday,
apparently there's a cookbook for creating security groups. Giant gaping
security issues aside, when I tried to use it, it fails building nokigiri. I
don't think that's ever compiled successfully.Is there an alternative?
Doug
Ruby has an XML library in the standard lib called REXML. It’s obviously
much slower, but on the client side I think most use cases can tolerate the
performance hit, but most tooling authors just use nokogiri for whatever
reason. You could file an issue asking the authors to investigate using
REXML instead (though this may not be possible if they’re relying on an
upstream AWS library).As for installing nokogiri, there are options for the installer to use
local libxml2 packages rather than vendoring its own, these should be
described in the nokogiri documentation. You might have to take over the
nokogiri installation step to get this to work, depending on how this
cookbook installs it.--
Daniel DeLeo--
Regards,Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627
I had a JIRA ticket open for ages asking for nokogiri to get included in
the omnibus build of chef (not chef-dk); I think for very little pain it
would solve a lot of pain...
On Mon, Nov 17, 2014 at 7:18 PM, George Miranda gmiranda@getchef.com
wrote:
Handling Nokogiri dependencies seems like a likely candidate for a new
community cookbook? I've run into this problem repeatedly, on
different platforms, any time there's a reliance on nokogiri somewhere
up the chain.Installing Nokogiri - Nokogiri
Doug, try fulfilling the deps for your platform based on the link
above and you'll proceed past the libiconv errors.George Miranda — Partner Engineering
512.481.2876 – gmiranda@getchef.com – Linkedin Twitter
CHEF
TM
getchef.com Blog Facebook Twitter Youtube
Learn more about CHEF at re:Invent
Chef is an exclusive partner at the AWS Pop-Up Loft in SF
On Sun, Nov 16, 2014 at 9:31 AM, Douglas Garstang
doug.garstang@gmail.com wrote:Eric,
They didn't specifically say at the conference which one they where
referring to, but this seems like the likely candidate:GitHub - searchspring/aws-security: Chef library cookbook for managing AWS Security Groups
I just took a look at the mkmf log file, and it seems to be failing here:
/tmp/ccGwgqii.o: In function `main':
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/nokogiri-1.6.4.1/ext/nokogiri/conftest.c:8:
undefined reference to `libiconv_open'
/opt/chef/embedded/lib/ruby/gems/1.9.1/gems/nokogiri-1.6.4.1/ext/nokogiri/conftest.c:9:
undefined reference to `libiconv'
collect2: ld returned 1 exit statusUsually this would mean a library is missing (how I'd get the package
installed first from chef would be a challenge), but I can't seem to find
any library called libiconv on CentOS 6.5. It looks like you can compile
nokogirl with --without-iconv, but I don't know how I could wrap the
aws_security cookbook to make it do that.Doug.
On Fri, Nov 14, 2014 at 10:27 AM, Eric Herot eric.opscode@herot.com
wrote:For those of us not at the conference, can you say what the name of the
cookbook you’re trying to use is?FYI there was a recent change to the Nokogiri gem that added another
external package dependency (specifically libghc-zlib-dev on Ubuntu).
I’d
suggest filing an issue with the cookbook author letting them know that
this
problem exists.On November 14, 2014 at 12:50:23 PM, Daniel DeLeo (dan@kallistec.com)
wrote:On Friday, November 14, 2014 at 9:43 AM, Douglas Garstang wrote:
So, I was at one of the chef sessions at AWS Re:Invent yesterday,
apparently there's a cookbook for creating security groups. Giant
gaping
security issues aside, when I tried to use it, it fails building
nokigiri. I
don't think that's ever compiled successfully.Is there an alternative?
Doug
Ruby has an XML library in the standard lib called REXML. It’s obviously
much slower, but on the client side I think most use cases can tolerate
the
performance hit, but most tooling authors just use nokogiri for whatever
reason. You could file an issue asking the authors to investigate using
REXML instead (though this may not be possible if they’re relying on an
upstream AWS library).As for installing nokogiri, there are options for the installer to use
local libxml2 packages rather than vendoring its own, these should be
described in the nokogiri documentation. You might have to take over the
nokogiri installation step to get this to work, depending on how this
cookbook installs it.--
Daniel DeLeo--
Regards,Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627
On Mon Nov 17 11:23:49 2014, Thom May wrote:
I had a JIRA ticket open for ages asking for nokogiri to get included
in the omnibus build of chef (not chef-dk); I think for very little
pain it would solve a lot of pain...
chef-dk supported O/Sen != chef supported O/Sen.
the required pain would be maintaining building nokogiri on centos 5,
solaris 9, aix 6, and whatever other crazy operating systems we wind up
supporting chef-client on. and if you think building nokogiri on the
latest ubuntu and centos is painful...
I'd prefer not to go down this route at all. My original goal was to
automate the creation of security groups around a cookbook. I just learned
that in order to use encrypted data bags (which the aws_security cookbook
requires since it can't work with IAM roles) I have to deposit the secret
on the client somehow, I don't think that scales.
If all that did scale, then my other concerns are around security. Should
an instance be able to add itself to security groups? Smells fishy. On top
of that, all the dependencies the aws_security cookbook requires take a
long time to download and install (gcc, c++, fog and so on) and also
increase the complexity and decrease the chances of an instance coming up
successfully.
The problem here is not a chef one. It's infrastructure automation.
CloudFormation has a number of limitations. One is that if you want to use
reuse code, each piece of reused code becomes a stack in unto inself, and
your very quickly going to reach your stack limit of 20. You can request an
increase, but I've also heard anecdotally that once you hit about 100
stacks per account, performance starts to suffer.
Chef provisioner is cool, and I have some stuff working with that. However,
it's hard to install. I've been working with a developer for a week now
trying to clearly document how to install the chef client, which is
required to use chef provisioner. The biggest problem with the chef
provisioner is that it doesn't do anything except instances (and load
balancers?). No security groups, no EBS volumes. Nothing.
I just took a look at Terraform today. I'm disappointed. It seems you can't
have programmatic resource (ie instance) names. I imagine most people, like
us, have multiple environments, and therefore multiple instances with the
same name. It looks like the only way to do this with Terraform at the
moment would be to have multiple files, once per environment, and hard code
the resource names, eg: web01.prod.foo.com and web01.qa.foo.com.
Very frustrated.
Doug.
On Mon, Nov 17, 2014 at 11:37 AM, Lamont Granquist lamont@opscode.com
wrote:
On Mon Nov 17 11:23:49 2014, Thom May wrote:
I had a JIRA ticket open for ages asking for nokogiri to get included
in the omnibus build of chef (not chef-dk); I think for very little
pain it would solve a lot of pain...chef-dk supported O/Sen != chef supported O/Sen.
the required pain would be maintaining building nokogiri on centos 5,
solaris 9, aix 6, and whatever other crazy operating systems we wind up
supporting chef-client on. and if you think building nokogiri on the
latest ubuntu and centos is painful...
--
Regards,
Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com
Cell: +1-805-340-5627
For what it's worth, the new version (2.0) of the aws-sdk gem that's coming out soon uses multi_xml so ReXML can be used if one does not want any dependencies with extensions. Hopefully the cookbooks will be able to catch up with that (I for one have been enjoying the new version quite a bit).
On Nov 17, 2014, at 3:18 PM, Douglas Garstang doug.garstang@gmail.com wrote:
I'd prefer not to go down this route at all. My original goal was to automate the creation of security groups around a cookbook. I just learned that in order to use encrypted data bags (which the aws_security cookbook requires since it can't work with IAM roles) I have to deposit the secret on the client somehow, I don't think that scales.
If all that did scale, then my other concerns are around security. Should an instance be able to add itself to security groups? Smells fishy. On top of that, all the dependencies the aws_security cookbook requires take a long time to download and install (gcc, c++, fog and so on) and also increase the complexity and decrease the chances of an instance coming up successfully.
The problem here is not a chef one. It's infrastructure automation. CloudFormation has a number of limitations. One is that if you want to use reuse code, each piece of reused code becomes a stack in unto inself, and your very quickly going to reach your stack limit of 20. You can request an increase, but I've also heard anecdotally that once you hit about 100 stacks per account, performance starts to suffer.
Chef provisioner is cool, and I have some stuff working with that. However, it's hard to install. I've been working with a developer for a week now trying to clearly document how to install the chef client, which is required to use chef provisioner. The biggest problem with the chef provisioner is that it doesn't do anything except instances (and load balancers?). No security groups, no EBS volumes. Nothing.
I just took a look at Terraform today. I'm disappointed. It seems you can't have programmatic resource (ie instance) names. I imagine most people, like us, have multiple environments, and therefore multiple instances with the same name. It looks like the only way to do this with Terraform at the moment would be to have multiple files, once per environment, and hard code the resource names, eg: web01.prod.foo.com http://web01.prod.foo.com/ and web01.qa.foo.com http://web01.qa.foo.com/.
Very frustrated.
Doug.
On Mon, Nov 17, 2014 at 11:37 AM, Lamont Granquist <lamont@opscode.com mailto:lamont@opscode.com> wrote:
On Mon Nov 17 11:23:49 2014, Thom May wrote:
I had a JIRA ticket open for ages asking for nokogiri to get included
in the omnibus build of chef (not chef-dk); I think for very little
pain it would solve a lot of pain...chef-dk supported O/Sen != chef supported O/Sen.
the required pain would be maintaining building nokogiri on centos 5, solaris 9, aix 6, and whatever other crazy operating systems we wind up supporting chef-client on. and if you think building nokogiri on the latest ubuntu and centos is painful...
--
Regards,Douglas Garstang
http://www.linkedin.com/in/garstang http://www.linkedin.com/in/garstang
Email: doug.garstang@gmail.com mailto:doug.garstang@gmail.com
Cell: +1-805-340-5627