We need to create various AWS components such as VPC, NAT, EC2 instances etc. There are several suggestions for tools, and the most highlighted ones are Ansible and Chef.
Ansible has got resources to create and destroy VPC, EC2 etc.
My questions are:
Does chef has any built in resources to create, destroy and manage AWS VPC, EC2 etc?
If yes then how do we use those resources. Do we just get them installed on Chef DK and start using chef-apply command or is there a better way to do it?
I would suggest Chef Provisioning. It can build out aws from the vpc up.
See https://docs.chef.io/provisioning.html for details. The best part of it is that provisioning works just like a normal chef resource, so it becomes just another cookbook.
Thanks @dreamnite I will look into it. @coderanger: More number of tools means more management required and new learning for team. We are trying to stick with minimum tools. So, it is either Chef or AWS SDK. I am using Chef for quite a while now. Could you help me understand What is the reason for not recommending chef-provisioning?
My 2 cts: Here we’re using cloudformation for the infrastructure part (VPC, ELB, ASG, EC2 instances, etc) and chef on the instances to configure what can’t be done at packaging time (we are trying to use the Golden Image paradigm).
I’ve no idea why couldformation is so unloved, but there’s probably reasons I didn’t already get bitten from.
Thanks for the article @coderanger. I was going through chef.io and I found this:“Chef gives you a single, unified way to automate AWS services and resources.” Does your article mean Chef does not has any reliable solution for AWS automation?
I’m very disappointed that Chef Automate in what looks like a terrific product in many ways, doesn’t address the critical element of specifying and instantiating the actual infrastructure.
I did have high hopes with Chef Provisioning. I did build a production infrastructure out of it and generally liked it. The main points where it lacked was documentation. But when I was told it wasn’t going any further at any time soon, I looked at the Noah’s suggestions.
And have been totally disappointed. Cloud formation and Terraform is a major step backwards being purely declarative, very brittle. Its “Infrastructure as JSON” not “Infrastructure as Code”.
SparkleFormation looked promising as it is truly programable, but it suffers even more than Chef Provisioning from not having critical mass and not enough eyes to find/fix bugs.
Does this mean I have to go back to funky wrappers around knife and spice weasel?
Has anyone found something better than Infrastructure as JSON or YAML?
none that I am aware of. Terraform is decent if you like declarative style,
and does not control the whole fleet but only provision/de-provision (i.e.
dont use state). If you like programmability, sparkle formation is still
your best bet.
I dont think we’ll have tools with larger community around this usecase.
Cause most users who churn lots of hosts, are likely to use cloud specific stuff
(cloud formation, or azure specific gears) either from CI or some ancillary
services. Given that there exists low level libraries like fog, boto and
jcloud it is relatively easier to roll something custom. And the
requirement of having something framework style (not library) as well as
cross cloud is slim. We had this requirement, and we rolled our own
provisioning gears on top of knife bootstrap. Existing stuff didnt fit, and
we didnt want to wait.
Concurrent provisioning has gremlins if done on chef provisioning layer,
due to chef config being global and other known ruby gotchas.
I think among the current solution, if terraform add mruby based plugins,
that will open up the possibility to have a turing complete DSL beyond the
json/hcl/yaml based stuff. … but thats some work , & hashicorp have to
be fine with something like that (considering you want this mainstream).
"Reliable" is a very loaded term. Chef Provisioning isn't unreliable, it's just not a project that we recommend anyone start new infrastructure projects around given the status I laid out in the blog post. As I said, Terraform or SparkleFormation are likely better options for most people. Both work great with Chef, so this isn't an either/or choice.
We use CloudFormation combined with jsonnet ( https://github.com/google/jsonnet), which basically lets you turn your
CloudFormation JSON templates into fully functional code. We created a
library out of it and it works great.
To disagree somewhat, provisioning has become an open source chef project. At this point I would say attracting new users is critical to reaching “critical mass” for attracting those who can contribute.
Please, try chef provisioning. It is full featured and stable. Be aware of coderanger’s comments, but do try it.
i too think that community wise we can make more progress on chef-provisioning. Also agree that terraform and so on are not that great of options as they really don’t help you with orchestration in any way other than creating some things etc… while a costly option exists in hashicorp’s atlas - you’re basically tying your entire infra/code/deployment to hashicorp at that point… it ‘can’ work but its very disconnected from some of the things I find chef provisioning can do for you… if your scenarios are simple I’m guessing any tool can do it - but its when you need more control that tools like terraform come short
SparkleFormation looked promising as it is truly programable, but it suffers even more than Chef Provisioning from not having critical mass and not enough eyes to find/fix bugs.
Critical mass no, but I've had very quick responses from those in the #sparkleformation irc on freenode.
To be clear, SparleFormation is neither AWS nor CloudFormation dependent. It provides a DSL and a cli tool, sfn, to work with cloud provider orchestration templates. It can generate equally Cloudformation, Heat, Azure Resource Manager or others. If you desire programmability and wish to avoid spending your time hunting down missing commas in a serialization format like json, then you might like it.
It pairs nicely with Chef since the DSL is still ruby based.
For all of Terraform’s short-comings we’ve enjoyed the separation of infrastructure from bootstrap.
Anything held in CF lacks post-change updates meaning the stack is the stack, changes are a new stack so drift is inevitable. We’ve also going way of packer+ Terraform for rapid deployment and the chef bootstrap is sub 1m to keep drift down and consistency. Changes to the TF plan are easy and while orchestration is hard I’ve managed to find a way. If recommend looking seriously at Terraform.
We currently utilize chef provisioning for all of our deployments including
infrastructure in AWS and physical hardware, it’s pretty easy to use and
like you said it won’t add any more tools to your belt.
dear @dreamnite@manishmehra ,
Hello, my project slightly relate to this which I want to use chef automation for VPC , RDS Instance on AWS cloud but I was not able to access the links so kindly share the link which is accessible.
Just my unofficial "2 cents" but I've found Pulumi to be a great alternative to Terraform. There is even a converter from Terraform called tf2pulumi and also a nice wrapper to use existing Terraform providers. kitchen-ec2 is nice but obviously somewhat limited in what it enables you to provision.
Back to the original question, would you consider leveraging kitchen if there were a kitchen-aws tool? that leveraged the underlying AWS SDK gem code? Or, alternatively, would you find it useful to have something like a 'predeployment' template much like the kitchen-azurerm driver provides for Azure but for AWS?
Thank you for questions, input and I hope this information was helpful!
, & hashicorp have to