Automate Apache Spark Cluster with Chef

Chef Infra (archive)
1

So I am using Morpheus as a CMP, and within this I can create a blueprint / app deployment that will automatically spit out 3 individual Spark nodes, however I need to automate the cluster configuration. I.e. setting node1 as master, and nodes 2 & 3 as slaves, and then configuring the workstreams between them.

Th end goal is our client will be able to do the deployment via Morpheus themselves via the app I have created, however as part of the deployment it will include the automated cluster config.

Any advice to try and complete?

Thanks
Nick

2

I will quickly try and grab the page that details as much as I've been able to find regarding the cluster config.

3

You might want to take a look at Habitat. Habitat excels at deploying and managing clustered applications and there is a Spark plan already available.

https://bldr.habitat.sh/#/pkgs/core/spark/latest

4

Thanks Matt,

I had come across Habitat as well as Apache's own deployment scripts etc. My main issue is trying to deploy through Morpheus AND configure the cluster in an AWS/Azure one-click type service for our client.

We may ultimately need to provide this particular service another way, but exploring all options first.

Thanks again
Nick