Azure Powershell BootStrapping


#1

Hi All, I am currently automating our Azure build process using PowerShell and am stuck trying to get Chef to bootstrap.

The commands I am using are:

$BootStrap = ‘{“chef_node_name”:"’ + $vmName + ‘",“chef_server_url”:“https:////chef-livingston.company.net//organizations//company”, “validation_client_name”:“win.self.bootstrap”}’

Set-AzureRMVMChefExtension -ResourceGroupName $RGPName -VMName $vmName -ValidationPem “E:\\Program Files\\Server Build\\New_Azure\\Pem\\win.self.bootstrap.pem” -BootstrapOptions $BootStrap -RunList $Runlist -ChefServerURL “https:////chef-livingston.company.net//organizations//company” -ValidationClientName “win.self.bootstrap” -ChefServiceInterval “60” -Windows

The Chef agent installs fine but it doesn’t want to bootstrap. I believe it’s down to SSL between the agent and the server. I want to turn this off. How do I do this? I took a stab in the dark and tried:

$BootStrap = ‘{“chef_node_name”:"’ + $vmName + ‘",“chef_server_url”:“https:////chef-livingston.company.net//organizations//company”, “validation_client_name”:“win.self.bootstrap”, “ssl_verify_mode”:“verify_none”}’

No luck though. Can anyone help?

Ta
Cam


#2

Here is the content of chef-stacktrace.out

Generated at 2017-03-09 11:50:00 +0000
OpenSSL::SSL::SSLError: SSL Error connecting to https://chef-livingston.company.net/organizations/company/clients - SSL_connect returned=1 errno=0 state=error: certificate verify failed
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http.rb:412:in rescue in retrying_http_errors' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http.rb:369:inretrying_http_errors’
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http.rb:333:in send_http_request' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http.rb:148:inrequest’
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http.rb:131:in post' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/api_client/registration.rb:96:increate’
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/api_client/registration.rb:87:in create_or_update' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/api_client/registration.rb:58:inrun’
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/client.rb:621:in register' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/client.rb:267:inrun’
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/application.rb:277:in run_with_graceful_exit_option' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/application.rb:253:inblock in run_chef_client’
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/local_mode.rb:44:in with_server_connectivity' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/application.rb:236:inrun_chef_client’
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/application/client.rb:427:in run_application' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/application.rb:59:inrun’
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/bin/chef-client:26:in <top (required)>' C:/opscode/chef/bin/chef-client:68:inload’
C:/opscode/chef/bin/chef-client:68:in `’

Caused by OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed
C:/opscode/chef/embedded/lib/ruby/2.3.0/net/http.rb:933:in connect_nonblock' C:/opscode/chef/embedded/lib/ruby/2.3.0/net/http.rb:933:inconnect’
C:/opscode/chef/embedded/lib/ruby/2.3.0/net/http.rb:863:in do_start' C:/opscode/chef/embedded/lib/ruby/2.3.0/net/http.rb:852:instart’
C:/opscode/chef/embedded/lib/ruby/2.3.0/net/http.rb:1398:in request' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http/basic_client.rb:70:inrequest’
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http.rb:340:in block in send_http_request' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http.rb:372:inblock in retrying_http_errors’
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http.rb:370:in loop' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http.rb:370:inretrying_http_errors’
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http.rb:333:in send_http_request' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http.rb:148:inrequest’
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http.rb:131:in post' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/api_client/registration.rb:96:increate’
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/api_client/registration.rb:87:in create_or_update' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/api_client/registration.rb:58:inrun’
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/client.rb:621:in register' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/client.rb:267:inrun’
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/application.rb:277:in run_with_graceful_exit_option' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/application.rb:253:inblock in run_chef_client’
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/local_mode.rb:44:in with_server_connectivity' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/application.rb:236:inrun_chef_client’
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/application/client.rb:427:in run_application' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/application.rb:59:inrun’
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/bin/chef-client:26:in <top (required)>' C:/opscode/chef/bin/chef-client:68:inload’
C:/opscode/chef/bin/chef-client:68:in `’


#3

Hi Cammmy,

You were pretty close I think. There’s a bootstrap_option that the VM Extension should recognise called node_ssl_verify_mode. It seems to be documented for ARM templates at https://docs.chef.io/azure_portal.html but not explicitly for the PowerShell cmdlets:

"node_ssl_verify_mode": "none"

Alternatively if you’re building an image for distribution don’t forget you can use knife ssl fetch https://chef-livingston.company.net and then try and pre-load the certificates into the c:\chef\trusted_certs folder within in the image.

Please let me know how you get on.

thanks,
Stuart


#4

Thanks for the response Stuart. We managed to find that by digging into the details of the VM after installing the extension through the GUI. Unfortunately it’s still not working. I’ve just tried this:

$BootStrap = ‘{“chef_node_name”: "’ + $vmName + ‘", “node_ssl_verify_mode”: “none”, “node_verify_api_cert”: “false”}’

Set-AzureRMVMChefExtension -ResourceGroupName $RGPName -VMName $vmName -ValidationPem “E:\Program Files\Server Build\New_Azure\Pem\win.self.bootstrap.pem” -BootstrapOptions $BootStrap -RunList $Runlist -ChefServerURL “https://chef-livingston.brewindolphin.net/organizations/brewin_dolphin” -ValidationClientName “win.self.bootstrap” -ChefServiceInterval “60” -Windows

I’m still getting this:

OpenSSL::SSL::SSLError: SSL Error connecting to https://chef-livingston.brewindolphin.net/organizations/brewin_dolphin/clients - SSL_connect returned=1 errno=0 state=error: certificate verify failed

Caused by OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed

We’re trying to use market place images rather than maintaining our own.


#5

Hi Cammy,

Unfortunately it doesn’t look like that is actually a bootstrap_option and is more of a directive to the extension, and it looks like it’s only settable using ARM, rather than being exposed as a parameter to the cmdlet.

One more thought is that you could try setting the ssl_verify_mode by specifying a custom client.rb (using the -client_rb parameter to the PowerShell cmdlet?). You may need to delimit each line with \n.

regards,
Stuart


#6

Thanks Stuart, that was my next thought but wanted to avoid it if possible. I may also try and get a certificate that the clients will trust. Issue is, they aren’t domain joined at this point. So we can’t use our CA.


#7

What’s the process you use to stand up the machines themselves? Do you use an ARM template or some other process. Perhaps you could add the settings by using node_ssl_verify_mode or even specifying chef_server_crt via use of the ARM template deployment? Or if you’re wedded to using PowerShell cmdlets everywhere with no external payload I’ve seen many more creative solutions out there in the field I’d be happy to talk you through directly!

regards,
Stuart


#8

I have a fair bit of PowerShell experience but am new to Azure and Chef, so my aim is to keep things as simple as possible and all in one place. In a nutshell the build process is as follows. An in house website will generate a template file for servers based on the requirements (machine spec, environment etc). This file looks like this:

$Location = “West Europe”
$Subscription = “Non-Production Subscription”
$global:vmQTY = 1
$vmName = “Test-Tes-App-xx”
$vmSize = “Standard_A1_v2”
$vmDomain = “Company.dev”
$OSpubName = “MicrosoftWindowsServer”
$OSofferName = “WindowsServer”
$OSskuName = “2012-R2-Datacenter”
$OSdiskName = “Test-Tes-App-xx-C”
$RGPName = “Test-Test”
$vNetRGPName = “NonProd-ExpressRouteConnection”
$vNetName = “Company-Azure-NonProd-Vnet”
$SubnetName = “NonProd-Back-End”
$maxdisks = 2
$diskmagic = $true
$PublicIP = $False
$AzureAS = “Test-Test-App-AS”
$vmPatchOption = 2
$vmPatchWeek = “Excluded”
$vmPatchDay = 1
$vmPatchHour = 0
$vmOSDisk = [pscustomobject]@{label = “OS”; STAName = “cpytesttesthdd”; STAType = “Standard_LRS”}
$vmDisks = @(

)
$vmLocalAdmins = ("")
$vmNotes = “Build test”
$vmCMDBApplication = “1eed7734db19ea0072aab6bffe961969”
$vmCMDBOwnerApp = “Server Infrastructure”
$vmRequester = “CN=Name,User,OU=Users,OU=London,OU=Sites,DC=Company,DC=Net”
$RunList = “production.base”
$UserEmail = "user@company.com"

This is then picked up by a series of scripts which build and configure the server and any of the prerequisites (resource groups, availability sets etc.). The servers are then built using an Azure RM VM Config passed to the New-AzureRMVM cmdlet

$vm = New-AzureRmVMConfig -VMName $vmName -VMSize $vmSize -AvailabilitySetId $AzureASId

Configs are then applied to the VMConfig, e.g.

$vm = Set-AzureRmVMOperatingSystem -VM $vm -Windows -ComputerName $vmName -Credential $cred -ProvisionVMAgent -EnableAutoUpdate

New-AzureRmVM -ResourceGroupName $RGPName -Location $Location -VM $vm

After the VM is built, I’m using Set-AzureRMVMChefExtension to try and bootsrap it.

Ta
Cam


#9

Okay, I’m closer! I have the agent bootstrapping with this

$BootStrap = ‘{“chef_node_name”: "’ + $vmName + '"}'
Set-AzureRMVMChefExtension -ResourceGroupName $RGPName -VMName $vmName -ValidationPem ‘E:\Program Files\Server Build\New_Azure\Pem\win.self.bootstrap.pem’ -ClientRB ‘E:\Program Files\Server Build\New_Azure\ClientRB\client.rb’ -BootstrapOptions $BootStrap -RunList $Runlist -ChefServerURL ‘https://chef-livingston.company.net/organizations/company’ -ValidationClientName “win.self.bootstrap” -ChefServiceInterval ‘60’ -Windows

My client.rb just contains:

ssl_verify_mode :verify_none

The server appears in Chef and gets it’s runlist assigned, but it never runs it. From then on I get:

Error Source/Chef EventID/10003

Failed Chef Client run UNKNOWN in UNKNOWN seconds.
Exception type: Net::HTTPServerException
Exception message: 403 "Forbidden"
Exception backtrace: C:/opscode/chef/embedded/lib/ruby/2.3.0/net/http/response.rb:120:in error!' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http.rb:150:inrequest’
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/http.rb:123:in put' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/api_client/registration.rb:102:inupdate’
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/api_client/registration.rb:92:in rescue in create_or_update' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/api_client/registration.rb:87:increate_or_update’
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/api_client/registration.rb:58:in run' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/client.rb:621:inregister’
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/client.rb:267:in run' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/application.rb:277:inrun_with_graceful_exit_option’
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/application.rb:253:in block in run_chef_client' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/local_mode.rb:44:inwith_server_connectivity’
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/application.rb:236:in run_chef_client' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/application/client.rb:427:inrun_application’
C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/lib/chef/application.rb:59:in run' C:/opscode/chef/embedded/lib/ruby/gems/2.3.0/gems/chef-12.19.36-universal-mingw32/bin/chef-client:26:in<top (required)>'
C:/opscode/chef/bin/chef-client:68:in load' C:/opscode/chef/bin/chef-client:68:in

I removed the runlist and restarted the machine, but I get the same issue. My searching has only turned up suggestions around permissions (which I’ve checked and seem to be okay AFAIK) and timeouts on long running cookbooks (not the issue here).

Any suggestions?

Ta
Cam


#10

I got this working in the end:

$BootStrap = ‘{“chef_node_name”:"’ + $vmName + ‘"}’

Set-AzureRMVMChefExtension -ResourceGroupName $RGPName -VMName $vmName -ValidationPem ‘E:\Program Files\Server Build\New_Azure\Pem\win.self.bootstrap.pem’ -ClientRB ‘E:\Program Files\Server Build\New_Azure\ClientRB\client.rb’ -BootstrapOptions $BootStrap -RunList $Runlist -ChefServerURL ‘https://chef.company.net/organizations/company’ -ValidationClientName “win.self.bootstrap” -ChefServiceInterval ‘60’ -Windows

The first cookbook that runs joins the machine to the domain. This was then breaking the client as the nodename would change. Just before the domain join, I put this line in:

$NodeName = “node_name`t” + “’” + $MachineName + "’"
Add-Content “C:\chef\client.rb” $NodeName

My machines now build and bootstrap/install cookbooks with 0 user interaction.


#11

Hi All

Well, this worked for a while but we now have an issue. The nodes have started registering with .reddog.microsoft.com in their nodename, despite what I have above.

Example build, Dev-TES-App-01

When the machine was provisioned this was fed as the bootstrap options:

$BootStrap = ‘{“chef_node_name”:"’ + $vmName + ‘"}’

Which resolves to:

{“chef_node_name”:“Dev-TES-App-01”}

-BootstrapOptions $BootStrap

It seems to pick this up as 0.Settings contains:

cSettings": {“bootstrap_options”:"{“chef_node_name”:“Dev-TES-App-01”}"

but the nodename in Chef is showing as:

Dev-TES-App-01.reddog.microsoft.com

This is breaking the client as when the machine joins our domain, it’s nodename changes. We also don’t want them to register with reddog.microsoft.com in the name anyway, as this isn’t our domain.

Does anyone know why the client would be ignoring the options I’m feeding into it?

Ta
Cam


#12

Hi Cam,

I don’t know if this would help but try to force it use your DNS instead of Azure before applying the name and joining it to the domain.
Azure provides DNS name resolution by default for all virtual machines so i wrote a cookbook for the Linux machine to use the DNS i wanted, I believe its still in the Chef server if no one deleted it :smile:

Regards,
Nick