Continue the chef recipe execution even after server reboot

Chef Infra (archive)
1

Hi there,

Can any one help us to Continue the chef recipe execution even after server reboot.

We have a recipe which will call around 5 recipes, in those recipe 1 requires reboot(Which perform domain join on Windows server). Once server is added to domain, we need to execute remaining 4 recipes which are dependent on domain join.

Can you please let us know how to run continuation of recipe execution once the server got restarted and came back to online,

Thanks
Venkat

2

create a recipe that will execute chef-client -d 300 service to run every 5 minutes and another recipe that will change it once the machine joins the domain.

3

More elaboration on this will be much appreciated as i am beginner to chef…

4

If you’re looking for reboot handlers, there is a cookbook which might help you to reboot immediately without aborting a chef-client run.

Here is this : https://github.com/vinyar/chef_win_reboots

A real example can be seen in pattern two – which was really the genesis for this repo from way back when

Hope this will work for you.

Cheers,
Waseem

5

I had faced the similar situation in past.
In my scenario, there were 5 recipes, and after 3rd recipe we want to
reboot the server and after that chef should execute the remaining two
recipes. this is how I handled this use case

I split the recipes into two sets, first set will contain 3 recipes and
second one will have 2 recipes. third recipe where reboot in called,
before that I created a scheduled task (in windows), that scheduled task
will be called only once and will execute the remaining second set of
recipes. once the last recipe is executed it will delete the scheduled task.

hope this will helps.

Thanks,
Sachin

6

Hi Sachin,

Thank you for your reply,

we have tried of creating windows task which is as below. And we called before calling recipe which requires reboot.

windows_task ‘chef ad-join’ do
task_name 'chef ad-join’
user 'SYSTEM’
command 'chef-client -L C:\chef\chef-ad-join.log’
run_level :highest
frequency :onstart
only_if { node[‘kernel’][‘cs_info’][‘domain_role’].to_i == 0 || node[‘kernel’][‘cs_info’][‘domain_role’].to_i == 2 }
notifies :create, ‘registry_key[warning]’, :immediately # http://bit.ly/1WDZ1kn
action :create
end

but the system is getting rebooted and after that its not getting executed remaining recipes.
Can you please help us here to get it sorted out.

Thanks

7

We had similar issue, Hostname change needs reboot, AD Join needs reboot, GPO for assigning WSUS server doesn’t apply till after reboot and is needed before running patches.

What we ended up doing is creating a scheduled task to run chef-client on system startup. If a server has been down for a while we want to make sure it converges right away before it starts handling production work loads anyway. This is seperate from the recurring scheduled task that may take a bit after server boot to run. Then we simply added our run list to the role cookbook. The resources that need a reboot we have set to :delayed so it reboots at the end of the chef client run, and the resources that need to wait till after the reboot we have wrapped in an “unless” statement checking on “reboot_pending?”

knifing a new server then we end up with 4 reboots before it’s finally ready for use, hostname gets set, chef-client on startup task gets created as well as the recurring scheduled task. Adjoin, and patches don’t run. Second server run chef kicks off right away and joins the domain with new host name, patches still don’t run. Third reboot GPO’s from domain are applied and in house WSUS server is set and chef server kicks off again and applies patches this time and reboots if any patches require it.

There is no easy way for chef to resume a client run where it left off after reboot. You’d have to create a system that saves the state of the chef client run and sets an autorun on next boot to kick off and read in the state and continue. Not an easy task. Otherwise building resources with good idempotency and logic you should be able to run every cookbook every time chef runs on a node to verify all resources have successfully converged.

1 Like
8

Hi Venkat,

Sorry for the late reply. here is the code snippet which i used in my
recipe , This will give you an brief idea.

code before reboot recipe
execute “Create startChefClient” do
command “SchTasks /Create /SC ONSTART /TN “startChefClient” /TR
"E:\test.bat” /ru ******** /rp *******"
action :run
end

execute “reboot windows machine” do
command 'shutdown /r’
action :run
end

9

Hi Venkat,

I have created the one and checked on few of my servers. I found it’s working absolutely fine for me.

---------------------------Script Start------------------------
powershell_script ‘Domain_Join’ do
guard_interpreter :powershell_script
code -domainJoin
$currentTime = Get-Date
$currentTimeString = $currentTime.ToString()

$systemDomain = (Get-WmiObject Win32_ComputerSystem).Domain
If (($systemDomain) -eq ‘domain_name’)
{
Try {
write-host "$env:computerName is DOMAIN MEMBER"
Remove-Item C:\DomainJoinFlag.txt -ErrorAction Stop
Unregister-ScheduledTask -TaskName “Chef client schedule_DJ” -Confirm:$false
} # end of Try
Catch [System.Management.Automation.ItemNotFoundException]
{ write-host "Server is already domain member, Or Exception raised due to either missing FLAG file or Server startup schedule task configuration."
eventcreate /t INFORMATION /ID 0909 /L APPLICATION /SO “ChefClient_$env:computerName” /D “Server is already domain member, Or Exception raised due to either missing FLAG file or Server startup schedule task configuration. Refer to the CHEF reciepie for DomainJoin or check Administrative credentials for creting schedule task”}
}
else { write-host "$env:computerName is NOT domain member, joining the server to the domain. Server will be rebooting in a while…"
eventcreate /t INFORMATION /ID 0909 /L APPLICATION /SO “ChefClient_$env:computerName” /D "Joining the server : $env:ComputerName to the domain ININLAB.COM (Server Time): $currentTimeString"
New-Item C:\DomainJoinFlag.txt -type file -force
write-host “$env:computerName DOMAIN JOIN INITIATED for the server”
$cred = New-Object System.Management.Automation.PsCredential(“domain\domain_user”, (ConvertTo-SecureString “Password” -AsPlainText -Force))
Add-Computer -DomainName “domain_name” -Credential $cred -OUPath "OU=HyperV,OU=Infrastructure,OU=Servers,DC=domain,DC=name"
shutdown -r -t 120
} #end_of_else
domainJoin
notifies :run, ‘windows_task[Chef client schedule_DJ]’, :immediately
end

windows_task ‘Chef client schedule_DJ’ do
user 'SYSTEM’
command 'chef-client -L C:\chef\chef-client_after_reboot_domainJoin.log’
run_level :highest
frequency :onstart
frequency_modifier 30
action :create
only_if { ::File.exist?(‘C:\DomainJoinFlag.txt’) }
end

---------------------------Script Ends------------------------

Recommendations: Use data_bags for domain_name, Domain Username, Domain Password.