Hi, I am using Chef server 12.5.0 and chef-sync 1.0.0-rc.7 to replicate chef server between 2 instances.
Sync starts and keep running for a while with no problem, but every day it crashes with connection refused error after some time and it seems like crash happens because master side cannot start database connection. (there are some errors like below in the master side)
2016-08-12_00:00:50.29844 =ERROR REPORT==== 12-Aug-2016::00:00:50 ===
2016-08-12_00:00:50.29844 ** Generic server <0.645.0> terminating
2016-08-12_00:00:50.29856 ** Reason for termination ==
2016-08-12_00:00:50.29856 ** {empty,[{queue,get,[{,}],[{file,"queue.erl"},{line,181}]},
2016-08-12_00:00:50.29857 {epgsql_sock,command_tag,1,
2016-08-12_00:00:50.29858 [{file,"src/epgsql_sock.erl"},{line,409}]},
2016-08-12_00:00:50.29858 {epgsql_sock,on_message,2,
2016-08-12_00:00:50.29859 [{file,"src/epgsql_sock.erl"},{line,684}]},
2016-08-12_00:00:50.29859 {epgsql_sock,loop,1,[{file,"src/epgsql_sock.erl"},{line,334}]},
2016-08-12_00:00:50.29859 {gen_server,handle_msg,5,[{file,"gen_server.erl"},{line,604}]},
2016-08-12_00:00:50.29859 {proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,239}]}]}
2016-08-12_00:00:50.36570 =ERROR REPORT==== 12-Aug-2016::00:00:50 ===
2016-08-12_00:00:50.36570 pool 'sqerl' failed to start member: {error,<<"57P03">>}
2016-08-12_00:00:50.36590 =ERROR REPORT==== 12-Aug-2016::00:00:50 ===
2016-08-12_00:00:50.36591 Unable to start database connection: <<"57P03">>
2016-08-12_00:00:50.37925 =SUPERVISOR REPORT==== 12-Aug-2016::00:0 0:50 ===
2016-08-12_00:00:50.37925 Supervisor: {local,pooler_sqerl_member_sup}
2016-08-12_00:00:50.37926 Context: child_terminated
2016-08-12_00:00:50.37926 Reason: <<"57P03">>
2016-08-12_00:00:50.37926 Offender: [{pid,<0.12165.0>},
2016-08-12_00:00:50.37926 {name,sqerl_client},
2016-08-12_00:00:50.37926 {mfargs,{sqerl_client,start_link,undefined}},
2016-08-12_00:00:50.37926 {restart_type,temporary},
2016-08-12_00:00:50.37927 {shutdown,brutal_kill},
2016-08-12_00:00:50.37927 {child_type,worker}]
2016-08-12_00:00:50.37927
2016-08-12_00:00:50.37945
2016-08-12_00:00:50.37945 =ERROR REPORT==== 12-Aug-2016::00:00:50 ===
according to the result of chef-server-ctl status, it seems like some of the server components including postgresql are restarted while ec_sync_server keeps running.
(this result was taken around 18:00)
run: bookshelf: (pid 17038) 67262s; run: log: (pid 20916) 5608899s
run: ec_sync_server: (pid 4321) 84867s; run: log: (pid 30600) 5607658s
run: nginx: (pid 17097) 67260s; run: log: (pid 21020) 5608896s
run: oc_bifrost: (pid 17114) 67258s; run: log: (pid 20626) 5608906s
run: oc_id: (pid 17163) 67256s; run: log: (pid 20646) 5608905s
run: opscode-erchef: (pid 17224) 67251s; run: log: (pid 20972) 5608898s
run: opscode-expander: (pid 17245) 67249s; run: log: (pid 20842) 5608900s
run: opscode-pushy-server: (pid 22468) 2155459s; run: log: (pid 22116) 5608841s
run: opscode-reporting: (pid 22483) 2155459s; run: log: (pid 22020) 5608843s
run: opscode-solr4: (pid 17267) 67247s; run: log: (pid 20786) 5608900s
run: postgresql: (pid 17300) 67244s; run: log: (pid 20579) 5608906s
run: rabbitmq: (pid 17331) 67242s; run: log: (pid 20487) 5608907s
run: redis_lb: (pid 17524) 67240s; run: log: (pid 21015) 5608897s
I suppose this indicates that ec_sync_server wasn't aware of postgresql's restart(happened around 0:00) and because of that, ec_sync_server returned connection error to the replica and sync crashed.
Same thing happens everyday a bit after 0:00.
Is postgresql supposed to be restarted at midnight every day?
Is there any way to prevent this crash?
to fix this, I need to restart ec_sync_server on the master side and stop/start sync on the replica side every day...