Need help on pushy server error

Chef Infra (archive)
1

Hi There,

Today we suddenly started having the following error on our push server. I
have tried reconfigure the push server, reinstalled the push server,
upgrade all Chef packages to the latest versions, but still have the
following errors

2015-06-04_04:20:51.27833 04:20:51.170 [error] (Undefined:Undefined)
<0.328.0> CRASH REPORT Process <0.328.0> with 0 neighbours exited with
reason: {error,invalid_password} in gen_server:init_it/6 line 320
2015-06-04_04:20:51.28245 04:20:51.170 [error] (Undefined:Undefined)
<0.277.0> pool ‘sqerl’ failed to start member:
{error,{error,invalid_password}}
2015-06-04_04:20:51.28729 04:20:51.175 [error] (Undefined:Undefined)
<0.332.0> Unable to start database connection: {error,invalid_password}
2015-06-04_04:20:51.28729
2015-06-04_04:20:51.29202 04:20:51.287 [error] (Undefined:Undefined)
<0.332.0> CRASH REPORT Process <0.332.0> with 0 neighbours exited with
reason: {error,invalid_password} in gen_server:init_it/6 line 320
2015-06-04_04:20:56.09388 04:20:56.093 [warning] (Undefined:Undefined)
<0.6.0> lager_error_logger_h dropped 43 messages in the last second that
exceeded the limit of 50 messages/sec
2015-06-04_04:20:56.09391 04:20:56.093 [info] (Undefined:Undefined) <0.7.0>
Application pushy exited with reason:
{shutdown,{pushy_app,start,[normal,[]]}}
2015-06-04_04:20:57.62211 {“Kernel pid
terminated”,application_controller,"{application_start_failure,pushy,{shutdown,{pushy_app,start,[normal,[]]}}}"}
2015-06-04_04:20:57.62302 Assertion failed: ok (mailbox.cpp:84)
2015-06-04_04:20:57.62303 Assertion failed: pfd.revents & POLLIN
(signaler.cpp:155)

It causes the server returns error “504 Gateway Time-out” now to all push
job api.

How can I fix this error to our push server? I assume I need to reset the
password for chef-push_user, but couldn’t figure it out how to do it.

Can someone help me please

Thanks so much.

–

Jing Li
Senior .Net Developer, The Network for Learning Ltd

D: +64 9 222 0170 | W:
A: Suite 306, Geyser Building, 100 Parnell Road, Parnell, Auckland 1052
P: PO Box 37118, Parnell, Auckland 1151

http://www.n4l.co.nz
https://twitter.com/@N4LNZ/ https://www.facebook.com/NetworkForLearning
https://plus.google.com/104649004276520409744/about
http://www.linkedin.com/company/the-network-for-learning-ltd-n4l-

2

This is an error with the Push Server application connecting to the PostgreSQL server it uses for storing state. Did any of this state recently change? Did you run a push-server-ctl reconfigure recently?

—
Stephen Delano - Engineering Lead, Chef

On Wed, Jun 3, 2015 at 9:27 PM, Jing Li jing.li@n4l.co.nz wrote:

Hi There,
Today we suddenly started having the following error on our push server. I
have tried reconfigure the push server, reinstalled the push server,
upgrade all Chef packages to the latest versions, but still have the
following errors
2015-06-04_04:20:51.27833 04:20:51.170 [error] (Undefined:Undefined)
<0.328.0> CRASH REPORT Process <0.328.0> with 0 neighbours exited with
reason: {error,invalid_password} in gen_server:init_it/6 line 320
2015-06-04_04:20:51.28245 04:20:51.170 [error] (Undefined:Undefined)
<0.277.0> pool 'sqerl' failed to start member:
{error,{error,invalid_password}}
2015-06-04_04:20:51.28729 04:20:51.175 [error] (Undefined:Undefined)
<0.332.0> Unable to start database connection: {error,invalid_password}
2015-06-04_04:20:51.28729
2015-06-04_04:20:51.29202 04:20:51.287 [error] (Undefined:Undefined)
<0.332.0> CRASH REPORT Process <0.332.0> with 0 neighbours exited with
reason: {error,invalid_password} in gen_server:init_it/6 line 320
2015-06-04_04:20:56.09388 04:20:56.093 [warning] (Undefined:Undefined)
<0.6.0> lager_error_logger_h dropped 43 messages in the last second that
exceeded the limit of 50 messages/sec
2015-06-04_04:20:56.09391 04:20:56.093 [info] (Undefined:Undefined) <0.7.0>
Application pushy exited with reason:
{shutdown,{pushy_app,start,[normal,]}}
2015-06-04_04:20:57.62211 {"Kernel pid
terminated",application_controller,"{application_start_failure,pushy,{shutdown,{pushy_app,start,[normal,]}}}"}
2015-06-04_04:20:57.62302 Assertion failed: ok (mailbox.cpp:84)
2015-06-04_04:20:57.62303 Assertion failed: pfd.revents & POLLIN
(signaler.cpp:155)
It causes the server returns error "504 Gateway Time-out" now to all push
job api.
How can I fix this error to our push server? I assume I need to reset the
password for chef-push_user, but couldn't figure it out how to do it.
Can someone help me please
Thanks so much.

Jing Li
Senior .Net Developer, The Network for Learning Ltd
D: +64 9 222 0170 | W:
A: Suite 306, Geyser Building, 100 Parnell Road, Parnell, Auckland 1052
P: PO Box 37118, Parnell, Auckland 1151
http://www.n4l.co.nz
https://twitter.com/@N4LNZ/ https://www.facebook.com/NetworkForLearning
https://plus.google.com/104649004276520409744/about
http://www.linkedin.com/company/the-network-for-learning-ltd-n4l-

3

Thanks, I just managed to update the passwords in PostgreSql db manually
according to the PushServer config file. I did run reconfigure in order to
upgrade the push-server package to the latest one, but it was yesterday.

Thanks

On Thu, Jun 4, 2015 at 5:20 PM, Stephen Delano stephen@opscode.com wrote:

This is an error with the Push Server application connecting to the
PostgreSQL server it uses for storing state. Did any of this state recently
change? Did you run a push-server-ctl reconfigure recently?

—
Stephen Delano - Engineering Lead, Chef

On Wed, Jun 3, 2015 at 9:27 PM, Jing Li jing.li@n4l.co.nz wrote:

Hi There,

Today we suddenly started having the following error on our push server.
I have tried reconfigure the push server, reinstalled the push server,
upgrade all Chef packages to the latest versions, but still have the
following errors

2015-06-04_04:20:51.27833 04:20:51.170 [error] (Undefined:Undefined)
<0.328.0> CRASH REPORT Process <0.328.0> with 0 neighbours exited with
reason: {error,invalid_password} in gen_server:init_it/6 line 320
2015-06-04_04:20:51.28245 04:20:51.170 [error] (Undefined:Undefined)
<0.277.0> pool 'sqerl' failed to start member:
{error,{error,invalid_password}}
2015-06-04_04:20:51.28729 04:20:51.175 [error] (Undefined:Undefined)
<0.332.0> Unable to start database connection: {error,invalid_password}
2015-06-04_04:20:51.28729
2015-06-04_04:20:51.29202 04:20:51.287 [error] (Undefined:Undefined)
<0.332.0> CRASH REPORT Process <0.332.0> with 0 neighbours exited with
reason: {error,invalid_password} in gen_server:init_it/6 line 320
2015-06-04_04:20:56.09388 04:20:56.093 [warning] (Undefined:Undefined)
<0.6.0> lager_error_logger_h dropped 43 messages in the last second that
exceeded the limit of 50 messages/sec
2015-06-04_04:20:56.09391 04:20:56.093 [info] (Undefined:Undefined)
<0.7.0> Application pushy exited with reason:
{shutdown,{pushy_app,start,[normal,]}}
2015-06-04_04:20:57.62211 {"Kernel pid
terminated",application_controller,"{application_start_failure,pushy,{shutdown,{pushy_app,start,[normal,]}}}"}
2015-06-04_04:20:57.62302 Assertion failed: ok (mailbox.cpp:84)
2015-06-04_04:20:57.62303 Assertion failed: pfd.revents & POLLIN
(signaler.cpp:155)

It causes the server returns error "504 Gateway Time-out" now to all
push job api.

How can I fix this error to our push server? I assume I need to reset the
password for chef-push_user, but couldn't figure it out how to do it.

Can someone help me please

Thanks so much.

--

Jing Li
Senior .Net Developer, The Network for Learning Ltd

D: +64 9 222 0170 | W:
A: Suite 306, Geyser Building, 100 Parnell Road, Parnell, Auckland 1052
P: PO Box 37118, Parnell, Auckland 1151

http://www.n4l.co.nz
https://twitter.com/@N4LNZ/
https://www.facebook.com/NetworkForLearning
https://plus.google.com/104649004276520409744/about
http://www.linkedin.com/company/the-network-for-learning-ltd-n4l-

--

Jing Li
Senior .Net Developer, The Network for Learning Ltd

D: +64 9 222 0170 | W:
A: Suite 306, Geyser Building, 100 Parnell Road, Parnell, Auckland 1052
P: PO Box 37118, Parnell, Auckland 1151

http://www.n4l.co.nz
https://twitter.com/@N4LNZ/ https://www.facebook.com/NetworkForLearning
https://plus.google.com/104649004276520409744/about
http://www.linkedin.com/company/the-network-for-learning-ltd-n4l-