Problem with "chef-sync-ctl sync-log": Unauthorized


#1

Hello Guys,

I’m testing the replication feature
https://docs.getchef.com/server/server_replication.html

Package: chef-server-core-12.0.0_rc.5-1.el5.x86_64.rpm
OS: CentOS release 6.5
chef12-server1 = master / us-east-1
chef12-server2 = replica / sa-east-1

The problem is:

============

[root@chef12-server2 ~]# chef-sync-ctl sync-log test

Logfile created on 2014-10-28 14:26:03 +0000 by logger.rb/31641

I, [2014-10-28T14:26:03.073474 #14407] INFO – : ec_sync_client: worker
for
https://chef12-server1.datac.test.com/organizations/test/organizations/test
-> https://127.0.0.1/organizations/test started
I, [2014-10-28T14:29:04.326504 #15188] INFO – : ec_sync_client: worker
for
https://chef12-server1.datac.test.com/organizations/test/organizations/test
-> https://127.0.0.1/organizations/test started
I, [2014-10-28T14:29:04.881830 #15188] INFO – : HTTP Request Returned 404
Not Found:
I, [2014-10-28T14:32:03.081657 #16173] INFO – : ec_sync_client: worker
for https://chef12-server1.datac.test.com/organizations/test ->
https://127.0.0.1/organizations/test started
I, [2014-10-28T14:32:03.697880 #16173] INFO – : Bootrapping started.
I, [2014-10-28T14:32:04.681163 #16173] INFO – : HTTP Request Returned 401
Unauthorized: error
I, [2014-10-28T14:32:05.294182 #16173] INFO – : HTTP Request Returned 401
Unauthorized: error
I, [2014-10-28T14:32:05.927196 #16173] INFO – : HTTP Request Returned 401
Unauthorized: error
I, [2014-10-28T14:32:06.564459 #16173] INFO – : HTTP Request Returned 401
Unauthorized: error
I, [2014-10-28T14:33:52.138190 #16938] INFO – : ec_sync_client: worker
for https://chef12-server1.datac.test.com/organizations/test ->
https://127.0.0.1/organizations/test started
I, [2014-10-28T14:33:52.756533 #16938] INFO – : Bootrapping started.
I, [2014-10-28T14:33:53.743989 #16938] INFO – : HTTP Request Returned 401
Unauthorized: error
I, [2014-10-28T14:33:54.377393 #16938] INFO – : HTTP Request Returned 401
Unauthorized: error
I, [2014-10-28T14:33:54.998467 #16938] INFO – : HTTP Request Returned 401
Unauthorized: error
I, [2014-10-28T14:33:55.630469 #16938] INFO – : HTTP Request Returned 401
Unauthorized: error
I, [2014-10-28T14:42:12.312655 #8098] INFO – : ec_sync_client: worker for
https://chef12-server1.datac.test.com/organizations/test ->
https://127.0.0.1/organizations/test started
I, [2014-10-28T14:42:12.925787 #8098] INFO – : Bootrapping started.
I, [2014-10-28T14:42:14.763716 #8098] INFO – : HTTP Request Returned 401
Unauthorized: error
I, [2014-10-28T14:42:15.422438 #8098] INFO – : HTTP Request Returned 401
Unauthorized: error
I, [2014-10-28T14:42:16.066673 #8098] INFO – : HTTP Request Returned 401
Unauthorized: error
I, [2014-10-28T14:42:16.664576 #8098] INFO – : HTTP Request Returned 401
Unauthorized: error

============

The /etc/chef-sync/ec_sync_user.pem is the same on both machines (generated
on master “server1” copied to replica “server2”), and the command
"chef-sync-ctl prepare-org test" was executed on both sides.

Also, the “show config” does not work, both in server1 or in server2:

[root@chef12-server2 ~]# chef-sync-ctl show-config
Starting Chef Client, version 11.12.2
Compiling Cookbooks…

================================================================================
Recipe Compile Error

Chef::Exceptions::RecipeNotFound

could not find recipe show_config for cookbook chef-sync

Running handlers:
Running handlers complete

[2014-10-28T14:59:08+00:00] FATAL: Stacktrace dumped to
/opt/chef-sync/embedded/cookbooks/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated in 3.848546996 seconds
[2014-10-28T14:59:08+00:00] FATAL: Chef::Exceptions::ChildConvergeError:
Chef run process exited unsuccessfully (exit code 1)

Can you help me please?

Thanks a lot!


– Tiago Cruz


#2

Hi Tiago,

The show-config issue looks like a bug we should address. I’ll make
sure that gets filed. As for the 401, your instinct to look at the
key is correct. Could you try the following to see if the key works
outside of the sync service:

From the replica:

/opt/chef-server/embedded/bin/knife node list -s
https://chef12-server1.datac.test.com/organizations/test -u
ec_sync_user -k /etc/chef-sync/ec_sync_user.pem
/opt/chef-server/embedded/bin/knife node list -s
https://localhost/organizations/test -u ec_sync_user -k
/etc/chef-sync/ec_sync_user.pem

Both of those command should succeed without error. Could you also
post the output of

chef-sync-ctl manager-log

Cheers,

Steven

On Tue, Oct 28, 2014 at 3:02 PM, Tiago Cruz tiago.tuxkiller@gmail.com wrote:

Hello Guys,

I’m testing the replication feature
https://docs.getchef.com/server/server_replication.html

Package: chef-server-core-12.0.0_rc.5-1.el5.x86_64.rpm
OS: CentOS release 6.5
chef12-server1 = master / us-east-1
chef12-server2 = replica / sa-east-1

The problem is:

============

[root@chef12-server2 ~]# chef-sync-ctl sync-log test

Logfile created on 2014-10-28 14:26:03 +0000 by logger.rb/31641

I, [2014-10-28T14:26:03.073474 #14407] INFO – : ec_sync_client: worker for
https://chef12-server1.datac.test.com/organizations/test/organizations/test
-> https://127.0.0.1/organizations/test started
I, [2014-10-28T14:29:04.326504 #15188] INFO – : ec_sync_client: worker for
https://chef12-server1.datac.test.com/organizations/test/organizations/test
-> https://127.0.0.1/organizations/test started
I, [2014-10-28T14:29:04.881830 #15188] INFO – : HTTP Request Returned 404
Not Found:
I, [2014-10-28T14:32:03.081657 #16173] INFO – : ec_sync_client: worker for
https://chef12-server1.datac.test.com/organizations/test ->
https://127.0.0.1/organizations/test started
I, [2014-10-28T14:32:03.697880 #16173] INFO – : Bootrapping started.
I, [2014-10-28T14:32:04.681163 #16173] INFO – : HTTP Request Returned 401
Unauthorized: error
I, [2014-10-28T14:32:05.294182 #16173] INFO – : HTTP Request Returned 401
Unauthorized: error
I, [2014-10-28T14:32:05.927196 #16173] INFO – : HTTP Request Returned 401
Unauthorized: error
I, [2014-10-28T14:32:06.564459 #16173] INFO – : HTTP Request Returned 401
Unauthorized: error
I, [2014-10-28T14:33:52.138190 #16938] INFO – : ec_sync_client: worker for
https://chef12-server1.datac.test.com/organizations/test ->
https://127.0.0.1/organizations/test started
I, [2014-10-28T14:33:52.756533 #16938] INFO – : Bootrapping started.
I, [2014-10-28T14:33:53.743989 #16938] INFO – : HTTP Request Returned 401
Unauthorized: error
I, [2014-10-28T14:33:54.377393 #16938] INFO – : HTTP Request Returned 401
Unauthorized: error
I, [2014-10-28T14:33:54.998467 #16938] INFO – : HTTP Request Returned 401
Unauthorized: error
I, [2014-10-28T14:33:55.630469 #16938] INFO – : HTTP Request Returned 401
Unauthorized: error
I, [2014-10-28T14:42:12.312655 #8098] INFO – : ec_sync_client: worker for
https://chef12-server1.datac.test.com/organizations/test ->
https://127.0.0.1/organizations/test started
I, [2014-10-28T14:42:12.925787 #8098] INFO – : Bootrapping started.
I, [2014-10-28T14:42:14.763716 #8098] INFO – : HTTP Request Returned 401
Unauthorized: error
I, [2014-10-28T14:42:15.422438 #8098] INFO – : HTTP Request Returned 401
Unauthorized: error
I, [2014-10-28T14:42:16.066673 #8098] INFO – : HTTP Request Returned 401
Unauthorized: error
I, [2014-10-28T14:42:16.664576 #8098] INFO – : HTTP Request Returned 401
Unauthorized: error

============

The /etc/chef-sync/ec_sync_user.pem is the same on both machines (generated
on master “server1” copied to replica “server2”), and the command
"chef-sync-ctl prepare-org test" was executed on both sides.

Also, the “show config” does not work, both in server1 or in server2:

[root@chef12-server2 ~]# chef-sync-ctl show-config
Starting Chef Client, version 11.12.2
Compiling Cookbooks…

================================================================================
Recipe Compile Error

Chef::Exceptions::RecipeNotFound

could not find recipe show_config for cookbook chef-sync

Running handlers:
Running handlers complete

[2014-10-28T14:59:08+00:00] FATAL: Stacktrace dumped to
/opt/chef-sync/embedded/cookbooks/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated in 3.848546996 seconds
[2014-10-28T14:59:08+00:00] FATAL: Chef::Exceptions::ChildConvergeError:
Chef run process exited unsuccessfully (exit code 1)

Can you help me please?

Thanks a lot!


– Tiago Cruz


#3

Hello Steven, nice tip thanks!

Well, I don’t have /opt/chef-server directory, but I used another
"/bin/knife" instead, I hope you don’t mind :slight_smile:

[root@chef12-server1 ~]# find /opt/ -name ‘knife’ | egrep ‘/bin/knife$’ |
wc -l
17

And you’re right the user/key does not match on replica host:

[root@chef12-server1 ~]# /opt/chef/bin/knife user list -s
https://localhost/organizations/test -u ec_sync_user -k
/etc/chef-sync/ec_sync_user.pem
WARNING: No knife configuration file found
ec_sync_user
tiago_cruz

[root@chef12-server2 ~]# /opt/chef/bin/knife user list -s
https://localhost/organizations/test -u ec_sync_user -k
/etc/chef-sync/ec_sync_user.pem
WARNING: No knife configuration file found
ERROR: Failed to authenticate to https://localhost/organizations/test as
ec_sync_user with key /etc/chef-sync/ec_sync_user.pem
Response: Invalid signature for user or client ‘ec_sync_user’

Strange, 'cause I can see the creation while I run the reconfigure:

[root@chef12-server2 ~]# chef-sync-ctl reconfigure

Recipe: chef-sync::sync_user

  • chef_user[ec_sync_user] action create (up to date)

So, how can I fix this?

Thanks a lot!

On Tue, Oct 28, 2014 at 2:00 PM, Steven Danna steve@opscode.com wrote:

Hi Tiago,

The show-config issue looks like a bug we should address. I’ll make
sure that gets filed. As for the 401, your instinct to look at the
key is correct. Could you try the following to see if the key works
outside of the sync service:

From the replica:

/opt/chef-server/embedded/bin/knife node list -s
https://chef12-server1.datac.test.com/organizations/test -u
ec_sync_user -k /etc/chef-sync/ec_sync_user.pem
/opt/chef-server/embedded/bin/knife node list -s
https://localhost/organizations/test -u ec_sync_user -k
/etc/chef-sync/ec_sync_user.pem

Both of those command should succeed without error. Could you also
post the output of

chef-sync-ctl manager-log

Cheers,

Steven

On Tue, Oct 28, 2014 at 3:02 PM, Tiago Cruz tiago.tuxkiller@gmail.com
wrote:

Hello Guys,

I’m testing the replication feature
https://docs.getchef.com/server/server_replication.html

Package: chef-server-core-12.0.0_rc.5-1.el5.x86_64.rpm
OS: CentOS release 6.5
chef12-server1 = master / us-east-1
chef12-server2 = replica / sa-east-1

The problem is:

============

[root@chef12-server2 ~]# chef-sync-ctl sync-log test

Logfile created on 2014-10-28 14:26:03 +0000 by logger.rb/31641

I, [2014-10-28T14:26:03.073474 #14407] INFO – : ec_sync_client: worker
for

https://chef12-server1.datac.test.com/organizations/test/organizations/test

-> https://127.0.0.1/organizations/test started
I, [2014-10-28T14:29:04.326504 #15188] INFO – : ec_sync_client: worker
for

https://chef12-server1.datac.test.com/organizations/test/organizations/test

-> https://127.0.0.1/organizations/test started
I, [2014-10-28T14:29:04.881830 #15188] INFO – : HTTP Request Returned
404
Not Found:
I, [2014-10-28T14:32:03.081657 #16173] INFO – : ec_sync_client: worker
for
https://chef12-server1.datac.test.com/organizations/test ->
https://127.0.0.1/organizations/test started
I, [2014-10-28T14:32:03.697880 #16173] INFO – : Bootrapping started.
I, [2014-10-28T14:32:04.681163 #16173] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:32:05.294182 #16173] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:32:05.927196 #16173] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:32:06.564459 #16173] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:33:52.138190 #16938] INFO – : ec_sync_client: worker
for
https://chef12-server1.datac.test.com/organizations/test ->
https://127.0.0.1/organizations/test started
I, [2014-10-28T14:33:52.756533 #16938] INFO – : Bootrapping started.
I, [2014-10-28T14:33:53.743989 #16938] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:33:54.377393 #16938] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:33:54.998467 #16938] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:33:55.630469 #16938] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:42:12.312655 #8098] INFO – : ec_sync_client: worker
for
https://chef12-server1.datac.test.com/organizations/test ->
https://127.0.0.1/organizations/test started
I, [2014-10-28T14:42:12.925787 #8098] INFO – : Bootrapping started.
I, [2014-10-28T14:42:14.763716 #8098] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:42:15.422438 #8098] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:42:16.066673 #8098] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:42:16.664576 #8098] INFO – : HTTP Request Returned
401
Unauthorized: error

============

The /etc/chef-sync/ec_sync_user.pem is the same on both machines
(generated
on master “server1” copied to replica “server2”), and the command
"chef-sync-ctl prepare-org test" was executed on both sides.

Also, the “show config” does not work, both in server1 or in server2:

[root@chef12-server2 ~]# chef-sync-ctl show-config
Starting Chef Client, version 11.12.2
Compiling Cookbooks…

================================================================================

Recipe Compile Error

================================================================================

Chef::Exceptions::RecipeNotFound

could not find recipe show_config for cookbook chef-sync

Running handlers:
Running handlers complete

[2014-10-28T14:59:08+00:00] FATAL: Stacktrace dumped to
/opt/chef-sync/embedded/cookbooks/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated in 3.848546996 seconds
[2014-10-28T14:59:08+00:00] FATAL: Chef::Exceptions::ChildConvergeError:
Chef run process exited unsuccessfully (exit code 1)

Can you help me please?

Thanks a lot!


– Tiago Cruz


– Tiago Cruz


#4

Sorry, I forgot to give you the output of “manager-log” command:

Master:

[tiago.cruz@chef12-server1 ~]$ sudo chef-sync-ctl manager-log
2014-10-28_14:20:14.37022 I, [2014-10-28T14:20:14.370091 #7976] INFO – :
ec_sync_client: manager starting
2014-10-28_14:20:14.37034 I, [2014-10-28T14:20:14.370298 #7976] INFO – :
ec_sync_client: listening on /var/opt/chef-sync/ec_sync_client/ec_sync.sock
2014-10-28_14:20:14.61700 I, [2014-10-28T14:20:14.616912 #7976] INFO – :
Sending fast shutdown to workers
2014-10-28_14:20:14.61704 I, [2014-10-28T14:20:14.617003 #7976] INFO – :
Waiting for workers to stop
2014-10-28_14:20:14.61708 I, [2014-10-28T14:20:14.617045 #7976] INFO – :
Shutdown complete.
2014-10-28_14:20:16.32615 I, [2014-10-28T14:20:16.326040 #7982] INFO – :
ec_sync_client: manager starting
2014-10-28_14:20:16.32626 I, [2014-10-28T14:20:16.326215 #7982] INFO – :
ec_sync_client: listening on /var/opt/chef-sync/ec_sync_client/ec_sync.sock
2014-10-28_14:20:18.57771 I, [2014-10-28T14:20:18.575169 #7982] INFO – :
Sending fast shutdown to workers
2014-10-28_14:20:18.57778 I, [2014-10-28T14:20:18.575246 #7982] INFO – :
Waiting for workers to stop
2014-10-28_14:20:18.57778 I, [2014-10-28T14:20:18.575270 #7982] INFO – :
Shutdown complete.
2014-10-28_14:20:19.34221 I, [2014-10-28T14:20:19.342121 #8045] INFO – :
ec_sync_client: manager starting
2014-10-28_14:20:19.34236 I, [2014-10-28T14:20:19.342334 #8045] INFO – :
ec_sync_client: listening on /var/opt/chef-sync/ec_sync_client/ec_sync.sock
2014-10-28_14:22:25.64480 I, [2014-10-28T14:22:25.644691 #8045] INFO – :
Sending fast shutdown to workers
2014-10-28_14:22:25.64482 I, [2014-10-28T14:22:25.644803 #8045] INFO – :
Waiting for workers to stop
2014-10-28_14:22:25.64486 I, [2014-10-28T14:22:25.644830 #8045] INFO – :
Shutdown complete.

======================

Replica:

[tiago.cruz@chef12-server2 ~]$ sudo chef-sync-ctl manager-log
2014-10-28_14:42:16.66748 from
/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/file_system.rb:426:in
parallel_do' 2014-10-28_14:42:16.66752 from /opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/file_system.rb:360:incopy_entries’
2014-10-28_14:42:16.66756 from
/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/file_system.rb:145:in
block in copy_to' 2014-10-28_14:42:16.66760 from /opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:267:incall’
2014-10-28_14:42:16.66764 from
/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:267:in
process_input' 2014-10-28_14:42:16.66767 from /opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:257:inprocess_one’
2014-10-28_14:42:16.66771 from
/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:195:in
each_with_exceptions_unordered' 2014-10-28_14:42:16.66775 from /opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:74:inwait’
2014-10-28_14:42:16.66779 from
/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer.rb:48:in
parallel_do' 2014-10-28_14:42:16.66783 from /opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer.rb:29:inparallel_do’
2014-10-28_14:42:16.66787 from
/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/file_system.rb:426:in
parallel_do' 2014-10-28_14:42:16.66792 from /opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/file_system.rb:142:incopy_to’
2014-10-28_14:42:16.66796 from
/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/orgsyncer.rb:132:in
bootstrap' 2014-10-28_14:42:16.66800 from /opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/orgsyncer.rb:54:instart’
2014-10-28_14:42:16.66804 from
/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:114:in
block in start_worker' 2014-10-28_14:42:16.66808 from /opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:110:infork’
2014-10-28_14:42:16.66812 from
/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:110:in
start_worker' 2014-10-28_14:42:16.66816 from /opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:69:inblock in start’
2014-10-28_14:42:16.66906 from
/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:66:in
each' 2014-10-28_14:42:16.66911 from /opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:66:instart’
2014-10-28_14:42:16.66915 from
/opt/chef-sync/embedded/service/ec_sync_client/bin/ec_sync_client:56:in
run' 2014-10-28_14:42:16.66919 from /opt/chef-sync/embedded/service/ec_sync_client/bin/ec_sync_client:62:in
2014-10-28_14:42:16.68830 I, [2014-10-28T14:42:16.688096 #8095] INFO – :
Worker for test died unexpectedly!
2014-10-28_14:44:04.81704 I, [2014-10-28T14:44:04.816897 #8095] INFO – :
ec_sync_client: starting new command connection handler
2014-10-28_14:44:04.82802 I, [2014-10-28T14:44:04.827847 #8095] INFO – :
ec_sync_client: closing command connection handler

On Tue, Oct 28, 2014 at 3:05 PM, Tiago Cruz tiago.tuxkiller@gmail.com
wrote:

Hello Steven, nice tip thanks!

Well, I don’t have /opt/chef-server directory, but I used another
"/bin/knife" instead, I hope you don’t mind :slight_smile:

[root@chef12-server1 ~]# find /opt/ -name ‘knife’ | egrep ‘/bin/knife$’ |
wc -l
17

And you’re right the user/key does not match on replica host:

[root@chef12-server1 ~]# /opt/chef/bin/knife user list -s
https://localhost/organizations/test -u ec_sync_user -k
/etc/chef-sync/ec_sync_user.pem
WARNING: No knife configuration file found
ec_sync_user
tiago_cruz

[root@chef12-server2 ~]# /opt/chef/bin/knife user list -s
https://localhost/organizations/test -u ec_sync_user -k
/etc/chef-sync/ec_sync_user.pem
WARNING: No knife configuration file found
ERROR: Failed to authenticate to https://localhost/organizations/test as
ec_sync_user with key /etc/chef-sync/ec_sync_user.pem
Response: Invalid signature for user or client ‘ec_sync_user’

Strange, 'cause I can see the creation while I run the reconfigure:

[root@chef12-server2 ~]# chef-sync-ctl reconfigure

Recipe: chef-sync::sync_user

  • chef_user[ec_sync_user] action create (up to date)

So, how can I fix this?

Thanks a lot!

On Tue, Oct 28, 2014 at 2:00 PM, Steven Danna steve@opscode.com wrote:

Hi Tiago,

The show-config issue looks like a bug we should address. I’ll make
sure that gets filed. As for the 401, your instinct to look at the
key is correct. Could you try the following to see if the key works
outside of the sync service:

From the replica:

/opt/chef-server/embedded/bin/knife node list -s
https://chef12-server1.datac.test.com/organizations/test -u
ec_sync_user -k /etc/chef-sync/ec_sync_user.pem
/opt/chef-server/embedded/bin/knife node list -s
https://localhost/organizations/test -u ec_sync_user -k
/etc/chef-sync/ec_sync_user.pem

Both of those command should succeed without error. Could you also
post the output of

chef-sync-ctl manager-log

Cheers,

Steven

On Tue, Oct 28, 2014 at 3:02 PM, Tiago Cruz tiago.tuxkiller@gmail.com
wrote:

Hello Guys,

I’m testing the replication feature
https://docs.getchef.com/server/server_replication.html

Package: chef-server-core-12.0.0_rc.5-1.el5.x86_64.rpm
OS: CentOS release 6.5
chef12-server1 = master / us-east-1
chef12-server2 = replica / sa-east-1

The problem is:

============

[root@chef12-server2 ~]# chef-sync-ctl sync-log test

Logfile created on 2014-10-28 14:26:03 +0000 by logger.rb/31641

I, [2014-10-28T14:26:03.073474 #14407] INFO – : ec_sync_client:
worker for

https://chef12-server1.datac.test.com/organizations/test/organizations/test

-> https://127.0.0.1/organizations/test started
I, [2014-10-28T14:29:04.326504 #15188] INFO – : ec_sync_client:
worker for

https://chef12-server1.datac.test.com/organizations/test/organizations/test

-> https://127.0.0.1/organizations/test started
I, [2014-10-28T14:29:04.881830 #15188] INFO – : HTTP Request Returned
404
Not Found:
I, [2014-10-28T14:32:03.081657 #16173] INFO – : ec_sync_client:
worker for
https://chef12-server1.datac.test.com/organizations/test ->
https://127.0.0.1/organizations/test started
I, [2014-10-28T14:32:03.697880 #16173] INFO – : Bootrapping started.
I, [2014-10-28T14:32:04.681163 #16173] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:32:05.294182 #16173] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:32:05.927196 #16173] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:32:06.564459 #16173] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:33:52.138190 #16938] INFO – : ec_sync_client:
worker for
https://chef12-server1.datac.test.com/organizations/test ->
https://127.0.0.1/organizations/test started
I, [2014-10-28T14:33:52.756533 #16938] INFO – : Bootrapping started.
I, [2014-10-28T14:33:53.743989 #16938] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:33:54.377393 #16938] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:33:54.998467 #16938] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:33:55.630469 #16938] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:42:12.312655 #8098] INFO – : ec_sync_client: worker
for
https://chef12-server1.datac.test.com/organizations/test ->
https://127.0.0.1/organizations/test started
I, [2014-10-28T14:42:12.925787 #8098] INFO – : Bootrapping started.
I, [2014-10-28T14:42:14.763716 #8098] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:42:15.422438 #8098] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:42:16.066673 #8098] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:42:16.664576 #8098] INFO – : HTTP Request Returned
401
Unauthorized: error

============

The /etc/chef-sync/ec_sync_user.pem is the same on both machines
(generated
on master “server1” copied to replica “server2”), and the command
"chef-sync-ctl prepare-org test" was executed on both sides.

Also, the “show config” does not work, both in server1 or in server2:

[root@chef12-server2 ~]# chef-sync-ctl show-config
Starting Chef Client, version 11.12.2
Compiling Cookbooks…

================================================================================

Recipe Compile Error

================================================================================

Chef::Exceptions::RecipeNotFound

could not find recipe show_config for cookbook chef-sync

Running handlers:
Running handlers complete

[2014-10-28T14:59:08+00:00] FATAL: Stacktrace dumped to
/opt/chef-sync/embedded/cookbooks/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated in 3.848546996 seconds
[2014-10-28T14:59:08+00:00] FATAL: Chef::Exceptions::ChildConvergeError:
Chef run process exited unsuccessfully (exit code 1)

Can you help me please?

Thanks a lot!


– Tiago Cruz


– Tiago Cruz


– Tiago Cruz


#5

Hi,

On chef12-server2 first make sure that /etc/chef-sync/ec_sync_user.pem
has the same md5sum as the one on server1. If it does, then try this
on server2:

chef-server-ctl user-delete ec_sync_user
chef-sync-ctl reconfigure

Cheers,

Steven

On Tue, Oct 28, 2014 at 5:27 PM, Tiago Cruz tiago.tuxkiller@gmail.com wrote:

Sorry, I forgot to give you the output of “manager-log” command:

Master:

[tiago.cruz@chef12-server1 ~]$ sudo chef-sync-ctl manager-log
2014-10-28_14:20:14.37022 I, [2014-10-28T14:20:14.370091 #7976] INFO – :
ec_sync_client: manager starting
2014-10-28_14:20:14.37034 I, [2014-10-28T14:20:14.370298 #7976] INFO – :
ec_sync_client: listening on /var/opt/chef-sync/ec_sync_client/ec_sync.sock
2014-10-28_14:20:14.61700 I, [2014-10-28T14:20:14.616912 #7976] INFO – :
Sending fast shutdown to workers
2014-10-28_14:20:14.61704 I, [2014-10-28T14:20:14.617003 #7976] INFO – :
Waiting for workers to stop
2014-10-28_14:20:14.61708 I, [2014-10-28T14:20:14.617045 #7976] INFO – :
Shutdown complete.
2014-10-28_14:20:16.32615 I, [2014-10-28T14:20:16.326040 #7982] INFO – :
ec_sync_client: manager starting
2014-10-28_14:20:16.32626 I, [2014-10-28T14:20:16.326215 #7982] INFO – :
ec_sync_client: listening on /var/opt/chef-sync/ec_sync_client/ec_sync.sock
2014-10-28_14:20:18.57771 I, [2014-10-28T14:20:18.575169 #7982] INFO – :
Sending fast shutdown to workers
2014-10-28_14:20:18.57778 I, [2014-10-28T14:20:18.575246 #7982] INFO – :
Waiting for workers to stop
2014-10-28_14:20:18.57778 I, [2014-10-28T14:20:18.575270 #7982] INFO – :
Shutdown complete.
2014-10-28_14:20:19.34221 I, [2014-10-28T14:20:19.342121 #8045] INFO – :
ec_sync_client: manager starting
2014-10-28_14:20:19.34236 I, [2014-10-28T14:20:19.342334 #8045] INFO – :
ec_sync_client: listening on /var/opt/chef-sync/ec_sync_client/ec_sync.sock
2014-10-28_14:22:25.64480 I, [2014-10-28T14:22:25.644691 #8045] INFO – :
Sending fast shutdown to workers
2014-10-28_14:22:25.64482 I, [2014-10-28T14:22:25.644803 #8045] INFO – :
Waiting for workers to stop
2014-10-28_14:22:25.64486 I, [2014-10-28T14:22:25.644830 #8045] INFO – :
Shutdown complete.

======================

Replica:

[tiago.cruz@chef12-server2 ~]$ sudo chef-sync-ctl manager-log
2014-10-28_14:42:16.66748 from
/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/file_system.rb:426:in
parallel_do' 2014-10-28_14:42:16.66752 from /opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/file_system.rb:360:incopy_entries’
2014-10-28_14:42:16.66756 from
/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/file_system.rb:145:in
block in copy_to' 2014-10-28_14:42:16.66760 from /opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:267:incall’
2014-10-28_14:42:16.66764 from
/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:267:in
process_input' 2014-10-28_14:42:16.66767 from /opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:257:inprocess_one’
2014-10-28_14:42:16.66771 from
/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:195:in
each_with_exceptions_unordered' 2014-10-28_14:42:16.66775 from /opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:74:inwait’
2014-10-28_14:42:16.66779 from
/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer.rb:48:in
parallel_do' 2014-10-28_14:42:16.66783 from /opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer.rb:29:inparallel_do’
2014-10-28_14:42:16.66787 from
/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/file_system.rb:426:in
parallel_do' 2014-10-28_14:42:16.66792 from /opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/file_system.rb:142:incopy_to’
2014-10-28_14:42:16.66796 from
/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/orgsyncer.rb:132:in
bootstrap' 2014-10-28_14:42:16.66800 from /opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/orgsyncer.rb:54:instart’
2014-10-28_14:42:16.66804 from
/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:114:in
block in start_worker' 2014-10-28_14:42:16.66808 from /opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:110:infork’
2014-10-28_14:42:16.66812 from
/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:110:in
start_worker' 2014-10-28_14:42:16.66816 from /opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:69:inblock in start’
2014-10-28_14:42:16.66906 from
/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:66:in
each' 2014-10-28_14:42:16.66911 from /opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:66:instart’
2014-10-28_14:42:16.66915 from
/opt/chef-sync/embedded/service/ec_sync_client/bin/ec_sync_client:56:in
run' 2014-10-28_14:42:16.66919 from /opt/chef-sync/embedded/service/ec_sync_client/bin/ec_sync_client:62:in
2014-10-28_14:42:16.68830 I, [2014-10-28T14:42:16.688096 #8095] INFO – :
Worker for test died unexpectedly!
2014-10-28_14:44:04.81704 I, [2014-10-28T14:44:04.816897 #8095] INFO – :
ec_sync_client: starting new command connection handler
2014-10-28_14:44:04.82802 I, [2014-10-28T14:44:04.827847 #8095] INFO – :
ec_sync_client: closing command connection handler

On Tue, Oct 28, 2014 at 3:05 PM, Tiago Cruz tiago.tuxkiller@gmail.com
wrote:

Hello Steven, nice tip thanks!

Well, I don’t have /opt/chef-server directory, but I used another
"/bin/knife" instead, I hope you don’t mind :slight_smile:

[root@chef12-server1 ~]# find /opt/ -name ‘knife’ | egrep ‘/bin/knife$’ |
wc -l
17

And you’re right the user/key does not match on replica host:

[root@chef12-server1 ~]# /opt/chef/bin/knife user list -s
https://localhost/organizations/test -u ec_sync_user -k
/etc/chef-sync/ec_sync_user.pem
WARNING: No knife configuration file found
ec_sync_user
tiago_cruz

[root@chef12-server2 ~]# /opt/chef/bin/knife user list -s
https://localhost/organizations/test -u ec_sync_user -k
/etc/chef-sync/ec_sync_user.pem
WARNING: No knife configuration file found
ERROR: Failed to authenticate to https://localhost/organizations/test as
ec_sync_user with key /etc/chef-sync/ec_sync_user.pem
Response: Invalid signature for user or client ‘ec_sync_user’

Strange, 'cause I can see the creation while I run the reconfigure:

[root@chef12-server2 ~]# chef-sync-ctl reconfigure

Recipe: chef-sync::sync_user

  • chef_user[ec_sync_user] action create (up to date)

So, how can I fix this?

Thanks a lot!

On Tue, Oct 28, 2014 at 2:00 PM, Steven Danna steve@opscode.com wrote:

Hi Tiago,

The show-config issue looks like a bug we should address. I’ll make
sure that gets filed. As for the 401, your instinct to look at the
key is correct. Could you try the following to see if the key works
outside of the sync service:

From the replica:

/opt/chef-server/embedded/bin/knife node list -s
https://chef12-server1.datac.test.com/organizations/test -u
ec_sync_user -k /etc/chef-sync/ec_sync_user.pem
/opt/chef-server/embedded/bin/knife node list -s
https://localhost/organizations/test -u ec_sync_user -k
/etc/chef-sync/ec_sync_user.pem

Both of those command should succeed without error. Could you also
post the output of

chef-sync-ctl manager-log

Cheers,

Steven

On Tue, Oct 28, 2014 at 3:02 PM, Tiago Cruz tiago.tuxkiller@gmail.com
wrote:

Hello Guys,

I’m testing the replication feature
https://docs.getchef.com/server/server_replication.html

Package: chef-server-core-12.0.0_rc.5-1.el5.x86_64.rpm
OS: CentOS release 6.5
chef12-server1 = master / us-east-1
chef12-server2 = replica / sa-east-1

The problem is:

============

[root@chef12-server2 ~]# chef-sync-ctl sync-log test

Logfile created on 2014-10-28 14:26:03 +0000 by logger.rb/31641

I, [2014-10-28T14:26:03.073474 #14407] INFO – : ec_sync_client:
worker for

https://chef12-server1.datac.test.com/organizations/test/organizations/test
-> https://127.0.0.1/organizations/test started
I, [2014-10-28T14:29:04.326504 #15188] INFO – : ec_sync_client:
worker for

https://chef12-server1.datac.test.com/organizations/test/organizations/test
-> https://127.0.0.1/organizations/test started
I, [2014-10-28T14:29:04.881830 #15188] INFO – : HTTP Request Returned
404
Not Found:
I, [2014-10-28T14:32:03.081657 #16173] INFO – : ec_sync_client:
worker for
https://chef12-server1.datac.test.com/organizations/test ->
https://127.0.0.1/organizations/test started
I, [2014-10-28T14:32:03.697880 #16173] INFO – : Bootrapping started.
I, [2014-10-28T14:32:04.681163 #16173] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:32:05.294182 #16173] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:32:05.927196 #16173] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:32:06.564459 #16173] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:33:52.138190 #16938] INFO – : ec_sync_client:
worker for
https://chef12-server1.datac.test.com/organizations/test ->
https://127.0.0.1/organizations/test started
I, [2014-10-28T14:33:52.756533 #16938] INFO – : Bootrapping started.
I, [2014-10-28T14:33:53.743989 #16938] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:33:54.377393 #16938] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:33:54.998467 #16938] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:33:55.630469 #16938] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:42:12.312655 #8098] INFO – : ec_sync_client: worker
for
https://chef12-server1.datac.test.com/organizations/test ->
https://127.0.0.1/organizations/test started
I, [2014-10-28T14:42:12.925787 #8098] INFO – : Bootrapping started.
I, [2014-10-28T14:42:14.763716 #8098] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:42:15.422438 #8098] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:42:16.066673 #8098] INFO – : HTTP Request Returned
401
Unauthorized: error
I, [2014-10-28T14:42:16.664576 #8098] INFO – : HTTP Request Returned
401
Unauthorized: error

============

The /etc/chef-sync/ec_sync_user.pem is the same on both machines
(generated
on master “server1” copied to replica “server2”), and the command
"chef-sync-ctl prepare-org test" was executed on both sides.

Also, the “show config” does not work, both in server1 or in server2:

[root@chef12-server2 ~]# chef-sync-ctl show-config
Starting Chef Client, version 11.12.2
Compiling Cookbooks…

================================================================================
Recipe Compile Error

================================================================================

Chef::Exceptions::RecipeNotFound

could not find recipe show_config for cookbook chef-sync

Running handlers:
Running handlers complete

[2014-10-28T14:59:08+00:00] FATAL: Stacktrace dumped to
/opt/chef-sync/embedded/cookbooks/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated in 3.848546996 seconds
[2014-10-28T14:59:08+00:00] FATAL:
Chef::Exceptions::ChildConvergeError:
Chef run process exited unsuccessfully (exit code 1)

Can you help me please?

Thanks a lot!


– Tiago Cruz


– Tiago Cruz


– Tiago Cruz


#6

Hello again Steven, thanks for all your support.

I can’t remove the user:

[root@chef12-server2 ~]# chef-server-ctl user-delete ec_sync_user
Do you want to delete the user ec_sync_user? (Y/N)y
ERROR: You authenticated successfully to https://127.0.0.1:443 as pivotal
but you are not authorized for this action
Response: Members of an organization’s admins group cannot delete
themselves. Remove yourself from the admins group, then retry this
operation.

But I used the “cleanse” option (Delete all chef-sync data, and start
from scratch) and now the replication is working fine, thanks!

Cheers,
Tiago Cruz

On Tue, Oct 28, 2014 at 4:04 PM, Steven Danna steve@opscode.com wrote:

Hi,

On chef12-server2 first make sure that /etc/chef-sync/ec_sync_user.pem
has the same md5sum as the one on server1. If it does, then try this
on server2:

chef-server-ctl user-delete ec_sync_user
chef-sync-ctl reconfigure

Cheers,

Steven

On Tue, Oct 28, 2014 at 5:27 PM, Tiago Cruz tiago.tuxkiller@gmail.com
wrote:

Sorry, I forgot to give you the output of “manager-log” command:

Master:

[tiago.cruz@chef12-server1 ~]$ sudo chef-sync-ctl manager-log
2014-10-28_14:20:14.37022 I, [2014-10-28T14:20:14.370091 #7976] INFO –
:
ec_sync_client: manager starting
2014-10-28_14:20:14.37034 I, [2014-10-28T14:20:14.370298 #7976] INFO –
:
ec_sync_client: listening on
/var/opt/chef-sync/ec_sync_client/ec_sync.sock
2014-10-28_14:20:14.61700 I, [2014-10-28T14:20:14.616912 #7976] INFO –
:
Sending fast shutdown to workers
2014-10-28_14:20:14.61704 I, [2014-10-28T14:20:14.617003 #7976] INFO –
:
Waiting for workers to stop
2014-10-28_14:20:14.61708 I, [2014-10-28T14:20:14.617045 #7976] INFO –
:
Shutdown complete.
2014-10-28_14:20:16.32615 I, [2014-10-28T14:20:16.326040 #7982] INFO –
:
ec_sync_client: manager starting
2014-10-28_14:20:16.32626 I, [2014-10-28T14:20:16.326215 #7982] INFO –
:
ec_sync_client: listening on
/var/opt/chef-sync/ec_sync_client/ec_sync.sock
2014-10-28_14:20:18.57771 I, [2014-10-28T14:20:18.575169 #7982] INFO –
:
Sending fast shutdown to workers
2014-10-28_14:20:18.57778 I, [2014-10-28T14:20:18.575246 #7982] INFO –
:
Waiting for workers to stop
2014-10-28_14:20:18.57778 I, [2014-10-28T14:20:18.575270 #7982] INFO –
:
Shutdown complete.
2014-10-28_14:20:19.34221 I, [2014-10-28T14:20:19.342121 #8045] INFO –
:
ec_sync_client: manager starting
2014-10-28_14:20:19.34236 I, [2014-10-28T14:20:19.342334 #8045] INFO –
:
ec_sync_client: listening on
/var/opt/chef-sync/ec_sync_client/ec_sync.sock
2014-10-28_14:22:25.64480 I, [2014-10-28T14:22:25.644691 #8045] INFO –
:
Sending fast shutdown to workers
2014-10-28_14:22:25.64482 I, [2014-10-28T14:22:25.644803 #8045] INFO –
:
Waiting for workers to stop
2014-10-28_14:22:25.64486 I, [2014-10-28T14:22:25.644830 #8045] INFO –
:
Shutdown complete.

======================

Replica:

[tiago.cruz@chef12-server2 ~]$ sudo chef-sync-ctl manager-log
2014-10-28_14:42:16.66748 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/file_system.rb:426:in

`parallel_do’
2014-10-28_14:42:16.66752 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/file_system.rb:360:in

`copy_entries’
2014-10-28_14:42:16.66756 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/file_system.rb:145:in

`block in copy_to’
2014-10-28_14:42:16.66760 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:267:in

`call’
2014-10-28_14:42:16.66764 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:267:in

`process_input’
2014-10-28_14:42:16.66767 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:257:in

`process_one’
2014-10-28_14:42:16.66771 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:195:in

`each_with_exceptions_unordered’
2014-10-28_14:42:16.66775 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:74:in

`wait’
2014-10-28_14:42:16.66779 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer.rb:48:in

`parallel_do’
2014-10-28_14:42:16.66783 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer.rb:29:in

`parallel_do’
2014-10-28_14:42:16.66787 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/file_system.rb:426:in

`parallel_do’
2014-10-28_14:42:16.66792 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/file_system.rb:142:in

`copy_to’
2014-10-28_14:42:16.66796 from

/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/orgsyncer.rb:132:in

`bootstrap’
2014-10-28_14:42:16.66800 from

/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/orgsyncer.rb:54:in

`start’
2014-10-28_14:42:16.66804 from

/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:114:in

`block in start_worker’
2014-10-28_14:42:16.66808 from

/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:110:in

`fork’
2014-10-28_14:42:16.66812 from

/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:110:in

`start_worker’
2014-10-28_14:42:16.66816 from

/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:69:in

`block in start’
2014-10-28_14:42:16.66906 from

/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:66:in

`each’
2014-10-28_14:42:16.66911 from

/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:66:in

start' 2014-10-28_14:42:16.66915 from /opt/chef-sync/embedded/service/ec_sync_client/bin/ec_sync_client:56:inrun’
2014-10-28_14:42:16.66919 from
/opt/chef-sync/embedded/service/ec_sync_client/bin/ec_sync_client:62:in
`'
2014-10-28_14:42:16.68830 I, [2014-10-28T14:42:16.688096 #8095] INFO –
:
Worker for test died unexpectedly!
2014-10-28_14:44:04.81704 I, [2014-10-28T14:44:04.816897 #8095] INFO –
:
ec_sync_client: starting new command connection handler
2014-10-28_14:44:04.82802 I, [2014-10-28T14:44:04.827847 #8095] INFO –
:
ec_sync_client: closing command connection handler

On Tue, Oct 28, 2014 at 3:05 PM, Tiago Cruz tiago.tuxkiller@gmail.com
wrote:

Hello Steven, nice tip thanks!

Well, I don’t have /opt/chef-server directory, but I used another
"/bin/knife" instead, I hope you don’t mind :slight_smile:

[root@chef12-server1 ~]# find /opt/ -name ‘knife’ | egrep
’/bin/knife$’ |

wc -l
17

And you’re right the user/key does not match on replica host:

[root@chef12-server1 ~]# /opt/chef/bin/knife user list -s
https://localhost/organizations/test -u ec_sync_user -k
/etc/chef-sync/ec_sync_user.pem
WARNING: No knife configuration file found
ec_sync_user
tiago_cruz

[root@chef12-server2 ~]# /opt/chef/bin/knife user list -s
https://localhost/organizations/test -u ec_sync_user -k
/etc/chef-sync/ec_sync_user.pem
WARNING: No knife configuration file found
ERROR: Failed to authenticate to https://localhost/organizations/test
as

ec_sync_user with key /etc/chef-sync/ec_sync_user.pem
Response: Invalid signature for user or client ‘ec_sync_user’

Strange, 'cause I can see the creation while I run the reconfigure:

[root@chef12-server2 ~]# chef-sync-ctl reconfigure

Recipe: chef-sync::sync_user

  • chef_user[ec_sync_user] action create (up to date)

So, how can I fix this?

Thanks a lot!

On Tue, Oct 28, 2014 at 2:00 PM, Steven Danna steve@opscode.com
wrote:

Hi Tiago,

The show-config issue looks like a bug we should address. I’ll make
sure that gets filed. As for the 401, your instinct to look at the
key is correct. Could you try the following to see if the key works
outside of the sync service:

From the replica:

/opt/chef-server/embedded/bin/knife node list -s
https://chef12-server1.datac.test.com/organizations/test -u
ec_sync_user -k /etc/chef-sync/ec_sync_user.pem
/opt/chef-server/embedded/bin/knife node list -s
https://localhost/organizations/test -u ec_sync_user -k
/etc/chef-sync/ec_sync_user.pem

Both of those command should succeed without error. Could you also
post the output of

chef-sync-ctl manager-log

Cheers,

Steven

On Tue, Oct 28, 2014 at 3:02 PM, Tiago Cruz <tiago.tuxkiller@gmail.com

wrote:

Hello Guys,

I’m testing the replication feature
https://docs.getchef.com/server/server_replication.html

Package: chef-server-core-12.0.0_rc.5-1.el5.x86_64.rpm
OS: CentOS release 6.5
chef12-server1 = master / us-east-1
chef12-server2 = replica / sa-east-1

The problem is:

============

[root@chef12-server2 ~]# chef-sync-ctl sync-log test

Logfile created on 2014-10-28 14:26:03 +0000 by logger.rb/31641

I, [2014-10-28T14:26:03.073474 #14407] INFO – : ec_sync_client:
worker for

https://chef12-server1.datac.test.com/organizations/test/organizations/test

-> https://127.0.0.1/organizations/test started
I, [2014-10-28T14:29:04.326504 #15188] INFO – : ec_sync_client:
worker for

https://chef12-server1.datac.test.com/organizations/test/organizations/test

-> https://127.0.0.1/organizations/test started
I, [2014-10-28T14:29:04.881830 #15188] INFO – : HTTP Request
Returned

404
Not Found:
I, [2014-10-28T14:32:03.081657 #16173] INFO – : ec_sync_client:
worker for
https://chef12-server1.datac.test.com/organizations/test ->
https://127.0.0.1/organizations/test started
I, [2014-10-28T14:32:03.697880 #16173] INFO – : Bootrapping
started.

I, [2014-10-28T14:32:04.681163 #16173] INFO – : HTTP Request
Returned

401
Unauthorized: error
I, [2014-10-28T14:32:05.294182 #16173] INFO – : HTTP Request
Returned

401
Unauthorized: error
I, [2014-10-28T14:32:05.927196 #16173] INFO – : HTTP Request
Returned

401
Unauthorized: error
I, [2014-10-28T14:32:06.564459 #16173] INFO – : HTTP Request
Returned

401
Unauthorized: error
I, [2014-10-28T14:33:52.138190 #16938] INFO – : ec_sync_client:
worker for
https://chef12-server1.datac.test.com/organizations/test ->
https://127.0.0.1/organizations/test started
I, [2014-10-28T14:33:52.756533 #16938] INFO – : Bootrapping
started.

I, [2014-10-28T14:33:53.743989 #16938] INFO – : HTTP Request
Returned

401
Unauthorized: error
I, [2014-10-28T14:33:54.377393 #16938] INFO – : HTTP Request
Returned

401
Unauthorized: error
I, [2014-10-28T14:33:54.998467 #16938] INFO – : HTTP Request
Returned

401
Unauthorized: error
I, [2014-10-28T14:33:55.630469 #16938] INFO – : HTTP Request
Returned

401
Unauthorized: error
I, [2014-10-28T14:42:12.312655 #8098] INFO – : ec_sync_client:
worker

for
https://chef12-server1.datac.test.com/organizations/test ->
https://127.0.0.1/organizations/test started
I, [2014-10-28T14:42:12.925787 #8098] INFO – : Bootrapping started.
I, [2014-10-28T14:42:14.763716 #8098] INFO – : HTTP Request
Returned

401
Unauthorized: error
I, [2014-10-28T14:42:15.422438 #8098] INFO – : HTTP Request
Returned

401
Unauthorized: error
I, [2014-10-28T14:42:16.066673 #8098] INFO – : HTTP Request
Returned

401
Unauthorized: error
I, [2014-10-28T14:42:16.664576 #8098] INFO – : HTTP Request
Returned

401
Unauthorized: error

============

The /etc/chef-sync/ec_sync_user.pem is the same on both machines
(generated
on master “server1” copied to replica “server2”), and the command
"chef-sync-ctl prepare-org test" was executed on both sides.

Also, the “show config” does not work, both in server1 or in server2:

[root@chef12-server2 ~]# chef-sync-ctl show-config
Starting Chef Client, version 11.12.2
Compiling Cookbooks…

================================================================================

Recipe Compile Error

================================================================================

Chef::Exceptions::RecipeNotFound

could not find recipe show_config for cookbook chef-sync

Running handlers:
Running handlers complete

[2014-10-28T14:59:08+00:00] FATAL: Stacktrace dumped to
/opt/chef-sync/embedded/cookbooks/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated in 3.848546996 seconds
[2014-10-28T14:59:08+00:00] FATAL:
Chef::Exceptions::ChildConvergeError:
Chef run process exited unsuccessfully (exit code 1)

Can you help me please?

Thanks a lot!


– Tiago Cruz


– Tiago Cruz


– Tiago Cruz


– Tiago Cruz


#7

Hum… unfortunately, after a reboot on replica server, the sync stopped to
work:

[root@chef12-server2 ~]# chef-sync-ctl sync-status
Organization Name Status Last Sync Time Last Sync Count Next
Wakeup Time
test sync_failed 2014-10-28
19:16:04 +0000

My user ‘ec_sync_user’ is not working any more:

[root@chef12-server2 ~]# /opt/chef/bin/knife user list -s
https://localhost/organizations/test -u ec_sync_user -k
/etc/chef-sync/ec_sync_user.pem
WARNING: No knife configuration file found
ERROR: You authenticated successfully to
https://localhost/organizations/test as ec_sync_user but you are not
authorized for this action
Response: ‘ec_sync_user’ not associated with organization ‘test’

Even with “prepare-org” command:

[root@chef12-server2 ~]# chef-sync-ctl prepare-org test
Adding ec_sync_user to test
ec_sync_user already associated to test
Adding ec_sync_user to admin group in test

Sounds like my user has an invite, but how can I accept or force to accept
this invite?

[root@chef12-server2 ~]# chef-server-ctl org-associate test ec_sync_user -V
INFO: HTTP Request Returned 409 Conflict: The invite already exists.
ERROR: Conflict
Response: The invite already exists.

I also tried to set password, and then accept using the web interface, but
does not work also :slight_smile:

[root@chef12-server2 ~]# chef-server-ctl password ec_sync_user
(eval):17:in block (2 levels) in load_files': undefined method[]’ for
nil:NilClass (NoMethodError)
from
/opt/opscode/embedded/lib/ruby/gems/1.9.1/gems/omnibus-ctl-0.3.0/lib/omnibus-ctl.rb:177:in
call' from /opt/opscode/embedded/lib/ruby/gems/1.9.1/gems/omnibus-ctl-0.3.0/lib/omnibus-ctl.rb:177:inblock in add_command_under_category’
from
/opt/opscode/embedded/lib/ruby/gems/1.9.1/gems/omnibus-ctl-0.3.0/lib/omnibus-ctl.rb:555:in
run' from /opt/opscode/embedded/lib/ruby/gems/1.9.1/gems/omnibus-ctl-0.3.0/bin/omnibus-ctl:31:in<top (required)>'
from /opt/opscode/embedded/bin/omnibus-ctl:23:in load' from /opt/opscode/embedded/bin/omnibus-ctl:23:in

Thanks a lot!

On Tue, Oct 28, 2014 at 4:29 PM, Tiago Cruz tiago.tuxkiller@gmail.com
wrote:

Hello again Steven, thanks for all your support.

I can’t remove the user:

[root@chef12-server2 ~]# chef-server-ctl user-delete ec_sync_user
Do you want to delete the user ec_sync_user? (Y/N)y
ERROR: You authenticated successfully to https://127.0.0.1:443 as pivotal
but you are not authorized for this action
Response: Members of an organization’s admins group cannot delete
themselves. Remove yourself from the admins group, then retry this
operation.

But I used the “cleanse” option (Delete all chef-sync data, and start
from scratch) and now the replication is working fine, thanks!

Cheers,
Tiago Cruz

On Tue, Oct 28, 2014 at 4:04 PM, Steven Danna steve@opscode.com wrote:

Hi,

On chef12-server2 first make sure that /etc/chef-sync/ec_sync_user.pem
has the same md5sum as the one on server1. If it does, then try this
on server2:

chef-server-ctl user-delete ec_sync_user
chef-sync-ctl reconfigure

Cheers,

Steven

On Tue, Oct 28, 2014 at 5:27 PM, Tiago Cruz tiago.tuxkiller@gmail.com
wrote:

Sorry, I forgot to give you the output of “manager-log” command:

Master:

[tiago.cruz@chef12-server1 ~]$ sudo chef-sync-ctl manager-log
2014-10-28_14:20:14.37022 I, [2014-10-28T14:20:14.370091 #7976] INFO
– :
ec_sync_client: manager starting
2014-10-28_14:20:14.37034 I, [2014-10-28T14:20:14.370298 #7976] INFO
– :
ec_sync_client: listening on
/var/opt/chef-sync/ec_sync_client/ec_sync.sock
2014-10-28_14:20:14.61700 I, [2014-10-28T14:20:14.616912 #7976] INFO
– :
Sending fast shutdown to workers
2014-10-28_14:20:14.61704 I, [2014-10-28T14:20:14.617003 #7976] INFO
– :
Waiting for workers to stop
2014-10-28_14:20:14.61708 I, [2014-10-28T14:20:14.617045 #7976] INFO
– :
Shutdown complete.
2014-10-28_14:20:16.32615 I, [2014-10-28T14:20:16.326040 #7982] INFO
– :
ec_sync_client: manager starting
2014-10-28_14:20:16.32626 I, [2014-10-28T14:20:16.326215 #7982] INFO
– :
ec_sync_client: listening on
/var/opt/chef-sync/ec_sync_client/ec_sync.sock
2014-10-28_14:20:18.57771 I, [2014-10-28T14:20:18.575169 #7982] INFO
– :
Sending fast shutdown to workers
2014-10-28_14:20:18.57778 I, [2014-10-28T14:20:18.575246 #7982] INFO
– :
Waiting for workers to stop
2014-10-28_14:20:18.57778 I, [2014-10-28T14:20:18.575270 #7982] INFO
– :
Shutdown complete.
2014-10-28_14:20:19.34221 I, [2014-10-28T14:20:19.342121 #8045] INFO
– :
ec_sync_client: manager starting
2014-10-28_14:20:19.34236 I, [2014-10-28T14:20:19.342334 #8045] INFO
– :
ec_sync_client: listening on
/var/opt/chef-sync/ec_sync_client/ec_sync.sock
2014-10-28_14:22:25.64480 I, [2014-10-28T14:22:25.644691 #8045] INFO
– :
Sending fast shutdown to workers
2014-10-28_14:22:25.64482 I, [2014-10-28T14:22:25.644803 #8045] INFO
– :
Waiting for workers to stop
2014-10-28_14:22:25.64486 I, [2014-10-28T14:22:25.644830 #8045] INFO
– :
Shutdown complete.

======================

Replica:

[tiago.cruz@chef12-server2 ~]$ sudo chef-sync-ctl manager-log
2014-10-28_14:42:16.66748 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/file_system.rb:426:in

`parallel_do’
2014-10-28_14:42:16.66752 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/file_system.rb:360:in

`copy_entries’
2014-10-28_14:42:16.66756 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/file_system.rb:145:in

`block in copy_to’
2014-10-28_14:42:16.66760 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:267:in

`call’
2014-10-28_14:42:16.66764 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:267:in

`process_input’
2014-10-28_14:42:16.66767 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:257:in

`process_one’
2014-10-28_14:42:16.66771 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:195:in

`each_with_exceptions_unordered’
2014-10-28_14:42:16.66775 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:74:in

`wait’
2014-10-28_14:42:16.66779 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer.rb:48:in

`parallel_do’
2014-10-28_14:42:16.66783 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer.rb:29:in

`parallel_do’
2014-10-28_14:42:16.66787 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/file_system.rb:426:in

`parallel_do’
2014-10-28_14:42:16.66792 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/file_system.rb:142:in

`copy_to’
2014-10-28_14:42:16.66796 from

/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/orgsyncer.rb:132:in

`bootstrap’
2014-10-28_14:42:16.66800 from

/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/orgsyncer.rb:54:in

`start’
2014-10-28_14:42:16.66804 from

/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:114:in

`block in start_worker’
2014-10-28_14:42:16.66808 from

/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:110:in

`fork’
2014-10-28_14:42:16.66812 from

/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:110:in

`start_worker’
2014-10-28_14:42:16.66816 from

/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:69:in

`block in start’
2014-10-28_14:42:16.66906 from

/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:66:in

`each’
2014-10-28_14:42:16.66911 from

/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:66:in

start' 2014-10-28_14:42:16.66915 from /opt/chef-sync/embedded/service/ec_sync_client/bin/ec_sync_client:56:inrun’
2014-10-28_14:42:16.66919 from
/opt/chef-sync/embedded/service/ec_sync_client/bin/ec_sync_client:62:in
`'
2014-10-28_14:42:16.68830 I, [2014-10-28T14:42:16.688096 #8095] INFO
– :
Worker for test died unexpectedly!
2014-10-28_14:44:04.81704 I, [2014-10-28T14:44:04.816897 #8095] INFO
– :
ec_sync_client: starting new command connection handler
2014-10-28_14:44:04.82802 I, [2014-10-28T14:44:04.827847 #8095] INFO
– :
ec_sync_client: closing command connection handler

On Tue, Oct 28, 2014 at 3:05 PM, Tiago Cruz tiago.tuxkiller@gmail.com
wrote:

Hello Steven, nice tip thanks!

Well, I don’t have /opt/chef-server directory, but I used another
"/bin/knife" instead, I hope you don’t mind :slight_smile:

[root@chef12-server1 ~]# find /opt/ -name ‘knife’ | egrep
’/bin/knife$’ |

wc -l
17

And you’re right the user/key does not match on replica host:

[root@chef12-server1 ~]# /opt/chef/bin/knife user list -s
https://localhost/organizations/test -u ec_sync_user -k
/etc/chef-sync/ec_sync_user.pem
WARNING: No knife configuration file found
ec_sync_user
tiago_cruz

[root@chef12-server2 ~]# /opt/chef/bin/knife user list -s
https://localhost/organizations/test -u ec_sync_user -k
/etc/chef-sync/ec_sync_user.pem
WARNING: No knife configuration file found
ERROR: Failed to authenticate to https://localhost/organizations/test
as

ec_sync_user with key /etc/chef-sync/ec_sync_user.pem
Response: Invalid signature for user or client ‘ec_sync_user’

Strange, 'cause I can see the creation while I run the reconfigure:

[root@chef12-server2 ~]# chef-sync-ctl reconfigure

Recipe: chef-sync::sync_user

  • chef_user[ec_sync_user] action create (up to date)

So, how can I fix this?

Thanks a lot!

On Tue, Oct 28, 2014 at 2:00 PM, Steven Danna steve@opscode.com
wrote:

Hi Tiago,

The show-config issue looks like a bug we should address. I’ll make
sure that gets filed. As for the 401, your instinct to look at the
key is correct. Could you try the following to see if the key works
outside of the sync service:

From the replica:

/opt/chef-server/embedded/bin/knife node list -s
https://chef12-server1.datac.test.com/organizations/test -u
ec_sync_user -k /etc/chef-sync/ec_sync_user.pem
/opt/chef-server/embedded/bin/knife node list -s
https://localhost/organizations/test -u ec_sync_user -k
/etc/chef-sync/ec_sync_user.pem

Both of those command should succeed without error. Could you also
post the output of

chef-sync-ctl manager-log

Cheers,

Steven

On Tue, Oct 28, 2014 at 3:02 PM, Tiago Cruz <
tiago.tuxkiller@gmail.com>

wrote:

Hello Guys,

I’m testing the replication feature
https://docs.getchef.com/server/server_replication.html

Package: chef-server-core-12.0.0_rc.5-1.el5.x86_64.rpm
OS: CentOS release 6.5
chef12-server1 = master / us-east-1
chef12-server2 = replica / sa-east-1

The problem is:

============

[root@chef12-server2 ~]# chef-sync-ctl sync-log test

Logfile created on 2014-10-28 14:26:03 +0000 by logger.rb/31641

I, [2014-10-28T14:26:03.073474 #14407] INFO – : ec_sync_client:
worker for

https://chef12-server1.datac.test.com/organizations/test/organizations/test

-> https://127.0.0.1/organizations/test started
I, [2014-10-28T14:29:04.326504 #15188] INFO – : ec_sync_client:
worker for

https://chef12-server1.datac.test.com/organizations/test/organizations/test

-> https://127.0.0.1/organizations/test started
I, [2014-10-28T14:29:04.881830 #15188] INFO – : HTTP Request
Returned

404
Not Found:
I, [2014-10-28T14:32:03.081657 #16173] INFO – : ec_sync_client:
worker for
https://chef12-server1.datac.test.com/organizations/test ->
https://127.0.0.1/organizations/test started
I, [2014-10-28T14:32:03.697880 #16173] INFO – : Bootrapping
started.

I, [2014-10-28T14:32:04.681163 #16173] INFO – : HTTP Request
Returned

401
Unauthorized: error
I, [2014-10-28T14:32:05.294182 #16173] INFO – : HTTP Request
Returned

401
Unauthorized: error
I, [2014-10-28T14:32:05.927196 #16173] INFO – : HTTP Request
Returned

401
Unauthorized: error
I, [2014-10-28T14:32:06.564459 #16173] INFO – : HTTP Request
Returned

401
Unauthorized: error
I, [2014-10-28T14:33:52.138190 #16938] INFO – : ec_sync_client:
worker for
https://chef12-server1.datac.test.com/organizations/test ->
https://127.0.0.1/organizations/test started
I, [2014-10-28T14:33:52.756533 #16938] INFO – : Bootrapping
started.

I, [2014-10-28T14:33:53.743989 #16938] INFO – : HTTP Request
Returned

401
Unauthorized: error
I, [2014-10-28T14:33:54.377393 #16938] INFO – : HTTP Request
Returned

401
Unauthorized: error
I, [2014-10-28T14:33:54.998467 #16938] INFO – : HTTP Request
Returned

401
Unauthorized: error
I, [2014-10-28T14:33:55.630469 #16938] INFO – : HTTP Request
Returned

401
Unauthorized: error
I, [2014-10-28T14:42:12.312655 #8098] INFO – : ec_sync_client:
worker

for
https://chef12-server1.datac.test.com/organizations/test ->
https://127.0.0.1/organizations/test started
I, [2014-10-28T14:42:12.925787 #8098] INFO – : Bootrapping
started.

I, [2014-10-28T14:42:14.763716 #8098] INFO – : HTTP Request
Returned

401
Unauthorized: error
I, [2014-10-28T14:42:15.422438 #8098] INFO – : HTTP Request
Returned

401
Unauthorized: error
I, [2014-10-28T14:42:16.066673 #8098] INFO – : HTTP Request
Returned

401
Unauthorized: error
I, [2014-10-28T14:42:16.664576 #8098] INFO – : HTTP Request
Returned

401
Unauthorized: error

============

The /etc/chef-sync/ec_sync_user.pem is the same on both machines
(generated
on master “server1” copied to replica “server2”), and the command
"chef-sync-ctl prepare-org test" was executed on both sides.

Also, the “show config” does not work, both in server1 or in
server2:

[root@chef12-server2 ~]# chef-sync-ctl show-config
Starting Chef Client, version 11.12.2
Compiling Cookbooks…

================================================================================

Recipe Compile Error

================================================================================

Chef::Exceptions::RecipeNotFound

could not find recipe show_config for cookbook chef-sync

Running handlers:
Running handlers complete

[2014-10-28T14:59:08+00:00] FATAL: Stacktrace dumped to
/opt/chef-sync/embedded/cookbooks/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated in 3.848546996 seconds
[2014-10-28T14:59:08+00:00] FATAL:
Chef::Exceptions::ChildConvergeError:
Chef run process exited unsuccessfully (exit code 1)

Can you help me please?

Thanks a lot!


– Tiago Cruz


– Tiago Cruz


– Tiago Cruz


– Tiago Cruz


– Tiago Cruz


#8

Hi Tiago,

My hunch is that this is unrelated to chef-sync; however, I’m happy to
get to the bottom of it. The command: chef-server-ctl gather-logs
should create a tarball of recent logs. Can you run that and send me
over the logs off-list?

Cheers,

Steven

On Tue, Oct 28, 2014 at 8:41 PM, Tiago Cruz tiago.tuxkiller@gmail.com wrote:

Hum… unfortunately, after a reboot on replica server, the sync stopped to
work:

[root@chef12-server2 ~]# chef-sync-ctl sync-status
Organization Name Status Last Sync Time Last Sync Count Next Wakeup
Time
test sync_failed 2014-10-28
19:16:04 +0000

My user ‘ec_sync_user’ is not working any more:

[root@chef12-server2 ~]# /opt/chef/bin/knife user list -s
https://localhost/organizations/test -u ec_sync_user -k
/etc/chef-sync/ec_sync_user.pem
WARNING: No knife configuration file found
ERROR: You authenticated successfully to
https://localhost/organizations/test as ec_sync_user but you are not
authorized for this action
Response: ‘ec_sync_user’ not associated with organization ‘test’

Even with “prepare-org” command:

[root@chef12-server2 ~]# chef-sync-ctl prepare-org test
Adding ec_sync_user to test
ec_sync_user already associated to test
Adding ec_sync_user to admin group in test

Sounds like my user has an invite, but how can I accept or force to accept
this invite?

[root@chef12-server2 ~]# chef-server-ctl org-associate test ec_sync_user -V
INFO: HTTP Request Returned 409 Conflict: The invite already exists.
ERROR: Conflict
Response: The invite already exists.

I also tried to set password, and then accept using the web interface, but
does not work also :slight_smile:

[root@chef12-server2 ~]# chef-server-ctl password ec_sync_user
(eval):17:in block (2 levels) in load_files': undefined method[]’ for
nil:NilClass (NoMethodError)
from
/opt/opscode/embedded/lib/ruby/gems/1.9.1/gems/omnibus-ctl-0.3.0/lib/omnibus-ctl.rb:177:in
call' from /opt/opscode/embedded/lib/ruby/gems/1.9.1/gems/omnibus-ctl-0.3.0/lib/omnibus-ctl.rb:177:inblock in add_command_under_category’
from
/opt/opscode/embedded/lib/ruby/gems/1.9.1/gems/omnibus-ctl-0.3.0/lib/omnibus-ctl.rb:555:in
run' from /opt/opscode/embedded/lib/ruby/gems/1.9.1/gems/omnibus-ctl-0.3.0/bin/omnibus-ctl:31:in<top (required)>'
from /opt/opscode/embedded/bin/omnibus-ctl:23:in load' from /opt/opscode/embedded/bin/omnibus-ctl:23:in

Thanks a lot!

On Tue, Oct 28, 2014 at 4:29 PM, Tiago Cruz tiago.tuxkiller@gmail.com
wrote:

Hello again Steven, thanks for all your support.

I can’t remove the user:

[root@chef12-server2 ~]# chef-server-ctl user-delete ec_sync_user
Do you want to delete the user ec_sync_user? (Y/N)y
ERROR: You authenticated successfully to https://127.0.0.1:443 as pivotal
but you are not authorized for this action
Response: Members of an organization’s admins group cannot delete
themselves. Remove yourself from the admins group, then retry this
operation.

But I used the “cleanse” option (Delete all chef-sync data, and start
from scratch) and now the replication is working fine, thanks!

Cheers,
Tiago Cruz

On Tue, Oct 28, 2014 at 4:04 PM, Steven Danna steve@opscode.com wrote:

Hi,

On chef12-server2 first make sure that /etc/chef-sync/ec_sync_user.pem
has the same md5sum as the one on server1. If it does, then try this
on server2:

chef-server-ctl user-delete ec_sync_user
chef-sync-ctl reconfigure

Cheers,

Steven

On Tue, Oct 28, 2014 at 5:27 PM, Tiago Cruz tiago.tuxkiller@gmail.com
wrote:

Sorry, I forgot to give you the output of “manager-log” command:

Master:

[tiago.cruz@chef12-server1 ~]$ sudo chef-sync-ctl manager-log
2014-10-28_14:20:14.37022 I, [2014-10-28T14:20:14.370091 #7976] INFO
– :
ec_sync_client: manager starting
2014-10-28_14:20:14.37034 I, [2014-10-28T14:20:14.370298 #7976] INFO
– :
ec_sync_client: listening on
/var/opt/chef-sync/ec_sync_client/ec_sync.sock
2014-10-28_14:20:14.61700 I, [2014-10-28T14:20:14.616912 #7976] INFO
– :
Sending fast shutdown to workers
2014-10-28_14:20:14.61704 I, [2014-10-28T14:20:14.617003 #7976] INFO
– :
Waiting for workers to stop
2014-10-28_14:20:14.61708 I, [2014-10-28T14:20:14.617045 #7976] INFO
– :
Shutdown complete.
2014-10-28_14:20:16.32615 I, [2014-10-28T14:20:16.326040 #7982] INFO
– :
ec_sync_client: manager starting
2014-10-28_14:20:16.32626 I, [2014-10-28T14:20:16.326215 #7982] INFO
– :
ec_sync_client: listening on
/var/opt/chef-sync/ec_sync_client/ec_sync.sock
2014-10-28_14:20:18.57771 I, [2014-10-28T14:20:18.575169 #7982] INFO
– :
Sending fast shutdown to workers
2014-10-28_14:20:18.57778 I, [2014-10-28T14:20:18.575246 #7982] INFO
– :
Waiting for workers to stop
2014-10-28_14:20:18.57778 I, [2014-10-28T14:20:18.575270 #7982] INFO
– :
Shutdown complete.
2014-10-28_14:20:19.34221 I, [2014-10-28T14:20:19.342121 #8045] INFO
– :
ec_sync_client: manager starting
2014-10-28_14:20:19.34236 I, [2014-10-28T14:20:19.342334 #8045] INFO
– :
ec_sync_client: listening on
/var/opt/chef-sync/ec_sync_client/ec_sync.sock
2014-10-28_14:22:25.64480 I, [2014-10-28T14:22:25.644691 #8045] INFO
– :
Sending fast shutdown to workers
2014-10-28_14:22:25.64482 I, [2014-10-28T14:22:25.644803 #8045] INFO
– :
Waiting for workers to stop
2014-10-28_14:22:25.64486 I, [2014-10-28T14:22:25.644830 #8045] INFO
– :
Shutdown complete.

======================

Replica:

[tiago.cruz@chef12-server2 ~]$ sudo chef-sync-ctl manager-log
2014-10-28_14:42:16.66748 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/file_system.rb:426:in
`parallel_do’
2014-10-28_14:42:16.66752 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/file_system.rb:360:in
`copy_entries’
2014-10-28_14:42:16.66756 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/file_system.rb:145:in
`block in copy_to’
2014-10-28_14:42:16.66760 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:267:in
`call’
2014-10-28_14:42:16.66764 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:267:in
`process_input’
2014-10-28_14:42:16.66767 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:257:in
`process_one’
2014-10-28_14:42:16.66771 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:195:in
`each_with_exceptions_unordered’
2014-10-28_14:42:16.66775 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer/parallel_enumerable.rb:74:in
`wait’
2014-10-28_14:42:16.66779 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer.rb:48:in
`parallel_do’
2014-10-28_14:42:16.66783 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/parallelizer.rb:29:in
`parallel_do’
2014-10-28_14:42:16.66787 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/file_system.rb:426:in
`parallel_do’
2014-10-28_14:42:16.66792 from

/opt/chef-sync/embedded/service/gem/ruby/1.9.1/bundler/gems/chef-fa77f1e10e85/lib/chef/chef_fs/file_system.rb:142:in
`copy_to’
2014-10-28_14:42:16.66796 from

/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/orgsyncer.rb:132:in
`bootstrap’
2014-10-28_14:42:16.66800 from

/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/orgsyncer.rb:54:in
`start’
2014-10-28_14:42:16.66804 from

/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:114:in
`block in start_worker’
2014-10-28_14:42:16.66808 from

/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:110:in
`fork’
2014-10-28_14:42:16.66812 from

/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:110:in
`start_worker’
2014-10-28_14:42:16.66816 from

/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:69:in
`block in start’
2014-10-28_14:42:16.66906 from

/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:66:in
`each’
2014-10-28_14:42:16.66911 from

/opt/chef-sync/embedded/service/ec_sync_client/lib/ec_sync_client/sync_manager.rb:66:in
start' 2014-10-28_14:42:16.66915 from /opt/chef-sync/embedded/service/ec_sync_client/bin/ec_sync_client:56:inrun’
2014-10-28_14:42:16.66919 from
/opt/chef-sync/embedded/service/ec_sync_client/bin/ec_sync_client:62:in
`'
2014-10-28_14:42:16.68830 I, [2014-10-28T14:42:16.688096 #8095] INFO
– :
Worker for test died unexpectedly!
2014-10-28_14:44:04.81704 I, [2014-10-28T14:44:04.816897 #8095] INFO
– :
ec_sync_client: starting new command connection handler
2014-10-28_14:44:04.82802 I, [2014-10-28T14:44:04.827847 #8095] INFO
– :
ec_sync_client: closing command connection handler

On Tue, Oct 28, 2014 at 3:05 PM, Tiago Cruz tiago.tuxkiller@gmail.com
wrote:

Hello Steven, nice tip thanks!

Well, I don’t have /opt/chef-server directory, but I used another
"/bin/knife" instead, I hope you don’t mind :slight_smile:

[root@chef12-server1 ~]# find /opt/ -name ‘knife’ | egrep
’/bin/knife$’ |
wc -l
17

And you’re right the user/key does not match on replica host:

[root@chef12-server1 ~]# /opt/chef/bin/knife user list -s
https://localhost/organizations/test -u ec_sync_user -k
/etc/chef-sync/ec_sync_user.pem
WARNING: No knife configuration file found
ec_sync_user
tiago_cruz

[root@chef12-server2 ~]# /opt/chef/bin/knife user list -s
https://localhost/organizations/test -u ec_sync_user -k
/etc/chef-sync/ec_sync_user.pem
WARNING: No knife configuration file found
ERROR: Failed to authenticate to https://localhost/organizations/test
as
ec_sync_user with key /etc/chef-sync/ec_sync_user.pem
Response: Invalid signature for user or client ‘ec_sync_user’

Strange, 'cause I can see the creation while I run the reconfigure:

[root@chef12-server2 ~]# chef-sync-ctl reconfigure

Recipe: chef-sync::sync_user

  • chef_user[ec_sync_user] action create (up to date)

So, how can I fix this?

Thanks a lot!

On Tue, Oct 28, 2014 at 2:00 PM, Steven Danna steve@opscode.com
wrote:

Hi Tiago,

The show-config issue looks like a bug we should address. I’ll make
sure that gets filed. As for the 401, your instinct to look at the
key is correct. Could you try the following to see if the key works
outside of the sync service:

From the replica:

/opt/chef-server/embedded/bin/knife node list -s
https://chef12-server1.datac.test.com/organizations/test -u
ec_sync_user -k /etc/chef-sync/ec_sync_user.pem
/opt/chef-server/embedded/bin/knife node list -s
https://localhost/organizations/test -u ec_sync_user -k
/etc/chef-sync/ec_sync_user.pem

Both of those command should succeed without error. Could you also
post the output of

chef-sync-ctl manager-log

Cheers,

Steven

On Tue, Oct 28, 2014 at 3:02 PM, Tiago Cruz
tiago.tuxkiller@gmail.com
wrote:

Hello Guys,

I’m testing the replication feature
https://docs.getchef.com/server/server_replication.html

Package: chef-server-core-12.0.0_rc.5-1.el5.x86_64.rpm
OS: CentOS release 6.5
chef12-server1 = master / us-east-1
chef12-server2 = replica / sa-east-1

The problem is:

============

[root@chef12-server2 ~]# chef-sync-ctl sync-log test

Logfile created on 2014-10-28 14:26:03 +0000 by logger.rb/31641

I, [2014-10-28T14:26:03.073474 #14407] INFO – : ec_sync_client:
worker for

https://chef12-server1.datac.test.com/organizations/test/organizations/test
-> https://127.0.0.1/organizations/test started
I, [2014-10-28T14:29:04.326504 #15188] INFO – : ec_sync_client:
worker for

https://chef12-server1.datac.test.com/organizations/test/organizations/test
-> https://127.0.0.1/organizations/test started
I, [2014-10-28T14:29:04.881830 #15188] INFO – : HTTP Request
Returned
404
Not Found:
I, [2014-10-28T14:32:03.081657 #16173] INFO – : ec_sync_client:
worker for
https://chef12-server1.datac.test.com/organizations/test ->
https://127.0.0.1/organizations/test started
I, [2014-10-28T14:32:03.697880 #16173] INFO – : Bootrapping
started.
I, [2014-10-28T14:32:04.681163 #16173] INFO – : HTTP Request
Returned
401
Unauthorized: error
I, [2014-10-28T14:32:05.294182 #16173] INFO – : HTTP Request
Returned
401
Unauthorized: error
I, [2014-10-28T14:32:05.927196 #16173] INFO – : HTTP Request
Returned
401
Unauthorized: error
I, [2014-10-28T14:32:06.564459 #16173] INFO – : HTTP Request
Returned
401
Unauthorized: error
I, [2014-10-28T14:33:52.138190 #16938] INFO – : ec_sync_client:
worker for
https://chef12-server1.datac.test.com/organizations/test ->
https://127.0.0.1/organizations/test started
I, [2014-10-28T14:33:52.756533 #16938] INFO – : Bootrapping
started.
I, [2014-10-28T14:33:53.743989 #16938] INFO – : HTTP Request
Returned
401
Unauthorized: error
I, [2014-10-28T14:33:54.377393 #16938] INFO – : HTTP Request
Returned
401
Unauthorized: error
I, [2014-10-28T14:33:54.998467 #16938] INFO – : HTTP Request
Returned
401
Unauthorized: error
I, [2014-10-28T14:33:55.630469 #16938] INFO – : HTTP Request
Returned
401
Unauthorized: error
I, [2014-10-28T14:42:12.312655 #8098] INFO – : ec_sync_client:
worker
for
https://chef12-server1.datac.test.com/organizations/test ->
https://127.0.0.1/organizations/test started
I, [2014-10-28T14:42:12.925787 #8098] INFO – : Bootrapping
started.
I, [2014-10-28T14:42:14.763716 #8098] INFO – : HTTP Request
Returned
401
Unauthorized: error
I, [2014-10-28T14:42:15.422438 #8098] INFO – : HTTP Request
Returned
401
Unauthorized: error
I, [2014-10-28T14:42:16.066673 #8098] INFO – : HTTP Request
Returned
401
Unauthorized: error
I, [2014-10-28T14:42:16.664576 #8098] INFO – : HTTP Request
Returned
401
Unauthorized: error

============

The /etc/chef-sync/ec_sync_user.pem is the same on both machines
(generated
on master “server1” copied to replica “server2”), and the command
"chef-sync-ctl prepare-org test" was executed on both sides.

Also, the “show config” does not work, both in server1 or in
server2:

[root@chef12-server2 ~]# chef-sync-ctl show-config
Starting Chef Client, version 11.12.2
Compiling Cookbooks…

================================================================================
Recipe Compile Error

================================================================================

Chef::Exceptions::RecipeNotFound

could not find recipe show_config for cookbook chef-sync

Running handlers:
Running handlers complete

[2014-10-28T14:59:08+00:00] FATAL: Stacktrace dumped to
/opt/chef-sync/embedded/cookbooks/cache/chef-stacktrace.out
Chef Client failed. 0 resources updated in 3.848546996 seconds
[2014-10-28T14:59:08+00:00] FATAL:
Chef::Exceptions::ChildConvergeError:
Chef run process exited unsuccessfully (exit code 1)

Can you help me please?

Thanks a lot!


– Tiago Cruz


– Tiago Cruz


– Tiago Cruz


– Tiago Cruz


– Tiago Cruz


#9

Hi,

On Tue, Oct 28, 2014 at 6:04 PM, Steven Danna steve@opscode.com wrote:

chef-server-ctl user-delete ec_sync_user

It turns out that a bug in the RC makes this rather bad advice. You
can read more about the bug here:

The comments include directions for recovering from the bad state you
get into if you happened to have followed this advice. Thanks to
Tiago for helping find this issue.

Cheers,

Steven