# Erchef errors

**URL:** <https://discourse.chef.io/t/erchef-errors/4749>\
**Category:** Chef Infra (archive)\
**Created:** [November 26, 2013, 5:28pm UTC](https://discourse.chef.io/t/erchef-errors/4749 "2013-11-26T17:28:10Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![silviu\_dicu](https://avatars.discourse-cdn.com/v4/letter/s/c4cdca/32.png) [@silviu\_dicu](https://discourse.chef.io/u/silviu_dicu)\
**Post date:** [November 26, 2013, 5:28pm UTC](https://discourse.chef.io/t/erchef-errors/4749/1 "2013-11-26T17:28:10Z")

</div>

Hi,

All the sudden my chef server version 11.4 started to act up - I get

## Postgresql

2013-11-26\_17:09:25.72986 STATEMENT: INSERT INTO checksums(org\_id,  
checksum)  
2013-11-26\_17:09:25.72986 VALUES ($1, $2)  
2013-11-26\_17:09:26.51887 ERROR: duplicate key value violates unique  
constraint "checksums\_pkey"  
2013-11-26\_17:09:26.51891 DETAIL: Key (org\_id,  
checksum)=(00000000000000000000000000000000,  
4dc4a7ba29b5ae8e48d6fe7745a5de0e) already exists.  
2013-11-26\_17:09:26.51892 STATEMENT: INSERT INTO checksums(org\_id,  
checksum)  
2013-11-26\_17:09:26.51893 VALUES ($1, $2)  
2013-11-26\_17:09:27.23903 ERROR: duplicate key value violates unique  
constraint "checksums\_pkey"  
2013-11-26\_17:09:27.23909 DETAIL: Key (org\_id,  
checksum)=(00000000000000000000000000000000,  
4e10085c11e681d11be36a169c5d10f4) already exists.  
2013-11-26\_17:09:27.23910 STATEMENT: INSERT INTO checksums(org\_id,  
checksum)  
2013-11-26\_17:09:27.23911 VALUES ($1, $2)

## Erchef

2013-11-26\_17:22:28.88338 webmachine error: path="/environments/  
web\_uat/cookbook\_versions"  
2013-11-26\_17:22:28.88338 {error,  
2013-11-26\_17:22:28.88338 {error,  
2013-11-26\_17:22:28.88339 {badrecord,chef\_cookbook\_version},  
2013-11-26\_17:22:28.88339  
[{chef\_wm\_depsolver,’-assemble\_response/3-lc$^0/1-0-’,1,  
2013-11-26\_17:22:28.88339  
[{file,“src/chef\_wm\_depsolver.erl”},{line,266}]},  
2013-11-26\_17:22:28.88340  
{chef\_wm\_depsolver,’-assemble\_response/3-lc$^0/1-0-’,1,  
2013-11-26\_17:22:28.88340  
[{file,“src/chef\_wm\_depsolver.erl”},{line,268}]},  
2013-11-26\_17:22:28.88340 {chef\_wm\_depsolver,assemble\_response,3,  
2013-11-26\_17:22:28.88341  
[{file,“src/chef\_wm\_depsolver.erl”},{line,266}]},  
2013-11-26\_17:22:28.88341 {webmachine\_resource,resource\_call,3,  
2013-11-26\_17:22:28.88342  
[{file,“src/webmachine\_resource.erl”},{line,166}]},  
2013-11-26\_17:22:28.88342 {webmachine\_resource,do,3,  
2013-11-26\_17:22:28.88342  
[{file,“src/webmachine\_resource.erl”},{line,125}]},  
2013-11-26\_17:22:28.88343  
{webmachine\_decision\_core,resource\_call,1,  
2013-11-26\_17:22:28.88343  
[{file,“src/webmachine\_decision\_core.erl”},{line,48}]},  
2013-11-26\_17:22:28.88343 {webmachine\_decision\_core,decision,1,  
2013-11-26\_17:22:28.88344  
[{file,“src/webmachine\_decision\_core.erl”},{line,458}]},  
2013-11-26\_17:22:28.88346  
{webmachine\_decision\_core,handle\_request,2,  
2013-11-26\_17:22:28.88348  
[{file,“src/webmachine\_decision\_core.erl”},{line,33}]}]}}

## Nginx

… truncated  
"POST /environments/web\_prod/cookbook\_versions HTTP/1.1" 500 “201.387” 36  
"-" "Chef Client/11.4.0

It all leads to the fact that there are some problems in relation to  
cookbook versions.  
However I don’t get what is it.

thanks!

-silviu

---

<div class="post-metadata">

**Author:** ![silviu\_dicu](https://avatars.discourse-cdn.com/v4/letter/s/c4cdca/32.png) [@silviu\_dicu](https://discourse.chef.io/u/silviu_dicu)\
**Post date:** [November 27, 2013, 2:08pm UTC](https://discourse.chef.io/t/erchef-errors/4749/2 "2013-11-27T14:08:56Z")

</div>

Hi,

Answering to myself ... hope will help somebody.

Symptoms on the chef server

- higher load then usual, from 0.5 - 1 to 4 - 5
- the beam.smp eating cpu constantly
- all the errors that I sent before into the log files

Spent some time looking at all components, read the erl files etc.  
Noticed as well exceptions for connection to the PostgreSQL from chef\_wm  
however the db was doing nothing - has a max of 200 connections with a  
pool of 20 and never used it at full.

The problem ... something as stupid as - somebody had written into  
/etc/hosts  
the wrong entry for chef server. After I corrected all went fine.

cheers!

ps: (/var/opt/chef-server/postgresql/data/pg\_hba.conf has ONLY ip addresses  
however  
there most be something else that uses the hostname/ip from hosts files)

On Tue, Nov 26, 2013 at 12:28 PM, Silviu Dicu [silviudicu@gmail.com](mailto:silviudicu@gmail.com) wrote:

> Hi,
> 
> All the sudden my chef server version 11.4 started to act up - I get
> 
> ## Postgresql
> 
> 2013-11-26\_17:09:25.72986 STATEMENT: INSERT INTO checksums(org\_id,  
> checksum)  
> 2013-11-26\_17:09:25.72986 VALUES ($1, $2)  
> 2013-11-26\_17:09:26.51887 ERROR: duplicate key value violates unique  
> constraint "checksums\_pkey"  
> 2013-11-26\_17:09:26.51891 DETAIL: Key (org\_id,  
> checksum)=(00000000000000000000000000000000,  
> 4dc4a7ba29b5ae8e48d6fe7745a5de0e) already exists.  
> 2013-11-26\_17:09:26.51892 STATEMENT: INSERT INTO checksums(org\_id,  
> checksum)  
> 2013-11-26\_17:09:26.51893 VALUES ($1, $2)  
> 2013-11-26\_17:09:27.23903 ERROR: duplicate key value violates unique  
> constraint "checksums\_pkey"  
> 2013-11-26\_17:09:27.23909 DETAIL: Key (org\_id,  
> checksum)=(00000000000000000000000000000000,  
> 4e10085c11e681d11be36a169c5d10f4) already exists.  
> 2013-11-26\_17:09:27.23910 STATEMENT: INSERT INTO checksums(org\_id,  
> checksum)  
> 2013-11-26\_17:09:27.23911 VALUES ($1, $2)
> 
> ## Erchef
> 
> 2013-11-26\_17:22:28.88338 webmachine error: path="/environments/  
> web\_uat/cookbook\_versions"  
> 2013-11-26\_17:22:28.88338 {error,  
> 2013-11-26\_17:22:28.88338 {error,  
> 2013-11-26\_17:22:28.88339 {badrecord,chef\_cookbook\_version},  
> 2013-11-26\_17:22:28.88339  
> [{chef\_wm\_depsolver,'-assemble\_response/3-lc$^0/1-0-',1,  
> 2013-11-26\_17:22:28.88339  
> [{file,"src/chef\_wm\_depsolver.erl"},{line,266}]},  
> 2013-11-26\_17:22:28.88340  
> {chef\_wm\_depsolver,'-assemble\_response/3-lc$^0/1-0-',1,  
> 2013-11-26\_17:22:28.88340  
> [{file,"src/chef\_wm\_depsolver.erl"},{line,268}]},  
> 2013-11-26\_17:22:28.88340 {chef\_wm\_depsolver,assemble\_response,3,  
> 2013-11-26\_17:22:28.88341  
> [{file,"src/chef\_wm\_depsolver.erl"},{line,266}]},  
> 2013-11-26\_17:22:28.88341 {webmachine\_resource,resource\_call,3,  
> 2013-11-26\_17:22:28.88342  
> [{file,"src/webmachine\_resource.erl"},{line,166}]},  
> 2013-11-26\_17:22:28.88342 {webmachine\_resource,do,3,  
> 2013-11-26\_17:22:28.88342  
> [{file,"src/webmachine\_resource.erl"},{line,125}]},  
> 2013-11-26\_17:22:28.88343  
> {webmachine\_decision\_core,resource\_call,1,  
> 2013-11-26\_17:22:28.88343  
> [{file,"src/webmachine\_decision\_core.erl"},{line,48}]},  
> 2013-11-26\_17:22:28.88343 {webmachine\_decision\_core,decision,1,  
> 2013-11-26\_17:22:28.88344  
> [{file,"src/webmachine\_decision\_core.erl"},{line,458}]},  
> 2013-11-26\_17:22:28.88346  
> {webmachine\_decision\_core,handle\_request,2,  
> 2013-11-26\_17:22:28.88348  
> [{file,"src/webmachine\_decision\_core.erl"},{line,33}]}]}}
> 
> ## Nginx
> 
> ... truncated  
> "POST /environments/web\_prod/cookbook\_versions HTTP/1.1" 500 "201.387" 36  
> "-" "Chef Client/11.4.0
> 
> It all leads to the fact that there are some problems in relation to  
> cookbook versions.  
> However I don't get what is it.
> 
> thanks!
> 
> -silviu

---

<div class="post-metadata">

**Author:** ![Seth\_Falcon\_01](https://sea2.discourse-cdn.com/flex016/user_avatar/discourse.chef.io/seth_falcon_01/32/429_2.png) [@Seth\_Falcon\_01](https://discourse.chef.io/u/Seth_Falcon_01)\
**Post date:** [November 27, 2013, 5:48pm UTC](https://discourse.chef.io/t/erchef-errors/4749/3 "2013-11-27T17:48:02Z")

</div>

Hi there,

[silviudicu@gmail.com](mailto:silviudicu@gmail.com) writes:

> Answering to myself ... hope will help somebody.

Sorry not to have been able to respond sooner.

> Symptoms on the chef server
> 
> - higher load then usual, from 0.5 - 1 to 4 - 5
> - the beam.smp eating cpu constantly
> - all the errors that I sent before into the log files

The beam.smp for erchef pegging the CPU on 11.0.8 is a strong indication  
that you're hitting a bug in the dependency solver [1].

[1] [https://tickets.opscode.com/browse/CHEF-3921](https://tickets.opscode.com/browse/CHEF-3921)

> Spent some time looking at all components, read the erl files etc.  
> Noticed as well exceptions for connection to the PostgreSQL from chef\_wm  
> however the db was doing nothing - has a max of 200 connections with a  
> pool of 20 and never used it at full.
> 
> The problem ... something as stupid as - somebody had written into  
> /etc/hosts  
> the wrong entry for chef server. After I corrected all went fine.

Chef Server needs to generate URLs to send to clients for cookbook  
content upload/download. So it is possible that changes in /etc/hosts  
could affect functionality there -- but unlikely to be related to high  
CPU use of erchef beam process.

> ps: (/var/opt/chef-server/postgresql/data/pg\_hba.conf has ONLY ip addresses  
> however  
> there most be something else that uses the hostname/ip from hosts  
> files)

Yep, erchef treats the bookshelf service as an external  
service. Cookbook upload and cookbook content download rely on erchef  
being able to generate a URL to the bookshelf service (by default at  
https://$SERVER\_FQDN/bookshelf) that is routable by clients.

> On Tue, Nov 26, 2013 at 12:28 PM, Silviu Dicu [silviudicu@gmail.com](mailto:silviudicu@gmail.com) wrote:
> 
> > Hi,
> > 
> > All the sudden my chef server version 11.4 started to act up - I get
> > 
> > ## Postgresql
> > 
> > 2013-11-26\_17:09:25.72986 STATEMENT: INSERT INTO checksums(org\_id,  
> > checksum)  
> > 2013-11-26\_17:09:25.72986 VALUES ($1, $2)  
> > 2013-11-26\_17:09:26.51887 ERROR: duplicate key value violates unique  
> > constraint "checksums\_pkey"  
> > 2013-11-26\_17:09:26.51891 DETAIL: Key (org\_id,  
> > checksum)=(00000000000000000000000000000000,  
> > 4dc4a7ba29b5ae8e48d6fe7745a5de0e) already exists.

These errors are (a bit sad to say) expected and happen under normal  
operation. We use pg's foreign key constraints to do reference counting  
on cookbook content (checksums). The checksummed cookbook content is  
shared across cookbooks and cookbook versions. When we update a cookbook  
version or delete one, the query we run attempts to insert or delete all  
of the associated checksums where we expect to only succeed if the  
constraints align -- side effect of this approach is errors in the  
log. We have a ticket to address this using a stored procedure.

> > ## Erchef
> > 
> > 2013-11-26\_17:22:28.88338 webmachine error: path="/environments/  
> > web\_uat/cookbook\_versions"  
> > 2013-11-26\_17:22:28.88338 {error,  
> > 2013-11-26\_17:22:28.88338 {error,  
> > 2013-11-26\_17:22:28.88339 {badrecord,chef\_cookbook\_version},  
> > 2013-11-26\_17:22:28.88339  
> > [{chef\_wm\_depsolver,'-assemble\_response/3-lc$^0/1-0-',1,  
> > 2013-11-26\_17:22:28.88339  
> > [{file,"src/chef\_wm\_depsolver.erl"},{line,266}]},  
> > 2013-11-26\_17:22:28.88340  
> > {chef\_wm\_depsolver,'-assemble\_response/3-lc$^0/1-0-',1,

This one is not expected but difficult to say from this trace if there's  
bad data in the db (I doubt it) and more likely that we encountered an  
error fetching cookbook content.

> > ## Nginx
> > 
> > ... truncated  
> > "POST /environments/web\_prod/cookbook\_versions HTTP/1.1" 500 "201.387" 36  
> > "-" "Chef Client/11.4.0

This is the call made by chef-client to resolve dependencies.

- seth

--  
Seth Falcon | Development Lead | Opscode | @sfalcon