Policyfiles broke Chef search

Chef Infra (archive)
1

We’re trying to move to policyfiles, but this has resulted in chef searches no longer returning a list of nodes. When using a policy, are node attributes saved differently?

Here are two nodes I built without a policyfile:

$ knife search node 'sid:SOED0115' -c ~/.chef/client.rb          
2 items found

Node Name:   soed0115
Environment: _default
FQDN:        soed0115.example.com
IP:          192.168.2.40
Run List:    recipe[csg_oracle_soe::primary]
Roles:       
Recipes:     csg_oracle_soe::primary, csg_oracle_soe::base, csg_oracle_soe::ora_dirs, csg_oracle_soe::grid_install, csg_oracle_soe::grid_opatch, csg_oracle_soe::grid_psu, csg_oracle_soe::grid_setup, csg_oracle_soe::db_install, csg_oracle_soe::db_opatch, csg_oracle_soe::db_psu, csg_oracle_soe::db_create_primary, csg_oracle_soe::oem_install, csg_oracle_soe::db_target_properties
Platform:    redhat 7.3
Tags:        oraclesoe, primary, SOED

Node Name:   soed0116
Environment: _default
FQDN:        soed0116.example.com
IP:          192.168.2.41
Run List:    recipe[csg_oracle_soe::standby]
Roles:       
Recipes:     csg_oracle_soe::standby, csg_oracle_soe::base, csg_oracle_soe::ora_dirs, csg_oracle_soe::grid_install, csg_oracle_soe::grid_opatch, csg_oracle_soe::grid_psu, csg_oracle_soe::grid_setup, csg_oracle_soe::db_install, csg_oracle_soe::db_opatch, csg_oracle_soe::db_psu, csg_oracle_soe::db_create_standby, csg_oracle_soe::oem_install, csg_oracle_soe::db_target_properties
Platform:    redhat 7.3
Tags:        oraclesoe, standby, SOED

Here is a similar search for nodes built with a policy:

$ knife search node 'sid:SOED0122' -c ~/.chef/client.rb
0 items found

Here are one of the policy node’s attributes:

$ knife node show `hostname -s` -m -c ~/.chef/client.rb          
Node Name:   soed0122
Policy Name:  oracle_soe_primary
Policy Group: oraclesoe-dev
FQDN:        soed0122.example.com
IP:          192.168.2.50
Run List:    recipe[poise-hoist::default], recipe[csg_oracle_soe::primary]
Recipes:     poise-hoist::default, csg_oracle_soe::primary, csg_oracle_soe::base, csg_oracle_soe::ora_dirs, csg_oracle_soe::grid_install, csg_oracle_soe::grid_opatch, csg_oracle_soe::grid_psu, csg_oracle_soe::grid_setup, csg_oracle_soe::db_install, csg_oracle_soe::db_opatch, csg_oracle_soe::db_psu, csg_oracle_soe::db_create_primary, csg_oracle_soe::oem_install, csg_oracle_soe::db_target_properties
Platform:    redhat 7.3
Tags:        oraclesoe, primary, SOED
Attributes:
csg:
  oracle_soe:
    db_instances:
      app:      Test_SoE
      charset:  AL32UTF8
      dg_mode:  MA
      dr_loc:   XXX
      fs_mode:  FS
      ncharset: UTF8
      sid:      SOED0122
      tier:     Test
      ver:      12.1.0.2
    observers:
      soea0025.example.com
      soea0026.example.com
    product_code:   SOED
    tshirt_size:    entry
policy_group: oraclesoe-dev
policy_name:  oracle_soe_primary
tags:
  oraclesoe
  primary
  SOED
2

I do not know if it gets indexed differently, but probably there extra fields indexed like policy name and group.
Could you try knife search node 'name:SOED0122' instead?

3

I can find the node by name:

$ knife search node 'name:soed0122' -a csg.oracle_soe.db_instances -c ~/.chef/client.rb
1 items found

soed0122:
  csg.oracle_soe.db_instances:
    app:      Test_SoE
    charset:  AL32UTF8
    dg_mode:  MA
    dr_loc:   XXX
    fs_mode:  FS
    ncharset: UTF8
    sid:      SOED0122
    tier:     Test
    ver:      12.1.0.2

Our recipes are based on the feature of nested attribute fields being extracted to the top-level (see https://docs.chef.io/knife_search.html#nested-fields). The use of a policy has somehow changed how attributes are stored.

4

I do not know what could be the issue…
How come the knife search node 'sid:SOED0115' returned 2 node objects? You have the same flattened sid values on 2 different nodes? Could you please also list the attributes of these nodes?

Have you tried searching for the flattened attributes with full path?
knife search node 'csg_oracle..._sid:soed0122'

5

The two nodes are a database cluster, so the same sid runs on both. The search is suppose to return two nodes, which the FQDNs are used to build the Oracle tnsnames.ora file from a template. Here are the attributes from the nodes built without a policyfile:

$ knife search node 'sid:SOED0115' -a csg.oracle_soe.db_instances -c ~/.chef/client.rb
2 items found

soed0115:
  csg.oracle_soe.db_instances:
    app:      Test_SoE
    charset:  AL32UTF8
    dg_mode:  MA
    dr_loc:   XXX
    fs_mode:  FS
    ncharset: UTF8
    sid:      SOED0115
    tier:     Test
    ver:      12.1.0.2

soed0116:
  csg.oracle_soe.db_instances:
    app:      Test_SoE
    charset:  AL32UTF8
    dg_mode:  MA
    dr_loc:   XXX
    fs_mode:  FS
    ncharset: UTF8
    sid:      SOED0115
    tier:     Test
    ver:      12.1.0.2

Trying to search by the full path doesn’t seem to work:

$ knife search node 'csg.oracle_soe.db_instances.sid:SOED0115' -c ~/.chef/client.rb                               
0 items found
6

Seems like our Chef server might have been having problems during my policyfile testing. Soon afterwards, I started receiving HTTP 500 Internal Server Errors due to filesystems filling up. This may have impacted the indexing of the two, new nodes (purely speculation) which is why they were not searchable by nested attributes. After cleaning up the filesystems, a brand new set of nodes built successfully with policyfiles. The search is behaving like I’d expect. Thanks for the suggestions, @simark.