Hello, I have a question about habitat, specifically around the --config-from flag when starting the supervisor. My original thinking was that I could specify a custom path to that flag, which has a custom slapd.ldif in it, which the core/openldap package would then pickup and use (instead of the built-in slapd.ldif from the core package itself). However, that doesn’t seem to be happening–the rendered /hab/svc/openldap/config/slapd.ldif always seems to be content from the pacakge file, not my local file that I want to override it from (no matter what I specify for the --config-from flag). Is this expected behaviour? Or maybe I am mis-understanding what the --config-from flag is supposed to do?
Thanks again for posting this! Taking a look at this now.
I can tell you that I generally use the --config-from flag only in development - when I am making changes to a config file and want to see the results without having to rebuild the entire package every time. (this process is detailed in this section of the docs)
That said, it is possible there might be a bug with the --config-from flag - taking a deeper look.
Just talked with some other members of the team, who concur that this flag is only meant for development purposes.
If you are using this flag outside of development, I recommend you use hab config apply (documented here) to apply your custom config file, rather than --config-from.
Let us know if using hab config apply works for you?
Yes, using hab config apply works fine. The process I am using right now for --config-from is to bootstrap a running OpenLDAP server in my Kitchen instance (which also has a separate package that I need to verify can make LDAP calls during authentication requests). The idea was to override the default slapd.ldif by importing some basic LDAP directory data automagically when the supervisor starts the core/openldap package. I then configure my app to point at that core/openldap instance, and can run my Kitchen Inspec tests against it, verifying authentication works end-to-end. Maybe there’s a better way I should be doing it?
Investigating further 
Typically I see ownership of seed data as belonging to the app and not the data store. That way you’re not having to create unique data store artifacts for each app and can reuse one thing but seed it based on app/environment. I don’t know enough about ldap to know if that’s feasible. Would it be possible to load the data after ldap has started, so you can push that responsibility into test-kitchen, or the provisioning layer for your other environments, possibly into the init hook of your app?
If that doens’t work, one thing you might try doing is creating a myapp-openldap-testharness package that has a pkg_deps=(core/openldap) and its sole responsibility is to ship your test slapd.ldif and load it in the init hook. You’d want to copy other config/hooks into your package as well.
I think that that is going to be an anti-pattern though, since now you’re maintaining an artifact just for testing against and not testing against what you’ll be running elsewhere.
After much thinking and reviewing suggestions from here, I think I’ve come up with an approach that will work, but is still somewhat abusive to Inspec. I don’t know of a better way to do this, so I am open to suggestions (I’m also still a bit hazy on the exact mechanics of testing Habitat packages, so that may be contributing to my confusion).
Assume for now we are creating a custom <origin>/jenkins package with support for LDAP authentication, which we want to test using core/openldap.
The basic issue is that I don’t have a way to bootstrap data in to a running core/openldap instance (whether that instance is running in the jenkins instance itself (alongside jenkins, under the supervisor), or in a separate kitchen-habitat instance (in Docker).
The most self-contained way to go about this is to add the habitat-supervisor.sh script to test/files location, which is then directly invoked by inspec using a command("/path/to/habitat-supervisor.sh") call. This is still a bit abusive to Inspec, but it seems to get the job done.
I am not sure how Inspec determines test suites to run (lexical sorting by file names?), but right now, the supervisor_test.rb is run before anything else in test/smoke/default/*, so it seems to be working.
Here’s my supervisor_test.rb, which has a block like this in it:
describe command('/src/test/files/habitat-census.sh 2 30') do
its('exit_status') { should eq 0 }
its('stderr') { should eq '' }
its('stdout') { should match (/SUCCESS: Got census/) }
end
In my habitat-supervisor.sh, I have to do quite a few things to get the system-under-test (SUT) setup:
- Need to wait for initial quorum of supervisors.
- Need to install dependent packages (
core/jq-static,core/curl,core/openldap, etc.) - Start core/openldap service, wait for it to be
state=up, and LDAP socket open (ready to receive requests). - Import LDAP data using
ldapmodify ... /src/test/files/import.ldif - And then, I can start performing tests. But wait, Jenkins itself is still trying to settle down (because it’s installing plugins and restarting). So, probe Jenkins API with CURL until we get more than 5 200 OK responses back, so we know it’s ready to receive requests.
- Now, we can
hab config applythe ldap.toml, and start to run the actual Inspec tests to test that our<origin>/jenkinspackage can be configured correctly for LDAP authentication using the TOML vars.
This still doesn’t answer my original question about the intent of the --config-from flag. If I could specify a custom location to bootstrap the package data from, that would be ideal–because I could then apply this same pattern to any other package in Habitat my own package(s) consume that I would need for testing, but not necessarily be directly dependent on them.
@kmott sorry I’m late to the party. Could you give an example of how you’re using --config-from and what the directory structure of the target config directory looks like?
Enter a blank studio (setup a Docker -> Habitat portmap if you want to query LDAP outside the studio):
kmott$ mkdir openldap-test
kmott$ cd openldap-test
kmott$ export HAB_DOCKER_OPTS="-p 10389:10389"
kmott$ hab studio enter
Make a new directory /src/habitat/config, and place the following files in it:
slapd.ldif:
dn: cn=config
objectClass: olcGlobal
cn: config
olcArgsFile: {{pkg.svc_var_path}}/run/slapd.args
olcPidFile: {{pkg.svc_var_path}}/run/slapd.pid
dn: cn=schema,cn=config
objectClass: olcSchemaConfig
cn: schema
include: file://{{pkg.path}}/etc/openldap/schema/core.ldif
dn: olcDatabase=frontend,cn=config
objectClass: olcDatabaseConfig
objectClass: olcFrontendConfig
olcDatabase: frontend
dn: olcDatabase=mdb,cn=config
objectClass: olcDatabaseConfig
objectClass: olcMdbConfig
olcDatabase: mdb
olcSuffix: {{cfg.suffix}}
olcRootDN: {{cfg.rootdn}}
olcRootPW: {{cfg.rootpw}}
olcDbDirectory: {{pkg.svc_data_path}}/openldap-data
olcDbIndex: objectClass eq
# Pull in our test data
include: file:///src/habitat/config/import.ldif
import.ldif:
dn: dc=my-domain,dc=com
changetype: add
objectclass: dcObject
objectclass: organization
o: my-domain
dc: my-domain
Terminate the current supervisor if it was started, and restart with --config-from:
kmott$ hab sup term
kmott$ hab sup run --config-from=/src/habitat/config > /hab/sup/default/sup.log 2>&1 &
Load the core/openldap package now:
kmott$ hab svc load core/openldap
Wait a few moments, and check the contents of /hab/svc/openldap/config/slapd.ldif. Note that the rendered content is from the base package slapd.ldif, not mine with the modified include: <path>/import.ldif statement.
Here’s the output from tree for the openldap-test dir in my environment:
kmott$ tree openldap-test
openldap-test
└── habitat
└── config
├── import.ldif
└── slapd.ldif
2 directories, 2 files
I actually think --config-from is broken post supervisor 0.55.0 (the last time I could get a successful test of it. I’m going to file this as an issue and link it back here.