Problem capturing Supervisor logs with core/fluentd

Habitat Help
1

I am attempting to use core/fluentd to capture the Habitat Supervisor logs to do… something… with them.

I am appending gem 'fluent-plugin-systemd' to the core plan.sh so that I can use the systemd source type in fluentd, as below:

<source>
  @type systemd
  tag hab-sup
  path /var/jog/journal
  match [{ "_SYSTEMD_UNIT": "hab-sup-default.service" }]
  read_from_head true
</source>

When loading my fluentd service, I am seeing an error:

Jun 08 15:57:32 aup1.tmeu.eunstg1.websys.tmcs hab[538]: fluentd.default(O): /hab/pkgs/tm/fluentd/1.0.2/20180608143101/vendor/bundle/ruby/2.4.0/gems/ffi-1.9.25/lib/ffi.rb:6:in `require': libffi.so.6: cannot open shared object file: No such file or directory - /hab/pkgs/tm/fluentd/1.0.2/20180608143101/vendor/bundle/ruby/2.4.0/gems/ffi-1.9.25/lib/ffi_c.so (LoadError)

The file in question appears to be present:

$ ll /hab/pkgs/tm/fluentd/1.0.2/20180608143101/vendor/bundle/ruby/2.4.0/gems/ffi-1.9.25/lib/ffi_c.so
-rwxr-xr-x 1 root root 144520 Jun  8 14:31 /hab/pkgs/tm/fluentd/1.0.2/20180608143101/vendor/bundle/ruby/2.4.0/gems/ffi-1.9.25/lib/ffi_c.so

Has anybody encountered this, or found an equivalent solution for consuming the Supervisor logs?

2

@oliversellers can you share your full plan? A link to a repo works too

3

@elliott-davis It is exactly as the core/fluentd plan.sh found here https://github.com/habitat-sh/core-plans/blob/master/fluentd/plan.sh with the addition of gem 'fluent-plugin-systemd' in the Gemfile, i.e.

pkg_name=fluentd
pkg_origin=tm
pkg_version=1.0.2
pkg_deps=(
  core/ruby
  core/coreutils
  core/bundler
)
pkg_build_deps=(
  core/make
  core/gcc
  core/gcc-libs
)
pkg_upstream_url=https://www.fluentd.org/
pkg_description="Fluentd is an open source data collector, which lets \
  you unify the data collection and consumption for a better use and \
  understanding of data."
pkg_license=('Apache-2.0')
pkg_maintainer="INTL Habitat <INTL.Habitat@ticketmaster.co.uk>"
pkg_source=nothing-downloaded-but-build-in-src-cache-anyway
pkg_bin_dirs=(bin)
pkg_svc_user="root"
pkg_svc_group="root"
do_download() {
  return 0
}

do_verify() {
  return 0
}

do_unpack() {
  return 0
}

do_build() {
  local _bundler_dir
  _bundler_dir=$(pkg_path_for bundler)

  export GEM_HOME=${pkg_path}/vendor/bundle
  export GEM_PATH=${_bundler_dir}:${GEM_HOME}

  cat > Gemfile <<-GEMFILE
    source 'https://rubygems.org'
    gem 'fluentd', '= ${pkg_version}'
    gem 'fluent-plugin-systemd'
GEMFILE

  bundle install --jobs 2 --retry 5 --path ./vendor/bundle --binstubs
}

do_install() {
  cp -R . "$pkg_prefix/"
  fix_interpreter "$pkg_prefix/bin/*" core/coreutils bin/env
}
4

I’m going to take a look at this today. I’ve got a couple things I need to finish up this morning but then I’ll see if I can’t help get you unblocked here!

5

Missing is a libffi.so.6. Seems like due gem has such static liking...

6

Yeah I think @epcim is correct. To start you’re going to need to take a runtim dep on core/libffi . That being said, testing this locally does get us past this initial error but i’m testing in a studio where systemd doesn’t exist and it bombs out the moment that systemd libraries get accessed. Can you give this a shot and let me know what you see?

7

We’ve added both core/libffi and core/systemd to the runtime dependencies… we no longer see Gem issues.

1 Like
8

dope! I’ll mark this resolved!