Hi folks,
I’m trying to do a very basic implementation of an
Elasticsearch/Logstash/Kibana stack, all running on the same box. I hoped
someone had done the hard work of figuring out the mostly undocumented
lusis/chef-logstash https://github.com/lusis/chef-logstash cookbooks and
found the elkstack cookbook https://github.com/rackspace-cookbooks/elkstack.
Ultimately, I just need Logstash to read an HAProxy log at
/var/log/haproxy.log.
I’m using the elkstack::recipe but removed the rsyslog::client include at
the end because I don’t need it.
I also don’t actually have HAProxy running, just a log file with a few
sample entries from an actual HAProxy instance.
Here’s what my recipe looks like. The kibana bits are disabled because I
don’t need them for troubleshooting and it just slows down the provisioning
process.
Create a dummy HAProxy file containing a few entries
template node[‘cloudmine’][‘haproxy_log_file’] do
source "haproxy.log.erb"
owner "syslog"
group "adm"
mode 00644
not_if { ::File.exist?(node[‘cloudmine’][‘haproxy_log_file’]) }
notifies :restart, “logstash_service[#{instance_name}]”, :delayed
end
At this point, there’s nothing set up to tell Logstash to read my HAProxy
log file. So I added this simple bit to the recipe:
cookbook_file “input_haproxy” do
path "/opt/logstash/server/etc/conf.d/input_haproxy"
owner "logstash"
group "logstash"
mode 00664
end
And this is the content of the input_haproxy file:
input {
file {
path => "/var/log/haproxy.log"
start_position => "beginning"
type => “haproxy-http”
}
}
If this recipe runs with the cookbook_file block and I restart the Logstash
service… I get nothing in Elasticsearch, like my HAProxy log file is not
being read at all, ever.
However, if I run the recipe without the cookbook_file block, SSH into the
instance, do a $ sudo su -l logstash, manually create the input_haproxy
file in the correct location (I just copy-paste the contents from the file
in my git repo) and assign it the right ownership/permissions, then restart
the Logstash service, after about 30 seconds I can see that my HAProxy log
was read and I can see it in Elasticsearch.
It’s incredibly frustrating and I don’t understand what I’m doing wrong.
It’s like Chef does things differently than me when I do it manually,
resulting in this silent “failure”. I realize Chef does do this a little
bit differently, since I believe it would create the file as root and then
set the correct permissions, whereas I’m doing it directly as the logstash
user… But still, I don’t see why this would matter since, in the end, the
file is owned by the logstash user, with the rw-rw-r-- permissions I want.
Hi folks,
I'm trying to do a very basic implementation of an
Elasticsearch/Logstash/Kibana stack, all running on the same box. I hoped
someone had done the hard work of figuring out the mostly undocumented
lusis/chef-logstash https://github.com/lusis/chef-logstash cookbooks
and found the elkstack cookbook https://github.com/rackspace-cookbooks/elkstack.
Ultimately, I just need Logstash to read an HAProxy log at
/var/log/haproxy.log.
I'm using the elkstack::recipe but removed the rsyslog::client include at
the end because I don't need it.
I also don't actually have HAProxy running, just a log file with a few
sample entries from an actual HAProxy instance.
Here's what my recipe looks like. The kibana bits are disabled because I
don't need them for troubleshooting and it just slows down the provisioning
process.
Create a dummy HAProxy file containing a few entries
template node['cloudmine']['haproxy_log_file'] do
source "haproxy.log.erb"
owner "syslog"
group "adm"
mode 00644
not_if { ::File.exist?(node['cloudmine']['haproxy_log_file']) }
notifies :restart, "logstash_service[#{instance_name}]", :delayed
end
At this point, there's nothing set up to tell Logstash to read my HAProxy
log file. So I added this simple bit to the recipe:
cookbook_file "input_haproxy" do
path "/opt/logstash/server/etc/conf.d/input_haproxy"
owner "logstash"
group "logstash"
mode 00664
end
And this is the content of the input_haproxy file:
input {
file {
path => "/var/log/haproxy.log"
start_position => "beginning"
type => "haproxy-http"
}
}
If this recipe runs with the cookbook_file block and I restart the
Logstash service... I get nothing in Elasticsearch, like my HAProxy log
file is not being read at all, ever.
However, if I run the recipe without the cookbook_file block, SSH into
the instance, do a $ sudo su -l logstash, manually create the
input_haproxy file in the correct location (I just copy-paste the
contents from the file in my git repo) and assign it the right
ownership/permissions, then restart the Logstash service, after about 30
seconds I can see that my HAProxy log was read and I can see it in
Elasticsearch.
It's incredibly frustrating and I don't understand what I'm doing wrong.
It's like Chef does things differently than me when I do it manually,
resulting in this silent "failure". I realize Chef does do this a little
bit differently, since I believe it would create the file as root and then
set the correct permissions, whereas I'm doing it directly as the logstash
user... But still, I don't see why this would matter since, in the end, the
file is owned by the logstash user, with the rw-rw-r-- permissions I want.
For what it’s worth, we’re likely to rewrite the elkstack cookbook to be a library cookbook very shortly.
Until then, did you see that you can provide the same data you’re trying to provide to elkstack with examples found at:
Alternatively, you could also just call the upstream logstash_config resource with your arguments (it’s what elkstack does):
As one of the elkstack authors, I’d be glad to help you diagnose the issue further if you opened an issue on elkstack. Unfortunately, it’s such a complex beast (until the library rewrite) that there’s a number of things that can go wrong.
When you use chef to create your logstash config snippet, can you SSH into the server and see if the file is even being created?
Again, an issue on the Github repo is probably the best way for us to troubleshoot. Beyond that, a minimal reproducible example would also be the best way forward. Perhaps in the form of a test-kitchen suite and serverspec test?
Please also feel free to hit me up on #chef on Freenode for more realtime troubleshooting
Martin B. Smith
From: Fabien Delpierre fabien.delpierre@gmail.com
Sent: Sunday, April 5, 2015 4:52 PM
To: chef
Subject: [chef] Mind-boggling issue with Logstash
Hi folks,
I’m trying to do a very basic implementation of an Elasticsearch/Logstash/Kibana stack, all running on the same box. I hoped someone had done the hard work of figuring out the mostly undocumented lusis/chef-logstashhttps://github.com/lusis/chef-logstash cookbooks and found the elkstack cookbookhttps://github.com/rackspace-cookbooks/elkstack.
Ultimately, I just need Logstash to read an HAProxy log at /var/log/haproxy.log.
I’m using the elkstack::recipe but removed the rsyslog::client include at the end because I don’t need it.
I also don’t actually have HAProxy running, just a log file with a few sample entries from an actual HAProxy instance.
Here’s what my recipe looks like. The kibana bits are disabled because I don’t need them for troubleshooting and it just slows down the provisioning process.
Create a dummy HAProxy file containing a few entries
template node[‘cloudmine’][‘haproxy_log_file’] do
source "haproxy.log.erb"
owner "syslog"
group "adm"
mode 00644
not_if { ::File.exist?(node[‘cloudmine’][‘haproxy_log_file’]) }
notifies :restart, “logstash_service[#{instance_name}]”, :delayed
end
At this point, there’s nothing set up to tell Logstash to read my HAProxy log file. So I added this simple bit to the recipe:
cookbook_file “input_haproxy” do
path "/opt/logstash/server/etc/conf.d/input_haproxy"
owner "logstash"
group "logstash"
mode 00664
end
And this is the content of the input_haproxy file:
input {
file {
path => "/var/log/haproxy.log"
start_position => "beginning"
type => “haproxy-http”
}
}
If this recipe runs with the cookbook_file block and I restart the Logstash service… I get nothing in Elasticsearch, like my HAProxy log file is not being read at all, ever.
However, if I run the recipe without the cookbook_file block, SSH into the instance, do a $ sudo su -l logstash, manually create the input_haproxy file in the correct location (I just copy-paste the contents from the file in my git repo) and assign it the right ownership/permissions, then restart the Logstash service, after about 30 seconds I can see that my HAProxy log was read and I can see it in Elasticsearch.
It’s incredibly frustrating and I don’t understand what I’m doing wrong. It’s like Chef does things differently than me when I do it manually, resulting in this silent “failure”. I realize Chef does do this a little bit differently, since I believe it would create the file as root and then set the correct permissions, whereas I’m doing it directly as the logstash user… But still, I don’t see why this would matter since, in the end, the file is owned by the logstash user, with the rw-rw-r-- permissions I want.
Thanks for the suggestion. That's actually already part of the default
configuration and I left it alone. But for some reason, I'm not seeing
anything in stdout, ever.
On Sun, Apr 5, 2015 at 6:12 PM, Benjamin Bytheway bbytheway@gmail.com
wrote:
I would add a debug output to logstash, might help see what's going on.
Hi folks,
I'm trying to do a very basic implementation of an
Elasticsearch/Logstash/Kibana stack, all running on the same box. I hoped
someone had done the hard work of figuring out the mostly undocumented
lusis/chef-logstash https://github.com/lusis/chef-logstash cookbooks
and found the elkstack cookbook https://github.com/rackspace-cookbooks/elkstack.
Ultimately, I just need Logstash to read an HAProxy log at
/var/log/haproxy.log.
I'm using the elkstack::recipe but removed the rsyslog::client include at
the end because I don't need it.
I also don't actually have HAProxy running, just a log file with a few
sample entries from an actual HAProxy instance.
Here's what my recipe looks like. The kibana bits are disabled because I
don't need them for troubleshooting and it just slows down the provisioning
process.
Create a dummy HAProxy file containing a few entries
template node['cloudmine']['haproxy_log_file'] do
source "haproxy.log.erb"
owner "syslog"
group "adm"
mode 00644
not_if { ::File.exist?(node['cloudmine']['haproxy_log_file']) }
notifies :restart, "logstash_service[#{instance_name}]", :delayed
end
At this point, there's nothing set up to tell Logstash to read my HAProxy
log file. So I added this simple bit to the recipe:
cookbook_file "input_haproxy" do
path "/opt/logstash/server/etc/conf.d/input_haproxy"
owner "logstash"
group "logstash"
mode 00664
end
And this is the content of the input_haproxy file:
input {
file {
path => "/var/log/haproxy.log"
start_position => "beginning"
type => "haproxy-http"
}
}
If this recipe runs with the cookbook_file block and I restart the
Logstash service... I get nothing in Elasticsearch, like my HAProxy log
file is not being read at all, ever.
However, if I run the recipe without the cookbook_file block, SSH into
the instance, do a $ sudo su -l logstash, manually create the
input_haproxy file in the correct location (I just copy-paste the
contents from the file in my git repo) and assign it the right
ownership/permissions, then restart the Logstash service, after about 30
seconds I can see that my HAProxy log was read and I can see it in
Elasticsearch.
It's incredibly frustrating and I don't understand what I'm doing wrong.
It's like Chef does things differently than me when I do it manually,
resulting in this silent "failure". I realize Chef does do this a little
bit differently, since I believe it would create the file as root and then
set the correct permissions, whereas I'm doing it directly as the logstash
user... But still, I don't see why this would matter since, in the end, the
file is owned by the logstash user, with the rw-rw-r-- permissions I want.
Hi Martin,
Thanks for your answer. I wasn't expecting a reply from someone who
actually maintains the elkstack cookbook!
I frankly do not think your cookbook is at fault here, it does everything I
ask from it -- which isn't much -- just fine, it's the customization part I
need that does not happen as expected, and that's outside of the scope of
what your cookbook does for me in the current state of my recipe.
Yes, when I use cookbook_file, the input_haproxy file absolutely gets
created. Whether I do it through the recipe or manually, the file ends up
there with the same permissions, ownership and contents, which is why it's
so frustrating that it doesn't work through the recipe.
I had a faint glimmer of hope when you mentioned the custom_logstash
attributes stack that's actually handled by your stack_commons cookbook --
I never figured out how all those input templates were being dropped into
Logstash's /etc/conf.d directory, so knowing that allows me to overwrite
them and only insert the one I need, which I just did. You're right that
the elkstack cookbook is rather complex, but I'm not one to judge, I'm a
beginner and actually reading your cookbook taught me a few things I never
knew were possible. The cookbook is coded with a rather higher level of
skill and complexity than I typically see in the Chef space.
Anyway, while overwriting the
default['elkstack']['config']['custom_logstash']['name']
attribute to include just my HAProxy input config helped me remove all the
other default inputs that were being set in Logstash, unfortunately it
didn't change the issue I'm having. It's maddening!
I'll try opening an issue on GH to troubleshoot, but I'm running out of
time, which of course has nothing to do with you.
As one of the elkstack authors, I'd be glad to help you diagnose the
issue further if you opened an issue on elkstack. Unfortunately, it's such
a complex beast (until the library rewrite) that there's a number of things
that can go wrong.
When you use chef to create your logstash config snippet, can you SSH
into the server and see if the file is even being created?
Again, an issue on the Github repo is probably the best way for us to
troubleshoot. Beyond that, a minimal reproducible example would also be the
best way forward. Perhaps in the form of a test-kitchen suite and
serverspec test?
Please also feel free to hit me up on Chef Infra (archive) on Freenode for more
realtime troubleshooting
- Martin B. Smith
------------------------------
From: Fabien Delpierre fabien.delpierre@gmail.com Sent: Sunday, April 5, 2015 4:52 PM To: chef Subject: [chef] Mind-boggling issue with Logstash
Hi folks,
I'm trying to do a very basic implementation of an
Elasticsearch/Logstash/Kibana stack, all running on the same box. I hoped
someone had done the hard work of figuring out the mostly undocumented
lusis/chef-logstash https://github.com/lusis/chef-logstash cookbooks
and found the elkstack cookbook https://github.com/rackspace-cookbooks/elkstack.
Ultimately, I just need Logstash to read an HAProxy log at
/var/log/haproxy.log.
I'm using the elkstack::recipe but removed the rsyslog::client include
at the end because I don't need it.
I also don't actually have HAProxy running, just a log file with a few
sample entries from an actual HAProxy instance.
Here's what my recipe looks like. The kibana bits are disabled because I
don't need them for troubleshooting and it just slows down the provisioning
process.
Create a dummy HAProxy file containing a few entries
template node['cloudmine']['haproxy_log_file'] do
source "haproxy.log.erb"
owner "syslog"
group "adm"
mode 00644
not_if { ::File.exist?(node['cloudmine']['haproxy_log_file']) }
notifies :restart, "logstash_service[#{instance_name}]", :delayed
end
At this point, there's nothing set up to tell Logstash to read my
HAProxy log file. So I added this simple bit to the recipe:
cookbook_file "input_haproxy" do
path "/opt/logstash/server/etc/conf.d/input_haproxy"
owner "logstash"
group "logstash"
mode 00664
end
And this is the content of the input_haproxy file:
input {
file {
path => "/var/log/haproxy.log"
start_position => "beginning"
type => "haproxy-http"
}
}
If this recipe runs with the cookbook_file block and I restart the
Logstash service... I get nothing in Elasticsearch, like my HAProxy log
file is not being read at all, ever.
However, if I run the recipe without the cookbook_file block, SSH into
the instance, do a $ sudo su -l logstash, manually create the
input_haproxy file in the correct location (I just copy-paste the
contents from the file in my git repo) and assign it the right
ownership/permissions, then restart the Logstash service, after about 30
seconds I can see that my HAProxy log was read and I can see it in
Elasticsearch.
It's incredibly frustrating and I don't understand what I'm doing wrong.
It's like Chef does things differently than me when I do it manually,
resulting in this silent "failure". I realize Chef does do this a little
bit differently, since I believe it would create the file as root and then
set the correct permissions, whereas I'm doing it directly as the logstash
user... But still, I don't see why this would matter since, in the end, the
file is owned by the logstash user, with the rw-rw-r-- permissions I want.
Hi Martin,
Thanks for your answer. I wasn't expecting a reply from someone who actually maintains the elkstack cookbook!
I frankly do not think your cookbook is at fault here, it does everything I ask from it -- which isn't much -- just fine, it's the customization part I need that does not happen as expected, and that's outside of the scope of what your cookbook does for me in the current state of my recipe.
Yes, when I use cookbook_file, the input_haproxy file absolutely gets created. Whether I do it through the recipe or manually, the file ends up there with the same permissions, ownership and contents, which is why it's so frustrating that it doesn't work through the recipe.
I had a faint glimmer of hope when you mentioned the custom_logstash attributes stack that's actually handled by your stack_commons cookbook -- I never figured out how all those input templates were being dropped into Logstash's /etc/conf.d directory, so knowing that allows me to overwrite them and only insert the one I need, which I just did. You're right that the elkstack cookbook is rather complex, but I'm not one to judge, I'm a beginner and actually reading your cookbook taught me a few things I never knew were possible. The cookbook is coded with a rather higher level of skill and complexity than I typically see in the Chef space.
Anyway, while overwriting the default['elkstack']['config']['custom_logstash']['name'] attribute to include just my HAProxy input config helped me remove all the other default inputs that were being set in Logstash, unfortunately it didn't change the issue I'm having. It's maddening!
I'll try opening an issue on GH to troubleshoot, but I'm running out of time, which of course has nothing to do with you.
As one of the elkstack authors, I'd be glad to help you diagnose the issue further if you opened an issue on elkstack. Unfortunately, it's such a complex beast (until the library rewrite) that there's a number of things that can go wrong.
When you use chef to create your logstash config snippet, can you SSH into the server and see if the file is even being created?
Again, an issue on the Github repo is probably the best way for us to troubleshoot. Beyond that, a minimal reproducible example would also be the best way forward. Perhaps in the form of a test-kitchen suite and serverspec test?
Please also feel free to hit me up on Chef Infra (archive) on Freenode for more realtime troubleshooting
Martin B. Smith
From: Fabien Delpierre fabien.delpierre@gmail.com
Sent: Sunday, April 5, 2015 4:52 PM
To: chef
Subject: [chef] Mind-boggling issue with Logstash
Hi folks,
I'm trying to do a very basic implementation of an Elasticsearch/Logstash/Kibana stack, all running on the same box. I hoped someone had done the hard work of figuring out the mostly undocumented lusis/chef-logstash cookbooks and found the elkstack cookbook.
Ultimately, I just need Logstash to read an HAProxy log at /var/log/haproxy.log.
I'm using the elkstack::recipe but removed the rsyslog::client include at the end because I don't need it.
I also don't actually have HAProxy running, just a log file with a few sample entries from an actual HAProxy instance.
Here's what my recipe looks like. The kibana bits are disabled because I don't need them for troubleshooting and it just slows down the provisioning process.
Create a dummy HAProxy file containing a few entries
template node['cloudmine']['haproxy_log_file'] do
source "haproxy.log.erb"
owner "syslog"
group "adm"
mode 00644
not_if { ::File.exist?(node['cloudmine']['haproxy_log_file']) }
notifies :restart, "logstash_service[#{instance_name}]", :delayed
end
At this point, there's nothing set up to tell Logstash to read my HAProxy log file. So I added this simple bit to the recipe:
cookbook_file "input_haproxy" do
path "/opt/logstash/server/etc/conf.d/input_haproxy"
owner "logstash"
group "logstash"
mode 00664
end
And this is the content of the input_haproxy file:
input {
file {
path => "/var/log/haproxy.log"
start_position => "beginning"
type => "haproxy-http"
}
}
If this recipe runs with the cookbook_file block and I restart the Logstash service... I get nothing in Elasticsearch, like my HAProxy log file is not being read at all, ever.
However, if I run the recipe without the cookbook_file block, SSH into the instance, do a $ sudo su -l logstash, manually create the input_haproxy file in the correct location (I just copy-paste the contents from the file in my git repo) and assign it the right ownership/permissions, then restart the Logstash service, after about 30 seconds I can see that my HAProxy log was read and I can see it in Elasticsearch.
It's incredibly frustrating and I don't understand what I'm doing wrong. It's like Chef does things differently than me when I do it manually, resulting in this silent "failure". I realize Chef does do this a little bit differently, since I believe it would create the file as root and then set the correct permissions, whereas I'm doing it directly as the logstash user... But still, I don't see why this would matter since, in the end, the file is owned by the logstash user, with the rw-rw-r-- permissions I want.
I trimmed about 10,000 lines that all looked like this:
==> default: [2015-04-06T23:56:26+00:00] DEBUG: Chef::HTTP::StreamHandler
calling Chef::HTTP::Decompressor::NoopInflater#handle_chunk
==> default: [2015-04-06T23:56:26+00:00] DEBUG: Chef::HTTP::StreamHandler
calling Chef::HTTP::ValidateContentLength::ContentLengthCounter#handle_chunk
It looks like that's what Chef logs while downloading a file.
Other than the debug level of output, the output looks the same as during
my earlier testing.
Interestingly, if you look at line 2607 and below, you can see an exception
was thrown and the Chef run exited unsuccessfully. I'd never seen that
failure so I just ran $ vagrant provision and it completed successfully
this time.
But in a strange twist of fate, and without me having changed any of the
code... I just did a run from scratch to produce this gist, and for ****s
and giggles tried to see if my problem had vanished at the end, and yep,
the problem has vanished. Or at least, at the end of this particular run,
I'm not seeing the problem. In a sense, that's even more frustrating
because I spent hours troubleshooting this. Finally today I deleted
everything from my computer, even the Vagrant box/template and the code
repository, then I re-grabbed everything to produce the above gist, only to
find that things are now working fine apparently...
However, for the sake of sanity, I had to do a $ vagrant destroy followed
by another $ vagrant up... and at the end, the app is not working again.
The only differences between the previous run and this one were that I
didn't have the Chef output verbosity set to debug, and I didn't have to
download the Vagrant box from the Internet as I already had it from the
previous run.
Here's the gist of the Chef output, but this time, I didn't have the
debug-level verbosity:
I guess there's nothing to do but try a third time, with debug verbosity
again...
Again I had the same failure mid-run, with the weird error involving
/var/chef/cache/server.tar.gz, but again I just ran $ vagrant provision and
the run completed fine. And the app is working as intended. I really,
really don't understand this.
On Mon, Apr 6, 2015 at 5:46 PM, Aimelyne Mochiron aimelynem@gmail.com
wrote:
Hi Fabien,
Could you please post the output you see from Chef in a gist? Pass it the
-ldebug switch for max verbosity.
Hi Martin,
Thanks for your answer. I wasn't expecting a reply from someone who
actually maintains the elkstack cookbook!
I frankly do not think your cookbook is at fault here, it does everything
I ask from it -- which isn't much -- just fine, it's the customization part
I need that does not happen as expected, and that's outside of the scope of
what your cookbook does for me in the current state of my recipe.
Yes, when I use cookbook_file, the input_haproxy file absolutely gets
created. Whether I do it through the recipe or manually, the file ends up
there with the same permissions, ownership and contents, which is why it's
so frustrating that it doesn't work through the recipe.
I had a faint glimmer of hope when you mentioned the custom_logstash
attributes stack that's actually handled by your stack_commons cookbook --
I never figured out how all those input templates were being dropped into
Logstash's /etc/conf.d directory, so knowing that allows me to overwrite
them and only insert the one I need, which I just did. You're right that
the elkstack cookbook is rather complex, but I'm not one to judge, I'm a
beginner and actually reading your cookbook taught me a few things I never
knew were possible. The cookbook is coded with a rather higher level of
skill and complexity than I typically see in the Chef space.
Anyway, while overwriting the default['elkstack']['config']['custom_logstash']['name']
attribute to include just my HAProxy input config helped me remove all the
other default inputs that were being set in Logstash, unfortunately it
didn't change the issue I'm having. It's maddening!
I'll try opening an issue on GH to troubleshoot, but I'm running out of
time, which of course has nothing to do with you.
As one of the elkstack authors, I'd be glad to help you diagnose the
issue further if you opened an issue on elkstack. Unfortunately, it's such
a complex beast (until the library rewrite) that there's a number of things
that can go wrong.
When you use chef to create your logstash config snippet, can you SSH
into the server and see if the file is even being created?
Again, an issue on the Github repo is probably the best way for us to
troubleshoot. Beyond that, a minimal reproducible example would also be the
best way forward. Perhaps in the form of a test-kitchen suite and
serverspec test?
Please also feel free to hit me up on Chef Infra (archive) on Freenode for more realtime
troubleshooting
- Martin B. Smith
------------------------------
From: Fabien Delpierre fabien.delpierre@gmail.com Sent: Sunday, April 5, 2015 4:52 PM To: chef Subject: [chef] Mind-boggling issue with Logstash
Hi folks,
I'm trying to do a very basic implementation of an
Elasticsearch/Logstash/Kibana stack, all running on the same box. I hoped
someone had done the hard work of figuring out the mostly undocumented
lusis/chef-logstash https://github.com/lusis/chef-logstash cookbooks
and found the elkstack cookbook https://github.com/rackspace-cookbooks/elkstack.
Ultimately, I just need Logstash to read an HAProxy log at
/var/log/haproxy.log.
I'm using the elkstack::recipe but removed the rsyslog::client include
at the end because I don't need it.
I also don't actually have HAProxy running, just a log file with a few
sample entries from an actual HAProxy instance.
Here's what my recipe looks like. The kibana bits are disabled because
I don't need them for troubleshooting and it just slows down the
provisioning process.
Create a dummy HAProxy file containing a few entries
template node['cloudmine']['haproxy_log_file'] do
source "haproxy.log.erb"
owner "syslog"
group "adm"
mode 00644
not_if { ::File.exist?(node['cloudmine']['haproxy_log_file']) }
notifies :restart, "logstash_service[#{instance_name}]", :delayed
end
At this point, there's nothing set up to tell Logstash to read my
HAProxy log file. So I added this simple bit to the recipe:
cookbook_file "input_haproxy" do
path "/opt/logstash/server/etc/conf.d/input_haproxy"
owner "logstash"
group "logstash"
mode 00664
end
And this is the content of the input_haproxy file:
input {
file {
path => "/var/log/haproxy.log"
start_position => "beginning"
type => "haproxy-http"
}
}
If this recipe runs with the cookbook_file block and I restart the
Logstash service... I get nothing in Elasticsearch, like my HAProxy log
file is not being read at all, ever.
However, if I run the recipe without the cookbook_file block, SSH into
the instance, do a $ sudo su -l logstash, manually create the
input_haproxy file in the correct location (I just copy-paste the
contents from the file in my git repo) and assign it the right
ownership/permissions, then restart the Logstash service, after about 30
seconds I can see that my HAProxy log was read and I can see it in
Elasticsearch.
It's incredibly frustrating and I don't understand what I'm doing
wrong. It's like Chef does things differently than me when I do it
manually, resulting in this silent "failure". I realize Chef does do this a
little bit differently, since I believe it would create the file as root
and then set the correct permissions, whereas I'm doing it directly as the
logstash user... But still, I don't see why this would matter since, in the
end, the file is owned by the logstash user, with the rw-rw-r-- permissions
I want.
Sure, why not. gist:c9e977d9cc36472abd56 · GitHub
I trimmed about 10,000 lines that all looked like this:
==> default: [2015-04-06T23:56:26+00:00] DEBUG: Chef::HTTP::StreamHandler calling Chef::HTTP::Decompressor::NoopInflater#handle_chunk
==> default: [2015-04-06T23:56:26+00:00] DEBUG: Chef::HTTP::StreamHandler calling Chef::HTTP::ValidateContentLength::ContentLengthCounter#handle_chunk
It looks like that's what Chef logs while downloading a file.
Other than the debug level of output, the output looks the same as during my earlier testing.
Interestingly, if you look at line 2607 and below, you can see an exception was thrown and the Chef run exited unsuccessfully. I'd never seen that failure so I just ran $ vagrant provision and it completed successfully this time.
But in a strange twist of fate, and without me having changed any of the code... I just did a run from scratch to produce this gist, and for ****s and giggles tried to see if my problem had vanished at the end, and yep, the problem has vanished. Or at least, at the end of this particular run, I'm not seeing the problem. In a sense, that's even more frustrating because I spent hours troubleshooting this. Finally today I deleted everything from my computer, even the Vagrant box/template and the code repository, then I re-grabbed everything to produce the above gist, only to find that things are now working fine apparently...
However, for the sake of sanity, I had to do a $ vagrant destroy followed by another $ vagrant up... and at the end, the app is not working again. The only differences between the previous run and this one were that I didn't have the Chef output verbosity set to debug, and I didn't have to download the Vagrant box from the Internet as I already had it from the previous run.
Here's the gist of the Chef output, but this time, I didn't have the debug-level verbosity: gist:dc2e9e8a4b7c5aec7297 · GitHub
I guess there's nothing to do but try a third time, with debug verbosity again...
Again I had the same failure mid-run, with the weird error involving /var/chef/cache/server.tar.gz, but again I just ran $ vagrant provision and the run completed fine. And the app is working as intended. I really, really don't understand this.
On Mon, Apr 6, 2015 at 5:46 PM, Aimelyne Mochiron aimelynem@gmail.com wrote:
Hi Fabien,
Could you please post the output you see from Chef in a gist? Pass it the -ldebug switch for max verbosity.
Hi Martin,
Thanks for your answer. I wasn't expecting a reply from someone who actually maintains the elkstack cookbook!
I frankly do not think your cookbook is at fault here, it does everything I ask from it -- which isn't much -- just fine, it's the customization part I need that does not happen as expected, and that's outside of the scope of what your cookbook does for me in the current state of my recipe.
Yes, when I use cookbook_file, the input_haproxy file absolutely gets created. Whether I do it through the recipe or manually, the file ends up there with the same permissions, ownership and contents, which is why it's so frustrating that it doesn't work through the recipe.
I had a faint glimmer of hope when you mentioned the custom_logstash attributes stack that's actually handled by your stack_commons cookbook -- I never figured out how all those input templates were being dropped into Logstash's /etc/conf.d directory, so knowing that allows me to overwrite them and only insert the one I need, which I just did. You're right that the elkstack cookbook is rather complex, but I'm not one to judge, I'm a beginner and actually reading your cookbook taught me a few things I never knew were possible. The cookbook is coded with a rather higher level of skill and complexity than I typically see in the Chef space.
Anyway, while overwriting the default['elkstack']['config']['custom_logstash']['name'] attribute to include just my HAProxy input config helped me remove all the other default inputs that were being set in Logstash, unfortunately it didn't change the issue I'm having. It's maddening!
I'll try opening an issue on GH to troubleshoot, but I'm running out of time, which of course has nothing to do with you.
As one of the elkstack authors, I'd be glad to help you diagnose the issue further if you opened an issue on elkstack. Unfortunately, it's such a complex beast (until the library rewrite) that there's a number of things that can go wrong.
When you use chef to create your logstash config snippet, can you SSH into the server and see if the file is even being created?
Again, an issue on the Github repo is probably the best way for us to troubleshoot. Beyond that, a minimal reproducible example would also be the best way forward. Perhaps in the form of a test-kitchen suite and serverspec test?
Please also feel free to hit me up on Chef Infra (archive) on Freenode for more realtime troubleshooting
Martin B. Smith
From: Fabien Delpierre fabien.delpierre@gmail.com
Sent: Sunday, April 5, 2015 4:52 PM
To: chef
Subject: [chef] Mind-boggling issue with Logstash
Hi folks,
I'm trying to do a very basic implementation of an Elasticsearch/Logstash/Kibana stack, all running on the same box. I hoped someone had done the hard work of figuring out the mostly undocumented lusis/chef-logstash cookbooks and found the elkstack cookbook.
Ultimately, I just need Logstash to read an HAProxy log at /var/log/haproxy.log.
I'm using the elkstack::recipe but removed the rsyslog::client include at the end because I don't need it.
I also don't actually have HAProxy running, just a log file with a few sample entries from an actual HAProxy instance.
Here's what my recipe looks like. The kibana bits are disabled because I don't need them for troubleshooting and it just slows down the provisioning process.
Create a dummy HAProxy file containing a few entries
template node['cloudmine']['haproxy_log_file'] do
source "haproxy.log.erb"
owner "syslog"
group "adm"
mode 00644
not_if { ::File.exist?(node['cloudmine']['haproxy_log_file']) }
notifies :restart, "logstash_service[#{instance_name}]", :delayed
end
At this point, there's nothing set up to tell Logstash to read my HAProxy log file. So I added this simple bit to the recipe:
cookbook_file "input_haproxy" do
path "/opt/logstash/server/etc/conf.d/input_haproxy"
owner "logstash"
group "logstash"
mode 00664
end
And this is the content of the input_haproxy file:
input {
file {
path => "/var/log/haproxy.log"
start_position => "beginning"
type => "haproxy-http"
}
}
If this recipe runs with the cookbook_file block and I restart the Logstash service... I get nothing in Elasticsearch, like my HAProxy log file is not being read at all, ever.
However, if I run the recipe without the cookbook_file block, SSH into the instance, do a $ sudo su -l logstash, manually create the input_haproxy file in the correct location (I just copy-paste the contents from the file in my git repo) and assign it the right ownership/permissions, then restart the Logstash service, after about 30 seconds I can see that my HAProxy log was read and I can see it in Elasticsearch.
It's incredibly frustrating and I don't understand what I'm doing wrong. It's like Chef does things differently than me when I do it manually, resulting in this silent "failure". I realize Chef does do this a little bit differently, since I believe it would create the file as root and then set the correct permissions, whereas I'm doing it directly as the logstash user... But still, I don't see why this would matter since, in the end, the file is owned by the logstash user, with the rw-rw-r-- permissions I want.
Nope, not behind a proxy. And what's interesting is I can run Chef ad
nauseam without the debug verbosity and never see this download fail, but
it fails every single time if I enable debug verbosity.
The root issue in the first instance was a Content-Length mismatch that
aborted the download of the logstash tarball (< gist:c9e977d9cc36472abd56 · GitHub>).
Are you behind a proxy?
am
On 7 Apr 2015, at 03:52, Fabien Delpierre <fabien.delpierre@gmail.com
<javascript:_e(%7B%7D,'cvml','fabien.delpierre@gmail.com');>> wrote:
Sure, why not. gist:c9e977d9cc36472abd56 · GitHub
I trimmed about 10,000 lines that all looked like this:
==> default: [2015-04-06T23:56:26+00:00] DEBUG: Chef::HTTP::StreamHandler
calling Chef::HTTP::Decompressor::NoopInflater#handle_chunk
==> default: [2015-04-06T23:56:26+00:00] DEBUG: Chef::HTTP::StreamHandler
calling Chef::HTTP::ValidateContentLength::ContentLengthCounter#handle_chunk
It looks like that's what Chef logs while downloading a file.
Other than the debug level of output, the output looks the same as during
my earlier testing.
Interestingly, if you look at line 2607 and below, you can see an
exception was thrown and the Chef run exited unsuccessfully. I'd never seen
that failure so I just ran $ vagrant provision and it completed
successfully this time.
But in a strange twist of fate, and without me having changed any of the
code... I just did a run from scratch to produce this gist, and for ****s
and giggles tried to see if my problem had vanished at the end, and yep,
the problem has vanished. Or at least, at the end of this particular run,
I'm not seeing the problem. In a sense, that's even more frustrating
because I spent hours troubleshooting this. Finally today I deleted
everything from my computer, even the Vagrant box/template and the code
repository, then I re-grabbed everything to produce the above gist, only to
find that things are now working fine apparently...
However, for the sake of sanity, I had to do a $ vagrant destroy followed
by another $ vagrant up... and at the end, the app is not working again.
The only differences between the previous run and this one were that I
didn't have the Chef output verbosity set to debug, and I didn't have to
download the Vagrant box from the Internet as I already had it from the
previous run.
Here's the gist of the Chef output, but this time, I didn't have the
debug-level verbosity: gist:dc2e9e8a4b7c5aec7297 · GitHub
I guess there's nothing to do but try a third time, with debug verbosity
again...
Again I had the same failure mid-run, with the weird error involving
/var/chef/cache/server.tar.gz, but again I just ran $ vagrant provision
and the run completed fine. And the app is working as intended. I really,
really don't understand this.
On Mon, Apr 6, 2015 at 5:46 PM, Aimelyne Mochiron <aimelynem@gmail.com
<javascript:_e(%7B%7D,'cvml','aimelynem@gmail.com');>> wrote:
Hi Fabien,
Could you please post the output you see from Chef in a gist? Pass it the
-ldebug switch for max verbosity.
Cheers,
am
On 6 Apr 2015, at 04:52, Fabien Delpierre <fabien.delpierre@gmail.com
<javascript:_e(%7B%7D,'cvml','fabien.delpierre@gmail.com');>> wrote:
Hi Martin,
Thanks for your answer. I wasn't expecting a reply from someone who
actually maintains the elkstack cookbook!
I frankly do not think your cookbook is at fault here, it does everything
I ask from it -- which isn't much -- just fine, it's the customization part
I need that does not happen as expected, and that's outside of the scope of
what your cookbook does for me in the current state of my recipe.
Yes, when I use cookbook_file, the input_haproxy file absolutely gets
created. Whether I do it through the recipe or manually, the file ends up
there with the same permissions, ownership and contents, which is why it's
so frustrating that it doesn't work through the recipe.
I had a faint glimmer of hope when you mentioned the custom_logstash
attributes stack that's actually handled by your stack_commons cookbook --
I never figured out how all those input templates were being dropped into
Logstash's /etc/conf.d directory, so knowing that allows me to overwrite
them and only insert the one I need, which I just did. You're right that
the elkstack cookbook is rather complex, but I'm not one to judge, I'm a
beginner and actually reading your cookbook taught me a few things I never
knew were possible. The cookbook is coded with a rather higher level of
skill and complexity than I typically see in the Chef space.
Anyway, while overwriting the default['elkstack']['config']['custom_logstash']['name']
attribute to include just my HAProxy input config helped me remove all
the other default inputs that were being set in Logstash, unfortunately it
didn't change the issue I'm having. It's maddening!
I'll try opening an issue on GH to troubleshoot, but I'm running out of
time, which of course has nothing to do with you.
On Sun, Apr 5, 2015 at 6:12 PM, Martin Smith <martin.smith@rackspace.com
<javascript:_e(%7B%7D,'cvml','martin.smith@rackspace.com');>> wrote:
Hi Fabien,
For what it's worth, we're likely to rewrite the elkstack cookbook to be
a library cookbook very shortly.
Until then, did you see that you can provide the same data you're trying
to provide to elkstack with examples found at:
As one of the elkstack authors, I'd be glad to help you diagnose the
issue further if you opened an issue on elkstack. Unfortunately, it's such
a complex beast (until the library rewrite) that there's a number of things
that can go wrong.
When you use chef to create your logstash config snippet, can you SSH
into the server and see if the file is even being created?
Again, an issue on the Github repo is probably the best way for us to
troubleshoot. Beyond that, a minimal reproducible example would also be the
best way forward. Perhaps in the form of a test-kitchen suite and
serverspec test?
Please also feel free to hit me up on Chef Infra (archive) on Freenode for more
realtime troubleshooting
- Martin B. Smith
------------------------------
From: Fabien Delpierre <fabien.delpierre@gmail.com
<javascript:_e(%7B%7D,'cvml','fabien.delpierre@gmail.com');>> Sent: Sunday, April 5, 2015 4:52 PM To: chef Subject: [chef] Mind-boggling issue with Logstash
Hi folks,
I'm trying to do a very basic implementation of an
Elasticsearch/Logstash/Kibana stack, all running on the same box. I hoped
someone had done the hard work of figuring out the mostly undocumented
lusis/chef-logstash https://github.com/lusis/chef-logstash cookbooks
and found the elkstack cookbook https://github.com/rackspace-cookbooks/elkstack.
Ultimately, I just need Logstash to read an HAProxy log at
/var/log/haproxy.log.
I'm using the elkstack::recipe but removed the rsyslog::client include
at the end because I don't need it.
I also don't actually have HAProxy running, just a log file with a few
sample entries from an actual HAProxy instance.
Here's what my recipe looks like. The kibana bits are disabled because
I don't need them for troubleshooting and it just slows down the
provisioning process.
Create a dummy HAProxy file containing a few entries
template node['cloudmine']['haproxy_log_file'] do
source "haproxy.log.erb"
owner "syslog"
group "adm"
mode 00644
not_if { ::File.exist?(node['cloudmine']['haproxy_log_file']) }
notifies :restart, "logstash_service[#{instance_name}]", :delayed
end
At this point, there's nothing set up to tell Logstash to read my
HAProxy log file. So I added this simple bit to the recipe:
cookbook_file "input_haproxy" do
path "/opt/logstash/server/etc/conf.d/input_haproxy"
owner "logstash"
group "logstash"
mode 00664
end
And this is the content of the input_haproxy file:
input {
file {
path => "/var/log/haproxy.log"
start_position => "beginning"
type => "haproxy-http"
}
}
If this recipe runs with the cookbook_file block and I restart the
Logstash service... I get nothing in Elasticsearch, like my HAProxy log
file is not being read at all, ever.
However, if I run the recipe without the cookbook_file block, SSH into
the instance, do a $ sudo su -l logstash, manually create the
input_haproxy file in the correct location (I just copy-paste the
contents from the file in my git repo) and assign it the right
ownership/permissions, then restart the Logstash service, after about 30
seconds I can see that my HAProxy log was read and I can see it in
Elasticsearch.
It's incredibly frustrating and I don't understand what I'm doing
wrong. It's like Chef does things differently than me when I do it
manually, resulting in this silent "failure". I realize Chef does do this a
little bit differently, since I believe it would create the file as root
and then set the correct permissions, whereas I'm doing it directly as the
logstash user... But still, I don't see why this would matter since, in the
end, the file is owned by the logstash user, with the rw-rw-r-- permissions
I want.
