Chef server load testing

Chef Infra (archive)
1

Hi,

I’d like to load test the chef-server (11 or 12) with simulated node registrations - ie. simulating chef-client process.
I’m wondering if anyone is doing anything like this?

My hack thus far is that I have created a jmeter plugin that creates a chef-client process per thread. I had to modify the run_lock.rb code to allow concurrent runs of chef-client on a single node. What I really need is the ability to simulate http payload(s) that ultimately simulates an entire chef-client process - and makes my plugin much lighter and more flexible.

All suggestions welcome!

-marcus

2

On Thu, Mar 12, 2015 at 9:11 AM, Marcus Simonsen msimonsen@blackberry.com
wrote:

Hi,

I'd like to load test the chef-server (11 or 12) with simulated node
registrations - ie. simulating chef-client process.
I'm wondering if anyone is doing anything like this?

My hack thus far is that I have created a jmeter plugin that creates a
chef-client process per thread. I had to modify the run_lock.rb code to
allow concurrent runs of chef-client on a single node. What I really need
is the ability to simulate http payload(s) that ultimately simulates an
entire chef-client process - and makes my plugin much lighter and more
flexible.

All suggestions welcome!

-marcus

Irving built a really cool load tester using Docker instances and
Provisioning(Metal) on AWS last fall:

Your best bet is probably to do things with containers, as that'll allow
you to have actual nodes to play with.

--

3

I wrote a thing like this quite a while ago. Unfortunately the code is not public. What I did was analyze our logs to find the approximate mix of different API calls we were getting and then made a couple of “stories” for simulated chef-client runs doing a mix of different calls (initial run with client registration vs. re-run of an existing client/node pair, search, data bag fetches and so on). The API calls were done using Chef::Rest directly rather than running chef client so I could skip the stuff that wasn’t relevant. Then I prepopulated some required data and ran the simulated chef-client runs in threads using JRuby.

Even if it was open source, I don’t know that the code would actually be usable for most people as there’s a lot of undocumented manual steps required in the setup. I also am not sure if the code has been kept up to date with API changes, as more recently I started using basho_bench with the erlang HTTP client for chef to benchmark server features (this benchmarks a proprietary add-on for chef-server so it is non-public as well).

Anyway, you may be able to approximate my approach by writing ruby scripts that just do the necessary API calls and skip the rest of the client code. One tricky aspect of that is for self-hosted chef servers, you need to put some load on bookshelf (the s3 clone that stores the cookbooks’ files). You’ll probably have to read through the Chef source to see how this works.

Good luck,

--
Daniel DeLeo

On Thursday, March 12, 2015 at 9:11 AM, Marcus Simonsen wrote:

Hi,

I'd like to load test the chef-server (11 or 12) with simulated node registrations - ie. simulating chef-client process.
I'm wondering if anyone is doing anything like this?

My hack thus far is that I have created a jmeter plugin that creates a chef-client process per thread. I had to modify the run_lock.rb code to allow concurrent runs of chef-client on a single node. What I really need is the ability to simulate http payload(s) that ultimately simulates an entire chef-client process - and makes my plugin much lighter and more flexible.

All suggestions welcome!

-marcus

4

Thanks Daniel.

This is more in line with what I want to achieve. A very granular approach to simulating chef client operations against the server.
I've been taking a look into the chef code to understand a bit more what these Chef::Rest calls would look like. If I simulate payload then I won't even have to write any ruby at all, I'll just need to be able to template the http headers and body. My objective is to write as little code as possible.
On the other hand leveraging already working client code would be easier. If you have some samples of how you would simulate this via the Chef api that would be appreciated.
Cheffish gem seems to be very helpful in this respect - but documentation lacking I do have to dig into source.

From: Daniel DeLeo [ddeleo@kallistec.com] on behalf of Daniel DeLeo [dan@kallistec.com]
Sent: 12 March 2015 14:56
To: chef@lists.opscode.com
Subject: [chef] Re: chef server load testing

I wrote a thing like this quite a while ago. Unfortunately the code is not public. What I did was analyze our logs to find the approximate mix of different API calls we were getting and then made a couple of “stories” for simulated chef-client runs doing a mix of different calls (initial run with client registration vs. re-run of an existing client/node pair, search, data bag fetches and so on). The API calls were done using Chef::Rest directly rather than running chef client so I could skip the stuff that wasn’t relevant. Then I prepopulated some required data and ran the simulated chef-client runs in threads using JRuby.

Even if it was open source, I don’t know that the code would actually be usable for most people as there’s a lot of undocumented manual steps required in the setup. I also am not sure if the code has been kept up to date with API changes, as more recently I started using basho_bench with the erlang HTTP client for chef to benchmark server features (this benchmarks a proprietary add-on for chef-server so it is non-public as well).

Anyway, you may be able to approximate my approach by writing ruby scripts that just do the necessary API calls and skip the rest of the client code. One tricky aspect of that is for self-hosted chef servers, you need to put some load on bookshelf (the s3 clone that stores the cookbooks’ files). You’ll probably have to read through the Chef source to see how this works.

Good luck,

--
Daniel DeLeo

On Thursday, March 12, 2015 at 9:11 AM, Marcus Simonsen wrote:

Hi,

I'd like to load test the chef-server (11 or 12) with simulated node registrations - ie. simulating chef-client process.
I'm wondering if anyone is doing anything like this?

My hack thus far is that I have created a jmeter plugin that creates a chef-client process per thread. I had to modify the run_lock.rb code to allow concurrent runs of chef-client on a single node. What I really need is the ability to simulate http payload(s) that ultimately simulates an entire chef-client process - and makes my plugin much lighter and more flexible.

All suggestions welcome!

-marcus

5

Chef::ApiClient::Registration appears promising, and your name on the code!
Am I heading in the right direction?

From: Marcus Simonsen [msimonsen@blackberry.com]
Sent: 12 March 2015 15:26
To: chef@lists.opscode.com
Subject: [chef] RE: Re: chef server load testing

Thanks Daniel.

This is more in line with what I want to achieve. A very granular approach to simulating chef client operations against the server.
I've been taking a look into the chef code to understand a bit more what these Chef::Rest calls would look like. If I simulate payload then I won't even have to write any ruby at all, I'll just need to be able to template the http headers and body. My objective is to write as little code as possible.
On the other hand leveraging already working client code would be easier. If you have some samples of how you would simulate this via the Chef api that would be appreciated.
Cheffish gem seems to be very helpful in this respect - but documentation lacking I do have to dig into source.

From: Daniel DeLeo [ddeleo@kallistec.com] on behalf of Daniel DeLeo [dan@kallistec.com]
Sent: 12 March 2015 14:56
To: chef@lists.opscode.com
Subject: [chef] Re: chef server load testing

I wrote a thing like this quite a while ago. Unfortunately the code is not public. What I did was analyze our logs to find the approximate mix of different API calls we were getting and then made a couple of “stories” for simulated chef-client runs doing a mix of different calls (initial run with client registration vs. re-run of an existing client/node pair, search, data bag fetches and so on). The API calls were done using Chef::Rest directly rather than running chef client so I could skip the stuff that wasn’t relevant. Then I prepopulated some required data and ran the simulated chef-client runs in threads using JRuby.

Even if it was open source, I don’t know that the code would actually be usable for most people as there’s a lot of undocumented manual steps required in the setup. I also am not sure if the code has been kept up to date with API changes, as more recently I started using basho_bench with the erlang HTTP client for chef to benchmark server features (this benchmarks a proprietary add-on for chef-server so it is non-public as well).

Anyway, you may be able to approximate my approach by writing ruby scripts that just do the necessary API calls and skip the rest of the client code. One tricky aspect of that is for self-hosted chef servers, you need to put some load on bookshelf (the s3 clone that stores the cookbooks’ files). You’ll probably have to read through the Chef source to see how this works.

Good luck,

--
Daniel DeLeo

On Thursday, March 12, 2015 at 9:11 AM, Marcus Simonsen wrote:

Hi,

I'd like to load test the chef-server (11 or 12) with simulated node registrations - ie. simulating chef-client process.
I'm wondering if anyone is doing anything like this?

My hack thus far is that I have created a jmeter plugin that creates a chef-client process per thread. I had to modify the run_lock.rb code to allow concurrent runs of chef-client on a single node. What I really need is the ability to simulate http payload(s) that ultimately simulates an entire chef-client process - and makes my plugin much lighter and more flexible.

All suggestions welcome!

-marcus

6

On Thursday, March 12, 2015 at 1:52 PM, Marcus Simonsen wrote:

Chef::ApiClient::Registration appears promising, and your name on the code!
Am I heading in the right direction?

Generally the “model” objects in Chef (node, role, etc.) follow an Active Record type pattern, where each object/class has methods for save/load/update/etc. So you can look though all those classes for how they use Chef::REST (doing git grep ‘def save’ might narrow it down a bit). In a few cases, the logic for running a request is a bit more complicated, and these things tend to have their own classes to encapsulate the behavior for just that one request. Chef::ApiClient::Registration is one of those, as it handles making a request with the validator key to create a new ApiClient object and store the private key on disk, which is what usually happens the very first time you run chef-client on a host. Likewise, the process of syncing cookbooks from the server to a host is a bit complicated, so that has it’s own CookbookSynchronizer class, and the process where chef-client queries the dependency solver to get the list of cookbooks to run is done by PolicyBuilder::ExpandNodeObject.

From: Marcus Simonsen [msimonsen@blackberry.com (mailto:msimonsen@blackberry.com)]
Sent: 12 March 2015 15:26
To: chef@lists.opscode.com (mailto:chef@lists.opscode.com)
Subject: [chef] RE: Re: chef server load testing

Thanks Daniel.

This is more in line with what I want to achieve. A very granular approach to simulating chef client operations against the server.
Cool. Just remember to validate the proportion of different API calls against your real-world usage. There’s a lot of ways that your data and the way you use the API can cause the Chef Server to have varying performance characteristics.

I've been taking a look into the chef code to understand a bit more what these Chef::Rest calls would look like. If I simulate payload then I won't even have to write any ruby at all, I'll just need to be able to template the http headers and body. My objective is to write as little code as possible.

The tricky part is that the HTTP headers include a timestamp, and that timestamp is part of the request signature. The server only allows a small deviation in the current time and the timestamp in the request as a mitigation against replay attacks. It’s possible that the amount of deviation allowed might be configurable, but there’s no documented config option for this. So you need to generate your request headers on-demand or at least a short time before your tests run.

On the other hand leveraging already working client code would be easier. If you have some samples of how you would simulate this via the Chef api that would be appreciated.
Cheffish gem seems to be very helpful in this respect - but documentation lacking I do have to dig into source.

If you are more familiar with some other language than ruby, then you might look around to see if there are any language bindings for whatever language you want to use. I know that at least python, go, php, and java have been written at some point, though I’m not sure if they’re well maintained.

--
Daniel DeLeo

7

Many Thanks Daniel.
I've been able to do basic create nodes and clients, with the rest api. And I'm sure with more digging that I can load cookbooks etc...
It appears that the cost of starting the ruby process/associated ruby classes is expensive, so I'm now wondering about the possibility of running jruby/chef inside of the JMeter JVM.
Results aren't promising:
./jruby my_sample_rest_client.rb
=>
NotImplementedError: fork is not available on this platform
fork at org/jruby/RubyKernel.java:1735
fork_subprocess at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/mixlib-shellout-2.0.1/lib/mixlib/shellout/unix.rb:284
run_command at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/mixlib-shellout-2.0.1/lib/mixlib/shellout/unix.rb:52
run_command at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/mixlib-shellout-2.0.1/lib/mixlib/shellout.rb:248
shell_out_command at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/chef-12.1.1/lib/chef/mixin/shell_out.rb:97
shell_out_with_systems_locale at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/chef-12.1.1/lib/chef/mixin/shell_out.rb:61
shell_out_with_systems_locale! at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/chef-12.1.1/lib/chef/mixin/shell_out.rb:65
guess_internal_locale at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/chef-12.1.1/lib/chef/config.rb:673
(root) at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/chef-12.1.1/lib/chef/config.rb:702
(root) at /opt/jruby-9.0.0.0.pre1/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
require at /opt/jruby-9.0.0.0.pre1/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54
require at org/jruby/RubyKernel.java:954
(root) at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/chef-12.1.1/lib/chef.rb:24
(root) at /opt/jruby-9.0.0.0.pre1/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
require at org/jruby/RubyKernel.java:954
require at /opt/jruby-9.0.0.0.pre1/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:128
<=
Unless you know how to suppress fork?

From: Daniel DeLeo [ddeleo@kallistec.com] on behalf of Daniel DeLeo [dan@kallistec.com]
Sent: 12 March 2015 22:03
To: chef@lists.opscode.com
Subject: [chef] Re: RE: RE: Re: chef server load testing

On Thursday, March 12, 2015 at 1:52 PM, Marcus Simonsen wrote:

Chef::ApiClient::Registration appears promising, and your name on the code!
Am I heading in the right direction?

Generally the “model” objects in Chef (node, role, etc.) follow an Active Record type pattern, where each object/class has methods for save/load/update/etc. So you can look though all those classes for how they use Chef::REST (doing git grep ‘def save’ might narrow it down a bit). In a few cases, the logic for running a request is a bit more complicated, and these things tend to have their own classes to encapsulate the behavior for just that one request. Chef::ApiClient::Registration is one of those, as it handles making a request with the validator key to create a new ApiClient object and store the private key on disk, which is what usually happens the very first time you run chef-client on a host. Likewise, the process of syncing cookbooks from the server to a host is a bit complicated, so that has it’s own CookbookSynchronizer class, and the process where chef-client queries the dependency solver to get the list of cookbooks to run is done by PolicyBuilder::ExpandNodeObject.

From: Marcus Simonsen [msimonsen@blackberry.com (mailto:msimonsen@blackberry.com)]
Sent: 12 March 2015 15:26
To: chef@lists.opscode.com (mailto:chef@lists.opscode.com)
Subject: [chef] RE: Re: chef server load testing

Thanks Daniel.

This is more in line with what I want to achieve. A very granular approach to simulating chef client operations against the server.
Cool. Just remember to validate the proportion of different API calls against your real-world usage. There’s a lot of ways that your data and the way you use the API can cause the Chef Server to have varying performance characteristics.

I've been taking a look into the chef code to understand a bit more what these Chef::Rest calls would look like. If I simulate payload then I won't even have to write any ruby at all, I'll just need to be able to template the http headers and body. My objective is to write as little code as possible.

The tricky part is that the HTTP headers include a timestamp, and that timestamp is part of the request signature. The server only allows a small deviation in the current time and the timestamp in the request as a mitigation against replay attacks. It’s possible that the amount of deviation allowed might be configurable, but there’s no documented config option for this. So you need to generate your request headers on-demand or at least a short time before your tests run.

On the other hand leveraging already working client code would be easier. If you have some samples of how you would simulate this via the Chef api that would be appreciated.
Cheffish gem seems to be very helpful in this respect - but documentation lacking I do have to dig into source.

If you are more familiar with some other language than ruby, then you might look around to see if there are any language bindings for whatever language you want to use. I know that at least python, go, php, and java have been written at some point, though I’m not sure if they’re well maintained.

--
Daniel DeLeo

8

The cost of ruby processes/classes is expensive > I know I'll run it
in JRuby; hehe, I love it - interesting approach.

I do not know that any work has been done to get Chef running on
JRuby. It is not likely to work, as mentioned, requiring the fork
syscall (this will spill over to affecting Ohai as well which relies
on it); you may be able to patch mixlib::shellout::unix to use Spoon
or a process_spawn type implementation which will be available to you.
Similar work would have to be undertaken for ohai. That's probably
only touching the surface.

https://rubygems.org/gems/spoon

I would not recommend this path to someone inexperienced with Ruby,
Chef and deeply familiar with implementation-specific challenges
regarding a JRuby application.

cheers,

--aj

On Sat, Mar 14, 2015 at 8:19 AM, Marcus Simonsen
msimonsen@blackberry.com wrote:

Many Thanks Daniel.
I've been able to do basic create nodes and clients, with the rest api. And I'm sure with more digging that I can load cookbooks etc...
It appears that the cost of starting the ruby process/associated ruby classes is expensive, so I'm now wondering about the possibility of running jruby/chef inside of the JMeter JVM.
Results aren't promising:
./jruby my_sample_rest_client.rb
=>
NotImplementedError: fork is not available on this platform
fork at org/jruby/RubyKernel.java:1735
fork_subprocess at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/mixlib-shellout-2.0.1/lib/mixlib/shellout/unix.rb:284
run_command at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/mixlib-shellout-2.0.1/lib/mixlib/shellout/unix.rb:52
run_command at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/mixlib-shellout-2.0.1/lib/mixlib/shellout.rb:248
shell_out_command at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/chef-12.1.1/lib/chef/mixin/shell_out.rb:97
shell_out_with_systems_locale at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/chef-12.1.1/lib/chef/mixin/shell_out.rb:61
shell_out_with_systems_locale! at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/chef-12.1.1/lib/chef/mixin/shell_out.rb:65
guess_internal_locale at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/chef-12.1.1/lib/chef/config.rb:673
(root) at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/chef-12.1.1/lib/chef/config.rb:702
(root) at /opt/jruby-9.0.0.0.pre1/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
require at /opt/jruby-9.0.0.0.pre1/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54
require at org/jruby/RubyKernel.java:954
(root) at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/chef-12.1.1/lib/chef.rb:24
(root) at /opt/jruby-9.0.0.0.pre1/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
require at org/jruby/RubyKernel.java:954
require at /opt/jruby-9.0.0.0.pre1/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:128
<=
Unless you know how to suppress fork?

From: Daniel DeLeo [ddeleo@kallistec.com] on behalf of Daniel DeLeo [dan@kallistec.com]
Sent: 12 March 2015 22:03
To: chef@lists.opscode.com
Subject: [chef] Re: RE: RE: Re: chef server load testing

On Thursday, March 12, 2015 at 1:52 PM, Marcus Simonsen wrote:

Chef::ApiClient::Registration appears promising, and your name on the code!
Am I heading in the right direction?

Generally the “model” objects in Chef (node, role, etc.) follow an Active Record type pattern, where each object/class has methods for save/load/update/etc. So you can look though all those classes for how they use Chef::REST (doing git grep ‘def save’ might narrow it down a bit). In a few cases, the logic for running a request is a bit more complicated, and these things tend to have their own classes to encapsulate the behavior for just that one request. Chef::ApiClient::Registration is one of those, as it handles making a request with the validator key to create a new ApiClient object and store the private key on disk, which is what usually happens the very first time you run chef-client on a host. Likewise, the process of syncing cookbooks from the server to a host is a bit complicated, so that has it’s own CookbookSynchronizer class, and the process where chef-client queries the dependency solver to get the list of cookbooks to run is done by PolicyBuilder::ExpandNodeObject.

From: Marcus Simonsen [msimonsen@blackberry.com (mailto:msimonsen@blackberry.com)]
Sent: 12 March 2015 15:26
To: chef@lists.opscode.com (mailto:chef@lists.opscode.com)
Subject: [chef] RE: Re: chef server load testing

Thanks Daniel.

This is more in line with what I want to achieve. A very granular approach to simulating chef client operations against the server.
Cool. Just remember to validate the proportion of different API calls against your real-world usage. There’s a lot of ways that your data and the way you use the API can cause the Chef Server to have varying performance characteristics.

I've been taking a look into the chef code to understand a bit more what these Chef::Rest calls would look like. If I simulate payload then I won't even have to write any ruby at all, I'll just need to be able to template the http headers and body. My objective is to write as little code as possible.

The tricky part is that the HTTP headers include a timestamp, and that timestamp is part of the request signature. The server only allows a small deviation in the current time and the timestamp in the request as a mitigation against replay attacks. It’s possible that the amount of deviation allowed might be configurable, but there’s no documented config option for this. So you need to generate your request headers on-demand or at least a short time before your tests run.

On the other hand leveraging already working client code would be easier. If you have some samples of how you would simulate this via the Chef api that would be appreciated.
Cheffish gem seems to be very helpful in this respect - but documentation lacking I do have to dig into source.

If you are more familiar with some other language than ruby, then you might look around to see if there are any language bindings for whatever language you want to use. I know that at least python, go, php, and java have been written at some point, though I’m not sure if they’re well maintained.

--
Daniel DeLeo

9

It was only an idea, simply to workaround the expense of loading chef/ruby.
I may be able to load and cache the ruby process from within JMeter, so that each sample (request) can re-use it somehow... Not sure how feasible this is.

The suggestion of using ec_metal is not available to me as we build out our own Chef servers in house (private cloud).

-marcus

From: AJ Christensen [aj@junglistheavy.industries]
Sent: 13 March 2015 15:26
To: chef@lists.opscode.com
Subject: [chef] Re: RE: Re: RE: RE: Re: chef server load testing

The cost of ruby processes/classes is expensive > I know I'll run it
in JRuby; hehe, I love it - interesting approach.

I do not know that any work has been done to get Chef running on
JRuby. It is not likely to work, as mentioned, requiring the fork
syscall (this will spill over to affecting Ohai as well which relies
on it); you may be able to patch mixlib::shellout::unix to use Spoon
or a process_spawn type implementation which will be available to you.
Similar work would have to be undertaken for ohai. That's probably
only touching the surface.

https://rubygems.org/gems/spoon

I would not recommend this path to someone inexperienced with Ruby,
Chef and deeply familiar with implementation-specific challenges
regarding a JRuby application.

cheers,

--aj

On Sat, Mar 14, 2015 at 8:19 AM, Marcus Simonsen
msimonsen@blackberry.com wrote:

Many Thanks Daniel.
I've been able to do basic create nodes and clients, with the rest api. And I'm sure with more digging that I can load cookbooks etc...
It appears that the cost of starting the ruby process/associated ruby classes is expensive, so I'm now wondering about the possibility of running jruby/chef inside of the JMeter JVM.
Results aren't promising:
./jruby my_sample_rest_client.rb
=>
NotImplementedError: fork is not available on this platform
fork at org/jruby/RubyKernel.java:1735
fork_subprocess at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/mixlib-shellout-2.0.1/lib/mixlib/shellout/unix.rb:284
run_command at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/mixlib-shellout-2.0.1/lib/mixlib/shellout/unix.rb:52
run_command at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/mixlib-shellout-2.0.1/lib/mixlib/shellout.rb:248
shell_out_command at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/chef-12.1.1/lib/chef/mixin/shell_out.rb:97
shell_out_with_systems_locale at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/chef-12.1.1/lib/chef/mixin/shell_out.rb:61
shell_out_with_systems_locale! at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/chef-12.1.1/lib/chef/mixin/shell_out.rb:65
guess_internal_locale at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/chef-12.1.1/lib/chef/config.rb:673
(root) at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/chef-12.1.1/lib/chef/config.rb:702
(root) at /opt/jruby-9.0.0.0.pre1/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
require at /opt/jruby-9.0.0.0.pre1/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54
require at org/jruby/RubyKernel.java:954
(root) at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/chef-12.1.1/lib/chef.rb:24
(root) at /opt/jruby-9.0.0.0.pre1/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
require at org/jruby/RubyKernel.java:954
require at /opt/jruby-9.0.0.0.pre1/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:128
<=
Unless you know how to suppress fork?

From: Daniel DeLeo [ddeleo@kallistec.com] on behalf of Daniel DeLeo [dan@kallistec.com]
Sent: 12 March 2015 22:03
To: chef@lists.opscode.com
Subject: [chef] Re: RE: RE: Re: chef server load testing

On Thursday, March 12, 2015 at 1:52 PM, Marcus Simonsen wrote:

Chef::ApiClient::Registration appears promising, and your name on the code!
Am I heading in the right direction?

Generally the “model” objects in Chef (node, role, etc.) follow an Active Record type pattern, where each object/class has methods for save/load/update/etc. So you can look though all those classes for how they use Chef::REST (doing git grep ‘def save’ might narrow it down a bit). In a few cases, the logic for running a request is a bit more complicated, and these things tend to have their own classes to encapsulate the behavior for just that one request. Chef::ApiClient::Registration is one of those, as it handles making a request with the validator key to create a new ApiClient object and store the private key on disk, which is what usually happens the very first time you run chef-client on a host. Likewise, the process of syncing cookbooks from the server to a host is a bit complicated, so that has it’s own CookbookSynchronizer class, and the process where chef-client queries the dependency solver to get the list of cookbooks to run is done by PolicyBuilder::ExpandNodeObject.

From: Marcus Simonsen [msimonsen@blackberry.com (mailto:msimonsen@blackberry.com)]
Sent: 12 March 2015 15:26
To: chef@lists.opscode.com (mailto:chef@lists.opscode.com)
Subject: [chef] RE: Re: chef server load testing

Thanks Daniel.

This is more in line with what I want to achieve. A very granular approach to simulating chef client operations against the server.
Cool. Just remember to validate the proportion of different API calls against your real-world usage. There’s a lot of ways that your data and the way you use the API can cause the Chef Server to have varying performance characteristics.

I've been taking a look into the chef code to understand a bit more what these Chef::Rest calls would look like. If I simulate payload then I won't even have to write any ruby at all, I'll just need to be able to template the http headers and body. My objective is to write as little code as possible.

The tricky part is that the HTTP headers include a timestamp, and that timestamp is part of the request signature. The server only allows a small deviation in the current time and the timestamp in the request as a mitigation against replay attacks. It’s possible that the amount of deviation allowed might be configurable, but there’s no documented config option for this. So you need to generate your request headers on-demand or at least a short time before your tests run.

On the other hand leveraging already working client code would be easier. If you have some samples of how you would simulate this via the Chef api that would be appreciated.
Cheffish gem seems to be very helpful in this respect - but documentation lacking I do have to dig into source.

If you are more familiar with some other language than ruby, then you might look around to see if there are any language bindings for whatever language you want to use. I know that at least python, go, php, and java have been written at some point, though I’m not sure if they’re well maintained.

--
Daniel DeLeo

10

On Friday, March 13, 2015 at 12:40 PM, Marcus Simonsen wrote:

It was only an idea, simply to workaround the expense of loading chef/ruby.
I may be able to load and cache the ruby process from within JMeter, so that each sample (request) can re-use it somehow... Not sure how feasible this is.

The suggestion of using ec_metal is not available to me as we build out our own Chef servers in house (private cloud).

-marcus

From: AJ Christensen [aj@junglistheavy.industries (mailto:aj@junglistheavy.industries)]
Sent: 13 March 2015 15:26
To: chef@lists.opscode.com (mailto:chef@lists.opscode.com)
Subject: [chef] Re: RE: Re: RE: RE: Re: chef server load testing

The cost of ruby processes/classes is expensive > I know I'll run it
in JRuby; hehe, I love it - interesting approach.

I do not know that any work has been done to get Chef running on
JRuby. It is not likely to work, as mentioned, requiring the fork
syscall (this will spill over to affecting Ohai as well which relies
on it); you may be able to patch mixlib::shellout::unix to use Spoon
or a process_spawn type implementation which will be available to you.
Similar work would have to be undertaken for ohai. That's probably
only touching the surface.

I’ve done this for a subset of the Chef codebase needed to use Chef::REST in the past. Though since I did that last we added some stuff like the locale check in Chef::Config that Marcus ran into.

spoon | RubyGems.org | your community gem host
GitHub - rtomayko/posix-spawn: Ruby process spawning library

I would not recommend this path to someone inexperienced with Ruby,
Chef and deeply familiar with implementation-specific challenges
regarding a JRuby application.

AJ is correct here, you’ll have to do some extra work to avoid any calls to Mixlib::ShellOut if you want to use JRuby. Hopefully you only need to make a few workarounds, but it’ll unfortunately require a little whack-a-mole.

cheers,

--aj

On Sat, Mar 14, 2015 at 8:19 AM, Marcus Simonsen
<msimonsen@blackberry.com (mailto:msimonsen@blackberry.com)> wrote:

Many Thanks Daniel.
I've been able to do basic create nodes and clients, with the rest api. And I'm sure with more digging that I can load cookbooks etc...
It appears that the cost of starting the ruby process/associated ruby classes is expensive, so I'm now wondering about the possibility of running jruby/chef inside of the JMeter JVM.
Results aren't promising:
./jruby my_sample_rest_client.rb
=>
NotImplementedError: fork is not available on this platform
fork at org/jruby/RubyKernel.java:1735
fork_subprocess at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/mixlib-shellout-2.0.1/lib/mixlib/shellout/unix.rb:284
run_command at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/mixlib-shellout-2.0.1/lib/mixlib/shellout/unix.rb:52
run_command at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/mixlib-shellout-2.0.1/lib/mixlib/shellout.rb:248
shell_out_command at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/chef-12.1.1/lib/chef/mixin/shell_out.rb:97
shell_out_with_systems_locale at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/chef-12.1.1/lib/chef/mixin/shell_out.rb:61
shell_out_with_systems_locale! at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/chef-12.1.1/lib/chef/mixin/shell_out.rb:65
guess_internal_locale at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/chef-12.1.1/lib/chef/config.rb:673
(root) at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/chef-12.1.1/lib/chef/config.rb:702
(root) at /opt/jruby-9.0.0.0.pre1/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
require at /opt/jruby-9.0.0.0.pre1/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:54
require at org/jruby/RubyKernel.java:954
(root) at /opt/jruby-9.0.0.0.pre1/lib/ruby/gems/shared/gems/chef-12.1.1/lib/chef.rb:24
(root) at /opt/jruby-9.0.0.0.pre1/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:1
require at org/jruby/RubyKernel.java:954
require at /opt/jruby-9.0.0.0.pre1/lib/ruby/stdlib/rubygems/core_ext/kernel_require.rb:128
<=
Unless you know how to suppress fork?

Like I hinted at before, you need to look at the code calling the shellout command and figure out how to stop it from doing so. In the case of this config option, you might be able to set the configuration explicitly in your config file to avoid this auto detection code.

--
Daniel DeLeo

11

Markus,

Would it be worth considering a Ruby load-testing framework? Alas, the only
one I know if is: https://github.com/brewster/front_end_loader

-Peter

–

Peter Burkholder — Customer Success Engineer

Unavailability: Travel March 2-3; Vacation March 16-20; ChefConf March
30-April 3

301-204-5767 – pburkholder@chef.io – *my: *Linkedin
http://www.linkedin.com/in/pburkholder Twitter
http://www.twitter.com/pburkholder Cal
https://www.google.com/calendar/embed?src=pburkholder%40chef.io&mode=WEEK
endar

CHEF

CHEF.IO http://www.chef.io/

TM

chef.io http://www.chef.io/ Blog http://www.chef.io/blog/ Facebook
https://www.facebook.com/getchefdotcom Twitter
https://twitter.com/chef Youtube https://www.youtube.com/getchef

12

Ya been thinking about it, and I will look into it.
It’s even possible that I’ll create the necessary concurrent and reporting features that I need in ruby.
But it’s likely I need a coordinated cluster of client bots to hammer the server., which JMeter provides.
Combined with my comfort with JMeter and java I simply don’t want to re-invent, rather than to bridge technologies.

Sent from my BlackBerry 10 smartphone on the Bell network.
From: Peter Burkholder
Sent: Friday, March 13, 2015 6:04 PM
To: chef@lists.opscode.com
Reply To: chef@lists.opscode.com
Subject: [chef] Re: Re: RE: Re: RE: Re: RE: RE: Re: chef server load testing

Markus,

Would it be worth considering a Ruby load-testing framework? Alas, the only one I know if is: https://github.com/brewster/front_end_loader

-Peter

–

Peter Burkholder — Customer Success Engineer

Unavailability: Travel March 2-3; Vacation March 16-20; ChefConf March 30-April 3

301-204-5767 – pburkholder@chef.iomailto:pburkholder@chef.io – my: Linkedinhttp://www.linkedin.com/in/pburkholder Twitterhttp://www.twitter.com/pburkholder Calhttps://www.google.com/calendar/embed?src=pburkholder%40chef.io&mode=WEEKendar

CHEF

CHEF.IOhttp://www.chef.io/

TM

chef.iohttp://www.chef.io/ Bloghttp://www.chef.io/blog/ Facebookhttps://www.facebook.com/getchefdotcom Twitterhttps://twitter.com/chef Youtubehttps://www.youtube.com/getchef

13

Here are my musings on the subject if anyone is interested:

There are 2 general approaches I took.

  1. Write a JMeter plugin calling instances chef-client
  2. Wrap the Chef server api in some concurrency operations

Feel free to mock me :slight_smile: But I wouldn’t mind constructive feedback either.

From: Marcus Simonsen [msimonsen@blackberry.com]
Sent: 13 March 2015 18:53
To: Peter Burkholder; chef@lists.opscode.com
Subject: [chef] Re: Re: Re: RE: Re: RE: Re: RE: RE: Re: chef server load testing

Ya been thinking about it, and I will look into it.
It’s even possible that I’ll create the necessary concurrent and reporting features that I need in ruby.
But it’s likely I need a coordinated cluster of client bots to hammer the server., which JMeter provides.
Combined with my comfort with JMeter and java I simply don’t want to re-invent, rather than to bridge technologies.

Sent from my BlackBerry 10 smartphone on the Bell network.
From: Peter Burkholder
Sent: Friday, March 13, 2015 6:04 PM
To: chef@lists.opscode.com
Reply To: chef@lists.opscode.com
Subject: [chef] Re: Re: RE: Re: RE: Re: RE: RE: Re: chef server load testing

Markus,

Would it be worth considering a Ruby load-testing framework? Alas, the only one I know if is: https://github.com/brewster/front_end_loader

-Peter

–

Peter Burkholder — Customer Success Engineer

Unavailability: Travel March 2-3; Vacation March 16-20; ChefConf March 30-April 3

301-204-5767 – pburkholder@chef.iomailto:pburkholder@chef.io – my: Linkedinhttp://www.linkedin.com/in/pburkholder Twitterhttp://www.twitter.com/pburkholder Calhttps://www.google.com/calendar/embed?src=pburkholder%40chef.io&mode=WEEKendar

CHEF

CHEF.IOhttp://www.chef.io/

TM

chef.iohttp://www.chef.io/ Bloghttp://www.chef.io/blog/ Facebookhttps://www.facebook.com/getchefdotcom Twitterhttps://twitter.com/chef Youtubehttps://www.youtube.com/getchef