Possible problem with parallel testing with test-kitchen


#1

Hi,

We’ve been having fun using the lxc driver for testing with test-kitchen (which makes testing really fast!). Unfortunately I forgot that GitHub has rate limiting imposed now. (60 requests were used up in less than 15 minutes (in part because LXC is so fast at creating and destroying guests and booting new guests ))

slikens@app379:~/git/haproxy_lwrp# curl https://api.github.com/rate_limit -u "damm"
Enter host password for user ‘damm’:
{
“rate”: {
“limit”: 5000,
“remaining”: 5000
}
}
slikens@app379:~/git/haproxy_lwrp# curl https://api.github.com/rate_limit
{
“rate”: {
“limit”: 60,
“remaining”: 60
}
}

I believe a few api requests occur in https://github.com/opscode/test-kitchen/blob/1.0/lib/kitchen/busser.rb#L100

I don’t think the right answer is ‘can we ship Kitchen Busser with Omnibus a/k/a Chef-full’.

Could we move https://raw.github.com/opscode/kb/go to opscode.com like we do with the full installer? curl -L https://www.opscode.com/chef/kb.sh? with that we could reduce the github api requests.

the basic error seen when getting this can be found https://gist.github.com/damm/5130887

P.S. I can create an issue for this, I just wanted to start a conversation about it and get a general idea of where we wanted to steer this before hand.

Thanks!


#2

Hi,

Following up it appears each test consumes roughly 7 requests, which leaves you roughly 8 tests per hour before GitHub will start to rate limit you. Are there any other people dealing with this or am I the only person that is trying to test this much?

Thanks!

Scott M. Likens

Before Test

slikens@app379:~/git/haproxy_lwrp# curl https://api.github.com/rate_limit
{
“rate”: {
“limit”: 60,
“remaining”: 60
}
}

After test

slikens@app379:~/git/haproxy_lwrp# curl https://api.github.com/rate_limit
{
“rate”: {
“limit”: 60,
“remaining”: 53
}
}

On Mar 10, 2013, at 3:58 PM, Scott M. Likens scott@likens.us wrote:

Hi,

We’ve been having fun using the lxc driver for testing with test-kitchen (which makes testing really fast!). Unfortunately I forgot that GitHub has rate limiting imposed now. (60 requests were used up in less than 15 minutes (in part because LXC is so fast at creating and destroying guests and booting new guests ))

slikens@app379:~/git/haproxy_lwrp# curl https://api.github.com/rate_limit -u "damm"
Enter host password for user ‘damm’:
{
“rate”: {
“limit”: 5000,
“remaining”: 5000
}
}
slikens@app379:~/git/haproxy_lwrp# curl https://api.github.com/rate_limit
{
“rate”: {
“limit”: 60,
“remaining”: 60
}
}

I believe a few api requests occur in https://github.com/opscode/test-kitchen/blob/1.0/lib/kitchen/busser.rb#L100

I don’t think the right answer is ‘can we ship Kitchen Busser with Omnibus a/k/a Chef-full’.

Could we move https://raw.github.com/opscode/kb/go to opscode.com like we do with the full installer? curl -L https://www.opscode.com/chef/kb.sh? with that we could reduce the github api requests.

the basic error seen when getting this can be found https://gist.github.com/damm/5130887

P.S. I can create an issue for this, I just wanted to start a conversation about it and get a general idea of where we wanted to steer this before hand.

Thanks!

!DSPAM:513d101f120561348188260!


#3

Do conditional GETs or HEAD requests count against the limit?


Daniel DeLeo

On Sunday, March 10, 2013 at 3:58 PM, Scott M. Likens wrote:

Hi,

We’ve been having fun using the lxc driver for testing with test-kitchen (which makes testing really fast!). Unfortunately I forgot that GitHub has rate limiting imposed now. (60 requests were used up in less than 15 minutes (in part because LXC is so fast at creating and destroying guests and booting new guests ))

slikens@app379:~/git/haproxy_lwrp# curl https://api.github.com/rate_limit -u "damm"
Enter host password for user ‘damm’:
{
“rate”: {
“limit”: 5000,
“remaining”: 5000
}
}
slikens@app379:~/git/haproxy_lwrp# curl https://api.github.com/rate_limit
{
“rate”: {
“limit”: 60,
“remaining”: 60
}
}

I believe a few api requests occur in https://github.com/opscode/test-kitchen/blob/1.0/lib/kitchen/busser.rb#L100

I don’t think the right answer is ‘can we ship Kitchen Busser with Omnibus a/k/a Chef-full’.

Could we move https://raw.github.com/opscode/kb/go to opscode.com (http://opscode.com) like we do with the full installer? curl -L https://www.opscode.com/chef/kb.sh? with that we could reduce the github api requests.

the basic error seen when getting this can be found https://gist.github.com/damm/5130887

P.S. I can create an issue for this, I just wanted to start a conversation about it and get a general idea of where we wanted to steer this before hand.

Thanks!


#4

Dan,

Unfortunately yes. I was able to reduce it to 1 GET (installation of bats, minitest is installed via rubygems so it does not hit github) by modifying neutering the installation script at https://github.com/opscode/kb/blob/go/install to https://gist.github.com/damm/c7e89264f3130b2c62fc and modify lib/kitchen/busser.rb to point to a different INSTALL_URL. With that small change this reduces the problem greatly.

I imagine there are many ways to reduce this pain I’m just not sure of the quickest and sanest path.

If there’s anyway I can help let me know

Thanks!

On Mar 11, 2013, at 9:50 PM, Daniel DeLeo dan@kallistec.com wrote:

Do conditional GETs or HEAD requests count against the limit?


Daniel DeLeo

On Sunday, March 10, 2013 at 3:58 PM, Scott M. Likens wrote:

Hi,

We’ve been having fun using the lxc driver for testing with test-kitchen (which makes testing really fast!). Unfortunately I forgot that GitHub has rate limiting imposed now. (60 requests were used up in less than 15 minutes (in part because LXC is so fast at creating and destroying guests and booting new guests ))

slikens@app379:~/git/haproxy_lwrp# curl https://api.github.com/rate_limit -u "damm"
Enter host password for user ‘damm’:
{
“rate”: {
“limit”: 5000,
“remaining”: 5000
}
}
slikens@app379:~/git/haproxy_lwrp# curl https://api.github.com/rate_limit
{
“rate”: {
“limit”: 60,
“remaining”: 60
}
}

I believe a few api requests occur in https://github.com/opscode/test-kitchen/blob/1.0/lib/kitchen/busser.rb#L100

I don’t think the right answer is ‘can we ship Kitchen Busser with Omnibus a/k/a Chef-full’.

Could we move https://raw.github.com/opscode/kb/go to opscode.com like we do with the full installer? curl -L https://www.opscode.com/chef/kb.sh? with that we could reduce the github api requests.
the basic error seen when getting this can be found https://gist.github.com/damm/5130887

P.S. I can create an issue for this, I just wanted to start a conversation about it and get a general idea of where we wanted to steer this before hand.

Thanks!

!DSPAM:513eb44a175911804284693!


#5

I mentioned this to Fletcher on Twitter, he is aware of the situation!
Tickets would be good.

Cheers,

AJ

On 13 March 2013 06:48, Scott M. Likens scott@likens.us wrote:

Dan,

Unfortunately yes. I was able to reduce it to 1 GET (installation of bats,
minitest is installed via rubygems so it does not hit github) by modifying
neutering the installation script at
https://github.com/opscode/kb/blob/go/install to
https://gist.github.com/damm/c7e89264f3130b2c62fc and modify
lib/kitchen/busser.rb to point to a different INSTALL_URL. With that small
change this reduces the problem greatly.

I imagine there are many ways to reduce this pain I’m just not sure of the
quickest and sanest path.

If there’s anyway I can help let me know

Thanks!

On Mar 11, 2013, at 9:50 PM, Daniel DeLeo dan@kallistec.com wrote:

Do conditional GETs or HEAD requests count against the limit?


Daniel DeLeo

On Sunday, March 10, 2013 at 3:58 PM, Scott M. Likens wrote:

Hi,

We’ve been having fun using the lxc driver for testing with test-kitchen
(which makes testing really fast!). Unfortunately I forgot that GitHub has
rate limiting imposed now. (60 requests were used up in less than 15
minutes (in part because LXC is so fast at creating and destroying guests
and booting new guests ))

slikens@app379:~/git/haproxy_lwrp# curl https://api.github.com/rate_limit -u
"damm"
Enter host password for user ‘damm’:
{
“rate”: {
“limit”: 5000,
“remaining”: 5000
}
}
slikens@app379:~/git/haproxy_lwrp# curl https://api.github.com/rate_limit
{
“rate”: {
“limit”: 60,
“remaining”: 60
}
}

I believe a few api requests occur in
https://github.com/opscode/test-kitchen/blob/1.0/lib/kitchen/busser.rb#L100

I don’t think the right answer is ‘can we ship Kitchen Busser with Omnibus
a/k/a Chef-full’.

Could we move https://raw.github.com/opscode/kb/go to opscode.com like we do
with the full installer? curl -L https://www.opscode.com/chef/kb.sh? with
that we could reduce the github api requests.
the basic error seen when getting this can be found
https://gist.github.com/damm/5130887

P.S. I can create an issue for this, I just wanted to start a conversation
about it and get a general idea of where we wanted to steer this before
hand.

Thanks!

!DSPAM:513eb44a175911804284693!


#6

Pppppppppmmmpplkllkp
On Mar 10, 2013 6:58 PM, “Scott M. Likens” scott@likens.us wrote:

Hi,

We’ve been having fun using the lxc driver for testing with test-kitchen
(which makes testing really fast!). Unfortunately I forgot that GitHub has
rate limiting imposed now. (60 requests were used up in less than 15
minutes (in part because LXC is so fast at creating and destroying guests
and booting new guests ))

slikens@app379:~/git/haproxy_lwrp# curl https://api.github.com/rate_limit-u "damm"
Enter host password for user ‘damm’:
{
“rate”: {
“limit”: 5000,
“remaining”: 5000
}
}
slikens@app379:~/git/haproxy_lwrp# curl https://api.github.com/rate_limit
{
“rate”: {
“limit”: 60,
“remaining”: 60
}
}

I believe a few api requests occur in
https://github.com/opscode/test-kitchen/blob/1.0/lib/kitchen/busser.rb#L100

I don’t think the right answer is 'can we ship Kitchen Busser with
Omnibus a/k/a Chef-full’
.

Could we move https://raw.github.com/opscode/kb/go to opscode.com like we
do with the full installer? curl -L https://www.opscode.com/chef/kb.sh?with that we could reduce the github api requests.

the basic error seen when getting this can be found
https://gist.github.com/damm/5130887

P.S. I can create an issue for this, I just wanted to start a conversation
about it and get a general idea of where we wanted to steer this before
hand.

Thanks!


#7

Hi,

No problem I created it https://github.com/opscode/test-kitchen/issues/68

Thanks!

On Mar 12, 2013, at 12:27 PM, AJ Christensen aj@junglist.gen.nz wrote:

I mentioned this to Fletcher on Twitter, he is aware of the situation!
Tickets would be good.

Cheers,

AJ

On 13 March 2013 06:48, Scott M. Likens scott@likens.us wrote:

Dan,

Unfortunately yes. I was able to reduce it to 1 GET (installation of bats,
minitest is installed via rubygems so it does not hit github) by modifying
neutering the installation script at
https://github.com/opscode/kb/blob/go/install to
https://gist.github.com/damm/c7e89264f3130b2c62fc and modify
lib/kitchen/busser.rb to point to a different INSTALL_URL. With that small
change this reduces the problem greatly.

I imagine there are many ways to reduce this pain I’m just not sure of the
quickest and sanest path.

If there’s anyway I can help let me know

Thanks!

On Mar 11, 2013, at 9:50 PM, Daniel DeLeo dan@kallistec.com wrote:

Do conditional GETs or HEAD requests count against the limit?


Daniel DeLeo

On Sunday, March 10, 2013 at 3:58 PM, Scott M. Likens wrote:

Hi,

We’ve been having fun using the lxc driver for testing with test-kitchen
(which makes testing really fast!). Unfortunately I forgot that GitHub has
rate limiting imposed now. (60 requests were used up in less than 15
minutes (in part because LXC is so fast at creating and destroying guests
and booting new guests ))

slikens@app379:~/git/haproxy_lwrp# curl https://api.github.com/rate_limit -u
"damm"
Enter host password for user ‘damm’:
{
“rate”: {
“limit”: 5000,
“remaining”: 5000
}
}
slikens@app379:~/git/haproxy_lwrp# curl https://api.github.com/rate_limit
{
“rate”: {
“limit”: 60,
“remaining”: 60
}
}

I believe a few api requests occur in
https://github.com/opscode/test-kitchen/blob/1.0/lib/kitchen/busser.rb#L100

I don’t think the right answer is ‘can we ship Kitchen Busser with Omnibus
a/k/a Chef-full’.

Could we move https://raw.github.com/opscode/kb/go to opscode.com like we do
with the full installer? curl -L https://www.opscode.com/chef/kb.sh? with
that we could reduce the github api requests.
the basic error seen when getting this can be found
https://gist.github.com/damm/5130887

P.S. I can create an issue for this, I just wanted to start a conversation
about it and get a general idea of where we wanted to steer this before
hand.

Thanks!

!DSPAM:513f81af199541804284693!


#8

On Tuesday, 12 March, 2013 at 6:23 PM, Scott M. Likens wrote:

Hi,

No problem I created it https://github.com/opscode/test-kitchen/issues/68

Thanks!

On Mar 12, 2013, at 12:27 PM, AJ Christensen <aj@junglist.gen.nz (mailto:aj@junglist.gen.nz)> wrote:

I mentioned this to Fletcher on Twitter, he is aware of the situation!
Tickets would be good.

Cheers,

AJ

On 13 March 2013 06:48, Scott M. Likens <scott@likens.us (mailto:scott@likens.us)> wrote:

Dan,

Unfortunately yes. I was able to reduce it to 1 GET (installation of bats,
minitest is installed via rubygems so it does not hit github) by modifying
neutering the installation script at
https://github.com/opscode/kb/blob/go/install to
https://gist.github.com/damm/c7e89264f3130b2c62fc and modify
lib/kitchen/busser.rb to point to a different INSTALL_URL. With that small
change this reduces the problem greatly.

I imagine there are many ways to reduce this pain I’m just not sure of the
quickest and sanest path.

If there’s anyway I can help let me know

Thanks!

On Mar 11, 2013, at 9:50 PM, Daniel DeLeo <dan@kallistec.com (mailto:dan@kallistec.com)> wrote:

Do conditional GETs or HEAD requests count against the limit?


Daniel DeLeo

On Sunday, March 10, 2013 at 3:58 PM, Scott M. Likens wrote:

Hi,

We’ve been having fun using the lxc driver for testing with test-kitchen
(which makes testing really fast!). Unfortunately I forgot that GitHub has
rate limiting imposed now. (60 requests were used up in less than 15
minutes (in part because LXC is so fast at creating and destroying guests
and booting new guests ))

slikens@app379:~/git/haproxy_lwrp# curl https://api.github.com/rate_limit -u
"damm"
Enter host password for user ‘damm’:
{
“rate”: {
“limit”: 5000,
“remaining”: 5000
}
}
slikens@app379:~/git/haproxy_lwrp# curl https://api.github.com/rate_limit
{
“rate”: {
“limit”: 60,
“remaining”: 60
}
}

I believe a few api requests occur in
https://github.com/opscode/test-kitchen/blob/1.0/lib/kitchen/busser.rb#L100

I don’t think the right answer is ‘can we ship Kitchen Busser with Omnibus
a/k/a Chef-full’.

Could we move https://raw.github.com/opscode/kb/go to opscode.com (http://opscode.com) like we do
with the full installer? curl -L https://www.opscode.com/chef/kb.sh? with
that we could reduce the github api requests.
the basic error seen when getting this can be found
https://gist.github.com/damm/5130887

P.S. I can create an issue for this, I just wanted to start a conversation
about it and get a general idea of where we wanted to steer this before
hand.

Thanks!

!DSPAM:513f81af199541804284693!

Personally I think it’s awesome someone has bumped up against GitHub API throttling because their testing is going that fast. I was taking a look at this last night and there are 2 places where the git tag API calls are going to hurt us:

If we had a way to avoid these calls or ask some other endpoint (static or dynamic), we’d only be asking the GitHub file servers to download the correct tarballs. I’d like to avoid adding too many moving parts however it feels like at least one more something is called for here (barring some re-implementation of kb).

Currently I’m toying with a very small web app that can answer a simple question: what is the latest version-tagging git tag on a github repo? This could make use of conditional GETs (which don’t fall under throttling) and the responses could be cached using ETAG headers. The only real change to the kb code would be to call this API to get an answer and proceed to download off GitHub as usual. Essentially this app would mitigate the API throttling for everyone in one place.

So why not do conditional GETs and ETAG caching on our workstations and remote nodes? The problem is that kb lives on those freshly created nodes which have no past history to draw from. Their first call to the GitHub API will always be a cache miss. Worse is that in certain environments (like LXC containers, NAT’ed subnets with VM instances) the source IP to GitHub could all be the same anyway.

Anyway that is what I’d like to spend a bit more time on tonight to flesh it out. Longer term there could be a more clever solution, but for now let’s get everyone back to fast, throttle-free tests :slight_smile:

Thoughts?

Fletcher


#9

On Mar 13, 2013, at 2:24 PM, Fletcher Nichol fnichol@nichol.ca wrote:

On Tuesday, 12 March, 2013 at 6:23 PM, Scott M. Likens wrote:

Hi,

No problem I created it https://github.com/opscode/test-kitchen/issues/68

Thanks!

On Mar 12, 2013, at 12:27 PM, AJ Christensen <aj@junglist.gen.nz (mailto:aj@junglist.gen.nz)> wrote:

I mentioned this to Fletcher on Twitter, he is aware of the situation!
Tickets would be good.

Cheers,

AJ

On 13 March 2013 06:48, Scott M. Likens <scott@likens.us (mailto:scott@likens.us)> wrote:

Dan,

Unfortunately yes. I was able to reduce it to 1 GET (installation of bats,
minitest is installed via rubygems so it does not hit github) by modifying
neutering the installation script at
https://github.com/opscode/kb/blob/go/install to
https://gist.github.com/damm/c7e89264f3130b2c62fc and modify
lib/kitchen/busser.rb to point to a different INSTALL_URL. With that small
change this reduces the problem greatly.

I imagine there are many ways to reduce this pain I’m just not sure of the
quickest and sanest path.

If there’s anyway I can help let me know

Thanks!

On Mar 11, 2013, at 9:50 PM, Daniel DeLeo <dan@kallistec.com (mailto:dan@kallistec.com)> wrote:

Do conditional GETs or HEAD requests count against the limit?


Daniel DeLeo

On Sunday, March 10, 2013 at 3:58 PM, Scott M. Likens wrote:

Hi,

We’ve been having fun using the lxc driver for testing with test-kitchen
(which makes testing really fast!). Unfortunately I forgot that GitHub has
rate limiting imposed now. (60 requests were used up in less than 15
minutes (in part because LXC is so fast at creating and destroying guests
and booting new guests ))

slikens@app379:~/git/haproxy_lwrp# curl https://api.github.com/rate_limit -u
"damm"
Enter host password for user ‘damm’:
{
“rate”: {
“limit”: 5000,
“remaining”: 5000
}
}
slikens@app379:~/git/haproxy_lwrp# curl https://api.github.com/rate_limit
{
“rate”: {
“limit”: 60,
“remaining”: 60
}
}

I believe a few api requests occur in
https://github.com/opscode/test-kitchen/blob/1.0/lib/kitchen/busser.rb#L100

I don’t think the right answer is ‘can we ship Kitchen Busser with Omnibus
a/k/a Chef-full’.

Could we move https://raw.github.com/opscode/kb/go to opscode.com (http://opscode.com) like we do
with the full installer? curl -L https://www.opscode.com/chef/kb.sh? with
that we could reduce the github api requests.
the basic error seen when getting this can be found
https://gist.github.com/damm/5130887

P.S. I can create an issue for this, I just wanted to start a conversation
about it and get a general idea of where we wanted to steer this before
hand.

Thanks!

Personally I think it’s awesome someone has bumped up against GitHub API throttling because their testing is going that fast. I was taking a look at this last night and there are 2 places where the git tag API calls are going to hurt us:

If we had a way to avoid these calls or ask some other endpoint (static or dynamic), we’d only be asking the GitHub file servers to download the correct tarballs. I’d like to avoid adding too many moving parts however it feels like at least one more something is called for here (barring some re-implementation of kb).

In my first implementation at reducing the GitHub API Requests I found reducing just the installation script was good enough to allow me to test without interruptions. However to scale this further beyond roughly 60 tests per hour (if you use bats, minitest does not incur an api request if i recall, which hints at some flexibility)

Currently I’m toying with a very small web app that can answer a simple question: what is the latest version-tagging git tag on a github repo? This could make use of conditional GETs (which don’t fall under throttling) and the responses could be cached using ETAG headers. The only real change to the kb code would be to call this API to get an answer and proceed to download off GitHub as usual. Essentially this app would mitigate the API throttling for everyone in one place.

I would say if we are trying to cache the sources, basically then let’s host all the source files for testing (including chef-full, kb, etc) as there might be someone who would want to do testing but cannot because their Firewall blocks Github (yes this number is still shockingly high).

So why not do conditional GETs and ETAG caching on our workstations and remote nodes? The problem is that kb lives on those freshly created nodes which have no past history to draw from. Their first call to the GitHub API will always be a cache miss. Worse is that in certain environments (like LXC containers, NAT’ed subnets with VM instances) the source IP to GitHub could all be the same anyway.

Anyway that is what I’d like to spend a bit more time on tonight to flesh it out. Longer term there could be a more clever solution, but for now let’s get everyone back to fast, throttle-free tests :slight_smile:

Thoughts?

Fletcher

My initial thought of implementing this was to stop hardcoding the kb installer and make it available in either .kitchen.local.yml or .kitchen.yml. Additionally from the evidence above, It’s possible to ship kb and it’s counterparts up to rubygems to help (this may be enough for people who no internet access to be able to test)

I hope that helps answer your question.

Thanks!

Scott M. Likens


#10

You can probably just chuck kb into your base containers too, no?

–AJ

On 17 March 2013 07:05, Scott M. Likens scott@likens.us wrote:

On Mar 13, 2013, at 2:24 PM, Fletcher Nichol fnichol@nichol.ca wrote:

On Tuesday, 12 March, 2013 at 6:23 PM, Scott M. Likens wrote:

Hi,

No problem I created it https://github.com/opscode/test-kitchen/issues/68

Thanks!

On Mar 12, 2013, at 12:27 PM, AJ Christensen <aj@junglist.gen.nz (mailto:aj@junglist.gen.nz)> wrote:

I mentioned this to Fletcher on Twitter, he is aware of the situation!
Tickets would be good.

Cheers,

AJ

On 13 March 2013 06:48, Scott M. Likens <scott@likens.us (mailto:scott@likens.us)> wrote:

Dan,

Unfortunately yes. I was able to reduce it to 1 GET (installation of bats,
minitest is installed via rubygems so it does not hit github) by modifying
neutering the installation script at
https://github.com/opscode/kb/blob/go/install to
https://gist.github.com/damm/c7e89264f3130b2c62fc and modify
lib/kitchen/busser.rb to point to a different INSTALL_URL. With that small
change this reduces the problem greatly.

I imagine there are many ways to reduce this pain I’m just not sure of the
quickest and sanest path.

If there’s anyway I can help let me know

Thanks!

On Mar 11, 2013, at 9:50 PM, Daniel DeLeo <dan@kallistec.com (mailto:dan@kallistec.com)> wrote:

Do conditional GETs or HEAD requests count against the limit?


Daniel DeLeo

On Sunday, March 10, 2013 at 3:58 PM, Scott M. Likens wrote:

Hi,

We’ve been having fun using the lxc driver for testing with test-kitchen
(which makes testing really fast!). Unfortunately I forgot that GitHub has
rate limiting imposed now. (60 requests were used up in less than 15
minutes (in part because LXC is so fast at creating and destroying guests
and booting new guests ))

slikens@app379:~/git/haproxy_lwrp# curl https://api.github.com/rate_limit -u
"damm"
Enter host password for user ‘damm’:
{
“rate”: {
“limit”: 5000,
“remaining”: 5000
}
}
slikens@app379:~/git/haproxy_lwrp# curl https://api.github.com/rate_limit
{
“rate”: {
“limit”: 60,
“remaining”: 60
}
}

I believe a few api requests occur in
https://github.com/opscode/test-kitchen/blob/1.0/lib/kitchen/busser.rb#L100

I don’t think the right answer is ‘can we ship Kitchen Busser with Omnibus
a/k/a Chef-full’.

Could we move https://raw.github.com/opscode/kb/go to opscode.com (http://opscode.com) like we do
with the full installer? curl -L https://www.opscode.com/chef/kb.sh? with
that we could reduce the github api requests.
the basic error seen when getting this can be found
https://gist.github.com/damm/5130887

P.S. I can create an issue for this, I just wanted to start a conversation
about it and get a general idea of where we wanted to steer this before
hand.

Thanks!

Personally I think it’s awesome someone has bumped up against GitHub API throttling because their testing is going that fast. I was taking a look at this last night and there are 2 places where the git tag API calls are going to hurt us:

If we had a way to avoid these calls or ask some other endpoint (static or dynamic), we’d only be asking the GitHub file servers to download the correct tarballs. I’d like to avoid adding too many moving parts however it feels like at least one more something is called for here (barring some re-implementation of kb).

In my first implementation at reducing the GitHub API Requests I found reducing just the installation script was good enough to allow me to test without interruptions. However to scale this further beyond roughly 60 tests per hour (if you use bats, minitest does not incur an api request if i recall, which hints at some flexibility)

Currently I’m toying with a very small web app that can answer a simple question: what is the latest version-tagging git tag on a github repo? This could make use of conditional GETs (which don’t fall under throttling) and the responses could be cached using ETAG headers. The only real change to the kb code would be to call this API to get an answer and proceed to download off GitHub as usual. Essentially this app would mitigate the API throttling for everyone in one place.

I would say if we are trying to cache the sources, basically then let’s host all the source files for testing (including chef-full, kb, etc) as there might be someone who would want to do testing but cannot because their Firewall blocks Github (yes this number is still shockingly high).

So why not do conditional GETs and ETAG caching on our workstations and remote nodes? The problem is that kb lives on those freshly created nodes which have no past history to draw from. Their first call to the GitHub API will always be a cache miss. Worse is that in certain environments (like LXC containers, NAT’ed subnets with VM instances) the source IP to GitHub could all be the same anyway.

Anyway that is what I’d like to spend a bit more time on tonight to flesh it out. Longer term there could be a more clever solution, but for now let’s get everyone back to fast, throttle-free tests :slight_smile:

Thoughts?

Fletcher

My initial thought of implementing this was to stop hardcoding the kb installer and make it available in either .kitchen.local.yml or .kitchen.yml. Additionally from the evidence above, It’s possible to ship kb and it’s counterparts up to rubygems to help (this may be enough for people who no internet access to be able to test)

I hope that helps answer your question.

Thanks!

Scott M. Likens


#11

AJ,

Certainly but that basically locks you into a version of ‘kb’ for testing. Short term that may be do-able but that seems like a slippery crevice i’d rather avoid.

There is no reason you cannot host kb on s3 and have it use directory indexes to ‘walk’ and determine the version to install, and i’m certain there are other ways to do this as well. So possibly making the INSTALL_URL exposed in .kitchen.local.yml could be a good beginning.

Thanks!

On Mar 16, 2013, at 10:27 PM, AJ Christensen aj@junglist.gen.nz wrote:

You can probably just chuck kb into your base containers too, no?

–AJ

On 17 March 2013 07:05, Scott M. Likens scott@likens.us wrote:

On Mar 13, 2013, at 2:24 PM, Fletcher Nichol fnichol@nichol.ca wrote:

On Tuesday, 12 March, 2013 at 6:23 PM, Scott M. Likens wrote:

Hi,

No problem I created it https://github.com/opscode/test-kitchen/issues/68

Thanks!

On Mar 12, 2013, at 12:27 PM, AJ Christensen <aj@junglist.gen.nz (mailto:aj@junglist.gen.nz)> wrote:

I mentioned this to Fletcher on Twitter, he is aware of the situation!
Tickets would be good.

Cheers,

AJ

On 13 March 2013 06:48, Scott M. Likens <scott@likens.us (mailto:scott@likens.us)> wrote:

Dan,

Unfortunately yes. I was able to reduce it to 1 GET (installation of bats,
minitest is installed via rubygems so it does not hit github) by modifying
neutering the installation script at
https://github.com/opscode/kb/blob/go/install to
https://gist.github.com/damm/c7e89264f3130b2c62fc and modify
lib/kitchen/busser.rb to point to a different INSTALL_URL. With that small
change this reduces the problem greatly.

I imagine there are many ways to reduce this pain I’m just not sure of the
quickest and sanest path.

If there’s anyway I can help let me know

Thanks!

On Mar 11, 2013, at 9:50 PM, Daniel DeLeo <dan@kallistec.com (mailto:dan@kallistec.com)> wrote:

Do conditional GETs or HEAD requests count against the limit?


Daniel DeLeo

On Sunday, March 10, 2013 at 3:58 PM, Scott M. Likens wrote:

Hi,

We’ve been having fun using the lxc driver for testing with test-kitchen
(which makes testing really fast!). Unfortunately I forgot that GitHub has
rate limiting imposed now. (60 requests were used up in less than 15
minutes (in part because LXC is so fast at creating and destroying guests
and booting new guests ))

slikens@app379:~/git/haproxy_lwrp# curl https://api.github.com/rate_limit -u
"damm"
Enter host password for user ‘damm’:
{
“rate”: {
“limit”: 5000,
“remaining”: 5000
}
}
slikens@app379:~/git/haproxy_lwrp# curl https://api.github.com/rate_limit
{
“rate”: {
“limit”: 60,
“remaining”: 60
}
}

I believe a few api requests occur in
https://github.com/opscode/test-kitchen/blob/1.0/lib/kitchen/busser.rb#L100

I don’t think the right answer is ‘can we ship Kitchen Busser with Omnibus
a/k/a Chef-full’.

Could we move https://raw.github.com/opscode/kb/go to opscode.com (http://opscode.com) like we do
with the full installer? curl -L https://www.opscode.com/chef/kb.sh? with
that we could reduce the github api requests.
the basic error seen when getting this can be found
https://gist.github.com/damm/5130887

P.S. I can create an issue for this, I just wanted to start a conversation
about it and get a general idea of where we wanted to steer this before
hand.

Thanks!

Personally I think it’s awesome someone has bumped up against GitHub API throttling because their testing is going that fast. I was taking a look at this last night and there are 2 places where the git tag API calls are going to hurt us:

If we had a way to avoid these calls or ask some other endpoint (static or dynamic), we’d only be asking the GitHub file servers to download the correct tarballs. I’d like to avoid adding too many moving parts however it feels like at least one more something is called for here (barring some re-implementation of kb).

In my first implementation at reducing the GitHub API Requests I found reducing just the installation script was good enough to allow me to test without interruptions. However to scale this further beyond roughly 60 tests per hour (if you use bats, minitest does not incur an api request if i recall, which hints at some flexibility)

Currently I’m toying with a very small web app that can answer a simple question: what is the latest version-tagging git tag on a github repo? This could make use of conditional GETs (which don’t fall under throttling) and the responses could be cached using ETAG headers. The only real change to the kb code would be to call this API to get an answer and proceed to download off GitHub as usual. Essentially this app would mitigate the API throttling for everyone in one place.

I would say if we are trying to cache the sources, basically then let’s host all the source files for testing (including chef-full, kb, etc) as there might be someone who would want to do testing but cannot because their Firewall blocks Github (yes this number is still shockingly high).

So why not do conditional GETs and ETAG caching on our workstations and remote nodes? The problem is that kb lives on those freshly created nodes which have no past history to draw from. Their first call to the GitHub API will always be a cache miss. Worse is that in certain environments (like LXC containers, NAT’ed subnets with VM instances) the source IP to GitHub could all be the same anyway.

Anyway that is what I’d like to spend a bit more time on tonight to flesh it out. Longer term there could be a more clever solution, but for now let’s get everyone back to fast, throttle-free tests :slight_smile:

Thoughts?

Fletcher

My initial thought of implementing this was to stop hardcoding the kb installer and make it available in either .kitchen.local.yml or .kitchen.yml. Additionally from the evidence above, It’s possible to ship kb and it’s counterparts up to rubygems to help (this may be enough for people who no internet access to be able to test)

I hope that helps answer your question.

Thanks!

Scott M. Likens

!DSPAM:51455479147001804284693!