How would I prevent python_pip from installing packages that are already installed?

Chef Infra (archive)
1

Hello folks,

I have the following bit of code:

reqs.each do |req|
python_pip req do
virtualenv /foo
action :upgrade
not_if "ls -1 /foo/lib/python33/site-packages | grep #{req}"
end
end

Now, reqs is a pretty large list of pip packages (probably about 30)
compiled from a requirements file. Additionally, each package has a
specific version requirement.
So whenever Chef runs, it painstakingly attempts to upgrade every package
listed in reqs despite my apparently feeble and incompetent not_if. It
causes my Chef run to take 5-7 minutes instead of 2-3. This is fine for the
first run, but this is a development VM for a team that iterates pretty
fast, so 5-7 minute “compile times” are going to be a drag.

One strange thing that might be relevant is that the Chef log reads
something like the below line for each pip package:

==> default: [2015-03-06T13:02:05-05:00] INFO: Upgrading
python_pip[alembic==0.6.4] version from uninstalled to latest

Note that it says “uninstalled” even though the package is definitely
installed.
pip list pip (6.0.8) setuptools (12.4) virtualenv (12.0.7) source /foo/bin/activate
(foo) pip list alembic (0.6.4) ... pip (6.0.8) ... setuptools (13.0.2) ... (foo)

Interestingly though, there is one package in reqs without a version
requirement, it’s gunicorn. When Chef tries to install it, it says this
instead:

==> default: [2015-03-06T13:01:57-05:00] INFO: Upgrading
python_pip[gunicorn] version from 19.3.0 to latest

Frankly it shouldn’t even be trying to install gunicorn anyway, but it’s
interesting/weird to me that for this one package, it accurately detects
that it’s installed and returns the installed version, whereas all the
other ones report that they’re uninstalled even though that’s wrong.

The ls command in my not_if returns the correct code whether the virtual
environment is active or not.

ls -1 /foo/lib/python3.3/site-packages | grep alem; echo ?
alembic
alembic-0.6.4-py3.3.egg-info
0
ls -1 /foo/lib/python3.3/site-packages | grep foo; echo ?
1

So unless I’m off the mark with my logic, the python_pip resource should
execute only if grep returns 1… but that’s not the case and I’m confused
and help please pretty please with powdered sugar and a cherry on top.

Thanks in advance!
Fabien

2

Does it work if you append '$' to the grep like so:

"ls -1 /foo/lib/python33/site-packages | grep '#{req}$'"

--
Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 10:29 AM, Fabien Delpierre <
fabien.delpierre@gmail.com> wrote:

Hello folks,

I have the following bit of code:

reqs.each do |req|
python_pip req do
virtualenv /foo
action :upgrade
not_if "ls -1 /foo/lib/python33/site-packages | grep #{req}"
end
end

Now, reqs is a pretty large list of pip packages (probably about 30)
compiled from a requirements file. Additionally, each package has a
specific version requirement.
So whenever Chef runs, it painstakingly attempts to upgrade every package
listed in reqs despite my apparently feeble and incompetent not_if. It
causes my Chef run to take 5-7 minutes instead of 2-3. This is fine for the
first run, but this is a development VM for a team that iterates pretty
fast, so 5-7 minute "compile times" are going to be a drag.

One strange thing that might be relevant is that the Chef log reads
something like the below line for each pip package:

==> default: [2015-03-06T13:02:05-05:00] INFO: Upgrading
python_pip[alembic==0.6.4] version from uninstalled to latest

Note that it says "uninstalled" even though the package is definitely
installed.
$ pip list
pip (6.0.8)
setuptools (12.4)
virtualenv (12.0.7)
$ source /foo/bin/activate
(foo)$ pip list
alembic (0.6.4)
...
pip (6.0.8)
...
setuptools (13.0.2)
...
(foo)$

Interestingly though, there is one package in reqs without a version
requirement, it's gunicorn. When Chef tries to install it, it says this
instead:

==> default: [2015-03-06T13:01:57-05:00] INFO: Upgrading
python_pip[gunicorn] version from 19.3.0 to latest

Frankly it shouldn't even be trying to install gunicorn anyway, but it's
interesting/weird to me that for this one package, it accurately detects
that it's installed and returns the installed version, whereas all the
other ones report that they're uninstalled even though that's wrong.

The ls command in my not_if returns the correct code whether the virtual
environment is active or not.

$ ls -1 /foo/lib/python3.3/site-packages | grep alem; echo $?
alembic
alembic-0.6.4-py3.3.egg-info
0
$ ls -1 /foo/lib/python3.3/site-packages | grep foo; echo $?
1

So unless I'm off the mark with my logic, the python_pip resource should
execute only if grep returns 1... but that's not the case and I'm confused
and help please pretty please with powdered sugar and a cherry on top.

Thanks in advance!
Fabien

3

Thanks for the suggestion!
No dice, unfortunately, it still took me for the full round of unneeded
"updates".

On Fri, Mar 6, 2015 at 1:38 PM, Michael Weinberg michael@hw-ops.com wrote:

Does it work if you append '$' to the grep like so:

"ls -1 /foo/lib/python33/site-packages | grep '#{req}$'"

--
Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 10:29 AM, Fabien Delpierre <
fabien.delpierre@gmail.com> wrote:

Hello folks,

I have the following bit of code:

reqs.each do |req|
python_pip req do
virtualenv /foo
action :upgrade
not_if "ls -1 /foo/lib/python33/site-packages | grep #{req}"
end
end

Now, reqs is a pretty large list of pip packages (probably about 30)
compiled from a requirements file. Additionally, each package has a
specific version requirement.
So whenever Chef runs, it painstakingly attempts to upgrade every package
listed in reqs despite my apparently feeble and incompetent not_if. It
causes my Chef run to take 5-7 minutes instead of 2-3. This is fine for the
first run, but this is a development VM for a team that iterates pretty
fast, so 5-7 minute "compile times" are going to be a drag.

One strange thing that might be relevant is that the Chef log reads
something like the below line for each pip package:

==> default: [2015-03-06T13:02:05-05:00] INFO: Upgrading
python_pip[alembic==0.6.4] version from uninstalled to latest

Note that it says "uninstalled" even though the package is definitely
installed.
$ pip list
pip (6.0.8)
setuptools (12.4)
virtualenv (12.0.7)
$ source /foo/bin/activate
(foo)$ pip list
alembic (0.6.4)
...
pip (6.0.8)
...
setuptools (13.0.2)
...
(foo)$

Interestingly though, there is one package in reqs without a version
requirement, it's gunicorn. When Chef tries to install it, it says this
instead:

==> default: [2015-03-06T13:01:57-05:00] INFO: Upgrading
python_pip[gunicorn] version from 19.3.0 to latest

Frankly it shouldn't even be trying to install gunicorn anyway, but it's
interesting/weird to me that for this one package, it accurately detects
that it's installed and returns the installed version, whereas all the
other ones report that they're uninstalled even though that's wrong.

The ls command in my not_if returns the correct code whether the virtual
environment is active or not.

$ ls -1 /foo/lib/python3.3/site-packages | grep alem; echo $?
alembic
alembic-0.6.4-py3.3.egg-info
0
$ ls -1 /foo/lib/python3.3/site-packages | grep foo; echo $?
1

So unless I'm off the mark with my logic, the python_pip resource should
execute only if grep returns 1... but that's not the case and I'm confused
and help please pretty please with powdered sugar and a cherry on top.

Thanks in advance!
Fabien

4

Fabien,

This may be a copy-paste typo restricted to your message to the list, I realise, but just in case: the path you specify in your guard is not the same as the one you use when running the command manually:

ls -1 /foo/lib/python33/site-packages
ls -1 /foo/lib/python3.3/site-packages

cf. 'python33' in the first one vs 'python3.3' (33 vs 3.3) in the second one. Is the typo present in your code?

am

On 6 Mar 2015, at 20:43, Fabien Delpierre fabien.delpierre@gmail.com wrote:

Thanks for the suggestion!
No dice, unfortunately, it still took me for the full round of unneeded "updates".

On Fri, Mar 6, 2015 at 1:38 PM, Michael Weinberg michael@hw-ops.com wrote:
Does it work if you append '$' to the grep like so:

"ls -1 /foo/lib/python33/site-packages | grep '#{req}$'"

--
Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 10:29 AM, Fabien Delpierre fabien.delpierre@gmail.com wrote:
Hello folks,

I have the following bit of code:

reqs.each do |req|
python_pip req do
virtualenv /foo
action :upgrade
not_if "ls -1 /foo/lib/python33/site-packages | grep #{req}"
end
end

Now, reqs is a pretty large list of pip packages (probably about 30) compiled from a requirements file. Additionally, each package has a specific version requirement.
So whenever Chef runs, it painstakingly attempts to upgrade every package listed in reqs despite my apparently feeble and incompetent not_if. It causes my Chef run to take 5-7 minutes instead of 2-3. This is fine for the first run, but this is a development VM for a team that iterates pretty fast, so 5-7 minute "compile times" are going to be a drag.

One strange thing that might be relevant is that the Chef log reads something like the below line for each pip package:

==> default: [2015-03-06T13:02:05-05:00] INFO: Upgrading python_pip[alembic==0.6.4] version from uninstalled to latest

Note that it says "uninstalled" even though the package is definitely installed.
$ pip list
pip (6.0.8)
setuptools (12.4)
virtualenv (12.0.7)
$ source /foo/bin/activate
(foo)$ pip list
alembic (0.6.4)
...
pip (6.0.8)
...
setuptools (13.0.2)
...
(foo)$

Interestingly though, there is one package in reqs without a version requirement, it's gunicorn. When Chef tries to install it, it says this instead:

==> default: [2015-03-06T13:01:57-05:00] INFO: Upgrading python_pip[gunicorn] version from 19.3.0 to latest

Frankly it shouldn't even be trying to install gunicorn anyway, but it's interesting/weird to me that for this one package, it accurately detects that it's installed and returns the installed version, whereas all the other ones report that they're uninstalled even though that's wrong.

The ls command in my not_if returns the correct code whether the virtual environment is active or not.

$ ls -1 /foo/lib/python3.3/site-packages | grep alem; echo $?
alembic
alembic-0.6.4-py3.3.egg-info
0
$ ls -1 /foo/lib/python3.3/site-packages | grep foo; echo $?
1

So unless I'm off the mark with my logic, the python_pip resource should execute only if grep returns 1... but that's not the case and I'm confused and help please pretty please with powdered sugar and a cherry on top.

Thanks in advance!
Fabien

5

On Friday, March 6, 2015 at 10:29 AM, Fabien Delpierre wrote:

Hello folks,

I have the following bit of code:

reqs.each do |req|
python_pip req do
virtualenv /foo
action :upgrade
not_if "ls -1 /foo/lib/python33/site-packages | grep #{req}"
end
end

Now, reqs is a pretty large list of pip packages (probably about 30) compiled from a requirements file. Additionally, each package has a specific version requirement.
So whenever Chef runs, it painstakingly attempts to upgrade every package listed in reqs despite my apparently feeble and incompetent not_if. It causes my Chef run to take 5-7 minutes instead of 2-3. This is fine for the first run, but this is a development VM for a team that iterates pretty fast, so 5-7 minute "compile times" are going to be a drag.

One strange thing that might be relevant is that the Chef log reads something like the below line for each pip package:

==> default: [2015-03-06T13:02:05-05:00] INFO: Upgrading python_pip[alembic==0.6.4] version from uninstalled to latest

Note that it says "uninstalled" even though the package is definitely installed.
$ pip list
pip (6.0.8)
setuptools (12.4)
virtualenv (12.0.7)
$ source /foo/bin/activate
(foo)$ pip list
alembic (0.6.4)
...
pip (6.0.8)
...
setuptools (13.0.2)
...
(foo)$

Interestingly though, there is one package in reqs without a version requirement, it's gunicorn. When Chef tries to install it, it says this instead:

==> default: [2015-03-06T13:01:57-05:00] INFO: Upgrading python_pip[gunicorn] version from 19.3.0 to latest

Frankly it shouldn't even be trying to install gunicorn anyway, but it's interesting/weird to me that for this one package, it accurately detects that it's installed and returns the installed version, whereas all the other ones report that they're uninstalled even though that's wrong.

All of these things sound like bugs, you should file an issue.

The ls command in my not_if returns the correct code whether the virtual environment is active or not.

$ ls -1 /foo/lib/python3.3/site-packages | grep alem; echo $?
alembic
alembic-0.6.4-py3.3.egg-info
0
$ ls -1 /foo/lib/python3.3/site-packages | grep foo; echo $?
1

So unless I'm off the mark with my logic, the python_pip resource should execute only if grep returns 1... but that's not the case and I'm confused and help please pretty please with powdered sugar and a cherry on top.
Since you know the version you want, have you tried setting the version on the package resource and specifying action :install ? Including the desired version makes the install action behave like upgrade or downgrade if your existing version isn’t the desired one (modulo weirdness in the underlying packager’s implementation, for example rpm needs an extra flag to allow downgrades). It may be that the pip provider is just naive and doesn’t yet support this functionality, in which case that would be a separate bug to file.

Thanks in advance!
Fabien

--
Daniel DeLeo

6

Aimelyne,
Thanks for pointing it out, but no, it's all good in the code, I think. In
fact the code uses attributes extensively, I just made it plainer just for
clarity. I guess I typo'd the 'python33', indeed it should read
'python3.3'. Only the "python" is hardcoded in the code but the digits come
from an attribute and should = 3.3, which is what I need since that's the
correct path :slight_smile:

default['python']['version'] = '3.3.5'
default['foo']['python_minor_version'] =
node['python']['version'].split(/(^\d+.\d+)/)[1]

reqs.each do |req|
python_pip req do
virtualenv node['foo']['venv']
action :install
not_if "ls -1
#{node['foo']['venv']}/lib/python#{node['foo']['python_minor_version']}/site-packages
| grep '#{req}$'"
end
end

On Fri, Mar 6, 2015 at 4:58 PM, Aimelyne Mochiron aimelynem@gmail.com
wrote:

Fabien,

This may be a copy-paste typo restricted to your message to the list, I
realise, but just in case: the path you specify in your guard is not the
same as the one you use when running the command manually:

ls -1 /foo/lib/python33/site-packages
ls -1 /foo/lib/python3.3/site-packages

cf. 'python33' in the first one vs 'python3.3' (33 vs 3.3) in the second
one. Is the typo present in your code?

am

On 6 Mar 2015, at 20:43, Fabien Delpierre fabien.delpierre@gmail.com
wrote:

Thanks for the suggestion!
No dice, unfortunately, it still took me for the full round of unneeded
"updates".

On Fri, Mar 6, 2015 at 1:38 PM, Michael Weinberg michael@hw-ops.com
wrote:

Does it work if you append '$' to the grep like so:

"ls -1 /foo/lib/python33/site-packages | grep '#{req}$'"

--
Michael F. Weinberg | Director of Operations
http://heavywaterops.com | @heavywaterops

On Fri, Mar 6, 2015 at 10:29 AM, Fabien Delpierre <
fabien.delpierre@gmail.com> wrote:

Hello folks,

I have the following bit of code:

reqs.each do |req|
python_pip req do
virtualenv /foo
action :upgrade
not_if "ls -1 /foo/lib/python33/site-packages | grep #{req}"
end
end

Now, reqs is a pretty large list of pip packages (probably about 30)
compiled from a requirements file. Additionally, each package has a
specific version requirement.
So whenever Chef runs, it painstakingly attempts to upgrade every
package listed in reqs despite my apparently feeble and incompetent
not_if. It causes my Chef run to take 5-7 minutes instead of 2-3. This
is fine for the first run, but this is a development VM for a team that
iterates pretty fast, so 5-7 minute "compile times" are going to be a drag.

One strange thing that might be relevant is that the Chef log reads
something like the below line for each pip package:

==> default: [2015-03-06T13:02:05-05:00] INFO: Upgrading
python_pip[alembic==0.6.4] version from uninstalled to latest

Note that it says "uninstalled" even though the package is definitely
installed.
$ pip list
pip (6.0.8)
setuptools (12.4)
virtualenv (12.0.7)
$ source /foo/bin/activate
(foo)$ pip list
alembic (0.6.4)
...
pip (6.0.8)
...
setuptools (13.0.2)
...
(foo)$

Interestingly though, there is one package in reqs without a version
requirement, it's gunicorn. When Chef tries to install it, it says this
instead:

==> default: [2015-03-06T13:01:57-05:00] INFO: Upgrading
python_pip[gunicorn] version from 19.3.0 to latest

Frankly it shouldn't even be trying to install gunicorn anyway, but it's
interesting/weird to me that for this one package, it accurately detects
that it's installed and returns the installed version, whereas all the
other ones report that they're uninstalled even though that's wrong.

The ls command in my not_if returns the correct code whether the
virtual environment is active or not.

$ ls -1 /foo/lib/python3.3/site-packages | grep alem; echo $?
alembic
alembic-0.6.4-py3.3.egg-info
0
$ ls -1 /foo/lib/python3.3/site-packages | grep foo; echo $?
1

So unless I'm off the mark with my logic, the python_pip resource should
execute only if grep returns 1... but that's not the case and I'm confused
and help please pretty please with powdered sugar and a cherry on top.

Thanks in advance!
Fabien

7

Daniel,
Setting the version like:
python_pip 'package' do
version 'x'
...
end
...sounds like something I should be able to figure out. I have a
requirements file that looks like this:
package==version
And a long list of packages.
I'd just need to parse it to separate the package names from the desired
versions, instead of what I'm doing now, which really is doing:
python_pip 'package==version' do
...
end
Maybe that's what I'm doing wrong to begin with, but since the conventional
method to install a specific version is to do $ pip install package==version,
I figured that doing it like I'm doing would pass the package==version
directly to the python_pip resource and it would take care of itself fine.

I don't think submitting bugs is going to do anything since the
maintainer(s) of the python cookbook have not quite abandoned it but keep
saying they mean to gut it and redo the whole thing from the ground up,
so... I dunno :slight_smile: Anyway, I will try parsing my requirements file. It's
funny because I'm refactoring the original code written by someone else,
and they just had:
bash "install packages" do
cwd "/foo"
code "source venv/bin/activate; pip install -r setup/reqs.txt"
end
And I thought that was horribly inelegant, but I've spent more time getting
this to work than I would have by leaving it alone. I'll have learned a few
things by the time I'm done, though, so that's alright.

Thanks for the suggestion, I'll have to try it next week!

On Fri, Mar 6, 2015 at 5:07 PM, Daniel DeLeo dan@kallistec.com wrote:

On Friday, March 6, 2015 at 10:29 AM, Fabien Delpierre wrote:

Hello folks,

I have the following bit of code:

reqs.each do |req|
python_pip req do
virtualenv /foo
action :upgrade
not_if "ls -1 /foo/lib/python33/site-packages | grep #{req}"
end
end

Now, reqs is a pretty large list of pip packages (probably about 30)
compiled from a requirements file. Additionally, each package has a
specific version requirement.
So whenever Chef runs, it painstakingly attempts to upgrade every
package listed in reqs despite my apparently feeble and incompetent not_if.
It causes my Chef run to take 5-7 minutes instead of 2-3. This is fine for
the first run, but this is a development VM for a team that iterates pretty
fast, so 5-7 minute "compile times" are going to be a drag.

One strange thing that might be relevant is that the Chef log reads
something like the below line for each pip package:

==> default: [2015-03-06T13:02:05-05:00] INFO: Upgrading
python_pip[alembic==0.6.4] version from uninstalled to latest

Note that it says "uninstalled" even though the package is definitely
installed.
$ pip list
pip (6.0.8)
setuptools (12.4)
virtualenv (12.0.7)
$ source /foo/bin/activate
(foo)$ pip list
alembic (0.6.4)
...
pip (6.0.8)
...
setuptools (13.0.2)
...
(foo)$

Interestingly though, there is one package in reqs without a version
requirement, it's gunicorn. When Chef tries to install it, it says this
instead:

==> default: [2015-03-06T13:01:57-05:00] INFO: Upgrading
python_pip[gunicorn] version from 19.3.0 to latest

Frankly it shouldn't even be trying to install gunicorn anyway, but it's
interesting/weird to me that for this one package, it accurately detects
that it's installed and returns the installed version, whereas all the
other ones report that they're uninstalled even though that's wrong.

All of these things sound like bugs, you should file an issue.

The ls command in my not_if returns the correct code whether the virtual
environment is active or not.

$ ls -1 /foo/lib/python3.3/site-packages | grep alem; echo $?
alembic
alembic-0.6.4-py3.3.egg-info
0
$ ls -1 /foo/lib/python3.3/site-packages | grep foo; echo $?
1

So unless I'm off the mark with my logic, the python_pip resource should
execute only if grep returns 1... but that's not the case and I'm confused
and help please pretty please with powdered sugar and a cherry on top.
Since you know the version you want, have you tried setting the version on
the package resource and specifying action :install ? Including the
desired version makes the install action behave like upgrade or downgrade
if your existing version isn’t the desired one (modulo weirdness in the
underlying packager’s implementation, for example rpm needs an extra flag
to allow downgrades). It may be that the pip provider is just naive and
doesn’t yet support this functionality, in which case that would be a
separate bug to file.

Thanks in advance!
Fabien

--
Daniel DeLeo

8

On Friday, March 6, 2015 at 2:39 PM, Fabien Delpierre wrote:

Daniel,
Setting the version like:
python_pip 'package' do
version 'x'
...
end
...sounds like something I should be able to figure out. I have a requirements file that looks like this:
package==version
And a long list of packages.

Ruby’s pretty good for quick and dirty scripting of this kind of thing. Something like this should work:

require ‘pp’ # just for debugging/showing the example

names_and_version_specs = IO.readlines('/tmp/example.txt')
names_and_versions = names_and_version_specs.map do |line|
line.strip.split('==')
end

debug

pp names_and_versions

do something like this in your recipe:

names_and_versions.each do |name, pkg_version|

python_pip name do

version pkg_version

end

end

HTH,

--
Daniel DeLeo

9

Success! Initially I had some trouble because I need to remove a few lines
from my pip requirements file before passing it to Chef, and after running
that .split, my code that did that no longer worked, so I struggled with
that for a bit, then it occurred to me that I could just clean up the
requirements file before splitting the package from the package version,
and now I have working code, and Chef runs after the first one now take
50-60 seconds. Seems like I don't even need that not_if anymore, either. So
here's my updated code if anyone cares. It's probably not very elegant but
it's the best this newbie programmer can come up with, and it works.

Install packages not loaded from the requirements file

node['foo']['venv_packages'].each do |pkg, pkg_version|
python_pip pkg do
virtualenv node['foo']['venv']
version pkg_version
action :install
end
end

Parse the pip requirements file and remove pyodbc, we'll deal with it

separately
reqs = IO.readlines("/foo/setup/dev-requirements.txt").map {|r| r.chomp}
pyodbc =

while reqs.index{|r| r.include?("pyodbc")} != nil
match = reqs.index{|r| r.include?("pyodbc")}
pyodbc.push(reqs[match])
reqs.delete_at(match)
end

Now re-parse the array to split the package names from the desired

versions
reqs = reqs.map {|r| r.strip.split('==')}

Install the packages we want in the virtual environment

reqs.each do |pkg, pkg_version|
python_pip pkg do
virtualenv node['foo']['venv']
version pkg_version
action :install
end
end

Now install pyodbc

match = pyodbc.index{|p| p.include?("pyodbc=")}
pyodbc_pkg = pyodbc[match].split("==") # this should now be =
["pyodbc","3.0.7"]
pyodbc.delete_at(match)
pyodbc_args = pyodbc * " " # this should now be = "--allow-external pyodbc
--allow-public pyodbc"

python_pip pyodbc_pkg[0] do
virtualenv node['foo']['venv']
version pyodbc_pkg[1]
options pyodbc_args
action :install
end

On Fri, Mar 6, 2015 at 7:32 PM, Daniel DeLeo dan@kallistec.com wrote:

On Friday, March 6, 2015 at 2:39 PM, Fabien Delpierre wrote:

Daniel,
Setting the version like:
python_pip 'package' do
version 'x'
...
end
...sounds like something I should be able to figure out. I have a
requirements file that looks like this:
package==version
And a long list of packages.

Ruby’s pretty good for quick and dirty scripting of this kind of thing.
Something like this should work:

require ‘pp’ # just for debugging/showing the example

names_and_version_specs = IO.readlines('/tmp/example.txt')
names_and_versions = names_and_version_specs.map do |line|
line.strip.split('==')
end

debug

pp names_and_versions

do something like this in your recipe:

names_and_versions.each do |name, pkg_version|

python_pip name do

version pkg_version

end

end

HTH,

--
Daniel DeLeo

10

The internal implementation of python_pip should be responsible for
figuring out if it needs to take action to repair its state.

If it doesn't, its a bug.

You shouldn't have to do any of that in your recipe.

-s

On Sun, Mar 8, 2015 at 4:53 PM, Fabien Delpierre
fabien.delpierre@gmail.com wrote:

Success! Initially I had some trouble because I need to remove a few lines
from my pip requirements file before passing it to Chef, and after running
that .split, my code that did that no longer worked, so I struggled with
that for a bit, then it occurred to me that I could just clean up the
requirements file before splitting the package from the package version, and
now I have working code, and Chef runs after the first one now take 50-60
seconds. Seems like I don't even need that not_if anymore, either. So here's
my updated code if anyone cares. It's probably not very elegant but it's the
best this newbie programmer can come up with, and it works.

Install packages not loaded from the requirements file

node['foo']['venv_packages'].each do |pkg, pkg_version|
python_pip pkg do
virtualenv node['foo']['venv']
version pkg_version
action :install
end
end

Parse the pip requirements file and remove pyodbc, we'll deal with it

separately
reqs = IO.readlines("/foo/setup/dev-requirements.txt").map {|r| r.chomp}
pyodbc =

while reqs.index{|r| r.include?("pyodbc")} != nil
match = reqs.index{|r| r.include?("pyodbc")}
pyodbc.push(reqs[match])
reqs.delete_at(match)
end

Now re-parse the array to split the package names from the desired

versions
reqs = reqs.map {|r| r.strip.split('==')}

Install the packages we want in the virtual environment

reqs.each do |pkg, pkg_version|
python_pip pkg do
virtualenv node['foo']['venv']
version pkg_version
action :install
end
end

Now install pyodbc

match = pyodbc.index{|p| p.include?("pyodbc=")}
pyodbc_pkg = pyodbc[match].split("==") # this should now be =
["pyodbc","3.0.7"]
pyodbc.delete_at(match)
pyodbc_args = pyodbc * " " # this should now be = "--allow-external pyodbc
--allow-public pyodbc"

python_pip pyodbc_pkg[0] do
virtualenv node['foo']['venv']
version pyodbc_pkg[1]
options pyodbc_args
action :install
end

On Fri, Mar 6, 2015 at 7:32 PM, Daniel DeLeo dan@kallistec.com wrote:

On Friday, March 6, 2015 at 2:39 PM, Fabien Delpierre wrote:

Daniel,
Setting the version like:
python_pip 'package' do
version 'x'
...
end
...sounds like something I should be able to figure out. I have a
requirements file that looks like this:
package==version
And a long list of packages.

Ruby’s pretty good for quick and dirty scripting of this kind of thing.
Something like this should work:

require ‘pp’ # just for debugging/showing the example

names_and_version_specs = IO.readlines('/tmp/example.txt')
names_and_versions = names_and_version_specs.map do |line|
line.strip.split('==')
end

debug

pp names_and_versions

do something like this in your recipe:

names_and_versions.each do |name, pkg_version|

python_pip name do

version pkg_version

end

end

HTH,

--
Daniel DeLeo

11

Thanks Sean.
I got it to work with with Daniel's help. First I was using the :upgrade
action because I encountered a problem with :install that using :upgrade
instead fixed, but it turns out using :install works fine after refactoring
the code some. No bugs as far as I can tell!

On Mon, Mar 9, 2015 at 9:30 AM, Sean OMeara someara@chef.io wrote:

The internal implementation of python_pip should be responsible for
figuring out if it needs to take action to repair its state.

If it doesn't, its a bug.

You shouldn't have to do any of that in your recipe.

-s

On Sun, Mar 8, 2015 at 4:53 PM, Fabien Delpierre
fabien.delpierre@gmail.com wrote:

Success! Initially I had some trouble because I need to remove a few
lines
from my pip requirements file before passing it to Chef, and after
running
that .split, my code that did that no longer worked, so I struggled with
that for a bit, then it occurred to me that I could just clean up the
requirements file before splitting the package from the package version,
and
now I have working code, and Chef runs after the first one now take 50-60
seconds. Seems like I don't even need that not_if anymore, either. So
here's
my updated code if anyone cares. It's probably not very elegant but it's
the
best this newbie programmer can come up with, and it works.

Install packages not loaded from the requirements file

node['foo']['venv_packages'].each do |pkg, pkg_version|
python_pip pkg do
virtualenv node['foo']['venv']
version pkg_version
action :install
end
end

Parse the pip requirements file and remove pyodbc, we'll deal with it

separately
reqs = IO.readlines("/foo/setup/dev-requirements.txt").map {|r| r.chomp}
pyodbc =

while reqs.index{|r| r.include?("pyodbc")} != nil
match = reqs.index{|r| r.include?("pyodbc")}
pyodbc.push(reqs[match])
reqs.delete_at(match)
end

Now re-parse the array to split the package names from the desired

versions
reqs = reqs.map {|r| r.strip.split('==')}

Install the packages we want in the virtual environment

reqs.each do |pkg, pkg_version|
python_pip pkg do
virtualenv node['foo']['venv']
version pkg_version
action :install
end
end

Now install pyodbc

match = pyodbc.index{|p| p.include?("pyodbc=")}
pyodbc_pkg = pyodbc[match].split("==") # this should now be =
["pyodbc","3.0.7"]
pyodbc.delete_at(match)
pyodbc_args = pyodbc * " " # this should now be = "--allow-external
pyodbc
--allow-public pyodbc"

python_pip pyodbc_pkg[0] do
virtualenv node['foo']['venv']
version pyodbc_pkg[1]
options pyodbc_args
action :install
end

On Fri, Mar 6, 2015 at 7:32 PM, Daniel DeLeo dan@kallistec.com wrote:

On Friday, March 6, 2015 at 2:39 PM, Fabien Delpierre wrote:

Daniel,
Setting the version like:
python_pip 'package' do
version 'x'
...
end
...sounds like something I should be able to figure out. I have a
requirements file that looks like this:
package==version
And a long list of packages.

Ruby’s pretty good for quick and dirty scripting of this kind of thing.
Something like this should work:

require ‘pp’ # just for debugging/showing the example

names_and_version_specs = IO.readlines('/tmp/example.txt')
names_and_versions = names_and_version_specs.map do |line|
line.strip.split('==')
end

debug

pp names_and_versions

do something like this in your recipe:

names_and_versions.each do |name, pkg_version|

python_pip name do

version pkg_version

end

end

HTH,

--
Daniel DeLeo