pip freeze does case sensitive comparsion

[abs51295]

Environment

• pip version: 18.0
• Python version: 3.6
• OS: Mac OS X

Description

Running pip freeze -l -r requirements.txt does a case sensitive comparison with dependencies mentioned in requirements.txt.

Expected behavior
Running pip freeze -l -r requirements.txt should do case insensitive comparison with dependencies mentioned in requirements.txt or at least allow that via a command line option.

How to Reproduce

1. Create a virtualenv.
2. Create a requirements.txt file.
3. Run pip install -r requirements.txt.
4. Run pip freeze -l -r requirements.txt.

Output

#
# This file is autogenerated by pip-compile
# To update, run:
#    pip-compile --output-file requirements.txt requirements.in
asn1crypto==0.24.0
attrs==18.1.0
certifi==2018.4.16
cffi==1.11.5
chardet==3.0.4
click==6.7
colorama==0.3.9
coverage==4.5.1
cryptography==2.2.2
flake8-polyfill==1.0.2
flake8==3.5.0
Requirement file [requirements.txt] contains flask-cors==3.0.4, but that package is not installed
Requirement file [requirements.txt] contains flask==1.0.2, but that package is not installed
gevent==1.3.1
greenlet==0.4.13
gunicorn==19.8.1
idna==2.6
itsdangerous==0.24
Requirement file [requirements.txt] contains jinja2==2.10, but that package is not installed
mando==0.6.4
Requirement file [requirements.txt] contains markupsafe==1.0, but that package is not installed
mccabe==0.6.1
more-itertools==4.1.0
pluggy==0.6.0
psycopg2==2.7.4
py==1.5.3
pycodestyle==2.3.1
pycparser==2.18
pycrypto==2.6.1
pyflakes==1.6.0
Requirement file [requirements.txt] contains pyjwt==1.6.3, but that package is not installed
pytest-cov==2.5.1
pytest-mock==1.10.0
pytest==3.5.1
radon==2.2.0
requests-futures==0.9.7
requests==2.18.4
semantic-version==2.6.0
six==1.11.0
Requirement file [requirements.txt] contains sqlalchemy==1.2.7, but that package is not installed
urllib3==1.22
Requirement file [requirements.txt] contains werkzeug==0.14.1, but that package is not installed
## The following requirements were added by pip freeze:
Flask==1.0.2
Flask-Cors==3.0.4
Jinja2==2.10
MarkupSafe==1.0
PyJWT==1.6.3
SQLAlchemy==1.2.7
Werkzeug==0.14.1

[abs51295]

@cjerdonek I will like to work on this issue. Let me know, when we have classified this as a bug or a feature request.

[pradyunsg]

Hey @abs51295!

I'd consider this a bug since this is essentially a case where pip is not normalizing the given package names correctly.

[abs51295]

Thanks @pradyunsg I will send a fix soon! 😄

[abs51295]

@pradyunsg how should the fix look like? Should we change the dependency names to lower case explicitly?

[pradyunsg]

There's a normalize_name or a canonical_name (or similarly named) function in pip's codebase, which should be used to normalize names.

(sorry I missed you on IRC - I slept off with the client open)

[abs51295]

Thanks @pradyunsg will look into this.

( I don't see you on #pypa-dev anymore on IRC)

[abs51295]

@pradyunsg I found this: https://github.com/pypa/pip/blob/master/src/pip/_vendor/packaging/requirements.py#L86-L89. Looks like this requires a discussion.

[pradyunsg]

Hey @abs51295! I was away from a PC for most of the time over the weekend.

I've not looked carefully but you'd want to start looking here: https://github.com/pypa/pip/blob/b13cee6388adea5ddaf1ddb982beeec37d07ffda/src/pip/_internal/commands/freeze.py

Basically, whenever pip's getting information from the user/distlib/pkg_resources, you'd want to normalize the name using pip._vendor.packaging.utils.canonicalize_name.

I found this: /src/pip/_vendor/packaging/requirements.py@master#L86-L89.

That's a vendored library (packaging) and while I do think that change needs discussion, the change in pip doesn't and they're fairly separate.

[abs51295]

Basically, whenever pip's getting information from the user/distlib/pkg_resources, you'd want to normalize the name using pip._vendor.packaging.utils.canonicalize_name

@pradyunsg I did that, but why I was referring to the https://github.com/pypa/pip/blob/master/src/pip/_vendor/packaging/requirements.py#L86-L89, is because of the following:

1. Canonicalizing the name when pip's getting the information, which can be achieved by canonicalizing the name here, solves only half the problem.

2. The other half of the problem lies when the requirement line is parsed from requirements.txt file and compared with existing installations as obtained in step 1.

3. During the above comparsion, if we don't canonicalize the name obtained from the requirements.txt file, then we will again run into the same problem of case sensitive comparison. To mitigate that, we need to canonicalize the name of this requirement which leads to this and hence the comment.

[abs51295]

@pradyunsg any updates on this?