Never use the pkg_resources metadata backend on python 3.13+

[sbidoul]

On python 3.13+ we don't support the fallback to the pkg_resources metadata backend, and don't attempt to detect .egg installed distributions (#12330).

In effect, this means pkg_resources is never used at all by pip on python 3.13 (to the best of my knowledge).

This does not mean we will not want to drop support for the pkg_resources backend before we drop support for python 3.12, but at least this places a limit on it.

@pypa/pip-committers what do you think?

[ichard26]

We should probably check in our redistributors to make sure they're OK with this change. /cc @FFY00 @eli-schwartz @stefanor @doko42 @kitterma @hroncok

TL;DR, we'd like to stop using pkg_resources as our metadata backend (e.g. used to discover installed packages) at some point. It's old and has warts that simply don't exist with the importlib-metadata backend. For more details, please see #7413.

Anyway, for a long time, pip has defaulted to using the importlib-metadata backend on Python 3.11 and higher, while sticking with pkg_resources otherwise. In addition, the pkg_resources backend is still used for all Python versions to discover .egg distributions (not .egg-info!) as importlib-metadata lacks support for them. Finally, there are various ways for end-users and redistributors to opt-out of the importlib-metadata backend.

We'd like to disable these escape hatches on Python 3.13 so we have a migration path to eliminating the pkg_resources backend entirely. Our question is whether you still need the pkg_resources backend on Python 3.13 or higher for any reasons? For example, if you still ship any .egg distributions on Python 3.13, pip would not be able to discover or uninstall those packages.

[eli-schwartz]

Gentoo does not ship any .egg files, which I believe means we shouldn't be affected by this change, correct?

[hroncok]

For example, if you still ship any .egg distributions on Python 3.13, pip would not be able to discover or uninstall those packages.

If we do, it's always some cmake-wrpapped-autotools-bash-script-calling-setup.py-with-weird-arguments situation. Such packages are hardly ever interacted with by the rest of the Python ecosystem.

And we don't want pip to uninstall RPM-installed packages. So that part is more than OK.

---

Anyway, on Fedora 42:

• conda-tests has eggs in /usr/share/conda/tests/data/env_metadata/envpy27osx/lib/python2.7/site-packages -- I don't care about Python 2.7 🙈
• deluge-common has some egg based plugins in /usr/lib/python3.13/site-packages/deluge/plugins/ but they don't seem to depend on pip to load them
• python3-distributed has test egg in /usr/lib/python3.13/site-packages/distributed/tests/
• Python has test egg in /usr/lib/python3.13/test/test_importlib/metadata/data
• /usr/lib64/python3.13/site-packages/urjtag-2021.3-py3.13-linux-x86_64.egg seems like a real egg installed by autotools, I will deal with this one

(based on repoquery -q --repo=rawhide -f '*.egg')

[hroncok]

/usr/lib64/python3.13/site-packages/urjtag-2021.3-py3.13-linux-x86_64.egg is caused by pypa/setuptools#3143 which has been without a response for a couple of years now.

I suggest it would be excellent to make setuptools not create eggs first before you land this, but perhaps you need to land this before you make this a priority for setuptools :/

[pfmoore]

We have no control over what setuptools does, unfortunately, but given that pypa/setuptools#3143 appears to be triggered only by using the deprecated setup.py install invocation, I don't think it's something we should hold pip up for.

[stefanor]

In Debian we have a couple of packages installed as .eggs. Usually these are expanded (.egg directories), but some are zip files:

$ apt-file search .egg | sed -ne 's,\.egg/.*,.egg/, p' | uniq
deluge-common: /usr/lib/python3/dist-packages/deluge/plugins/AutoAdd-1.8.egg/
deluge-common: /usr/lib/python3/dist-packages/deluge/plugins/Blocklist-1.4.egg/
deluge-common: /usr/lib/python3/dist-packages/deluge/plugins/Execute-1.3.egg/
deluge-common: /usr/lib/python3/dist-packages/deluge/plugins/Extractor-0.7.egg/
deluge-common: /usr/lib/python3/dist-packages/deluge/plugins/Label-0.3.egg/
deluge-common: /usr/lib/python3/dist-packages/deluge/plugins/Notifications-0.4.egg/
deluge-common: /usr/lib/python3/dist-packages/deluge/plugins/Scheduler-0.3.egg/
deluge-common: /usr/lib/python3/dist-packages/deluge/plugins/Stats-0.4.egg/
deluge-common: /usr/lib/python3/dist-packages/deluge/plugins/Toggle-0.4.egg/
deluge-common: /usr/lib/python3/dist-packages/deluge/plugins/WebUi-0.2.egg/
python3-astra-toolbox: /usr/lib/python3/dist-packages/astra_toolbox-2.1.0-py3.11-linux-x86_64.egg/
python3-ferret: /usr/lib/python3/dist-packages/pyferret-7.65-py3.12-linux-x86_64.egg/
python3-nose2: /usr/lib/python3/dist-packages/nose2/tests/functional/support/scenario/tests_in_unzipped_eggs/pkgunegg-0.0.0-py2.7.egg/
python3-pkg-resources: /usr/lib/python3/dist-packages/pkg_resources/tests/data/my-test-package_unpacked-egg/my_test_package-1.0-py3.7.egg/
python3-pmix: /usr/lib/python3/dist-packages/pypmix-5.0.5-py3.12-linux-x86_64.egg/
python3-zope.testrunner: /usr/lib/python3/dist-packages/zope/testrunner/tests/testrunner-ex-251759/eggs/foo.bar-1.2-py2.5.egg/
quantlib-python: /usr/lib/python3/dist-packages/QuantLib-1.35-py3.12-linux-x86_64.egg/
quantlib-python: /usr/lib/python3/dist-packages/QuantLib-1.36-py3.12-linux-x86_64.egg/

$ apt-file search .egg | grep '\.egg$'
cappuccino: /usr/lib/python3/dist-packages/cappuccino-0.5.1-py3.12.egg
libpython3.12-testsuite: /usr/lib/python3.12/test/test_importlib/data/example-21.12-py3.6.egg
libpython3.13-testsuite: /usr/lib/python3.13/test/test_importlib/metadata/data/example-21.12-py3.6.egg
librust-python-pkginfo-dev: /usr/share/cargo/registry/python-pkginfo-0.6.3/tests/fixtures/build-0.4.0-py3.9.egg
pypy3-lib-testsuite: /usr/lib/pypy3.10/test/test_importlib/data/example-21.12-py3.6.egg
python3-distributed: /usr/lib/python3/dist-packages/distributed/tests/testegg-1.0.0-py3.4.egg
python3-ferret: /usr/lib/python3/dist-packages/gcircle-7.65-py3.12.egg
python3-ferret: /usr/lib/python3/dist-packages/pipedviewer-7.65-py3.12.egg
python3-nose2: /usr/lib/python3/dist-packages/nose2/tests/functional/support/scenario/tests_in_zipped_eggs/pkgegg-0.0.0-py2.7.egg
python3-pkg-resources: /usr/lib/python3/dist-packages/pkg_resources/tests/data/my-test-package_zipped-egg/my_test_package-1.0-py3.7.egg

Many of these are off sys.path, inside test suites (that in many cases should be packaged somewhere else).

We use EXTERNALLY_MANAGED to disallow pip installs, so I'm not really worried by these remaining corner cases.

[ichard26]

Thanks @eli-schwartz @hroncok and @stefanor for responding!

I suggest it would be excellent to make setuptools not create eggs first before you land this, but perhaps you need to land this before you make this a priority for setuptools :/

@hroncok I did reach out to the the setuptools folks to flag pypa/setuptools#3143, but ultimately whether they decide to do anything is entirely up to them. We have no control over what setuptools does, sadly.

Gentoo does not ship any .egg files, which I believe means we shouldn't be affected by this change, correct?

@eli-schwartz Assuming Gentoo doesn't patch pip or CPython to force the use of the legacy pkg_resources backend for any other reason, yeah, y'all should be fine. 👍

We use EXTERNALLY_MANAGED to disallow pip installs, so I'm not really worried by these remaining corner cases.

@stefanor Sounds great! Thanks for letting us know!

I don't think we've heard from the Arch folks yet? Filipe is probably busy so I'll leave it for a little bit longer and then ping someone else.

Please note that I am purposefully making no comment on whether we should actually stop using the pkg_resources backend entirely on Python 3.13. I haven't looked at the issue and the context surrounding it in close enough detail yet. I just want to make sure we're making an informed decision.

[eli-schwartz]

@hroncok

/usr/lib64/python3.13/site-packages/urjtag-2021.3-py3.13-linux-x86_64.egg seems like a real egg installed by autotools, I will deal with this one

Interestingly, Gentoo has this package but ours installs egg-info, not an egg. It is still a legacy package and runs setup.py install directly, but the recipe does it explicitly and doesn't utilize the upstream Makefile. The difference is that DESTDIR is passed as --root, whereas the Makefile tries to include it in --prefix. Gentoo avoids the Makefile primarily since we support building for multiple versions of python and the Makefile isn't set up to do that.

Arguably it should be migrated from legacy setup.py with egg-info to a wheel builder/installer anyway, even though the pkg_resources backend for pip isn't affected. I guess this ancient unmaintained JTAG package is not commonly configured by users to enable the non-default, optional python backend. :)

deluge-common has some egg based plugins in /usr/lib/python3.13/site-packages/deluge/plugins/ but they don't seem to depend on pip to load them

Yup, I agree this shouldn't matter.

@stefanor

python3-pmix: /usr/lib/python3/dist-packages/pypmix-5.0.5-py3.12-linux-x86_64.egg/

Another automake installation of a setup.py using --prefix="$(DESTDIR)$(prefix)", Gentoo doesn't build the python bindings at all though.