Add support for Python dependency groups

[redeboer]

Problem description

PEP 735 recently got accepted (see #2211 (comment)). It can be used as a mechanism for defining Python dependencies for the developer environment without exposing them in the built distribution. I expect this will be quickly adopted by Python package managers, see e.g. python-poetry/poetry#9751, astral-sh/uv#8090, pypa/pip#12963.

Would it be possible to include dependency groups in the Pixi lock file and recognize it in features, similarly like how Pixi handles optional-dependencies?

[tdejager]

Yeah I think we should support, we could make an environment per dependency-group, or something else.

The most interesting thing is this part of the PEP is:

The mutual compatibility of Dependency Groups is not guaranteed. For example, the Data Science example above shows conflicting versions of scikit-learn. Therefore, installing multiple locked dependency groups in tandem may require that tools apply additional constraints or generate additional lockfile data. These problems are considered out of scope for this PEP.
As two examples of how combinations might be locked:

• A tool might require that lockfile data be explicitly generated for any combination to be considered valid
• Poetry implements the requirement that all Dependency Groups be mutually compatible, and generates only one locked version. (Meaning it finds a single solution, rather than a set or matrix of solutions.)

If you want to mix and match dependency groups in the same environment that would pose some problems.

[sirosen]

Hi there, PEP 735 author here! 👋

If you want to mix and match dependency groups in the same environment that would pose some problems.

To make sure that this is clear, the spec doesn't say anything to forbid a tool from having tighter constraints on dependency groups. For example, I'm aware that poetry requires that all dependency groups are mutually compatible.

So you don't need to support this if it poses a problem -- but that does put you in a position of having to tell your users what the behavior is (e.g., "it will fail when you try to lock dependencies").

It's possible that some users will misread the spec as saying that "tools are required to support this usage". But it's much more open than that, "tools should think about the fact that the spec doesn't protect against groups conflicting."

I hope that perspective helps!
(Also, I only learned about pixi a few months ago, but it looks very cool. Awesome work!)

[olivier-lacroix]

@tdejager , I think the dependency groups being composable, they should rather be mapped to features, as the current extras are.

[olivier-lacroix]

Logged a similar request against the pyproject-toml crate. Better to handle deserialisation there.

[tdejager]

@sirosen thank you for the reply and kind words :)! Great to see you here!

@olivier-lacroix I see that makes sense! We would still need to think if we want to instantiate some of these features by default in an evironment. Not sure yet!

[olivier-lacroix]

Started by implementing the group recursion in the pyproject-toml crate PyO3/pyproject-toml-rs#26

note: Maybe it would make sense to implement a similar logic to resolve optional extras that include self references

@tdejager , this now means that adding a pypi dependency to a feature when using a pyproject.toml manifest could go to either the optional-dependencies table, or the dependency-groups table. I would propose to:

• default to adding the dependency to a group in the dependency-groups table, as it most naturally maps to pixi's feature concept
• unless an optional dependency group of the same name already exist, in which case we add the dependency there
• add a CLI flag to force adding it as an optional-dependency group. Maybe we should also take that opportunity to allow forcing the use of the tool.pixi.feature table?

We should most likely make sure dependency groups and optional extras have no overlapping names.

Let me know what you think!