Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Shared external list of path sources for use as an "index" analogue #10098

Open
smackesey opened this issue Dec 22, 2024 · 0 comments
Open

Shared external list of path sources for use as an "index" analogue #10098

smackesey opened this issue Dec 22, 2024 · 0 comments
Labels
configuration Settings and such enhancement New feature or improvement to existing functionality

Comments

@smackesey
Copy link

smackesey commented Dec 22, 2024

I'd like the ability to provide a list of sources in a file outside the package pyproject.toml to uv pip install. The reason the sources should be in an external file is that these sources will contain paths for all the packages in a monorepo, and the same file will be used whenever a package is installed in a test environment to ensure dependencies within the monorepo are resolved within the checked-out commit of the monorepo.

Details

In monorepo development, often one has many interdependent packages in the same repo. Let's call a dependency across packages within the monorepo an "internal dependency".

When testing, we'd typically like to resolve internal dependencies against the checked out version of the code.

Consider a toy version of the much more complicated scenario we have in Dagster. Suppose we have three packages in the monorepo: dagster-foo dagster-bar, and dagster-baz. dagster-foo depends on dagster-bar, and dagster-bar depends on dagster-baz:

### dagster-foo/pyproject.toml

[project]
...
name = "dagster-foo"
dependencies = ["dagster-bar"]
### dagster-bar/pyproject.toml

[project]
...
name = "dagster-bar"
dependencies = ["dagster-baz"]

When I install dagster-foo into a test environment from a local path, I want to resolve both it's first order dep dagster-bar and second-order dep dagster-baz to the local versions within the monorepo.

One way to do this is to use [tool.uv.sources]. For that to work, we'd have to do this:

### dagster-foo/pyproject.toml

[project]
...
name = "dagster-foo",
dependencies = [
  "dagster-bar",
  "dagster-baz",  # have to include this now because tool.uv.sources won't work for 2nd-order deps
]

[tool.uv.sources]
dagster-bar = { path = "/path/to/dagster/python_modules/dagster-bar", editable = true }
dagster-baz = { path = "/path/to/dagster/python_modules/dagster-baz", editable = true }

This is not ideal for a few reasons:

  • I have to list indirect dependencies under project.dependencies to make them work with uv.tool.sources
  • I need to manage the [tool.uv.sources] separately for every dagster-* package.

What would be better is if I could somehow maintain a single list of sources in some external file and link that file when installing any dagster-* into a test environment. This would effectively function like an "index" that shadows PyPI:

### SOURCES.toml

dagster-foo = { path = "/path/to/dagster/python_modules/dagster-foo", editable = true }
dagster-bar = { path = "/path/to/dagster/python_modules/dagster-bar", editable = true }
dagster-baz = { path = "/path/to/dagster/python_modules/dagster-baz", editable = true }
# ... any other dagster-* packages

Then when installing dagster-foo, I would not have any tool.uv.sources listed in its pyproject.toml, but would just do:

uv pip install -e path/to/dagster-foo --sources SOURCES.toml

# analogous to the below command, _if_ we had a private index https://dagster.monorepo.index that somehow contained the 
# local versions of all dagster-* packages (which we don't)
uv pip install -e path/to/dagster-foo --index https://dagster.monorepo.index

Is this functionality already supported through some pattern I haven't thought of? If not, consider this a feature request.

@charliermarsh charliermarsh added enhancement New feature or improvement to existing functionality configuration Settings and such labels Dec 26, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
configuration Settings and such enhancement New feature or improvement to existing functionality
Projects
None yet
Development

No branches or pull requests

2 participants