Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

No packages inside simple or packages folder #1836

Open
qomhmd opened this issue Dec 22, 2024 · 4 comments
Open

No packages inside simple or packages folder #1836

qomhmd opened this issue Dec 22, 2024 · 4 comments

Comments

@qomhmd
Copy link

qomhmd commented Dec 22, 2024 •
edited
Loading

With this config file:

[mirror]
directory = /srv/pypi
json = true
release-files = true
cleanup = false
master = https://pypi.org
timeout = 10
global-timeout = 1800
workers = 3
hash-index = false
simple-format = ALL
stop-on-error = false
storage-backend = filesystem
verifiers = 3
compare-method = hash

[plugins]
enabled = all

[size_project_metadata]
max_package_size = 1G

[blocklist]
platforms =
        windows
        py2.6
        py2.7

[allowlist]
packages =
        black
        numpy
        pandas

I do not see any packages downloaded. Empty folders are there only:

.
└── pypi
├── generation
├── mirrored-files
├── status
└── web
├── json
│   └── black
├── last-modified
├── local-stats
│   └── days
├── packages
├── pypi
│   └── black
│   └── json
└── simple
├── black
│   ├── index.html
│   ├── index.v1_html
│   └── index.v1_json
├── index.html
├── index.v1_html
└── index.v1_json

image
@flyinghyrax
Copy link
Contributor

Your configuration file looks fine to me, it seems like mostly defaults. 🤔

What command are you running, and what is the log output? It may also help to use the —-debug option to enable more detailed logs.

@qomhmd
Copy link
Author

qomhmd commented Dec 23, 2024

Thanks! I'm running mirror, and with the --debug flag I get:

(.venv) [root@L-66SHLY3 pypi]# bandersnatch --debug mirror
2024-12-23 10:11:42,620 DEBUG: Checking config for storage backend... (configuration.py:125)
2024-12-23 10:11:42,620 DEBUG: Found storage backend in config! (configuration.py:127)
2024-12-23 10:11:42,620 INFO: Selected storage backend: filesystem (configuration.py:133)
2024-12-23 10:11:42,621 DEBUG: Using digest sha256 by default ... (configuration.py:139)
2024-12-23 10:11:42,621 DEBUG: Checking config for compare method... (configuration.py:167)
2024-12-23 10:11:42,621 DEBUG: Found compare method in config! (configuration.py:169)
2024-12-23 10:11:42,621 INFO: Selected compare method: hash (configuration.py:181)
2024-12-23 10:11:42,621 DEBUG: Checking config for alternative download mirror... (configuration.py:184)
2024-12-23 10:11:42,621 DEBUG: No alternative download mirror found in config. (configuration.py:189)
2024-12-23 10:11:42,621 DEBUG: Skip checking download-mirror-no-fallback because dependent optionis not set in config. (configuration.py:209)
2024-12-23 10:11:42,648 DEBUG: Initializing Master's aiohttp ClientSession (master.py:80)
2024-12-23 10:11:42,650 INFO: Initialized project plugin allowlist_project, filtering ['black'] (allowlist_name.py:34)
2024-12-23 10:11:42,650 DEBUG: Project blocklist is [] (blocklist_name.py:62)
2024-12-23 10:11:42,651 INFO: Initialized project plugin blocklist_project, filtering [] (blocklist_name.py:27)
2024-12-23 10:11:42,651 INFO: Initialized project plugin project_requirements, filtering [] (allowlist_name.py:34)
2024-12-23 10:11:42,658 INFO: Initialized metadata plugin size_project_metadata to block projects > 1073741824 bytes; except packages in the allowlist: ['black'] (metadata_filter.py:231)
2024-12-23 10:11:42,659 INFO: Initialized release plugin allowlist_release, filtering [<Requirement('black')>] (allowlist_name.py:181)
2024-12-23 10:11:42,660 INFO: Initialized release plugin blocklist_release, filtering [] (blocklist_name.py:110)
2024-12-23 10:11:42,662 INFO: Initialized prerelease plugin with [re.compile('.+rc\d+$'), re.compile('.+a(lpha)?\d+$'), re.compile('.+b(eta)?\d+$'), re.compile('.+dev\d+$')] (prerelease_name.py:36)
2024-12-23 10:11:42,662 INFO: Initialized prerelease plugin prerelease_release, filtering all packages (prerelease_name.py:48)
2024-12-23 10:11:42,662 INFO: Initialized release plugin project_requirements_pinned, filtering [] (allowlist_name.py:181)
2024-12-23 10:11:42,664 INFO: Initialized exclude_platform plugin with ['.win32', '-win32', 'win_amd64', 'win-amd64', '-cp26-', '-pp26-', '-ip26-', '-jy26-', '-py2.6-', '-py2.6.', '-cp27-', '-pp27-', '-ip27-', '-jy27-', '-py2.7-', '-py2.7.'] (filename_name.py:111)
2024-12-23 10:11:42,665 DEBUG: Adding json directories to bootstrap (mirror.py:495)
2024-12-23 10:11:42,665 DEBUG: Retrieving FileLock instance @ /srv/pypi/.lock (filesystem.py:37)
2024-12-23 10:11:42,665 DEBUG: Acquiring FLock with timeout: 1 (mirror.py:510)
2024-12-23 10:11:42,666 DEBUG: File not changed...deleting temporary file: /srv/pypi/generation.pq_qyhj6 (filesystem.py:120)
2024-12-23 10:11:42,666 INFO: Status file /srv/pypi/status missing. Starting over. (mirror.py:567)
2024-12-23 10:11:42,666 INFO: Syncing with https://pypi.org. (mirror.py:58)
2024-12-23 10:11:42,666 INFO: Current mirror serial: 0 (mirror.py:279)
2024-12-23 10:11:42,666 INFO: Syncing all packages. (mirror.py:294)
2024-12-23 10:11:56,392 INFO: Package 'black' is allowlisted (allowlist_name.py:91)
2024-12-23 10:11:56,464 INFO: Trying to reach serial: 26644022 (mirror.py:311)
2024-12-23 10:11:56,465 INFO: 1 packages to sync. (mirror.py:313)
2024-12-23 10:11:56,474 DEBUG: Package syncer 0 started for duty (mirror.py:126)
2024-12-23 10:11:56,474 INFO: Fetching metadata for package: black (serial 25362798) (package.py:58)
2024-12-23 10:11:56,474 DEBUG: Getting /pypi/black/json (serial 25362798) (master.py:127)
2024-12-23 10:11:56,475 DEBUG: Package syncer 1 started for duty (mirror.py:126)
2024-12-23 10:11:56,475 DEBUG: Package syncer 1 emptied queue (mirror.py:133)
2024-12-23 10:11:56,475 DEBUG: Package syncer 2 started for duty (mirror.py:126)
2024-12-23 10:11:56,475 DEBUG: Package syncer 2 emptied queue (mirror.py:133)
2024-12-23 10:11:56,487 INFO: Package 'black' is allowlisted (allowlist_name.py:91)
2024-12-23 10:11:56,505 DEBUG: Writing temporary file /srv/pypi/web/json/.black.lnd5irh3 to target destination: /srv/pypi/web/json/black (filesystem.py:94)
2024-12-23 10:11:56,510 DEBUG: MATCH: Release black==18.3a0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,510 DEBUG: MATCH: Release black==18.3a1 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,510 DEBUG: MATCH: Release black==18.3a2 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,510 DEBUG: MATCH: Release black==18.3a3 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,510 DEBUG: MATCH: Release black==18.3a4 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,510 DEBUG: MATCH: Release black==18.4a0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,510 DEBUG: MATCH: Release black==18.4a1 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,510 DEBUG: MATCH: Release black==18.4a2 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,510 DEBUG: MATCH: Release black==18.4a3 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,510 DEBUG: MATCH: Release black==18.4a4 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,510 DEBUG: MATCH: Release black==18.5b0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,510 DEBUG: MATCH: Release black==18.5b1 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,510 DEBUG: MATCH: Release black==18.6b0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,510 DEBUG: MATCH: Release black==18.6b1 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,510 DEBUG: MATCH: Release black==18.6b2 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,510 DEBUG: MATCH: Release black==18.6b3 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,510 DEBUG: MATCH: Release black==18.6b4 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,510 DEBUG: MATCH: Release black==18.9b0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,511 DEBUG: MATCH: Release black==19.10b0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,511 DEBUG: MATCH: Release black==19.3b0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,511 DEBUG: MATCH: Release black==20.8b0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,511 DEBUG: MATCH: Release black==20.8b1 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,511 DEBUG: MATCH: Release black==21.10b0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,511 DEBUG: MATCH: Release black==21.11b0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,511 DEBUG: MATCH: Release black==21.11b1 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,511 DEBUG: MATCH: Release black==21.12b0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,511 DEBUG: MATCH: Release black==21.4b0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,511 DEBUG: MATCH: Release black==21.4b1 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,511 DEBUG: MATCH: Release black==21.4b2 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,511 DEBUG: MATCH: Release black==21.5b0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,511 DEBUG: MATCH: Release black==21.5b1 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,511 DEBUG: MATCH: Release black==21.5b2 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,511 DEBUG: MATCH: Release black==21.6b0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,511 DEBUG: MATCH: Release black==21.7b0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,511 DEBUG: MATCH: Release black==21.8b0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,511 DEBUG: MATCH: Release black==21.9b0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,511 DEBUG: MATCH: Release black==22.1.0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,511 DEBUG: MATCH: Release black==22.10.0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,512 DEBUG: MATCH: Release black==22.12.0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,512 DEBUG: MATCH: Release black==22.3.0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,512 DEBUG: MATCH: Release black==22.6.0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,512 DEBUG: MATCH: Release black==22.8.0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,512 DEBUG: MATCH: Release black==23.1.0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,512 DEBUG: MATCH: Release black==23.10.0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,512 DEBUG: MATCH: Release black==23.10.1 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,512 DEBUG: MATCH: Release black==23.11.0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,512 DEBUG: MATCH: Release black==23.12.0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,512 DEBUG: MATCH: Release black==23.12.1 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,512 DEBUG: MATCH: Release black==23.1a1 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,512 DEBUG: MATCH: Release black==23.3.0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,512 DEBUG: MATCH: Release black==23.7.0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,512 DEBUG: MATCH: Release black==23.9.0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,512 DEBUG: MATCH: Release black==23.9.1 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,512 DEBUG: MATCH: Release black==24.1.0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,512 DEBUG: MATCH: Release black==24.1.1 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,512 DEBUG: MATCH: Release black==24.10.0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,512 DEBUG: MATCH: Release black==24.1a1 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,513 DEBUG: MATCH: Release black==24.2.0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,513 DEBUG: MATCH: Release black==24.3.0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,513 DEBUG: MATCH: Release black==24.4.0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,513 DEBUG: MATCH: Release black==24.4.1 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,513 DEBUG: MATCH: Release black==24.4.2 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,513 DEBUG: MATCH: Release black==24.8.0 matches specifier (allowlist_name.py:252)
2024-12-23 10:11:56,513 INFO: Storing index page(s): black - in /srv/pypi/web/simple/black (mirror.py:699)
2024-12-23 10:11:56,513 DEBUG: There are 0 releases for black (simple.py:184)
2024-12-23 10:11:56,513 DEBUG: Generated simple HTML format for black (simple.py:251)
2024-12-23 10:11:56,513 DEBUG: Generated simple JSON format for black (simple.py:254)
2024-12-23 10:11:56,513 DEBUG: Attempting to write PEP691 simple pages for black (mirror.py:714)
2024-12-23 10:11:56,514 DEBUG: Writing temporary file /srv/pypi/web/simple/black/.index.html.lndkwbmc to target destination: /srv/pypi/web/simple/black/index.html (filesystem.py:94)
2024-12-23 10:11:56,515 DEBUG: Writing temporary file /srv/pypi/web/simple/black/.index.v1_html.vb4pn4bz to target destination: /srv/pypi/web/simple/black/index.v1_html (filesystem.py:94)
2024-12-23 10:11:56,515 DEBUG: Writing temporary file /srv/pypi/web/simple/black/.index.v1_json.jeydg_go to target destination: /srv/pypi/web/simple/black/index.v1_json (filesystem.py:94)
2024-12-23 10:11:56,516 DEBUG: Modifying destination: /srv/pypi/todo with: /srv/pypi/todo.9y7qg1db (filesystem.py:123)
2024-12-23 10:11:56,517 DEBUG: Running Non PEP503 path cleanup for black (mirror.py:444)
2024-12-23 10:11:56,517 DEBUG: Package syncer 0 emptied queue (mirror.py:133)
2024-12-23 10:11:56,517 INFO: Generating global index page. (simple.py:264)
2024-12-23 10:11:56,517 DEBUG: Writing temporary file /srv/pypi/web/simple/.index.html.6ukq7hqx to target destination: /srv/pypi/web/simple/index.html (filesystem.py:94)
2024-12-23 10:11:56,518 DEBUG: Writing temporary file /srv/pypi/web/simple/.index.v1_json.fw3djuhi to target destination: /srv/pypi/web/simple/index.v1_json (filesystem.py:94)
2024-12-23 10:11:56,518 INFO: New mirror serial: 26644022 (mirror.py:473)
2024-12-23 10:11:56,519 DEBUG: Modifying destination: /srv/pypi/web/last-modified with: /srv/pypi/web/last-modified._6qnjn0y (filesystem.py:123)
2024-12-23 10:11:56,519 DEBUG: Modifying destination: /srv/pypi/status with: /srv/pypi/status.k4e7jphu (filesystem.py:123)
2024-12-23 10:11:56,519 DEBUG: Closing Master's aiohttp ClientSession and waiting 0.1 seconds (master.py:100)
2024-12-23 10:11:56,621 INFO: 1 packages had changes (mirror.py:991)
2024-12-23 10:11:56,621 DEBUG: black added: [] (mirror.py:998)
2024-12-23 10:11:56,621 INFO: Writing diff file to mirrored-files (mirror.py:1001)

I made some parts boldface which I think are contradictory and the second one shows the problem? 🤔

@flyinghyrax
Copy link
Contributor

Thank you for following up! I agree; the allowlist_release plugin is matching and the allowlist_project seems to be also. Then something is filtering out all the releases. 🤔

Unfortunately right now not all the filter plugins have as much debug logging as allowlist_release, so my guess is another plugin is rejecting the releases but not logging that.

What I would try first is to enable only selected plugins:

[plugins]
enabled =
    allowlist_project
    exclude_platform
    prerelease_release
    size_project_metadata

This will rule out other plugins that were using their default options. You can also start with just the allowlist_release plugin and enable the others one at a time, which should help identify which one is doing the filtering.

@qomhmd
Copy link
Author

qomhmd commented Dec 25, 2024 

[plugins]
enabled =
    allowlist_project
    exclude_platform
    prerelease_release
    size_project_metadata

This worked! Thanks a lot.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@flyinghyrax @qomhmd