Very slow builds on CI machine

[buhrmann]

Hi,
I'm curious about great differences in build times for a package of mine (using conda mambabuild), between my local machine (macOS Arm M1), and a CI machine.

The greatest time seems to be spend on "Reloading output folder" steps. E.g. locally:

...
17:39:42 > #12 0.973 Attempting to finalize metadata for grapy-core
17:39:42 > #12 0.973 INFO:conda_build.metadata:Attempting to finalize metadata for grapy-core
17:39:48 > #12 6.455 Reloading output folder: /opt/conda/conda-bld
17:39:48 > #12 6.497 Reloading output folder: /opt/conda/conda-bld
17:40:30 > #12 48.90 Reloading output folder: /opt/conda/conda-bld
17:40:30 > #12 49.11 BUILD START: ['grapy-core-0.1.0-py_16.tar.bz2']
17:40:30 > #12 49.11 Reloading output folder: /opt/conda/conda-bld
17:40:32 > #12 50.76
17:40:32 > #12 50.76 ## Package Plan ##
...

where 42s of the total build time of 2:40s is spend on a single reload step.

And In CI it's much worse:

Here, the same reload step takes a whopping 166s, i.e. almost 3 min. And the total build takes more than 7 min, instead of the 2:40s locally.

Do you have any experience with such differences depending on the machine/environment?

[jpfeuffer]

We have exactly the same problem. I am not sure if it's really stuck at the "Reloading output folder" step or just not reporting what it is doing anymore.
I am pretty desperate at this point.
Running with --debug does not help either.
Just no output at all. Until suddenly after hours it splurges all the output and the conda build summary tells me that the "actual build" took a few seconds...

What is this?

[jpfeuffer]

For reference, this is one of the runs with debug:
https://github.com/bigbio/quantms-utils/actions/runs/11929175706/job/33247451809

[jpfeuffer]

Even conda build is faster than boa 😂