Replace Python eggs with a universal wheel - #6341
Conversation
|
Important Review skippedDraft detected. Please check the settings in the CodeRabbit UI or the ⚙️ Run configurationConfiguration used: Organization UI Review profile: CHILL Plan: Pro Plus Run ID: You can disable this status message by setting the Use the checkbox below for a quick retry:
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
7f569fd to
4eade76
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #6341 +/- ##
==========================================
+ Coverage 70.60% 70.70% +0.10%
==========================================
Files 207 208 +1
Lines 23637 23782 +145
==========================================
+ Hits 16688 16816 +128
- Misses 6949 6966 +17 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
229ada7 to
d320180
Compare
Python eggs are CPython-minor specific, need pkg_resources, and already fail inside Fedora 39+ / Python 3.12 containers. Avocado is pure Python, so one py3-none-any wheel is enough: unpack it on the host and put that directory on PYTHONPATH. When running from a source tree the helper builds the wheel with pip; otherwise it fetches the GitHub release asset. Eggs remain as a deprecated zipimport fallback. Reference: avocado-framework#6108 Reference: avocado-framework#6115 Signed-off-by: Harvey Lynden <harveylynden@gmail.com>
Stop copying per-CPython eggs plus setuptools 59.2 into every container. Unpack one wheel on the host and bind-mount it at /opt/avocado-wheel so nrunner can still start with python -m avocado.plugins.runners... task-run. --spawner-podman-avocado-wheel selects the package; --spawner-podman-avocado-egg is kept as a deprecated alias. Reference: avocado-framework#6108 Reference: avocado-framework#6115 Signed-off-by: Harvey Lynden <harveylynden@gmail.com>
Unit tests build a py3-none-any wheel, unpack it, and prove import avocado plus python -m avocado.plugins.runners.exec_test work from PYTHONPATH. Functional Podman jobs now target fedora:latest so they keep exercising current CPython. Reference: avocado-framework#6115 Signed-off-by: Harvey Lynden <harveylynden@gmail.com>
Record the egg-to-wheel switch for the next release notes, the nrunner reference, and the man page. Point users at --spawner-podman-avocado-wheel. Reference: avocado-framework#6108 Signed-off-by: Harvey Lynden <harveylynden@gmail.com>
bdist_egg plus fedora:40 cannot prove the bootstrap still works on current images. Build a universal wheel and pass it with --spawner-podman-avocado-wheel against fedora:latest. Reference: avocado-framework#6115 Signed-off-by: Harvey Lynden <harveylynden@gmail.com>
avocado-fetch-eggs.py pulled one egg per CPython minor plus setuptools 59.2. Add avocado-fetch-wheels.py for the single py3-none-any wheel, and warn on the old script. Reference: avocado-framework#6108 Signed-off-by: Harvey Lynden <harveylynden@gmail.com>
Gate 3 on a Fedora 44 VM showed fedora:41+ default container images no longer include python3, so nrunner cannot start. fedora:40 still has Python 3.12, the first interpreter where eggs already failed. Reference: avocado-framework#6115 Signed-off-by: Harvey Lynden <harveylynden@gmail.com>
Avocado 113.0 GitHub releases only attached eggs. pip-installed Avocado therefore could not bootstrap Podman until we also try PyPI. Snapshot the source tree before pip wheel so read-only mounts and parallel selftests still build. Copy README.rst as a regular file because it is a symlink into docs/. Reference: avocado-framework#6108 Signed-off-by: Harvey Lynden <harveylynden@gmail.com>
Drop the per-interpreter egg-build matrix and Makefile.gh bdist_egg. The release pipeline uploads py3-none-any wheels to the GitHub release instead of eggs. Reference: avocado-framework#6108 Signed-off-by: Harvey Lynden <harveylynden@gmail.com>
Avocado core no longer imports pkg_resources. Document that releases publish wheels, not eggs, and that docs can use current setuptools. Reference: avocado-framework#6108 Signed-off-by: Harvey Lynden <harveylynden@gmail.com>
Stop downloading per-CPython eggs and setuptools 59.2. The old script warns and uses the same resolver as avocado-fetch-wheels. Reference: avocado-framework#6108 Signed-off-by: Harvey Lynden <harveylynden@gmail.com>
d320180 to
fc67554
Compare
|
Hi @harvey0100, I can see the various rpm-builds failing and the GH action with the podman spawner even errors entirely. I assume I should take a look at these once the CI passes and is marked as non-draft. |
PraveenPenguin
left a comment
There was a problem hiding this comment.
@harvey0100 thanks for the patch .. have review comment see if those make sense
| shutil.rmtree(dest) | ||
| dest.mkdir(parents=True) | ||
| with zipfile.ZipFile(wheel_path) as archive: | ||
| archive.extractall(dest) |
There was a problem hiding this comment.
zipfile.ZipFile.extractall(dest) is called with no path-traversal guard. can we have like
with zipfile.ZipFile(wheel_path) as archive:
dest_real = os.path.realpath(dest)
for member in archive.namelist():
member_path = os.path.realpath(
os.path.join(dest_real, member)
)
if not member_path.startswith(dest_real + os.sep):
raise RuntimeError(
f"Wheel contains path-traversal entry: {member!r}"
)
archive.extractall(dest)
| if os.path.isdir(package): | ||
| host_dir = os.path.abspath(package) | ||
| else: | ||
| cache_root = ( |
There was a problem hiding this comment.
prepare_bootstrap() constructs its own cache_root expression here using cache_dirs[0] if cache_dirs else Path.cwd(). This duplicates the same conditional already resolved inside resolve_wheel_file().
when cache_dirs is None (the default), the guard if cache_dirs else treats None the same as an empty list — but resolve_wheel_file normalises None to [] first, so the two computations are not equivalent if callers pass None. The extracted site directory therefore ends up in a different location than the cached wheel, breaking the extraction marker cache across calls.
thinking can we do this way
if os.path.isdir(package):
host_dir = os.path.abspath(package)
else:
_cache_dirs = cache_dirs or []
_cache_root = (
Path(_cache_dirs[0] if _cache_dirs else Path.cwd())
/ "wheel-bootstrap"
)
host_dir = _extract_wheel(package, _cache_root / "site")
| def _bootstrap_url(self): | ||
| return self.config.get( | ||
| "spawner.podman.avocado_spawner_wheel" | ||
| ) or self.config.get("spawner.podman.avocado_spawner_egg") |
There was a problem hiding this comment.
can we deprecation path as well something like
def _bootstrap_url(self):
wheel = self.config.get("spawner.podman.avocado_spawner_wheel")
if wheel:
return wheel
egg = self.config.get("spawner.podman.avocado_spawner_egg")
if egg:
LOG.warning(
"--spawner-podman-avocado-egg is deprecated; "
"use --spawner-podman-avocado-wheel instead."
)
return egg
return None
Fixes #6108
Fixes #6115
nrunner
--spawner=podmandeployed Avocado with per-CPython eggs plus setuptools 59.2 onPYTHONPATH. That fails on Fedora 39+ / Python 3.12+ (hang /ModuleNotFoundError: avocado). Eggs are a discontinued format; GitHub 113.0 still attached eggs and no wheel.This switches isolated spawners to one
avocado_framework-{ver}-py3-none-any.whl: unpack on the host, bind-mount/opt/avocado-wheel:ro,z, keeppython -m avocado.plugins.runners… task-run. Not a venv inside the container. Notavocado runas the entrypoint.Resolver: explicit URL →
pip wheelfrom the git tree → GitHub release wheel → PyPI. CI/release stopbdist_eggand attach the wheel to GitHub releases. Docs drop thesetuptools<82pin.--spawner-podman-avocado-eggstays as a deprecated alias for one release.The container image must still provide
python3. Fedora 41+ default images do not; selftests usefedora:40.fedora-toolboxstill has an interpreter.Not in this PR:
setup.py→pyproject.toml/ CIsetuptools<80(#5962), remaining distutils (#5159), changing the default Podman image tofedora-toolbox.Test plan
selftests.unit.plugin.wheel_bootstrappasstest.py+/bin/trueonfedora:40(Python 3.12)fedora-toolbox:44(Python 3.14)pip wheelwhen the tree is on a read-only mountavocado run --spawner=podman --spawner-podman-image=fedora:40 examples/tests/passtest.pyMade with Cursor