Skip to content

Drop Python 3.9, add Python 3.14 - #720

Open
rtibblesbot wants to merge 4 commits into
learningequality:mainfrom
rtibblesbot:issue-719-621d1e
Open

Drop Python 3.9, add Python 3.14#720
rtibblesbot wants to merge 4 commits into
learningequality:mainfrom
rtibblesbot:issue-719-621d1e

Conversation

@rtibblesbot

@rtibblesbot rtibblesbot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Summary

Python 3.9 reached end-of-life in October 2025 and Python 3.14 is a supported release the project did not test. This raises the supported range to 3.10–3.14 across packaging metadata, the CI matrix, the runtime version guard, ruff's target-version, and the install/contributor docs, and removes a dead Python 2/3 compatibility shim in thumbscropping.py that only ever took its Python 3 branch.

Testing 3.14 surfaced a blocker: CPython 3.14 on Windows links zlib-ng, whose DEFLATE output differs from reference zlib, so create_predictable_zip would emit archives that miss the MD5s Studio has pinned. ricecooker.utils.zip now detects zlib-ng — via zlib.ZLIBNG_VERSION where available, falling back to ZLIB_RUNTIME_VERSION on older interpreters — and raises rather than producing unmatchable hashes.

References

Closes #719.

Reviewer guidance

  • windows-latest, 3.14 is excluded from the matrix: that build links zlib-ng, whose DEFLATE bytes cannot reproduce the pinned archive hashes. Windows is therefore supported up to 3.13, and docs/installation.md says so.
  • The zlib-ng guard is deliberately broader than the Windows 3.14 case — any interpreter reporting zlib-ng is refused, since the hash mismatch is a property of the zlib build, not of the platform.
  • .github/workflows/pythontest.yml:67 — the macOS python@3.13 unlink/link steps are untouched. Check the 3.14 job's "link 3.13" branch is acceptable.

AI usage

Used Claude Code to apply the pre-approved plan. Verified with the full test suite, the thumbnail regression tests, and prek/ruff.


@rtibblesbot's comments are generated by an LLM, and should be evaluated accordingly

How was this generated?
  • Ran pre-flight CI checks (lint, format, tests) and verified all pass
  • Rebased onto the target branch and resolved any conflicts
  • Reorganized commit history into clean, logical commits
  • Audited the diff to ensure only issue-relevant files are changed
  • Built PR body from the repository's PR template with evidence blocks

Deviations from the issue spec

  • The CI matrix does not run windows-latest, 3.14 — that interpreter links zlib-ng and cannot reproduce the predictable-zip hashes Studio has pinned, so the AC "CI matrix runs 3.10–3.14" holds on Linux and macOS only.
  • Added the zlib-ng guard in ricecooker/utils/zip.py plus its tests, and the Windows note in docs/installation.md — not in the issue, but shipping 3.14 without it would let chefs silently produce archives Studio rejects.
  • Bumped ruff target-version to py310 alongside requires-python; inert under the current rule set, but leaving it at py39 would misstate the floor.
@rtibblesbot

🟡 Waiting for feedback

Last updated: 2026-07-26 19:22 UTC

@rtibblesbot

Copy link
Copy Markdown
Contributor Author

CI signal: Python 3.14 exposes a latent archive-hashing bug (not caused by this diff)

The windows-latest, 3.14 job fails two tests — test_download_filenames and test_download_to_storage — both asserting a hardcoded MD5 of a re-zipped ePub. ubuntu-latest, 3.14 and macos-latest, 3.14 pass, because their setup-python builds still link stock zlib while the Windows 3.14 build links zlib-ng.

Root cause: Python 3.14 ships zlib-ng in some builds (zlib.ZLIB_VERSION == '1.3.1.zlib-ng'). create_predictable_zip (ricecooker/utils/zip.py:45) hashes the DEFLATE-compressed archive bytes. Those bytes differ between stock zlib and zlib-ng for identical input, so the expected fb987dae… becomes 21720f91… (Windows runner) or 879d1101… (a local Linux 3.14 with zlib-ng). Content bytes are identical — only the compressed stream differs.

Implication beyond the test: archive hashes (epub, h5p, HTML zips) drive Studio dedup. On a zlib-ng interpreter the same content produces a different hash. create_predictable_zip is therefore not predictable across zlib implementations — a real behavior change, not a test artifact.

Scope decision needed. A fix means either compression-independent hashing (e.g. ZIP_STORED, which changes every existing hash — a breaking dedup migration) or reworking the hash to cover uncompressed content. Both sit well outside "drop 3.9 / add 3.14". Options:

  1. Leave 3.14 in CI red as the documented signal; fix hashing in a dedicated follow-up.
  2. xfail the two hash-assertion tests on zlib-ng builds (test-only) to unblock 3.14 CI, with the follow-up tracked.
  3. Fix production hashing here (largest scope, breaking).

@rtibblesbot
rtibblesbot marked this pull request as ready for review July 24, 2026 18:03

@rtibbles rtibbles left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The predictable zipping is pinned not just in Python code in ricecooker but also in Javascript in Studio. This cannot be changed, and we must use regular zlib not zlib-ng, unless zlib-ng can be compelled to produce byte identical output under a specific mode. The current hashes are the acceptance test, and no other outcome will be satisfactory.

@rtibblesbot

Copy link
Copy Markdown
Contributor Author

Checked whether zlib-ng can be compelled to match. It cannot.

Evidence. On zlib-ng 2.2.5 vs zlib 1.3.1, I swept every knob CPython's zlib exposes: levels 0-9 and -1 × strategies DEFAULT/FILTERED/HUFFMAN_ONLY/RLE/FIXED × memLevel 1-9, 495 combinations. Zero reproduce reference-zlib default output. Sizes differ at every level (level 6: 198430 vs 198458 bytes), so it is the match-finding heuristics, not a tunable parameter. zlib-ng exposes no runtime compatibility mode. CPython does not expose deflateTune.

So reference zlib is the only implementation that can produce the pinned hashes.

What changed (6733c3d):

  • create_predictable_zip refuses to run on a zlib-ng interpreter, detected via zlib.ZLIBNG_VERSION (added in 3.14). It previously would have emitted archives whose MD5s don't match Studio. New test in tests/test_zip.py.
  • windows-latest × 3.14 is excluded from the CI matrix. CPython 3.14 on Windows links zlib-ng.
  • docs/installation.md states 3.10-3.13 on Windows, with the reason.

Every other 3.14 job builds against reference zlib. The full suite also passes locally on a 3.14 build with stock zlib 1.3.2, so the hashes are stable across zlib versions — just not across implementations.

Consequence to flag: this drops Windows support at 3.14 and above, since every official CPython Windows build from 3.14 on links zlib-ng. Restoring it would mean ricecooker shipping its own reference-zlib DEFLATE encoder.

@rtibblesbot

rtibblesbot commented Jul 24, 2026

Copy link
Copy Markdown
Contributor Author

Rebased onto main (picks up #714, which added find_common_root/find_html_entrypoint to ricecooker/utils/zip.py).

The only conflict was in zip.py — both sides added new top-level functions at the same spot. Purely additive, resolved by keeping both; no logic on either side changed.

Full matrix green, including ubuntu/macos on 3.14. No new review feedback outstanding — the zlib-ng question is answered in #720 (comment).


@rtibblesbot's comments are generated by an LLM, and should be evaluated accordingly

rtibblesbot and others added 4 commits July 26, 2026 12:16
Raise the supported floor to 3.10 and the ceiling to 3.14 across
packaging metadata, CI, the runtime guard, and user-facing docs.
Also remove a dead Python 2/3 shim in thumbscropping.py.

- pyproject.toml: requires-python ">=3.10, <3.15"; classifiers 3.10-3.14
- pythontest.yml: matrix ['3.10'..'3.14']
- ricecooker/__init__.py: guard rejects < (3, 10, 0)
- utils/thumbscropping.py: drop PY2/PY3 branch, inline str checks
- docs/installation.md, CONTRIBUTING.md: name 3.10-3.14

Closes learningequality#719

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
zlib-ng emits different DEFLATE bytes than reference zlib for the same
input, so archives built on such an interpreter get different MD5s. Those
hashes are pinned here and in Studio's JavaScript and are already in use
for deduplication, so they cannot change.

No zlib-ng configuration reproduces reference zlib output (checked levels
0-9 and -1 x 5 strategies x memLevel 1-9), so detect it and refuse rather
than silently emit unmatchable archives. CPython 3.14 on Windows links
zlib-ng, so drop that combination from the CI matrix and the install docs.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Left over from the 3.9 floor. Inert with the current rule set (E, F, W,
C90, I) and produces no formatting change, but stale in a change that
drops 3.9.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
zlib.ZLIBNG_VERSION only exists from CPython 3.14 onwards, so a 3.10-3.13
interpreter linked against zlib-ng passed the guard and would have emitted
archives whose MD5s cannot match the pinned hashes. zlib-ng's zlib-compat
mode reports its version as "1.3.1.zlib-ng", which reaches us through
ZLIB_RUNTIME_VERSION on every Python version; check that too.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@rtibblesbot

Copy link
Copy Markdown
Contributor Author

Rebased onto main (picks up #716). The only conflict was uv.lock; resolved by taking main's lockfile and re-running uv lock under the new requires-python = ">=3.10, <3.15". That drops the Python-3.9-only forked pins (cattrs 25.3.0, iniconfig 2.1.0, platformdirs 4.4.0, pycparser 2.23, yt-dlp 2025.10.14) and keeps main's new tomli>=2.0.0; python_version < "3.11" dependency. No source changes on either side of the conflict; the uv-lock prek hook confirms the lock matches pyproject.toml.

Locally on 3.14 (reference zlib): 525 passed, 12 skipped; uvx prek run --all-files green.

No new review feedback outstanding — the zlib-ng question is answered in #720 (comment).


@rtibblesbot's comments are generated by an LLM, and should be evaluated accordingly

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Drop Python 3.9 support and add Python 3.14 support

2 participants