DO_NOT_MERGE : Cherry-pick PR403#415
Conversation
Signed-off-by: arekay-nv <230885705+arekay-nv@users.noreply.github.com>
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces provisioning and staging support for the VBench (WAN 2.2) accuracy scorer, including setting up an isolated Python 3.11 environment in the development Dockerfile, handling PyTorch 2.6+ weight loading restrictions, and ensuring staged videos are renamed to .mp4 to prevent VBench errors. Feedback on these changes suggests installing the required wget and unzip tools directly in the Dockerfile, committing the pyproject.toml compatibility patches to the repository instead of using sed dynamically, using a distinct exit code for missing system dependencies to avoid conflicts, and unconditionally setting the PyTorch weight-loading environment variable to prevent external overrides.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| # reader, needs libGL.so.1 / libgthread at import time (only used when PROVISION_VBENCH=1). | ||
| RUN apt-get update && \ | ||
| apt-get install -y --no-install-recommends build-essential procps git curl ca-certificates \ | ||
| libgl1 libglib2.0-0 \ |
There was a problem hiding this comment.
The vbench_runner.py script checks for the presence of wget and unzip to download model weights on a cold cache. However, these tools are not installed in the development Docker image, which will cause the evaluation to fail. Adding them to the apt-get install command ensures they are available.
libgl1 libglib2.0-0 wget unzip \
| RUN if [ "${PROVISION_VBENCH}" = "1" ]; then \ | ||
| cd /opt/vbench_accuracy && \ | ||
| sed -i 's#requires-python = ">=3.12"#requires-python = ">=3.10"#' pyproject.toml && \ | ||
| sed -i 's# "vbench==0.1.5",# "vbench==0.1.5",\n "decord2>=3.4.0",#' pyproject.toml && \ | ||
| sed -i "s#^package = false#package = false\noverride-dependencies = [\"decord; sys_platform == 'never'\"]#" pyproject.toml && \ | ||
| UV_PROJECT_ENVIRONMENT="$(pwd)/.venv" uv lock --python 3.11 && \ | ||
| UV_PROJECT_ENVIRONMENT="$(pwd)/.venv" uv sync --python 3.11 && \ | ||
| UV_PROJECT_ENVIRONMENT="$(pwd)/.venv" uv run python -c "import torch, decord, tokenizers, vbench; print('vbench', vbench.__file__, '| decord', decord.__version__, '| tokenizers', tokenizers.__version__, '| torch', torch.__version__)" ; \ | ||
| else \ | ||
| echo "PROVISION_VBENCH=${PROVISION_VBENCH}: skipping VBench provisioning" ; \ | ||
| fi |
There was a problem hiding this comment.
Using sed to dynamically patch pyproject.toml during the Docker build is fragile and hard to maintain. If the formatting or version pins in pyproject.toml change in the future, these sed commands will silently fail or break the build.
Instead, you should commit these compatibility settings directly to examples/09_Wan22_VideoGen_Example/accuracy/pyproject.toml in this PR:
- Change
requires-pythonto">=3.10". - Add
"decord2>=3.4.0"todependencies. - Add
override-dependencies = ["decord; sys_platform == 'never'"]under[tool.uv].
This allows you to simplify the Dockerfile and avoid dynamic patching.
RUN if [ "${PROVISION_VBENCH}" = "1" ]; then \
cd /opt/vbench_accuracy && \
UV_PROJECT_ENVIRONMENT="$(pwd)/.venv" uv lock --python 3.11 && \
UV_PROJECT_ENVIRONMENT="$(pwd)/.venv" uv sync --python 3.11 && \
UV_PROJECT_ENVIRONMENT="$(pwd)/.venv" uv run python -c "import torch, decord, tokenizers, vbench; print('vbench', vbench.__file__, '| decord', decord.__version__, '| tokenizers', tokenizers.__version__, '| torch', torch.__version__)" ; \
else \
echo "PROVISION_VBENCH=${PROVISION_VBENCH}: skipping VBench provisioning" ; \
fi
| file=sys.stderr, | ||
| flush=True, | ||
| ) | ||
| return 2 |
There was a problem hiding this comment.
Returning 2 for missing system dependencies conflicts with the documented exit code 2 (which is reserved for CUDA required but unavailable in the file's header docstring). Using a distinct exit code like 3 avoids this ambiguity.
| return 2 | |
| return 3 |
| # weights_only=True default. This escape hatch is read at torch.load call time, | ||
| # only applies when the callsite does not set weights_only explicitly, and is | ||
| # scoped to this subprocess. | ||
| os.environ.setdefault("TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD", "1") |
There was a problem hiding this comment.
Since this subprocess is dedicated to running VBench, and VBench's reference checkpoints must load full pickles, we should unconditionally set TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD to "1". Using setdefault would allow an inherited environment variable (if set to "0") to override this, which would cause VBench to crash under PyTorch 2.6+.
| os.environ.setdefault("TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD", "1") | |
| os.environ["TORCH_FORCE_NO_WEIGHTS_ONLY_LOAD"] = "1" |
1 participant
Cherry pick 403 for container build.
Type of change
Related issues
Testing
Checklist