Skip to content

[CI] Parity: preserve partial reports on download failure #3557

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ethanwee1 wants to merge 1 commit into
base: ethanwee/parity-preview-replacement
Choose a base branch
from
Open

[CI] Parity: preserve partial reports on download failure #3557

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 26 additions & 0 deletions .github/workflows/parity.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -154,6 +154,7 @@ jobs:
run: pip install -r requirements.txt

- name: Download artifacts
id: download
working-directory: .automation_scripts/pytorch-unit-test-scripts
env:
GITHUB_TOKEN: ${{ secrets.PARITY_GITHUB_TOKEN || github.token }}
Expand Down Expand Up @@ -200,7 +201,20 @@ jobs:
fi

echo "Running: python3 ./download_testlogs $ARGS"
# Don't abort the job here on a partial download (e.g. a config whose
# upstream job was skipped -- ROCm distributed during GPU capacity
# crunches). Capture download_testlogs' real exit code, keep going so
# the report still generates and uploads from whatever downloaded,
# then fail the job at the end (see "Flag incomplete download") so the
# run is still clearly marked failed.
set +e
python3 ./download_testlogs $ARGS 2>&1 | tee download_${{ matrix.arch }}.log
DL_RC=${PIPESTATUS[0]}
set -e
echo "download_rc=$DL_RC" >> "$GITHUB_OUTPUT"
if [ "$DL_RC" -ne 0 ]; then
echo "::warning::download_testlogs exited $DL_RC for ${{ matrix.arch }}; building a partial report, the job will be failed at the end."
fi

- name: Identify output folder
id: folder
Expand Down Expand Up @@ -284,6 +298,7 @@ jobs:

- name: Collect upload paths
id: upload-paths
if: ${{ always() && steps.folder.outputs.folder != '' }}
run: |
FOLDER=".automation_scripts/pytorch-unit-test-scripts/${{ steps.folder.outputs.folder }}"
PATHS="${FOLDER}/*.csv
Expand All @@ -302,11 +317,22 @@ jobs:
echo "EOF" >> "$GITHUB_OUTPUT"

- name: Upload artifacts
if: ${{ always() && steps.upload-paths.outputs.paths != '' }}
uses: actions/upload-artifact@v4
with:
name: ${{ needs.setup-matrix.outputs.prefix }}-results-${{ matrix.arch }}
path: ${{ steps.upload-paths.outputs.paths }}

- name: Flag incomplete download
# A partial download (some config's upstream jobs missing) still produces
# and uploads a report above; fail the job here so the run is clearly
# marked failed for that arch without losing the report.
if: ${{ always() && steps.download.outputs.download_rc != '0' && steps.download.outputs.download_rc != '' }}
run: |
echo "download_testlogs reported an incomplete download (rc=${{ steps.download.outputs.download_rc }}) for ${{ matrix.arch }}."
echo "A partial report was still generated and uploaded; failing the job to flag the incomplete download."
exit 1

summarize:
needs: [setup-matrix, generate-parity]
if: ${{ !cancelled() }}
Expand Down