Skip to content

Add workflow to run SDGym monthly and publish results - #427

Merged
R-Palazzo merged 68 commits into
mainfrom
issue-425-workflow-sdgym
Aug 28, 2025
Merged

Add workflow to run SDGym monthly and publish results#427
R-Palazzo merged 68 commits into
mainfrom
issue-425-workflow-sdgym

Conversation

@R-Palazzo

Copy link
Copy Markdown
Collaborator

CU-86b5qr8wj
Resolve #425

@R-Palazzo
R-Palazzo requested review from amontanez24 and fealho July 24, 2025 14:43
@R-Palazzo R-Palazzo self-assigned this Jul 24, 2025
@R-Palazzo
R-Palazzo requested a review from a team as a code owner July 24, 2025 14:43
@sdv-team

Copy link
Copy Markdown
Contributor

@codecov

codecov Bot commented Jul 24, 2025

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 77.73973% with 65 lines in your changes missing coverage. Please review.
✅ Project coverage is 71.79%. Comparing base (1616a40) to head (f23428c).
⚠️ Report is 1 commits behind head on main.

Files with missing lines Patch % Lines
sdgym/result_writer.py 33.33% 20 Missing ⚠️
sdgym/run_benchmark/upload_benchmark_results.py 81.73% 19 Missing ⚠️
sdgym/benchmark.py 70.37% 8 Missing ⚠️
sdgym/sdgym_result_explorer/result_handler.py 33.33% 8 Missing ⚠️
sdgym/cli/utils.py 42.85% 4 Missing ⚠️
sdgym/cli/collect.py 33.33% 2 Missing ⚠️
sdgym/run_benchmark/run_benchmark.py 93.54% 2 Missing ⚠️
sdgym/cli/__main__.py 0.00% 1 Missing ⚠️
sdgym/sdgym_result_explorer/result_explorer.py 50.00% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #427      +/-   ##
==========================================
+ Coverage   70.97%   71.79%   +0.81%     
==========================================
  Files          24       27       +3     
  Lines        1840     2088     +248     
==========================================
+ Hits         1306     1499     +193     
- Misses        534      589      +55     
Flag Coverage Δ
integration 56.47% <39.17%> (-0.71%) ⬇️
unit 62.35% <77.73%> (+2.08%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Comment thread .github/workflows/upload_benchmark_results.yml Outdated
Comment thread sdgym/_run_benchmark/__init__.py Outdated
Comment thread sdgym/benchmark.py
Comment thread sdgym/_run_benchmark/__init__.py Outdated
@R-Palazzo
R-Palazzo removed the request for review from a team July 24, 2025 16:20

@fealho fealho left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job!

Comment thread .github/workflows/run_benchmark.yml Outdated
Comment thread sdgym/_run_benchmark/__init__.py Outdated
Comment thread sdgym/_run_benchmark/run_benchmark.py Outdated
Comment thread sdgym/_run_benchmark/run_benchmark.py
Comment thread sdgym/_run_benchmark/upload_benchmark_results.py Outdated
Comment thread sdgym/_run_benchmark/upload_benchmark_results.py Outdated
Comment thread tests/unit/_run_benchmark/test__utils.py Outdated
@R-Palazzo
R-Palazzo force-pushed the issue-425-workflow-sdgym branch from e928db0 to e63a58d Compare August 4, 2025 13:46
Comment thread .github/workflows/run_benchmark.yml Outdated
from sdgym.s3 import parse_s3_path

OUTPUT_DESTINATION_AWS = 's3://sdgym-benchmark/Benchmarks/'
UPLOAD_DESTINATION_AWS = 's3://sdgym-benchmark/Benchmarks/'

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Along with the Github location, I'm also uploading the summary there so we have a backup.

Comment thread .github/workflows/upload_benchmark_results.yml Outdated
@R-Palazzo
R-Palazzo requested a review from amontanez24 August 19, 2025 11:53
@R-Palazzo
R-Palazzo force-pushed the issue-425-workflow-sdgym branch from 7eb91af to fc7321b Compare August 25, 2025 10:22
@R-Palazzo
R-Palazzo changed the base branch from main to issue-434-bump-torch August 25, 2025 10:23
Comment thread pyproject.toml Outdated
@R-Palazzo
R-Palazzo force-pushed the issue-425-workflow-sdgym branch from c1c2830 to 3791993 Compare August 25, 2025 14:34
Base automatically changed from issue-434-bump-torch to main August 25, 2025 15:27

@amontanez24 amontanez24 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking really good! Should e the last round of comments

Comment thread .github/workflows/readme.yml Outdated
Comment thread .github/workflows/run_benchmark.yml Outdated
Comment thread .github/workflows/upload_benchmark_results.yml Outdated
--base gatsby-home \
--title "Upload SDGym Benchmark Results ($FOLDER_NAME)" \
--body "Automated benchmark results upload" \
--reviewer "amontanez24"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

you can remove this before merging

Comment on lines +60 to +62
- name: Checkout target repo (sdv-dev.github.io)
if: env.SKIP_UPLOAD != 'true'
run: |
git clone https://github.com/sdv-dev/sdv-dev.github.io.git target-repo
cd target-repo
git checkout gatsby-home

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why don't we use the github checkout action for this?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @amontanez24, I've reinstated the command to directly commit fb5b32a. Last time it was this commit:
sdv-dev/sdv-dev.github.io@0911425

@R-Palazzo
R-Palazzo force-pushed the issue-425-workflow-sdgym branch from 3442f44 to f23428c Compare August 27, 2025 18:20
@R-Palazzo
R-Palazzo merged commit b4e8813 into main Aug 28, 2025
55 checks passed
@R-Palazzo
R-Palazzo deleted the issue-425-workflow-sdgym branch August 28, 2025 09:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add workflow to run SDGym monthly and publish results

4 participants