Bug-2038705: Replace requests-based GitHub API calls with PyGithub across ETL modules#9664
Draft
moijes12 wants to merge 5 commits into
Draft
Bug-2038705: Replace requests-based GitHub API calls with PyGithub across ETL modules#9664moijes12 wants to merge 5 commits into
moijes12 wants to merge 5 commits into
Conversation
Updated the below functions in treeherder/utils/github.py to use PyGithub - get_repository - compare_shas - get_all_commits - get_commit - get_pull_request - get_pull_request_commits - get_releases Pending work: - Callers need to be updated. - Failing tests need to be examined - New tests needed for `github.py` Fixes [Bug-2038705](https://bugzilla.mozilla.org/show_bug.cgi?id=2038705)
- Fixed failing unit tests for push_loader - Updated changelog/collector.py
Fixed collector.py and its unit tests Also, updated tasks.py
Migrated the `query_data` function in ingest.py to use the library written over PyGithub. Also, updated the unit tests for the same.
- Added tests for treeherder/utils/github.py - Removed fetch_api and fetch_api_full_url
moijes12
changed the title
|
|
||
| commits = [] | ||
| for _commit in compare_response["commits"]: | ||
| for temp_commit in compare_result.commits: |
Collaborator
There was a problem hiding this comment.
Could the old variable name be kept? With temp_commit, I'd associate commit local changes which will not be pushed upstream but be removed with e.g. git reset --hard ...
| from django.conf import settings | ||
| from django.core.management.base import BaseCommand, CommandError | ||
| from django.db import connection | ||
| from github.Commit import Commit |
Collaborator
There was a problem hiding this comment.
Could the github API calls in this file also be routed through the helper github.py which sets the authentication token if available? There might be cases when the ingest command needs to be run on non-prod (for testing, ...) or production (to recover or for other reasons)? This can be done in this PR or in a separate one.
| for release in github.get_releases(owner, repository): | ||
| # no "since" option for releases() we filter manually here | ||
| if "since" in kw and release["published_at"] <= kw["since"]: | ||
| print(f"kw={kw}") |
Collaborator
There was a problem hiding this comment.
Debug code which needs removal.
Contributor
Author
|
Thanks @Archaeopteryx I will move this to draft as I work on the review feedback |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR migrates all GitHub REST API interactions from direct
requestslibrarycalls to the PyGithub library, providing a more maintainable and type-safe approach
to GitHub API interactions.
Changes
treeherder/utils/github.pyto use PyGithub objects insteadof raw REST API responses with
requestslibrarytreeherder/changelog/collector.py- Updated to work with PyGithub commit andrelease objects
treeherder/changelog/tasks.py- Updated datetime handling for PyGithubdatetime objects
treeherder/etl/management/commands/ingest.py- Refactoredquery_data()andingest_git_pushes()to use PyGithub comparison and commit objectstests/utils/test_github.pyandupdated existing tests in changelog and ingest modules to mock PyGithub objects
Benefits
Related Issue
Fixes Bug-2038705