Skip to content

Split microservices into separate packages - #11324

Merged
amaltaro merged 1 commit into
dmwm:masterfrom
goughes:split-ms-pypi
Oct 7, 2022
Merged

Split microservices into separate packages#11324
amaltaro merged 1 commit into
dmwm:masterfrom
goughes:split-ms-pypi

Conversation

@goughes

@goughes goughes commented Oct 6, 2022

Copy link
Copy Markdown
Contributor

Fixes #10159

Status

Not tested

Description

This PR updates build_dependencies.py, creates a new reqmgr2ms-core system and adds separate packages for all the current microservices.

Is it backward compatible (if not, which system it affects?)

Yes, we leave the current reqmgr2ms package which is still used for RPM based builds.

Related PRs

Follow up to #10357 as I have created a "MSCore" directory and moved MSCore, MSManager and TaskManager modules to it.

External dependencies / deployment changes

We will need to update each microservice's config.py since the path for MSManager has changed.
https://gitlab.cern.ch/cmsweb-k8s/services_config/-/merge_requests/168
https://gitlab.cern.ch/cmsweb-k8s/services_config/-/merge_requests/167
https://gitlab.cern.ch/cmsweb-k8s/services_config/-/merge_requests/166
dmwm/deployment#1192

@cmsdmwmbot

Copy link
Copy Markdown

Jenkins results:

  • Python3 Unit tests: failed
    • 30 new failures
    • 2 changes in unstable tests
  • Python3 Pylint check: failed
    • 18 warnings and errors that must be fixed
    • 39 warnings
    • 276 comments to review
  • Pylint py3k check: failed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 83 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13615/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot

Copy link
Copy Markdown

Jenkins results:

  • Python3 Unit tests: failed
    • 26 new failures
    • 2 changes in unstable tests
  • Python3 Pylint check: failed
    • 18 warnings and errors that must be fixed
    • 39 warnings
    • 276 comments to review
  • Pylint py3k check: failed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 83 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13617/artifact/artifacts/PullRequestReport.html

@goughes
goughes force-pushed the split-ms-pypi branch 3 times, most recently from 2b7a3f4 to 01bf2a3 Compare October 6, 2022 19:06
@cmsdmwmbot

Copy link
Copy Markdown

Jenkins results:

  • Python3 Unit tests: failed
    • 4 new failures
    • 3 changes in unstable tests
  • Python3 Pylint check: failed
    • 18 warnings and errors that must be fixed
    • 39 warnings
    • 276 comments to review
  • Pylint py3k check: failed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 83 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13618/artifact/artifacts/PullRequestReport.html

@amaltaro amaltaro left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Erik, changes look good to me. I left one comment along the code though.
I think we should stop building reqmgr2ms for PyPi, even though we still need it for RPMs.

Comment thread tools/build_pypi_packages.sh Outdated
TOBUILD=$1
# list of packages that can be built and uploaded to pypi
PACKAGES="wmagent wmagent-devtools wmcore reqmon reqmgr2 reqmgr2ms global-workqueue acdcserver"
PACKAGES="wmagent wmagent-devtools wmcore reqmon reqmgr2 reqmgr2ms global-workqueue acdcserver

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perhaps we should stop building/uploading reqmgr2ms to PyPi?

@cmsdmwmbot

Copy link
Copy Markdown

Jenkins results:

  • Python3 Unit tests: failed
    • 4 new failures
    • 3 changes in unstable tests
  • Python3 Pylint check: failed
    • 15 warnings and errors that must be fixed
    • 39 warnings
    • 276 comments to review
  • Pylint py3k check: failed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 84 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13619/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot

Copy link
Copy Markdown

Jenkins results:

  • Python3 Unit tests: failed
    • 1 new failures
    • 2 changes in unstable tests
  • Python3 Pylint check: failed
    • 14 warnings and errors that must be fixed
    • 39 warnings
    • 276 comments to review
  • Pylint py3k check: failed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 84 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13620/artifact/artifacts/PullRequestReport.html

@goughes

goughes commented Oct 6, 2022

Copy link
Copy Markdown
Contributor Author

Thanks @amaltaro, I have removed reqmgr2ms from requirements.txt and the PyPI build script.

@cmsdmwmbot

Copy link
Copy Markdown

Jenkins results:

  • Python3 Unit tests: succeeded
    • 4 changes in unstable tests
  • Python3 Pylint check: failed
    • 14 warnings and errors that must be fixed
    • 39 warnings
    • 276 comments to review
  • Pylint py3k check: failed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 84 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13621/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot

Copy link
Copy Markdown

Jenkins results:

  • Python3 Unit tests: succeeded
    • 3 changes in unstable tests
  • Python3 Pylint check: failed
    • 14 warnings and errors that must be fixed
    • 39 warnings
    • 276 comments to review
  • Pylint py3k check: failed
    • 1 warnings
  • Pycodestyle check: succeeded
    • 84 comments to review

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13622/artifact/artifacts/PullRequestReport.html

@amaltaro
amaltaro self-requested a review October 6, 2022 20:44

@amaltaro amaltaro left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks Erik, this looks good to go!

@amaltaro

amaltaro commented Oct 7, 2022

Copy link
Copy Markdown
Contributor

Thanks Erik!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Split reqmgr2ms into separate PyPI packages

3 participants