Skip to content

Build WMCore services and publish them to pypy, usinig to tag pushes - #11318

Merged
amaltaro merged 1 commit into
dmwm:masterfrom
khurtado:pypy
Oct 7, 2022
Merged

Build WMCore services and publish them to pypy, usinig to tag pushes#11318
amaltaro merged 1 commit into
dmwm:masterfrom
khurtado:pypy

Conversation

@khurtado

@khurtado khurtado commented Oct 5, 2022 •
edited
Loading

Copy link
Copy Markdown

Fixes #11276

Notes:
Packages supported: [wmagent, wmagent-devtools, wmcore, reqmon, reqmgr2, reqmgr2ms, global-workqueue, acdcserver]

Status

tested in a fork.
We need to upload the pypy secret token inside the WMCore repository for this to work

Description

These changes create a set of github actions to automatically build and publish WMCore services ([wmagent, wmagent-devtools, wmcore, reqmon, reqmgr2, reqmgr2ms, global-workqueue, acdcserver]) to pypy: https://pypi.org/user/cms-oc-dmwm/

Is it backward compatible (if not, which system it affects?)

YES

External dependencies / deployment changes

#11324

@cmsdmwmbot

cmsdmwmbot commented Oct 5, 2022

Copy link
Copy Markdown

Jenkins results:

  • Python3 Unit tests: succeeded
    • 1 tests no longer failing
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13607/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot

cmsdmwmbot commented Oct 5, 2022

Copy link
Copy Markdown

Jenkins results:

  • Python3 Unit tests: failed
    • 2 new failures
    • 2 tests no longer failing
    • 1 changes in unstable tests
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13608/artifact/artifacts/PullRequestReport.html

@vkuznet

vkuznet commented Oct 5, 2022

Copy link
Copy Markdown
Contributor

For the references:

FROM debian:latest
MAINTAINER Valentin Kuznetsov vkuznet@gmail.com
RUN apt-get update
RUN apt-get install -y curl libcurl4 libcurl4-openssl-dev python3-pycurl pip
RUN ln -s /usr/bin/python3 /usr/bin/python
RUN pip install reqmgr2
ENV WDIR=/data
WORKDIR $WDIR
CMD ["python3"]

Please note, the image is debian based, it does not have RPMs, it uses reqmgr2 PyPi package, it is 628MB image size.

But using reqmgr2ms package (and similar Dockefile) I get the following issue:

INFO: pip is looking at multiple versions of py to determine which version is compatible with other requirements. This could take a while.
Collecting py<2.0.0,>=1.4.26
  Downloading py-1.10.0-py2.py3-none-any.whl (97 kB)
  Downloading py-1.9.0-py2.py3-none-any.whl (99 kB)
  Downloading py-1.8.2-py2.py3-none-any.whl (83 kB)
  Downloading py-1.8.1-py2.py3-none-any.whl (83 kB)
  Downloading py-1.8.0-py2.py3-none-any.whl (83 kB)
  Downloading py-1.7.0-py2.py3-none-any.whl (83 kB)
  Downloading py-1.6.0-py2.py3-none-any.whl (83 kB)
  Downloading py-1.5.4-py2.py3-none-any.whl (83 kB)
  Downloading py-1.5.3-py2.py3-none-any.whl (84 kB)
  Downloading py-1.5.2-py2.py3-none-any.whl (88 kB)
  Downloading py-1.5.1-py2.py3-none-any.whl (88 kB)
  Downloading py-1.4.34-py2.py3-none-any.whl (84 kB)
  Downloading py-1.4.33-py2.py3-none-any.whl (83 kB)
  Downloading py-1.4.32-py2.py3-none-any.whl (82 kB)
  Downloading py-1.4.31-py2.py3-none-any.whl (81 kB)
  Downloading py-1.4.30-py2.py3-none-any.whl (81 kB)
  Downloading py-1.4.29-py2.py3-none-any.whl (82 kB)
  Downloading py-1.4.28-py2.py3-none-any.whl (81 kB)
  Downloading py-1.4.27-py2.py3-none-any.whl (81 kB)
  Downloading py-1.4.26.tar.gz (190 kB)
INFO: pip is looking at multiple versions of py to determine which version is compatible with other requirements. This could take a while.
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: pip is looking at multiple versions of decorator to determine which version is compatible with other requirements. This could take a while.
Collecting decorator>=3.4.2
  Downloading decorator-5.1.0-py3-none-any.whl (9.1 kB)
  Downloading decorator-5.0.9-py3-none-any.whl (8.9 kB)
  Downloading decorator-5.0.8-py3-none-any.whl (8.9 kB)
  Downloading decorator-5.0.7-py3-none-any.whl (8.8 kB)
  Downloading decorator-5.0.6-py3-none-any.whl (8.8 kB)
  Downloading decorator-5.0.5-py3-none-any.whl (8.8 kB)
  Downloading decorator-4.4.2-py2.py3-none-any.whl (9.2 kB)
  Downloading decorator-4.4.1-py2.py3-none-any.whl (9.2 kB)
  Downloading decorator-4.4.0-py2.py3-none-any.whl (8.3 kB)
  Downloading decorator-4.3.2-py2.py3-none-any.whl (9.1 kB)
  Downloading decorator-4.3.1-py2.py3-none-any.whl (8.8 kB)
  Downloading decorator-4.3.0-py2.py3-none-any.whl (9.2 kB)
  Downloading decorator-4.2.1-py2.py3-none-any.whl (9.3 kB)
  Downloading decorator-4.1.2-py2.py3-none-any.whl (9.1 kB)
  Downloading decorator-4.1.1-py2.py3-none-any.whl (9.0 kB)
  Downloading decorator-4.1.0-py2.py3-none-any.whl (9.0 kB)
  Downloading decorator-4.0.11-py2.py3-none-any.whl (8.9 kB)
  Downloading decorator-4.0.10-py2.py3-none-any.whl (9.2 kB)
  Downloading decorator-4.0.9-py2.py3-none-any.whl (22 kB)
  Downloading decorator-4.0.6-py2.py3-none-any.whl (21 kB)
  Downloading decorator-4.0.4-py2.py3-none-any.whl (8.7 kB)
  Downloading decorator-4.0.3-py2.py3-none-any.whl (8.5 kB)
  Downloading decorator-4.0.2-py2.py3-none-any.whl (8.5 kB)
  Downloading decorator-4.0.1-py2.py3-none-any.whl (8.5 kB)
  Downloading decorator-4.0.0-py2.py3-none-any.whl (8.6 kB)
  Downloading decorator-3.4.2.tar.gz (6.1 kB)
INFO: pip is looking at multiple versions of decorator to determine which version is compatible with other requirements. This could take a while.
INFO: This is taking longer than usual. You might need to provide the dependency resolver with stricter constraints to reduce runtime. If you want to abort this run, you can press Ctrl + C to do so. To improve how pip performs, tell us what happened here: https://pip.pypa.io/surveys/backtracking
INFO: pip is looking at multiple versions of retry to determine which version is compatible with other requirements. This could take a while.

The bottom line is that we still have dependency chain issue with python packages which will require special attention since building and pushing packages to pypi is half way through, we need to install them to see if our builds provide valid packages.

@khurtado

khurtado commented Oct 5, 2022

Copy link
Copy Markdown
Author

@vkuznet Wonderful! @ericvaandering is working on splitting reqmgr2ms into different pypy packages at present. Let's come back to the issue you are seeing after he manages to do #10159

@khurtado khurtado mentioned this pull request Oct 5, 2022
Closed
@cmsdmwmbot

cmsdmwmbot commented Oct 5, 2022

Copy link
Copy Markdown

Jenkins results:

  • Python3 Unit tests: failed
    • 12 new failures
    • 2 tests no longer failing
    • 2 changes in unstable tests
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13610/artifact/artifacts/PullRequestReport.html

@khurtado
khurtado requested a review from amaltaro October 5, 2022 18:15
@cmsdmwmbot

cmsdmwmbot commented Oct 5, 2022

Copy link
Copy Markdown

Jenkins results:

  • Python3 Unit tests: failed
    • 1 new failures
    • 2 tests no longer failing
    • 2 changes in unstable tests
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13611/artifact/artifacts/PullRequestReport.html

@khurtado

khurtado commented Oct 6, 2022

Copy link
Copy Markdown
Author

@germanfgv Do you have any insight on the t0 pypy package that exists already?
https://pypi.org/project/t0/

Do we need to provide support on WMCore or does T0 have that covered?

amaltaro
amaltaro reviewed Oct 7, 2022
View reviewed changes

@amaltaro amaltaro left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@khurtado Kenyi, can you please update this PR with the recent changes made by Erik in this PR:
#11324
?

We now have separated microservice builds, and reqmgr2ms no longer needs to be packaged and uploaded to PyPi. @goughes can correct me here as well, in case I am missing anything.

@vkuznet

vkuznet commented Oct 7, 2022

Copy link
Copy Markdown
Contributor

I created a separate issue about packages which are already in pypi but for some reason which cannot be installed, see #11327 So far I only was able to install reqmgr2, but reqmgr2ms and wmagent are not installable.

@khurtado

khurtado commented Oct 7, 2022

Copy link
Copy Markdown
Author

@amaltaro I have updated the list of pypy packages to go according to Erik changes.

amaltaro
amaltaro requested changes Oct 7, 2022
View reviewed changes

@amaltaro amaltaro left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess GH actions are only available for this architecture:

    runs-on: ubuntu-latest

? Don't we have the possibility to use an RPM-based system (RH/Fedora/CentOS)?

Can you please squash these commits? And also ping me on slack, such that we can define the PyPi token. Thanks

Comment thread .github/workflows/pypy_build_publish_template.yaml
@cmsdmwmbot

cmsdmwmbot commented Oct 7, 2022

Copy link
Copy Markdown

Jenkins results:

  • Python3 Unit tests: failed
    • 2 new failures
    • 1 tests no longer failing
    • 1 changes in unstable tests
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13627/artifact/artifacts/PullRequestReport.html

Kenyi Hurtado Kenyi Hurtado
Build WMCore services (except for t0 right now, due to name conflicts…
d50c2d5 
…) and publish them to pypy, using tag pushes
@cmsdmwmbot

cmsdmwmbot commented Oct 7, 2022

Copy link
Copy Markdown

Jenkins results:

  • Python3 Unit tests: succeeded
    • 1 tests no longer failing
    • 2 changes in unstable tests
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13628/artifact/artifacts/PullRequestReport.html

@cmsdmwmbot

cmsdmwmbot commented Oct 7, 2022

Copy link
Copy Markdown

Jenkins results:

  • Python3 Unit tests: succeeded
    • 1 tests no longer failing
    • 1 changes in unstable tests
  • Python3 Pylint check: succeeded
  • Pylint py3k check: succeeded
  • Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13629/artifact/artifacts/PullRequestReport.html

amaltaro
amaltaro approved these changes Oct 7, 2022
View reviewed changes

@amaltaro amaltaro left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clarification, Kenyi and Valentin.
From our chat on slack, I have created the Pypi token and it should be all set now. I'm creating a new tag soon to run a real test.

@amaltaro
amaltaro merged commit 28be7dc into dmwm:master Oct 7, 2022
@amaltaro

amaltaro commented Oct 7, 2022

Copy link
Copy Markdown
Contributor

Haaa, nice! In 1min, we managed to build and upload everything to PyPi. Here is one example:
https://pypi.org/project/wmagent/2.1.4rc2/

I know it's unrelated to this PR, but I wonder if the PyPi release number should be 2.1.4.rc2, or the one created 2.1.4rc2? @goughes any thoughts? We might as well refactor our pre-releases to actually release candidates.

@khurtado

khurtado commented Oct 9, 2022

Copy link
Copy Markdown
Author

@amaltaro I think only .postN and .devN are allowed to have a . after the numerical version numbers. Anything else like rc, a, b, etc can't have a dot, see the PEP 440 naming convention below:

https://peps.python.org/pep-0440/#public-version-identifiers

@amaltaro amaltaro mentioned this pull request Oct 10, 2022
Closed
@ericvaandering

ericvaandering commented Oct 11, 2022 via email

Copy link
Copy Markdown
Member

@khurtado khurtado mentioned this pull request Oct 11, 2022
Merged
@vkuznet vkuznet mentioned this pull request Oct 11, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@vkuznet vkuznet vkuznet left review comments
@amaltaro amaltaro amaltaro approved these changes

Assignees

No one assigned

Labels

PR: Do not merge yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Automated packaging and upload of WMCore services to Pypi

5 participants

@khurtado @cmsdmwmbot @vkuznet @amaltaro @ericvaandering