Add new GH Action workflows to build WMCore images

[vkuznet]

Fixes #10921 , #11374

Status

In development

Description

Provide new set of GH Action workflows to build WMCore images. So far it is done as independent GH workflow but it re-uses tar ball creation steps from pypi packaging workflow, i.e. in particular it re-uses the following steps https://github.com/dmwm/WMCore/blob/master/.github/workflows/pypy_build_publish_template.yaml#L16-L33 from pypi packaging workflow. This is why my initial suggestion was to create images along with pypi packaging but based on request from @amaltaro I made this PR to have separated GH Action workflows. To avoid re-using the same code within these workflows (if we want to separate pypi from docker images) I suggest to create later an issue to have separate template which we will re-use in both (pypi and image creation) workflows.

Note: so far the upload to registry is commented out since it requires setup of specific CERN credentials. Once we'll enable GH Action and successfully build all images we may enable upload part.

Is it backward compatible (if not, which system it affects?)

YES

Related PRs

#11339

External dependencies / deployment changes

None

[cmsdmwmbot]

Jenkins results:

• Python3 Unit tests: succeeded
  • 2 changes in unstable tests
• Python3 Pylint check: succeeded
• Pylint py3k check: succeeded
• Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13761/artifact/artifacts/PullRequestReport.html

[vkuznet]

@amaltaro , I made the following modifications per our discussion:

• new workflow yaml files pypi_build_and_images.yaml is a clone of pypy_build_publish.yaml with the following changes:
  • it uses original build_and_publish_services section which builds pypi packages and upload them to pypi
  • it adds a new job build_images which now depends on build_and_publish_services, i.e. it will only starts when we build and upload to pypi
  • the build_images uses template from pypi_images_template.yaml which by itself now relies on presents of packages in pypi and uses Dockerfile which relies on pip install

As I explained I can't test the changes therefore I'll rely on your review. The pypi_build_and_images.yaml will eventually replace pypy_build_publish.yaml because it incorporates the latter. So far I keep them separate, and once we get proper build we can remove the latter.

[cmsdmwmbot]

Jenkins results:

• Python3 Unit tests: failed
  • 4 new failures
  • 3 changes in unstable tests
• Python3 Pylint check: succeeded
• Pylint py3k check: succeeded
• Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13773/artifact/artifacts/PullRequestReport.html

[amaltaro]

@vkuznet Valentin, I left a few comments along the code.
In addition to those, I think we discussed that we should provide everything in the same workflow, didn't we?

With the current proposal, we would be building and uploading images to PyPi twice, which might actually mark one of the build workflows as failed.

[vkuznet]

@amaltaro , everything is done within one workflow pypi_build_and_images.yaml but it uses two templates for two different jobs. I kept it separate from existing one pypy_build_publish.yaml to ease review process and to try it out first. Once we'll see that it is working we can simply delete pypy_build_publish.yaml one in favor of pypi_build_and_images.yaml. Is it ok? All other stuff you asked for is already implemented, please have another look.

[cmsdmwmbot]

Jenkins results:

• Python3 Unit tests: succeeded
  • 2 tests no longer failing
  • 2 tests added
  • 1 changes in unstable tests
• Python3 Pylint check: succeeded
• Pylint py3k check: succeeded
• Pycodestyle check: succeeded

Details at https://cmssdt.cern.ch/dmwm-jenkins/view/All/job/DMWM-WMCore-PR-test/13778/artifact/artifacts/PullRequestReport.html

[amaltaro]

@vkuznet that's fine with me. However, this workflow duplicates the PyPi build and upload to the PyPi repository and I am not sure whether that will actually cause a terminal failure for one of the workflows (which could be this one).

If we have to re-run this workflow manually, and PyPi packages already exist in the target version, it will keep failing. This is just a suspicious though, I never tried to upload the same package and version twice to PyPi.

[vkuznet]

@amaltaro , ok, I removed existing GH workflow as we are replacing it. And, I changed prefix in existing template name from pypy to pypi. Now this PR contains everything we need. Please review again.

[amaltaro]

A change has been requested along the code. Thanks

[amaltaro]

Okay, it's time to see how it goes for real now. Thanks

[vkuznet]

@amaltaro , can you try to run it manually, I seethe menu now appears in GH Action tab

[amaltaro]

I am going to make a new tag in a couple of minutes. Let's see how it goes.

[amaltaro]

@vkuznet you might want to have a look at the failure. I just double checked and those secrets are available in the repository settings, so it must be some syntax or something like that that we missed in this GH workflow.

[vkuznet]

Alan, from yaml file syntax point of view I do not see anything unusual. Said that, I do not have access to Settings area of dmwm repo and can't see which secrets we have over there. I suggest that we convene on a zoom tomorrow or next week to clarify that, e.g. you may check PYPY_PRODUCTION secret as we use here: https://github.com/dmwm/WMCore/actions/runs/3596748595/workflow#L35 This line is identical to line https://github.com/dmwm/WMCore/actions/runs/3596748595/workflow#L50 syntax wise and therefore I do not know what is a cause of the issue yet.

[amaltaro]

Just to update this ticket.
Valentin and I made a few interactive changes (through the GH web UI) to the GitHub workflows and it's not fully functional, thus building and uploading the PyPi package as well as the docker image to the Gitlab registry.

Relevant commits are:
8c40303
26f83a5
a960e58

Thanks again Valentin.

[amaltaro]

And now I have a confirmation about this suspicious:
#11377 (comment)

indeed (trying to) uploading the same package to PyPi causes the HTTP request to fail, hence failing the action workflow, as can be seen here:
https://github.com/dmwm/WMCore/actions/runs/3687511820

A manual re-run action will fail, unless none of the PyPi images made it to the PyPi repository.

This workflow case, docker image build failed with the following message:

  Installing build dependencies: started
  Installing build dependencies: finished with status 'done'
  Getting requirements to build wheel: started
  Getting requirements to build wheel: finished with status 'done'
  Installing backend dependencies: started
Error: The operation was canceled.

for now, I'd say we don't try to make improvements to this workflow. But if it comes back again in the coming weeks, then we should investigate it further and make it more robust.