Skip to content

Refactor docker build/push workflow action #11653

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
amaltaro merged 1 commit into from
Jul 10, 2023
Merged

Refactor docker build/push workflow action #11653

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
46 changes: 19 additions & 27 deletions .github/workflows/docker_images_template.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,8 @@ on:
jobs:
build_images_from_template:
runs-on: ubuntu-latest
env:
SERVICE_NAME: ${{inputs.wmcore_component}}
steps:
- name: Get the Ref
id: get-ref
Expand All @@ -23,37 +25,27 @@ jobs:
ref: ${{ github.ref }}
head_ref: ${{ github.head_ref }}

- name: Build image
env:
PYPI_TAG: ${{steps.get-ref.outputs.tag}}
run: |
echo "Building service: ${{inputs.wmcore_component}}, with tag: ${PYPI_TAG}"
svn checkout https://github.com/dmwm/CMSKubernetes/trunk/docker/pypi/${{inputs.wmcore_component}}
cd ${{inputs.wmcore_component}}
cat Dockerfile
echo "Sleeping 5min to ensure that PyPi packages are available..."
sleep 300
docker build --build-arg TAG=${PYPI_TAG} --tag registry.cern.ch/cmsweb/${{inputs.wmcore_component}}:${PYPI_TAG} .

- name: Images
run: |
docker images

- name: Login to registry.cern.ch
uses: docker/login-action@v2
with:
registry: registry.cern.ch
username: ${{ secrets.cern_user }}
password: ${{ secrets.cern_token }}

- name: Publish image to registry.cern.ch
uses: docker/build-push-action@v1
with:
path: ${{inputs.wmcore_component}}
build_args: |
TAG=${{steps.get-ref.outputs.tag}}
registry: registry.cern.ch
username: ${{ secrets.cern_user }}
password: ${{ secrets.cern_token }}
repository: cmsweb/${{inputs.wmcore_component}}
tag_with_ref: true
- name: Build and publish docker image
env:
PYPI_TAG: ${{steps.get-ref.outputs.tag}}
CERN_REGISTRY: registry.cern.ch
run: |
echo "Building service: ${SERVICE_NAME}, with tag: ${PYPI_TAG}"
svn checkout https://github.com/dmwm/CMSKubernetes/trunk/docker/pypi/${SERVICE_NAME}

@vkuznet vkuznet Jul 10, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need svn to checkout github repo? Why not to use git for that?

@amaltaro amaltaro Jul 10, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For this question, please refer to: #11639 for full context.

@vkuznet vkuznet Jul 10, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Alan, even though ticket lists that you swap curl with svn it does not provide a reason, and neither address why you use curl/svn to get git repo files, why not to use (tool designed for that) git? Please note that in all CMSKubernetes yaml file I always relies on git, e.g. RUN git checkout tags/$TAG -b build see https://github.com/dmwm/CMSKubernetes/blob/master/docker/dbs2go/Dockerfile#L33

@amaltaro amaltaro Jul 10, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@todor-ivanov has done the research on the svn util. My understanding is that with svn, we can fetch a sub-directory of the repository, while git does not provide this capability and we would have to clone the whole repository for each of the 11(?) services that we have setup in the actions workflow.

@vkuznet vkuznet Jul 10, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ok, thanks for providing the reason. Said that, this line is not doing what the comment is saying, i.e. it will fetch latest subdir and not a particular tag. I do not know svn enough to answer if it can fetch proper tag of sub-dir, but curl cat fetch tarball of particular tag, and git indeed will fetch entire repo for that tag. So, we should fix either svn or switch back to curl to fetch exact tag of the repo.

@amaltaro amaltaro Jul 10, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Valentin, the CMSKubernetes code that we check out isn't really tagged (we always fetch what is in HEAD).
The PYPI_TAG is indeed used, but then in the scope of WMCore (used for the tag in pypi and the docker build argument).

Having said that, I would say it works as expected and there is nothing to be changed here.

@todor-ivanov todor-ivanov Jul 11, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @vkuznet @amaltaro

with svn, we can fetch a sub-directory of the repository, while git does not provide this capability and we would have to clone the whole repository for each of the 11(?) services that we have setup

Yes this is the exact reason.

So, we should fix either svn or switch back to curl to fetch exact tag of the repo

SVN is fully capable of fetching particular tag.

cd ${SERVICE_NAME}
echo "Retrieved Dockerfile with content:"
cat Dockerfile
echo "Sleeping 5min to ensure that PyPi packages are available..."
sleep 300

@vkuznet vkuznet Jul 10, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do we need to sleep, what do you mean that PyPi packages are available? Once you download them they should be available and I don't see any needs for sleep. Why 300 and not any other number?

@amaltaro amaltaro Jul 10, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this we discussed when we were commissioning the build workflow. There is a delay between publishing a package to the PyPi repository and fetching it. This 5min is a fair enough commitment that we came up with in the past and which allows us to use the PyPi package when building the docker image.

docker build --build-arg TAG=${PYPI_TAG} --tag ${CERN_REGISTRY}/cmsweb/${SERVICE_NAME}:${PYPI_TAG} .
echo "Image build process completed. Current images are:"
docker images
echo "Now push new image to the CERN registry"
docker push ${CERN_REGISTRY}/cmsweb/${SERVICE_NAME}:${PYPI_TAG}