Gitlab Image Build to ECIR

docs/services/gitlab/cicd.md

@@ -0,0 +1,66 @@
+# Continuous Integration and Continuous Deployment (CI/CD) with EIDF GitLab
+
+## GitLab Harbor Integration and the ECIR
+
+We recommend that users set up Harbor Integration to automate the use of the EIDF Container Image Registry (ECIR) as the pull and push location for container images.
+The ECIR is recommended as it is part of EIDF infrastructure and hence provides speed over external registries as well as a number of built-in security features such as software bills of materials (SBOMs) and security scanning via Trivy.
+More documentation around the ECIR can be found in the [ECIR documentation](../registry/index.md) and the [ECIR working with documentation](../registry/working-with.md).
+
+We assume users have set up an ECIR project and ECIR push robot account for the project which can be done by contacting the [EIDF Helpdesk](https://portal.eidf.ac.uk/queries/submit).
+
+Users should follow the [GitLab documentation on Harbor Integration](https://docs.gitlab.com/user/project/integrations/harbor/), making note of the following values to be input:
+
+- Harbor URL: `https://registry.eidf.ac.uk`
+- Harbor Project Name: Typically the same as the EIDF project you are working in e.g. `eidf123`
+- Username: The username of the Push Robot for your ECIR project, typically has a name like `eidf123-push-robot`
+- Password: The secret key of the push robot
+
+## Using an ECIR image in GitLab CI/CD
+
+The ECIR is a private registry and so the use of images stored in the ECIR requires authentication. A number of methods for this are documented in the [GitLab documentation](https://docs.gitlab.com/ci/docker/using_docker_images/#access-an-image-from-a-private-container-registry). We recommend setting up a variable defining the `DOCKER_AUTH_CONFIG` as described in the GitLab documentation above. When making use of the Harbor Integration this variable should be set as follows:
+
+```.gitlab-ci.yml
+variables:
+  DOCKER_AUTH_CONFIG: "{\"auths\":{\"$HARBOR_HOST\":{\"username\":\"$HARBOR_USERNAME\",\"password\":\"$HARBOR_PASSWORD\"}}}"
+```
+
+This will then authorise you to make use of the ECIR in your CI/CD pipelines, for example by pulling an image from the ECIR as the image for a job:
+
+```.gitlab-ci.yml
+
+variables:
+  DOCKER_AUTH_CONFIG: "{\"auths\":{\"$HARBOR_HOST\":{\"username\":\"$HARBOR_USERNAME\",\"password\":\"$HARBOR_PASSWORD\"}}}"
+
+my-job:
+  image: $HARBOR_HOST/$HARBOR_PROJECT/hello-world:latest
+  script:
+    - echo "This job uses an image from the ECIR"
+```
+
+## Building a Container Image with GitLab CI/CD
+
+Automation of building of container images is possible using GitLab CI/CD in the default GitLab Instance runner.
+
+Images can be built from a Dockerfile using [BuildKit](https://docs.docker.com/build/buildkit/). This requires users to include a stage in the .gitlab-ci.yml using the following template, taking special note of replacing parts in `< >` and if not using Harbor integration ensuring the Environment Variables are set correctly:
+
+```.gitlab-ci.yml
+build-rootless:
+  image:
+    name: moby/buildkit:rootless
+  stage: build
+  variables:
+    BUILDKITD_FLAGS: --oci-worker-no-process-sandbox
+    REG_IMAGE: "$HARBOR_HOST/$HARBOR_PROJECT/<image-name>:$CI_COMMIT_SHA"
+  before_script:
+    - mkdir -p ~/.docker
+    - echo "{\"auths\":{\"$HARBOR_HOST\":{\"username\":\"$HARBOR_USERNAME\",\"password\":\"$HARBOR_PASSWORD\"}}}" > ~/.docker/config.json
+  script:
+    - |
+      buildctl-daemonless.sh build \
+        --frontend dockerfile.v0 \
+        --local context=. \
+        --local dockerfile=<. or Dockerfile path to use> \
+        --output type=image,name=$REG_IMAGE,push=true
+```
+
+This will build an image using the specified Dockerfile and push it to the specified registry. The template is lightly adapted from the GitLab documentation on [building and pushing Docker images with BuildKit](https://docs.gitlab.com/ee/ci/docker/using_docker_buildx.html#build-and-push-docker-images-with-buildkit) where more examples and details can be found.

docs/services/gitlab/quickstart.md

@@ -2,7 +2,7 @@
 
 ## Accessing
 
-Access the EIDF gitlab in your browser by opening [https://gitlab.eidf.ac.uk/](https://gitlab.eidf.ac.uk/). You must be a member of an active EIDF project and have a user account to use the EIDF gitlab Service.
+Access the EIDF GitLab in your browser by opening [https://gitlab.eidf.ac.uk/](https://gitlab.eidf.ac.uk/). You must be a member of an active EIDF project and have a user account to use the EIDF GitLab Service.
 
 ![GitLab Login Page](../../images/access/gitlab-login.png)
 
@@ -34,8 +34,8 @@ From the command line, you can clone this repository by adding your username and
 
 To clone a repository using git over SSH, click on the "Code" button to get the git+ssh URL. You will need to have added an SSH key to your account for this method to work.
 
-For a more complete set of documentation relating to adding and using SSH keys with Gitlab, see the upstream [Gitlab Documentation](https://docs.gitlab.com/user/ssh/)
+For a more complete set of documentation relating to adding and using SSH keys with GitLab, see the upstream [GitLab Documentation](https://docs.gitlab.com/user/ssh/)
 
-## CI/CD Examples
+## Where to go next?
 
-A repository containing some common Gitlab CI/CD configurations and relevant examples is maintained at at [Gitlab CI/Cd Examples](https://gitlab.eidf.ac.uk/Liz/cicd-examples)
+CI/CD pipelines for automation of workflows are documented in the [GitLab CI/CD documentation](cicd.md).

docs/services/registry/working-with.md

@@ -1,4 +1,4 @@
-# Working with ECIR
+# Working with the EIDF Container Image Registry (ECIR)
 
 ## The Registry Interface and Accounts
 
@@ -13,6 +13,10 @@ User tokens can be accessed from the User Profile from the dropdown under your U
 
 See the FAQ section ['Unauthorised' error when logging into the registry from Docker](./faq.md#unauthorised-error-when-logging-into-the-registry-from-docker) for help with token expiry and authorisation issues.
 
+## ECIR Project
+
+Each EIDF project can have a private space on the registry, this is called an ECIR project. An ECIR project can be used to store images and artifacts that are private to the project. All projects should, by default, have an ECIR project created for them but if your project does not have one, you can request one by contacting the [EIDF Helpdesk](https://portal.eidf.ac.uk/queries/submit).
+
 ## Push Commands
 
 In your project, there is a PUSH Command option which will give you the command templates for pushing to the Project repositories from different clients.
@@ -62,19 +66,13 @@ https://docs.docker.com/engine/reference/commandline/login/#credentials-store
 Login Succeeded
 ```
 
-From your command line, you can now push and pull images to the registry.
-
-## Kubernetes/GPU Service Access
+Note that user tokens have a limited validity period and therefore, robot accounts are recommended for long-term use. Contact the [EIDF Helpdesk](https://portal.eidf.ac.uk/queries/submit) to create a robot account for your project.
 
-To pull images from the registry, from private or authenticated projects, you will need to add a secret to the namespace you are using and reference it in your job definition. Note that user tokens have a limited validity period.
+From your command line, you may now push and pull images to the registry using docker commands as described in the [docker documentation](https://docs.docker.com/reference/cli/docker/).
 
-If you are regularly using a repository from a project where you are sharing resources, it is recommended to create a robot account with limited read only privileges, this can be requested via a Helpdesk Request for your project.
-
-!!! important "Portal Management"
-
-    There will be new functionality soon added to the EIDF Portal to allow for project users to create read only robot accounts and for PI/Managers to create read/write robot accounts for use in CI/CD pipelines for image building.
+## Kubernetes/GPU Service Access
 
-This is then treated like a normal user secret when you have the robot credentials.
+To pull images from the registry, including from private or authenticated projects, you will need to add a secret to the namespace you are using and reference it in your job definition. Note that user tokens have a limited validity period and hence ECIR robot accounts are recommended for long-term use. Contact the [EIDF Helpdesk](https://portal.eidf.ac.uk/queries/submit) to create a robot account for your project.
 
 Secrets can be created in one of two ways, as detailed below, either directly via kubectl from your Docker config.json file, or by creating a YAML file.
 

mkdocs.yml

@@ -106,8 +106,8 @@ nav:
     #    - "Quickstart": services/mft/quickstart.md
     #    - "Using the MFT": services/mft/using-the-mft.md
     - "Code Collaboration":
-      - "Gitlab Overview": services/gitlab/index.md
-      - "Gitlab Quickstart": services/gitlab/quickstart.md
+      - "GitLab Overview": services/gitlab/index.md
+      - "GitLab Quickstart": services/gitlab/quickstart.md
       - "Container Image Registry Overview": services/registry/index.md
       - "Container Image Registry Projects and Caches": services/registry/projects.md
       - "Container Image Registry Use": services/registry/working-with.md