From c248cd9e8fec0e2e271156620f05931969ad0692 Mon Sep 17 00:00:00 2001 From: Nir Soffer Date: Thu, 20 Jun 2024 21:00:18 +0300 Subject: [PATCH] Introduce docs directory We want to add more workloads (stalefulset, daemonset), more storage (CephFS), and the new OCM discovered applications. Each variant require specific instructions or example commands so it is best done using one file per variant, where you can experiment with complete disaster recovery flow for a single variant. - Move the content we had in the main README.md to `docs/initial-setup.md` and `docs/ocm-rbd.md` - Move the content we had in `workloads/kubevirt/README.md` to `docs/ovm-kubevirt.md`. - Add `docs/create-environment.md` with short instructions how to create a testing environment for playing with the samples in this repo. - Change `README.md` to a short overview linking to all other documents. The new docs directory can be used next to create more structure documentation like https://docs.readthedocs.io. Signed-off-by: Nir Soffer --- README.md | 210 ++----------------- docs/create-environment.md | 217 ++++++++++++++++++++ docs/initial-setup.md | 48 +++++ docs/ocm-kubevirt.md | 387 +++++++++++++++++++++++++++++++++++ docs/ocm-rbd.md | 142 +++++++++++++ workloads/kubevirt/README.md | 377 +--------------------------------- 6 files changed, 812 insertions(+), 569 deletions(-) create mode 100644 docs/create-environment.md create mode 100644 docs/initial-setup.md create mode 100644 docs/ocm-kubevirt.md create mode 100644 docs/ocm-rbd.md diff --git a/README.md b/README.md index 90fc033c..968a11da 100644 --- a/README.md +++ b/README.md @@ -1,204 +1,24 @@ # ocm-ramen-samples -OCM Stateful application samples, including Ramen resources. +OCM Stateful application samples, including *Ramen* resources. -## Initial setup - -1. Clone this git repository to get started: - - ``` - git clone https://github.com/RamenDR/ocm-ramen-samples.git - cd ocm-ramen-samples - ``` - -1. Switch kubeconfig to point to the OCM Hub cluster - - ``` - kubectl config use-context hub - ``` - -1. Create DRClusters and DRPolicy - - When using the ramen testing environment this is not needed, but if - you are using your own Kubernetes clusters you need to create the - resources. - - Modify the DRCluster and DRpolicy resources in the `ramen` directory - to match the actual cluster names in your environment, and apply - the kustomization: - - ``` - kubectl apply -k ramen - ``` - - This creates DRPolicy and DRCluster resources in the cluster - namespace that can be viewed using: - - ``` - kubectl get drcluster,drpolicy - ``` - -1. Setup the common OCM channel resources on the hub: - - ``` - kubectl apply -k channel - ``` - - This creates a Channel resource in the `ramen-samples` namespace and - can be viewed using: - - ``` - kubectl get channel ramen-gitops -n ramen-samples - ``` - -## Sample applications - -In the workloads directory provides samples that can be deployed on -Kubernetes and OpenShift. - -- deployment - busybox deployment -- kubevirt - - vm-pvc - PVC based VM - - vm-dv - DataVolume based VM - - vm-dvt - DataVolumeTemplate based VM - -## Deploying a sample application - -In the example we use the busybox deployment for Kubernetes regional DR -environment using RBD storage: - - subscription/deployment-k8s-regional-rbd - -This application is deployed in the `deployment-rbd` namespace on the -hub and managed clusters. - -You can use other overlays to deploy on other cluster types or use -different storage class. You can also create your own overlays based on -the examples. - -1. Deploy an OCM application subscription on hub: - - ``` - kubectl apply -k subscription/deployment-k8s-regional-rbd - ``` - - This creates the required Subscription, Placement, and - ManagedClusterSetBinding resources for the deployment in the - `deployment-rbd` namespace and can be viewed using: +## Create an environment - ``` - kubectl get subscription,placement -n deployment-rbd - ``` +The easiest way to start is to use the *Ramen* testing environment +created by the *drenv* tool. See +[create environment](create-environment.md) to learn how to +quickly create a your disaster recovery playground. -1. Inspect subscribed resources from the channel created in the same - namespace on the ManagedCluster selected by the Placement. - - The busybox deployment Placement `status` can be viewed on the hub - using: - - ``` - kubectl get placement placement -n deployment-rbd - ``` - - The Busybox deployment subscribed resources, like the pod and the PVC - can be viewed on the ManagedCluster using (example ManagedCluster - `dr1`): - - ``` - kubectl get pod,pvc -n deployment-rbd --context dr1 - ``` - -## Undeploying a sample application - -To undeploy an application delete the subscription overlay used to -deploy the application: - -``` -kubectl delete -k subscription/deployment-k8s-regional-rbd -``` - -## Enable DR for a deployed application - -1. Change the Placement to be reconciled by Ramen - - ``` - kubectl annotate placement placement -n deployment-rbd \ - cluster.open-cluster-management.io/experimental-scheduling-disable=true - ``` - -1. Deploy a DRPlacementControl resource for the OCM application on the - hub, for example: - - ``` - kubectl apply -k dr/deployment-k8s-regional-rbd - ``` - - This creates a DRPlacementControl resource for the busybox deployment - in the `deployment-rbd` namespace and can be viewed using: - - ``` - kubectl get drpc -n deployment-rbd - ``` - - At this point the placement of the application is managed by *Ramen*. - -## Disable DR for a DR enabled application - -1. Delete the drpc resource for the OCM application on the hub: - - ``` - kubectl delete -k dr/deployment-k8s-regional-rbd - ``` - - This deletes the DRPlacementControl resource for the busybox - deployment, disabling replication and removing replicated data. - -> [!IMPORTANT] -> Do not delete the Placement annotation -> `cluster.open-cluster-management.io/experimental-scheduling-disable` -> to ensure that OCM will not change the placement of the application, -> which can result in data loss. - -### Optional: enabling OCM scheduling for the application - -It is not recommended to enable OCM scheduling on after disabling DR, -since OCM does not support moving workload storage between clusters. If -the placement point to wrong cluster, OCM will delete the application -and its storage from the current cluster, and deploy the application -with new storage on the cluster selected by the placement. - -Find the current placement of the application: - -``` -kubectl get placementdecisions -n deployment-rbd --context hub \ - -o jsonpath='{.items[0].status.decisions[0].clusterName}{"\n"}' -``` - -Ensure that the `Placement` predicates is pointing to the cluster where -the workload is currently placed. Here is example predicates selecting -the cluster `dr1`: +## Initial setup -``` -spec: - clusterSets: - - default - numberOfClusters: 1 - predicates: - - requiredClusterSelector: - claimSelector: {} - labelSelector: - matchExpressions: - - key: name - operator: In - values: - - dr1 -``` +Before experimenting with disaster recovery we need to configure the +clusters. See [initial setup](docs/initial-setup.md) to learn how to set +up your environment. -Change the Placement to be reconciled by OCM: +## Experiments -``` -kubectl annotate placement placement -n deployment-rbd \ - cluster.open-cluster-management.io/experimental-scheduling-disable- -``` +After setting up your environment you can experiments with various +workloads and storage types: -At this point the application is managed again by *OCM*. +- Experiment with *OCM* managed [deployment](docs/ocm-rbd.md) +- Experiment with *OCM* managed [virtual machine](docs/ocm-kubevirt.md) diff --git a/docs/create-environment.md b/docs/create-environment.md new file mode 100644 index 00000000..558c48eb --- /dev/null +++ b/docs/create-environment.md @@ -0,0 +1,217 @@ +# Creating a test environment + +This page will help you to set up an environment for experimenting with +disaster recovery. + +## What you’ll need + +- Bare metal or virtual machine with nested virtualization enabled +- 8 CPUs or more +- 20 GiB of free memory +- 100 GiB of free space +- Internet connection +- Linux - tested on *Fedora* 37, 38, and 39 +- non-root user with sudo privileges (all instructions are for non-root user) + +## Setting up your machine + +### Install libvirt + +Install the `@virtualization` group - on Fedora you can use: + +``` +sudo dnf install @virtualization +``` + +Enable the libvirtd service. + +``` +sudo systemctl enable libvirtd --now +``` + +Add yourself to the libvirt group (required for minikube kvm2 driver). + +``` +sudo usermod -a -G libvirt $(whoami) +``` + +Logout and login again for the change above to be in effect. + +### Install required packages + +On Fedora you can use: + +``` +sudo dnf install git make golang helm podman +``` + +### Clone *Ramen* source locally + +``` +git clone https://github.com/RamenDR/ramen.git +``` + +Enter the `ramen` directory - all the commands in this guide assume you +are in ramen root directory. + +``` +cd ramen +``` + +### Create a python virtual environment + +To keep the ramen tools separate from your host python, we create a +python virtual environment. + +``` +make venv +``` + +To activate the environment use: + +``` +source venv +``` + +To exit virtual environment issue command *deactivate*. + +### Installing required tools + +The drenv tool requires various tool for deploying the testing clusters. + +#### minikube + +On Fedora you can use: + +``` +sudo dnf install https://storage.googleapis.com/minikube/releases/latest/minikube-latest.x86_64.rpm +``` + +Tested with version v1.31.1. + +#### kubectl + +See [Install and Set Up kubectl on Linux](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/) +for details. + +Tested with version v1.30.2. + +#### clusteradm + +See [Install clusteradm CLI tool](https://open-cluster-management.io/getting-started/installation/start-the-control-plane/#install-clusteradm-cli-tool) +for the details. + +Version v0.81 or later is required. + +#### subctl + +See [Submariner subctl installation](https://submariner.io/operations/deployment/subctl/) +for the details. + +Version v0.17.0 or later is required. + +#### velero + +See [Velero Basic Install](https://velero.io/docs/v1.12/basic-install/) +for the details. + +Tested with version v1.12.2. + +#### virtctl + +``` +curl -L -o virtctl https://github.com/kubevirt/kubevirt/releases/download/v1.2.1/virtctl-v1.2.1-linux-amd64 +sudo install virtctl /usr/local/bin +rm virtctl +``` + +#### mc + +``` +curl -L -o mc https://dl.min.io/client/mc/release/linux-amd64/mc +sudo install mc /usr/local/bin +rm mc +``` + +For more info see +[MinIO Client Quickstart](https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart) + +#### kustomize + +``` +curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash +sudo install kustomize /usr/local/bin +rm kustomize +``` + +For more info see +[kustomize install](https://kubectl.docs.kubernetes.io/installation/kustomize/) + +#### argocd + +``` +curl -L -o argocd https://github.com/argoproj/argo-cd/releases/download/v2.11.3/argocd-linux-amd64 +sudo install argocd /usr/local/bin/ +rm argocd +``` + +For more info see [argocd installation](https://argo-cd.readthedocs.io/en/stable/cli_installation/) + +## Starting the test environment + +Before using the `drenv` tool to start a test environment, you need to +activate the python virtual environment: + +``` +source venv +``` + +Available environment files: + +- `envs/regional-dr.yaml` - regional dr for testing workloads using RBD and CephFS storage +- `envs/regional-dr-kubevirt.yaml` - regional dr for testing virtual machines using RBD storage + +To start a Regional-DR environment use: + +``` +(cd test; drenv start envs/regional-dr.yaml) +``` + +Starting the environment takes 8-15 minutes, depending on your machine +and internet connection. + +## Build the ramen operator image + +Build the *Ramen* operator container image: + +``` +make docker-build +``` + +> [!NOTE] +> Select `docker.io/library/golang:1.21` when prompted. + +This builds the image `quay.io/ramendr/ramen-operator:latest` + +## Deploy and Configure the ramen operator + +To deploy the *Ramen* operator in the test environment: + +``` +ramenctl deploy test/envs/regional-dr.yaml +ramenctl config test/envs/regional-dr.yaml +``` + +Your environment is ready! + +See [initial-setup](initial-setup.md) to learn how to set it up for +experimenting with disaster recovery. + +## Deleting the environment + +To stop and delete the minikube clusters use `drenv delete` with the +same environment file used to start the environment: + +``` +(cd test; drenv delete envs/regional-dr.yaml) +``` diff --git a/docs/initial-setup.md b/docs/initial-setup.md new file mode 100644 index 00000000..9681e189 --- /dev/null +++ b/docs/initial-setup.md @@ -0,0 +1,48 @@ +# Initial setup + +## Clone this git repository to get started: + +``` +git clone https://github.com/RamenDR/ocm-ramen-samples.git +cd ocm-ramen-samples +``` + +## Switch kubeconfig to point to the OCM Hub cluster + +``` +kubectl config use-context hub +``` + +## Create DRClusters and DRPolicy + +When using the ramen testing environment this is not needed, but if +you are using your own Kubernetes clusters you need to create the +resources. + +Modify the DRCluster and DRpolicy resources in the `ramen` directory +to match the actual cluster names in your environment, and apply +the kustomization: + +``` +kubectl apply -k ramen +``` + +This creates DRPolicy and DRCluster resources in the cluster +namespace that can be viewed using: + +``` +kubectl get drcluster,drpolicy +``` + +## Setup the common OCM channel resources on the hub: + +``` +kubectl apply -k channel +``` + +This creates a Channel resource in the `ramen-samples` namespace and +can be viewed using: + +``` +kubectl get channel ramen-gitops -n ramen-samples +``` diff --git a/docs/ocm-kubevirt.md b/docs/ocm-kubevirt.md new file mode 100644 index 00000000..71061bd5 --- /dev/null +++ b/docs/ocm-kubevirt.md @@ -0,0 +1,387 @@ +# Experiment with OCM managed virtual machines + +In the guide we experiment with an OCM managed PVC based virtual machine +on a regional DR environment: + + workloads/kubevirt/vm-pvc/k8s-regional + +This virtual machine is deployed in the `vm-pvc` namespace on the +hub and managed clusters. + +To experiment with other virtual machine configurations adapt the +commands to use `vm-dv` or `vm-dvt` instead of `vm-pvc`. + +## Setting up a testing environment + +When starting the testing environment, use the +`regional-dr-kubvirt.yaml` environment. + +## Adding your SSH key + +To connect to the test vm using `virtctl ssh`, you need to replace the +included SSH public key with your own public key. + +1. Fork this repo in github + +1. Replace the content of `workloads/kubevirt/vm-*/base/test_rsa.pub` + with the content of *your* public key (e.g. `~/.ssh/id_rsa.pub`) + +1. Update `channel/channel.yaml` to point to *your* repository + (e.g. https://github.com/MY-GITHUB-USER/ocm-ramen-samples.git) + +1. If you are not using the `main` branch update + `subscription/base/subscription.yaml` to point to the right branch. + +## Deploying the VM subscription + +To start a PVC based VM `vm-pvc` use the +`subscription/kubevirt/vm-pvc-k8s-regional` overlay: + +```sh +kubectl apply -k subscription/kubevirt/vm-pvc-k8s-regional --context hub +``` + +This subscription starts the VM `vm-pvc` in the namespace `vm-pvc` on +one of the clusters in the default clusterset. + +This overlay is customized for *Ramen* minikube based test environment. +To use in another setup you may want to create a new overlay. + +## Inspecting the VM status + +To find where the VM was placed look at the PlacementDecisions status: + +```sh +kubectl get placementdecisions -n vm-pvc --context hub \ + -o jsonpath='{.items[0].status.decisions[0].clusterName}{"\n"}' +``` + +To inspect the VM and DR resources use: + +```sh +watch -n 5 kubectl get vm,vmi,pod,pvc,vrg,vr -n vm-pvc --context dr1 +``` + +Example output: + +```console +Every 5.0s: kubectl get vm,vmi,pod,pvc,vrg,vr -n vm-pvc --context dr1 + +NAME AGE STATUS READY +virtualmachine.kubevirt.io/vm 5m51s Running True + +NAME AGE PHASE IP NODENAME READY +virtualmachineinstance.kubevirt.io/vm 5m51s Running 10.244.0.61 dr1 True + +NAME READY STATUS RESTARTS AGE +pod/virt-launcher-vm-chsrh 1/1 Running 0 5m51s + +NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE +persistentvolumeclaim/vm-pvc Bound pvc-149bd414-c1b8-443c-90f7-7e30fc519eb4 128Mi RWX rook-ceph-block 5m51s +``` + +At this point there there are no `vrg` and `vr` resources, since we did +not enable DR for the VM yet. + +We can inspect the underlying RBD image backing the VM PVC using the +`rook-ceph` kubectl krew plugin: + +``` +kubectl rook-ceph --context dr1 rbd du -p replicapool +``` + +Example output: + +```console +NAME PROVISIONED USED +csi-vol-a3fdb384-2e31-49f1-bd48-a97b2d79f981 128 MiB 80 MiB +``` + +## Enabling DR for the VM + +To allow *Ramen* to protect the VM, you need to disable *OCM* +scheduling by adding an annotation to the VM placement: + +```sh +kubectl annotate placement placement \ + cluster.open-cluster-management.io/experimental-scheduling-disable=true \ + --namespace vm-pvc \ + --context hub +``` + +Deploy the DR resources to enable DR: + +```sh +kubectl apply -k dr/kubevirt/vm-pvc-k8s-regional --context hub +``` + +At this point *Ramen* controls the VM placement and protects the VM data +by replicating it to the secondary cluster ("dr2"). + +To wait until the VM data is replicating to the secondary cluster, wait +for the `PeerReady` condition: + +```sh +kubectl wait drpc drpc \ + --for condition=PeerReady \ + --namespace vm-pvc \ + --timeout 5m \ + --context hub +``` + +## Inspecting the VM DR status + +We can inspect the VM DR status using the `DRPlacementControl` resource +on the hub cluster: + +```sh +watch -n 5 kubectl get drpc -n vm-pvc --context hub -o wide +``` + +Example output: + +```console +Every 5.0s: kubectl get drpc -n vm-pvc --context hub -o wide + +NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY +drpc 51s dr1 Deployed Completed 2023-11-19T20:26:53Z 5.035609263s True +``` + +To get more details we can watch the VM and DR resources on the managed +cluster: + +```sh +watch -n 5 kubectl get vm,vmi,pod,pvc,vrg,vr -n vm-pvc --context dr1 +``` + +Example output: + +```console +Every 5.0s: kubectl get vm,vmi,pod,pvc,vrg,vr -n vm-pvc --context dr1 + +NAME AGE STATUS READY +virtualmachine.kubevirt.io/vm 16m Running True + +NAME AGE PHASE IP NODENAME READY +virtualmachineinstance.kubevirt.io/vm 16m Running 10.244.0.61 dr1 True + +NAME READY STATUS RESTARTS AGE +pod/virt-launcher-vm-chsrh 1/1 Running 0 16m + +NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE +persistentvolumeclaim/vm-pvc Bound pvc-149bd414-c1b8-443c-90f7-7e30fc519eb4 128Mi RWX rook-ceph-block 16m + +NAME DESIREDSTATE CURRENTSTATE +volumereplicationgroup.ramendr.openshift.io/drpc primary Primary + +NAME AGE VOLUMEREPLICATIONCLASS PVCNAME DESIREDSTATE CURRENTSTATE +volumereplication.replication.storage.openshift.io/vm-pvc 2m46s vrc-sample vm-pvc primary Primary +``` + +The cirros VM used by this example includes a logger service logging a +message every 10 seconds: + +```sh +virtctl ssh cirros@vm -n vm-pvc --known-hosts= --context dr1 -c 'head /var/log/ramen.log' +``` + +Example output: + +```console +Sun Nov 19 20:13:20 UTC 2023 START uptime=3.33 +Sun Nov 19 20:13:30 UTC 2023 UPDATE +Sun Nov 19 20:13:40 UTC 2023 UPDATE +Sun Nov 19 20:13:50 UTC 2023 UPDATE +Sun Nov 19 20:14:00 UTC 2023 UPDATE +Sun Nov 19 20:14:10 UTC 2023 UPDATE +Sun Nov 19 20:14:20 UTC 2023 UPDATE +Sun Nov 19 20:14:30 UTC 2023 UPDATE +Sun Nov 19 20:14:40 UTC 2023 UPDATE +Sun Nov 19 20:14:50 UTC 2023 UPDATE +``` + +Ramen set up RBD mirroring for the underlying RBD image. The same image +is created on the secondary cluster ("dr2"): + +``` +kubectl rook-ceph --context dr2 rbd du -p replicapool +``` + +Example output: + +```console +NAME PROVISIONED USED +csi-vol-a3fdb384-2e31-49f1-bd48-a97b2d79f981 128 MiB 80 MiB +``` + +In case of a disaster in the primary cluster, we can start the VM using +the replica on the secondary cluster. + +## Failing over to another cluster + +In case of disaster you can force the VM to run on the other cluster. +The VM will start on the other cluster using the data from the last +replication. Data since the last replication is lost. + +To simulate a disaster we can pause the minkube VM running cluster +`dr1`: + +```sh +virsh -c qemu:///system suspend dr1 +``` + +To start `Failover` action, patch the VM `DRPlacementControl` resource +to set `action` and `failoverCluster`: + +```sh +kubectl patch drpc drpc \ + --patch '{"spec": {"action": "Failover", "failoverCluster": "dr2"}}' \ + --type merge \ + --namespace vm-pvc \ + --context hub +``` + +The VM will start on the failover cluster ("dr2"). Nothing will change +on the primary cluster ("dr1") since it is still paused. + +Inspecting the `/var/log/ramen.log` via SSH show how much data was lost +during the failover: + +```sh +virtctl ssh cirros@vm -n vm-pvc --known-hosts= --context dr2 -c 'tail -f /var/log/ramen.log' +``` + +Example output: + +```console +Sun Nov 19 20:33:42 UTC 2023 UPDATE +Sun Nov 19 20:33:53 UTC 2023 UPDATE +Sun Nov 19 20:34:03 UTC 2023 UPDATE +Sun Nov 19 20:34:13 UTC 2023 UPDATE +Sun Nov 19 20:34:23 UTC 2023 UPDATE +Sun Nov 19 20:34:33 UTC 2023 UPDATE +Sun Nov 19 20:34:43 UTC 2023 UPDATE +Sun Nov 19 20:34:53 UTC 2023 UPDATE +Sun Nov 19 20:37:43 UTC 2023 START uptime=3.18 +Sun Nov 19 20:37:53 UTC 2023 UPDATE +Sun Nov 19 20:38:03 UTC 2023 UPDATE +``` + +To enable replication from the secondary cluster to the primary cluster, +we need to recover the primary cluster. In this example we can resume +the minikube VM: + +```sh +virsh -c qemu:///system resume dr1 +``` + +*Ramen* will clean up the VM resources from the primary cluster and +enable RBD mirroring from the secondary cluster ("dr2") to the primary +cluster ("dr1"). + +To wait until the VM data is replicated again to the other cluster: + +```sh +kubectl wait drpc drpc \ + --for condition=PeerReady \ + --namespace vm-pvc \ + --timeout 5m \ + --context hub +``` + +Since the primary cluster is recovered, we can move the VM back to the +primary cluster. + +## Relocating to another cluster + +To move the VM back to the original cluster after a disaster +you can use the `Relocate` action. The VM will be terminated on +the current cluster, and started on the other cluster. No data is lost +during this operation. + +Patch the VM `DRPlacementControl` resource to set `action` and +if needed, `preferredCluster`. + +```sh +kubectl patch drpc drpc \ + --patch '{"spec": {"action": "Relocate", "preferredCluster": "dr1"}}' \ + --type merge \ + --namespace vm-pvc \ + --context hub +``` + +The VM will terminate on the current cluster and will start on the +preferred cluster. + +To wait until the VM is relocated to the primary cluster, wait until the +drpc phase is `Relocated`: + +```sh +kubectl wait drpc drpc \ + --for jsonpath='{.status.phase}=Relocated' \ + --namespace vm-pvc \ + --timeout 5m \ + --context hub +``` + +Inspecting `/var/log/ramen.log` shows that the VM was terminated cleanly +on the secondary cluster ("dr2") and started on the primary cluster +("dr1"). No data was lost! + +```sh +virtctl ssh cirros@vm -n vm-pvc --known-hosts= --context dr1 -c 'tail -f /var/log/ramen.log' +``` + +Example output: + +```console +Sun Nov 19 20:41:53 UTC 2023 UPDATE +Sun Nov 19 20:42:03 UTC 2023 UPDATE +Sun Nov 19 20:42:13 UTC 2023 UPDATE +Sun Nov 19 20:42:23 UTC 2023 UPDATE +Sun Nov 19 20:42:33 UTC 2023 UPDATE +Sun Nov 19 20:42:43 UTC 2023 UPDATE +Sun Nov 19 20:42:53 UTC 2023 UPDATE +Sun Nov 19 20:42:58 UTC 2023 STOP +Sun Nov 19 20:41:17 UTC 2023 START uptime=3.45 +Sun Nov 19 20:41:27 UTC 2023 UPDATE +Sun Nov 19 20:41:37 UTC 2023 UPDATE +``` + +To wait until the VM is replicating data again to the secondary cluster, +wait for the `PeerReady` condition: + +```sh +kubectl wait drpc drpc \ + --for condition=PeerReady \ + --namespace vm-pvc \ + --timeout 5m \ + --context hub +``` + +## Disable DR for the VM + +Delete the `dr` resources to disable DR: + +```sh +kubectl delete -k dr/kubevirt/vm-pvc-k8s-regional --context hub +``` + +At this point ramen is not protecting the virtual machine, and the +storage used for replicating the virtual machine data on the DR clusters +will be reclaimed. The virtual machine will continue to run on the +current cluster. + +> [!IMPORTANT] +> Do not delete the Placement annotation +> `cluster.open-cluster-management.io/experimental-scheduling-disable` +> to ensure that OCM will not change the placement of the virtual +> machine, which can result in data loss. + +## Undeploying the VM + +Delete the subscription to stop and delete the VM: + +```sh +kubectl delete -k subscription/kubevirt/vm-pvc-k8s-regional --context hub +``` diff --git a/docs/ocm-rbd.md b/docs/ocm-rbd.md new file mode 100644 index 00000000..35239466 --- /dev/null +++ b/docs/ocm-rbd.md @@ -0,0 +1,142 @@ +# Experiment with OCM managed application using RBD storage + +In the example we use the busybox deployment for Kubernetes regional DR +environment using RBD storage: + + subscription/deployment-k8s-regional-rbd + +This application is deployed in the `deployment-rbd` namespace on the +hub and managed clusters. + +You can use other overlays to deploy on other cluster types or use +different storage class. You can also create your own overlays based on +the examples. + +## Deploying a sample application + +1. Deploy an OCM application subscription on hub: + + ``` + kubectl apply -k subscription/deployment-k8s-regional-rbd + ``` + + This creates the required Subscription, Placement, and + ManagedClusterSetBinding resources for the deployment in the + `deployment-rbd` namespace and can be viewed using: + + ``` + kubectl get subscription,placement -n deployment-rbd + ``` + +1. Inspect subscribed resources from the channel created in the same + namespace on the ManagedCluster selected by the Placement. + + The busybox deployment Placement `status` can be viewed on the hub + using: + + ``` + kubectl get placement placement -n deployment-rbd + ``` + + The Busybox deployment subscribed resources, like the pod and the PVC + can be viewed on the ManagedCluster using (example ManagedCluster + `dr1`): + + ``` + kubectl get pod,pvc -n deployment-rbd --context dr1 + ``` + +## Undeploying a sample application + +To undeploy an application delete the subscription overlay used to +deploy the application: + +``` +kubectl delete -k subscription/deployment-k8s-regional-rbd +``` + +## Enable DR for a deployed application + +1. Change the Placement to be reconciled by Ramen + + ``` + kubectl annotate placement placement -n deployment-rbd \ + cluster.open-cluster-management.io/experimental-scheduling-disable=true + ``` + +1. Deploy a DRPlacementControl resource for the OCM application on the + hub, for example: + + ``` + kubectl apply -k dr/deployment-k8s-regional-rbd + ``` + + This creates a DRPlacementControl resource for the busybox deployment + in the `deployment-rbd` namespace and can be viewed using: + + ``` + kubectl get drpc -n deployment-rbd + ``` + + At this point the placement of the application is managed by *Ramen*. + +## Disable DR for a DR enabled application + +1. Delete the drpc resource for the OCM application on the hub: + + ``` + kubectl delete -k dr/deployment-k8s-regional-rbd + ``` + + This deletes the DRPlacementControl resource for the busybox + deployment, disabling replication and removing replicated data. + +> [!IMPORTANT] +> Do not delete the Placement annotation +> `cluster.open-cluster-management.io/experimental-scheduling-disable` +> to ensure that OCM will not change the placement of the application, +> which can result in data loss. + +### Optional: enabling OCM scheduling for the application + +It is not recommended to enable OCM scheduling on after disabling DR, +since OCM does not support moving workload storage between clusters. If +the placement point to wrong cluster, OCM will delete the application +and its storage from the current cluster, and deploy the application +with new storage on the cluster selected by the placement. + +Find the current placement of the application: + +``` +kubectl get placementdecisions -n deployment-rbd --context hub \ + -o jsonpath='{.items[0].status.decisions[0].clusterName}{"\n"}' +``` + +Ensure that the `Placement` predicates is pointing to the cluster where +the workload is currently placed. Here is example predicates selecting +the cluster `dr1`: + +``` +spec: + clusterSets: + - default + numberOfClusters: 1 + predicates: + - requiredClusterSelector: + claimSelector: {} + labelSelector: + matchExpressions: + - key: name + operator: In + values: + - dr1 +``` + +Change the Placement to be reconciled by OCM: + +``` +kubectl annotate placement placement -n deployment-rbd \ + cluster.open-cluster-management.io/experimental-scheduling-disable- +``` + +At this point the application is managed again by *OCM*. diff --git a/workloads/kubevirt/README.md b/workloads/kubevirt/README.md index eacadb82..070e3e44 100644 --- a/workloads/kubevirt/README.md +++ b/workloads/kubevirt/README.md @@ -3,377 +3,6 @@ This directory contains sample kubevirt resources for deployment in Kubernetes or OpenShift clusters. -## Setting up a testing environment - -When starting the testing environment, use the -`regional-dr-kubvirt.yaml` configuration. - -## Adding your SSH key - -If you want to inject your SSH key into the VM, you need to replace -the included SSH public key with your own public key. - -1. Fork this repo in github - -1. Replace the content of `workloads/kubevirt/vm-*/base/test_rsa.pub` - with the content of *your* public key (e.g. `~/.ssh/id_rsa.pub`) - -1. Update `channel/channel.yaml` to point to *your* repository - (e.g. https://github.com/MY-GITHUB-USER/ocm-ramen-samples.git) - -1. If you are not using the `main` branch update - `subscription/base/subscription.yaml` to point to the right branch. - -## Deploying the VM subscription - -To start a PVC based VM `vm-pvc` use the -`subscription/kubevirt/vm-pvc-k8s-regional` overlay: - -```sh -kubectl apply -k subscription/kubevirt/vm-pvc-k8s-regional --context hub -``` - -This subscription starts the VM `vm-pvc` in the namespace `vm-pvc` on -one of the clusters in the default clusterset. - -This overlay is customized for *Ramen* minikube based test environment. -To use in another setup you may want to create a new overlay. - -## Inspecting the VM status - -To find where the VM was placed look at the PlacementDecisions status: - -```sh -kubectl get placementdecisions -n vm-pvc --context hub \ - -o jsonpath='{.items[0].status.decisions[0].clusterName}{"\n"}' -``` - -To inspect the VM and DR resources use: - -```sh -watch -n 5 kubectl get vm,vmi,pod,pvc,vrg,vr -n vm-pvc --context dr1 -``` - -Example output: - -```console -Every 5.0s: kubectl get vm,vmi,pod,pvc,vrg,vr -n vm-pvc --context dr1 - -NAME AGE STATUS READY -virtualmachine.kubevirt.io/vm 5m51s Running True - -NAME AGE PHASE IP NODENAME READY -virtualmachineinstance.kubevirt.io/vm 5m51s Running 10.244.0.61 dr1 True - -NAME READY STATUS RESTARTS AGE -pod/virt-launcher-vm-chsrh 1/1 Running 0 5m51s - -NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE -persistentvolumeclaim/vm-pvc Bound pvc-149bd414-c1b8-443c-90f7-7e30fc519eb4 128Mi RWX rook-ceph-block 5m51s -``` - -At this point there there are no `vrg` and `vr` resources, since we did -not enable DR for the VM yet. - -We can inspect the underlying RBD image backing the VM PVC using the -`rook-ceph` kubectl krew plugin: - -``` -kubectl rook-ceph --context dr1 rbd du -p replicapool -``` - -Example output: - -```console -NAME PROVISIONED USED -csi-vol-a3fdb384-2e31-49f1-bd48-a97b2d79f981 128 MiB 80 MiB -``` - -## Enabling DR for the VM - -To allow *Ramen* to protect the VM, you need to disable *OCM* -scheduling by adding an annotation to the VM placement: - -```sh -kubectl annotate placement placement \ - cluster.open-cluster-management.io/experimental-scheduling-disable=true \ - --namespace vm-pvc \ - --context hub -``` - -Deploy the DR resources to enable DR: - -```sh -kubectl apply -k dr/kubevirt/vm-pvc-k8s-regional --context hub -``` - -At this point *Ramen* controls the VM placement and protects the VM data -by replicating it to the secondary cluster ("dr2"). - -To wait until the VM data is replicating to the secondary cluster, wait -for the `PeerReady` condition: - -```sh -kubectl wait drpc drpc \ - --for condition=PeerReady \ - --namespace vm-pvc \ - --timeout 5m \ - --context hub -``` - -## Inspecting the VM DR status - -We can inspect the VM DR status using the `DRPlacementControl` resource -on the hub cluster: - -```sh -watch -n 5 kubectl get drpc -n vm-pvc --context hub -o wide -``` - -Example output: - -```console -Every 5.0s: kubectl get drpc -n vm-pvc --context hub -o wide - -NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY -drpc 51s dr1 Deployed Completed 2023-11-19T20:26:53Z 5.035609263s True -``` - -To get more details we can watch the VM and DR resources on the managed -cluster: - -```sh -watch -n 5 kubectl get vm,vmi,pod,pvc,vrg,vr -n vm-pvc --context dr1 -``` - -Example output: - -```console -Every 5.0s: kubectl get vm,vmi,pod,pvc,vrg,vr -n vm-pvc --context dr1 - -NAME AGE STATUS READY -virtualmachine.kubevirt.io/vm 16m Running True - -NAME AGE PHASE IP NODENAME READY -virtualmachineinstance.kubevirt.io/vm 16m Running 10.244.0.61 dr1 True - -NAME READY STATUS RESTARTS AGE -pod/virt-launcher-vm-chsrh 1/1 Running 0 16m - -NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE -persistentvolumeclaim/vm-pvc Bound pvc-149bd414-c1b8-443c-90f7-7e30fc519eb4 128Mi RWX rook-ceph-block 16m - -NAME DESIREDSTATE CURRENTSTATE -volumereplicationgroup.ramendr.openshift.io/drpc primary Primary - -NAME AGE VOLUMEREPLICATIONCLASS PVCNAME DESIREDSTATE CURRENTSTATE -volumereplication.replication.storage.openshift.io/vm-pvc 2m46s vrc-sample vm-pvc primary Primary -``` - -The cirros VM used by this example includes a logger service logging a -message every 10 seconds: - -```sh -virtctl ssh cirros@vm -n vm-pvc --known-hosts= --context dr1 -c 'head /var/log/ramen.log' -``` - -Example output: - -```console -Sun Nov 19 20:13:20 UTC 2023 START uptime=3.33 -Sun Nov 19 20:13:30 UTC 2023 UPDATE -Sun Nov 19 20:13:40 UTC 2023 UPDATE -Sun Nov 19 20:13:50 UTC 2023 UPDATE -Sun Nov 19 20:14:00 UTC 2023 UPDATE -Sun Nov 19 20:14:10 UTC 2023 UPDATE -Sun Nov 19 20:14:20 UTC 2023 UPDATE -Sun Nov 19 20:14:30 UTC 2023 UPDATE -Sun Nov 19 20:14:40 UTC 2023 UPDATE -Sun Nov 19 20:14:50 UTC 2023 UPDATE -``` - -Ramen set up RBD mirroring for the underlying RBD image. The same image -is created on the secondary cluster ("dr2"): - -``` -kubectl rook-ceph --context dr2 rbd du -p replicapool -``` - -Example output: - -```console -NAME PROVISIONED USED -csi-vol-a3fdb384-2e31-49f1-bd48-a97b2d79f981 128 MiB 80 MiB -``` - -In case of a disaster in the primary cluster, we can start the VM using -the replica on the secondary cluster. - -## Failing over to another cluster - -In case of disaster you can force the VM to run on the other cluster. -The VM will start on the other cluster using the data from the last -replication. Data since the last replication is lost. - -To simulate a disaster we can pause the minkube VM running cluster -`dr1`: - -```sh -virsh -c qemu:///system suspend dr1 -``` - -To start `Failover` action, patch the VM `DRPlacementControl` resource -to set `action` and `failoverCluster`: - -```sh -kubectl patch drpc drpc \ - --patch '{"spec": {"action": "Failover", "failoverCluster": "dr2"}}' \ - --type merge \ - --namespace vm-pvc \ - --context hub -``` - -The VM will start on the failover cluster ("dr2"). Nothing will change -on the primary cluster ("dr1") since it is still paused. - -Inspecting the `/var/log/ramen.log` via SSH show how much data was lost -during the failover: - -```sh -virtctl ssh cirros@vm -n vm-pvc --known-hosts= --context dr2 -c 'tail -f /var/log/ramen.log' -``` - -Example output: - -```console -Sun Nov 19 20:33:42 UTC 2023 UPDATE -Sun Nov 19 20:33:53 UTC 2023 UPDATE -Sun Nov 19 20:34:03 UTC 2023 UPDATE -Sun Nov 19 20:34:13 UTC 2023 UPDATE -Sun Nov 19 20:34:23 UTC 2023 UPDATE -Sun Nov 19 20:34:33 UTC 2023 UPDATE -Sun Nov 19 20:34:43 UTC 2023 UPDATE -Sun Nov 19 20:34:53 UTC 2023 UPDATE -Sun Nov 19 20:37:43 UTC 2023 START uptime=3.18 -Sun Nov 19 20:37:53 UTC 2023 UPDATE -Sun Nov 19 20:38:03 UTC 2023 UPDATE -``` - -To enable replication from the secondary cluster to the primary cluster, -we need to recover the primary cluster. In this example we can resume -the minikube VM: - -```sh -virsh -c qemu:///system resume dr1 -``` - -*Ramen* will clean up the VM resources from the primary cluster and -enable RBD mirroring from the secondary cluster ("dr2") to the primary -cluster ("dr1"). - -To wait until the VM data is replicated again to the other cluster: - -```sh -kubectl wait drpc drpc \ - --for condition=PeerReady \ - --namespace vm-pvc \ - --timeout 5m \ - --context hub -``` - -Since the primary cluster is recovered, we can move the VM back to the -primary cluster. - -## Relocating to another cluster - -To move the VM back to the original cluster after a disaster -you can use the `Relocate` action. The VM will be terminated on -the current cluster, and started on the other cluster. No data is lost -during this operation. - -Patch the VM `DRPlacementControl` resource to set `action` and -if needed, `preferredCluster`. - -```sh -kubectl patch drpc drpc \ - --patch '{"spec": {"action": "Relocate", "preferredCluster": "dr1"}}' \ - --type merge \ - --namespace vm-pvc \ - --context hub -``` - -The VM will terminate on the current cluster and will start on the -preferred cluster. - -To wait until the VM is relocated to the primary cluster, wait until the -drpc phase is `Relocated`: - -```sh -kubectl wait drpc drpc \ - --for jsonpath='{.status.phase}=Relocated' \ - --namespace vm-pvc \ - --timeout 5m \ - --context hub -``` - -Inspecting `/var/log/ramen.log` shows that the VM was terminated cleanly -on the secondary cluster ("dr2") and started on the primary cluster -("dr1"). No data was lost! - -```sh -virtctl ssh cirros@vm -n vm-pvc --known-hosts= --context dr1 -c 'tail -f /var/log/ramen.log' -``` - -Example output: - -```console -Sun Nov 19 20:41:53 UTC 2023 UPDATE -Sun Nov 19 20:42:03 UTC 2023 UPDATE -Sun Nov 19 20:42:13 UTC 2023 UPDATE -Sun Nov 19 20:42:23 UTC 2023 UPDATE -Sun Nov 19 20:42:33 UTC 2023 UPDATE -Sun Nov 19 20:42:43 UTC 2023 UPDATE -Sun Nov 19 20:42:53 UTC 2023 UPDATE -Sun Nov 19 20:42:58 UTC 2023 STOP -Sun Nov 19 20:41:17 UTC 2023 START uptime=3.45 -Sun Nov 19 20:41:27 UTC 2023 UPDATE -Sun Nov 19 20:41:37 UTC 2023 UPDATE -``` - -To wait until the VM is replicating data again to the secondary cluster, -wait for the `PeerReady` condition: - -```sh -kubectl wait drpc drpc \ - --for condition=PeerReady \ - --namespace vm-pvc \ - --timeout 5m \ - --context hub -``` - -## Disable DR for the VM - -Delete the `dr` resources to disable DR: - -```sh -kubectl delete -k dr/kubevirt/vm-pvc-k8s-regional --context hub -``` - -At this point ramen is not protecting the virtual machine, and the -storage used for replicating the virtual machine data on the DR clusters -will be reclaimed. The virtual machine will continue to run on the -current cluster. - -> [!IMPORTANT] -> Do not delete the Placement annotation -> `cluster.open-cluster-management.io/experimental-scheduling-disable` -> to ensure that OCM will not change the placement of the virtual -> machine, which can result in data loss. - -## Undeploying the VM - -Delete the subscription to stop and delete the VM: - -```sh -kubectl delete -k subscription/kubevirt/vm-pvc-k8s-regional --context hub -``` +- vm-pvc - PVC based VM +- vm-dv - DataVolume based VM +- vm-dvt - DataVolumeTemplate based VM