diff --git a/README.md b/README.md index 90fc033c..968a11da 100644 --- a/README.md +++ b/README.md @@ -1,204 +1,24 @@ # ocm-ramen-samples -OCM Stateful application samples, including Ramen resources. +OCM Stateful application samples, including *Ramen* resources. -## Initial setup - -1. Clone this git repository to get started: - - ``` - git clone https://github.com/RamenDR/ocm-ramen-samples.git - cd ocm-ramen-samples - ``` - -1. Switch kubeconfig to point to the OCM Hub cluster - - ``` - kubectl config use-context hub - ``` - -1. Create DRClusters and DRPolicy - - When using the ramen testing environment this is not needed, but if - you are using your own Kubernetes clusters you need to create the - resources. - - Modify the DRCluster and DRpolicy resources in the `ramen` directory - to match the actual cluster names in your environment, and apply - the kustomization: - - ``` - kubectl apply -k ramen - ``` - - This creates DRPolicy and DRCluster resources in the cluster - namespace that can be viewed using: - - ``` - kubectl get drcluster,drpolicy - ``` - -1. Setup the common OCM channel resources on the hub: - - ``` - kubectl apply -k channel - ``` - - This creates a Channel resource in the `ramen-samples` namespace and - can be viewed using: - - ``` - kubectl get channel ramen-gitops -n ramen-samples - ``` - -## Sample applications - -In the workloads directory provides samples that can be deployed on -Kubernetes and OpenShift. - -- deployment - busybox deployment -- kubevirt - - vm-pvc - PVC based VM - - vm-dv - DataVolume based VM - - vm-dvt - DataVolumeTemplate based VM - -## Deploying a sample application - -In the example we use the busybox deployment for Kubernetes regional DR -environment using RBD storage: - - subscription/deployment-k8s-regional-rbd - -This application is deployed in the `deployment-rbd` namespace on the -hub and managed clusters. - -You can use other overlays to deploy on other cluster types or use -different storage class. You can also create your own overlays based on -the examples. - -1. Deploy an OCM application subscription on hub: - - ``` - kubectl apply -k subscription/deployment-k8s-regional-rbd - ``` - - This creates the required Subscription, Placement, and - ManagedClusterSetBinding resources for the deployment in the - `deployment-rbd` namespace and can be viewed using: +## Create an environment - ``` - kubectl get subscription,placement -n deployment-rbd - ``` +The easiest way to start is to use the *Ramen* testing environment +created by the *drenv* tool. See +[create environment](create-environment.md) to learn how to +quickly create a your disaster recovery playground. -1. Inspect subscribed resources from the channel created in the same - namespace on the ManagedCluster selected by the Placement. - - The busybox deployment Placement `status` can be viewed on the hub - using: - - ``` - kubectl get placement placement -n deployment-rbd - ``` - - The Busybox deployment subscribed resources, like the pod and the PVC - can be viewed on the ManagedCluster using (example ManagedCluster - `dr1`): - - ``` - kubectl get pod,pvc -n deployment-rbd --context dr1 - ``` - -## Undeploying a sample application - -To undeploy an application delete the subscription overlay used to -deploy the application: - -``` -kubectl delete -k subscription/deployment-k8s-regional-rbd -``` - -## Enable DR for a deployed application - -1. Change the Placement to be reconciled by Ramen - - ``` - kubectl annotate placement placement -n deployment-rbd \ - cluster.open-cluster-management.io/experimental-scheduling-disable=true - ``` - -1. Deploy a DRPlacementControl resource for the OCM application on the - hub, for example: - - ``` - kubectl apply -k dr/deployment-k8s-regional-rbd - ``` - - This creates a DRPlacementControl resource for the busybox deployment - in the `deployment-rbd` namespace and can be viewed using: - - ``` - kubectl get drpc -n deployment-rbd - ``` - - At this point the placement of the application is managed by *Ramen*. - -## Disable DR for a DR enabled application - -1. Delete the drpc resource for the OCM application on the hub: - - ``` - kubectl delete -k dr/deployment-k8s-regional-rbd - ``` - - This deletes the DRPlacementControl resource for the busybox - deployment, disabling replication and removing replicated data. - -> [!IMPORTANT] -> Do not delete the Placement annotation -> `cluster.open-cluster-management.io/experimental-scheduling-disable` -> to ensure that OCM will not change the placement of the application, -> which can result in data loss. - -### Optional: enabling OCM scheduling for the application - -It is not recommended to enable OCM scheduling on after disabling DR, -since OCM does not support moving workload storage between clusters. If -the placement point to wrong cluster, OCM will delete the application -and its storage from the current cluster, and deploy the application -with new storage on the cluster selected by the placement. - -Find the current placement of the application: - -``` -kubectl get placementdecisions -n deployment-rbd --context hub \ - -o jsonpath='{.items[0].status.decisions[0].clusterName}{"\n"}' -``` - -Ensure that the `Placement` predicates is pointing to the cluster where -the workload is currently placed. Here is example predicates selecting -the cluster `dr1`: +## Initial setup -``` -spec: - clusterSets: - - default - numberOfClusters: 1 - predicates: - - requiredClusterSelector: - claimSelector: {} - labelSelector: - matchExpressions: - - key: name - operator: In - values: - - dr1 -``` +Before experimenting with disaster recovery we need to configure the +clusters. See [initial setup](docs/initial-setup.md) to learn how to set +up your environment. -Change the Placement to be reconciled by OCM: +## Experiments -``` -kubectl annotate placement placement -n deployment-rbd \ - cluster.open-cluster-management.io/experimental-scheduling-disable- -``` +After setting up your environment you can experiments with various +workloads and storage types: -At this point the application is managed again by *OCM*. +- Experiment with *OCM* managed [deployment](docs/ocm-rbd.md) +- Experiment with *OCM* managed [virtual machine](docs/ocm-kubevirt.md) diff --git a/docs/create-environment.md b/docs/create-environment.md new file mode 100644 index 00000000..558c48eb --- /dev/null +++ b/docs/create-environment.md @@ -0,0 +1,217 @@ +# Creating a test environment + +This page will help you to set up an environment for experimenting with +disaster recovery. + +## What you’ll need + +- Bare metal or virtual machine with nested virtualization enabled +- 8 CPUs or more +- 20 GiB of free memory +- 100 GiB of free space +- Internet connection +- Linux - tested on *Fedora* 37, 38, and 39 +- non-root user with sudo privileges (all instructions are for non-root user) + +## Setting up your machine + +### Install libvirt + +Install the `@virtualization` group - on Fedora you can use: + +``` +sudo dnf install @virtualization +``` + +Enable the libvirtd service. + +``` +sudo systemctl enable libvirtd --now +``` + +Add yourself to the libvirt group (required for minikube kvm2 driver). + +``` +sudo usermod -a -G libvirt $(whoami) +``` + +Logout and login again for the change above to be in effect. + +### Install required packages + +On Fedora you can use: + +``` +sudo dnf install git make golang helm podman +``` + +### Clone *Ramen* source locally + +``` +git clone https://github.com/RamenDR/ramen.git +``` + +Enter the `ramen` directory - all the commands in this guide assume you +are in ramen root directory. + +``` +cd ramen +``` + +### Create a python virtual environment + +To keep the ramen tools separate from your host python, we create a +python virtual environment. + +``` +make venv +``` + +To activate the environment use: + +``` +source venv +``` + +To exit virtual environment issue command *deactivate*. + +### Installing required tools + +The drenv tool requires various tool for deploying the testing clusters. + +#### minikube + +On Fedora you can use: + +``` +sudo dnf install https://storage.googleapis.com/minikube/releases/latest/minikube-latest.x86_64.rpm +``` + +Tested with version v1.31.1. + +#### kubectl + +See [Install and Set Up kubectl on Linux](https://kubernetes.io/docs/tasks/tools/install-kubectl-linux/) +for details. + +Tested with version v1.30.2. + +#### clusteradm + +See [Install clusteradm CLI tool](https://open-cluster-management.io/getting-started/installation/start-the-control-plane/#install-clusteradm-cli-tool) +for the details. + +Version v0.81 or later is required. + +#### subctl + +See [Submariner subctl installation](https://submariner.io/operations/deployment/subctl/) +for the details. + +Version v0.17.0 or later is required. + +#### velero + +See [Velero Basic Install](https://velero.io/docs/v1.12/basic-install/) +for the details. + +Tested with version v1.12.2. + +#### virtctl + +``` +curl -L -o virtctl https://github.com/kubevirt/kubevirt/releases/download/v1.2.1/virtctl-v1.2.1-linux-amd64 +sudo install virtctl /usr/local/bin +rm virtctl +``` + +#### mc + +``` +curl -L -o mc https://dl.min.io/client/mc/release/linux-amd64/mc +sudo install mc /usr/local/bin +rm mc +``` + +For more info see +[MinIO Client Quickstart](https://min.io/docs/minio/linux/reference/minio-mc.html#quickstart) + +#### kustomize + +``` +curl -s "https://raw.githubusercontent.com/kubernetes-sigs/kustomize/master/hack/install_kustomize.sh" | bash +sudo install kustomize /usr/local/bin +rm kustomize +``` + +For more info see +[kustomize install](https://kubectl.docs.kubernetes.io/installation/kustomize/) + +#### argocd + +``` +curl -L -o argocd https://github.com/argoproj/argo-cd/releases/download/v2.11.3/argocd-linux-amd64 +sudo install argocd /usr/local/bin/ +rm argocd +``` + +For more info see [argocd installation](https://argo-cd.readthedocs.io/en/stable/cli_installation/) + +## Starting the test environment + +Before using the `drenv` tool to start a test environment, you need to +activate the python virtual environment: + +``` +source venv +``` + +Available environment files: + +- `envs/regional-dr.yaml` - regional dr for testing workloads using RBD and CephFS storage +- `envs/regional-dr-kubevirt.yaml` - regional dr for testing virtual machines using RBD storage + +To start a Regional-DR environment use: + +``` +(cd test; drenv start envs/regional-dr.yaml) +``` + +Starting the environment takes 8-15 minutes, depending on your machine +and internet connection. + +## Build the ramen operator image + +Build the *Ramen* operator container image: + +``` +make docker-build +``` + +> [!NOTE] +> Select `docker.io/library/golang:1.21` when prompted. + +This builds the image `quay.io/ramendr/ramen-operator:latest` + +## Deploy and Configure the ramen operator + +To deploy the *Ramen* operator in the test environment: + +``` +ramenctl deploy test/envs/regional-dr.yaml +ramenctl config test/envs/regional-dr.yaml +``` + +Your environment is ready! + +See [initial-setup](initial-setup.md) to learn how to set it up for +experimenting with disaster recovery. + +## Deleting the environment + +To stop and delete the minikube clusters use `drenv delete` with the +same environment file used to start the environment: + +``` +(cd test; drenv delete envs/regional-dr.yaml) +``` diff --git a/docs/initial-setup.md b/docs/initial-setup.md new file mode 100644 index 00000000..9681e189 --- /dev/null +++ b/docs/initial-setup.md @@ -0,0 +1,48 @@ +# Initial setup + +## Clone this git repository to get started: + +``` +git clone https://github.com/RamenDR/ocm-ramen-samples.git +cd ocm-ramen-samples +``` + +## Switch kubeconfig to point to the OCM Hub cluster + +``` +kubectl config use-context hub +``` + +## Create DRClusters and DRPolicy + +When using the ramen testing environment this is not needed, but if +you are using your own Kubernetes clusters you need to create the +resources. + +Modify the DRCluster and DRpolicy resources in the `ramen` directory +to match the actual cluster names in your environment, and apply +the kustomization: + +``` +kubectl apply -k ramen +``` + +This creates DRPolicy and DRCluster resources in the cluster +namespace that can be viewed using: + +``` +kubectl get drcluster,drpolicy +``` + +## Setup the common OCM channel resources on the hub: + +``` +kubectl apply -k channel +``` + +This creates a Channel resource in the `ramen-samples` namespace and +can be viewed using: + +``` +kubectl get channel ramen-gitops -n ramen-samples +``` diff --git a/docs/ocm-kubevirt.md b/docs/ocm-kubevirt.md new file mode 100644 index 00000000..71061bd5 --- /dev/null +++ b/docs/ocm-kubevirt.md @@ -0,0 +1,387 @@ +# Experiment with OCM managed virtual machines + +In the guide we experiment with an OCM managed PVC based virtual machine +on a regional DR environment: + + workloads/kubevirt/vm-pvc/k8s-regional + +This virtual machine is deployed in the `vm-pvc` namespace on the +hub and managed clusters. + +To experiment with other virtual machine configurations adapt the +commands to use `vm-dv` or `vm-dvt` instead of `vm-pvc`. + +## Setting up a testing environment + +When starting the testing environment, use the +`regional-dr-kubvirt.yaml` environment. + +## Adding your SSH key + +To connect to the test vm using `virtctl ssh`, you need to replace the +included SSH public key with your own public key. + +1. Fork this repo in github + +1. Replace the content of `workloads/kubevirt/vm-*/base/test_rsa.pub` + with the content of *your* public key (e.g. `~/.ssh/id_rsa.pub`) + +1. Update `channel/channel.yaml` to point to *your* repository + (e.g. https://github.com/MY-GITHUB-USER/ocm-ramen-samples.git) + +1. If you are not using the `main` branch update + `subscription/base/subscription.yaml` to point to the right branch. + +## Deploying the VM subscription + +To start a PVC based VM `vm-pvc` use the +`subscription/kubevirt/vm-pvc-k8s-regional` overlay: + +```sh +kubectl apply -k subscription/kubevirt/vm-pvc-k8s-regional --context hub +``` + +This subscription starts the VM `vm-pvc` in the namespace `vm-pvc` on +one of the clusters in the default clusterset. + +This overlay is customized for *Ramen* minikube based test environment. +To use in another setup you may want to create a new overlay. + +## Inspecting the VM status + +To find where the VM was placed look at the PlacementDecisions status: + +```sh +kubectl get placementdecisions -n vm-pvc --context hub \ + -o jsonpath='{.items[0].status.decisions[0].clusterName}{"\n"}' +``` + +To inspect the VM and DR resources use: + +```sh +watch -n 5 kubectl get vm,vmi,pod,pvc,vrg,vr -n vm-pvc --context dr1 +``` + +Example output: + +```console +Every 5.0s: kubectl get vm,vmi,pod,pvc,vrg,vr -n vm-pvc --context dr1 + +NAME AGE STATUS READY +virtualmachine.kubevirt.io/vm 5m51s Running True + +NAME AGE PHASE IP NODENAME READY +virtualmachineinstance.kubevirt.io/vm 5m51s Running 10.244.0.61 dr1 True + +NAME READY STATUS RESTARTS AGE +pod/virt-launcher-vm-chsrh 1/1 Running 0 5m51s + +NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE +persistentvolumeclaim/vm-pvc Bound pvc-149bd414-c1b8-443c-90f7-7e30fc519eb4 128Mi RWX rook-ceph-block 5m51s +``` + +At this point there there are no `vrg` and `vr` resources, since we did +not enable DR for the VM yet. + +We can inspect the underlying RBD image backing the VM PVC using the +`rook-ceph` kubectl krew plugin: + +``` +kubectl rook-ceph --context dr1 rbd du -p replicapool +``` + +Example output: + +```console +NAME PROVISIONED USED +csi-vol-a3fdb384-2e31-49f1-bd48-a97b2d79f981 128 MiB 80 MiB +``` + +## Enabling DR for the VM + +To allow *Ramen* to protect the VM, you need to disable *OCM* +scheduling by adding an annotation to the VM placement: + +```sh +kubectl annotate placement placement \ + cluster.open-cluster-management.io/experimental-scheduling-disable=true \ + --namespace vm-pvc \ + --context hub +``` + +Deploy the DR resources to enable DR: + +```sh +kubectl apply -k dr/kubevirt/vm-pvc-k8s-regional --context hub +``` + +At this point *Ramen* controls the VM placement and protects the VM data +by replicating it to the secondary cluster ("dr2"). + +To wait until the VM data is replicating to the secondary cluster, wait +for the `PeerReady` condition: + +```sh +kubectl wait drpc drpc \ + --for condition=PeerReady \ + --namespace vm-pvc \ + --timeout 5m \ + --context hub +``` + +## Inspecting the VM DR status + +We can inspect the VM DR status using the `DRPlacementControl` resource +on the hub cluster: + +```sh +watch -n 5 kubectl get drpc -n vm-pvc --context hub -o wide +``` + +Example output: + +```console +Every 5.0s: kubectl get drpc -n vm-pvc --context hub -o wide + +NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY +drpc 51s dr1 Deployed Completed 2023-11-19T20:26:53Z 5.035609263s True +``` + +To get more details we can watch the VM and DR resources on the managed +cluster: + +```sh +watch -n 5 kubectl get vm,vmi,pod,pvc,vrg,vr -n vm-pvc --context dr1 +``` + +Example output: + +```console +Every 5.0s: kubectl get vm,vmi,pod,pvc,vrg,vr -n vm-pvc --context dr1 + +NAME AGE STATUS READY +virtualmachine.kubevirt.io/vm 16m Running True + +NAME AGE PHASE IP NODENAME READY +virtualmachineinstance.kubevirt.io/vm 16m Running 10.244.0.61 dr1 True + +NAME READY STATUS RESTARTS AGE +pod/virt-launcher-vm-chsrh 1/1 Running 0 16m + +NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE +persistentvolumeclaim/vm-pvc Bound pvc-149bd414-c1b8-443c-90f7-7e30fc519eb4 128Mi RWX rook-ceph-block 16m + +NAME DESIREDSTATE CURRENTSTATE +volumereplicationgroup.ramendr.openshift.io/drpc primary Primary + +NAME AGE VOLUMEREPLICATIONCLASS PVCNAME DESIREDSTATE CURRENTSTATE +volumereplication.replication.storage.openshift.io/vm-pvc 2m46s vrc-sample vm-pvc primary Primary +``` + +The cirros VM used by this example includes a logger service logging a +message every 10 seconds: + +```sh +virtctl ssh cirros@vm -n vm-pvc --known-hosts= --context dr1 -c 'head /var/log/ramen.log' +``` + +Example output: + +```console +Sun Nov 19 20:13:20 UTC 2023 START uptime=3.33 +Sun Nov 19 20:13:30 UTC 2023 UPDATE +Sun Nov 19 20:13:40 UTC 2023 UPDATE +Sun Nov 19 20:13:50 UTC 2023 UPDATE +Sun Nov 19 20:14:00 UTC 2023 UPDATE +Sun Nov 19 20:14:10 UTC 2023 UPDATE +Sun Nov 19 20:14:20 UTC 2023 UPDATE +Sun Nov 19 20:14:30 UTC 2023 UPDATE +Sun Nov 19 20:14:40 UTC 2023 UPDATE +Sun Nov 19 20:14:50 UTC 2023 UPDATE +``` + +Ramen set up RBD mirroring for the underlying RBD image. The same image +is created on the secondary cluster ("dr2"): + +``` +kubectl rook-ceph --context dr2 rbd du -p replicapool +``` + +Example output: + +```console +NAME PROVISIONED USED +csi-vol-a3fdb384-2e31-49f1-bd48-a97b2d79f981 128 MiB 80 MiB +``` + +In case of a disaster in the primary cluster, we can start the VM using +the replica on the secondary cluster. + +## Failing over to another cluster + +In case of disaster you can force the VM to run on the other cluster. +The VM will start on the other cluster using the data from the last +replication. Data since the last replication is lost. + +To simulate a disaster we can pause the minkube VM running cluster +`dr1`: + +```sh +virsh -c qemu:///system suspend dr1 +``` + +To start `Failover` action, patch the VM `DRPlacementControl` resource +to set `action` and `failoverCluster`: + +```sh +kubectl patch drpc drpc \ + --patch '{"spec": {"action": "Failover", "failoverCluster": "dr2"}}' \ + --type merge \ + --namespace vm-pvc \ + --context hub +``` + +The VM will start on the failover cluster ("dr2"). Nothing will change +on the primary cluster ("dr1") since it is still paused. + +Inspecting the `/var/log/ramen.log` via SSH show how much data was lost +during the failover: + +```sh +virtctl ssh cirros@vm -n vm-pvc --known-hosts= --context dr2 -c 'tail -f /var/log/ramen.log' +``` + +Example output: + +```console +Sun Nov 19 20:33:42 UTC 2023 UPDATE +Sun Nov 19 20:33:53 UTC 2023 UPDATE +Sun Nov 19 20:34:03 UTC 2023 UPDATE +Sun Nov 19 20:34:13 UTC 2023 UPDATE +Sun Nov 19 20:34:23 UTC 2023 UPDATE +Sun Nov 19 20:34:33 UTC 2023 UPDATE +Sun Nov 19 20:34:43 UTC 2023 UPDATE +Sun Nov 19 20:34:53 UTC 2023 UPDATE +Sun Nov 19 20:37:43 UTC 2023 START uptime=3.18 +Sun Nov 19 20:37:53 UTC 2023 UPDATE +Sun Nov 19 20:38:03 UTC 2023 UPDATE +``` + +To enable replication from the secondary cluster to the primary cluster, +we need to recover the primary cluster. In this example we can resume +the minikube VM: + +```sh +virsh -c qemu:///system resume dr1 +``` + +*Ramen* will clean up the VM resources from the primary cluster and +enable RBD mirroring from the secondary cluster ("dr2") to the primary +cluster ("dr1"). + +To wait until the VM data is replicated again to the other cluster: + +```sh +kubectl wait drpc drpc \ + --for condition=PeerReady \ + --namespace vm-pvc \ + --timeout 5m \ + --context hub +``` + +Since the primary cluster is recovered, we can move the VM back to the +primary cluster. + +## Relocating to another cluster + +To move the VM back to the original cluster after a disaster +you can use the `Relocate` action. The VM will be terminated on +the current cluster, and started on the other cluster. No data is lost +during this operation. + +Patch the VM `DRPlacementControl` resource to set `action` and +if needed, `preferredCluster`. + +```sh +kubectl patch drpc drpc \ + --patch '{"spec": {"action": "Relocate", "preferredCluster": "dr1"}}' \ + --type merge \ + --namespace vm-pvc \ + --context hub +``` + +The VM will terminate on the current cluster and will start on the +preferred cluster. + +To wait until the VM is relocated to the primary cluster, wait until the +drpc phase is `Relocated`: + +```sh +kubectl wait drpc drpc \ + --for jsonpath='{.status.phase}=Relocated' \ + --namespace vm-pvc \ + --timeout 5m \ + --context hub +``` + +Inspecting `/var/log/ramen.log` shows that the VM was terminated cleanly +on the secondary cluster ("dr2") and started on the primary cluster +("dr1"). No data was lost! + +```sh +virtctl ssh cirros@vm -n vm-pvc --known-hosts= --context dr1 -c 'tail -f /var/log/ramen.log' +``` + +Example output: + +```console +Sun Nov 19 20:41:53 UTC 2023 UPDATE +Sun Nov 19 20:42:03 UTC 2023 UPDATE +Sun Nov 19 20:42:13 UTC 2023 UPDATE +Sun Nov 19 20:42:23 UTC 2023 UPDATE +Sun Nov 19 20:42:33 UTC 2023 UPDATE +Sun Nov 19 20:42:43 UTC 2023 UPDATE +Sun Nov 19 20:42:53 UTC 2023 UPDATE +Sun Nov 19 20:42:58 UTC 2023 STOP +Sun Nov 19 20:41:17 UTC 2023 START uptime=3.45 +Sun Nov 19 20:41:27 UTC 2023 UPDATE +Sun Nov 19 20:41:37 UTC 2023 UPDATE +``` + +To wait until the VM is replicating data again to the secondary cluster, +wait for the `PeerReady` condition: + +```sh +kubectl wait drpc drpc \ + --for condition=PeerReady \ + --namespace vm-pvc \ + --timeout 5m \ + --context hub +``` + +## Disable DR for the VM + +Delete the `dr` resources to disable DR: + +```sh +kubectl delete -k dr/kubevirt/vm-pvc-k8s-regional --context hub +``` + +At this point ramen is not protecting the virtual machine, and the +storage used for replicating the virtual machine data on the DR clusters +will be reclaimed. The virtual machine will continue to run on the +current cluster. + +> [!IMPORTANT] +> Do not delete the Placement annotation +> `cluster.open-cluster-management.io/experimental-scheduling-disable` +> to ensure that OCM will not change the placement of the virtual +> machine, which can result in data loss. + +## Undeploying the VM + +Delete the subscription to stop and delete the VM: + +```sh +kubectl delete -k subscription/kubevirt/vm-pvc-k8s-regional --context hub +``` diff --git a/docs/ocm-rbd.md b/docs/ocm-rbd.md new file mode 100644 index 00000000..35239466 --- /dev/null +++ b/docs/ocm-rbd.md @@ -0,0 +1,142 @@ +# Experiment with OCM managed application using RBD storage + +In the example we use the busybox deployment for Kubernetes regional DR +environment using RBD storage: + + subscription/deployment-k8s-regional-rbd + +This application is deployed in the `deployment-rbd` namespace on the +hub and managed clusters. + +You can use other overlays to deploy on other cluster types or use +different storage class. You can also create your own overlays based on +the examples. + +## Deploying a sample application + +1. Deploy an OCM application subscription on hub: + + ``` + kubectl apply -k subscription/deployment-k8s-regional-rbd + ``` + + This creates the required Subscription, Placement, and + ManagedClusterSetBinding resources for the deployment in the + `deployment-rbd` namespace and can be viewed using: + + ``` + kubectl get subscription,placement -n deployment-rbd + ``` + +1. Inspect subscribed resources from the channel created in the same + namespace on the ManagedCluster selected by the Placement. + + The busybox deployment Placement `status` can be viewed on the hub + using: + + ``` + kubectl get placement placement -n deployment-rbd + ``` + + The Busybox deployment subscribed resources, like the pod and the PVC + can be viewed on the ManagedCluster using (example ManagedCluster + `dr1`): + + ``` + kubectl get pod,pvc -n deployment-rbd --context dr1 + ``` + +## Undeploying a sample application + +To undeploy an application delete the subscription overlay used to +deploy the application: + +``` +kubectl delete -k subscription/deployment-k8s-regional-rbd +``` + +## Enable DR for a deployed application + +1. Change the Placement to be reconciled by Ramen + + ``` + kubectl annotate placement placement -n deployment-rbd \ + cluster.open-cluster-management.io/experimental-scheduling-disable=true + ``` + +1. Deploy a DRPlacementControl resource for the OCM application on the + hub, for example: + + ``` + kubectl apply -k dr/deployment-k8s-regional-rbd + ``` + + This creates a DRPlacementControl resource for the busybox deployment + in the `deployment-rbd` namespace and can be viewed using: + + ``` + kubectl get drpc -n deployment-rbd + ``` + + At this point the placement of the application is managed by *Ramen*. + +## Disable DR for a DR enabled application + +1. Delete the drpc resource for the OCM application on the hub: + + ``` + kubectl delete -k dr/deployment-k8s-regional-rbd + ``` + + This deletes the DRPlacementControl resource for the busybox + deployment, disabling replication and removing replicated data. + +> [!IMPORTANT] +> Do not delete the Placement annotation +> `cluster.open-cluster-management.io/experimental-scheduling-disable` +> to ensure that OCM will not change the placement of the application, +> which can result in data loss. + +### Optional: enabling OCM scheduling for the application + +It is not recommended to enable OCM scheduling on after disabling DR, +since OCM does not support moving workload storage between clusters. If +the placement point to wrong cluster, OCM will delete the application +and its storage from the current cluster, and deploy the application +with new storage on the cluster selected by the placement. + +Find the current placement of the application: + +``` +kubectl get placementdecisions -n deployment-rbd --context hub \ + -o jsonpath='{.items[0].status.decisions[0].clusterName}{"\n"}' +``` + +Ensure that the `Placement` predicates is pointing to the cluster where +the workload is currently placed. Here is example predicates selecting +the cluster `dr1`: + +``` +spec: + clusterSets: + - default + numberOfClusters: 1 + predicates: + - requiredClusterSelector: + claimSelector: {} + labelSelector: + matchExpressions: + - key: name + operator: In + values: + - dr1 +``` + +Change the Placement to be reconciled by OCM: + +``` +kubectl annotate placement placement -n deployment-rbd \ + cluster.open-cluster-management.io/experimental-scheduling-disable- +``` + +At this point the application is managed again by *OCM*. diff --git a/workloads/kubevirt/README.md b/workloads/kubevirt/README.md index eacadb82..070e3e44 100644 --- a/workloads/kubevirt/README.md +++ b/workloads/kubevirt/README.md @@ -3,377 +3,6 @@ This directory contains sample kubevirt resources for deployment in Kubernetes or OpenShift clusters. -## Setting up a testing environment - -When starting the testing environment, use the -`regional-dr-kubvirt.yaml` configuration. - -## Adding your SSH key - -If you want to inject your SSH key into the VM, you need to replace -the included SSH public key with your own public key. - -1. Fork this repo in github - -1. Replace the content of `workloads/kubevirt/vm-*/base/test_rsa.pub` - with the content of *your* public key (e.g. `~/.ssh/id_rsa.pub`) - -1. Update `channel/channel.yaml` to point to *your* repository - (e.g. https://github.com/MY-GITHUB-USER/ocm-ramen-samples.git) - -1. If you are not using the `main` branch update - `subscription/base/subscription.yaml` to point to the right branch. - -## Deploying the VM subscription - -To start a PVC based VM `vm-pvc` use the -`subscription/kubevirt/vm-pvc-k8s-regional` overlay: - -```sh -kubectl apply -k subscription/kubevirt/vm-pvc-k8s-regional --context hub -``` - -This subscription starts the VM `vm-pvc` in the namespace `vm-pvc` on -one of the clusters in the default clusterset. - -This overlay is customized for *Ramen* minikube based test environment. -To use in another setup you may want to create a new overlay. - -## Inspecting the VM status - -To find where the VM was placed look at the PlacementDecisions status: - -```sh -kubectl get placementdecisions -n vm-pvc --context hub \ - -o jsonpath='{.items[0].status.decisions[0].clusterName}{"\n"}' -``` - -To inspect the VM and DR resources use: - -```sh -watch -n 5 kubectl get vm,vmi,pod,pvc,vrg,vr -n vm-pvc --context dr1 -``` - -Example output: - -```console -Every 5.0s: kubectl get vm,vmi,pod,pvc,vrg,vr -n vm-pvc --context dr1 - -NAME AGE STATUS READY -virtualmachine.kubevirt.io/vm 5m51s Running True - -NAME AGE PHASE IP NODENAME READY -virtualmachineinstance.kubevirt.io/vm 5m51s Running 10.244.0.61 dr1 True - -NAME READY STATUS RESTARTS AGE -pod/virt-launcher-vm-chsrh 1/1 Running 0 5m51s - -NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE -persistentvolumeclaim/vm-pvc Bound pvc-149bd414-c1b8-443c-90f7-7e30fc519eb4 128Mi RWX rook-ceph-block 5m51s -``` - -At this point there there are no `vrg` and `vr` resources, since we did -not enable DR for the VM yet. - -We can inspect the underlying RBD image backing the VM PVC using the -`rook-ceph` kubectl krew plugin: - -``` -kubectl rook-ceph --context dr1 rbd du -p replicapool -``` - -Example output: - -```console -NAME PROVISIONED USED -csi-vol-a3fdb384-2e31-49f1-bd48-a97b2d79f981 128 MiB 80 MiB -``` - -## Enabling DR for the VM - -To allow *Ramen* to protect the VM, you need to disable *OCM* -scheduling by adding an annotation to the VM placement: - -```sh -kubectl annotate placement placement \ - cluster.open-cluster-management.io/experimental-scheduling-disable=true \ - --namespace vm-pvc \ - --context hub -``` - -Deploy the DR resources to enable DR: - -```sh -kubectl apply -k dr/kubevirt/vm-pvc-k8s-regional --context hub -``` - -At this point *Ramen* controls the VM placement and protects the VM data -by replicating it to the secondary cluster ("dr2"). - -To wait until the VM data is replicating to the secondary cluster, wait -for the `PeerReady` condition: - -```sh -kubectl wait drpc drpc \ - --for condition=PeerReady \ - --namespace vm-pvc \ - --timeout 5m \ - --context hub -``` - -## Inspecting the VM DR status - -We can inspect the VM DR status using the `DRPlacementControl` resource -on the hub cluster: - -```sh -watch -n 5 kubectl get drpc -n vm-pvc --context hub -o wide -``` - -Example output: - -```console -Every 5.0s: kubectl get drpc -n vm-pvc --context hub -o wide - -NAME AGE PREFERREDCLUSTER FAILOVERCLUSTER DESIREDSTATE CURRENTSTATE PROGRESSION START TIME DURATION PEER READY -drpc 51s dr1 Deployed Completed 2023-11-19T20:26:53Z 5.035609263s True -``` - -To get more details we can watch the VM and DR resources on the managed -cluster: - -```sh -watch -n 5 kubectl get vm,vmi,pod,pvc,vrg,vr -n vm-pvc --context dr1 -``` - -Example output: - -```console -Every 5.0s: kubectl get vm,vmi,pod,pvc,vrg,vr -n vm-pvc --context dr1 - -NAME AGE STATUS READY -virtualmachine.kubevirt.io/vm 16m Running True - -NAME AGE PHASE IP NODENAME READY -virtualmachineinstance.kubevirt.io/vm 16m Running 10.244.0.61 dr1 True - -NAME READY STATUS RESTARTS AGE -pod/virt-launcher-vm-chsrh 1/1 Running 0 16m - -NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE -persistentvolumeclaim/vm-pvc Bound pvc-149bd414-c1b8-443c-90f7-7e30fc519eb4 128Mi RWX rook-ceph-block 16m - -NAME DESIREDSTATE CURRENTSTATE -volumereplicationgroup.ramendr.openshift.io/drpc primary Primary - -NAME AGE VOLUMEREPLICATIONCLASS PVCNAME DESIREDSTATE CURRENTSTATE -volumereplication.replication.storage.openshift.io/vm-pvc 2m46s vrc-sample vm-pvc primary Primary -``` - -The cirros VM used by this example includes a logger service logging a -message every 10 seconds: - -```sh -virtctl ssh cirros@vm -n vm-pvc --known-hosts= --context dr1 -c 'head /var/log/ramen.log' -``` - -Example output: - -```console -Sun Nov 19 20:13:20 UTC 2023 START uptime=3.33 -Sun Nov 19 20:13:30 UTC 2023 UPDATE -Sun Nov 19 20:13:40 UTC 2023 UPDATE -Sun Nov 19 20:13:50 UTC 2023 UPDATE -Sun Nov 19 20:14:00 UTC 2023 UPDATE -Sun Nov 19 20:14:10 UTC 2023 UPDATE -Sun Nov 19 20:14:20 UTC 2023 UPDATE -Sun Nov 19 20:14:30 UTC 2023 UPDATE -Sun Nov 19 20:14:40 UTC 2023 UPDATE -Sun Nov 19 20:14:50 UTC 2023 UPDATE -``` - -Ramen set up RBD mirroring for the underlying RBD image. The same image -is created on the secondary cluster ("dr2"): - -``` -kubectl rook-ceph --context dr2 rbd du -p replicapool -``` - -Example output: - -```console -NAME PROVISIONED USED -csi-vol-a3fdb384-2e31-49f1-bd48-a97b2d79f981 128 MiB 80 MiB -``` - -In case of a disaster in the primary cluster, we can start the VM using -the replica on the secondary cluster. - -## Failing over to another cluster - -In case of disaster you can force the VM to run on the other cluster. -The VM will start on the other cluster using the data from the last -replication. Data since the last replication is lost. - -To simulate a disaster we can pause the minkube VM running cluster -`dr1`: - -```sh -virsh -c qemu:///system suspend dr1 -``` - -To start `Failover` action, patch the VM `DRPlacementControl` resource -to set `action` and `failoverCluster`: - -```sh -kubectl patch drpc drpc \ - --patch '{"spec": {"action": "Failover", "failoverCluster": "dr2"}}' \ - --type merge \ - --namespace vm-pvc \ - --context hub -``` - -The VM will start on the failover cluster ("dr2"). Nothing will change -on the primary cluster ("dr1") since it is still paused. - -Inspecting the `/var/log/ramen.log` via SSH show how much data was lost -during the failover: - -```sh -virtctl ssh cirros@vm -n vm-pvc --known-hosts= --context dr2 -c 'tail -f /var/log/ramen.log' -``` - -Example output: - -```console -Sun Nov 19 20:33:42 UTC 2023 UPDATE -Sun Nov 19 20:33:53 UTC 2023 UPDATE -Sun Nov 19 20:34:03 UTC 2023 UPDATE -Sun Nov 19 20:34:13 UTC 2023 UPDATE -Sun Nov 19 20:34:23 UTC 2023 UPDATE -Sun Nov 19 20:34:33 UTC 2023 UPDATE -Sun Nov 19 20:34:43 UTC 2023 UPDATE -Sun Nov 19 20:34:53 UTC 2023 UPDATE -Sun Nov 19 20:37:43 UTC 2023 START uptime=3.18 -Sun Nov 19 20:37:53 UTC 2023 UPDATE -Sun Nov 19 20:38:03 UTC 2023 UPDATE -``` - -To enable replication from the secondary cluster to the primary cluster, -we need to recover the primary cluster. In this example we can resume -the minikube VM: - -```sh -virsh -c qemu:///system resume dr1 -``` - -*Ramen* will clean up the VM resources from the primary cluster and -enable RBD mirroring from the secondary cluster ("dr2") to the primary -cluster ("dr1"). - -To wait until the VM data is replicated again to the other cluster: - -```sh -kubectl wait drpc drpc \ - --for condition=PeerReady \ - --namespace vm-pvc \ - --timeout 5m \ - --context hub -``` - -Since the primary cluster is recovered, we can move the VM back to the -primary cluster. - -## Relocating to another cluster - -To move the VM back to the original cluster after a disaster -you can use the `Relocate` action. The VM will be terminated on -the current cluster, and started on the other cluster. No data is lost -during this operation. - -Patch the VM `DRPlacementControl` resource to set `action` and -if needed, `preferredCluster`. - -```sh -kubectl patch drpc drpc \ - --patch '{"spec": {"action": "Relocate", "preferredCluster": "dr1"}}' \ - --type merge \ - --namespace vm-pvc \ - --context hub -``` - -The VM will terminate on the current cluster and will start on the -preferred cluster. - -To wait until the VM is relocated to the primary cluster, wait until the -drpc phase is `Relocated`: - -```sh -kubectl wait drpc drpc \ - --for jsonpath='{.status.phase}=Relocated' \ - --namespace vm-pvc \ - --timeout 5m \ - --context hub -``` - -Inspecting `/var/log/ramen.log` shows that the VM was terminated cleanly -on the secondary cluster ("dr2") and started on the primary cluster -("dr1"). No data was lost! - -```sh -virtctl ssh cirros@vm -n vm-pvc --known-hosts= --context dr1 -c 'tail -f /var/log/ramen.log' -``` - -Example output: - -```console -Sun Nov 19 20:41:53 UTC 2023 UPDATE -Sun Nov 19 20:42:03 UTC 2023 UPDATE -Sun Nov 19 20:42:13 UTC 2023 UPDATE -Sun Nov 19 20:42:23 UTC 2023 UPDATE -Sun Nov 19 20:42:33 UTC 2023 UPDATE -Sun Nov 19 20:42:43 UTC 2023 UPDATE -Sun Nov 19 20:42:53 UTC 2023 UPDATE -Sun Nov 19 20:42:58 UTC 2023 STOP -Sun Nov 19 20:41:17 UTC 2023 START uptime=3.45 -Sun Nov 19 20:41:27 UTC 2023 UPDATE -Sun Nov 19 20:41:37 UTC 2023 UPDATE -``` - -To wait until the VM is replicating data again to the secondary cluster, -wait for the `PeerReady` condition: - -```sh -kubectl wait drpc drpc \ - --for condition=PeerReady \ - --namespace vm-pvc \ - --timeout 5m \ - --context hub -``` - -## Disable DR for the VM - -Delete the `dr` resources to disable DR: - -```sh -kubectl delete -k dr/kubevirt/vm-pvc-k8s-regional --context hub -``` - -At this point ramen is not protecting the virtual machine, and the -storage used for replicating the virtual machine data on the DR clusters -will be reclaimed. The virtual machine will continue to run on the -current cluster. - -> [!IMPORTANT] -> Do not delete the Placement annotation -> `cluster.open-cluster-management.io/experimental-scheduling-disable` -> to ensure that OCM will not change the placement of the virtual -> machine, which can result in data loss. - -## Undeploying the VM - -Delete the subscription to stop and delete the VM: - -```sh -kubectl delete -k subscription/kubevirt/vm-pvc-k8s-regional --context hub -``` +- vm-pvc - PVC based VM +- vm-dv - DataVolume based VM +- vm-dvt - DataVolumeTemplate based VM