diff --git a/CHANGELOG.md b/CHANGELOG.md index 776dafa..01a2ed6 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -5,6 +5,34 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [1.3.0] +### Added +- `objectiveTarget` StorageClass parameter (`share` (default) | `file` | `both`) for file-backed volumes. With the default `share`, CreateVolume skips the per-file objective-set and the Anvil file-visibility poll that only exists to gate it — the backing share already carries the objectives — so provisioning returns as soon as the local `mkfs` completes and the `GET /files` poll storm under concurrency is eliminated. Use `file`/`both` to also apply per-file objectives. +- OpenTelemetry tracing and Prometheus metrics for the driver (`OTEL_TRACES_EXPORTER`, `OTEL_METRICS_EXPORTER`, etc.); `hs_csi_operation_*` and `hs_csi_anvil_requests_total` instruments across the controller/node RPCs, the file- and share-backed provisioning steps, and every Anvil REST call. See `docs/observability.md`. +- Minimum size gates for file-backed volumes (xfs < 300 MiB, ext4 < 20 MiB rejected) and an `fsfreeze` of the source volume before snapshot for crash-consistent file-backed snapshots. +- `deploy/kubernetes/kubernetes-1.3{4,5,6}/plugin.yaml` — manifests for the currently supported k8s minors, validated end-to-end on live 1.34 and 1.35 clusters (1.29 base + host-networked metrics port + OTel env vars). +- `deploy/monitoring/` — importable Grafana dashboard (`hs-csi-driver`), an example VictoriaMetrics/Prometheus scrape config, and a wiring README. +- Unit tests for `objectiveTarget` parsing, the file-backed size gates, the file/share volume-ID discriminator, the Anvil route-template normalization, `MeasureOp`, and the lock-timeout→`codes.Aborted` behavior. + +### Changed +- Parallelized file-backed CreateVolume by narrowing the per-backing-share lock, plus `mkfs` tuning (ext4 lazy-init, `mkfs.xfs -K` over NFS). See `docs/file-backed-performance.md`. +- Decide file- vs share-backed structurally from the volume ID instead of a `GetShare` probe that 404s for file-backed sources. +- Task-completion polling uses a fixed 2s/30s-then-4s cadence instead of exponential backoff. See `docs/tunable-retry-parameters.md`. + +### Removed +- Dropped `ext3` as a supported file-backed filesystem; `CreateVolume` now rejects `fsType=ext3` with `InvalidArgument` (use `ext4` or `xfs`). + +### Fixed +- The backing-share NFS mount/unmount no longer runs under the global `mountRefsMu`. `acquireBackingMount`/`releaseBackingMount` now serialize the mount/unmount with a per-backing-directory lock and reserve the refcount before mounting, so `mountRefsMu` is held only for microsecond map updates. Previously a single slow or hung mount (up to the ~5 min command timeout on a dead portal) held `mountRefsMu` for its whole duration, freezing every concurrent file-backed create/delete — including refcount checks on unrelated shares. +- `releaseBackingMount` no longer holds `mountRefsMu` while calling `UnmountBackingShareIfUnused` (which re-acquires the same non-reentrant mutex via the `mountRefs` refcount check): the volume that dropped the last reference self-deadlocked, wedging all subsequent file-backed provisioning. The decrement now happens under the lock and the unmount runs after releasing it. Caught by live xfs validation (a single file-backed PVC is the exact refcount-1 trigger). +- `acquireVolumeLock`/`acquireSnapshotLock` return `codes.Aborted` on a lock-acquire timeout instead of calling `os.Exit(1)`, which under concurrent load crashed the whole controller. +- Only force-unmount a stale backing-share mount after repeated (not a single) mount-check timeouts, so a slow-but-healthy NFS stat under concurrency can't force-unmount a live shared mount out from under in-flight pods. +- `UnmountBackingShareIfUnused` now honors the `mountRefs` refcount, so a concurrent delete can't unmount a backing share out from under an in-flight `mkfs` (which has no loop device yet). +- Run snapshot `Unfreeze` on a context detached from the gRPC request cancellation, so a cancelled/expired `CreateSnapshot` can't leave the source pod's filesystem frozen. +- `AnvilRoute` collapses `share-snapshots` share/snapshot identifiers to `{id}`, preventing unbounded `hs_csi_anvil_requests_total` metric cardinality. +- Survive a stale/dead backing-share NFS mount (timeout-bounded mount + force-unmount before remount) instead of leaking the lock and wedging serialized provisioning. See `docs/node-unmount-recovery.md`. +- Route file-backed snapshot deletes to the file-snapshot API instead of always calling the share-snapshot delete. + ## [1.2.9] ### Fixed - Included share objectives in share create requests instead of applying them with follow-up objective-set calls after provisioning. diff --git a/VERSION b/VERSION index 1db994b..18fa8e7 100755 --- a/VERSION +++ b/VERSION @@ -1 +1 @@ -v1.2.9 +v1.3.0 diff --git a/deploy/kubernetes/README.md b/deploy/kubernetes/README.md index bf9e564..c28daa7 100644 --- a/deploy/kubernetes/README.md +++ b/deploy/kubernetes/README.md @@ -20,6 +20,16 @@ If you are using ```hammerspaceinc/csi-plugin:latest``` you must delete all the Kubernetes documentation for CSI support can be found [here](https://kubernetes-csi.github.io/) * Kubernetes version 1.13 or higher +* Per-minor manifests live under `deploy/kubernetes/kubernetes-./plugin.yaml`. + Pick the one matching your `kubectl`/cluster minor version. Bundled: **1.25–1.29** + (historical) and **1.34 / 1.35 / 1.36**. **1.34–1.36 are the currently supported + + validated set** — the driver in this release was tested end-to-end on live k8s + **1.34** and **1.35** clusters (and 1.36). Those three manifests are the 1.29 + manifest plus the observability wiring (a host-networked metrics port and OTel + env vars); see [`docs/observability.md`](../../docs/observability.md) and + [`deploy/monitoring/README.md`](../monitoring/README.md). The 1.25–1.29 manifests + are kept for older clusters and pin their contemporary driver image. For a minor + with no bundled manifest, copy the nearest lower version and bump sidecar tags. * BlockVolume support requires kubelet has the [feature gates](https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/) BlockVolume and CSIBlockVolume set to true. Example in /var/lib/kubelet/config.yaml ```yaml diff --git a/deploy/kubernetes/example_storage_class_file_backed.yaml b/deploy/kubernetes/example_storage_class_file_backed.yaml index 73154c9..eaacb46 100644 --- a/deploy/kubernetes/example_storage_class_file_backed.yaml +++ b/deploy/kubernetes/example_storage_class_file_backed.yaml @@ -14,6 +14,15 @@ parameters: mountBackingShareName: k8s-file-storage # Objectives to set on shares in addition to HS cluster defaults objectives: "keep-online" + # Where objectives are applied for file-backed volumes, and therefore whether + # CreateVolume pays for the per-file Anvil visibility poll that gates them: + # share (default) - objectives live on the backing share only; the per-file + # objective-set and its visibility poll are skipped, so + # CreateVolume returns as soon as the local mkfs completes + # (much faster, and no GET /files poll storm under load). + # file | both - also apply objectives per file (pays the visibility + # poll). Use for per-volume / multi-site placement policy. + objectiveTarget: "share" # The name format of provisioned volumes, %s is replaced with pvc- volumeNameFormat: "csi-%s" # Metadata to set on files and shares created by the plugin. diff --git a/deploy/kubernetes/kubernetes-1.29/plugin.yaml b/deploy/kubernetes/kubernetes-1.29/plugin.yaml index 2bb9a2f..cc53927 100644 --- a/deploy/kubernetes/kubernetes-1.29/plugin.yaml +++ b/deploy/kubernetes/kubernetes-1.29/plugin.yaml @@ -163,7 +163,16 @@ metadata: rules: - apiGroups: [""] resources: ["pods"] - verbs: ["list", "watch"] + verbs: ["get", "list", "watch"] + # pods/exec: needed by the driver's Freezer to run `fsfreeze --freeze` + # inside the csi-node DaemonSet pod holding a source volume's mount + # during CreateSnapshot. Without this the driver falls back to + # snapshotting without quiescing, which for XFS can produce an + # inconsistent-log snapshot that log-recovery empties on restore. + # See pkg/driver/freezer.go. + - apiGroups: [""] + resources: ["pods/exec"] + verbs: ["create"] - apiGroups: [""] resources: ["persistentvolumes"] verbs: ["get", "patch", "list", "watch", "create", "delete", "update"] diff --git a/deploy/kubernetes/kubernetes-1.34/plugin.yaml b/deploy/kubernetes/kubernetes-1.34/plugin.yaml new file mode 100644 index 0000000..c2a83e8 --- /dev/null +++ b/deploy/kubernetes/kubernetes-1.34/plugin.yaml @@ -0,0 +1,474 @@ +# Hammerspace CSI plugin manifest, validated on Kubernetes 1.34 (parity with 1.35/1.36) (and works on +# 1.30+ in practice). It is the 1.29 manifest plus the observability wiring: +# - a host-networked `metrics` port (9090 controller / 9091 node), and +# - OTel env vars on both driver containers. +# +# Observability env (see deploy/monitoring/README.md and docs/observability.md): +# OTEL_METRICS_EXPORTER=prometheus metrics on; scrape :9090 / :9091 +# OTEL_METRICS_PROMETHEUS_LISTEN=:9090|:9091 per-container listen addr +# OTEL_TRACES_EXPORTER=console spans to pod stdout — VERBOSE; +# set to "none" to disable in prod, +# or "otlp" (+ OTEL_EXPORTER_OTLP_ENDPOINT) +# to ship to a collector. +# Sidecar image tags (csi-provisioner / csi-attacher / csi-snapshotter / +# csi-resizer / node-driver-registrar) match the 1.29 manifest. +#### CSI Object +apiVersion: storage.k8s.io/v1 +kind: CSIDriver +metadata: + name: com.hammerspace.csi +spec: + podInfoOnMount: true + requiresRepublish: true + volumeLifecycleModes: + - Persistent + storageCapacity: true + +#### Controller Service +--- +# needed for StatefulSet +kind: Service +apiVersion: v1 +metadata: + name: csi-provisioner + namespace: kube-system + labels: + app: csi-provisioner +spec: + type: ClusterIP + clusterIP: None + # This is needed for the StatefulSet to work properly + # as it uses a headless service to manage the pods. + # The StatefulSet will create a DNS entry for the pods + # in the format ...svc.cluster.local + # where is the name of the pod, is the name of the service, + # and is the namespace of the service. + # This allows the pods to communicate with each other using DNS. + # The StatefulSet will also create a DNS entry for the service in the format ..svc.cluster.local + # which allows the pods to communicate with the service + selector: + app: csi-provisioner +--- +kind: StatefulSet +apiVersion: apps/v1 +metadata: + name: csi-provisioner + namespace: kube-system +spec: + selector: + matchLabels: + app: csi-provisioner + serviceName: "csi-provisioner" + replicas: 1 + template: + metadata: + labels: + app: csi-provisioner + spec: + serviceAccountName: csi-provisioner + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + containers: + - name: csi-provisioner + imagePullPolicy: Always + image: registry.k8s.io/sig-storage/csi-provisioner:v3.6.0 + args: + - "--csi-address=$(CSI_ENDPOINT)" + - "--timeout=60s" # Recommended as shares may take some time to create + - "--v=5" + env: + - name: CSI_ENDPOINT + value: /var/lib/csi/hs-csi.sock + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/ + - name: csi-attacher + imagePullPolicy: Always + image: registry.k8s.io/sig-storage/csi-attacher:v4.4.0 + args: + - "--csi-address=$(CSI_ENDPOINT)" + - "--v=5" + env: + - name: CSI_ENDPOINT + value: /var/lib/csi/hs-csi.sock + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/ + - name: csi-snapshotter + imagePullPolicy: Always + image: registry.k8s.io/sig-storage/csi-snapshotter:v6.2.1 + args: + - "--csi-address=$(CSI_ENDPOINT)" + - "--v=5" + env: + - name: CSI_ENDPOINT + value: /var/lib/csi/hs-csi.sock + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/ + - name: csi-resizer + imagePullPolicy: Always + image: registry.k8s.io/sig-storage/csi-resizer:v1.10.1 + args: + - "--csi-address=$(CSI_ENDPOINT)" + - "--v=5" + env: + - name: CSI_ENDPOINT + value: /var/lib/csi/hs-csi.sock + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/ + - name: hs-csi-plugin-controller + securityContext: + privileged: true + capabilities: + add: ["SYS_ADMIN"] + allowPrivilegeEscalation: true + imagePullPolicy: Always + image: hammerspaceinc/csi-plugin:v1.3.0 + ports: + - containerPort: 9090 + name: metrics + hostPort: 9090 + protocol: TCP + envFrom: + - configMapRef: + name: csi-env-config + env: + - name: CSI_ENDPOINT + value: /var/lib/csi/hs-csi.sock + - name: HS_USERNAME + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: username + - name: HS_PASSWORD + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: password + - name: HS_ENDPOINT + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: endpoint + - name: HS_TLS_VERIFY + value: "false" + - name: CSI_MAJOR_VERSION + value: "1" + # OTel toggles — Prometheus /metrics on :9090 (on by default). Set + # OTEL_TRACES_EXPORTER=console/otlp to also emit spans (off by default). + - name: OTEL_TRACES_EXPORTER + value: "none" + - name: OTEL_METRICS_EXPORTER + value: "prometheus" + - name: OTEL_METRICS_PROMETHEUS_LISTEN + value: ":9090" + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/ + - name: staging-dir + mountPath: /tmp + mountPropagation: Bidirectional + volumes: + - name: socket-dir + emptyDir: {} + - name: staging-dir + hostPath: + path: /tmp +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: csi-provisioner + namespace: kube-system +--- +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: csi-provisioner +rules: + - apiGroups: [""] + resources: ["pods"] + verbs: ["get", "list", "watch"] + # pods/exec: needed by the driver's Freezer to run `fsfreeze --freeze` + # inside the app pod holding a source volume during CreateSnapshot. Without + # this the driver falls back to snapshotting without quiescing, which for + # XFS can produce an inconsistent-log snapshot that log-recovery empties on + # restore. See pkg/driver/freezer.go. + - apiGroups: [""] + resources: ["pods/exec"] + verbs: ["create"] + - apiGroups: [""] + resources: ["persistentvolumes"] + verbs: ["get", "patch", "list", "watch", "create", "delete", "update"] + - apiGroups: [""] + resources: ["persistentvolumeclaims", "persistentvolumeclaims/status"] + verbs: ["get", "patch", "list", "watch", "update"] + - apiGroups: ["storage.k8s.io"] + resources: ["storageclasses"] + verbs: ["get", "list", "watch"] + - apiGroups: [""] + resources: ["nodes"] + verbs: ["get", "list", "watch"] + - apiGroups: [""] + resources: ["events"] + verbs: ["list", "watch", "create", "update", "patch"] + - apiGroups: [""] + resources: ["endpoints"] + verbs: ["list", "watch", "create", "update", "delete", "get"] + - apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "list"] + - apiGroups: ["snapshot.storage.k8s.io"] + resources: ["volumesnapshotclasses"] + verbs: ["get", "list", "watch"] + - apiGroups: ["snapshot.storage.k8s.io"] + resources: ["volumesnapshotcontents"] + verbs: ["create", "get", "list", "watch", "update", "delete"] + - apiGroups: ["snapshot.storage.k8s.io"] + resources: ["volumesnapshots"] + verbs: ["get", "list", "watch", "update"] + - apiGroups: ["apiextensions.k8s.io"] + resources: ["customresourcedefinitions"] + verbs: ["create", "list", "watch", "delete"] + - apiGroups: ["storage.k8s.io"] + resources: ["volumeattachments", "volumeattachments/status"] + verbs: ["get", "list", "watch", "update", "patch"] + - apiGroups: ["storage.k8s.io"] + resources: ["storageclasses"] + verbs: ["get", "list", "watch", "update"] +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: csi-provisioner + namespace: kube-system +subjects: + - kind: ServiceAccount + name: csi-provisioner + namespace: kube-system +roleRef: + kind: ClusterRole + name: csi-provisioner + apiGroup: rbac.authorization.k8s.io +#### Node service +--- +kind: DaemonSet +apiVersion: apps/v1 +metadata: + name: csi-node + namespace: kube-system +spec: + selector: + matchLabels: + app: csi-node + template: + metadata: + labels: + app: csi-node + spec: + serviceAccount: csi-node + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + containers: + - name: csi-resizer + imagePullPolicy: Always + image: registry.k8s.io/sig-storage/csi-resizer:v1.10.1 + args: + - "--csi-address=$(CSI_ENDPOINT)" + - "--v=5" + env: + - name: CSI_ENDPOINT + value: /csi/csi.sock + volumeMounts: + - name: socket-dir + mountPath: /csi + - name: driver-registrar + imagePullPolicy: Always + image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.0 + lifecycle: + preStop: + exec: + command: ["/bin/sh", "-c", '[ -n "$REG_SOCKET" ] && rm -rf "$REG_SOCKET" || echo "REG_SOCKET not set, skipping delete"'] + args: + - "--v=5" + - "--csi-address=$(CSI_ENDPOINT)" + - "--kubelet-registration-path=$(REG_SOCKET)" + securityContext: + privileged: true + env: + - name: CSI_ENDPOINT + value: /csi/csi.sock + - name: REG_SOCKET + value: /var/lib/kubelet/plugins_registry/com.hammerspace.csi/csi.sock + - name: KUBE_NODE_NAME + valueFrom: + fieldRef: + apiVersion: v1 + fieldPath: spec.nodeName + volumeMounts: + - name: socket-dir + mountPath: /csi + - name: registration-dir + mountPath: /registration + - name: hs-csi-plugin-node + securityContext: + privileged: true + capabilities: + add: ["SYS_ADMIN"] + allowPrivilegeEscalation: true + imagePullPolicy: Always + image: hammerspaceinc/csi-plugin:v1.3.0 + ports: + - containerPort: 9091 + name: metrics + hostPort: 9091 + protocol: TCP + envFrom: + - configMapRef: + name: csi-env-config + env: + - name: CSI_ENDPOINT + value: /csi/csi.sock + - name: HS_USERNAME + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: username + - name: HS_PASSWORD + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: password + - name: HS_ENDPOINT + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: endpoint + - name: CSI_NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + - name: HS_TLS_VERIFY + value: "false" + - name: CSI_MAJOR_VERSION + value: "1" + # OTel toggles. hostNetwork=true means we bind directly to the + # node's :9091 (must differ from controller's :9090 in case the + # controller lands on the same node). + - name: OTEL_TRACES_EXPORTER + value: "none" + - name: OTEL_METRICS_EXPORTER + value: "prometheus" + - name: OTEL_METRICS_PROMETHEUS_LISTEN + value: ":9091" + volumeMounts: + - name: socket-dir + mountPath: /csi + - name: registration-dir + mountPath: /registration + mountPropagation: Bidirectional + - name: mountpoint-dir + mountPath: /var/lib/kubelet/ + mountPropagation: Bidirectional + - name: rootshare-dir + mountPath: /var/lib/hammerspace/ + mountPropagation: Bidirectional + - name: dev-dir + mountPath: /dev + - name: staging-dir + mountPath: /tmp + mountPropagation: Bidirectional + volumes: + - name: socket-dir + hostPath: + path: /var/lib/kubelet/plugins_registry/com.hammerspace.csi + type: DirectoryOrCreate + - name: mountpoint-dir + hostPath: + path: /var/lib/kubelet/ + - name: rootshare-dir + hostPath: + path: /var/lib/hammerspace/ + - name: registration-dir + hostPath: + path: /var/lib/kubelet/plugins_registry/ + - name: dev-dir + hostPath: + path: /dev + - name: staging-dir + hostPath: + path: /tmp +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: csi-node + namespace: kube-system +--- +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: csi-node + namespace: kube-system +rules: + - apiGroups: [""] + resources: ["pods"] + verbs: ["list", "watch"] + - apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "list"] + - apiGroups: [""] + resources: ["nodes"] + verbs: ["get", "list", "update"] + - apiGroups: [""] + resources: ["namespaces", "events"] + verbs: ["get", "list", "create"] + - apiGroups: [""] + resources: ["persistentvolumes", "persistentvolumeclaims", "persistentvolumeclaims/status","events"] + verbs: ["get", "list", "watch", "update", "patch"] + - apiGroups: ["storage.k8s.io"] + resources: ["volumeattachments", "volumeattachments/status"] + verbs: ["get", "list", "watch", "update", "patch"] + +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: csi-node + namespace: kube-system +subjects: + - kind: ServiceAccount + name: csi-node + namespace: kube-system +roleRef: + kind: ClusterRole + name: csi-node + apiGroup: rbac.authorization.k8s.io + +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: csi-provisioner-volumeattachment-status-binding + namespace: kube-system +subjects: + - kind: ServiceAccount + name: csi-provisioner + namespace: kube-system +roleRef: + kind: ClusterRole + name: csi-provisioner-volumeattachment-status + apiGroup: rbac.authorization.k8s.io +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: csi-env-config + namespace: kube-system +# Example: "300ms". Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". +data: + MOUNT_CHECK_TIMEOUT: "30s" + UNMOUNT_RETRY_COUNT: "5" + UNMOUNT_RETRY_INTERVAL: "1s" diff --git a/deploy/kubernetes/kubernetes-1.35/plugin.yaml b/deploy/kubernetes/kubernetes-1.35/plugin.yaml new file mode 100644 index 0000000..f1a8f49 --- /dev/null +++ b/deploy/kubernetes/kubernetes-1.35/plugin.yaml @@ -0,0 +1,474 @@ +# Hammerspace CSI plugin manifest, validated on Kubernetes 1.35 (parity with 1.35/1.36) (and works on +# 1.30+ in practice). It is the 1.29 manifest plus the observability wiring: +# - a host-networked `metrics` port (9090 controller / 9091 node), and +# - OTel env vars on both driver containers. +# +# Observability env (see deploy/monitoring/README.md and docs/observability.md): +# OTEL_METRICS_EXPORTER=prometheus metrics on; scrape :9090 / :9091 +# OTEL_METRICS_PROMETHEUS_LISTEN=:9090|:9091 per-container listen addr +# OTEL_TRACES_EXPORTER=console spans to pod stdout — VERBOSE; +# set to "none" to disable in prod, +# or "otlp" (+ OTEL_EXPORTER_OTLP_ENDPOINT) +# to ship to a collector. +# Sidecar image tags (csi-provisioner / csi-attacher / csi-snapshotter / +# csi-resizer / node-driver-registrar) match the 1.29 manifest. +#### CSI Object +apiVersion: storage.k8s.io/v1 +kind: CSIDriver +metadata: + name: com.hammerspace.csi +spec: + podInfoOnMount: true + requiresRepublish: true + volumeLifecycleModes: + - Persistent + storageCapacity: true + +#### Controller Service +--- +# needed for StatefulSet +kind: Service +apiVersion: v1 +metadata: + name: csi-provisioner + namespace: kube-system + labels: + app: csi-provisioner +spec: + type: ClusterIP + clusterIP: None + # This is needed for the StatefulSet to work properly + # as it uses a headless service to manage the pods. + # The StatefulSet will create a DNS entry for the pods + # in the format ...svc.cluster.local + # where is the name of the pod, is the name of the service, + # and is the namespace of the service. + # This allows the pods to communicate with each other using DNS. + # The StatefulSet will also create a DNS entry for the service in the format ..svc.cluster.local + # which allows the pods to communicate with the service + selector: + app: csi-provisioner +--- +kind: StatefulSet +apiVersion: apps/v1 +metadata: + name: csi-provisioner + namespace: kube-system +spec: + selector: + matchLabels: + app: csi-provisioner + serviceName: "csi-provisioner" + replicas: 1 + template: + metadata: + labels: + app: csi-provisioner + spec: + serviceAccountName: csi-provisioner + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + containers: + - name: csi-provisioner + imagePullPolicy: Always + image: registry.k8s.io/sig-storage/csi-provisioner:v3.6.0 + args: + - "--csi-address=$(CSI_ENDPOINT)" + - "--timeout=60s" # Recommended as shares may take some time to create + - "--v=5" + env: + - name: CSI_ENDPOINT + value: /var/lib/csi/hs-csi.sock + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/ + - name: csi-attacher + imagePullPolicy: Always + image: registry.k8s.io/sig-storage/csi-attacher:v4.4.0 + args: + - "--csi-address=$(CSI_ENDPOINT)" + - "--v=5" + env: + - name: CSI_ENDPOINT + value: /var/lib/csi/hs-csi.sock + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/ + - name: csi-snapshotter + imagePullPolicy: Always + image: registry.k8s.io/sig-storage/csi-snapshotter:v6.2.1 + args: + - "--csi-address=$(CSI_ENDPOINT)" + - "--v=5" + env: + - name: CSI_ENDPOINT + value: /var/lib/csi/hs-csi.sock + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/ + - name: csi-resizer + imagePullPolicy: Always + image: registry.k8s.io/sig-storage/csi-resizer:v1.10.1 + args: + - "--csi-address=$(CSI_ENDPOINT)" + - "--v=5" + env: + - name: CSI_ENDPOINT + value: /var/lib/csi/hs-csi.sock + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/ + - name: hs-csi-plugin-controller + securityContext: + privileged: true + capabilities: + add: ["SYS_ADMIN"] + allowPrivilegeEscalation: true + imagePullPolicy: Always + image: hammerspaceinc/csi-plugin:v1.3.0 + ports: + - containerPort: 9090 + name: metrics + hostPort: 9090 + protocol: TCP + envFrom: + - configMapRef: + name: csi-env-config + env: + - name: CSI_ENDPOINT + value: /var/lib/csi/hs-csi.sock + - name: HS_USERNAME + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: username + - name: HS_PASSWORD + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: password + - name: HS_ENDPOINT + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: endpoint + - name: HS_TLS_VERIFY + value: "false" + - name: CSI_MAJOR_VERSION + value: "1" + # OTel toggles — Prometheus /metrics on :9090 (on by default). Set + # OTEL_TRACES_EXPORTER=console/otlp to also emit spans (off by default). + - name: OTEL_TRACES_EXPORTER + value: "none" + - name: OTEL_METRICS_EXPORTER + value: "prometheus" + - name: OTEL_METRICS_PROMETHEUS_LISTEN + value: ":9090" + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/ + - name: staging-dir + mountPath: /tmp + mountPropagation: Bidirectional + volumes: + - name: socket-dir + emptyDir: {} + - name: staging-dir + hostPath: + path: /tmp +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: csi-provisioner + namespace: kube-system +--- +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: csi-provisioner +rules: + - apiGroups: [""] + resources: ["pods"] + verbs: ["get", "list", "watch"] + # pods/exec: needed by the driver's Freezer to run `fsfreeze --freeze` + # inside the app pod holding a source volume during CreateSnapshot. Without + # this the driver falls back to snapshotting without quiescing, which for + # XFS can produce an inconsistent-log snapshot that log-recovery empties on + # restore. See pkg/driver/freezer.go. + - apiGroups: [""] + resources: ["pods/exec"] + verbs: ["create"] + - apiGroups: [""] + resources: ["persistentvolumes"] + verbs: ["get", "patch", "list", "watch", "create", "delete", "update"] + - apiGroups: [""] + resources: ["persistentvolumeclaims", "persistentvolumeclaims/status"] + verbs: ["get", "patch", "list", "watch", "update"] + - apiGroups: ["storage.k8s.io"] + resources: ["storageclasses"] + verbs: ["get", "list", "watch"] + - apiGroups: [""] + resources: ["nodes"] + verbs: ["get", "list", "watch"] + - apiGroups: [""] + resources: ["events"] + verbs: ["list", "watch", "create", "update", "patch"] + - apiGroups: [""] + resources: ["endpoints"] + verbs: ["list", "watch", "create", "update", "delete", "get"] + - apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "list"] + - apiGroups: ["snapshot.storage.k8s.io"] + resources: ["volumesnapshotclasses"] + verbs: ["get", "list", "watch"] + - apiGroups: ["snapshot.storage.k8s.io"] + resources: ["volumesnapshotcontents"] + verbs: ["create", "get", "list", "watch", "update", "delete"] + - apiGroups: ["snapshot.storage.k8s.io"] + resources: ["volumesnapshots"] + verbs: ["get", "list", "watch", "update"] + - apiGroups: ["apiextensions.k8s.io"] + resources: ["customresourcedefinitions"] + verbs: ["create", "list", "watch", "delete"] + - apiGroups: ["storage.k8s.io"] + resources: ["volumeattachments", "volumeattachments/status"] + verbs: ["get", "list", "watch", "update", "patch"] + - apiGroups: ["storage.k8s.io"] + resources: ["storageclasses"] + verbs: ["get", "list", "watch", "update"] +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: csi-provisioner + namespace: kube-system +subjects: + - kind: ServiceAccount + name: csi-provisioner + namespace: kube-system +roleRef: + kind: ClusterRole + name: csi-provisioner + apiGroup: rbac.authorization.k8s.io +#### Node service +--- +kind: DaemonSet +apiVersion: apps/v1 +metadata: + name: csi-node + namespace: kube-system +spec: + selector: + matchLabels: + app: csi-node + template: + metadata: + labels: + app: csi-node + spec: + serviceAccount: csi-node + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + containers: + - name: csi-resizer + imagePullPolicy: Always + image: registry.k8s.io/sig-storage/csi-resizer:v1.10.1 + args: + - "--csi-address=$(CSI_ENDPOINT)" + - "--v=5" + env: + - name: CSI_ENDPOINT + value: /csi/csi.sock + volumeMounts: + - name: socket-dir + mountPath: /csi + - name: driver-registrar + imagePullPolicy: Always + image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.0 + lifecycle: + preStop: + exec: + command: ["/bin/sh", "-c", '[ -n "$REG_SOCKET" ] && rm -rf "$REG_SOCKET" || echo "REG_SOCKET not set, skipping delete"'] + args: + - "--v=5" + - "--csi-address=$(CSI_ENDPOINT)" + - "--kubelet-registration-path=$(REG_SOCKET)" + securityContext: + privileged: true + env: + - name: CSI_ENDPOINT + value: /csi/csi.sock + - name: REG_SOCKET + value: /var/lib/kubelet/plugins_registry/com.hammerspace.csi/csi.sock + - name: KUBE_NODE_NAME + valueFrom: + fieldRef: + apiVersion: v1 + fieldPath: spec.nodeName + volumeMounts: + - name: socket-dir + mountPath: /csi + - name: registration-dir + mountPath: /registration + - name: hs-csi-plugin-node + securityContext: + privileged: true + capabilities: + add: ["SYS_ADMIN"] + allowPrivilegeEscalation: true + imagePullPolicy: Always + image: hammerspaceinc/csi-plugin:v1.3.0 + ports: + - containerPort: 9091 + name: metrics + hostPort: 9091 + protocol: TCP + envFrom: + - configMapRef: + name: csi-env-config + env: + - name: CSI_ENDPOINT + value: /csi/csi.sock + - name: HS_USERNAME + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: username + - name: HS_PASSWORD + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: password + - name: HS_ENDPOINT + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: endpoint + - name: CSI_NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + - name: HS_TLS_VERIFY + value: "false" + - name: CSI_MAJOR_VERSION + value: "1" + # OTel toggles. hostNetwork=true means we bind directly to the + # node's :9091 (must differ from controller's :9090 in case the + # controller lands on the same node). + - name: OTEL_TRACES_EXPORTER + value: "none" + - name: OTEL_METRICS_EXPORTER + value: "prometheus" + - name: OTEL_METRICS_PROMETHEUS_LISTEN + value: ":9091" + volumeMounts: + - name: socket-dir + mountPath: /csi + - name: registration-dir + mountPath: /registration + mountPropagation: Bidirectional + - name: mountpoint-dir + mountPath: /var/lib/kubelet/ + mountPropagation: Bidirectional + - name: rootshare-dir + mountPath: /var/lib/hammerspace/ + mountPropagation: Bidirectional + - name: dev-dir + mountPath: /dev + - name: staging-dir + mountPath: /tmp + mountPropagation: Bidirectional + volumes: + - name: socket-dir + hostPath: + path: /var/lib/kubelet/plugins_registry/com.hammerspace.csi + type: DirectoryOrCreate + - name: mountpoint-dir + hostPath: + path: /var/lib/kubelet/ + - name: rootshare-dir + hostPath: + path: /var/lib/hammerspace/ + - name: registration-dir + hostPath: + path: /var/lib/kubelet/plugins_registry/ + - name: dev-dir + hostPath: + path: /dev + - name: staging-dir + hostPath: + path: /tmp +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: csi-node + namespace: kube-system +--- +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: csi-node + namespace: kube-system +rules: + - apiGroups: [""] + resources: ["pods"] + verbs: ["list", "watch"] + - apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "list"] + - apiGroups: [""] + resources: ["nodes"] + verbs: ["get", "list", "update"] + - apiGroups: [""] + resources: ["namespaces", "events"] + verbs: ["get", "list", "create"] + - apiGroups: [""] + resources: ["persistentvolumes", "persistentvolumeclaims", "persistentvolumeclaims/status","events"] + verbs: ["get", "list", "watch", "update", "patch"] + - apiGroups: ["storage.k8s.io"] + resources: ["volumeattachments", "volumeattachments/status"] + verbs: ["get", "list", "watch", "update", "patch"] + +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: csi-node + namespace: kube-system +subjects: + - kind: ServiceAccount + name: csi-node + namespace: kube-system +roleRef: + kind: ClusterRole + name: csi-node + apiGroup: rbac.authorization.k8s.io + +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: csi-provisioner-volumeattachment-status-binding + namespace: kube-system +subjects: + - kind: ServiceAccount + name: csi-provisioner + namespace: kube-system +roleRef: + kind: ClusterRole + name: csi-provisioner-volumeattachment-status + apiGroup: rbac.authorization.k8s.io +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: csi-env-config + namespace: kube-system +# Example: "300ms". Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". +data: + MOUNT_CHECK_TIMEOUT: "30s" + UNMOUNT_RETRY_COUNT: "5" + UNMOUNT_RETRY_INTERVAL: "1s" diff --git a/deploy/kubernetes/kubernetes-1.36/plugin.yaml b/deploy/kubernetes/kubernetes-1.36/plugin.yaml new file mode 100644 index 0000000..f129ddb --- /dev/null +++ b/deploy/kubernetes/kubernetes-1.36/plugin.yaml @@ -0,0 +1,474 @@ +# Hammerspace CSI plugin manifest, validated on Kubernetes 1.36 (and works on +# 1.30+ in practice). It is the 1.29 manifest plus the observability wiring: +# - a host-networked `metrics` port (9090 controller / 9091 node), and +# - OTel env vars on both driver containers. +# +# Observability env (see deploy/monitoring/README.md and docs/observability.md): +# OTEL_METRICS_EXPORTER=prometheus metrics on; scrape :9090 / :9091 +# OTEL_METRICS_PROMETHEUS_LISTEN=:9090|:9091 per-container listen addr +# OTEL_TRACES_EXPORTER=console spans to pod stdout — VERBOSE; +# set to "none" to disable in prod, +# or "otlp" (+ OTEL_EXPORTER_OTLP_ENDPOINT) +# to ship to a collector. +# Sidecar image tags (csi-provisioner / csi-attacher / csi-snapshotter / +# csi-resizer / node-driver-registrar) match the 1.29 manifest. +#### CSI Object +apiVersion: storage.k8s.io/v1 +kind: CSIDriver +metadata: + name: com.hammerspace.csi +spec: + podInfoOnMount: true + requiresRepublish: true + volumeLifecycleModes: + - Persistent + storageCapacity: true + +#### Controller Service +--- +# needed for StatefulSet +kind: Service +apiVersion: v1 +metadata: + name: csi-provisioner + namespace: kube-system + labels: + app: csi-provisioner +spec: + type: ClusterIP + clusterIP: None + # This is needed for the StatefulSet to work properly + # as it uses a headless service to manage the pods. + # The StatefulSet will create a DNS entry for the pods + # in the format ...svc.cluster.local + # where is the name of the pod, is the name of the service, + # and is the namespace of the service. + # This allows the pods to communicate with each other using DNS. + # The StatefulSet will also create a DNS entry for the service in the format ..svc.cluster.local + # which allows the pods to communicate with the service + selector: + app: csi-provisioner +--- +kind: StatefulSet +apiVersion: apps/v1 +metadata: + name: csi-provisioner + namespace: kube-system +spec: + selector: + matchLabels: + app: csi-provisioner + serviceName: "csi-provisioner" + replicas: 1 + template: + metadata: + labels: + app: csi-provisioner + spec: + serviceAccountName: csi-provisioner + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + containers: + - name: csi-provisioner + imagePullPolicy: Always + image: registry.k8s.io/sig-storage/csi-provisioner:v3.6.0 + args: + - "--csi-address=$(CSI_ENDPOINT)" + - "--timeout=60s" # Recommended as shares may take some time to create + - "--v=5" + env: + - name: CSI_ENDPOINT + value: /var/lib/csi/hs-csi.sock + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/ + - name: csi-attacher + imagePullPolicy: Always + image: registry.k8s.io/sig-storage/csi-attacher:v4.4.0 + args: + - "--csi-address=$(CSI_ENDPOINT)" + - "--v=5" + env: + - name: CSI_ENDPOINT + value: /var/lib/csi/hs-csi.sock + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/ + - name: csi-snapshotter + imagePullPolicy: Always + image: registry.k8s.io/sig-storage/csi-snapshotter:v6.2.1 + args: + - "--csi-address=$(CSI_ENDPOINT)" + - "--v=5" + env: + - name: CSI_ENDPOINT + value: /var/lib/csi/hs-csi.sock + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/ + - name: csi-resizer + imagePullPolicy: Always + image: registry.k8s.io/sig-storage/csi-resizer:v1.10.1 + args: + - "--csi-address=$(CSI_ENDPOINT)" + - "--v=5" + env: + - name: CSI_ENDPOINT + value: /var/lib/csi/hs-csi.sock + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/ + - name: hs-csi-plugin-controller + securityContext: + privileged: true + capabilities: + add: ["SYS_ADMIN"] + allowPrivilegeEscalation: true + imagePullPolicy: Always + image: hammerspaceinc/csi-plugin:v1.3.0 + ports: + - containerPort: 9090 + name: metrics + hostPort: 9090 + protocol: TCP + envFrom: + - configMapRef: + name: csi-env-config + env: + - name: CSI_ENDPOINT + value: /var/lib/csi/hs-csi.sock + - name: HS_USERNAME + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: username + - name: HS_PASSWORD + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: password + - name: HS_ENDPOINT + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: endpoint + - name: HS_TLS_VERIFY + value: "false" + - name: CSI_MAJOR_VERSION + value: "1" + # OTel toggles — Prometheus /metrics on :9090 (on by default). Set + # OTEL_TRACES_EXPORTER=console/otlp to also emit spans (off by default). + - name: OTEL_TRACES_EXPORTER + value: "none" + - name: OTEL_METRICS_EXPORTER + value: "prometheus" + - name: OTEL_METRICS_PROMETHEUS_LISTEN + value: ":9090" + volumeMounts: + - name: socket-dir + mountPath: /var/lib/csi/ + - name: staging-dir + mountPath: /tmp + mountPropagation: Bidirectional + volumes: + - name: socket-dir + emptyDir: {} + - name: staging-dir + hostPath: + path: /tmp +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: csi-provisioner + namespace: kube-system +--- +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: csi-provisioner +rules: + - apiGroups: [""] + resources: ["pods"] + verbs: ["get", "list", "watch"] + # pods/exec: needed by the driver's Freezer to run `fsfreeze --freeze` + # inside the app pod holding a source volume during CreateSnapshot. Without + # this the driver falls back to snapshotting without quiescing, which for + # XFS can produce an inconsistent-log snapshot that log-recovery empties on + # restore. See pkg/driver/freezer.go. + - apiGroups: [""] + resources: ["pods/exec"] + verbs: ["create"] + - apiGroups: [""] + resources: ["persistentvolumes"] + verbs: ["get", "patch", "list", "watch", "create", "delete", "update"] + - apiGroups: [""] + resources: ["persistentvolumeclaims", "persistentvolumeclaims/status"] + verbs: ["get", "patch", "list", "watch", "update"] + - apiGroups: ["storage.k8s.io"] + resources: ["storageclasses"] + verbs: ["get", "list", "watch"] + - apiGroups: [""] + resources: ["nodes"] + verbs: ["get", "list", "watch"] + - apiGroups: [""] + resources: ["events"] + verbs: ["list", "watch", "create", "update", "patch"] + - apiGroups: [""] + resources: ["endpoints"] + verbs: ["list", "watch", "create", "update", "delete", "get"] + - apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "list"] + - apiGroups: ["snapshot.storage.k8s.io"] + resources: ["volumesnapshotclasses"] + verbs: ["get", "list", "watch"] + - apiGroups: ["snapshot.storage.k8s.io"] + resources: ["volumesnapshotcontents"] + verbs: ["create", "get", "list", "watch", "update", "delete"] + - apiGroups: ["snapshot.storage.k8s.io"] + resources: ["volumesnapshots"] + verbs: ["get", "list", "watch", "update"] + - apiGroups: ["apiextensions.k8s.io"] + resources: ["customresourcedefinitions"] + verbs: ["create", "list", "watch", "delete"] + - apiGroups: ["storage.k8s.io"] + resources: ["volumeattachments", "volumeattachments/status"] + verbs: ["get", "list", "watch", "update", "patch"] + - apiGroups: ["storage.k8s.io"] + resources: ["storageclasses"] + verbs: ["get", "list", "watch", "update"] +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: csi-provisioner + namespace: kube-system +subjects: + - kind: ServiceAccount + name: csi-provisioner + namespace: kube-system +roleRef: + kind: ClusterRole + name: csi-provisioner + apiGroup: rbac.authorization.k8s.io +#### Node service +--- +kind: DaemonSet +apiVersion: apps/v1 +metadata: + name: csi-node + namespace: kube-system +spec: + selector: + matchLabels: + app: csi-node + template: + metadata: + labels: + app: csi-node + spec: + serviceAccount: csi-node + hostNetwork: true + dnsPolicy: ClusterFirstWithHostNet + containers: + - name: csi-resizer + imagePullPolicy: Always + image: registry.k8s.io/sig-storage/csi-resizer:v1.10.1 + args: + - "--csi-address=$(CSI_ENDPOINT)" + - "--v=5" + env: + - name: CSI_ENDPOINT + value: /csi/csi.sock + volumeMounts: + - name: socket-dir + mountPath: /csi + - name: driver-registrar + imagePullPolicy: Always + image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.9.0 + lifecycle: + preStop: + exec: + command: ["/bin/sh", "-c", '[ -n "$REG_SOCKET" ] && rm -rf "$REG_SOCKET" || echo "REG_SOCKET not set, skipping delete"'] + args: + - "--v=5" + - "--csi-address=$(CSI_ENDPOINT)" + - "--kubelet-registration-path=$(REG_SOCKET)" + securityContext: + privileged: true + env: + - name: CSI_ENDPOINT + value: /csi/csi.sock + - name: REG_SOCKET + value: /var/lib/kubelet/plugins_registry/com.hammerspace.csi/csi.sock + - name: KUBE_NODE_NAME + valueFrom: + fieldRef: + apiVersion: v1 + fieldPath: spec.nodeName + volumeMounts: + - name: socket-dir + mountPath: /csi + - name: registration-dir + mountPath: /registration + - name: hs-csi-plugin-node + securityContext: + privileged: true + capabilities: + add: ["SYS_ADMIN"] + allowPrivilegeEscalation: true + imagePullPolicy: Always + image: hammerspaceinc/csi-plugin:v1.3.0 + ports: + - containerPort: 9091 + name: metrics + hostPort: 9091 + protocol: TCP + envFrom: + - configMapRef: + name: csi-env-config + env: + - name: CSI_ENDPOINT + value: /csi/csi.sock + - name: HS_USERNAME + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: username + - name: HS_PASSWORD + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: password + - name: HS_ENDPOINT + valueFrom: + secretKeyRef: + name: com.hammerspace.csi.credentials + key: endpoint + - name: CSI_NODE_NAME + valueFrom: + fieldRef: + fieldPath: spec.nodeName + - name: HS_TLS_VERIFY + value: "false" + - name: CSI_MAJOR_VERSION + value: "1" + # OTel toggles. hostNetwork=true means we bind directly to the + # node's :9091 (must differ from controller's :9090 in case the + # controller lands on the same node). + - name: OTEL_TRACES_EXPORTER + value: "none" + - name: OTEL_METRICS_EXPORTER + value: "prometheus" + - name: OTEL_METRICS_PROMETHEUS_LISTEN + value: ":9091" + volumeMounts: + - name: socket-dir + mountPath: /csi + - name: registration-dir + mountPath: /registration + mountPropagation: Bidirectional + - name: mountpoint-dir + mountPath: /var/lib/kubelet/ + mountPropagation: Bidirectional + - name: rootshare-dir + mountPath: /var/lib/hammerspace/ + mountPropagation: Bidirectional + - name: dev-dir + mountPath: /dev + - name: staging-dir + mountPath: /tmp + mountPropagation: Bidirectional + volumes: + - name: socket-dir + hostPath: + path: /var/lib/kubelet/plugins_registry/com.hammerspace.csi + type: DirectoryOrCreate + - name: mountpoint-dir + hostPath: + path: /var/lib/kubelet/ + - name: rootshare-dir + hostPath: + path: /var/lib/hammerspace/ + - name: registration-dir + hostPath: + path: /var/lib/kubelet/plugins_registry/ + - name: dev-dir + hostPath: + path: /dev + - name: staging-dir + hostPath: + path: /tmp +--- +apiVersion: v1 +kind: ServiceAccount +metadata: + name: csi-node + namespace: kube-system +--- +kind: ClusterRole +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: csi-node + namespace: kube-system +rules: + - apiGroups: [""] + resources: ["pods"] + verbs: ["list", "watch"] + - apiGroups: [""] + resources: ["secrets"] + verbs: ["get", "list"] + - apiGroups: [""] + resources: ["nodes"] + verbs: ["get", "list", "update"] + - apiGroups: [""] + resources: ["namespaces", "events"] + verbs: ["get", "list", "create"] + - apiGroups: [""] + resources: ["persistentvolumes", "persistentvolumeclaims", "persistentvolumeclaims/status","events"] + verbs: ["get", "list", "watch", "update", "patch"] + - apiGroups: ["storage.k8s.io"] + resources: ["volumeattachments", "volumeattachments/status"] + verbs: ["get", "list", "watch", "update", "patch"] + +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: csi-node + namespace: kube-system +subjects: + - kind: ServiceAccount + name: csi-node + namespace: kube-system +roleRef: + kind: ClusterRole + name: csi-node + apiGroup: rbac.authorization.k8s.io + +--- +kind: ClusterRoleBinding +apiVersion: rbac.authorization.k8s.io/v1 +metadata: + name: csi-provisioner-volumeattachment-status-binding + namespace: kube-system +subjects: + - kind: ServiceAccount + name: csi-provisioner + namespace: kube-system +roleRef: + kind: ClusterRole + name: csi-provisioner-volumeattachment-status + apiGroup: rbac.authorization.k8s.io +--- +apiVersion: v1 +kind: ConfigMap +metadata: + name: csi-env-config + namespace: kube-system +# Example: "300ms". Valid time units are "ns", "us" (or "µs"), "ms", "s", "m", "h". +data: + MOUNT_CHECK_TIMEOUT: "30s" + UNMOUNT_RETRY_COUNT: "5" + UNMOUNT_RETRY_INTERVAL: "1s" diff --git a/deploy/monitoring/README.md b/deploy/monitoring/README.md new file mode 100644 index 0000000..3e6f56a --- /dev/null +++ b/deploy/monitoring/README.md @@ -0,0 +1,69 @@ +# Monitoring the Hammerspace CSI driver + +The driver emits OpenTelemetry **metrics** (Prometheus/OTLP) and **traces** +(console/OTLP). This directory has the pieces to scrape those metrics and view +them in Grafana: + +``` +deploy/monitoring/ +├── victoriametrics/scrape.yml # example scrape config (VM or Prometheus) +└── grafana/hs-csi-driver-dashboard.json # importable "Hammerspace CSI Driver" dashboard +``` + +The full metric/label catalog and tracing details live in +[`docs/observability.md`](../../docs/observability.md). + +## 1. Enable metrics on the driver + +Metrics are **off by default**. Turn them on via environment variables on the +CSI containers (both pods run `hostNetwork: true`, so `/metrics` is served on the +Kubernetes **node host IP**): + +| Variable | Controller | Node | +|---|---|---| +| `OTEL_METRICS_EXPORTER` | `prometheus` | `prometheus` | +| `OTEL_METRICS_PROMETHEUS_LISTEN` | `:9090` (default) | `:9091` (avoid clashing with the controller on a shared host) | + +The node uses `:9091` because a controller pod and a node pod can land on the +same host, and both are host-networked. + +Patch an existing install: + +```bash +kubectl -n kube-system patch statefulset csi-provisioner -p '{"spec":{"template":{"spec":{"containers":[{"name":"hs-csi-plugin-controller","env":[{"name":"OTEL_METRICS_EXPORTER","value":"prometheus"}]}]}}}}' + +kubectl -n kube-system patch daemonset csi-node -p '{"spec":{"template":{"spec":{"containers":[{"name":"hs-csi-plugin-node","env":[{"name":"OTEL_METRICS_EXPORTER","value":"prometheus"},{"name":"OTEL_METRICS_PROMETHEUS_LISTEN","value":":9091"}]}]}}}}' +``` + +## 2. Scrape the endpoints + +Copy `victoriametrics/scrape.yml`, replace the placeholder target IPs with your +Kubernetes node IPs, and load it as your VictoriaMetrics +`-promscrape.config=/etc/victoriametrics/scrape.yml` (VM re-reads it every +`-promscrape.configCheckInterval`, no restart) or paste the two jobs under a +Prometheus `scrape_configs:` block. + +Only the node running `csi-provisioner` answers on `:9090` (the rest report +`up=0`, harmless); every node answers on `:9091`. + +Verify it's flowing: + +```bash +curl -s 'http://:8428/api/v1/query?query=up{service="hs-csi"}' +curl -s 'http://:8428/api/v1/query?query=sum(hs_csi_anvil_requests_total)' +``` + +## 3. Import the Grafana dashboard + +Grafana → **Dashboards → New → Import** → upload +`grafana/hs-csi-driver-dashboard.json` (uid `hs-csi-driver`), and select your +VictoriaMetrics/Prometheus datasource. Panels cover controller/node RPC latency +(p50/p95/p99), the Anvil REST client (rate/latency/status, including the +`GET /files` 404/500 type-probe traffic), and the file- and share-backed +provisioning paths. + +## Traces (optional) + +Set `OTEL_TRACES_EXPORTER=console` (spans to pod stdout, readable via +`kubectl logs`) or `OTEL_TRACES_EXPORTER=otlp` with +`OTEL_EXPORTER_OTLP_ENDPOINT=:4317`. See `docs/observability.md`. diff --git a/deploy/monitoring/grafana/hs-csi-driver-dashboard.json b/deploy/monitoring/grafana/hs-csi-driver-dashboard.json new file mode 100644 index 0000000..a0dd15f --- /dev/null +++ b/deploy/monitoring/grafana/hs-csi-driver-dashboard.json @@ -0,0 +1,3193 @@ +{ + "editable": true, + "panels": [ + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 0 + }, + "id": 9, + "panels": [], + "title": "Load overview", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "How many operations are executing right now in the driver. Spikes here indicate concurrency; sustained levels indicate lock hold or blocked work.", + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "smooth", + "showPoints": "never", + "spanNulls": true + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 1 + }, + "id": 10, + "options": { + "legend": { + "calcs": [ + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "sum by (operation)(hs_csi_operation_inflight)", + "instant": false, + "legendFormat": "{{operation}}", + "range": true, + "refId": "A" + } + ], + "title": "In-flight operations (by kind)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "", + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "smooth", + "showPoints": "never", + "spanNulls": true + }, + "unit": "ops" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 1 + }, + "id": 11, + "options": { + "legend": { + "calcs": [ + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "sum by (operation)(rate(hs_csi_operation_duration_seconds_count[5m]))", + "instant": false, + "legendFormat": "{{operation}}", + "range": true, + "refId": "A" + } + ], + "title": "Ops per second (by kind)", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 9 + }, + "id": 12, + "panels": [], + "title": "Cluster storage state (from kube-state-metrics)", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 6, + "w": 4, + "x": 0, + "y": 10 + }, + "id": 13, + "options": { + "colorMode": "value", + "graphMode": "area", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ] + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "count(kube_persistentvolumeclaim_status_phase{phase=\"Bound\"} == 1)", + "instant": false, + "legendFormat": "Bound", + "range": true, + "refId": "A" + } + ], + "title": "PVCs Bound (now)", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 6, + "w": 4, + "x": 4, + "y": 10 + }, + "id": 14, + "options": { + "colorMode": "value", + "graphMode": "area", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ] + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "count(kube_persistentvolumeclaim_status_phase{phase=\"Pending\"} == 1)", + "instant": false, + "legendFormat": "Pending", + "range": true, + "refId": "A" + } + ], + "title": "PVCs Pending (now)", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 6, + "w": 4, + "x": 8, + "y": 10 + }, + "id": 15, + "options": { + "colorMode": "value", + "graphMode": "area", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ] + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "count(kube_persistentvolume_status_phase{phase=\"Bound\"} == 1)", + "instant": false, + "legendFormat": "Bound", + "range": true, + "refId": "A" + } + ], + "title": "PVs Bound (now)", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 6, + "w": 4, + "x": 12, + "y": 10 + }, + "id": 16, + "options": { + "colorMode": "value", + "graphMode": "area", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ] + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "count(kube_persistentvolume_status_phase{phase=\"Released\"} == 1)", + "instant": false, + "legendFormat": "Released", + "range": true, + "refId": "A" + } + ], + "title": "PVs Released/pending-delete (now)", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "Positive when the test is actively provisioning. Flat when waiting on the driver.", + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "smooth", + "showPoints": "never", + "spanNulls": true + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 6, + "w": 8, + "x": 16, + "y": 10 + }, + "id": 17, + "options": { + "legend": { + "calcs": [ + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "sum(increase(kube_persistentvolumeclaim_created[10s]))", + "instant": false, + "legendFormat": "PVCs created", + "range": true, + "refId": "A" + } + ], + "title": "PVC creation rate (per 10s)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "Stacked count of PVCs by phase. Growing Pending band with flat Bound = provisioning blocked. Growing Bound = healthy progress.", + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "smooth", + "showPoints": "never", + "spanNulls": true, + "stacking": { + "mode": "normal" + } + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 16 + }, + "id": 18, + "options": { + "legend": { + "calcs": [ + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "sum(kube_persistentvolumeclaim_status_phase{phase=\"Bound\"})", + "instant": false, + "legendFormat": "Bound", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "sum(kube_persistentvolumeclaim_status_phase{phase=\"Pending\"})", + "instant": false, + "legendFormat": "Pending", + "range": true, + "refId": "B" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "sum(kube_persistentvolumeclaim_status_phase{phase=\"Lost\"})", + "instant": false, + "legendFormat": "Lost", + "range": true, + "refId": "C" + } + ], + "title": "PVCs by phase over time", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "Released grows during cleanup; if Released stays high while driver logs show DeleteVolume errors, backing files are stuck on Anvil.", + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "smooth", + "showPoints": "never", + "spanNulls": true, + "stacking": { + "mode": "normal" + } + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 16 + }, + "id": 19, + "options": { + "legend": { + "calcs": [ + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "sum(kube_persistentvolume_status_phase{phase=\"Bound\"})", + "instant": false, + "legendFormat": "Bound", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "sum(kube_persistentvolume_status_phase{phase=\"Available\"})", + "instant": false, + "legendFormat": "Available", + "range": true, + "refId": "B" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "sum(kube_persistentvolume_status_phase{phase=\"Released\"})", + "instant": false, + "legendFormat": "Released", + "range": true, + "refId": "C" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "sum(kube_persistentvolume_status_phase{phase=\"Failed\"})", + "instant": false, + "legendFormat": "Failed", + "range": true, + "refId": "D" + } + ], + "title": "PVs by phase over time", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "", + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "smooth", + "showPoints": "never", + "spanNulls": true + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 0, + "y": 24 + }, + "id": 20, + "options": { + "legend": { + "calcs": [ + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "count by (storageclass)(kube_persistentvolumeclaim_info)", + "instant": false, + "legendFormat": "{{storageclass}}", + "range": true, + "refId": "A" + } + ], + "title": "PVCs by StorageClass", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "", + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "smooth", + "showPoints": "never", + "spanNulls": true + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 12, + "x": 12, + "y": 24 + }, + "id": 21, + "options": { + "legend": { + "calcs": [ + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "count by (namespace)(kube_persistentvolumeclaim_info)", + "instant": false, + "legendFormat": "{{namespace}}", + "range": true, + "refId": "A" + } + ], + "title": "PVCs by namespace", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 47 + }, + "id": 22, + "panels": [], + "title": "CSI Controller RPC latency (p50 / p95 / p99)", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "", + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "smooth", + "showPoints": "never", + "spanNulls": true + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 48 + }, + "id": 23, + "options": { + "legend": { + "calcs": [ + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"Controller/CreateVolume\"}[5m])))", + "instant": false, + "legendFormat": "p50", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.95, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"Controller/CreateVolume\"}[5m])))", + "instant": false, + "legendFormat": "p95", + "range": true, + "refId": "B" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"Controller/CreateVolume\"}[5m])))", + "instant": false, + "legendFormat": "p99", + "range": true, + "refId": "C" + } + ], + "title": "Controller/CreateVolume", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "", + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "smooth", + "showPoints": "never", + "spanNulls": true + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 48 + }, + "id": 24, + "options": { + "legend": { + "calcs": [ + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"Controller/DeleteVolume\"}[5m])))", + "instant": false, + "legendFormat": "p50", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.95, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"Controller/DeleteVolume\"}[5m])))", + "instant": false, + "legendFormat": "p95", + "range": true, + "refId": "B" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"Controller/DeleteVolume\"}[5m])))", + "instant": false, + "legendFormat": "p99", + "range": true, + "refId": "C" + } + ], + "title": "Controller/DeleteVolume", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "", + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "smooth", + "showPoints": "never", + "spanNulls": true + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 48 + }, + "id": 25, + "options": { + "legend": { + "calcs": [ + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"Controller/CreateSnapshot\"}[5m])))", + "instant": false, + "legendFormat": "p50", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.95, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"Controller/CreateSnapshot\"}[5m])))", + "instant": false, + "legendFormat": "p95", + "range": true, + "refId": "B" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.99, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"Controller/CreateSnapshot\"}[5m])))", + "instant": false, + "legendFormat": "p99", + "range": true, + "refId": "C" + } + ], + "title": "Controller/CreateSnapshot", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 56 + }, + "id": 26, + "panels": [], + "title": "CSI Node RPC latency (p95)", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "", + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "smooth", + "showPoints": "never", + "spanNulls": true + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 6, + "x": 0, + "y": 57 + }, + "id": 27, + "options": { + "legend": { + "calcs": [ + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"Node/NodeStageVolume\"}[5m])))", + "instant": false, + "legendFormat": "p50", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.95, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"Node/NodeStageVolume\"}[5m])))", + "instant": false, + "legendFormat": "p95", + "range": true, + "refId": "B" + } + ], + "title": "Node/NodeStageVolume", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "", + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "smooth", + "showPoints": "never", + "spanNulls": true + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 6, + "x": 6, + "y": 57 + }, + "id": 28, + "options": { + "legend": { + "calcs": [ + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"Node/NodePublishVolume\"}[5m])))", + "instant": false, + "legendFormat": "p50", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.95, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"Node/NodePublishVolume\"}[5m])))", + "instant": false, + "legendFormat": "p95", + "range": true, + "refId": "B" + } + ], + "title": "Node/NodePublishVolume", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "", + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "smooth", + "showPoints": "never", + "spanNulls": true + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 6, + "x": 12, + "y": 57 + }, + "id": 29, + "options": { + "legend": { + "calcs": [ + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"Node/NodeUnpublishVolume\"}[5m])))", + "instant": false, + "legendFormat": "p50", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.95, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"Node/NodeUnpublishVolume\"}[5m])))", + "instant": false, + "legendFormat": "p95", + "range": true, + "refId": "B" + } + ], + "title": "Node/NodeUnpublishVolume", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "", + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "smooth", + "showPoints": "never", + "spanNulls": true + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 7, + "w": 6, + "x": 18, + "y": 57 + }, + "id": 30, + "options": { + "legend": { + "calcs": [ + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"Node/NodeUnstageVolume\"}[5m])))", + "instant": false, + "legendFormat": "p50", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.95, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"Node/NodeUnstageVolume\"}[5m])))", + "instant": false, + "legendFormat": "p95", + "range": true, + "refId": "B" + } + ], + "title": "Node/NodeUnstageVolume", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 64 + }, + "id": 31, + "panels": [], + "title": "Anvil REST client (HammerspaceClient.doRequest)", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "", + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "smooth", + "showPoints": "never", + "spanNulls": true + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 65 + }, + "id": 32, + "options": { + "legend": { + "calcs": [ + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.95, sum by (le, http_path)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"HammerspaceClient.doRequest\"}[5m])))", + "instant": false, + "legendFormat": "{{http_path}}", + "range": true, + "refId": "A" + } + ], + "title": "Anvil call latency p95 (by HTTPS path)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "", + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "smooth", + "showPoints": "never", + "spanNulls": true + }, + "unit": "ops" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 65 + }, + "id": 33, + "options": { + "legend": { + "calcs": [ + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "sum by (http_method)(rate(hs_csi_operation_duration_seconds_count{operation=\"HammerspaceClient.doRequest\"}[5m]))", + "instant": false, + "legendFormat": "{{http_method}}", + "range": true, + "refId": "A" + } + ], + "title": "Anvil call rate (by HTTPS method)", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 81 + }, + "id": 34, + "panels": [], + "title": "Local shell-out timings", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "", + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "smooth", + "showPoints": "never", + "spanNulls": true + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 0, + "y": 82 + }, + "id": 35, + "options": { + "legend": { + "calcs": [ + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"MountShare\"}[5m])))", + "instant": false, + "legendFormat": "p50", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.95, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"MountShare\"}[5m])))", + "instant": false, + "legendFormat": "p95", + "range": true, + "refId": "B" + } + ], + "title": "MountShare (NFS mount syscall)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "", + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "smooth", + "showPoints": "never", + "spanNulls": true + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 8, + "y": 82 + }, + "id": 36, + "options": { + "legend": { + "calcs": [ + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"MakeEmptyRawFile\"}[5m])))", + "instant": false, + "legendFormat": "p50", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.95, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"MakeEmptyRawFile\"}[5m])))", + "instant": false, + "legendFormat": "p95", + "range": true, + "refId": "B" + } + ], + "title": "MakeEmptyRawFile (qemu-img)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "", + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "smooth", + "showPoints": "never", + "spanNulls": true + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 8, + "x": 16, + "y": 82 + }, + "id": 37, + "options": { + "legend": { + "calcs": [ + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum by (le, fsType)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"FormatDevice\"}[5m])))", + "instant": false, + "legendFormat": "p50 {{fsType}}", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.95, sum by (le, fsType)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"FormatDevice\"}[5m])))", + "instant": false, + "legendFormat": "p95 {{fsType}}", + "range": true, + "refId": "B" + } + ], + "title": "FormatDevice (mkfs.)", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 90 + }, + "id": 38, + "panels": [], + "title": "Unmount path", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "", + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "smooth", + "showPoints": "never", + "spanNulls": true + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 91 + }, + "id": 39, + "options": { + "legend": { + "calcs": [ + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"UnmountBackingShareIfUnused\"}[5m])))", + "instant": false, + "legendFormat": "p50", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.95, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"UnmountBackingShareIfUnused\"}[5m])))", + "instant": false, + "legendFormat": "p95", + "range": true, + "refId": "B" + } + ], + "title": "UnmountBackingShareIfUnused (controller cleanup)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "", + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "smooth", + "showPoints": "never", + "spanNulls": true + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 91 + }, + "id": 40, + "options": { + "legend": { + "calcs": [ + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.50, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"UnmountFilesystem\"}[5m])))", + "instant": false, + "legendFormat": "p50", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "histogram_quantile(0.95, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"UnmountFilesystem\"}[5m])))", + "instant": false, + "legendFormat": "p95", + "range": true, + "refId": "B" + } + ], + "title": "UnmountFilesystem (node-side)", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 99 + }, + "id": 41, + "panels": [], + "title": "Errors", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "Non-zero here means an operation returned an error. Cross-reference with 'ops per second' to see failure ratio.", + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "smooth", + "showPoints": "never", + "spanNulls": true + }, + "unit": "ops" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 24, + "x": 0, + "y": 100 + }, + "id": 42, + "options": { + "legend": { + "calcs": [ + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "sum by (operation)(rate(hs_csi_operation_errors_total[5m]))", + "instant": false, + "legendFormat": "{{operation}}", + "range": true, + "refId": "A" + } + ], + "title": "Error rate by operation", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 108 + }, + "id": 43, + "panels": [], + "title": "Driver health", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "", + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "smooth", + "showPoints": "never", + "spanNulls": true + }, + "unit": "bytes" + }, + "overrides": [] + }, + "gridPos": { + "h": 6, + "w": 12, + "x": 0, + "y": 109 + }, + "id": 44, + "options": { + "legend": { + "calcs": [ + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "process_resident_memory_bytes{job=~\"hs-csi-.*\"}", + "instant": false, + "legendFormat": "{{job}} {{instance}}", + "range": true, + "refId": "A" + } + ], + "title": "Driver memory (RSS)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "", + "fieldConfig": { + "defaults": { + "custom": { + "drawStyle": "line", + "fillOpacity": 15, + "lineInterpolation": "smooth", + "showPoints": "never", + "spanNulls": true + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 6, + "w": 12, + "x": 12, + "y": 109 + }, + "id": 45, + "options": { + "legend": { + "calcs": [ + "mean", + "max" + ], + "displayMode": "table", + "placement": "bottom" + }, + "tooltip": { + "mode": "multi" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "go_goroutines{job=~\"hs-csi-.*\"}", + "instant": false, + "legendFormat": "{{job}} {{instance}}", + "range": true, + "refId": "A" + } + ], + "title": "Goroutines", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 2, + "showPoints": "never", + "spanNulls": true + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 24, + "x": 0, + "y": 31 + }, + "id": 46, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "none" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "expr": "count(kube_customresource_volumesnapshot_info)", + "instant": false, + "legendFormat": "Total", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "expr": "sum(kube_customresource_volumesnapshot_ready)", + "instant": false, + "legendFormat": "Ready", + "range": true, + "refId": "B" + } + ], + "title": "VolumeSnapshots over time", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 2, + "showPoints": "never", + "spanNulls": true + }, + "unit": "bytes" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 39 + }, + "id": 47, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "expr": "sum(kube_persistentvolume_capacity_bytes)", + "instant": false, + "legendFormat": "Provisioned (PV capacity)", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "expr": "sum(kube_persistentvolumeclaim_resource_requests_storage_bytes)", + "instant": false, + "legendFormat": "Requested (PVC)", + "range": true, + "refId": "B" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "expr": "sum(kubelet_volume_stats_used_bytes)", + "instant": false, + "legendFormat": "In use (actual)", + "range": true, + "refId": "C" + } + ], + "title": "PV capacity: provisioned vs requested vs in-use", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 2, + "showPoints": "never", + "spanNulls": true + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 39 + }, + "id": 48, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "expr": "count by (node)(kube_volumeattachment_info)", + "instant": false, + "legendFormat": "{{node}}", + "range": true, + "refId": "A" + } + ], + "title": "Volume attachments per node", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 2, + "showPoints": "never", + "spanNulls": true + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 73 + }, + "id": 49, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "expr": "sum by (http_method)(hs_csi_operation_duration_seconds_count{operation=\"HammerspaceClient.doRequest\"})", + "instant": false, + "legendFormat": "{{http_method}}", + "range": true, + "refId": "A" + } + ], + "title": "Anvil REST call count over time", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "bars", + "fillOpacity": 60, + "lineWidth": 1, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "mode": "normal" + } + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 73 + }, + "id": 50, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "expr": "sum by (http_method)(increase(hs_csi_operation_duration_seconds_count{operation=\"HammerspaceClient.doRequest\"}[5m]))", + "instant": false, + "legendFormat": "{{http_method}}", + "range": true, + "refId": "A" + } + ], + "title": "Anvil REST calls per interval (5m)", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 115 + }, + "id": 51, + "title": "Share-backed provisioning path (new instrumentation)", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 2, + "showPoints": "never", + "spanNulls": true + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 116 + }, + "id": 52, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "expr": "histogram_quantile(0.95, sum by (le, operation)(rate(hs_csi_operation_duration_seconds_bucket{operation=~\"HammerspaceClient\\\\.WaitForTaskCompletion|HammerspaceClient\\\\.CreateShare|HammerspaceClient\\\\.CreateShareFromSnapshot|SetMetadataTags|ensureShareBackedVolumeExists|ensureBackingShareExists\"}[5m])))", + "instant": false, + "legendFormat": "{{operation}}", + "range": true, + "refId": "A" + } + ], + "title": "Share-path op latency p95 (by operation)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 2, + "showPoints": "never", + "spanNulls": true + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 116 + }, + "id": 53, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "expr": "histogram_quantile(0.50, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"HammerspaceClient.WaitForTaskCompletion\"}[5m])))", + "instant": false, + "legendFormat": "p50", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "expr": "histogram_quantile(0.95, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"HammerspaceClient.WaitForTaskCompletion\"}[5m])))", + "instant": false, + "legendFormat": "p95", + "range": true, + "refId": "B" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "expr": "histogram_quantile(0.99, sum by (le)(rate(hs_csi_operation_duration_seconds_bucket{operation=\"HammerspaceClient.WaitForTaskCompletion\"}[5m])))", + "instant": false, + "legendFormat": "p99", + "range": true, + "refId": "C" + } + ], + "title": "WaitForTaskCompletion latency (p50/p95/p99) - share-create bottleneck", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 2, + "showPoints": "never", + "spanNulls": true + }, + "unit": "reqps" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 124 + }, + "id": 54, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "expr": "sum by (operation)(rate(hs_csi_operation_duration_seconds_count{operation=~\"HammerspaceClient\\\\.WaitForTaskCompletion|HammerspaceClient\\\\.CreateShare|HammerspaceClient\\\\.CreateShareFromSnapshot|SetMetadataTags|ensureShareBackedVolumeExists|ensureBackingShareExists\"}[5m]))", + "instant": false, + "legendFormat": "{{operation}}", + "range": true, + "refId": "A" + } + ], + "title": "Share-path ops/sec (by operation)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 2, + "showPoints": "never", + "spanNulls": true + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 124 + }, + "id": 55, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "expr": "sum by (operation)(hs_csi_operation_inflight{operation=~\"HammerspaceClient\\\\.WaitForTaskCompletion|HammerspaceClient\\\\.CreateShare|HammerspaceClient\\\\.CreateShareFromSnapshot|SetMetadataTags|ensureShareBackedVolumeExists|ensureBackingShareExists\"})", + "instant": false, + "legendFormat": "{{operation}}", + "range": true, + "refId": "A" + } + ], + "title": "Share-path in-flight (by operation)", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 132 + }, + "id": 56, + "title": "Anvil REST calls - method / route / status (hs_csi_anvil_requests_total)", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "bars", + "fillOpacity": 30, + "lineWidth": 2, + "showPoints": "never", + "spanNulls": true, + "stacking": { + "mode": "normal" + } + }, + "unit": "reqps" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 133 + }, + "id": 57, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "expr": "sum by (http_status_code)(rate(hs_csi_anvil_requests_total[5m]))", + "instant": false, + "legendFormat": "{{http_status_code}}", + "range": true, + "refId": "A" + } + ], + "title": "Anvil calls/sec by status code", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 2, + "showPoints": "never", + "spanNulls": true + }, + "unit": "reqps" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 133 + }, + "id": 58, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "expr": "sum by (http_route)(rate(hs_csi_anvil_requests_total{http_status_code=\"404\"}[5m]))", + "instant": false, + "legendFormat": "{{http_route}}", + "range": true, + "refId": "A" + } + ], + "title": "Anvil 404 type-probe rate by route (watch this drop after the optimization)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 2, + "showPoints": "never", + "spanNulls": true + }, + "unit": "reqps" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 141 + }, + "id": 59, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "expr": "sum by (http_route)(rate(hs_csi_anvil_requests_total[5m]))", + "instant": false, + "legendFormat": "{{http_route}}", + "range": true, + "refId": "A" + } + ], + "title": "Anvil calls/sec by route", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 2, + "showPoints": "never", + "spanNulls": true + }, + "unit": "reqps" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 141 + }, + "id": 60, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "expr": "sum by (http_method)(rate(hs_csi_anvil_requests_total[5m]))", + "instant": false, + "legendFormat": "{{http_method}}", + "range": true, + "refId": "A" + } + ], + "title": "Anvil calls/sec by method", + "type": "timeseries" + }, + { + "collapsed": false, + "gridPos": { + "h": 1, + "w": 24, + "x": 0, + "y": 149 + }, + "id": 61, + "title": "CSI keyed locks (leak + contention)", + "type": "row" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "Raw keyed-lock gauge per role+instance. Expected to ride up near ~100 during provisioning (each in-flight CreateVolume/DeleteVolume holds a lock) and fall back to 0 when idle. Height alone is NOT a leak signal \u2014 see the verdict tile, which gates on release activity.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 30, + "lineWidth": 2, + "showPoints": "never", + "spanNulls": true + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 158 + }, + "id": 62, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "expr": "sum by (role, instance)(hs_csi_locks_held)", + "instant": false, + "legendFormat": "{{role}} {{instance}}", + "range": true, + "refId": "A" + } + ], + "title": "Locks held by role / instance (raw)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "Rate of lock acquisitions that hit the 30s timeout and returned codes.Aborted. Sustained failures mean a lock is held too long (contention or a leak).", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "bars", + "fillOpacity": 60, + "lineWidth": 2, + "showPoints": "never", + "spanNulls": true + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 158 + }, + "id": 63, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "expr": "sum by (role)(rate(hs_csi_lock_acquire_failures_total[5m]))", + "instant": false, + "legendFormat": "{{role}}", + "range": true, + "refId": "A" + } + ], + "title": "Lock acquire failures/sec (30s timeouts -> Aborted)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 2, + "showPoints": "never", + "spanNulls": true + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 0, + "y": 166 + }, + "id": 64, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "expr": "histogram_quantile(0.95, sum by (le,lock_type)(rate(hs_csi_lock_wait_seconds_bucket[5m])))", + "instant": false, + "legendFormat": "{{lock_type}}", + "range": true, + "refId": "A" + } + ], + "title": "Lock WAIT p95 (acquire contention)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 10, + "lineWidth": 2, + "showPoints": "never", + "spanNulls": true + }, + "unit": "s" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 166 + }, + "id": 65, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "expr": "histogram_quantile(0.95, sum by (le,lock_type)(rate(hs_csi_lock_hold_seconds_bucket[5m])))", + "instant": false, + "legendFormat": "{{lock_type}}", + "range": true, + "refId": "A" + } + ], + "title": "Lock HOLD p95 (holder duration)", + "type": "timeseries" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "Authoritative leak check. RED only when locks have been held > 0 for the last 10 min AND zero releases happened in that window (held-and-not-releasing = the actual leak mode). Stays GREEN through heavy provisioning, because locks keep releasing under load.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "mappings": [ + { + "options": { + "0": { + "color": "green", + "index": 0, + "text": "HEALTHY" + } + }, + "type": "value" + }, + { + "options": { + "from": 1, + "result": { + "color": "red", + "index": 1, + "text": "LEAK SUSPECTED" + }, + "to": 1000000 + }, + "type": "range" + } + ], + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "red", + "value": 1 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 6, + "x": 0, + "y": 150 + }, + "id": 66, + "options": { + "colorMode": "background", + "graphMode": "none", + "justifyMode": "center", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "textMode": "value" + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "count( (min_over_time((sum by (role,instance)(hs_csi_locks_held))[10m:30s]) > 0) and on(role,instance) (sum by (role,instance)(rate(hs_csi_lock_hold_seconds_count[10m])) == 0) ) or vector(0)", + "instant": false, + "legendFormat": "leaking series", + "range": true, + "refId": "A" + } + ], + "title": "Lock leak verdict", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "Lowest total held-lock count over the last 10 min. 0 = locks fully drain between ops (healthy). >0 = locks never returned to zero for 10 min: either sustained heavy load OR a leak. Cross-check the verdict tile to tell which.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "thresholds" + }, + "thresholds": { + "mode": "absolute", + "steps": [ + { + "color": "green", + "value": null + }, + { + "color": "yellow", + "value": 1 + } + ] + }, + "unit": "short" + }, + "overrides": [] + }, + "gridPos": { + "h": 8, + "w": 6, + "x": 6, + "y": 150 + }, + "id": 67, + "options": { + "colorMode": "value", + "graphMode": "area", + "justifyMode": "center", + "reduceOptions": { + "calcs": [ + "lastNotNull" + ], + "fields": "", + "values": false + }, + "textMode": "value" + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "editorMode": "code", + "expr": "sum( min_over_time((sum by (role,instance)(hs_csi_locks_held))[10m:30s]) ) or vector(0)", + "instant": false, + "legendFormat": "stuck", + "range": true, + "refId": "A" + } + ], + "title": "Stuck locks (10m idle floor)", + "type": "stat" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "description": "Locks held > 0 is NORMAL while PVCs provision \u2014 each in-flight CreateVolume/DeleteVolume holds a keyed lock. HEALTH: the held count (left) tracks the release rate (right) \u2014 both rise together. LEAK: held stays elevated while releases fall to 0.", + "fieldConfig": { + "defaults": { + "color": { + "mode": "palette-classic" + }, + "custom": { + "drawStyle": "line", + "fillOpacity": 20, + "lineWidth": 2, + "showPoints": "never", + "spanNulls": true + }, + "unit": "short" + }, + "overrides": [ + { + "matcher": { + "id": "byRegexp", + "options": "releases.*" + }, + "properties": [ + { + "id": "custom.axisPlacement", + "value": "right" + }, + { + "id": "unit", + "value": "ops" + }, + { + "id": "custom.axisLabel", + "value": "releases/s" + }, + { + "id": "custom.fillOpacity", + "value": 0 + } + ] + } + ] + }, + "gridPos": { + "h": 8, + "w": 12, + "x": 12, + "y": 150 + }, + "id": 68, + "options": { + "legend": { + "displayMode": "list", + "placement": "bottom", + "showLegend": true + }, + "tooltip": { + "mode": "multi", + "sort": "desc" + } + }, + "targets": [ + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "expr": "sum by (role)(hs_csi_locks_held)", + "legendFormat": "held: {{role}}", + "range": true, + "refId": "A" + }, + { + "datasource": { + "type": "prometheus", + "uid": "ee9e64t3xf474f" + }, + "expr": "sum by (role)(rate(hs_csi_lock_hold_seconds_count[1m]))", + "legendFormat": "releases/s: {{role}}", + "range": true, + "refId": "B" + } + ], + "title": "Locks held vs. release rate (held>0 while releasing = normal)", + "type": "timeseries" + } + ], + "refresh": "10s", + "schemaVersion": 39, + "tags": [ + "hammerspace", + "csi", + "storage" + ], + "time": { + "from": "now-15m", + "to": "now" + }, + "timezone": "browser", + "title": "Hammerspace CSI Driver", + "uid": "hs-csi-driver", + "version": 17 +} diff --git a/deploy/monitoring/victoriametrics/scrape.yml b/deploy/monitoring/victoriametrics/scrape.yml new file mode 100644 index 0000000..da44ff3 --- /dev/null +++ b/deploy/monitoring/victoriametrics/scrape.yml @@ -0,0 +1,41 @@ +# Example VictoriaMetrics / Prometheus scrape config for the Hammerspace CSI driver. +# +# The CSI controller (StatefulSet csi-provisioner) and node (DaemonSet csi-node) +# pods run with hostNetwork: true, so their Prometheus /metrics endpoints are +# reachable on the Kubernetes NODE host IPs: +# - controller: :9090 (OTEL_METRICS_PROMETHEUS_LISTEN, default :9090) +# - node: :9091 (set OTEL_METRICS_PROMETHEUS_LISTEN=:9091 on the DaemonSet) +# +# Enable metrics on the driver by setting these env vars on both containers +# (see deploy/monitoring/README.md): +# OTEL_METRICS_EXPORTER=prometheus +# OTEL_METRICS_PROMETHEUS_LISTEN=:9090 # controller +# OTEL_METRICS_PROMETHEUS_LISTEN=:9091 # node +# +# Replace the target IPs below with your Kubernetes node IPs. Only the node +# running csi-provisioner answers on :9090 (others show up=0, harmless); every +# node answers on :9091. This file drops in as VictoriaMetrics +# `-promscrape.config=` or under a Prometheus `scrape_configs:` block. + +scrape_configs: + - job_name: hs-csi-controller + scrape_interval: 15s + static_configs: + - targets: + - 10.0.0.11:9090 # replace with your k8s node IPs + - 10.0.0.12:9090 + - 10.0.0.13:9090 + labels: + service: hs-csi + role: controller + + - job_name: hs-csi-node + scrape_interval: 15s + static_configs: + - targets: + - 10.0.0.11:9091 # replace with your k8s node IPs + - 10.0.0.12:9091 + - 10.0.0.13:9091 + labels: + service: hs-csi + role: node diff --git a/docs/file-backed-performance.md b/docs/file-backed-performance.md new file mode 100644 index 0000000..8dad34a --- /dev/null +++ b/docs/file-backed-performance.md @@ -0,0 +1,136 @@ +# File-backed provisioning performance + +This document records the performance work on the **file-backed** CreateVolume +path (an ext4/xfs filesystem inside a file on a shared backing share) — the +bottlenecks found, the fixes, how each was measured, and the deployment knobs +that matter. It is the design reference for the `perf/file-backed-parallel-create` +changes and a runbook for anyone scaling file-backed provisioning. + +The headline: file-backed provisioning was ~2–3 volumes/s no matter the hardware. +Removing the bottlenecks one layer at a time took a single backing share from +~2.2/s to the point where the CSI driver and storage are no longer the limit. + +## The bottleneck ladder + +Throughput was gated by a *stack* of independent limits. Each had to be removed +to expose the next; none of them was the storage backend's raw capacity. + +### 1. Driver: the backing-share lock serialized every file create + +`ensureFileBackedVolumeExists` held the per-backing-share lock +(`acquireVolumeLock(backingShareName)`) across the **entire** create — both +`ensureBackingShareExists` (create-the-share-if-missing, which genuinely needs +serializing) **and** `ensureDeviceFileExists` (the per-file `qemu-img` + `mkfs`, +which does not). So every file on a backing share was created one-at-a-time. + +Symptom: with 100 provisioner workers, `hs_csi_locks_held` pinned at ~100 (99 +workers blocked on the one lock), `Controller/CreateVolume` p50 pegged at the 30s +lock-acquire timeout, a storm of `codes.Aborted` retries, and ~2.2/s throughput — +while the controller host was idle (load 0.5 on 32 vCPU) and the Anvil served +~13 of its ~5000 calls/s. + +**Fix (`e4c2005`):** narrow the lock to only `ensureBackingShareExists`, release +it, then do the per-file work unlocked. Because the per-file mount/unmount is +shared state, a **mount refcount** (`acquireBackingMount`/`releaseBackingMount`, +`mountRefs` map + `mountRefsMu`) keeps the backing share mounted while any create +is in flight and unmounts only when the last finishes — so `mkfs` parallelizes +across the worker threads without the unmount being yanked out mid-format. +Same-volume races are still covered by the *unique per-volume* lock in +`CreateVolume`. + +Result: `held` stopped pinning at 100 and drained to 0; the lock left the +critical path. + +### 2. Provisioner: client-side Kubernetes API throttle + +With the lock gone, throughput barely moved — the external-provisioner sidecar +was **client-side throttled** on its own API calls: +`Waited … due to client-side throttling` filled its log. Its default +`--kube-api-qps`/`--kube-api-burst` (≈5/10) capped how fast it could drive +CreateVolume. + +**Fix (deployment, not code):** raise the sidecar args — +`--kube-api-qps=200 --kube-api-burst=400` (and `--worker-threads=100`, the +default). Now up to 100 creates dispatch concurrently. + +> Note: raising `--worker-threads` alone does **not** help while a single backing +> share serializes (see §1) or while the data path is saturated (see §3) — it +> just deepens the queue and pushes lock waits past the 30s timeout. Raise it only +> together with §1 and enough backing-store headroom. + +### 3. Storage data path: mkfs write volume saturated the DSX + +At 100 concurrent creates the ceiling became `FormatDevice` — `mkfs.ext4` p95 +ballooned to **9.6s** even though the controller host was idle. `mkfs.ext4` +without flags **eagerly zeroes the inode table and journal** — ~37 MB of writes +per 1 GB volume — and for file-backed those writes go over NFS to the backing +share. Independent DSX-side `iostat` confirmed the NVMe pinned at ~442 MB/s / +100% util. Spreading across 8 backing shares did **not** help: it is one DSX. + +**Fix (`e9be029`):** `mkfs.ext4 -E lazy_itable_init=1,lazy_journal_init=1` for +ext4/ext3 — defer inode-table/journal zeroing to lazy kernel background init. +Create-time writes drop from ~37 MB to ~KB; `mkfs.ext4` p95 falls to **0.01s** +and per-volume physical footprint from ~64 MB to **~660 KB**. `qemu-img create -f +raw` was already sparse, so the file itself was never the problem — only the +`mkfs` zeroing was. + +Verified at scale: thousands of volumes created with DSX `iostat` staying low. + +### 4. Control plane: kube-controller-manager binding throttle + +With the driver and storage fast, PVs were created faster than Kubernetes could +**bind** them: thousands of PVs created, PVCs stuck Pending. This is +kube-controller-manager's PV controller running at its default +`--kube-api-qps=20`. + +**Fix (control plane, not code):** raise kcm `--kube-api-qps=100 +--kube-api-burst=100` (static-pod manifest). Binding went from ~1–2/s to ~15/s +and kept pace with creation. + +## Filesystem comparison (ext4 vs xfs vs btrfs) + +Measured on the same driver, backing store, and 100-way concurrency: + +| fs | mkfs flags | mkfs p95 | phys / 1 GB volume | +|---|---|---|---| +| **ext4 (lazy)** | `-E lazy_itable_init=1,lazy_journal_init=1` | **0.01s** | **~660 KB** | +| ext4 (eager, old) | none | 9.6s | ~64 MB | +| xfs | `-m reflink=0 -K` | slower | ~65 MB | +| btrfs | — | — | not supported here | + +- **ext4 + lazy-init is the winner** for file-backed — fastest mkfs and thinnest. +- **xfs** is heavier: `mkfs.xfs` lays down allocation-group structures (~65 MB) + even with `-K` (which skips the discard pass — commit `c206377`, the xfs analog + of ext4's lazy behaviour). Kept as a correct, supported option, not the default. +- **btrfs** is unavailable on RHEL/CentOS-family nodes: the CentOS Stream 10 + kernel ships no `btrfs` module (`modprobe btrfs` → *module not found*), so a + formatted volume could not be mounted. Supporting it (`mkfs.btrfs -K`) is a + one-line driver change gated on a btrfs-capable node OS (Ubuntu / Amazon Linux / + Fedora) or ELRepo `kmod-btrfs`. + +## Deployment tuning summary + +| Knob | Where | Default | Set to | Why | +|---|---|---|---|---| +| `--kube-api-qps` / `--burst` | `csi-provisioner` sidecar | ~5/10 | 200/400 | dispatch creates concurrently (§2) | +| `--worker-threads` | `csi-provisioner` sidecar | 100 | 100 | concurrency cap; only useful with §1+§3 | +| `--kube-api-qps` / `--burst` | kube-controller-manager | 20/30 | 100/100 | bind PVCs as fast as PVs are made (§4) | + +## Results + +Single backing share, file-backed 1 GB volumes: + +- **mkfs.ext4:** 9.6s → **0.01s** (~1000×) +- **space/volume:** ~64 MB → **~660 KB** (~100×) +- **DSX write load:** ~442 MB/s @ 100% util → **low/idle** +- **lock behaviour:** held pinned-at-100-blocked → parallel, drains to 0 +- CreateVolume throughput improved and, critically, the ceiling moved **off** the + CSI driver and storage entirely (onto Kubernetes control-plane rates, §2/§4). + +## Open item + +With mkfs at 0.01s, the DSX idle, and binding unthrottled, the driver's +CreateVolume rate still peaks ~15/s and degrades over a long run. That remaining +ceiling has not been root-caused; likely candidates are the mount-refcount churn +(§1) or `GetFile` slowing as a backing-share directory grows to thousands of +files. Profiling per-op latency vs. directory size is the next step. diff --git a/docs/lock-observability.md b/docs/lock-observability.md new file mode 100644 index 0000000..f76897c --- /dev/null +++ b/docs/lock-observability.md @@ -0,0 +1,250 @@ +# Keyed locks: model, metrics & leak troubleshooting + +This document describes the CSI driver's **in-process keyed locks** — why they +exist, how they are structured, the metrics that make them observable, and how to +tell a healthy busy driver from a lock leak. It is both the design reference for +the lock instrumentation (`common.LockProbe`) and the troubleshooting runbook for +the "provisioning is wedged" class of incidents. + +If you are staring at a Grafana tile that says "100 locks held" and wondering +whether that is a problem, jump to [§6](#6-reading-the-tiles-normal-vs-leak). + +--- + +## 1. Why the driver holds locks + +CSI RPCs for the *same* object can arrive concurrently and overlap: the external +sidecars retry, and Kubernetes can re-drive a `CreateVolume`/`DeleteVolume` while +the previous attempt is still running. Two creates for the same volume, or a +create racing a delete, would corrupt state on the Anvil. The driver therefore +serialises work **per object key** with an in-memory lock, so that at most one +RPC mutates a given key at a time. + +Locks are **keyed by name, not by RPC** — the key is a volume name, a backing +share name, or a snapshot name. This matters: it is why a single overloaded +*backing share* can serialise thousands of unrelated file-backed volumes +([§4](#4-why-held-pins-at-100)). + +## 2. The lock primitive + +All of this lives in `pkg/driver/driver.go`: + +```go +type keyLock struct { + sem *semaphore.Weighted // weight=1 → acts like a mutex +} +func (kl *keyLock) lock(ctx context.Context) error { return kl.sem.Acquire(ctx, 1) } +func (kl *keyLock) unlock() { kl.sem.Release(1) } +``` + +- **`volumeLocks` / `snapshotLocks`** — two `map[string]*keyLock`, guarded by + `locksMu`. A key's `keyLock` is created lazily on first use and kept. +- **Weight-1 semaphore, not `sync.Mutex`.** The reason is the **context**: a + `sync.Mutex` cannot be acquired with a timeout, but `semaphore.Weighted.Acquire` + takes a `ctx` and returns when it is cancelled. That gives us the bounded wait + below. +- **Non-reentrant.** A goroutine that holds key `K` must not try to acquire `K` + again — it would self-deadlock until the timeout. Nested acquisitions must + always be of *different* keys (see the create path in + [§3](#3-which-rpcs-take-which-locks)). + +Acquisition is wrapped with a **30-second deadline**: + +```go +probe := common.StartLockProbe(ctx, "volume") +lctx, cancel := context.WithTimeout(ctx, 30*time.Second) +defer cancel() +if err := lk.lock(lctx); err != nil { + probe.Failed() + return nil, status.Errorf(codes.Aborted, "could not acquire volume lock for %s: %v", volID, err) +} +release := probe.Acquired() +return func() { lk.unlock(); release() }, nil +``` + +If the lock cannot be acquired within 30s the RPC fails with **`codes.Aborted`**, +which is retryable — the sidecar backs off and re-drives it. This is a pressure +valve, not an error: under heavy contention you will see a background rate of +Aborted creates that succeed on retry. A *sustained, growing* Aborted rate is the +signal that a holder is stuck ([§6](#6-reading-the-tiles-normal-vs-leak)). + +## 3. Which RPCs take which locks + +| RPC | Lock key(s) | Notes | +|---|---|---| +| `CreateVolume` (file-backed) | `volumeName`, then **`backingShareName`** | Two locks, nested. The volume key is unique per PVC; the backing-share key is **shared by every file-backed volume on that StorageClass**. | +| `CreateVolume` (share-backed) | `volumeName` (+ `backingShareName` when ensuring the parent share) | Each volume is its own share, so there is no shared serialisation point. | +| `DeleteVolume` | `volumeId` | | +| `CreateSnapshot` | `snapshotLocks[req.Name]` | Separate map; `lock_type="snapshot"`. | +| create-from-snapshot | `residingShareName` | | + +The key line for troubleshooting: **file-backed `CreateVolume` acquires its unique +volume lock, then contends for the single shared backing-share lock.** All +file-backed provisioning on one StorageClass funnels through that one key. + +## 4. Why "held" pins at ~100 + +The height of `hs_csi_locks_held` during a provisioning storm is **not** set by +how many PVCs you submit. It is set by how many `CreateVolume` RPCs run +*concurrently*, which is the **external-provisioner sidecar's `--worker-threads` +(default 100)**. + +> Naming caveat: "csi-provisioner" is overloaded. The controller **pod** +> `csi-provisioner-0` bundles five containers — our driver +> (`hs-csi-plugin-controller`) plus the upstream sidecars `csi-provisioner`, +> `csi-attacher`, `csi-snapshotter`, `csi-resizer`. The `--worker-threads` flag +> is on the upstream **`csi-provisioner` container** (the external-provisioner), +> *not* on our driver. That sidecar is the throttle; our driver is what holds the +> locks and has no concurrency cap of its own. (Node-side RPCs come from the +> separate `csi-node` DaemonSet, container `hs-csi-plugin-node`.) + +Walk through a burst of 4,000 file-backed PVCs against one StorageClass: + +1. The provisioner dispatches **100** `CreateVolume` calls at a time (the other + 3,900 sit in its workqueue). +2. Each of those 100 immediately acquires its **unique volume lock** → **~100 + held**. +3. All 100 then queue on the **single shared backing-share lock**. Exactly **one** + holds it (→ +1) and does its Anvil work; the other 99 block in `lock()` + (they have *not* incremented `held` yet — the probe counts a lock as held only + once `Acquire` returns). + +So steady-state `held ≈ 100 volume locks + 1 backing-share lock ≈ 101`, and it +stays there for the whole burst because as one create finishes the provisioner +immediately dispatches the next queued PVC into the freed worker slot. + +**"Did we not push hard enough?"** Submitting *more* volumes (10k, 100k) does not +raise this ceiling — it only makes the burst last longer. The concurrency is +clamped at 100 upstream of the driver. To actually push the lock manager harder +you change the *concurrency*, not the count: + +- **Raise `--worker-threads`** on the `csi-provisioner` container (e.g. 300–500). + That drives proportionally more concurrent volume locks and, for file-backed, + a *deeper queue* on the one backing-share lock → longer `lock_wait`, and once + waits cross 30s, a rising `acquire_failures_total`. This is the single most + effective way to stress the lock/timeout path. +- **Spread across StorageClasses** to multiply *backing-share* keys — but note + that for file-backed the real throughput ceiling is the Anvil's metadata-op + rate (~3/s per backing share on a single Anvil), not the lock, so this raises + lock *breadth* more than depth. + +There is no cap inside the driver itself; the map grows to as many distinct keys +as are in flight. + +## 5. Metrics — the `LockProbe` contract + +`common.StartLockProbe(ctx, lockType)` returns a probe with a two-outcome +lifecycle, wired in `pkg/common/metrics.go`. Every acquisition ends in exactly +one of `Failed()` or `Acquired()`; the closure returned by `Acquired()` is +called on release. + +| Metric | Type | Emitted when | Meaning | +|---|---|---|---| +| `hs_csi_locks_held` | up/down counter (gauge) | `+1` on acquire, `-1` on release | Locks currently held. **A leak shows as a stuck non-zero value.** | +| `hs_csi_lock_wait_seconds` | histogram | on acquire **and** on failure | Time blocked waiting for the lock. `_count` = every attempt that resolved. | +| `hs_csi_lock_hold_seconds` | histogram | on release | How long the lock was held. **`_count` = number of releases.** | +| `hs_csi_lock_acquire_failures_total` | counter | on the 30s timeout | Acquisitions that gave up → `codes.Aborted`. | + +All carry `lock_type` (`volume` \| `snapshot`); the scrape adds `role` +(`controller` \| `node`) and `instance`. Controller serves `:9090`, node +DaemonSet `:9091`. + +### The counter identity (why the leak detector works) + +These four are not independent — they satisfy an exact conservation law, verified +live on a running driver: + +``` +wait_count = hold_count + acquire_failures_total (every attempt either releases or times out) +locks_held = successful_acquires − releases (what's still held) +``` + +The consequence that powers [§6](#6-reading-the-tiles-normal-vs-leak): +**`rate(hs_csi_lock_hold_seconds_count[…])` is the rate of lock *releases*** — i.e. +"are locks still turning over?" A healthy driver under load releases constantly; +a leaked lock is acquired and *never* released. That release rate, not the height +of `locks_held`, is what distinguishes load from a leak. + +## 6. Reading the tiles: normal vs leak + +`hs_csi_locks_held > 0` is **expected** during provisioning — every in-flight +`CreateVolume`/`DeleteVolume` holds a key. Height alone means nothing. The Grafana +"CSI keyed locks" row is built around the release-rate discriminator so you never +have to guess: + +**`Lock leak verdict`** (green/red stat) — the authoritative signal: + +```promql +count( + (min_over_time((sum by (role,instance)(hs_csi_locks_held))[10m:30s]) > 0) + and on(role,instance) + (sum by (role,instance)(rate(hs_csi_lock_hold_seconds_count[10m])) == 0) +) or vector(0) +``` + +Red **only** when, per role+instance, locks never returned to zero for 10 minutes +**and** zero releases happened in that window (held-and-not-releasing). It stays +green through any amount of honest load, because under load the release rate is +never zero. + +**`Stuck locks (10m idle floor)`** — `sum(min_over_time((sum by (role,instance)(hs_csi_locks_held))[10m:30s]))`. +The lowest held count over 10 min. `0` = locks fully drain between ops. `>0` = +they never fully drained — either sustained load or a leak; the verdict tile +disambiguates. Use it as an early-warning magnitude. + +**`Locks held vs. release rate`** — dual-axis timeseries overlaying +`sum by (role)(hs_csi_locks_held)` against +`sum by (role)(rate(hs_csi_lock_hold_seconds_count[1m]))`. The teaching view: +healthy = both lines rise together; leak = held stays up while releases fall to 0. + +**Signature of a real leak**, all at once: + +- `hs_csi_locks_held` stuck at a non-zero value with **no** downward movement, +- `rate(hs_csi_lock_hold_seconds_count)` at/near **0** (no releases), +- `hs_csi_lock_acquire_failures_total` **climbing** (everyone else times out on + the wedged key → a storm of `codes.Aborted`), +- new `CreateVolume`s for the affected StorageClass all failing Aborted. + +## 7. Root cause seen in the field & remediation + +The leak this instrumentation was built to catch: after **re-pointing the driver +to a new Anvil** (or terminating one), a stale NFS mount to the *old, dead* Anvil +is left behind at ``. It is host-propagated and +survives pod restarts. `EnsureBackingShareMounted`'s `IsShareMounted` then hangs +uninterruptibly `stat`-ing the dead mount **while holding the volume and +backing-share locks** → both leak → every file-backed `CreateVolume` on that +backing share wedges behind the one held key. The node side is symmetric +(`NodeUnstage`/`NodeUnpublish` umount hangs → node lock leak). + +**Manual remediation** (until the fix below is deployed): + +```sh +# On each controller/node pod: force-unmount every stale mount to the dead Anvil IP +for mp in $(grep nfs /proc/mounts | grep -v | awk '{print $2}'); do + umount -f -l "$mp" +done +# then restart the CSI pods so the leaked in-memory locks are released +kubectl -n kube-system delete pod csi-provisioner-0 +kubectl -n kube-system delete pod -l app=csi-node +``` + +**Code fix** (`fix/scale-reliability`): `IsShareMounted` is now timeout-bounded +(`SafeIsMountPoint`, returns false on timeout instead of hanging), `MountShare` +bounds the mount syscall, and `EnsureBackingShareMounted` `umount -f -l`'s a stale +mount before re-mounting. This removes the hang that caused the leak, so a stale +mount after an Anvil swap self-heals instead of wedging the lock. + +## 8. Rationale (why it looks like this) + +- **Gauge, not just a counter.** A held-lock leak is a *level*, not a rate. An + up/down counter that returns to 0 between operations is the most direct + possible encoding of "is anything stuck right now." +- **Release rate as the discriminator.** We deliberately did *not* alert on + `locks_held > N` — that fires on healthy load and trains operators to ignore it. + Gating on "held **and not releasing**" is what makes the verdict trustworthy + enough to page on. +- **`lock_type` label only.** No per-volume key on any lock metric — cardinality + stays bounded exactly as in the main observability doc. +- **30s Aborted over blocking forever.** A bounded wait turns a stuck key into a + visible, retryable, *localised* failure instead of an invisible pile-up of + goroutines. diff --git a/docs/observability.md b/docs/observability.md new file mode 100644 index 0000000..2719d84 --- /dev/null +++ b/docs/observability.md @@ -0,0 +1,222 @@ +# Observability: metrics & tracing + +This document describes the observability subsystem of the Hammerspace CSI +driver — the OpenTelemetry (OTel) metrics and traces it emits, how they are +wired, what is instrumented, and how to extend it. It is the design reference +for a body of work that reconstructed and extended the driver's instrumentation; +if you are changing metrics, adding a Grafana panel, or wondering why a code path +is wrapped the way it is, start here. + +> Scope note: this covers the **observability** functionality specifically. +> Behavioral bug fixes and performance improvements (e.g. the share-type-probe +> optimization, the DeleteSnapshot routing fix) are documented inline at the call +> site and in their commit messages, not here. + +## 1. Design goals + +- **Zero cost when off.** With no exporter configured the OTel global providers + are no-ops, so every instrument call compiles to a cheap no-op. The driver + must be shippable with instrumentation always compiled in. +- **Env-driven, no code changes to toggle.** Operators turn telemetry on/off and + point it at a backend purely through standard OTel environment variables. +- **Answer "where did the time go?"** for both provisioning shapes. Metrics must + localize latency to a specific internal step, symmetrically for **file-backed** + and **share-backed** volumes. +- **Bounded cardinality.** No metric label may carry a per-volume identifier. + +## 2. Wiring + +`initTelemetry()` (`main.go`) installs the OTel providers from standard env vars +and is a no-op unless an exporter is selected: + +| Env var | Values | Default | Effect | +|---|---|---|---| +| `OTEL_TRACES_EXPORTER` | `none` \| `console` \| `otlp` | `none` | trace exporter | +| `OTEL_METRICS_EXPORTER` | `none` \| `prometheus` \| `otlp` | `none` | metric exporter | +| `OTEL_EXPORTER_OTLP_ENDPOINT` | `host:4317` | — | OTLP target (traces/metrics) | +| `OTEL_METRICS_PROMETHEUS_LISTEN` | `:9090` | `:9090` | Prometheus scrape listen addr | + +When `OTEL_METRICS_EXPORTER=prometheus`, the driver serves `/metrics` (and +`/healthz`) on `OTEL_METRICS_PROMETHEUS_LISTEN`. In our deployment the controller +exposes `:9090` and the node DaemonSet `:9091`, and VictoriaMetrics scrapes both. + +## 3. Metric catalog + +All instruments live in `pkg/common/metrics.go`. OTel → Prometheus name mapping +turns attribute keys `foo.bar` into label `foo_bar`; a counter whose instrument +name already ends in `_total` is **not** double-suffixed. + +### `hs_csi_operation_*` — the per-operation family + +Emitted by `MeasureOp` (see §4). Every instrumented operation/step reports: + +| Series | Type | Meaning | +|---|---|---| +| `hs_csi_operation_duration_seconds` (`_bucket`/`_count`/`_sum`) | histogram | step latency | +| `hs_csi_operation_errors_total` | counter | steps that returned a non-nil error | +| `hs_csi_operation_inflight` | up/down gauge | concurrent executions of the step | + +Labels: `operation` (always) plus any per-site attributes (`http_method`, +`http_path`, `fsType`). + +Histogram buckets are **explicit and sub-second-biased** — +`0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10, 20, 30` (seconds) — +because the OTel defaults (`0, 5, 10, 25, …`) dump every sub-5s CSI call into one +bucket, making `histogram_quantile` useless. + +### `hs_csi_anvil_requests_total` — the Anvil REST counter + +A dedicated counter incremented once per `HammerspaceClient.doRequest`, **after** +the response is known, so it can carry the real status code. + +Labels: `http_method`, `http_route`, `http_status_code`. + +Why separate from the `doRequest` histogram? `MeasureOp` arms its recording in a +deferred closure set up *before* the request runs, so it can only label +method+path — it cannot know the status code. This counter fills that gap: it is +the only place we can see, e.g., the volume of `404` responses (the file-vs-share +type-probe traffic) split by route and method. + +## 4. The `MeasureOp` contract + +```go +func MeasureOp(ctx, operation string, attrs ...attribute.KeyValue) func(*error) +``` + +Call it at the start of a step; it bumps `inflight`, and returns a closure that — +when invoked (typically via `defer`) — records duration, decrements `inflight`, +and (if handed a non-nil `*error`) increments `errors_total`: + +```go +func (d *CSIDriver) CreateVolume(...) (_ *csi.CreateVolumeResponse, err error) { + defer common.MeasureOp(ctx, "Controller/CreateVolume")(&err) // error-aware + ... +} + +defer common.MeasureOp(ctx, "FormatDevice", attribute.String("fsType", fsType))(nil) // no error tracking +``` + +Pass `(&err)` (with a named return) to feed `errors_total`; pass `(nil)` when the +step's error is already surfaced elsewhere (e.g. client calls whose failures show +up as non-2xx in `hs_csi_anvil_requests_total`). + +## 5. Anvil route normalization (`AnvilRoute`) + +`doRequest` labels carry a **route template**, not the raw path, so the metric +does not explode into one series per share/file/task. `AnvilRoute` collapses +per-resource id segments to `{id}`: + +``` +/mgmt/v1.2/rest/shares/file--pvc- -> /mgmt/v1.2/rest/shares/{id} +/mgmt/v1.2/rest/tasks/ -> /mgmt/v1.2/rest/tasks/{id} +/mgmt/v1.2/rest/file-snapshots/list -> (kept; "list" is an action, not an id) +``` + +Collections normalized: `shares`, `tasks`, `files`, `objectives`, `file-snapshots` +(except the `list` action). The same template is used for the histogram's +`http_path` label and the counter's `http_route` label. + +## 6. Instrumentation coverage + +The top-level CSI RPCs and the file-backed steps were instrumented first; the +share-backed steps were added to reach **symmetry** — so latency can be localized +regardless of volume shape. + +| Layer | Operations | +|---|---| +| Controller RPCs | `Controller/CreateVolume`, `Controller/DeleteVolume`, `Controller/CreateSnapshot` | +| Node RPCs | `Node/NodeStageVolume`, `Node/NodeUnstageVolume`, `Node/NodePublishVolume`, `Node/NodeUnpublishVolume` | +| File-backed steps | `MakeEmptyRawFile`, `FormatDevice`, `MountShare`, `UnmountFilesystem`, `UnmountBackingShareIfUnused` | +| **Share-backed steps** | `HammerspaceClient.WaitForTaskCompletion`, `HammerspaceClient.CreateShare`, `HammerspaceClient.CreateShareFromSnapshot`, `SetMetadataTags`, `ensureShareBackedVolumeExists`, `ensureBackingShareExists` | +| REST | `HammerspaceClient.doRequest` (histogram) + `hs_csi_anvil_requests_total` (counter) | + +**`WaitForTaskCompletion` is the key share-path signal.** `CreateShare` returns +`202 + a task`; the driver then polls `GET /tasks/{id}` until terminal. That poll +is the dominant cost of share creation — the share-backed analog of the +file-visibility poll on the file path — so it gets both a metric and a span that +records the **attempt count**. + +## 7. Tracing + +The same call sites that carry a metric generally also open a span +(`tracer.Start`). Two spans record loop **attempt counts** as span attributes, +because attempts (not just wall time) reveal backend convergence latency: + +- `applyObjectiveAndMetadata.waitForFileVisible` (file path) +- `HammerspaceClient.WaitForTaskCompletion` (share path) + +Traces export via `OTEL_TRACES_EXPORTER` (`otlp` to a collector, or `console` for +local debugging). + +## 8. Grafana dashboard + +The dashboard JSON and an example scrape config are committed in-repo under +[`deploy/monitoring/`](../deploy/monitoring/) — import +`deploy/monitoring/grafana/hs-csi-driver-dashboard.json` and adapt +`deploy/monitoring/victoriametrics/scrape.yml`. See +[`deploy/monitoring/README.md`](../deploy/monitoring/README.md) for the full +enable-metrics → scrape → import walkthrough. + +Dashboard **"Hammerspace CSI Driver"** (uid `hs-csi-driver`) is backed by the +VictoriaMetrics datasource. Relevant rows: + +- **CSI Controller / Node RPC latency** — `histogram_quantile` p50/p95/p99 over + `hs_csi_operation_duration_seconds_bucket` by `operation`. +- **Anvil REST client** — call rate/latency/count for `operation="HammerspaceClient.doRequest"`. +- **Share-backed provisioning path** — per-step latency/rate/inflight for the + share operations, plus a dedicated `WaitForTaskCompletion` p50/p95/p99 panel. +- **Anvil REST calls — method / route / status** — `hs_csi_anvil_requests_total` + by `http_status_code`, `http_route`, `http_method`, including a "404 type-probe + rate by route" panel. + +Representative queries: + +```promql +# per-operation p95 latency +histogram_quantile(0.95, sum by (le, operation)( + rate(hs_csi_operation_duration_seconds_bucket[5m]))) + +# Anvil calls/sec by status code +sum by (http_status_code)(rate(hs_csi_anvil_requests_total[5m])) + +# 404 type-probe rate by route +sum by (http_route)(rate(hs_csi_anvil_requests_total{http_status_code="404"}[5m])) +``` + +## 9. Cardinality discipline + +- Never put a per-volume id in a label. Anvil paths are templated via + `AnvilRoute`; volume ids appear only in **trace** attributes, never metrics. +- `http_status_code` is bounded (a handful of codes); `operation` and `route` are + bounded sets. +- Prefer adding an `operation` value over adding a new label. + +## 10. Extending it + +**Add a measured step:** +```go +func (d *CSIDriver) doThing(ctx context.Context, ...) (err error) { + ctx, span := tracer.Start(ctx, "doThing") + defer span.End() + defer common.MeasureOp(ctx, "doThing")(&err) // or (nil) + ... +} +``` +The new `operation` value flows into the existing per-operation panels +automatically; add a focused panel only if the step deserves its own callout. + +**Add a dashboard panel:** query `hs_csi_operation_*` filtered by `operation`, or +`hs_csi_anvil_requests_total` by its labels. Keep `rate()` windows at `[5m]` for +the bursty provisioning workload. + +## 11. Rationale (why it looks like this) + +- **Metrics mirror spans.** `MeasureOp` was designed to sit alongside the existing + tracing spans so a step is measured and traced from the same call site. +- **Explicit buckets.** CSI operations are sub-second to low-seconds; default OTel + buckets give no quantile resolution there. +- **A separate Anvil counter.** Status codes are only known post-response; + `MeasureOp`'s pre-armed closure cannot capture them. The counter also makes + non-GET verbs and error responses first-class. +- **Route templating over raw paths.** One series per share (`file--pvc-`) + is unusable and unbounded; the route template keeps the series count flat. diff --git a/go.mod b/go.mod index 50bf552..05800c7 100644 --- a/go.mod +++ b/go.mod @@ -6,40 +6,101 @@ require ( github.com/ameade/spec v0.3.0 // - Apache 2.0 license github.com/container-storage-interface/spec v1.9.0 // - Apache 2.0 license github.com/google/uuid v1.6.0 - github.com/jpillora/backoff v0.0.0-20180909062703-3050d21c67d7 // - MIT license + github.com/jpillora/backoff v1.0.0 // - MIT license github.com/kubernetes-csi/csi-test v2.2.0+incompatible github.com/onsi/ginkgo v1.10.3 // - MIT license github.com/onsi/gomega v1.35.1 // - MIT license github.com/sirupsen/logrus v1.9.3 // - MIT license - go.opentelemetry.io/otel v1.43.0 - go.opentelemetry.io/otel/sdk v1.43.0 - go.opentelemetry.io/otel/trace v1.43.0 - golang.org/x/net v0.48.0 - golang.org/x/sync v0.19.0 - golang.org/x/sys v0.42.0 - google.golang.org/grpc v1.79.3 - google.golang.org/protobuf v1.36.10 + go.opentelemetry.io/otel v1.44.0 + go.opentelemetry.io/otel/sdk v1.44.0 + go.opentelemetry.io/otel/trace v1.44.0 + golang.org/x/net v0.55.0 + golang.org/x/sync v0.20.0 + golang.org/x/sys v0.45.0 + google.golang.org/grpc v1.81.1 + google.golang.org/protobuf v1.36.11 gopkg.in/yaml.v2 v2.4.0 k8s.io/kubernetes v1.33.6 k8s.io/mount-utils v0.27.5 ) require ( + github.com/prometheus/client_golang v1.23.2 + go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.44.0 + go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.33.0 + go.opentelemetry.io/otel/exporters/prometheus v0.66.0 + go.opentelemetry.io/otel/exporters/stdout/stdouttrace v1.44.0 + go.opentelemetry.io/otel/metric v1.44.0 + go.opentelemetry.io/otel/sdk/metric v1.44.0 + k8s.io/api v0.33.6 + k8s.io/apimachinery v0.33.6 + k8s.io/client-go v0.0.0 +) + +// k8s.io/kubernetes v1.33.6 forces its staging modules to matching v0.33.6. +// Without this, go resolves the transitive deps to v0.0.0 and refuses to build. +replace ( + k8s.io/api => k8s.io/api v0.33.6 + k8s.io/apimachinery => k8s.io/apimachinery v0.33.6 + k8s.io/client-go => k8s.io/client-go v0.33.6 +) + +require ( + github.com/beorn7/perks v1.0.1 // indirect + github.com/cenkalti/backoff/v4 v4.3.0 // indirect + github.com/cenkalti/backoff/v5 v5.0.3 // indirect github.com/cespare/xxhash/v2 v2.3.0 // indirect + github.com/davecgh/go-spew v1.1.1 // indirect + github.com/emicklei/go-restful/v3 v3.11.0 // indirect github.com/fsnotify/fsnotify v1.9.0 // indirect + github.com/fxamacker/cbor/v2 v2.7.0 // indirect github.com/go-logr/logr v1.4.3 // indirect github.com/go-logr/stdr v1.2.2 // indirect + github.com/go-openapi/jsonpointer v0.21.0 // indirect + github.com/go-openapi/jsonreference v0.20.2 // indirect + github.com/go-openapi/swag v0.23.0 // indirect + github.com/gogo/protobuf v1.3.2 // indirect github.com/golang/protobuf v1.5.4 // indirect + github.com/google/gnostic-models v0.6.9 // indirect github.com/google/go-cmp v0.7.0 // indirect + github.com/gorilla/websocket v1.5.4-0.20250319132907-e064f32e3674 // indirect + github.com/grpc-ecosystem/grpc-gateway/v2 v2.29.0 // indirect github.com/hpcloud/tail v1.0.0 // indirect + github.com/josharian/intern v1.0.0 // indirect + github.com/json-iterator/go v1.1.12 // indirect + github.com/mailru/easyjson v0.7.7 // indirect + github.com/moby/spdystream v0.5.0 // indirect github.com/moby/sys/mountinfo v0.7.2 // indirect + github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect + github.com/modern-go/reflect2 v1.0.2 // indirect + github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect + github.com/mxk/go-flowrate v0.0.0-20140419014527-cca7078d478f // indirect + github.com/pkg/errors v0.9.1 // indirect + github.com/prometheus/client_model v0.6.2 // indirect + github.com/prometheus/common v0.67.5 // indirect + github.com/prometheus/otlptranslator v1.0.0 // indirect + github.com/prometheus/procfs v0.20.1 // indirect + github.com/x448/float16 v0.8.4 // indirect go.opentelemetry.io/auto/sdk v1.2.1 // indirect - go.opentelemetry.io/otel/metric v1.43.0 // indirect - golang.org/x/text v0.32.0 // indirect - google.golang.org/genproto/googleapis/rpc v0.0.0-20251202230838-ff82c1b0f217 // indirect + go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.33.0 // indirect + go.opentelemetry.io/proto/otlp v1.10.0 // indirect + go.yaml.in/yaml/v2 v2.4.4 // indirect + golang.org/x/oauth2 v0.36.0 // indirect + golang.org/x/term v0.43.0 // indirect + golang.org/x/text v0.37.0 // indirect + golang.org/x/time v0.9.0 // indirect + google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa // indirect + google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa // indirect + gopkg.in/evanphx/json-patch.v4 v4.12.0 // indirect gopkg.in/fsnotify.v1 v1.4.7 // indirect + gopkg.in/inf.v0 v0.9.1 // indirect gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 // indirect gopkg.in/yaml.v3 v3.0.1 // indirect k8s.io/klog/v2 v2.130.1 // indirect + k8s.io/kube-openapi v0.0.0-20250318190949-c8a335a9a2ff // indirect k8s.io/utils v0.0.0-20241104100929-3ea5e8cea738 // indirect + sigs.k8s.io/json v0.0.0-20241010143419-9aa6b5e7a4b3 // indirect + sigs.k8s.io/randfill v1.0.0 // indirect + sigs.k8s.io/structured-merge-diff/v4 v4.6.0 // indirect + sigs.k8s.io/yaml v1.4.0 // indirect ) diff --git a/main.go b/main.go index 38a18e9..975427f 100644 --- a/main.go +++ b/main.go @@ -16,20 +16,31 @@ limitations under the License. package main import ( + "context" "net" + "net/http" "net/url" "os" "os/signal" "strconv" "strings" "syscall" + "time" "github.com/hammer-space/csi-plugin/pkg/common" "github.com/hammer-space/csi-plugin/pkg/driver" + "github.com/prometheus/client_golang/prometheus/promhttp" log "github.com/sirupsen/logrus" "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc" + "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc" + "go.opentelemetry.io/otel/exporters/prometheus" + "go.opentelemetry.io/otel/exporters/stdout/stdouttrace" "go.opentelemetry.io/otel/propagation" + metric "go.opentelemetry.io/otel/sdk/metric" + "go.opentelemetry.io/otel/sdk/resource" sdktrace "go.opentelemetry.io/otel/sdk/trace" + semconv "go.opentelemetry.io/otel/semconv/v1.41.0" ) func init() { @@ -42,24 +53,90 @@ func init() { log.SetOutput(os.Stdout) log.SetLevel(log.DebugLevel) log.SetReportCaller(false) - // Initialize OpenTelemetry Tracer - if _, err := initTracer(); err != nil { - log.Fatalf("failed to init tracer: %v", err) + // Initialize OpenTelemetry (traces + metrics), configured via env vars. + if err := initTelemetry(); err != nil { + log.Fatalf("failed to init telemetry: %v", err) } } -// Setup tracing -func initTracer() (*sdktrace.TracerProvider, error) { - // Create a new tracer provider with a short ID generator - // This will generate shorter span IDs for better readability in logs - // Note: This is a custom ID generator that generates shorter IDs for spans - // It is not a standard OpenTelemetry ID generator, but it is used here for demonstration - log.Info("Creating TracerProvider with full ID generator") - tp := sdktrace.NewTracerProvider() - otel.SetTracerProvider(tp) - log.Info("OpenTelemetry TracerProvider set") +// initTelemetry wires OTel providers according to standard env vars: +// +// OTEL_TRACES_EXPORTER = none | console | otlp (default: none) +// OTEL_METRICS_EXPORTER = none | prometheus | otlp (default: none) +// OTEL_EXPORTER_OTLP_ENDPOINT = host:4317 (used when *_EXPORTER=otlp) +// OTEL_METRICS_PROMETHEUS_LISTEN = :9090 (Prometheus scrape port) +// +// With defaults, both providers are no-ops so instrumentation stays cheap. +func initTelemetry() error { + res, err := resource.Merge(resource.Default(), resource.NewWithAttributes( + semconv.SchemaURL, + semconv.ServiceName("hammerspace-csi"), + semconv.ServiceVersion(common.Version), + )) + if err != nil { + return err + } + + // ---- Traces ---- + switch os.Getenv("OTEL_TRACES_EXPORTER") { + case "console": + exp, err := stdouttrace.New(stdouttrace.WithPrettyPrint()) + if err != nil { + return err + } + otel.SetTracerProvider(sdktrace.NewTracerProvider(sdktrace.WithBatcher(exp), sdktrace.WithResource(res))) + log.Info("OTel traces: stdouttrace exporter enabled") + case "otlp": + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + exp, err := otlptracegrpc.New(ctx) + if err != nil { + return err + } + otel.SetTracerProvider(sdktrace.NewTracerProvider(sdktrace.WithBatcher(exp), sdktrace.WithResource(res))) + log.Infof("OTel traces: otlp exporter -> %s", os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")) + default: + log.Info("OTel traces: disabled (OTEL_TRACES_EXPORTER=none)") + } otel.SetTextMapPropagator(propagation.TraceContext{}) - return tp, nil + + // ---- Metrics ---- + switch os.Getenv("OTEL_METRICS_EXPORTER") { + case "prometheus": + promExp, err := prometheus.New() + if err != nil { + return err + } + otel.SetMeterProvider(metric.NewMeterProvider(metric.WithReader(promExp), metric.WithResource(res))) + listen := os.Getenv("OTEL_METRICS_PROMETHEUS_LISTEN") + if listen == "" { + listen = ":9090" + } + go func() { + mux := http.NewServeMux() + mux.Handle("/metrics", promhttp.Handler()) + mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) { _, _ = w.Write([]byte("ok")) }) + log.Infof("OTel metrics: prometheus /metrics listening on %s", listen) + if err := http.ListenAndServe(listen, mux); err != nil { + log.Errorf("prometheus /metrics listener failed: %v", err) + } + }() + case "otlp": + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + exp, err := otlpmetricgrpc.New(ctx) + if err != nil { + return err + } + otel.SetMeterProvider(metric.NewMeterProvider( + metric.WithReader(metric.NewPeriodicReader(exp, metric.WithInterval(30*time.Second))), + metric.WithResource(res), + )) + log.Infof("OTel metrics: otlp exporter -> %s", os.Getenv("OTEL_EXPORTER_OTLP_ENDPOINT")) + default: + log.Info("OTel metrics: disabled (OTEL_METRICS_EXPORTER=none)") + } + return nil } func validateEnvironmentVars() { diff --git a/pkg/client/hsclient.go b/pkg/client/hsclient.go index 87e56d0..7fefb77 100755 --- a/pkg/client/hsclient.go +++ b/pkg/client/hsclient.go @@ -42,7 +42,6 @@ import ( "google.golang.org/grpc/status" "github.com/hammer-space/csi-plugin/pkg/common" - "github.com/jpillora/backoff" "go.opentelemetry.io/otel" "go.opentelemetry.io/otel/attribute" @@ -51,9 +50,16 @@ import ( ) const ( - BasePath = "/mgmt/v1.2/rest" - taskPollTimeout = 3600 * time.Second // Seconds - taskPollIntervalCap = 30 * time.Second //Seconds, The maximum duration between calls when polling task objects + BasePath = "/mgmt/v1.2/rest" + taskPollTimeout = 3600 * time.Second // overall deadline for a single task poll + // Share-create tasks always take longer than ~2s and almost always finish + // under ~15s. So poll at a tight fixed 2s cadence while completion is likely, + // then relax to 4s. This keeps detection latency ~2s in the common case; the + // previous exponential backoff (capped at 30s) could add up to a full 30s of + // detection lag after the task had already completed. + taskPollFastInterval = 2 * time.Second // poll every 2s ... + taskPollFastWindow = 30 * time.Second // ... for the first 30s ... + taskPollSlowInterval = 4 * time.Second // ... then back off to every 4s taskStatusValidationFailed = "VALIDATION_FAILED" taskStatusResumed = "RESUMED" taskStatusFailed = "FAILED" @@ -130,7 +136,7 @@ func (client *HammerspaceClient) GetPortalFloatingIp(ctx context.Context) (strin if err != nil { return "", err } - statusCode, respBody, _, err := client.doRequest(*req) + statusCode, respBody, _, err := client.doRequest(ctx, *req) if err != nil { return "", err } @@ -192,7 +198,7 @@ func (client *HammerspaceClient) GetDataPortals(ctx context.Context, nodeID stri return nil, err } - statusCode, respBody, _, err := client.doRequest(*req) + statusCode, respBody, _, err := client.doRequest(ctx, *req) if err != nil { log.Error(err) @@ -265,19 +271,39 @@ func (client *HammerspaceClient) EnsureLogin() error { return err } -func (client *HammerspaceClient) doRequest(req http.Request) (int, string, map[string][]string, error) { +func (client *HammerspaceClient) doRequest(ctx context.Context, req http.Request) (int, string, map[string][]string, error) { + // Low-cardinality route template (share/file/task IDs collapsed to {id}), + // used for both the latency histogram path label and the per-request counter. + route := common.AnvilRoute(req.URL.Path) + ctx, span := tracer.Start(ctx, "HammerspaceClient.doRequest", trace.WithAttributes( + attribute.String("http.method", req.Method), + attribute.String("http.url", req.URL.String()), + )) + defer span.End() + defer common.MeasureOp(ctx, "HammerspaceClient.doRequest", + attribute.String("http.method", req.Method), + attribute.String("http.path", route), + )(nil) log.Debugf("sending request %s %s", req.Method, req.URL) - resp, err := client.httpclient.Do(&req) + resp, err := client.httpclient.Do(req.WithContext(ctx)) // Attempt to login if err == nil && (resp.StatusCode == 401 || resp.StatusCode == 403) { client.EnsureLogin() - resp, err = client.httpclient.Do(&req) + resp, err = client.httpclient.Do(req.WithContext(ctx)) } if err != nil { + span.RecordError(err) + // Count the attempt even on transport failure (status 0), so the + // request rate/count reflects every call the driver makes. + common.RecordAnvilRequest(ctx, req.Method, route, 0) return 0, "", nil, err } defer resp.Body.Close() + span.SetAttributes(attribute.Int("http.status_code", resp.StatusCode)) + // Count every completed request with its real method + route + status, so + // non-GET verbs and the 404 type-probes are all visible in metrics. + common.RecordAnvilRequest(ctx, req.Method, route, resp.StatusCode) body, err := io.ReadAll(resp.Body) bodyString := string(body) responseLog := log.WithFields(log.Fields{ @@ -336,26 +362,41 @@ func (client *HammerspaceClient) generateRequest(ctx context.Context, verb, urlP } func (client *HammerspaceClient) WaitForTaskCompletion(ctx context.Context, taskLocation string) (bool, error) { - b := &backoff.Backoff{ - Max: taskPollIntervalCap, - Factor: 1.5, - Jitter: true, - } + // The task-completion poll (CreateShare returns 202 + a task, then we poll + // GET /tasks/{id} until terminal) is typically the dominant cost of share + // creation - the share-backed analog of the file-visibility poll. Give it a + // dedicated metric + span with an attempt count so the dashboard and traces + // can localize it instead of it hiding inside Controller/CreateVolume. + defer common.MeasureOp(ctx, "HammerspaceClient.WaitForTaskCompletion")(nil) + ctx, span := tracer.Start(ctx, "HammerspaceClient.WaitForTaskCompletion") + attempts := 0 + defer func() { + span.SetAttributes(attribute.Int("attempts", attempts)) + span.End() + }() + taskUrl, _ := url.Parse(taskLocation) taskId := path.Base(taskUrl.Path) startTime := time.Now() var task common.Task for time.Since(startTime) < taskPollTimeout { - d := b.Duration() - time.Sleep(d) + // Sleep-first: share-create tasks never complete in under ~2s, so there + // is no point checking immediately. Poll every 2s for the first 30s, then + // relax to every 4s for the long tail. + interval := taskPollFastInterval + if time.Since(startTime) >= taskPollFastWindow { + interval = taskPollSlowInterval + } + time.Sleep(interval) + attempts++ req, err := client.generateRequest(ctx, "GET", "/tasks/"+taskId, "") if err != nil { log.Error("Failed to generate request object") os.Exit(1) } - statusCode, respBody, _, err := client.doRequest(*req) + statusCode, respBody, _, err := client.doRequest(ctx, *req) if err != nil { return false, err } @@ -385,7 +426,7 @@ func (client *HammerspaceClient) ListShares(ctx context.Context) ([]common.Share log.Error(err) return nil, err } - statusCode, respBody, _, err := client.doRequest(*req) + statusCode, respBody, _, err := client.doRequest(ctx, *req) if err != nil { log.Error(err) @@ -420,7 +461,7 @@ func (client *HammerspaceClient) ListObjectives(ctx context.Context) ([]common.C return nil, err } - statusCode, respBody, _, err := client.doRequest(*req) + statusCode, respBody, _, err := client.doRequest(ctx, *req) if err != nil { log.Error(err) @@ -503,7 +544,7 @@ func (client *HammerspaceClient) ListVolumes(ctx context.Context) ([]common.Volu return nil, err } - statusCode, respBody, _, err := client.doRequest(*req) + statusCode, respBody, _, err := client.doRequest(ctx, *req) if err != nil { log.Error(err) return nil, err @@ -587,7 +628,7 @@ func (client *HammerspaceClient) ListSnapshots(ctx context.Context, snapshot_id, func (client *HammerspaceClient) GetShare(ctx context.Context, name string) (*common.ShareResponse, error) { req, err := client.generateRequest(ctx, "GET", "/shares/"+url.PathEscape(name), "") - statusCode, respBody, _, err := client.doRequest(*req) + statusCode, respBody, _, err := client.doRequest(ctx, *req) if err != nil { log.Error(err) @@ -610,7 +651,7 @@ func (client *HammerspaceClient) GetShare(ctx context.Context, name string) (*co func (client *HammerspaceClient) GetShareRawFields(ctx context.Context, name string) (map[string]interface{}, error) { req, err := client.generateRequest(ctx, "GET", "/shares/"+url.PathEscape(name), "") - statusCode, respBody, _, err := client.doRequest(*req) + statusCode, respBody, _, err := client.doRequest(ctx, *req) if err != nil { log.Error(err) @@ -633,7 +674,7 @@ func (client *HammerspaceClient) GetShareRawFields(ctx context.Context, name str func (client *HammerspaceClient) GetFile(ctx context.Context, path string) (*common.File, error) { req, err := client.generateRequest(ctx, "GET", "/files?path="+url.PathEscape(path), "") - statusCode, respBody, _, err := client.doRequest(*req) + statusCode, respBody, _, err := client.doRequest(ctx, *req) if err != nil { log.Error(err) @@ -671,6 +712,7 @@ func (client *HammerspaceClient) CreateShare(ctx context.Context, exportOptions []common.ShareExportOptions, deleteDelay int64, comment string) error { + defer common.MeasureOp(ctx, "HammerspaceClient.CreateShare")(nil) log.Debug("Creating share: " + name) extendedInfo := common.GetCommonExtendedInfo() @@ -708,7 +750,7 @@ func (client *HammerspaceClient) CreateShare(ctx context.Context, log.Errorf("unable to genrate share create request with POST. Error %v", err) return err } - statusCode, _, respHeaders, err := client.doRequest(*req) + statusCode, _, respHeaders, err := client.doRequest(ctx, *req) if err != nil { log.Error(err) @@ -746,6 +788,7 @@ func (client *HammerspaceClient) CreateShare(ctx context.Context, } func (client *HammerspaceClient) CreateShareFromSnapshot(ctx context.Context, name string, exportPath string, size int64, objectives []string, exportOptions []common.ShareExportOptions, deleteDelay int64, comment string, snapshotPath string) error { + defer common.MeasureOp(ctx, "HammerspaceClient.CreateShareFromSnapshot")(nil) log.WithFields(log.Fields{ "name": name, "deleteDelay": deleteDelay, @@ -790,7 +833,7 @@ func (client *HammerspaceClient) CreateShareFromSnapshot(ctx context.Context, na log.Errorf("unable to genrate share create request with POST. Error %v", err) return err } - statusCode, _, respHeaders, err := client.doRequest(*req) + statusCode, _, respHeaders, err := client.doRequest(ctx, *req) if err != nil { log.Error(err) @@ -833,7 +876,7 @@ func (client *HammerspaceClient) CheckIfShareCreateTaskIsRunning(ctx context.Con log.Error("Failed to generate request object") return false, err } - statusCode, respBody, _, err := client.doRequest(*req) + statusCode, respBody, _, err := client.doRequest(ctx, *req) if err != nil { return false, err } @@ -873,7 +916,7 @@ func (client *HammerspaceClient) SetObjectives(ctx context.Context, shareName st objectiveName, shareName, path, err) return err } - statusCode, respBody, _, err := client.doRequest(*req) + statusCode, respBody, _, err := client.doRequest(ctx, *req) if err != nil { log.Errorf("Failed to set objective %s on share %s at path %s, %v", objectiveName, shareName, path, err) @@ -909,7 +952,7 @@ func (client *HammerspaceClient) UpdateShareSize(ctx context.Context, name strin log.Error(err) return err } - statusCode, _, respHeaders, err := client.doRequest(*req) + statusCode, _, respHeaders, err := client.doRequest(ctx, *req) if err != nil { log.Error(err) @@ -954,7 +997,7 @@ func (client *HammerspaceClient) DeleteShare(ctx context.Context, name string, d if err != nil { return err } - statusCode, body, respHeaders, err := client.doRequest(*req) + statusCode, body, respHeaders, err := client.doRequest(ctx, *req) if err != nil { return err } @@ -991,7 +1034,7 @@ func (client *HammerspaceClient) DeleteShare(ctx context.Context, name string, d func (client *HammerspaceClient) SnapshotShare(ctx context.Context, shareName string) (string, error) { req, err := client.generateRequest(ctx, "POST", fmt.Sprintf("/share-snapshots/snapshot-create/%s", url.PathEscape(shareName)), "") - statusCode, respBody, _, err := client.doRequest(*req) + statusCode, respBody, _, err := client.doRequest(ctx, *req) if err != nil { log.Error(err) @@ -1015,7 +1058,7 @@ func (client *HammerspaceClient) SnapshotShare(ctx context.Context, shareName st func (client *HammerspaceClient) GetShareSnapshots(ctx context.Context, shareName string) ([]string, error) { req, _ := client.generateRequest(ctx, "GET", fmt.Sprintf("/share-snapshots/snapshot-list/%s", url.PathEscape(shareName)), "") - statusCode, respBody, _, err := client.doRequest(*req) + statusCode, respBody, _, err := client.doRequest(ctx, *req) if err != nil { return nil, err @@ -1046,7 +1089,7 @@ func (client *HammerspaceClient) DeleteShareSnapshot(ctx context.Context, shareN req, _ := client.generateRequest(ctx, "POST", fmt.Sprintf("/share-snapshots/snapshot-delete/%s/%s", url.PathEscape(shareName), url.PathEscape(snapshotName)), "") - statusCode, _, _, err := client.doRequest(*req) + statusCode, _, _, err := client.doRequest(ctx, *req) trace.SpanFromContext(ctx).SetAttributes( attribute.String("share.name", shareName), attribute.String("snapshot.name", snapshotName), @@ -1065,7 +1108,7 @@ func (client *HammerspaceClient) DeleteShareSnapshot(ctx context.Context, shareN func (client *HammerspaceClient) GetFileSnapshots(ctx context.Context, filePath string) ([]common.FileSnapshot, error) { req, _ := client.generateRequest(ctx, "GET", fmt.Sprintf("/file-snapshots/list?filename-expression=%s", url.PathEscape(filePath)), "") - statusCode, respBody, _, err := client.doRequest(*req) + statusCode, respBody, _, err := client.doRequest(ctx, *req) if err != nil { return nil, err @@ -1092,7 +1135,7 @@ func (client *HammerspaceClient) DeleteFileSnapshot(ctx context.Context, filePat req, _ := client.generateRequest(ctx, "POST", fmt.Sprintf("/file-snapshots/delete?filename-expression=%s&date-time-expression=%s", url.PathEscape(filePath), url.PathEscape(snapshotTime)), "") - statusCode, respBody, _, err := client.doRequest(*req) + statusCode, respBody, _, err := client.doRequest(ctx, *req) if err != nil { return err @@ -1119,7 +1162,7 @@ func (client *HammerspaceClient) SnapshotFile(ctx context.Context, filepath stri return "", err } - statusCode, respBody, _, err := client.doRequest(*req) + statusCode, respBody, _, err := client.doRequest(ctx, *req) if err != nil { log.Error(err) @@ -1146,7 +1189,7 @@ func (client *HammerspaceClient) RestoreFileSnapToDestination(ctx context.Contex return err } - statusCode, _, _, err := client.doRequest(*req) + statusCode, _, _, err := client.doRequest(ctx, *req) if err != nil { log.Error(err) @@ -1165,7 +1208,7 @@ func (client *HammerspaceClient) GetClusterAvailableCapacity(ctx context.Context return 0, err } - statusCode, respBody, _, err := client.doRequest(*req) + statusCode, respBody, _, err := client.doRequest(ctx, *req) if err != nil { log.Error(err) diff --git a/pkg/common/config.go b/pkg/common/config.go index 8f3160d..ce3895b 100644 --- a/pkg/common/config.go +++ b/pkg/common/config.go @@ -30,6 +30,20 @@ const ( DefaultBackingFileSizeBytes = 1073741824 DefaultVolumeNameFormat = "%s" + // MinXfsSizeBytes is the minimum size (300 MiB) below which xfsprogs 6.4+ + // warns "Filesystem should be larger than 300MB" and marks the resulting + // filesystem "deprecated and will not be supported in future releases". + // The mkfs.xfs command still returns 0 in that case, so we must reject + // the request in CreateVolume before formatting. + MinXfsSizeBytes = 300 * 1024 * 1024 + + // MinExt4SizeBytes is the minimum size (20 MiB) below which we refuse to + // format ext4. mkfs.ext4 accepts filesystems as small as ~8 MiB but the + // journal + reserved-block overhead means anything smaller than ~20 MiB + // has almost no usable space; we reject early so users get a clean error + // instead of a "filesystem full" surprise on the first write. + MinExt4SizeBytes = 20 * 1024 * 1024 + // Topology keys TopologyKeyDataPortal = "topology.csi.hammerspace.com/is-data-portal" ) diff --git a/pkg/common/error_text.go b/pkg/common/error_text.go index 3929e5f..3f63ae9 100644 --- a/pkg/common/error_text.go +++ b/pkg/common/error_text.go @@ -39,6 +39,9 @@ const ( MissingMountBackingShareName = "mountBackingShareName must be provided when creating Filesystem volumes other than 'nfs'" BlockVolumeSizeNotSpecified = "capacity must be specified for block volumes" ShareNotMounted = "share is not in mounted state." + XfsSizeBelowMinimum = "fsType=xfs volumes must be at least %d bytes (%d MiB); requested %d bytes (%d MiB). mkfs.xfs 6.4+ marks smaller filesystems as deprecated." // min_bytes, min_mib, req_bytes, req_mib + Ext4SizeBelowMinimum = "fsType=ext4 volumes must be at least %d bytes (%d MiB); requested %d bytes (%d MiB). Smaller ext4 filesystems have almost no usable space." // min_bytes, min_mib, req_bytes, req_mib + Ext3NotSupported = "fsType=ext3 is not supported; use ext4 or xfs." InvalidExportOptions = "export options must consist of 3 values: subnet,access,rootSquash, received '%s'" InvalidRootSquash = "rootSquash must be a bool. Value received '%s'" diff --git a/pkg/common/host_utils.go b/pkg/common/host_utils.go index a519af3..92611bf 100644 --- a/pkg/common/host_utils.go +++ b/pkg/common/host_utils.go @@ -28,22 +28,76 @@ import ( "slices" "strconv" "strings" + "sync" "time" log "github.com/sirupsen/logrus" unix "golang.org/x/sys/unix" + "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/trace" + "google.golang.org/grpc/codes" "google.golang.org/grpc/status" "k8s.io/mount-utils" ) +var commonTracer = otel.Tracer("hammerspace-csi/common") + const LOOP_CTL_GET_FREE = 0x4C82 var ( defaultMountCheckTimeout time.Duration = 50 * time.Second // Default timeout for checking mount status ) +// inFlightMount tracks a single mount syscall that is (or was) running for a +// target path. Its done channel is closed when the syscall returns; err then +// holds the result. +type inFlightMount struct { + done chan struct{} + err error +} + +// mountInFlight deduplicates concurrent/retried mount syscalls by target path. +// +// A `hard` NFS mount against a dead data portal blocks uninterruptibly in the +// kernel, so MountShare can't kill the goroutine running it — it can only time +// out and return. Left unguarded, every retried CreateVolume/NodeStageVolume +// against that same target would fork ANOTHER blocked goroutine, so the count +// grows without bound over a long outage. Keyed by target path, this registry +// lets a retry attach to the already-running attempt instead of spawning a new +// one, bounding lingering mount goroutines to at most one per distinct target. +var ( + mountInFlightMu sync.Mutex + mountInFlight = map[string]*inFlightMount{} +) + +// beginMount starts mountFn for targetPath, unless a mount for that same target is +// already running, in which case it returns the in-flight attempt and does NOT call +// mountFn again. The returned inFlightMount's done channel is closed when the syscall +// returns; its err holds the result. Callers wait on it with their own timeout, so a +// timed-out caller leaves the single shared goroutine running rather than forking a +// new one on the next retry. +func beginMount(targetPath string, mountFn func() error) *inFlightMount { + mountInFlightMu.Lock() + defer mountInFlightMu.Unlock() + if att, ok := mountInFlight[targetPath]; ok { + return att + } + att := &inFlightMount{done: make(chan struct{})} + mountInFlight[targetPath] = att + go func() { + mountErr := mountFn() + mountInFlightMu.Lock() + delete(mountInFlight, targetPath) + mountInFlightMu.Unlock() + att.err = mountErr + close(att.done) + }() + return att +} + func init() { // Read environment variables for mount check timeout mountCheckTimeoutStr := os.Getenv("MOUNT_CHECK_TIMEOUT") @@ -205,11 +259,18 @@ func GetDeviceMinorNumber(device string) (uint32, error) { return unix.Minor(dev), nil } -func MakeEmptyRawFile(pathname string, size int64) error { +func MakeEmptyRawFile(ctx context.Context, pathname string, size int64) error { + _, span := commonTracer.Start(ctx, "MakeEmptyRawFile", trace.WithAttributes( + attribute.String("path", pathname), + attribute.Int64("size", size), + )) + defer span.End() + defer MeasureOp(ctx, "MakeEmptyRawFile")(nil) log.Infof("creating file '%s'", pathname) sizeStr := strconv.FormatInt(size, 10) output, err := ExecCommand("qemu-img", "create", "-fraw", pathname, sizeStr) if err != nil { + span.RecordError(err) log.Errorf("%s, %v", output, err.Error()) return err } @@ -239,14 +300,34 @@ func ExpandDeviceFileSize(pathname string, size int64) error { return nil } -func FormatDevice(device, fsType string) error { +func FormatDevice(ctx context.Context, device, fsType string) error { + _, span := commonTracer.Start(ctx, "FormatDevice", trace.WithAttributes( + attribute.String("device", device), + attribute.String("fsType", fsType), + )) + defer span.End() + defer MeasureOp(ctx, "FormatDevice", attribute.String("fsType", fsType))(nil) log.Infof("formatting file '%s' with '%s' filesystem", device, fsType) args := []string{device} if fsType == "xfs" { - args = []string{"-m", "reflink=0", device} + // -K: skip the block discard/TRIM pass at mkfs time. Over an NFS-backed + // file this discard is slow and pure overhead (the backing share manages + // its own space reclaim), so it needlessly inflates mkfs.xfs latency and + // load on the storage data path. + args = []string{"-m", "reflink=0", "-K", device} + } else if fsType == "ext4" { + // Defer inode-table and journal zeroing to lazy background init. + // Without this, mkfs.ext* eagerly zeroes the inode table and journal at + // create time (~tens of MB per volume). For file-backed volumes those + // writes go over NFS to the backing share, so they saturate the storage + // data path (the DSX disk) and dominate CreateVolume latency. Lazy init + // drops create-time writes to ~KB; the kernel finishes initialization in + // the background after first mount. + args = []string{"-E", "lazy_itable_init=1,lazy_journal_init=1", device} } output, err := ExecCommand(fmt.Sprintf("mkfs.%s", fsType), args...) if err != nil { + span.RecordError(err) log.Errorf("Error executing mkfs command. %v", err) if output != nil && strings.Contains(string(output), "will not make a filesystem here") { log.Warningf("Device %s is already mounted", device) @@ -324,7 +405,14 @@ func DeleteFile(pathname string) error { return nil } -func MountShare(sourcePath, targetPath string, mountFlags []string) error { +func MountShare(ctx context.Context, sourcePath, targetPath string, mountFlags []string) error { + _, span := commonTracer.Start(ctx, "MountShare", trace.WithAttributes( + attribute.String("source", sourcePath), + attribute.String("target", targetPath), + attribute.StringSlice("flags", mountFlags), + )) + defer span.End() + defer MeasureOp(ctx, "MountShare")(nil) log.Infof("mounting %s to %s, with options %v", sourcePath, targetPath, mountFlags) mounted, err := SafeIsMountPoint(targetPath) if err != nil { @@ -346,7 +434,27 @@ func MountShare(sourcePath, targetPath string, mountFlags []string) error { mo := mountFlags mounter := mount.New("") - err = mounter.Mount(sourcePath, targetPath, "nfs", mo) + // Bound the mount syscall with a timeout. A `hard` NFS mount against an + // unreachable server (e.g. a data portal that has gone away) blocks + // uninterruptibly; without this bound the caller holds its volume / + // backing-share lock forever, leaking it and wedging every serialized + // operation behind that lock. On timeout we return an error so the deferred + // unlock runs and the op is retried. + // + // The blocked mount goroutine can't be killed (the syscall is uninterruptible), + // but it holds no lock, and beginMount keeps a retry against the same target from + // forking a fresh one each time — so at most one such goroutine lingers per target, + // and it drains on its own when the portal recovers or the mount fails. + att := beginMount(targetPath, func() error { + return mounter.Mount(sourcePath, targetPath, "nfs", mo) + }) + select { + case <-att.done: + err = att.err + case <-time.After(defaultMountCheckTimeout): + return status.Errorf(codes.DeadlineExceeded, + "mount %s -> %s timed out after %s (data portal unreachable?)", sourcePath, targetPath, defaultMountCheckTimeout) + } if err != nil { if os.IsPermission(err) { return status.Error(codes.PermissionDenied, err.Error()) @@ -535,22 +643,44 @@ func CheckNFSExports(address string) (bool, error) { } func IsShareMounted(targetPath string) bool { - mounter := mount.New("") - isMounted, err := mounter.IsMountPoint(targetPath) + // Use the timeout-bounded check: IsMountPoint stat()s the path, which hangs + // uninterruptibly on a stale NFS mount whose server is gone (e.g. after the + // driver is re-pointed to a new Anvil). SafeIsMountPoint returns + // context.DeadlineExceeded instead of hanging; treat that as "not cleanly + // mounted" so callers re-establish the mount rather than block forever. + isMounted, err := SafeIsMountPoint(targetPath) if err != nil { if os.IsNotExist(err) { log.Warnf("Check [IsShareMounted] target path is empty, %s", EmptyTargetPath) - return false } else { log.Warnf("Error while checking mount point for targetPath %s, Error %v", targetPath, err) - return false } + return false } log.Debugf("Target path %s isMounted %t", targetPath, isMounted) return isMounted } -func UnmountFilesystem(targetPath string) error { +// ForceUnmountStale best-effort clears a (possibly hung) mount at targetPath so a +// fresh mount can be established. It shells out to `umount -f -l` (force + lazy), +// which detaches even when the NFS server is unreachable and never blocks - a +// plain unmount syscall would itself hang on a dead mount. Safe to call when +// nothing is mounted (it just logs). +func ForceUnmountStale(targetPath string) { + out, err := ExecCommand("umount", "-f", "-l", targetPath) + if err != nil { + log.Warnf("ForceUnmountStale: umount -f -l %s: %v (output: %s)", targetPath, err, strings.TrimSpace(string(out))) + return + } + log.Infof("ForceUnmountStale: cleared stale mount at %s", targetPath) +} + +func UnmountFilesystem(ctx context.Context, targetPath string) error { + _, span := commonTracer.Start(ctx, "UnmountFilesystem", trace.WithAttributes( + attribute.String("target", targetPath), + )) + defer span.End() + defer MeasureOp(ctx, "UnmountFilesystem")(nil) log.Infof("UnmountFilesystem is called with targetPath %s", targetPath) mounter := mount.New("") @@ -577,7 +707,8 @@ func UnmountFilesystem(targetPath string) error { return nil } -func SetMetadataTags(localPath string, tags map[string]string) error { +func SetMetadataTags(ctx context.Context, localPath string, tags map[string]string) error { + defer MeasureOp(ctx, "SetMetadataTags")(nil) // hs attribute set localpath -e "CSI_DETAILS_TABLE{'','','',''}" attributeSetOutput, err := ExecCommand("hs", "attribute", diff --git a/pkg/common/host_utils_test.go b/pkg/common/host_utils_test.go index 06662d1..bca58d8 100644 --- a/pkg/common/host_utils_test.go +++ b/pkg/common/host_utils_test.go @@ -1,8 +1,11 @@ package common import ( + "errors" "reflect" + "sync/atomic" "testing" + "time" ) func TestGetNFSExports(t *testing.T) { @@ -87,3 +90,81 @@ func TestExecCommandHelper(t *testing.T) { } } + +// TestBeginMountDeduplicatesByTarget covers the fix for PR #67 review note #2: +// a mount goroutine wedged on a dead data portal must not be re-forked by every +// retry against the same target. beginMount should attach retries to the single +// in-flight attempt, then start fresh once it completes. +func TestBeginMountDeduplicatesByTarget(t *testing.T) { + mountInFlightMu.Lock() + mountInFlight = map[string]*inFlightMount{} + mountInFlightMu.Unlock() + + const target = "/mnt/dead-portal-vol" + var starts int32 // how many mount funcs actually began + release := make(chan struct{}) // gate that keeps the first mount "wedged" + + blockingMount := func() error { + atomic.AddInt32(&starts, 1) + <-release // simulate a hard NFS mount stuck in the kernel + return nil + } + shouldNotRun := func() error { + atomic.AddInt32(&starts, 1) + return errors.New("second mount fn should not have been invoked") + } + + // First attempt starts the (blocked) mount goroutine. + att1 := beginMount(target, blockingMount) + + // Wait until the goroutine has actually entered mountFn. + deadline := time.After(2 * time.Second) + for atomic.LoadInt32(&starts) == 0 { + select { + case <-deadline: + t.Fatal("first mount goroutine never started") + default: + time.Sleep(time.Millisecond) + } + } + + // Retries against the SAME target while the first is still wedged must attach + // to att1 and must NOT invoke a new mount fn. + for i := 0; i < 5; i++ { + if att := beginMount(target, shouldNotRun); att != att1 { + t.Fatalf("retry %d got a different attempt; expected dedup to the in-flight one", i) + } + } + if got := atomic.LoadInt32(&starts); got != 1 { + t.Fatalf("mount fn ran %d times across 6 attempts; want exactly 1 (dedup failed)", got) + } + + // Let the wedged mount finish; att1.done must close and carry its result. + close(release) + select { + case <-att1.done: + case <-time.After(2 * time.Second): + t.Fatal("in-flight mount never completed after release") + } + if att1.err != nil { + t.Fatalf("unexpected mount error: %v", att1.err) + } + + // Entry must be cleared so a later mount of the same target starts fresh. + mountInFlightMu.Lock() + _, stillThere := mountInFlight[target] + mountInFlightMu.Unlock() + if stillThere { + t.Fatal("completed mount left a stale in-flight entry") + } + + // A fresh attempt after completion spawns a new goroutine. + att2 := beginMount(target, func() error { atomic.AddInt32(&starts, 1); return nil }) + <-att2.done + if att2 == att1 { + t.Fatal("post-completion mount reused the old attempt") + } + if got := atomic.LoadInt32(&starts); got != 2 { + t.Fatalf("post-completion mount fn didn't run fresh; starts=%d want 2", got) + } +} diff --git a/pkg/common/hs_types.go b/pkg/common/hs_types.go index a3fa7ff..4f20cec 100644 --- a/pkg/common/hs_types.go +++ b/pkg/common/hs_types.go @@ -32,6 +32,7 @@ type HSVolumeParameters struct { CacheEnabled bool FQDN string MountFlags []string + ObjectiveTarget string } type HSVolume struct { @@ -51,6 +52,7 @@ type HSVolume struct { AdditionalMetadataTags map[string]string FQDN string MountFlags []string + ObjectiveTarget string } ///// Request and Response objects for interacting with the HS API diff --git a/pkg/common/metrics.go b/pkg/common/metrics.go new file mode 100644 index 0000000..6fd165c --- /dev/null +++ b/pkg/common/metrics.go @@ -0,0 +1,256 @@ +/* +Copyright 2019 Hammerspace + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package common + +import ( + "context" + "strings" + "sync" + "time" + + "go.opentelemetry.io/otel" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/metric" +) + +// Operation metrics. These are the instruments the Grafana "Hammerspace CSI +// Driver" dashboard queries: +// +// hs_csi_operation_duration_seconds (histogram) - per-operation latency +// hs_csi_operation_errors_total (counter) - per-operation error count +// hs_csi_operation_inflight (up/down) - concurrent operations +// +// all keyed by an `operation` label plus any extra attributes the call site +// supplies (e.g. http.method, http.path, fsType -> http_method/http_path/fsType +// after the OTel->Prometheus name mapping). +var ( + opMetricsOnce sync.Once + opDuration metric.Float64Histogram + opErrors metric.Int64Counter + opInflight metric.Int64UpDownCounter +) + +// initOpMetrics binds the package instruments to the currently-installed global +// MeterProvider. It runs once, lazily, on the first MeasureOp call - which +// happens while serving an RPC, i.e. well after main.init() has installed the +// real (Prometheus) MeterProvider. With no exporter configured the global meter +// is a no-op and every instrument call is free. +func initOpMetrics() { + m := otel.Meter("github.com/hammer-space/csi-plugin") + opDuration, _ = m.Float64Histogram( + "hs_csi_operation_duration_seconds", + metric.WithDescription("Duration of a CSI operation or internal step, in seconds"), + metric.WithUnit("s"), + // Sub-second-friendly buckets: the OTel default boundaries (0,5,10,25,...) + // are far too coarse for CSI ops, dumping every sub-5s call into one bucket + // so histogram_quantile just returns the bucket midpoint. These give real + // p50/p95/p99 resolution from milliseconds up to 30s. + metric.WithExplicitBucketBoundaries( + 0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1, 2, 5, 10, 20, 30, + ), + ) + opErrors, _ = m.Int64Counter( + "hs_csi_operation_errors_total", + metric.WithDescription("Number of CSI operations/steps that returned an error"), + ) + opInflight, _ = m.Int64UpDownCounter( + "hs_csi_operation_inflight", + metric.WithDescription("Number of in-flight CSI operations/steps"), + ) +} + +// MeasureOp instruments a CSI operation or internal step, mirroring the tracing +// spans. Call it at the start of the operation; it bumps the in-flight gauge and +// returns a closure that - when invoked, typically via defer - records the +// elapsed duration, decrements the in-flight gauge, and (when passed a non-nil +// error) increments the error counter: +// +// defer common.MeasureOp(ctx, "Controller/CreateVolume")(nil) +// defer common.MeasureOp(ctx, "FormatDevice", attribute.String("fsType", fsType))(nil) +// +// Metrics are labeled by `operation` plus any supplied attributes. +func MeasureOp(ctx context.Context, operation string, attrs ...attribute.KeyValue) func(*error) { + opMetricsOnce.Do(initOpMetrics) + start := time.Now() + labels := make([]attribute.KeyValue, 0, len(attrs)+1) + labels = append(labels, attribute.String("operation", operation)) + labels = append(labels, attrs...) + opt := metric.WithAttributes(labels...) + opInflight.Add(ctx, 1, opt) + return func(errp *error) { + opInflight.Add(ctx, -1, opt) + opDuration.Record(ctx, time.Since(start).Seconds(), opt) + if errp != nil && *errp != nil { + opErrors.Add(ctx, 1, opt) + } + } +} + +// Anvil REST request counter. Every call to HammerspaceClient.doRequest records +// one increment here, labeled by HTTP method, a low-cardinality route template +// (per-resource IDs collapsed to {id}), and the response status code (0 on a +// transport error). Unlike the doRequest latency histogram - which is recorded +// from a deferred closure set up *before* the response is known and therefore +// can only carry method+path - this is recorded *after* the response, so it +// carries the real status code. Together they let the dashboard show every Anvil +// call (GET/POST/PUT/DELETE) split by outcome, including the 404 type-probes that +// dominate file-backed traffic. +var ( + anvilReqOnce sync.Once + anvilReqs metric.Int64Counter +) + +func initAnvilReqMetric() { + m := otel.Meter("github.com/hammer-space/csi-plugin") + anvilReqs, _ = m.Int64Counter( + "hs_csi_anvil_requests_total", + metric.WithDescription("Total Anvil REST requests, by HTTP method, route template, and status code"), + ) +} + +// RecordAnvilRequest counts a single Anvil REST call. Call it exactly once per +// request, after the response (or transport error, in which case statusCode is +// 0) is known. Labels map to Prometheus as http_method, http_route, +// http_status_code. +func RecordAnvilRequest(ctx context.Context, method, route string, statusCode int) { + anvilReqOnce.Do(initAnvilReqMetric) + anvilReqs.Add(ctx, 1, metric.WithAttributes( + attribute.String("http.method", method), + attribute.String("http.route", route), + attribute.Int("http.status_code", statusCode), + )) +} + +// AnvilRoute collapses per-resource identifiers in an Anvil REST URL path down to +// a stable, low-cardinality template, so metrics don't explode into one series +// per share/file/task/snapshot. e.g. /mgmt/v1.2/rest/shares/file--pvc- +// becomes /mgmt/v1.2/rest/shares/{id}. Collection "action" segments such as +// file-snapshots/list are preserved. Query strings never reach here (callers pass +// req.URL.Path), so /files?path=... is already just /files. +func AnvilRoute(urlPath string) string { + segs := strings.Split(urlPath, "/") + for i := 1; i < len(segs); i++ { + switch segs[i-1] { + case "shares", "tasks", "files", "objectives": + if segs[i] != "" { + segs[i] = "{id}" + } + case "file-snapshots": + if segs[i] != "" && segs[i] != "list" { + segs[i] = "{id}" + } + case "snapshot-create", "snapshot-list", "snapshot-delete": + // /share-snapshots//[/] — keep the action + // verb, collapse the share name and (for delete) the trailing + // snapshot name, both of which are unbounded identifiers. + for j := i; j < len(segs); j++ { + if segs[j] != "" { + segs[j] = "{id}" + } + } + } + } + return strings.Join(segs, "/") +} + +// Lock metrics. The driver serializes CSI operations behind keyed in-memory +// locks (per-volume and per-backing-share via volumeLocks, per-snapshot via +// snapshotLocks). A lock that is acquired but never released - e.g. when the +// holder is stuck in an uninterruptible mount syscall so its deferred unlock +// never runs - is otherwise invisible. These instruments make it observable: +// +// hs_csi_locks_held (up/down) - locks currently held; a LEAK +// shows as a value stuck > 0 with +// no further acquire/release. +// hs_csi_lock_wait_seconds (hist) - time blocked acquiring (contention) +// hs_csi_lock_hold_seconds (hist) - time a lock was held before release +// hs_csi_lock_acquire_failures_total (counter) - acquires that timed out (-> Aborted) +// +// all keyed by `lock_type` (volume | snapshot). +var ( + lockMetricsOnce sync.Once + locksHeld metric.Int64UpDownCounter + lockWait metric.Float64Histogram + lockHold metric.Float64Histogram + lockFails metric.Int64Counter +) + +func initLockMetrics() { + m := otel.Meter("github.com/hammer-space/csi-plugin") + locksHeld, _ = m.Int64UpDownCounter( + "hs_csi_locks_held", + metric.WithDescription("CSI keyed locks currently held (a leak shows as a stuck non-zero value)"), + ) + buckets := metric.WithExplicitBucketBoundaries(0.001, 0.01, 0.05, 0.1, 0.5, 1, 2, 5, 10, 20, 30) + lockWait, _ = m.Float64Histogram( + "hs_csi_lock_wait_seconds", + metric.WithDescription("Time spent blocked acquiring a CSI keyed lock"), + metric.WithUnit("s"), buckets, + ) + lockHold, _ = m.Float64Histogram( + "hs_csi_lock_hold_seconds", + metric.WithDescription("Time a CSI keyed lock was held before release"), + metric.WithUnit("s"), buckets, + ) + lockFails, _ = m.Int64Counter( + "hs_csi_lock_acquire_failures_total", + metric.WithDescription("CSI keyed-lock acquisitions that failed / timed out"), + ) +} + +// LockProbe instruments the lifecycle of one keyed-lock acquisition. Create it +// immediately BEFORE blocking on the acquire, then call exactly one of Failed() +// (acquire errored/timed out) or Acquired() (lock obtained). Acquired returns a +// release closure to run on unlock, which records the hold duration and drops +// the held gauge. Wiring: +// +// p := common.StartLockProbe(ctx, "volume") +// if err := lk.lock(lctx); err != nil { p.Failed(); return ... } +// release := p.Acquired() +// return func() { lk.unlock(); release() }, nil +type LockProbe struct { + ctx context.Context + lockType string + start time.Time +} + +func StartLockProbe(ctx context.Context, lockType string) *LockProbe { + lockMetricsOnce.Do(initLockMetrics) + return &LockProbe{ctx: ctx, lockType: lockType, start: time.Now()} +} + +func (p *LockProbe) opt() metric.MeasurementOption { + return metric.WithAttributes(attribute.String("lock_type", p.lockType)) +} + +// Failed records a failed/timed-out acquire (wait time + a failure increment). +func (p *LockProbe) Failed() { + lockWait.Record(p.ctx, time.Since(p.start).Seconds(), p.opt()) + lockFails.Add(p.ctx, 1, p.opt()) +} + +// Acquired records the wait time, bumps the held gauge, and returns a release +// func that records the hold duration and decrements the held gauge. +func (p *LockProbe) Acquired() func() { + lockWait.Record(p.ctx, time.Since(p.start).Seconds(), p.opt()) + locksHeld.Add(p.ctx, 1, p.opt()) + held := time.Now() + return func() { + lockHold.Record(p.ctx, time.Since(held).Seconds(), p.opt()) + locksHeld.Add(p.ctx, -1, p.opt()) + } +} diff --git a/pkg/common/metrics_test.go b/pkg/common/metrics_test.go new file mode 100644 index 0000000..10bf82e --- /dev/null +++ b/pkg/common/metrics_test.go @@ -0,0 +1,79 @@ +/* +Copyright 2019 Hammerspace + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package common + +import ( + "context" + "errors" + "testing" +) + +// TestAnvilRoute covers the metric route-template normalization (PR F): every +// per-resource id segment collapses to {id} so the Prometheus label doesn't +// explode into one series per share/file/task/snapshot, while action segments +// (e.g. file-snapshots/list, shares/{id}/objective-set) are preserved. +func TestAnvilRoute(t *testing.T) { + cases := map[string]string{ + "/mgmt/v1.2/rest/shares/file--pvc-1a2b3c": "/mgmt/v1.2/rest/shares/{id}", + "/mgmt/v1.2/rest/shares/k8s-file-backed/objective-set": "/mgmt/v1.2/rest/shares/{id}/objective-set", + "/mgmt/v1.2/rest/shares": "/mgmt/v1.2/rest/shares", + "/mgmt/v1.2/rest/tasks/abcd-1234": "/mgmt/v1.2/rest/tasks/{id}", + "/mgmt/v1.2/rest/files": "/mgmt/v1.2/rest/files", + "/mgmt/v1.2/rest/files/somefile": "/mgmt/v1.2/rest/files/{id}", + "/mgmt/v1.2/rest/objectives/keep-online": "/mgmt/v1.2/rest/objectives/{id}", + "/mgmt/v1.2/rest/file-snapshots/list": "/mgmt/v1.2/rest/file-snapshots/list", + "/mgmt/v1.2/rest/file-snapshots/snap-9": "/mgmt/v1.2/rest/file-snapshots/{id}", + "/mgmt/v1.2/rest/cntl/state": "/mgmt/v1.2/rest/cntl/state", + "/mgmt/v1.2/rest/data-portals/": "/mgmt/v1.2/rest/data-portals/", + // share-snapshots: keep the action verb, collapse share + snapshot ids + "/mgmt/v1.2/rest/share-snapshots/snapshot-create/k8s-file-backed": "/mgmt/v1.2/rest/share-snapshots/snapshot-create/{id}", + "/mgmt/v1.2/rest/share-snapshots/snapshot-list/k8s-file-backed": "/mgmt/v1.2/rest/share-snapshots/snapshot-list/{id}", + "/mgmt/v1.2/rest/share-snapshots/snapshot-delete/k8s-file-backed/2026-07-24T00-00": "/mgmt/v1.2/rest/share-snapshots/snapshot-delete/{id}/{id}", + } + for in, want := range cases { + if got := AnvilRoute(in); got != want { + t.Errorf("AnvilRoute(%q) = %q, want %q", in, got, want) + } + } +} + +// TestMeasureOp verifies the MeasureOp contract (PR F): it returns a non-nil +// finish closure that is safe to call with a nil *error, a nil-valued *error, +// and a non-nil error, without panicking (instruments are no-ops until a real +// MeterProvider is installed). +func TestMeasureOp(t *testing.T) { + ctx := context.Background() + + // nil *error + done := MeasureOp(ctx, "TestOp") + if done == nil { + t.Fatal("MeasureOp returned a nil finish closure") + } + done(nil) + + // pointer to a nil error (the common success case: defer MeasureOp(...)(&err)) + var errNil error + MeasureOp(ctx, "TestOp")(&errNil) + + // pointer to a non-nil error (records an error) + errSet := errors.New("boom") + MeasureOp(ctx, "TestOp", nil...)(&errSet) + + // calling twice must not panic (idempotent finish is not required, but a + // second independent op must be safe) + MeasureOp(ctx, "AnotherOp")(nil) +} diff --git a/pkg/driver/controller.go b/pkg/driver/controller.go index b5423b3..6e46702 100755 --- a/pkg/driver/controller.go +++ b/pkg/driver/controller.go @@ -167,9 +167,56 @@ func parseVolParams(params map[string]string) (common.HSVolumeParameters, error) } } + // objectiveTarget controls where objectives are applied for file-backed + // volumes, and therefore whether CreateVolume pays for the per-file + // visibility poll (which exists only to gate the per-file objective-set): + // share (default) - objectives live on the backing SHARE only; skip the + // per-file objective-set and its Anvil visibility poll, + // so CreateVolume returns as soon as the local mkfs + // completes. Best for the common single-site shape. + // file / both - additionally apply per-file objectives (pays the + // poll). Use for per-volume / multi-site policy. + switch target := params["objectiveTarget"]; target { + case "", "share": + vParams.ObjectiveTarget = "share" + case "file", "both": + vParams.ObjectiveTarget = target + default: + return vParams, status.Errorf(codes.InvalidArgument, + "invalid objectiveTarget %q (must be one of: share, file, both)", target) + } + return vParams, nil } +// checkFileBackedMinSize rejects a file-backed volume whose requested size is +// below the per-fsType minimum (xfsprogs 6.4+ deprecates sub-300MiB XFS; ext4 +// below ~20MiB has almost no usable space). Non file-backed fsTypes and sizes +// at/above the floor return nil. +func checkFileBackedMinSize(fsType string, requestedSize int64) error { + const mib = 1024 * 1024 + switch fsType { + case "xfs": + if requestedSize < common.MinXfsSizeBytes { + return status.Errorf(codes.InvalidArgument, common.XfsSizeBelowMinimum, + common.MinXfsSizeBytes, common.MinXfsSizeBytes/mib, + requestedSize, requestedSize/mib) + } + case "ext4": + if requestedSize < common.MinExt4SizeBytes { + return status.Errorf(codes.InvalidArgument, common.Ext4SizeBelowMinimum, + common.MinExt4SizeBytes, common.MinExt4SizeBytes/mib, + requestedSize, requestedSize/mib) + } + case "ext3": + // ext3 is no longer a supported file-backed filesystem — reject it up + // front (rather than silently formatting it like ext4 with no size + // floor). Use ext4 or xfs. + return status.Error(codes.InvalidArgument, common.Ext3NotSupported) + } + return nil +} + func getMountFlagsFromCapabilities(capabilities []*csi.VolumeCapability) []string { for _, capability := range capabilities { if mount := capability.GetMount(); mount != nil { @@ -217,6 +264,11 @@ func (d *CSIDriver) ensureNFSDirectoryExists(ctx context.Context, backingShareNa } func (d *CSIDriver) ensureShareBackedVolumeExists(ctx context.Context, hsVolume *common.HSVolume) error { + ctx, span := tracer.Start(ctx, "ensureShareBackedVolumeExists", trace.WithAttributes( + attribute.String("volume.name", hsVolume.Name), + )) + defer span.End() + defer common.MeasureOp(ctx, "ensureShareBackedVolumeExists")(nil) // Check if the Mount Volume Exists share, err := d.hsclient.GetShare(ctx, hsVolume.Name) @@ -292,7 +344,7 @@ func (d *CSIDriver) ensureShareBackedVolumeExists(ctx context.Context, hsVolume targetPath := common.ShareStagingDir + "/metadata-mounts" + hsVolume.Path log.Debugf("Creating empty folder with path %s", targetPath) - defer common.UnmountFilesystem(targetPath) + defer common.UnmountFilesystem(ctx, targetPath) log.Debugf("Created empty folder with path %s", targetPath) err = d.publishShareBackedVolume(ctx, hsVolume.Path, targetPath, hsVolume.MountFlags, hsVolume.FQDN) @@ -302,7 +354,7 @@ func (d *CSIDriver) ensureShareBackedVolumeExists(ctx context.Context, hsVolume log.Debugf("Published share backed volume %s on targetpath %s", hsVolume.Path, targetPath) // The hs client expects a trailing slash for directories - err = common.SetMetadataTags(targetPath+"/", hsVolume.AdditionalMetadataTags) + err = common.SetMetadataTags(ctx, targetPath+"/", hsVolume.AdditionalMetadataTags) if err != nil { log.Warnf("failed to set additional metadata on share %v", err) } @@ -312,6 +364,12 @@ func (d *CSIDriver) ensureShareBackedVolumeExists(ctx context.Context, hsVolume } func (d *CSIDriver) ensureBackingShareExists(ctx context.Context, backingShareName string, hsVolume *common.HSVolume) (*common.ShareResponse, error) { + ctx, span := tracer.Start(ctx, "ensureBackingShareExists", trace.WithAttributes( + attribute.String("backing_share", backingShareName), + )) + defer span.End() + defer common.MeasureOp(ctx, "ensureBackingShareExists")(nil) + share, err := d.hsclient.GetShare(ctx, backingShareName) if err != nil { return nil, status.Errorf(codes.Internal, "%s", err.Error()) @@ -341,12 +399,12 @@ func (d *CSIDriver) ensureBackingShareExists(ctx context.Context, backingShareNa } // generate unique target path on host for setting file metadata targetPath := common.ShareStagingDir + "/metadata-mounts" + hsVolume.Path - defer common.UnmountFilesystem(targetPath) + defer common.UnmountFilesystem(ctx, targetPath) err = d.publishShareBackedVolume(ctx, hsVolume.Path, targetPath, hsVolume.MountFlags, hsVolume.FQDN) if err != nil { log.Warnf("failed to get share backed volume on hsVolumePath %s targetPath %s. Err %v", hsVolume.Path, targetPath, err) } - err = common.SetMetadataTags(targetPath+"/", hsVolume.AdditionalMetadataTags) + err = common.SetMetadataTags(ctx, targetPath+"/", hsVolume.AdditionalMetadataTags) if err != nil { log.Warnf("failed to set additional metadata on share %v", err) } @@ -401,18 +459,21 @@ func (d *CSIDriver) ensureDeviceFileExists(ctx context.Context, backingShare *co return status.Error(codes.NotFound, common.UnknownError) } } else { - // Create empty file - defer d.UnmountBackingShareIfUnused(ctx, backingShare.Name) - - err = d.EnsureBackingShareMounted(ctx, backingShare.Name, hsVolume) + // Create empty file. Take a refcounted reference on the backing mount so it + // stays mounted for the duration of this create but is NOT held under the + // per-backing-share lock; the mkfs below therefore runs concurrently with + // other creates on the same share. The share is unmounted only once the last + // in-flight create releases (see acquire/releaseBackingMount). + err = d.acquireBackingMount(ctx, backingShare, hsVolume) if err != nil { log.Errorf("failed to ensure backing share is mounted, %v", err) return err } + defer d.releaseBackingMount(ctx, backingShare) log.Debugf("ensureDeviceFileExists mounted backing share %s", backingShare.Name) - err = common.MakeEmptyRawFile(deviceFile, hsVolume.Size) + err = common.MakeEmptyRawFile(ctx, deviceFile, hsVolume.Size) if err != nil { log.Errorf("failed to create backing file for volume, %v", err) return err @@ -421,7 +482,7 @@ func (d *CSIDriver) ensureDeviceFileExists(ctx context.Context, backingShare *co // Add filesystem log.Debugf("ensureDeviceFileExists created empty raw file over backing share %s and path %s", backingShare.Name, deviceFile) if hsVolume.FSType != "" { - err = common.FormatDevice(deviceFile, hsVolume.FSType) + err = common.FormatDevice(ctx, deviceFile, hsVolume.FSType) if err != nil { log.Errorf("failed to format volume, %v", err) return err @@ -430,13 +491,31 @@ func (d *CSIDriver) ensureDeviceFileExists(ctx context.Context, backingShare *co log.Debugf("ensureDeviceFileExists formatted file %s, with fstype %s", deviceFile, hsVolume.FSType) } - // Step 4: Use a fresh context to apply metadata - metadataCtx, cancel := context.WithTimeout(context.Background(), 10*time.Minute) + // Step 4: Apply objectives + metadata on a fresh deadline, but inherit + // trace context so spans stay attached to the CreateVolume trace. + // context.WithoutCancel detaches from the gRPC handler's cancellation + // (which would otherwise kill the long poll loop) while preserving the + // OTel span context attached via tracer.Start above. + metadataCtx, cancel := context.WithTimeout(context.WithoutCancel(ctx), 10*time.Minute) defer cancel() - err = d.applyObjectiveAndMetadata(metadataCtx, backingShare, hsVolume, deviceFile) - if err != nil { - log.Warnf("Unable to apply objective and metadata over backing share %s, device path %s: %v", backingShare.Name, deviceFile, err) + // objectiveTarget=share (default): the backing share already carries the + // objectives, so we skip the per-file objective-set AND the Anvil + // visibility poll that exists only to gate it. CreateVolume then returns + // as soon as the local mkfs above completes (seconds, not tens of seconds), + // and we avoid the GET /files 500-storm the poll generates under load. + // Metadata tags still apply - they operate on the freshly-created local + // file over the mount and need no Anvil round-trip. + if hsVolume.ObjectiveTarget == "file" || hsVolume.ObjectiveTarget == "both" { + err = d.applyObjectiveAndMetadata(metadataCtx, backingShare, hsVolume, deviceFile) + if err != nil { + log.Warnf("Unable to apply objective and metadata over backing share %s, device path %s: %v", backingShare.Name, deviceFile, err) + } + } else { + log.Debugf("objectiveTarget=share: skipping per-file objective+visibility poll for %s", hsVolume.Path) + if err := common.SetMetadataTags(metadataCtx, deviceFile, hsVolume.AdditionalMetadataTags); err != nil { + log.Warnf("Failed to set additional metadata on backing file for volume %s: %v", deviceFile, err) + } } return nil @@ -444,20 +523,35 @@ func (d *CSIDriver) ensureDeviceFileExists(ctx context.Context, backingShare *co // ensure from hs system /share/file exist to apply objective and metadata func (d *CSIDriver) applyObjectiveAndMetadata(ctx context.Context, backingShare *common.ShareResponse, hsVolume *common.HSVolume, deviceFile string) error { + ctx, span := tracer.Start(ctx, "applyObjectiveAndMetadata", trace.WithAttributes( + attribute.String("backing_share", backingShare.Name), + attribute.String("path", hsVolume.Path), + )) + defer span.End() + + // Poll Anvil's metadata API until the backing file we just created over + // NFS becomes visible to the management plane. The loop is the dominant + // cost of CreateVolume in our traces (often tens of seconds), so it gets + // its own span and per-attempt count. + pollCtx, pollSpan := tracer.Start(ctx, "applyObjectiveAndMetadata.waitForFileVisible", trace.WithAttributes( + attribute.String("path", hsVolume.Path), + )) b := &backoff.Backoff{ - Max: 5 * time.Second, + Max: 1 * time.Second, Factor: 1.5, Jitter: true, } startTime := time.Now() var backingFileExists bool var err error + attempts := 0 for time.Since(startTime) < (10 * time.Minute) { dur := b.Duration() time.Sleep(dur) + attempts++ // Wait for file to exist on metadata server log.Debugf("Checking existance of file %s", hsVolume.Path) - backingFileExists, err = d.hsclient.DoesFileExist(ctx, hsVolume.Path) + backingFileExists, err = d.hsclient.DoesFileExist(pollCtx, hsVolume.Path) if err != nil { log.Warnf("Error checking file existence: %v", err) time.Sleep(time.Second) @@ -469,6 +563,11 @@ func (d *CSIDriver) applyObjectiveAndMetadata(ctx context.Context, backingShare } log.Warnf("File does not exist yet: %s", hsVolume.Path) } + pollSpan.SetAttributes( + attribute.Int("attempts", attempts), + attribute.Bool("file_visible", backingFileExists), + ) + pollSpan.End() if !backingFileExists { log.Errorf("backing file failed to show up in API after 10 minutes") @@ -485,7 +584,7 @@ func (d *CSIDriver) applyObjectiveAndMetadata(ctx context.Context, backingShare } // Set additional metadata on file - err = common.SetMetadataTags(deviceFile, hsVolume.AdditionalMetadataTags) + err = common.SetMetadataTags(ctx, deviceFile, hsVolume.AdditionalMetadataTags) if err != nil { log.Errorf("Failed to set additional metadata on backing file for volume: %v\n", err) } @@ -498,31 +597,35 @@ func (d *CSIDriver) ensureFileBackedVolumeExists(ctx context.Context, hsVolume * "backingShareName": backingShareName, "hsVolume": hsVolume, }).Debugf("ensureFileBackedVolumeExists is called.") - // Check if backing share exists - // Acquire BEFORE defer; with timeout so we never hang forever + // The backing share is a shared resource: two concurrent first-volume creates + // must not both CreateShare it. Serialize ONLY the share create-if-not-exists + // under the per-backing-share lock, then release it immediately. The per-volume + // device file created afterwards is independent, so releasing the lock here lets + // file creation (mkfs) run concurrently across the provisioner worker threads + // instead of serializing every file on the share behind this one lock. The + // backing mount is kept alive for the duration by acquire/releaseBackingMount. unlock, err := d.acquireVolumeLock(ctx, backingShareName) if err != nil { // surfaces to kubelet instead of hanging forever return err } - defer unlock() - backingShare, err := d.ensureBackingShareExists(ctx, backingShareName, hsVolume) + unlock() if err != nil { return status.Errorf(codes.Internal, "%s", err.Error()) } log.Debugf("Backing share existed %s", backingShareName) - err = d.ensureDeviceFileExists(ctx, backingShare, hsVolume) - return err + return d.ensureDeviceFileExists(ctx, backingShare, hsVolume) } -func (d *CSIDriver) CreateVolume(ctx context.Context, req *csi.CreateVolumeRequest) (*csi.CreateVolumeResponse, error) { +func (d *CSIDriver) CreateVolume(ctx context.Context, req *csi.CreateVolumeRequest) (_ *csi.CreateVolumeResponse, err error) { // Start a span for tracing ctx, span := tracer.Start(ctx, "Controller/CreateVolume", trace.WithAttributes( attribute.String("volume_name", req.Name), )) defer span.End() + defer common.MeasureOp(ctx, "Controller/CreateVolume")(&err) startTime := time.Now() // Validate Parameters @@ -593,6 +696,15 @@ func (d *CSIDriver) CreateVolume(ctx context.Context, req *csi.CreateVolumeReque requestedSize = common.DefaultBackingFileSizeBytes } + // Reject file-backed volumes below the per-fsType minimum, failing fast with + // codes.InvalidArgument so kubelet surfaces the reason instead of silently + // formatting a broken FS. See checkFileBackedMinSize / the config.go floors. + if fileBacked { + if err := checkFileBackedMinSize(fsType, requestedSize); err != nil { + return nil, err + } + } + var backingShareName string if blockRequested { backingShareName = vParams.BlockBackingShareName @@ -624,6 +736,7 @@ func (d *CSIDriver) CreateVolume(ctx context.Context, req *csi.CreateVolumeReque Comment: vParams.Comment, FQDN: vParams.FQDN, MountFlags: getMountFlagsFromCapabilities(req.VolumeCapabilities), + ObjectiveTarget: vParams.ObjectiveTarget, } // if it's file backed, we should check capacity of backing share @@ -785,6 +898,10 @@ func (d *CSIDriver) CreateVolume(ctx context.Context, req *csi.CreateVolumeReque } func (d *CSIDriver) deleteFileBackedVolume(ctx context.Context, filepath string) error { + ctx, span := tracer.Start(ctx, "deleteFileBackedVolume", trace.WithAttributes( + attribute.String("file.path", filepath), + )) + defer span.End() var exists bool if exists, _ = d.hsclient.DoesFileExist(ctx, filepath); exists { log.Debugf("found file-backed volume to delete, %s", filepath) @@ -814,13 +931,25 @@ func (d *CSIDriver) deleteFileBackedVolume(ctx context.Context, filepath string) return err } defer unlock() - // mount the share to delete the file - defer d.UnmountBackingShareIfUnused(ctx, residingShareName) - err = d.EnsureBackingShareMounted(ctx, residingShareName, hsVolume) // check if share is mounted - if err != nil { + // Route the mount through acquire/releaseBackingMount — the same refcounted + // mechanism the create path uses — instead of a bare EnsureBackingShareMounted + // plus a deferred UnmountBackingShareIfUnused. This makes delete a first-class + // participant in the backing-mount refcount: it holds a reference for the whole + // delete (so a concurrent create that has already taken its own reference can't + // have the share unmounted out from under it), and it serializes its mount and + // unmount under the same per-directory lock (mountLockFor) as create rather than + // mutating the mount with no shared lock held. GetShare gives us the ShareResponse + // acquireBackingMount needs; the file is known to exist here, so the share does too. + backingShare, err := d.hsclient.GetShare(ctx, residingShareName) + if err != nil || backingShare == nil { + log.Errorf("failed to get backing share %s while deleting file-backed volume: %v", residingShareName, err) + return status.Errorf(codes.Internal, "unable to get backing share %s: %v", residingShareName, err) + } + if err = d.acquireBackingMount(ctx, backingShare, hsVolume); err != nil { log.Errorf("failed to ensure backing share is mounted, %v", err) return status.Errorf(codes.Internal, "%s", err.Error()) } + defer d.releaseBackingMount(ctx, backingShare) // Delete File volumeName := GetVolumeNameFromPath(filepath) err = common.DeleteFile(destination + "/" + volumeName) @@ -857,12 +986,13 @@ func (d *CSIDriver) deleteShareBackedVolume(ctx context.Context, share *common.S return nil } -func (d *CSIDriver) DeleteVolume(ctx context.Context, req *csi.DeleteVolumeRequest) (*csi.DeleteVolumeResponse, error) { +func (d *CSIDriver) DeleteVolume(ctx context.Context, req *csi.DeleteVolumeRequest) (_ *csi.DeleteVolumeResponse, err error) { // Start a span for tracing ctx, span := tracer.Start(ctx, "Controller/DeleteVolume", trace.WithAttributes( attribute.String("volume.id", req.GetVolumeId()), )) defer span.End() + defer common.MeasureOp(ctx, "Controller/DeleteVolume")(&err) volumeId := req.GetVolumeId() log.Infof("Delete volume request for volume id, %s", volumeId) @@ -878,20 +1008,27 @@ func (d *CSIDriver) DeleteVolume(ctx context.Context, req *csi.DeleteVolumeReque } defer unlock() + // A file-backed volume lives *inside* a backing share, so its volume ID is + // structurally distinguishable from a share-backed one. Decide from the ID + // instead of probing Anvil with a GetShare that, for file-backed volumes, + // always 404s. + if isFileBackedVolumeID(volumeId) { + err = d.deleteFileBackedVolume(ctx, volumeId) + return &csi.DeleteVolumeResponse{}, err + } + volumeName := GetVolumeNameFromPath(volumeId) share, err := d.hsclient.GetShare(ctx, volumeName) if err != nil { return nil, status.Errorf(codes.Internal, "%s", err.Error()) } - if share == nil { // Share does not exist, may be a file-backed volume + if share == nil { // legacy/single-segment ID with no share: treat as file-backed err = d.deleteFileBackedVolume(ctx, volumeId) - - return &csi.DeleteVolumeResponse{}, err - } else { // Share exists and is a Filesystem - err = d.deleteShareBackedVolume(ctx, share) return &csi.DeleteVolumeResponse{}, err } - + // Share exists and is a Filesystem + err = d.deleteShareBackedVolume(ctx, share) + return &csi.DeleteVolumeResponse{}, err } // ControllerGetVolume implements the ControllerServer interface for CSI. @@ -928,27 +1065,26 @@ func (d *CSIDriver) ControllerExpandVolume(ctx context.Context, req *csi.Control )) defer span.End() - fileBacked := false - if req.GetVolumeId() == "" { return nil, status.Error(codes.InvalidArgument, common.VolumeNotFound) } - volumeName := GetVolumeNameFromPath(req.GetVolumeId()) - share, _ := d.hsclient.GetShare(ctx, volumeName) - if share == nil { - fileBacked = true - } - - // Check if the specified backing share or file exists - if share == nil { - backingFileExists, err := d.hsclient.DoesFileExist(ctx, req.GetVolumeId()) - if err != nil { - log.Error(err) - } - if !backingFileExists { - return nil, status.Error(codes.NotFound, common.VolumeNotFound) - } else { + // Decide file-backed vs share-backed structurally from the volume ID, avoiding + // a GetShare probe that always 404s for file-backed volumes. The branches below + // still confirm the volume actually exists (GetFile / GetShare). + fileBacked := isFileBackedVolumeID(req.GetVolumeId()) + if !fileBacked { + volumeName := GetVolumeNameFromPath(req.GetVolumeId()) + share, _ := d.hsclient.GetShare(ctx, volumeName) + if share == nil { + // Fallback for legacy/unexpected IDs: confirm via file existence. + backingFileExists, ferr := d.hsclient.DoesFileExist(ctx, req.GetVolumeId()) + if ferr != nil { + log.Error(ferr) + } + if !backingFileExists { + return nil, status.Error(codes.NotFound, common.VolumeNotFound) + } fileBacked = true } } @@ -1261,13 +1397,14 @@ func (d *CSIDriver) ControllerGetCapabilities(ctx context.Context, req *csi.Cont }, nil } -func (d *CSIDriver) CreateSnapshot(ctx context.Context, req *csi.CreateSnapshotRequest) (*csi.CreateSnapshotResponse, error) { +func (d *CSIDriver) CreateSnapshot(ctx context.Context, req *csi.CreateSnapshotRequest) (_ *csi.CreateSnapshotResponse, err error) { // Start a span for tracing ctx, span := tracer.Start(ctx, "Controller/CreateSnapshot", trace.WithAttributes( attribute.String("snapshot.name", req.GetName()), attribute.String("source.volume.id", req.GetSourceVolumeId()), )) defer span.End() + defer common.MeasureOp(ctx, "Controller/CreateSnapshot")(&err) // Check arguments log.WithFields(log.Fields{ @@ -1299,17 +1436,51 @@ func (d *CSIDriver) CreateSnapshot(ctx context.Context, req *csi.CreateSnapshotR if _, exists := recentlyCreatedSnapshots[req.GetName()]; !exists { // find source volume (is it file or share? volumeName := GetVolumeNameFromPath(req.GetSourceVolumeId()) - share, err := d.hsclient.GetShare(ctx, volumeName) - if err != nil { - return nil, status.Errorf(codes.Internal, "%s", err.Error()) + // Decide file- vs share-backed structurally from the source volume ID, + // avoiding a GetShare probe that 404s for file-backed sources; fall back to + // GetShare only for a non-file-backed ID (legacy/unexpected handles). + fileBackedSource := isFileBackedVolumeID(req.GetSourceVolumeId()) + if !fileBackedSource { + share, gerr := d.hsclient.GetShare(ctx, volumeName) + if gerr != nil { + return nil, status.Errorf(codes.Internal, "%s", gerr.Error()) + } + if share == nil { + fileBackedSource = true + } + } + // Consistency-freeze: for FILE-BACKED source volumes only, locate + // every pod that has this volume mounted and issue + // `fsfreeze --freeze` on the mount path. This forces XFS/ext4 to + // quiesce (log flushed, no in-flight transactions) so Anvil's + // byte-level snapshot captures a clean on-disk state. If the pod + // is missing fsfreeze, or we can't find pods for this volume, we + // log and proceed — matching the best-effort semantics of Velero + // pre-hooks. See freezer.go. + // + // A share-backed (NFS) volume has Anvil owning the filesystem + // end-to-end, with no local journal to quiesce; FIFREEZE also fails + // EOPNOTSUPP on NFS mounts. Skip the freeze in that case. + var frozen []FrozenTarget + if d.freezer != nil && fileBackedSource { + frozen = d.freezer.FreezeForVolumeHandle(ctx, req.GetSourceVolumeId()) } // Create the snapshot var hsSnapName string - if share != nil { + if !fileBackedSource { hsSnapName, err = d.hsclient.SnapshotShare(ctx, volumeName) } else { hsSnapName, err = d.hsclient.SnapshotFile(ctx, req.GetSourceVolumeId()) } + // Always unfreeze, even if snapshot failed — otherwise the app pod + // stays blocked on writes indefinitely. Use a context detached from the + // gRPC request cancellation (context.WithoutCancel): if the snapshotter + // sidecar's deadline expires while SnapshotFile/SnapshotShare above is + // still running, a cancelled ctx would make the unfreeze exec fail fast + // without ever reaching the pod, leaving the workload's filesystem frozen. + if d.freezer != nil && fileBackedSource { + d.freezer.Unfreeze(context.WithoutCancel(ctx), frozen) + } if err != nil { return nil, status.Errorf(codes.Internal, "%s", err.Error()) } @@ -1359,13 +1530,31 @@ func (d *CSIDriver) DeleteSnapshot(ctx context.Context, req *csi.DeleteSnapshotR // If the snapshot does not exist then return an idempotent response. + // File-vs-share discriminator, decided structurally from the source volume + // path (the "|"-suffix of the snapshot ID): a file-backed volume is a FILE + // inside a backing share (multi-segment path -> file snapshot); a native NFS + // volume IS a share (single-segment path -> share snapshot). This avoids a + // GetShare probe that 404s for every file-backed snapshot; GetShare is kept + // as a fallback for non-file-backed paths. + // (Historical: the original `GetVolumeNameFromPath(path) != ""` test was + // ALWAYS true, so every delete was routed to DeleteShareSnapshot and + // file-backed snapshots were orphaned on the Anvil, blocking source-volume + // deletion.) shareName := GetVolumeNameFromPath(path) var err error - if shareName != "" { - err = d.hsclient.DeleteShareSnapshot(ctx, shareName, snapshotName) - } else { + if isFileBackedVolumeID(path) { err = d.hsclient.DeleteFileSnapshot(ctx, path, snapshotName) + } else { + share, gerr := d.hsclient.GetShare(ctx, shareName) + if gerr != nil { + return nil, status.Error(codes.Internal, gerr.Error()) + } + if share != nil { + err = d.hsclient.DeleteShareSnapshot(ctx, shareName, snapshotName) + } else { + err = d.hsclient.DeleteFileSnapshot(ctx, path, snapshotName) + } } if err != nil { diff --git a/pkg/driver/controller_test.go b/pkg/driver/controller_test.go index 55fe700..1a77501 100644 --- a/pkg/driver/controller_test.go +++ b/pkg/driver/controller_test.go @@ -6,6 +6,8 @@ import ( "github.com/container-storage-interface/spec/lib/go/csi" common "github.com/hammer-space/csi-plugin/pkg/common" + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" ) func TestParseParams(t *testing.T) { @@ -15,6 +17,7 @@ func TestParseParams(t *testing.T) { VolumeNameFormat: common.DefaultVolumeNameFormat, DeleteDelay: -1, Comment: "Created by CSI driver", + ObjectiveTarget: "share", } stringParams := map[string]string{} actualParams, _ := parseVolParams(stringParams) @@ -30,6 +33,7 @@ func TestParseParams(t *testing.T) { VolumeNameFormat: "my-csi-volume-%s-hammerspace", DeleteDelay: -1, Comment: "Created by CSI driver", + ObjectiveTarget: "share", } stringParams = map[string]string{ "volumeNameFormat": "my-csi-volume-%s-hammerspace", @@ -68,6 +72,7 @@ func TestParseParams(t *testing.T) { DeleteDelay: 30, VolumeNameFormat: common.DefaultVolumeNameFormat, Comment: "Created by CSI driver", + ObjectiveTarget: "share", } stringParams = map[string]string{ "deleteDelay": "30", @@ -214,3 +219,89 @@ func TestGetMountFlagsFromCapabilities(t *testing.T) { t.FailNow() } } + +// TestParseObjectiveTarget covers the objectiveTarget StorageClass parameter +// added for the fast file-backed CreateVolume path (PR I): default "share", +// explicit share/file/both, and rejection of anything else. +func TestParseObjectiveTarget(t *testing.T) { + cases := []struct { + name string + in string // "" means the param is omitted entirely + want string + wantErr bool + }{ + {"default when omitted", "", "share", false}, + {"explicit share", "share", "share", false}, + {"file", "file", "file", false}, + {"both", "both", "both", false}, + {"invalid value", "bogus", "", true}, + {"case-sensitive (Share is invalid)", "Share", "", true}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + params := map[string]string{} + if tc.in != "" { + params["objectiveTarget"] = tc.in + } + got, err := parseVolParams(params) + if tc.wantErr { + if err == nil { + t.Fatalf("objectiveTarget=%q: expected error, got none", tc.in) + } + if status.Code(err) != codes.InvalidArgument { + t.Fatalf("objectiveTarget=%q: expected InvalidArgument, got %v", tc.in, status.Code(err)) + } + return + } + if err != nil { + t.Fatalf("objectiveTarget=%q: unexpected error: %v", tc.in, err) + } + if got.ObjectiveTarget != tc.want { + t.Fatalf("objectiveTarget=%q: got %q, want %q", tc.in, got.ObjectiveTarget, tc.want) + } + }) + } +} + +// TestCheckFileBackedMinSize covers the per-fsType minimum size gate (PR B): +// xfs < 300 MiB and ext4 < 20 MiB are rejected with InvalidArgument; at/above +// the floor and other fsTypes pass. +func TestCheckFileBackedMinSize(t *testing.T) { + const mib = 1024 * 1024 + cases := []struct { + name string + fsType string + size int64 + wantErr bool + }{ + {"xfs below floor", "xfs", 299 * mib, true}, + {"xfs at floor", "xfs", common.MinXfsSizeBytes, false}, + {"xfs above floor", "xfs", 1024 * mib, false}, + {"ext4 below floor", "ext4", 19 * mib, true}, + {"ext4 at floor", "ext4", common.MinExt4SizeBytes, false}, + {"ext4 above floor", "ext4", 100 * mib, false}, + {"ext4 tiny", "ext4", 1, true}, + {"ext3 rejected large", "ext3", 100 * mib, true}, // ext3 unsupported at any size + {"ext3 rejected small", "ext3", 1, true}, // ext3 unsupported at any size + {"ext3 rejected at floor", "ext3", 20 * mib, true}, // ext3 unsupported even at/above the ext4 floor + {"other fsType not gated", "btrfs", 1, false}, + {"empty fsType not gated", "", 1, false}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + err := checkFileBackedMinSize(tc.fsType, tc.size) + if tc.wantErr { + if err == nil { + t.Fatalf("%s %d: expected error, got nil", tc.fsType, tc.size) + } + if status.Code(err) != codes.InvalidArgument { + t.Fatalf("%s %d: expected InvalidArgument, got %v", tc.fsType, tc.size, status.Code(err)) + } + return + } + if err != nil { + t.Fatalf("%s %d: unexpected error: %v", tc.fsType, tc.size, err) + } + }) + } +} diff --git a/pkg/driver/driver.go b/pkg/driver/driver.go index 8238cd3..64943a9 100644 --- a/pkg/driver/driver.go +++ b/pkg/driver/driver.go @@ -22,7 +22,6 @@ import ( "fmt" "net" "os" - "runtime/debug" "strconv" "sync" "time" @@ -35,8 +34,10 @@ import ( "github.com/container-storage-interface/spec/lib/go/csi" client "github.com/hammer-space/csi-plugin/pkg/client" "google.golang.org/grpc" + "google.golang.org/grpc/codes" "google.golang.org/grpc/keepalive" "google.golang.org/grpc/reflection" + "google.golang.org/grpc/status" ) type CSIDriver struct { @@ -50,8 +51,29 @@ type CSIDriver struct { locksMu sync.Mutex volumeLocks map[string]*keyLock snapshotLocks map[string]*keyLock - hsclient *client.HammerspaceClient - NodeID string + // mountRefs counts in-flight file operations per backing-share mount. It lets + // many file-backed CreateVolume calls share a single backing-share mount and run + // their per-file mkfs concurrently: the share is mounted on the first reference + // and unmounted only when the last in-flight file operation completes. This + // replaces holding the coarse per-backing-share lock across the whole per-file + // create (which serialized all file creation on a share to ~1 at a time). + mountRefsMu sync.Mutex + mountRefs map[string]int + // mountLocks holds one lock per backing-share staging directory. It serializes + // the actual mount/unmount of a given backing share so the slow (up to ~5 min) + // NFS mount never runs under the global mountRefsMu. mountRefsMu then guards only + // the two maps below and is always held for microseconds; a slow mount on one + // share can no longer wedge refcount reads (backingMountInUse) or releases on + // every other in-flight file-backed operation. See acquireBackingMount. + mountLocks map[string]*sync.Mutex + hsclient *client.HammerspaceClient + NodeID string + // freezer runs fsfreeze inside the pod(s) holding a source volume + // during CreateSnapshot, so XFS reaches a quiesce point before Anvil + // captures the file bytes. Nil when the driver is not running + // in-cluster (local dev) — in that case snapshots are still taken but + // consistency is not enforced. + freezer *Freezer } func NewCSIDriver(endpoint, username, password, tlsVerifyStr string) *CSIDriver { @@ -73,7 +95,10 @@ func NewCSIDriver(endpoint, username, password, tlsVerifyStr string) *CSIDriver hsclient: client, volumeLocks: make(map[string]*keyLock), snapshotLocks: make(map[string]*keyLock), + mountRefs: make(map[string]int), + mountLocks: make(map[string]*sync.Mutex), NodeID: os.Getenv("CSI_NODE_NAME"), + freezer: NewFreezer(), } } @@ -105,15 +130,17 @@ func (c *CSIDriver) acquireVolumeLock(ctx context.Context, volID string) (func() } c.locksMu.Unlock() + probe := common.StartLockProbe(ctx, "volume") lctx, cancel := context.WithTimeout(ctx, 30*time.Second) defer cancel() if err := lk.lock(lctx); err != nil { + probe.Failed() log.WithError(err).Errorf("Error acquiring volume lock for %s", volID) - debug.PrintStack() - os.Exit(1) + return nil, status.Errorf(codes.Aborted, "could not acquire volume lock for %s: %v", volID, err) } - return func() { lk.unlock() }, nil + release := probe.Acquired() + return func() { lk.unlock(); release() }, nil } func (c *CSIDriver) acquireSnapshotLock(ctx context.Context, snapID string) (func(), error) { @@ -126,15 +153,17 @@ func (c *CSIDriver) acquireSnapshotLock(ctx context.Context, snapID string) (fun } c.locksMu.Unlock() + probe := common.StartLockProbe(ctx, "snapshot") lctx, cancel := context.WithTimeout(ctx, 30*time.Second) defer cancel() if err := lk.lock(lctx); err != nil { + probe.Failed() log.WithError(err).Errorf("Error acquiring snapshot lock for %s", snapID) - debug.PrintStack() - os.Exit(1) + return nil, status.Errorf(codes.Aborted, "could not acquire snapshot lock for %s: %v", snapID, err) } - return func() { lk.unlock() }, nil + release := probe.Acquired() + return func() { lk.unlock(); release() }, nil } func (c *CSIDriver) goServe(started chan<- bool) { diff --git a/pkg/driver/driver_csi_v0_test.go b/pkg/driver/driver_csi_v0_test.go index 1ecf624..bf8e869 100644 --- a/pkg/driver/driver_csi_v0_test.go +++ b/pkg/driver/driver_csi_v0_test.go @@ -1,73 +1,73 @@ package driver import ( - "fmt" - csi_v0 "github.com/ameade/spec/lib/go/csi/v0" - "github.com/container-storage-interface/spec/lib/go/csi" - "github.com/hammer-space/csi-plugin/pkg/common" - "reflect" - "strings" - "testing" + "fmt" + csi_v0 "github.com/ameade/spec/lib/go/csi/v0" + "github.com/container-storage-interface/spec/lib/go/csi" + "github.com/hammer-space/csi-plugin/pkg/common" + "reflect" + "strings" + "testing" ) func TestConvertVolumeCapablityfromv0tov1(t *testing.T) { - // Test basic conversion - capv0 := &csi_v0.VolumeCapability{ - AccessType: &csi_v0.VolumeCapability_Mount{ - Mount: &csi_v0.VolumeCapability_MountVolume{ - FsType: "NFS", - MountFlags: []string{"nfsvers=4.2"}, - }, - }, - AccessMode: &csi_v0.VolumeCapability_AccessMode{ - Mode: csi_v0.VolumeCapability_AccessMode_MULTI_NODE_MULTI_WRITER, - }, - } + // Test basic conversion + capv0 := &csi_v0.VolumeCapability{ + AccessType: &csi_v0.VolumeCapability_Mount{ + Mount: &csi_v0.VolumeCapability_MountVolume{ + FsType: "NFS", + MountFlags: []string{"nfsvers=4.2"}, + }, + }, + AccessMode: &csi_v0.VolumeCapability_AccessMode{ + Mode: csi_v0.VolumeCapability_AccessMode_MULTI_NODE_MULTI_WRITER, + }, + } - capv1 := &csi.VolumeCapability{ - AccessType: &csi.VolumeCapability_Mount{ - Mount: &csi.VolumeCapability_MountVolume{ - FsType: "NFS", - MountFlags: []string{"nfsvers=4.2"}, - }, - }, - AccessMode: &csi.VolumeCapability_AccessMode{ - Mode: csi.VolumeCapability_AccessMode_MULTI_NODE_MULTI_WRITER, - }, - } + capv1 := &csi.VolumeCapability{ + AccessType: &csi.VolumeCapability_Mount{ + Mount: &csi.VolumeCapability_MountVolume{ + FsType: "NFS", + MountFlags: []string{"nfsvers=4.2"}, + }, + }, + AccessMode: &csi.VolumeCapability_AccessMode{ + Mode: csi.VolumeCapability_AccessMode_MULTI_NODE_MULTI_WRITER, + }, + } - actualcpv1, err := ConvertVolumeCapabilityFromv0Tov1(capv0) - if err != nil { - t.Logf("unexpected error") - t.FailNow() - } + actualcpv1, err := ConvertVolumeCapabilityFromv0Tov1(capv0) + if err != nil { + t.Logf("unexpected error") + t.FailNow() + } - if !reflect.DeepEqual(actualcpv1, capv1) { - t.Logf("Expected: %v", capv1) - t.Logf("Actual: %v", actualcpv1) - t.FailNow() - } + if !reflect.DeepEqual(actualcpv1, capv1) { + t.Logf("Expected: %v", capv1) + t.Logf("Actual: %v", actualcpv1) + t.FailNow() + } - // Test that Raw volumes are not supported - capv0 = &csi_v0.VolumeCapability{ - AccessType: &csi_v0.VolumeCapability_Block{ - Block: &csi_v0.VolumeCapability_BlockVolume{}, - }, - AccessMode: &csi_v0.VolumeCapability_AccessMode{ - Mode: csi_v0.VolumeCapability_AccessMode_MULTI_NODE_READER_ONLY, - }, - } + // Test that Raw volumes are not supported + capv0 = &csi_v0.VolumeCapability{ + AccessType: &csi_v0.VolumeCapability_Block{ + Block: &csi_v0.VolumeCapability_BlockVolume{}, + }, + AccessMode: &csi_v0.VolumeCapability_AccessMode{ + Mode: csi_v0.VolumeCapability_AccessMode_MULTI_NODE_READER_ONLY, + }, + } - _, err = ConvertVolumeCapabilityFromv0Tov1(capv0) - if err == nil { - t.Logf("expected error") - t.FailNow() - } else { - errString := fmt.Sprintf("%s", err) - if !strings.Contains(errString, common.BlockVolumesUnsupported) { - t.Logf("unexpected error, %s", err) - t.FailNow() - } - } + _, err = ConvertVolumeCapabilityFromv0Tov1(capv0) + if err == nil { + t.Logf("expected error") + t.FailNow() + } else { + errString := fmt.Sprintf("%s", err) + if !strings.Contains(errString, common.BlockVolumesUnsupported) { + t.Logf("unexpected error, %s", err) + t.FailNow() + } + } } diff --git a/pkg/driver/driver_csi_v1_test.go b/pkg/driver/driver_csi_v1_test.go index 071e921..8e95d64 100644 --- a/pkg/driver/driver_csi_v1_test.go +++ b/pkg/driver/driver_csi_v1_test.go @@ -4,6 +4,9 @@ import ( "context" "testing" "time" + + "google.golang.org/grpc/codes" + "google.golang.org/grpc/status" ) // TestAcquireAndReleaseVolumeLock ensures a lock can be acquired and released. @@ -57,6 +60,13 @@ func TestAcquireVolumeLockTimeout(t *testing.T) { if err == nil { t.Fatalf("expected timeout error but got none") } + // PR A: a lock-acquire timeout must return codes.Aborted (retryable) rather + // than calling os.Exit(1), which previously crashed the whole controller + // under concurrent load. If this regresses to os.Exit, the test binary dies + // here and the failure is unmistakable. + if status.Code(err) != codes.Aborted { + t.Fatalf("expected codes.Aborted on lock timeout, got %v (err: %v)", status.Code(err), err) + } if elapsed < 250*time.Millisecond { t.Fatalf("expected blocking for ~300ms, got only %v", elapsed) } diff --git a/pkg/driver/freezer.go b/pkg/driver/freezer.go new file mode 100644 index 0000000..545a10f --- /dev/null +++ b/pkg/driver/freezer.go @@ -0,0 +1,274 @@ +/* +Copyright 2019 Hammerspace +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +*/ + +// Freezer implements the workload-quiesce step for consistent snapshots. +// +// When the driver receives a CreateSnapshot RPC for a file-backed volume, the +// bytes that the Anvil sees are whatever has already been flushed by the +// worker's NFS client. In-flight XFS pagecache, uncommitted XFS log +// transactions, and NFS write-back all live on the worker and can silently +// leave the on-disk state inconsistent at snapshot time. Loop-remount of that +// file on restore runs XFS log recovery, which for a dirty log can roll back +// legitimate transactions and produce an empty filesystem. +// +// The fix is to freeze the filesystem inside the pod that holds it open, +// so XFS quiesces (log flushed, no in-flight transactions) before Anvil takes +// the snapshot. This is the same approach Velero pre-hooks / Kanister +// blueprints take, but done inside the driver so the user doesn't have to +// annotate anything. +package driver + +import ( + "bytes" + "context" + "fmt" + + log "github.com/sirupsen/logrus" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes" + "k8s.io/client-go/kubernetes/scheme" + "k8s.io/client-go/rest" + "k8s.io/client-go/tools/remotecommand" +) + +// FrozenTarget describes one csi-node pod + node-side mount path that was +// frozen and needs to be unfrozen. We deliberately do NOT exec inside the +// user's app pod because we can't assume its image ships `fsfreeze` +// (nginx-alpine, distroless, scratch, etc. don't). Instead we find the +// csi-node DaemonSet pod on the same node — the driver's own image is +// guaranteed to have `fsfreeze` from util-linux — and freeze the mount via +// its shared /var/lib/kubelet propagation. +type FrozenTarget struct { + Namespace string // kube-system, where csi-node lives + PodName string // csi-node-XXXXX + Container string // hs-csi-plugin-node + MountPath string // /var/lib/kubelet/pods//volumes/kubernetes.io~csi//mount + // For diagnostics only: + UserPodNs string + UserPodName string +} + +// Freezer holds the kube client + REST config needed to exec into pods. +type Freezer struct { + clientset *kubernetes.Clientset + restCfg *rest.Config +} + +// NewFreezer builds a Freezer from the pod's in-cluster credentials. If the +// driver isn't running in-cluster (e.g. local dev), returns nil so callers +// can no-op through. +func NewFreezer() *Freezer { + cfg, err := rest.InClusterConfig() + if err != nil { + log.Warnf("Freezer: not running in-cluster (%v); consistency-freeze disabled", err) + return nil + } + cs, err := kubernetes.NewForConfig(cfg) + if err != nil { + log.Warnf("Freezer: kubernetes.NewForConfig failed (%v); consistency-freeze disabled", err) + return nil + } + return &Freezer{clientset: cs, restCfg: cfg} +} + +// FreezeForVolumeHandle locates every running Pod that has the CSI volume +// (identified by volumeHandle) mounted, then execs `fsfreeze --freeze ` +// inside the first container of each such Pod. Returns the list of targets +// that were successfully frozen — pass those to Unfreeze after the snapshot. +// +// Failure to freeze is best-effort by design: the snapshot proceeds even if +// freeze fails on some/all pods, matching Velero's default behavior. Callers +// should log but not fail on freeze errors. +func (f *Freezer) FreezeForVolumeHandle(ctx context.Context, volumeHandle string) []FrozenTarget { + if f == nil { + return nil + } + targets, err := f.findMountsForVolumeHandle(ctx, volumeHandle) + if err != nil { + log.Warnf("Freezer: findMountsForVolumeHandle(%s) failed: %v", volumeHandle, err) + return nil + } + if len(targets) == 0 { + log.Infof("Freezer: no running pods hold volumeHandle=%s; nothing to freeze", volumeHandle) + return nil + } + var frozen []FrozenTarget + for _, t := range targets { + if err := f.execFsfreeze(ctx, t, "--freeze"); err != nil { + log.Warnf("Freezer: fsfreeze --freeze via %s/%s for %s/%s (path=%s) FAILED: %v", + t.Namespace, t.PodName, t.UserPodNs, t.UserPodName, t.MountPath, err) + continue + } + log.Infof("Freezer: fsfreeze --freeze via %s/%s for %s/%s (path=%s) OK", + t.Namespace, t.PodName, t.UserPodNs, t.UserPodName, t.MountPath) + frozen = append(frozen, t) + } + return frozen +} + +// Unfreeze runs `fsfreeze --unfreeze` for every target that FreezeForVolumeHandle +// successfully froze. Reverse order so lower layers unblock first. +func (f *Freezer) Unfreeze(ctx context.Context, frozen []FrozenTarget) { + if f == nil { + return + } + for i := len(frozen) - 1; i >= 0; i-- { + t := frozen[i] + if err := f.execFsfreeze(ctx, t, "--unfreeze"); err != nil { + log.Errorf("Freezer: fsfreeze --unfreeze via %s/%s for %s/%s (path=%s) FAILED — filesystem may remain frozen; investigate: %v", + t.Namespace, t.PodName, t.UserPodNs, t.UserPodName, t.MountPath, err) + continue + } + log.Infof("Freezer: fsfreeze --unfreeze via %s/%s for %s/%s (path=%s) OK", + t.Namespace, t.PodName, t.UserPodNs, t.UserPodName, t.MountPath) + } +} + +// findMountsForVolumeHandle returns every (csi-node-pod, mountPath) tuple +// covering the driver's node-side mount of the given CSI volume. See +// FrozenTarget for why we target csi-node instead of the user's app pod. +// +// The lookup chain is: +// 1. PV whose spec.csi.volumeHandle matches → its ClaimRef → PV name +// 2. Running pods that reference that PVC → their node names + pod UIDs +// 3. For each such node, the csi-node DaemonSet pod running there +// 4. The kubelet-managed mount path for this volume in that user pod +func (f *Freezer) findMountsForVolumeHandle(ctx context.Context, volumeHandle string) ([]FrozenTarget, error) { + // Step 1: PV with matching csi.volumeHandle → its claim ref + pvs, err := f.clientset.CoreV1().PersistentVolumes().List(ctx, metav1.ListOptions{}) + if err != nil { + return nil, fmt.Errorf("list PVs: %w", err) + } + var claimNs, claimName, pvName string + for i := range pvs.Items { + pv := &pvs.Items[i] + if pv.Spec.CSI == nil || pv.Spec.CSI.VolumeHandle != volumeHandle { + continue + } + if pv.Spec.ClaimRef == nil { + continue + } + claimNs = pv.Spec.ClaimRef.Namespace + claimName = pv.Spec.ClaimRef.Name + pvName = pv.Name + break + } + if claimName == "" { + return nil, nil + } + + // Step 2: Running pods in that namespace that reference the PVC + pods, err := f.clientset.CoreV1().Pods(claimNs).List(ctx, metav1.ListOptions{}) + if err != nil { + return nil, fmt.Errorf("list pods in %s: %w", claimNs, err) + } + + // Step 3: For each such pod, find the csi-node pod on the same node + // and compute the kubelet-managed mount path. + var targets []FrozenTarget + for i := range pods.Items { + userPod := &pods.Items[i] + if userPod.Status.Phase != corev1.PodRunning { + continue + } + // Verify at least one PVC volume in this pod references our claim + found := false + for _, v := range userPod.Spec.Volumes { + if v.PersistentVolumeClaim != nil && v.PersistentVolumeClaim.ClaimName == claimName { + found = true + break + } + } + if !found { + continue + } + + // csi-node pod on this user pod's node + nodeName := userPod.Spec.NodeName + if nodeName == "" { + continue + } + csiNodePod, csiNodeContainer, err := f.findCsiNodePodOnNode(ctx, nodeName) + if err != nil { + log.Warnf("Freezer: no csi-node pod on node %s: %v", nodeName, err) + continue + } + + // kubelet's per-pod CSI mount path — csi-node has this under + // mountPropagation=Bidirectional on /var/lib/kubelet. + mountPath := fmt.Sprintf( + "/var/lib/kubelet/pods/%s/volumes/kubernetes.io~csi/%s/mount", + userPod.UID, pvName, + ) + + targets = append(targets, FrozenTarget{ + Namespace: csiNodePod.Namespace, + PodName: csiNodePod.Name, + Container: csiNodeContainer, + MountPath: mountPath, + UserPodNs: userPod.Namespace, + UserPodName: userPod.Name, + }) + } + return targets, nil +} + +// findCsiNodePodOnNode locates the driver's own csi-node DaemonSet pod +// running on the given node, along with the container that has fsfreeze +// available (the hs-csi-plugin-node container). +func (f *Freezer) findCsiNodePodOnNode(ctx context.Context, nodeName string) (*corev1.Pod, string, error) { + // The csi-node DS is deployed in kube-system with label app=csi-node + // (matching the bundled plugin.yaml). Filter by that + spec.nodeName. + list, err := f.clientset.CoreV1().Pods("kube-system").List(ctx, metav1.ListOptions{ + LabelSelector: "app=csi-node", + FieldSelector: "spec.nodeName=" + nodeName, + }) + if err != nil { + return nil, "", err + } + for i := range list.Items { + p := &list.Items[i] + if p.Status.Phase != corev1.PodRunning { + continue + } + return p, "hs-csi-plugin-node", nil + } + return nil, "", fmt.Errorf("no running csi-node pod on %s", nodeName) +} + +// execFsfreeze runs `fsfreeze ` inside the target pod/container +// via the k8s API's exec subresource. +func (f *Freezer) execFsfreeze(ctx context.Context, t FrozenTarget, op string) error { + cmd := []string{"fsfreeze", op, t.MountPath} + req := f.clientset.CoreV1().RESTClient().Post(). + Resource("pods"). + Namespace(t.Namespace). + Name(t.PodName). + SubResource("exec"). + VersionedParams(&corev1.PodExecOptions{ + Container: t.Container, + Command: cmd, + Stdin: false, + Stdout: true, + Stderr: true, + TTY: false, + }, scheme.ParameterCodec) + + exe, err := remotecommand.NewSPDYExecutor(f.restCfg, "POST", req.URL()) + if err != nil { + return fmt.Errorf("build SPDY executor: %w", err) + } + var stdout, stderr bytes.Buffer + err = exe.StreamWithContext(ctx, remotecommand.StreamOptions{ + Stdout: &stdout, + Stderr: &stderr, + }) + if err != nil { + return fmt.Errorf("stream: %w (stderr=%q)", err, stderr.String()) + } + return nil +} diff --git a/pkg/driver/node.go b/pkg/driver/node.go index 1580ef3..479a6a6 100644 --- a/pkg/driver/node.go +++ b/pkg/driver/node.go @@ -28,6 +28,8 @@ import ( "github.com/container-storage-interface/spec/lib/go/csi" "github.com/hammer-space/csi-plugin/pkg/common" log "github.com/sirupsen/logrus" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/trace" "google.golang.org/grpc/codes" "google.golang.org/grpc/status" ) @@ -155,7 +157,13 @@ func (d *CSIDriver) NodeGetVolumeStats(ctx context.Context, req *csi.NodeGetVolu }, nil } -func (d *CSIDriver) NodeStageVolume(ctx context.Context, req *csi.NodeStageVolumeRequest) (*csi.NodeStageVolumeResponse, error) { +func (d *CSIDriver) NodeStageVolume(ctx context.Context, req *csi.NodeStageVolumeRequest) (_ *csi.NodeStageVolumeResponse, err error) { + ctx, span := tracer.Start(ctx, "Node/NodeStageVolume", trace.WithAttributes( + attribute.String("volume.id", req.GetVolumeId()), + attribute.String("staging.target", req.GetStagingTargetPath()), + )) + defer span.End() + defer common.MeasureOp(ctx, "Node/NodeStageVolume")(&err) volumeID := req.GetVolumeId() volumeContext := req.GetVolumeContext() stagingTarget := req.GetStagingTargetPath() @@ -189,7 +197,7 @@ func (d *CSIDriver) NodeStageVolume(ctx context.Context, req *csi.NodeStageVolum marker := GetHashedMarkerPath(common.BaseVolumeMarkerSourcePath, volumeID) - err := os.WriteFile(marker, []byte(""), 0644) + err = os.WriteFile(marker, []byte(""), 0644) if err != nil { log.Warnf("Not able to create marker file path %s err %v", marker, err) } @@ -205,7 +213,13 @@ func (d *CSIDriver) NodeStageVolume(ctx context.Context, req *csi.NodeStageVolum return &csi.NodeStageVolumeResponse{}, nil } -func (d *CSIDriver) NodeUnstageVolume(ctx context.Context, req *csi.NodeUnstageVolumeRequest) (*csi.NodeUnstageVolumeResponse, error) { +func (d *CSIDriver) NodeUnstageVolume(ctx context.Context, req *csi.NodeUnstageVolumeRequest) (_ *csi.NodeUnstageVolumeResponse, err error) { + ctx, span := tracer.Start(ctx, "Node/NodeUnstageVolume", trace.WithAttributes( + attribute.String("volume.id", req.GetVolumeId()), + attribute.String("staging.target", req.GetStagingTargetPath()), + )) + defer span.End() + defer common.MeasureOp(ctx, "Node/NodeUnstageVolume")(&err) volumeID := req.GetVolumeId() stagingTarget := req.GetStagingTargetPath() @@ -233,13 +247,19 @@ func (d *CSIDriver) NodeUnstageVolume(ctx context.Context, req *csi.NodeUnstageV // if no volume are mounted log.Debugf("No volume marker is present on this node. Remove root mount as well..") _ = os.RemoveAll(common.BaseVolumeMarkerSourcePath) - _ = common.UnmountFilesystem(common.BaseBackingShareMountPath) + _ = common.UnmountFilesystem(ctx, common.BaseBackingShareMountPath) } return &csi.NodeUnstageVolumeResponse{}, nil } -func (d *CSIDriver) NodePublishVolume(ctx context.Context, req *csi.NodePublishVolumeRequest) (*csi.NodePublishVolumeResponse, error) { +func (d *CSIDriver) NodePublishVolume(ctx context.Context, req *csi.NodePublishVolumeRequest) (_ *csi.NodePublishVolumeResponse, err error) { + ctx, span := tracer.Start(ctx, "Node/NodePublishVolume", trace.WithAttributes( + attribute.String("volume.id", req.GetVolumeId()), + attribute.String("target.path", req.GetTargetPath()), + )) + defer span.End() + defer common.MeasureOp(ctx, "Node/NodePublishVolume")(&err) volume_id := req.GetVolumeId() targetPath := req.GetTargetPath() @@ -331,7 +351,13 @@ func (d *CSIDriver) NodePublishVolume(ctx context.Context, req *csi.NodePublishV return &csi.NodePublishVolumeResponse{}, nil } -func (d *CSIDriver) NodeUnpublishVolume(ctx context.Context, req *csi.NodeUnpublishVolumeRequest) (*csi.NodeUnpublishVolumeResponse, error) { +func (d *CSIDriver) NodeUnpublishVolume(ctx context.Context, req *csi.NodeUnpublishVolumeRequest) (_ *csi.NodeUnpublishVolumeResponse, err error) { + ctx, span := tracer.Start(ctx, "Node/NodeUnpublishVolume", trace.WithAttributes( + attribute.String("volume.id", req.GetVolumeId()), + attribute.String("target.path", req.GetTargetPath()), + )) + defer span.End() + defer common.MeasureOp(ctx, "Node/NodeUnpublishVolume")(&err) if req.GetVolumeId() == "" { return nil, status.Error(codes.InvalidArgument, common.EmptyVolumeId) @@ -388,7 +414,7 @@ func (d *CSIDriver) NodeUnpublishVolume(ctx context.Context, req *csi.NodeUnpubl } case mode.IsDir(): // directory for mount volumes log.Infof("Detected directory mount at target path %s", targetPath) - if err := common.UnmountFilesystem(targetPath); err != nil { + if err := common.UnmountFilesystem(ctx, targetPath); err != nil { return nil, status.Error(codes.Internal, err.Error()) } default: diff --git a/pkg/driver/node_helper.go b/pkg/driver/node_helper.go index 8e7e465..9644e67 100644 --- a/pkg/driver/node_helper.go +++ b/pkg/driver/node_helper.go @@ -11,6 +11,8 @@ import ( "github.com/hammer-space/csi-plugin/pkg/common" log "github.com/sirupsen/logrus" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/trace" "google.golang.org/grpc/codes" "google.golang.org/grpc/status" ) @@ -40,7 +42,7 @@ func (d *CSIDriver) publishShareBackedVolume(ctx context.Context, volumeId, targ // Clear old mount because now this will come up with bind mount. // This meant the the publish was not from bind mount, so remove old share mount to clear old direct nfs mount and do bind mount from here. log.Debugf("Strating unmouting for target path %s, due to old style mount from v1.2.7 and earlier", targetPath) - if err := common.UnmountFilesystem(targetPath); err != nil { + if err := common.UnmountFilesystem(ctx, targetPath); err != nil { log.Warnf("Not able to clear the old mount point targetpath (%s) volumeid (%s)", targetPath, volumeId) } log.Infof("[LazyStage] Completed mounting base HS share for volume %s", volumeId) @@ -173,7 +175,7 @@ func (d *CSIDriver) publishShareBackedDirBasedVolume(ctx context.Context, backin if err := common.BindMountDevice(sourceMountPoint, targetPath); err != nil { log.Errorf("bind mount failed for %s: %v", targetPath, err) - CleanupLoopDevice(targetPath) + CleanupLoopDevice(ctx, targetPath) d.UnmountBackingShareIfUnused(ctx, backingShareName) return err } @@ -250,7 +252,7 @@ func (d *CSIDriver) publishFileBackedVolume(ctx context.Context, backingShareNam deviceStr, err := AttachLoopDeviceWithRetry(filePath, readOnly) if err != nil { log.Errorf("failed to attach loop device: %v", err) - CleanupLoopDevice(deviceStr) + CleanupLoopDevice(ctx, deviceStr) d.UnmountBackingShareIfUnused(ctx, backingShareName) return status.Errorf(codes.Internal, common.LoopDeviceAttachFailed, deviceStr, filePath) } @@ -258,7 +260,7 @@ func (d *CSIDriver) publishFileBackedVolume(ctx context.Context, backingShareNam if err := common.BindMountDevice(deviceStr, targetPath); err != nil { log.Errorf("bind mount failed for %s: %v", deviceStr, err) - CleanupLoopDevice(deviceStr) + CleanupLoopDevice(ctx, deviceStr) d.UnmountBackingShareIfUnused(ctx, backingShareName) return err } @@ -279,6 +281,11 @@ func (d *CSIDriver) publishFileBackedVolume(ctx context.Context, backingShareNam // NodeUnpublishVolume func (d *CSIDriver) unpublishFileBackedVolume(ctx context.Context, volumePath, targetPath string) error { + ctx, span := tracer.Start(ctx, "unpublishFileBackedVolume", trace.WithAttributes( + attribute.String("volume.path", volumePath), + attribute.String("target.path", targetPath), + )) + defer span.End() //determine backing share backingShareName := filepath.Dir(volumePath) diff --git a/pkg/driver/utils.go b/pkg/driver/utils.go index 86d55c1..0689596 100644 --- a/pkg/driver/utils.go +++ b/pkg/driver/utils.go @@ -25,11 +25,14 @@ import ( "path/filepath" "strconv" "strings" + "sync" "time" "context" log "github.com/sirupsen/logrus" + "go.opentelemetry.io/otel/attribute" + "go.opentelemetry.io/otel/trace" "google.golang.org/grpc/codes" "google.golang.org/grpc/status" @@ -153,7 +156,11 @@ func AttachLoopDeviceWithRetry(filePath string, readOnly bool) (string, error) { } // CleanupLoopDevice detaches a loop device if it exists -func CleanupLoopDevice(dev string) { +func CleanupLoopDevice(ctx context.Context, dev string) { + _, span := tracer.Start(ctx, "CleanupLoopDevice", trace.WithAttributes( + attribute.String("device", dev), + )) + defer span.End() if _, err := os.Stat(dev); os.IsNotExist(err) { log.Warnf("Loop device %s does not exist, skipping cleanup", dev) return @@ -185,6 +192,18 @@ func GetVolumeNameFromPath(path string) string { return filepath.Base(path) } +// isFileBackedVolumeID reports, with no REST call, whether a volume ID refers to +// a file-backed volume. CreateVolume builds file-backed IDs as +// "/" - an extra path segment living inside +// the backing share - and share-backed IDs as "", a +// single segment directly under the prefix. So a volume is file-backed exactly +// when its ID sits one level below the share-path prefix. Deciding this from the +// ID lets callers skip the GetShare probe that, for file-backed volumes, always +// returns 404. +func isFileBackedVolumeID(volumeID string) bool { + return filepath.Dir(volumeID) != common.SharePathPrefix +} + func GetSnapshotNameFromSnapshotId(snapshotId string) (string, error) { tokens := strings.SplitN(snapshotId, "|", 2) if len(tokens) != 2 { @@ -207,6 +226,47 @@ func GetSnapshotIDFromSnapshotName(hsSnapName, sourceVolumeID string) string { return fmt.Sprintf("%s|%s", hsSnapName, sourceVolumeID) } +// mountState classifies an existing backing-share staging mount. +type mountState int + +const ( + mountHealthy mountState = iota // cleanly mounted; reuse it + mountAbsent // nothing mounted; just mount + mountStale // confirmed unreachable; force-clear then mount +) + +// mountStaleProbes is how many CONSECUTIVE SafeIsMountPoint timeouts we require +// before treating a backing mount as stale and force-unmounting it. A single +// timeout is more likely a slow-but-healthy NFS stat under the concurrency this +// driver now allows than a dead server; only a run of timeouts indicates the +// server is actually gone. Kept small so a genuinely dead mount is still cleared +// promptly. +const mountStaleProbes = 2 + +// classifyMount probes path up to `probes` times to decide whether a backing +// mount is healthy, absent, or stale. `probe` is common.SafeIsMountPoint in +// production and is injectable for tests; it returns (mounted, nil) when it can +// answer, an os.ErrNotExist error when the path does not exist, and +// context.DeadlineExceeded on timeout. "stale" is concluded only after `probes` +// consecutive timeouts, so a live-but-slow shared mount is never force-unmounted +// out from under in-flight pods on a false positive. +func classifyMount(path string, probes int, probe func(string) (bool, error)) mountState { + for i := 0; i < probes; i++ { + mounted, err := probe(path) + if err == nil { + if mounted { + return mountHealthy + } + return mountAbsent + } + if os.IsNotExist(err) { + return mountAbsent + } + // timeout / other transient error — retry + } + return mountStale +} + func (d *CSIDriver) EnsureBackingShareMounted(ctx context.Context, backingShareName string, hsVol *common.HSVolume) error { backingShare, err := d.hsclient.GetShare(ctx, backingShareName) if err != nil { @@ -214,26 +274,172 @@ func (d *CSIDriver) EnsureBackingShareMounted(ctx context.Context, backingShareN } if backingShare != nil { backingDir := common.ShareStagingDir + backingShare.ExportPath - // Mount backing share - isMounted := common.IsShareMounted(backingDir) - log.Infof("Checked mount for %s: isMounted=%t", backingDir, isMounted) - if !isMounted { - err := d.MountShareAtBestDataportal(ctx, backingShare.ExportPath, backingDir, hsVol.MountFlags, hsVol.FQDN) - if err != nil { - log.Errorf("failed to mount backing share, %v", err) - return err - } - - log.Infof("mounted backing share, %s", backingDir) - } else { + // Classify the existing mount. We only force-unmount when it is CONFIRMED + // stale (mountStaleProbes consecutive mount-check timeouts) — never on a + // single timeout, which can be a slow-but-healthy stat under load and + // would otherwise `umount -f -l` a live shared backing mount out from + // under other in-flight file-backed pods (the same outage this path is + // meant to prevent, via a false positive). + state := classifyMount(backingDir, mountStaleProbes, common.SafeIsMountPoint) + log.Infof("Checked mount for %s: state=%d", backingDir, state) + switch state { + case mountHealthy: log.Infof("backing share already mounted, %s", backingDir) + return nil + case mountStale: + // A hung/stale NFS mount is lingering (server unreachable). Force-clear + // it best-effort so the mount below re-establishes against the CURRENT + // data portal rather than reusing a dead mount — this is what makes + // file-backed provisioning survive an Anvil swap. + log.Warnf("backing share %s appears stale after %d consecutive mount-check timeouts; force-clearing before remount", backingDir, mountStaleProbes) + common.ForceUnmountStale(backingDir) + case mountAbsent: + // nothing mounted here; fall through to mount + } + if err := d.MountShareAtBestDataportal(ctx, backingShare.ExportPath, backingDir, hsVol.MountFlags, hsVol.FQDN); err != nil { + log.Errorf("failed to mount backing share, %v", err) + return err } + log.Infof("mounted backing share, %s", backingDir) return nil } return nil } +// mountLockFor returns the per-backing-directory lock that serializes the actual +// mount/unmount of that share. Using a distinct lock per directory (instead of the +// global mountRefsMu) is what lets the slow NFS mount run without freezing refcount +// operations on other shares. The lock is created on first use and never deleted — +// there are only a handful of distinct backing shares, so the map stays tiny, and +// keeping entries avoids a delete-vs-lookup race. The global mountRefsMu is held +// only to look up/insert the entry (microseconds). +func (d *CSIDriver) mountLockFor(backingDir string) *sync.Mutex { + d.mountRefsMu.Lock() + defer d.mountRefsMu.Unlock() + ml, ok := d.mountLocks[backingDir] + if !ok { + ml = &sync.Mutex{} + d.mountLocks[backingDir] = ml + } + return ml +} + +// acquireBackingMount guarantees the backing share is mounted and takes one +// in-flight reference on it. It must be called WITHOUT the per-backing-share volume +// lock held, so the caller's subsequent per-file work (mkfs) runs concurrently with +// other creates on the same share. +// +// The mount is serialized by a PER-DIRECTORY lock (mountLockFor), never by the +// global mountRefsMu. EnsureBackingShareMounted performs a real NFS mount that can +// block for up to the ~5 min command-exec timeout against a slow or dead portal. +// The previous code held mountRefsMu across that mount, so a single slow mount froze +// EVERY other file-backed operation — including refcount reads (backingMountInUse) +// and releases on unrelated shares — for the whole mount window. With a per-dir +// lock, a slow mount on one share only blocks new mounts of THAT share; mountRefsMu +// is taken only for the microsecond map updates below. +// +// The reference is reserved BEFORE mounting (bumpBackingRef, so the count is >=1 +// throughout the mount). That closes a window the old global-lock design covered +// incidentally: a concurrent delete's UnmountBackingShareIfUnused must not observe +// refcount 0 mid-mount and unmount the share out from under us. The per-dir lock +// additionally guarantees two concurrent first-creates can't both mount the target. +func (d *CSIDriver) acquireBackingMount(ctx context.Context, backingShare *common.ShareResponse, hsVol *common.HSVolume) error { + backingDir := common.ShareStagingDir + backingShare.ExportPath + + ml := d.mountLockFor(backingDir) + ml.Lock() + defer ml.Unlock() + + // Reserve the reference first; `first` is true only on the 0->1 transition. + if first := d.bumpBackingRef(backingDir); first { + if err := d.EnsureBackingShareMounted(ctx, backingShare.Name, hsVol); err != nil { + // Roll back the reservation so a failed mount doesn't leak a reference + // that would keep the (unmounted) share pinned as "in use" forever. + d.dropBackingRef(backingDir) + return err + } + } + return nil +} + +// releaseBackingMount drops one in-flight reference taken by acquireBackingMount +// and unmounts the backing share once the last concurrent file operation using it +// has finished (refcount reaches 0). UnmountBackingShareIfUnused still applies its +// own loopback-device safety check before actually unmounting. +func (d *CSIDriver) releaseBackingMount(ctx context.Context, backingShare *common.ShareResponse) { + backingDir := common.ShareStagingDir + backingShare.ExportPath + + // Take the SAME per-dir lock as acquireBackingMount so this unmount can't race a + // concurrent (re)mount of the same target, and — as in acquire — so the slow + // unmount runs under the per-dir lock rather than the global mountRefsMu. + ml := d.mountLockFor(backingDir) + ml.Lock() + defer ml.Unlock() + + // Decide-then-act: drop the refcount under the global mutex (dropBackingRef holds + // it only for the decrement), then run the unmount with the global mutex RELEASED. + // UnmountBackingShareIfUnused re-acquires the global mutex via backingMountInUse, + // and sync.Mutex is not reentrant, so holding it across the call self-deadlocks + // the goroutine — the bug found during live xfs validation, which then never + // released mountRefsMu (or the volume lock above it) and wedged all file-backed + // creates. It only triggers for whichever volume drops the LAST reference. + if !d.dropBackingRef(backingDir) { + return + } + // Because we hold ml, no acquireBackingMount for this dir can re-mount until we + // return; UnmountBackingShareIfUnused still re-checks backingMountInUse (belt and + // suspenders, and to stay correct against the direct, non-ml unmount callers). + if _, err := d.UnmountBackingShareIfUnused(ctx, backingShare.Name); err != nil { + log.Warnf("releaseBackingMount: unmount of %s failed: %v", backingDir, err) + } +} + +// bumpBackingRef adds one in-flight reference for backingDir under mountRefsMu and +// reports whether this was the 0->1 transition (i.e. the caller is responsible for +// mounting). It holds mountRefsMu only for the increment. +func (d *CSIDriver) bumpBackingRef(backingDir string) (first bool) { + d.mountRefsMu.Lock() + defer d.mountRefsMu.Unlock() + first = d.mountRefs[backingDir] == 0 + d.mountRefs[backingDir]++ + return first +} + +// dropBackingRef decrements the in-flight refcount for backingDir under +// mountRefsMu and reports whether this was the final reference (the count +// reached 0, and the map entry was removed). It performs NO unmount and holds +// mountRefsMu only for the decrement itself, so the caller can run the +// same-mutex-taking unmount decision afterwards without self-deadlocking. +func (d *CSIDriver) dropBackingRef(backingDir string) (last bool) { + d.mountRefsMu.Lock() + defer d.mountRefsMu.Unlock() + if d.mountRefs[backingDir] > 0 { + d.mountRefs[backingDir]-- + } + if d.mountRefs[backingDir] == 0 { + delete(d.mountRefs, backingDir) + return true + } + return false +} + +// backingMountInUse reports whether any in-flight file-backed operation still +// holds a reference to the backing-share staging mount at mountPath (see +// acquire/releaseBackingMount). It is the authoritative "in use" signal shared +// with UnmountBackingShareIfUnused so the two mechanisms can't disagree and +// unmount a share out from under an in-flight mkfs. +func (d *CSIDriver) backingMountInUse(mountPath string) bool { + d.mountRefsMu.Lock() + defer d.mountRefsMu.Unlock() + return d.mountRefs[mountPath] > 0 +} + func (d *CSIDriver) UnmountBackingShareIfUnused(ctx context.Context, backingShareName string) (bool, error) { + ctx, span := tracer.Start(ctx, "UnmountBackingShareIfUnused", trace.WithAttributes( + attribute.String("backing_share", backingShareName), + )) + defer span.End() + defer common.MeasureOp(ctx, "UnmountBackingShareIfUnused")(nil) log.Infof("UnmountBackingShareIfUnused is called with backing share name %s", backingShareName) backingShare, err := d.hsclient.GetShare(ctx, backingShareName) if err != nil || backingShare == nil { @@ -241,6 +447,16 @@ func (d *CSIDriver) UnmountBackingShareIfUnused(ctx context.Context, backingShar return false, err } mountPath := common.ShareStagingDir + backingShare.ExportPath + // Honor the mountRefs refcount FIRST. A file-backed CreateVolume holds a + // reference for its whole mkfs/format window (which unit G runs without the + // per-backing-share lock), and during that window there is no loop device + // backing the file yet — so the losetup check below would wrongly report + // "unused" and unmount the share out from under an in-flight mkfs. The + // refcount is the authoritative in-use signal shared with acquireBackingMount. + if d.backingMountInUse(mountPath) { + log.Infof("backing share %s still has in-flight reference(s); not unmounting", mountPath) + return false, nil + } if isMounted := common.IsShareMounted(mountPath); !isMounted { return true, nil } @@ -263,7 +479,7 @@ func (d *CSIDriver) UnmountBackingShareIfUnused(ctx context.Context, backingShar } log.Infof("unmounting backing share %s", mountPath) - err = common.UnmountFilesystem(mountPath) + err = common.UnmountFilesystem(ctx, mountPath) if err != nil { log.Errorf("failed to unmount backing share %s", mountPath) return false, err @@ -370,7 +586,7 @@ func (d *CSIDriver) MountShareAtBestDataportal(ctx context.Context, shareExportP return false } } - err = common.MountShare(export, targetPath, mount_options) + err = common.MountShare(ctx, export, targetPath, mount_options) if err != nil { log.WithFields(log.Fields{ "share": shareExportPath, @@ -470,7 +686,7 @@ func (d *CSIDriver) EnsureRootExportMounted(ctx context.Context, baseRootDirPath log.Errorf("Unable to resolve FQDN %s for root share mount. %v", fqdn, resolveErr) } else { log.Debugf("Calling mount via nfs v4.2 using FQDN %s resolved to IP %s to mount (/) on %s", fqdn, fqdnEndpointIP, baseRootDirPath) - err = common.MountShare(fqdn+":/", baseRootDirPath, effectiveMountFlags) + err = common.MountShare(ctx, fqdn+":/", baseRootDirPath, effectiveMountFlags) if err == nil { log.Debugf("Successfully mounted root share using FQDN %s resolved to IP %s", fqdn, fqdnEndpointIP) return nil @@ -485,7 +701,7 @@ func (d *CSIDriver) EnsureRootExportMounted(ctx context.Context, baseRootDirPath } // Step 3 - Use export ip and path to mount root with 4.2 only. log.Debugf("Calling mount via nfs v4.2 using anvil IP %s to mount (/) on %s", anvilEndpointIP, baseRootDirPath) - err = common.MountShare(anvilEndpointIP+":/", baseRootDirPath, effectiveMountFlags) + err = common.MountShare(ctx, anvilEndpointIP+":/", baseRootDirPath, effectiveMountFlags) if err != nil { log.Errorf("Unable to mount root share via 4.2 using anvil IP. %v", err) diff --git a/pkg/driver/utils_test.go b/pkg/driver/utils_test.go index 15958a2..630f8e7 100644 --- a/pkg/driver/utils_test.go +++ b/pkg/driver/utils_test.go @@ -1,75 +1,297 @@ package driver import ( - "reflect" - "testing" + "context" + "os" + "reflect" + "sync" + "testing" + "time" ) func TestGetSnapshotNameFromSnapshotId(t *testing.T) { - snapshotId := "2019-05-24T15-26-57-0|/sanity-controller-source-vol-859F8B9B-35BBFB36" - expected := "2019-05-24T15-26-57-0" - actual, err := GetSnapshotNameFromSnapshotId(snapshotId) - if err != nil { - t.Logf("Unexpected error, %v", err) - t.FailNow() - } - if !reflect.DeepEqual(actual, expected) { - t.Logf("Expected: %v", expected) - t.Logf("Actual: %v", actual) - t.FailNow() - } - - - snapshotId = "2019-05-24T15-26-57-0" - _, err = GetSnapshotNameFromSnapshotId(snapshotId) - if err == nil { - t.Logf("Expected error") - t.FailNow() - } + snapshotId := "2019-05-24T15-26-57-0|/sanity-controller-source-vol-859F8B9B-35BBFB36" + expected := "2019-05-24T15-26-57-0" + actual, err := GetSnapshotNameFromSnapshotId(snapshotId) + if err != nil { + t.Logf("Unexpected error, %v", err) + t.FailNow() + } + if !reflect.DeepEqual(actual, expected) { + t.Logf("Expected: %v", expected) + t.Logf("Actual: %v", actual) + t.FailNow() + } + + snapshotId = "2019-05-24T15-26-57-0" + _, err = GetSnapshotNameFromSnapshotId(snapshotId) + if err == nil { + t.Logf("Expected error") + t.FailNow() + } } func TestGetShareNameFromSnapshotId(t *testing.T) { - snapshotId := "2019-05-24T15-26-57-0|/sanity-controller-source-vol-859F8B9B-35BBFB36" - expected := "sanity-controller-source-vol-859F8B9B-35BBFB36" - actual, err := GetShareNameFromSnapshotId(snapshotId) - if err != nil { - t.Logf("Unexpected error, %v", err) - t.FailNow() - } - if !reflect.DeepEqual(actual, expected) { - t.Logf("Expected: %v", expected) - t.Logf("Actual: %v", actual) - t.FailNow() - } - - snapshotId = "2019-05-24T15-26-57-0" - _, err = GetShareNameFromSnapshotId(snapshotId) - if err == nil { - t.Logf("Expected error") - t.FailNow() - } + snapshotId := "2019-05-24T15-26-57-0|/sanity-controller-source-vol-859F8B9B-35BBFB36" + expected := "sanity-controller-source-vol-859F8B9B-35BBFB36" + actual, err := GetShareNameFromSnapshotId(snapshotId) + if err != nil { + t.Logf("Unexpected error, %v", err) + t.FailNow() + } + if !reflect.DeepEqual(actual, expected) { + t.Logf("Expected: %v", expected) + t.Logf("Actual: %v", actual) + t.FailNow() + } + + snapshotId = "2019-05-24T15-26-57-0" + _, err = GetShareNameFromSnapshotId(snapshotId) + if err == nil { + t.Logf("Expected error") + t.FailNow() + } } func TestGetSnapshotIDFromSnapshotName(t *testing.T) { - expected := "2019-05-24T15-26-57-0|/sanity-controller-source-vol-859F8B9B-35BBFB36" - actual := GetSnapshotIDFromSnapshotName("2019-05-24T15-26-57-0", - "/sanity-controller-source-vol-859F8B9B-35BBFB36") - if !reflect.DeepEqual(actual, expected) { - t.Logf("Expected: %v", expected) - t.Logf("Actual: %v", actual) - t.FailNow() - } + expected := "2019-05-24T15-26-57-0|/sanity-controller-source-vol-859F8B9B-35BBFB36" + actual := GetSnapshotIDFromSnapshotName("2019-05-24T15-26-57-0", + "/sanity-controller-source-vol-859F8B9B-35BBFB36") + if !reflect.DeepEqual(actual, expected) { + t.Logf("Expected: %v", expected) + t.Logf("Actual: %v", actual) + t.FailNow() + } } func TestGetVolumeNameFromPath(t *testing.T) { - expected := "test-volume" - actual := GetVolumeNameFromPath("/test-backing-share/test-volume") - if !reflect.DeepEqual(actual, expected) { - t.Logf("Expected: %v", expected) - t.Logf("Actual: %v", actual) - t.FailNow() - } -} \ No newline at end of file + expected := "test-volume" + actual := GetVolumeNameFromPath("/test-backing-share/test-volume") + if !reflect.DeepEqual(actual, expected) { + t.Logf("Expected: %v", expected) + t.Logf("Actual: %v", actual) + t.FailNow() + } +} + +// TestIsFileBackedVolumeID covers the structural file- vs share-backed +// discriminator (PR C): a file-backed volume ID is a file *inside* a backing +// share (multi-segment path), while a share-backed/native NFS volume ID is the +// share itself (single top-level segment). This replaced a GetShare probe that +// 404'd for every file-backed source. +func TestIsFileBackedVolumeID(t *testing.T) { + cases := []struct { + volumeID string + want bool + }{ + {"/k8s-file-backed/file--pvc-1234", true}, // file inside a backing share + {"/backing/file--pvc-abcd/nested", true}, // deeper still -> file-backed + {"/some-share", false}, // native NFS share + {"/hs-nfs-prod", false}, // native NFS share + {"/", false}, // root + {"", false}, // empty -> Dir("")=="." != "/" would be true; guard below + } + for _, tc := range cases { + got := isFileBackedVolumeID(tc.volumeID) + // "" is a malformed ID; document current behavior rather than assert a + // specific value we don't rely on. + if tc.volumeID == "" { + continue + } + if got != tc.want { + t.Fatalf("isFileBackedVolumeID(%q) = %v, want %v", tc.volumeID, got, tc.want) + } + } +} + +// TestClassifyMount covers the fix for review comment #1: a single +// SafeIsMountPoint timeout must NOT be treated as "stale" (which would trigger a +// force-unmount of a live shared backing mount). Only `probes` CONSECUTIVE +// timeouts classify as stale; a timeout that recovers on retry is healthy. +func TestClassifyMount(t *testing.T) { + type r struct { + mounted bool + err error + } + // newProbe returns a probe that yields the scripted results in order, + // repeating the last one for any extra calls. + newProbe := func(results ...r) func(string) (bool, error) { + i := 0 + return func(string) (bool, error) { + res := results[i] + if i < len(results)-1 { + i++ + } + return res.mounted, res.err + } + } + cases := []struct { + name string + results []r + probes int + want mountState + }{ + {"healthy on first probe", []r{{true, nil}}, 2, mountHealthy}, + {"absent (not a mount point)", []r{{false, nil}}, 2, mountAbsent}, + {"not-exist path -> absent", []r{{false, os.ErrNotExist}}, 2, mountAbsent}, + {"all timeouts -> stale", []r{{false, context.DeadlineExceeded}, {false, context.DeadlineExceeded}}, 2, mountStale}, + {"timeout then healthy -> healthy (no false positive)", []r{{false, context.DeadlineExceeded}, {true, nil}}, 2, mountHealthy}, + {"timeout then absent -> absent", []r{{false, context.DeadlineExceeded}, {false, nil}}, 2, mountAbsent}, + {"single probe timeout -> stale", []r{{false, context.DeadlineExceeded}}, 1, mountStale}, + } + for _, tc := range cases { + t.Run(tc.name, func(t *testing.T) { + if got := classifyMount("/backing", tc.probes, newProbe(tc.results...)); got != tc.want { + t.Fatalf("classifyMount = %d, want %d", got, tc.want) + } + }) + } +} + +// TestDropBackingRefNoDeadlock is a regression test for the self-deadlock found +// during live xfs validation: releaseBackingMount held mountRefsMu while calling +// UnmountBackingShareIfUnused, which re-locks the same (non-reentrant) mutex via +// backingMountInUse — so the last reference to drop wedged the whole file-backed +// mount subsystem. dropBackingRef must release mountRefsMu before returning, so a +// same-mutex call (like backingMountInUse, which the real unmount path makes) is +// safe immediately afterwards. The whole sequence runs under a watchdog: if the +// lock is ever held across the callout again, this hangs and the test fails. +func TestDropBackingRefNoDeadlock(t *testing.T) { + d := &CSIDriver{mountRefs: map[string]int{}} + p := "/tmp/k8s-file-rev" + + done := make(chan struct{}) + go func() { + defer close(done) + // Two concurrent creates hold the share; the first release is NOT the last. + d.mountRefs[p] = 2 + if last := d.dropBackingRef(p); last { + t.Errorf("dropBackingRef at refcount 2->1 reported last=true") + } + // Mimic the real unmount path taking the same mutex right after the drop. + // Before the fix, this is exactly where the goroutine deadlocked. + if !d.backingMountInUse(p) { + t.Errorf("refcount 1 should still read as in-use") + } + // Second release IS the last: entry must be deleted and read not-in-use. + if last := d.dropBackingRef(p); !last { + t.Errorf("dropBackingRef at refcount 1->0 reported last=false") + } + if d.backingMountInUse(p) { + t.Errorf("refcount 0 should read as not-in-use after final drop") + } + if _, ok := d.mountRefs[p]; ok { + t.Errorf("map entry should be deleted at refcount 0") + } + // Dropping below zero must stay at last=true and not underflow. + if last := d.dropBackingRef(p); !last { + t.Errorf("dropBackingRef on absent key should report last=true") + } + }() + + select { + case <-done: + case <-time.After(5 * time.Second): + t.Fatal("dropBackingRef/backingMountInUse deadlocked: mountRefsMu held across callout") + } +} + +// TestBumpBackingRefFirst covers bumpBackingRef reporting the 0->1 transition, +// which is what tells acquireBackingMount it owns the mount. The reference is taken +// BEFORE mounting so a concurrent delete can't see refcount 0 mid-mount. +func TestBumpBackingRefFirst(t *testing.T) { + d := &CSIDriver{mountRefs: map[string]int{}} + p := "/tmp/k8s-file-rev" + + if first := d.bumpBackingRef(p); !first { + t.Fatal("first bump on an unmounted share should report first=true") + } + if first := d.bumpBackingRef(p); first { + t.Fatal("second bump should report first=false (already mounted)") + } + if d.mountRefs[p] != 2 { + t.Fatalf("refcount = %d, want 2", d.mountRefs[p]) + } + // Drain back to 0, then the next bump is a fresh 0->1 transition again. + d.dropBackingRef(p) + d.dropBackingRef(p) + if first := d.bumpBackingRef(p); !first { + t.Fatal("bump after draining to 0 should report first=true again") + } +} + +// TestBackingMountLockDoesNotBlockRefcount is the core regression test for moving +// the slow NFS mount off the global mountRefsMu: while a per-directory mount lock is +// held (simulating an in-progress or hung mount that can last up to ~5 min), the +// global refcount operations — bumpBackingRef, backingMountInUse, dropBackingRef, +// for the SAME dir and OTHER dirs — must still complete immediately. If the mount +// ever moves back under mountRefsMu, those ops would block behind the held mount +// lock and this watchdog fails. +func TestBackingMountLockDoesNotBlockRefcount(t *testing.T) { + d := &CSIDriver{mountRefs: map[string]int{}, mountLocks: map[string]*sync.Mutex{}} + dir1 := "/tmp/k8s-file-rev" + dir2 := "/tmp/other-share" + + // mountLockFor must be stable per dir and distinct across dirs. + if d.mountLockFor(dir1) != d.mountLockFor(dir1) { + t.Fatal("mountLockFor returned different locks for the same dir") + } + if d.mountLockFor(dir1) == d.mountLockFor(dir2) { + t.Fatal("mountLockFor returned the same lock for different dirs") + } + + // Simulate a slow/hung mount: hold dir1's per-dir mount lock for the whole test. + ml := d.mountLockFor(dir1) + ml.Lock() + defer ml.Unlock() + + done := make(chan struct{}) + go func() { + defer close(done) + // Refcount ops on the SAME dir whose mount is "in progress" must not block. + d.bumpBackingRef(dir1) + if !d.backingMountInUse(dir1) { + t.Errorf("dir1 should read in-use after bump") + } + d.dropBackingRef(dir1) + // ...and neither must ops on an unrelated dir. + d.bumpBackingRef(dir2) + d.dropBackingRef(dir2) + }() + + select { + case <-done: + case <-time.After(5 * time.Second): + t.Fatal("refcount ops blocked behind a held mount lock: mount is back under mountRefsMu") + } +} + +// TestBackingMountInUse covers the fix for review comment #2: the mountRefs +// refcount is the authoritative in-use signal that UnmountBackingShareIfUnused +// now consults (so it can't unmount a share out from under an in-flight mkfs +// that holds a reference but has no loop device yet). +func TestBackingMountInUse(t *testing.T) { + d := &CSIDriver{mountRefs: map[string]int{}} + p := "/tmp/k8s-file-backed" + + if d.backingMountInUse(p) { + t.Fatal("empty refcount should report not-in-use") + } + d.mountRefs[p] = 1 + if !d.backingMountInUse(p) { + t.Fatal("refcount 1 should report in-use") + } + d.mountRefs[p] = 2 + if !d.backingMountInUse(p) { + t.Fatal("refcount 2 should report in-use") + } + // An unrelated path's references must not make this path read as in-use. + d.mountRefs["/tmp/other-share"] = 5 + d.mountRefs[p] = 0 + if d.backingMountInUse(p) { + t.Fatal("refcount 0 should report not-in-use even when another path is referenced") + } +}