diff --git a/charts/shield/docs/host-profiles.md b/charts/shield/docs/host-profiles.md new file mode 100644 index 000000000..b3c2f7260 --- /dev/null +++ b/charts/shield/docs/host-profiles.md @@ -0,0 +1,225 @@ +# Host Shield Node Profiles + +## Problem + +The host shield runs as a DaemonSet - one pod per node. On clusters with mixed +node types (dense/heavy nodes alongside lighter ones), a single resource +configuration forces a trade-off: size for the heaviest node and waste resources +on lighter ones, or size for the lightest and risk OOMKilled on heavier nodes. + +`host.profiles` solves this by letting a single Helm release deploy one +right-sized host shield DaemonSet per node type. + +## How it works + +A DaemonSet with a `nodeSelector` only runs on nodes whose labels match. By +declaring multiple profiles - each with its own `nodeSelector` and resource +overrides - you get N independently-sized DaemonSets from one release. Helm +does not need to know anything about your nodes at deploy time; the Kubernetes +scheduler handles placement at runtime. + +``` +Nodes labeled sysdig.com/node-size=large -> DaemonSet shield-host-large (1000m / 1000Mi) +Nodes labeled sysdig.com/node-size=medium -> DaemonSet shield-host-medium ( 900m / 900Mi) +Nodes labeled sysdig.com/node-size=small -> DaemonSet shield-host-small ( 800m / 800Mi) +``` + +Everything else - RBAC, ConfigMap, Secrets, the cluster shield Deployment - +remains singular. Only the host DaemonSet multiplies. + +## Prerequisites: identify or add node labels + +The label you use for targeting must be unique per node type (mutually +exclusive). On managed Kubernetes this label usually already exists: + +| Platform | Label key | +|----------|------------------------------------| +| EKS | `eks.amazonaws.com/nodegroup` | +| GKE | `cloud.google.com/gke-nodepool` | +| AKS | `agentpool` | +| Any | `node.kubernetes.io/instance-type` | + +Check your nodes: +```bash +kubectl get nodes --show-labels +# or filter by a specific key: +kubectl get nodes -L eks.amazonaws.com/nodegroup +kubectl get nodes -L cloud.google.com/gke-nodepool +``` + +If your nodes don't already have a distinguishing label, add one: +```bash +kubectl label node sysdig.com/node-size=large +kubectl label node sysdig.com/node-size=medium +kubectl label node sysdig.com/node-size=small +``` + +> **Important:** each profile's `node_selector` must target a **disjoint** set +> of nodes. If two profiles can match the same node, two host shields will land +> on it and conflict over `hostNetwork`/`hostPID`. + +## Configuration + +Add `host.profiles` to your values file. The existing `host.resources`, +`host.node_selector`, `host.affinity`, and `host.tolerations` become the +**base** that every profile inherits from. A profile only needs to specify what +differs from the base. + +```yaml +host: + # Base resources - inherited by all profiles unless overridden + resources: + shield: + limits: + cpu: 1000m + memory: 1Gi + requests: + cpu: 250m + memory: 384Mi + + profiles: + - name: large # required, unique, DNS-label safe + node_selector: # required + sysdig.com/node-size: large + resources: # optional - only what differs from base + shield: + limits: { cpu: 1000m, memory: 1000Mi } + requests: { cpu: 350m, memory: 400Mi } + + - name: medium + node_selector: + sysdig.com/node-size: medium + resources: + shield: + limits: { cpu: 900m, memory: 900Mi } + requests: { cpu: 250m, memory: 300Mi } + + - name: small + node_selector: + sysdig.com/node-size: small + resources: + shield: + limits: { cpu: 800m, memory: 800Mi } + requests: { cpu: 150m, memory: 200Mi } +``` + +Each profile can also override `affinity` and `tolerations` (both inherit from +the base when omitted). This is useful if, for example, a node pool carries a +taint that only the corresponding profile should tolerate. + +```yaml + - name: spot + node_selector: + node.kubernetes.io/instance-type: t3.medium + tolerations: + - key: spot + operator: Exists + effect: NoSchedule + resources: + shield: + limits: { memory: 512Mi } +``` + +## Deployment + +Profiles are supplied as a separate values file alongside your base values: + +```bash +# Preview what will be deployed (no cluster changes) +helm template my-release aaronm-sysdig/shield \ + -f values-shield.yaml \ + -f values-shield-profiles.yaml \ + --show-only templates/host/daemonset.yaml + +# Deploy +helm upgrade --install --create-namespace \ + -n sysdig-agent my-release aaronm-sysdig/shield \ + -f values-shield.yaml \ + -f values-shield-profiles.yaml +``` + +## Verifying deployment + +```bash +# One DaemonSet per profile, each pinned to its node type +kubectl get daemonsets -n sysdig-agent + +# Confirm each pod landed on the right node +kubectl get pods -n sysdig-agent -o wide | grep host + +# Inspect resources on a specific pod +kubectl get pod -n sysdig-agent \ + -o jsonpath='{.spec.containers[0].resources}' | python3 -m json.tool +``` + +Expected output (3-profile example): +``` +NAME DESIRED CURRENT READY NODE SELECTOR +sysdig-agent-shield-host-large 1 1 1 sysdig.com/node-size=large +sysdig-agent-shield-host-medium 1 1 1 sysdig.com/node-size=medium +sysdig-agent-shield-host-small 1 1 1 sysdig.com/node-size=small +``` + +## Backward compatibility + +`host.profiles` defaults to `[]`. When empty, the chart behaves exactly as +before - a single host DaemonSet with the base `host.resources`. Existing +deployments are unaffected until profiles are explicitly configured. + +## Common patterns + +**Two pools (heavy/light):** +```yaml +host: + profiles: + - name: heavy + node_selector: + eks.amazonaws.com/nodegroup: heavy-nodes + resources: + shield: + limits: { memory: 2Gi } + requests: { memory: 768Mi } + - name: light + node_selector: + eks.amazonaws.com/nodegroup: light-nodes + resources: + shield: + limits: { memory: 512Mi } + requests: { memory: 192Mi } +``` + +**GKE node pools:** +```yaml +host: + profiles: + - name: standard + node_selector: + cloud.google.com/gke-nodepool: standard-pool + - name: highmem + node_selector: + cloud.google.com/gke-nodepool: highmem-pool + resources: + shield: + limits: { memory: 2Gi } + requests: { memory: 1Gi } +``` + +**Instance type targeting:** +```yaml +host: + profiles: + - name: xlarge + node_selector: + node.kubernetes.io/instance-type: m5.4xlarge + resources: + shield: + limits: { cpu: "2", memory: 2Gi } + requests: { cpu: 500m, memory: 768Mi } + - name: small + node_selector: + node.kubernetes.io/instance-type: t3.medium + resources: + shield: + limits: { cpu: 500m, memory: 512Mi } + requests: { cpu: 100m, memory: 192Mi } +``` diff --git a/charts/shield/templates/NOTES.txt b/charts/shield/templates/NOTES.txt index 1a89de544..2125c9073 100644 --- a/charts/shield/templates/NOTES.txt +++ b/charts/shield/templates/NOTES.txt @@ -40,3 +40,12 @@ Secrets, SOPS, or sealed-secrets) and reference it instead: See https://github.com/sysdiglabs/charts/issues/2622 for context. {{- end }} +{{ if .Values.host.profiles }} +NOTE: You have configured {{ len .Values.host.profiles }} host shield node profiles: +{{- range .Values.host.profiles }} + - {{ .name }} +{{- end }} +Ensure each profile's node_selector targets a DISJOINT set of nodes. If two +profiles can match the same node, two host shields will be scheduled on it and +conflict over hostNetwork/hostPID. +{{ end -}} diff --git a/charts/shield/templates/host/_helpers.tpl b/charts/shield/templates/host/_helpers.tpl index 98f6ec7db..722007e06 100644 --- a/charts/shield/templates/host/_helpers.tpl +++ b/charts/shield/templates/host/_helpers.tpl @@ -7,6 +7,18 @@ If release name contains chart name it will be used as a full name. {{- printf "%s-host" (include "shield.fullname" . | trunc 57 | trimSuffix "-") }} {{- end }} +{{/* +DaemonSet name. When rendered for a named profile, suffix the profile name and +truncate to the 63-char DNS limit. Without a profile it is exactly host.fullname. +*/}} +{{- define "host.daemonset.name" -}} +{{- if .ProfileName -}} +{{- printf "%s-%s" (include "host.fullname" .) .ProfileName | trunc 63 | trimSuffix "-" -}} +{{- else -}} +{{- include "host.fullname" . -}} +{{- end -}} +{{- end }} + {{- define "host.windows.fullname" -}} {{- printf "%s-host-windows" (include "shield.fullname" . | trunc 50 | trimSuffix "-") }} {{- end }} diff --git a/charts/shield/templates/host/daemonset.yaml b/charts/shield/templates/host/daemonset.yaml index 359283392..70be71b34 100644 --- a/charts/shield/templates/host/daemonset.yaml +++ b/charts/shield/templates/host/daemonset.yaml @@ -1,7 +1,8 @@ +{{- define "host.daemonset" -}} apiVersion: apps/v1 kind: DaemonSet metadata: - name: {{ template "host.fullname" . }} + name: {{ include "host.daemonset.name" . }} namespace: {{ .Release.Namespace }} labels: {{- include "host.workload_labels" . | nindent 4 }} @@ -11,13 +12,19 @@ spec: selector: matchLabels: {{- include "host.selector_labels" . | nindent 6 }} + {{- if .ProfileName }} + sysdig/host-profile: {{ .ProfileName }} + {{- end }} updateStrategy: {{- toYaml .Values.host.update_strategy | nindent 4 }} template: metadata: - name: {{ template "host.fullname" . }} + name: {{ include "host.daemonset.name" . }} labels: {{- include "host.pod_labels" . | nindent 8 }} + {{- if .ProfileName }} + sysdig/host-profile: {{ .ProfileName }} + {{- end }} annotations: {{- include "host.pod_annotations" . | nindent 8 }} spec: @@ -346,3 +353,15 @@ spec: {{- end }} {{- end }} {{- include "host.volumes" . | nindent 8 }} +{{- end -}} +{{- $profiles := .Values.host.profiles | default (list) -}} +{{- if eq (len $profiles) 0 -}} +{{- $profiles = list (dict) -}} +{{- end -}} +{{- range $profile := $profiles }} +{{- $hostMerged := mergeOverwrite (deepCopy $.Values.host) (omit $profile "name") -}} +{{- $valuesMerged := merge (dict "host" $hostMerged) (omit $.Values "host") -}} +{{- $ctx := merge (dict "Values" $valuesMerged "ProfileName" ($profile.name | default "")) (omit $ "Values") -}} +--- +{{ include "host.daemonset" $ctx | trim }} +{{ end -}} diff --git a/charts/shield/tests/host/daemonset_test.yaml b/charts/shield/tests/host/daemonset_test.yaml index 02eef90e5..694b0cd58 100644 --- a/charts/shield/tests/host/daemonset_test.yaml +++ b/charts/shield/tests/host/daemonset_test.yaml @@ -1057,3 +1057,69 @@ tests: mountPath: /opt/draios/etc/local_forwarder_config.yaml subPath: local_forwarder_config.yaml template: host/daemonset.yaml + + - it: Renders exactly one DaemonSet with the default name when profiles is empty + asserts: + - hasDocuments: + count: 1 + - equal: + path: metadata.name + value: release-name-shield-host + - notExists: + path: spec.selector.matchLabels["sysdig/host-profile"] + + - it: Renders one DaemonSet per profile with merged resources and node selectors + set: + host: + node_selector: + base-label: shared + profiles: + - name: heavy + node_selector: + sysdig.com/node-size: heavy + resources: + shield: + limits: + memory: 2Gi + - name: light + node_selector: + sysdig.com/node-size: light + resources: + shield: + limits: + memory: 512Mi + asserts: + - hasDocuments: + count: 2 + - equal: + path: metadata.name + value: release-name-shield-host-heavy + documentIndex: 0 + - equal: + path: metadata.name + value: release-name-shield-host-light + documentIndex: 1 + - equal: + path: spec.selector.matchLabels["sysdig/host-profile"] + value: heavy + documentIndex: 0 + - equal: + path: spec.template.spec.nodeSelector["sysdig.com/node-size"] + value: heavy + documentIndex: 0 + - equal: + path: spec.template.spec.nodeSelector.base-label + value: shared + documentIndex: 0 + - equal: + path: spec.template.spec.containers[0].resources.limits.memory + value: 2Gi + documentIndex: 0 + - equal: + path: spec.template.spec.containers[0].resources.requests.memory + value: 384Mi + documentIndex: 0 + - equal: + path: spec.template.spec.containers[0].resources.limits.memory + value: 512Mi + documentIndex: 1 diff --git a/charts/shield/values.schema.json b/charts/shield/values.schema.json index 087353ebe..e32cab8b8 100644 --- a/charts/shield/values.schema.json +++ b/charts/shield/values.schema.json @@ -426,6 +426,35 @@ }, "volume_mounts": { "$ref": "#/$defs/VolumeMounts" + }, + "profiles": { + "type": "array", + "description": "Node-type profiles. When set, one host shield DaemonSet is rendered per profile, each deep-merging its overrides over the base host settings. Each profile's node_selector must target a disjoint set of nodes.", + "items": { + "type": "object", + "required": [ + "name", + "node_selector" + ], + "properties": { + "name": { + "type": "string" + }, + "node_selector": { + "type": "object" + }, + "affinity": { + "type": "object" + }, + "tolerations": { + "type": "array" + }, + "resources": { + "type": "object" + } + }, + "additionalProperties": false + } } }, "required": [ diff --git a/charts/shield/values.yaml b/charts/shield/values.yaml index 2033280b6..a6fb1db0f 100644 --- a/charts/shield/values.yaml +++ b/charts/shield/values.yaml @@ -370,6 +370,32 @@ host: cpu: 250m # The memory request for the host shield memory: 384Mi + # A list of node-type profiles. When empty (default), a single host shield + # DaemonSet is rendered from the settings above. When set, the settings above + # act as the BASE and ONE DaemonSet is rendered per profile, each deep-merging + # its overrides over the base. + # + # IMPORTANT: each profile's node_selector MUST target a disjoint set of nodes. + # If two profiles can match the same node, two host shields will be scheduled + # on it and fight over hostNetwork/hostPID. Use labels that are mutually + # exclusive (e.g. distinct node-pool or instance-type labels). + # + # Example: + # profiles: + # - name: heavy # required, unique, DNS-label safe + # node_selector: # required - targeting + exclusivity + # sysdig.com/node-size: heavy + # resources: # optional - only what differs from base + # shield: + # limits: { cpu: "2", memory: 2Gi } + # # affinity / tolerations optional - inherit the base when omitted + # - name: light + # node_selector: + # sysdig.com/node-size: light + # resources: + # shield: + # limits: { memory: 512Mi } + profiles: [] # The host aliases for the linux host shield workloads host_aliases: [] # The annotations for the host shield workloads (metadata.annotations)