config-reloader ContainerPort ignores configReloaderExtraArgs http.listenAddr (hostNetwork port conflict)

[cesium147]

Summary

The operator always sets containerPort to the constant 8435 (ConfigReloaderDefaultPort in internal/controller/operator/factory/build/container.go) when building the config-reloader sidecar. Changing the process listen address with spec.configReloaderExtraArgs["http.listenAddr"] updates container args only. Neither containerPort nor the generated HTTP liveness/readiness probes respect the overridden port — both stay locked to 8435, so the kube-scheduler still sees a host port conflict even when the process listens elsewhere.

On nodes with hostNetwork: true, that blocks scheduling a second DaemonSet that also needs a config-reloader: Kubernetes reports that the node did not have free ports for the requested pod ports (didn't have free ports…), even when one process listens on 8435 and another on 8436 via http.listenAddr.

CommonConfigReloaderParams is shared across VMAgent, VMAuth, VMSingle, VMAlertmanager, and VMAlert. The bug lives in the common builder build.ConfigReloaderContainer and affects any component that uses configReloaderExtraArgs to change the listen address.

Environment

• Operator: v0.74.0 (also reproduced in source of v0.74.1, v0.75.0-rc1, and current master as of 2026-09-04)
• Resources: two VMAgent DaemonSets with hostNetwork: true on the same node
• Sidecar: default VictoriaMetrics config-reloader

Steps to reproduce

1. Deploy two VMAgent CRs as DaemonSets with hostNetwork: true (and otherwise valid scrape config).
2. On the second agent, override the reloader listen port, for example:

spec:
  hostNetwork: true
  configReloaderExtraArgs:
    http.listenAddr: "127.0.0.1:8436"

3. Observe PodSpec of the second DaemonSet: config-reloader still has containerPort: 8435 and probes targeting 8435, while args include --http.listenAddr=127.0.0.1:8436.
4. Scheduler fails to place the second pod on a node that already runs the first agent’s reloader on host port 8435.

Expected

ContainerPort and HTTP probe ports for config-reloader should follow the port from configReloaderExtraArgs["http.listenAddr"] when set (with fallback to 8435).

Actual

Process listens on the overridden address (127.0.0.1:8436). PodSpec still declares containerPort: 8435 and probes 8435. Scheduler rejects the pod on a node that already exposes host port 8435 for another agent’s reloader.

Proposed fix

1. Parse the port from ConfigReloaderExtraArgs["http.listenAddr"] (support :PORT, IP:PORT, [IPv6]:PORT).
2. Use that port for ContainerPort and liveness/readiness probes in addPortProbesToConfigReloaderContainer (default 8435 if unset/invalid).
3. Align scrape endpoint builders that hardcode 8435 if they scrape the sidecar by numeric port.
4. Update configReloaderMetricsURL in internal/controller/operator/factory/reconcile/config_reload.go, which also hardcodes build.ConfigReloaderDefaultPort (8435). Controllers that call WaitForConfigReloadHash (e.g. VMAuth) poll /metrics on that URL; when the sidecar listen port is overridden, they keep polling 8435 and eventually time out. configReloaderMetricsURL should resolve the configured port the same way as ContainerPort and probes.
5. Add unit tests in container_test.go for default, overridden, IPv6, and invalid values.

Related

• VictoriaMetrics/operator#1581 — IPv6 / enableTCP6 / http.listenAddr interaction (different symptom; related listen-address handling).
• VictoriaMetrics/operator#1308 — hardcoded config-reloader probes with proxy-protocol (probe port behavior; not hostNetwork / ContainerPort scheduling).

Related code

• internal/controller/operator/factory/build/container.go: ConfigReloaderDefaultPort, ConfigReloaderContainer, addPortProbesToConfigReloaderContainer

[vrutkovs]

Please don't use AI to file issues, its really hard to understand the core of the problem.

Deploy two VMAgent CRs as DaemonSets with hostNetwork: true (and otherwise valid scrape config).

What is the point of having two VMAgents on the same node, with both having hostNetwork set? Is it a theoretical issue or actually preventing some usecase?

[cesium147]

Yes, it blocks a real cluster. We run two hostNetwork VMAgent DaemonSets, and on every control-plane node they share the host network namespace.

kubeadm binds etcd, apiserver, controller-manager and scheduler metrics to 127.0.0.1. We bind kubelet and node-exporter the same way so those ports never show up on the node NIC. The scraper has to use hostNetwork and hit loopback.

vmagent-node sits on every node. Listen 127.0.0.1:8429, reloader 127.0.0.1:8435.

• node-exporter http://127.0.0.1:9100/metrics
• kubelet https://127.0.0.1:10250/metrics
• kubelet-cadvisor https://127.0.0.1:10250/metrics/cadvisor
• kubelet-probes https://127.0.0.1:10250/metrics/probes
• kubelet-resource https://127.0.0.1:10250/metrics/resource
• vmagent-node http://127.0.0.1:8429/metrics
• vmagent-node-config-reloader http://127.0.0.1:8435/metrics

vmagent-control-plane sits only on nodes with node-role.kubernetes.io/control-plane. Listen 127.0.0.1:8430, reloader 127.0.0.1:8436. Workers do not have these sockets.

• kube-etcd http://127.0.0.1:2381/metrics
• apiserver https://127.0.0.1:6443/metrics
• kube-controller-manager https://127.0.0.1:10257/metrics
• kube-scheduler https://127.0.0.1:10259/metrics
• vmagent-control-plane http://127.0.0.1:8430/metrics
• vmagent-control-plane-config-reloader http://127.0.0.1:8436/metrics

We already gave vmagent different ports, 8429 and 8430, and the reloader processes 8435 and 8436. The operator still writes containerPort: 8435 on both sidecars. Scheduler then fails the second pod with "didn't have free ports".

I can't collapse this into one DaemonSet. Put the CP jobs on a worker and they scrape dead endpoints. Drop kubelet and node-exporter from the node DS and worker nodes go dark.

[vrutkovs]

Why would you want to have vmagent-control-plane at all? You can customize the scrape configuration to avoid scraping the dead endpoints.

Also why does it need to run as a hostNetwork?

[cesium147]

node-exporter needs hostNetwork to see the host network interfaces from the pod. In the pod netns it only sees eth0 and lo, not the actual netdev. We bind node-exporter to 127.0.0.1:9100 so it does not open ports on the node NIC. The node VMAgent therefore runs hostNetwork to scrape node-exporter and kubelet (127.0.0.1:10250) via loopback on every node.

kubeadm binds etcd, apiserver, controller-manager and scheduler metrics to 127.0.0.1. Those sockets exist only on control-plane nodes. A separate VMAgent DaemonSet with a control-plane nodeSelector scrapes 127.0.0.1:2381/6443/10257/10259 there. Worker nodes do not run those services.

One DaemonSet with relabeling does not fit because the jobs use static loopback targets. There is no node metadata to drop CP targets on workers. We would have to switch discovery models or inject synthetic labels just to filter. Scheduling the CP scrape jobs only where the sockets exist avoids polling closed ports. The cluster-level VMAgent stays a normal Deployment for the ~50 VMServiceScrape platform targets; daemonSetMode on that instance would drop them.

On the PR: agreed it was too complex. I pushed a commit that finds the port by the stable reloader-http name and drops the static container-name check and the first-port fallback.

[vrutkovs]

Thank you, that explains it,

We would have to switch discovery models or inject synthetic labels just to filter

That seems to be a good trade-off for avoiding running two instances in two different modes.

Just to be clear - I'm not denying the issue doesn't exist, I just want to make sure its a niche corner-case to assign correct priority for it. An issue is an issue anyway, but its priority affects the scrutiny of PR review