host-device: copy host interface IP addresses and routes into container by SchSeba · Pull Request #1257 · containernetworking/plugins

SchSeba · 2026-05-04T14:29:25Z

Add a new configuration option useInterfaceNetwork that instructs the host-device plugin to capture the interface's IP addresses and routes from the host before moving the device into the container namespace, and then apply them inside the container.

This is critical for virtual environments (AWS, IBM Cloud, GPC) where the cloud provider configures IP addresses and routes directly on the network device. In these environments, there is no traditional IPAM source; the ground truth for L3 configuration lives on the host interface itself.

When useInterfaceNetwork is enabled, the plugin:

Captures all global-scope addresses and non-local routes from the host device before moving it into the container namespace.
Applies the captured addresses and routes to the interface inside the container.
Reports the addresses and routes in the CNI result (merged with any IPAM result if an IPAM plugin is also configured).

NOTE: The interface configuration on the host node must be persistent. When the device is moved back to the host (via DEL) and renamed to its original name, the system's network management service (e.g. NetworkManager, systemd-networkd, cloud-init, or cloud-specific agents) is expected to detect the device and re-apply the IP addresses and routes. This plugin does NOT re-configure the host interface on DEL; it relies on the node's network configuration being declarative and reconciled by the platform's networking stack.

Also implements the STATUS command to verify the host device exists, replacing the previous TODO stub.

SchSeba · 2026-05-04T18:47:35Z

Hi @s1061123 @squeed @LionelJouin if you have time please take a look on the PR.
This is critical for us to support virtual clusters running on clouds where the VFs are pass into the cluster VMs nodes with network configuration.

karampok · 2026-05-05T08:15:33Z

+	localRouteTable = 255
+)
+
+// HostNetworkStateFile holds the captured host-side L3 configuration


personal preference no comment unless exported

Done - removed doc comments from all unexported symbols.

karampok · 2026-05-05T08:15:33Z

+
+// HostNetworkStateFile holds the captured host-side L3 configuration
+// (addresses, routes, and rules) that should be applied to the container interface.
+type HostNetworkStateFile struct {


prefix ^File$ not very nice not really a file. May InterfaceInfo, InterfaceConfig, or just Interface

Good call - renamed HostNetworkStateFile to HostNetworkState throughout.

karampok · 2026-05-05T08:15:33Z

+type HostNetworkStateFile struct {
+	HostIfName    string       `json:"hostIfName"`
+	HostLinkWasUp bool         `json:"hostLinkWasUp"`
+	Addresses     []string     `json:"addresses,omitempty"`


can we use netlink.Addr, netlink.routes, rule

The string-based representation is intentional here - netlink.Addr, netlink.Route and netlink.Rule contain net.IP / *net.IPNet fields that don't round-trip cleanly through JSON (net.IP marshals as a base64 byte array, *net.IPNet isn't directly marshalable). Using strings gives us portable, human-readable JSON and avoids coupling the serialization format to the netlink library's internal types.

We do convert back to netlink types when applying (applyOnLink), so the actual netlink interaction is the same.

I still do not think we need serialization or string

karampok · 2026-05-05T08:15:34Z

+)
+
+// TestUseInterfaceNetwork verifies useInterfaceNetwork boolean behavior.
+func TestUseInterfaceNetwork(t *testing.T) {


what exactly this test is doing?

Removed this test - it was only validating the trivial boolean guard function which is already covered implicitly by the integration tests.

karampok · 2026-05-05T08:15:34Z

+}
+
+// TestStateJSONHasNoNeighbors verifies state serialization excludes neighbors.
+func TestStateJSONHasNoNeighbors(t *testing.T) {


what exactly this test is doing? Not sure the tests in the file really add value

Removed this test as well. Kept the TestMergeNetworkState* and TestLoadConf* tests since those exercise actual logic (result merging, config parsing, DPDK rejection).

s1061123

This PR introduces new parameter, 'useInterfaceNetwork', so could you please create another PR in https://github.com/containernetworking/cni.dev/pulls to modify host-device CNI document as well?

s1061123 · 2026-05-11T13:42:02Z

 	RuntimeConfig struct {
 		DeviceID string `json:"deviceID,omitempty"`
 	} `json:"runtimeConfig,omitempty"`
+	UseInterfaceNetwork bool `json:"useInterfaceNetwork,omitempty"`


Recommend to add comment to quickly mention what is 'UseInterfaceNetwork' because the option name is not intuitive (what 'useInterfaceNetwork' is used?)

Done - added an inline comment: // When true, copy the host interface's IP addresses and routes into the container before IPAM runs.

I also open the PR for the documentation site containernetworking/cni.dev#156

karampok · 2026-05-12T12:42:26Z

I am a bit unsure if we capture everything that need to be re-applied or if capture something we should not.
For example, when IPv6, when SLAAC, routes from RA, nothing should be copied imo. (IPv6 test are missing)

Some AI generated list which seems possible

1. Neighbours not captured — GCP assigns /32 to NICs; all egress needs ARP entry for gateway; LinkSetNsFd wipes neighbour table; traffic blackholes.
  proof
  2. rp_filter not captured — AWS multi-ENI requires loose (2); container gets netns default strict (1); asymmetric return traffic silently dropped.
  proof
  3. RTPROT_RA routes copied — RA routes have expiry (expires Nsec); re-applied without lifetime; no RA daemon in container to refresh; stale routes
  persist forever. proof
  4. SLAAC addresses copied without IFA_F_PERMANENT filter — dynamic addrs (ip addr show → dynamic) become permanent in container; no renewal; never
  expire. [proof](/usr/include/linux/if_addr.h:54 — IFA_F_PERMANENT 0x80]

SchSeba · 2026-06-24T14:11:56Z

Thanks for the thorough review @karampok - addressing your IPv6/SLAAC concerns:

Addressed in the latest push:

RTPROT_RA routes — now filtered out in captureHostNetworkState. RA-learned routes carry kernel-managed lifetimes and there's no RA daemon inside the container to refresh them, so copying them would create stale permanent routes.
SLAAC addresses — now filtered via IFA_F_PERMANENT flag check. Only permanently configured addresses are captured; dynamic (SLAAC/DHCPv6) addresses are skipped since they won't be renewed inside the container.

Out of scope for this PR (can be follow-ups):

Neighbours — On GCP with /32 NICs, the gateway ARP entry is indeed wiped when the device moves namespaces. However, once traffic flows in the container, the kernel will re-do ARP resolution (the gateway is reachable via the copied routes). If specific cloud setups require pre-populated neighbour entries, that can be added as a follow-up with a new captureNeighbours option. Adding it here would increase scope and complexity significantly.
rp_filter — This is a per-interface sysctl, not part of the netlink L3 configuration. Managing sysctls is the responsibility of the tuning CNI plugin, which is the standard approach in CNI. Users on AWS multi-ENI setups can chain tuning after host-device to set rp_filter=2.

Also added IPv6-related filtering (link-local unicast routes were already skipped).

karampok

About the DEL and not restoring: this means that once we have pod using this interface and then delete the pod, the secondary interface remains unusable until node reboots. right?

karampok · 2026-06-25T13:08:17Z

 			return fmt.Errorf("failed to find host device: %v", err)
 		}

+		networkState = &HostNetworkState{


HostNetworkState is partially initialized here from hostDev (name, link-up flag), then hostDev is passed again to captureHostNetworkState which extracts more fields from the same object. Consider a single constructor (e.g. newHostNetworkState(hostDev, interfaceNetworkEnabled)) that does all initialization in one place.

karampok · 2026-06-25T13:08:17Z

+				return err
+			}
+			if cfg.IPAM.Type == "" {
+				return printLinkWithNetworkState(contDev, cfg.CNIVersion, containerNs, networkState)


This early return for the no-IPAM + useInterfaceNetwork case is buried inside the interfaceNetworkEnabled block. Consider consolidating all no-IPAM exit paths together:

if cfg.IPAM.Type == "" { if cfg.DPDKMode { return types.PrintResult(result, cfg.CNIVersion) } if interfaceNetworkEnabled { return printLinkWithNetworkState(contDev, cfg.CNIVersion, containerNs, networkState) } return printLink(contDev, cfg.CNIVersion, containerNs) }

The control flow reads top-down: no IPAM? pick the right printer.

karampok · 2026-06-25T13:08:17Z

 	return types.PrintResult(&result, cniVersion)
 }

+func routeStateToCNIRoute(route routeState) *types.Route {


routeStateToCNIRoute belongs in host-network-state.go alongside the routeState type definition and the rest of the state handling logic.

karampok · 2026-06-25T13:08:17Z

+	Rules         []ruleState  `json:"rules,omitempty"`
+}
+
+type routeState struct {


Why is routeState needed as an intermediate type instead of using types.Route directly?

The HostNetworkState is never serialized to JSON -- it is captured in cmdAdd, applied in the same call, and discarded. And routeStateToCNIRoute converts back to types.Route for printing anyway.

The only field routeState has that types.Route lacks is Source. That could be handled with a thin wrapper or by keeping netlink.Route directly for the capture/apply path.

Also, the field naming (Destination/Gateway/Metric) diverges from both CNI (Dst/GW/Priority) and netlink (Dst/Gw/Priority) conventions.

karampok · 2026-06-25T13:08:17Z

+		if route.Protocol == syscall.RTPROT_KERNEL {
+			continue
+		}
+		// Skip RA-learned routes: they carry a kernel-managed lifetime and


The comment is incorrect. There is no "RA daemon" on the receiving side -- the kernel processes Router Advertisements natively via the accept_ra sysctl.

When the interface moves into the container namespace, the router on the link continues sending RAs. If accept_ra is enabled in the container netns (kernel default), the kernel will install fresh RA routes with proper lifetimes automatically.

Skipping RA routes is correct, but for the opposite reason: you skip them because the container will get new ones with proper lifetimes, not because it won't.

karampok · 2026-06-25T13:08:17Z

+	return nil
+}
+
+func isAlreadyExistsErr(err error) bool {


The string matching fallback in isAlreadyExistsErr is unnecessary. errors.Is(err, syscall.EEXIST) is sufficient for netlink operations (vishvananda/netlink returns syscall.Errno values). The "object already exists" string does not come from any standard errno path.

Consider the controller-runtime IgnoreAlreadyExists pattern:

func ignoreExists(err error) error { if errors.Is(err, syscall.EEXIST) { return nil } return err }

Then: return ignoreExists(netlink.AddrAdd(link, &addr)) -- cleaner than the bool check and branch.

karampok · 2026-06-25T13:09:57Z

+type HostNetworkStateFile struct {
+	HostIfName    string       `json:"hostIfName"`
+	HostLinkWasUp bool         `json:"hostLinkWasUp"`
+	Addresses     []string     `json:"addresses,omitempty"`


I still do not think we need serialization or string

SchSeba · 2026-07-01T12:07:50Z

Hi @karampok thanks for the comments!
about

About the DEL and not restoring: this means that once we have pod using this interface and then delete the pod, the secondary interface remains unusable until node reboots. right?

no when the interface is back on the host network namespace networkManager for example sees the device and run the configuration again on the device it can be static,dhcp or what every is convigured in the device

karampok

Small commments but this PR look lgtm from my side

karampok · 2026-07-03T10:43:34Z

+	}
+
+	for _, rule := range state.Rules {
+		nlRule := netlink.NewRule()


state.Rules is []netlink.Rule and nlRule is also netlink.Rule -- creating a new rule and copying fields from the same type is redundant leftover from the old ruleState string type. Simplify:

for _, rule := range state.Rules { if err := ignoreExists(netlink.RuleAdd(&rule)); err != nil { return fmt.Errorf("failed to add copied rule (src=%v table=%d): %w", rule.Src, rule.Table, err) } }

karampok · 2026-07-03T10:48:11Z

Hi @karampok thanks for the comments! about

About the DEL and not restoring: this means that once we have pod using this interface and then delete the pod, the secondary interface remains unusable until node reboots. right?

no when the interface is back on the host network namespace networkManager for example sees the device and run the configuration again on the device it can be static,dhcp or what every is convigured in the device

you assume they are all using NM (and auto connect feature) which probably is not true. I would have thought there is a cloudinit script that does once on boot the network config but I have not worked with clouds for many years, I do not have strong opinion here.

SchSeba · 2026-07-05T12:25:15Z

Hi @karampok thanks for the comments! about

About the DEL and not restoring: this means that once we have pod using this interface and then delete the pod, the secondary interface remains unusable until node reboots. right?

no when the interface is back on the host network namespace networkManager for example sees the device and run the configuration again on the device it can be static,dhcp or what every is convigured in the device

you assume they are all using NM (and auto connect feature) which probably is not true. I would have thought there is a cloudinit script that does once on boot the network config but I have not worked with clouds for many years, I do not have strong opinion here.

I tested this on two cluster provides for k8s deployments (GCP,IBM cloud) and they both has the same beavior.
also as commented if we see a user request in the future where this solution doesn't work we can iterate again and implement the full cache file for the CNI_DEL to restore the configuration.

squeed · 2026-07-06T14:17:12Z

nit: looks like you accidentally added a binary file.

shame on me :P
done

Add a new configuration option `useInterfaceNetwork` that instructs the host-device plugin to capture the interface's IP addresses and routes from the host before moving the device into the container namespace, and then apply them inside the container. This is critical for virtual environments (AWS, IBM Cloud, GPC) where the cloud provider configures IP addresses and routes directly on the network device. In these environments, there is no traditional IPAM source; the ground truth for L3 configuration lives on the host interface itself. When `useInterfaceNetwork` is enabled, the plugin: - Captures all global-scope addresses and non-local routes from the host device before moving it into the container namespace. - Applies the captured addresses and routes to the interface inside the container. - Reports the addresses and routes in the CNI result (merged with any IPAM result if an IPAM plugin is also configured). NOTE: The interface configuration on the host node must be persistent. When the device is moved back to the host (via DEL) and renamed to its original name, the system's network management service (e.g. NetworkManager, systemd-networkd, cloud-init, or cloud-specific agents) is expected to detect the device and re-apply the IP addresses and routes. This plugin does NOT re-configure the host interface on DEL; it relies on the node's network configuration being declarative and reconciled by the platform's networking stack. Also implements the STATUS command to verify the host device exists, replacing the previous TODO stub. Signed-off-by: Sebastian Sch <sebassch@gmail.com>

SchSeba · 2026-07-14T14:25:51Z

Hi @s1061123 @squeed let me know if we can merge this PR thanks! :)

SchSeba force-pushed the host-device-l3-info branch 2 times, most recently from 0feea32 to df398aa Compare May 4, 2026 17:27

karampok reviewed May 5, 2026

View reviewed changes

SchSeba force-pushed the host-device-l3-info branch from df398aa to 20dc60e Compare May 11, 2026 13:04

s1061123 requested changes May 11, 2026

View reviewed changes

SchSeba force-pushed the host-device-l3-info branch 3 times, most recently from 72eec2d to 10936b3 Compare May 11, 2026 13:47

squeed requested a review from mlguerrero12 June 22, 2026 14:24

SchSeba force-pushed the host-device-l3-info branch 2 times, most recently from a2cbe4c to c7b50c6 Compare June 25, 2026 06:10

SchSeba mentioned this pull request Jun 25, 2026

docs: document host-device useInterfaceNetwork option containernetworking/cni.dev#156

Open

SchSeba force-pushed the host-device-l3-info branch from c7b50c6 to 5272625 Compare June 29, 2026 07:10

karampok reviewed Jun 30, 2026

View reviewed changes

SchSeba force-pushed the host-device-l3-info branch 2 times, most recently from 0f94e10 to d388d1f Compare July 1, 2026 23:59

karampok reviewed Jul 3, 2026

View reviewed changes

SchSeba force-pushed the host-device-l3-info branch from d388d1f to 2f1cd7e Compare July 5, 2026 14:53

squeed reviewed Jul 6, 2026

View reviewed changes

SchSeba force-pushed the host-device-l3-info branch 2 times, most recently from cfbf85c to d14016b Compare July 6, 2026 14:22

SchSeba force-pushed the host-device-l3-info branch from d14016b to 3aa77ea Compare July 7, 2026 09:22

Uh oh!

Conversation

SchSeba commented May 4, 2026

Uh oh!

SchSeba commented May 4, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

s1061123 left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

karampok commented May 12, 2026

Uh oh!

SchSeba commented Jun 24, 2026

Uh oh!

karampok left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SchSeba commented Jul 1, 2026

Uh oh!

karampok left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

karampok commented Jul 3, 2026

Uh oh!

SchSeba commented Jul 5, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

SchSeba commented Jul 14, 2026

karampok left a comment •

edited

Loading