fix(operator): stop advertised cert data removing a site from routing

docs/architecture/crds.md

@@ -322,7 +322,7 @@ A discovered SWIM peer is not automatically authorized for routing.
 | `IdentityMismatch` | Connecting | Server SAN does not match configured `serverName` |
 | `CertificateExpired` / `CertificateNotYetValid` | Connecting | Server certificate is outside its validity period |
 | `PinMismatch` | Connecting | Canonical fingerprint does not match a configured pin |
-| `AdvertisedCertMismatch` | Connecting | SWIM-advertised certificate does not match either configured rotation pin |
+| `AdvertisedCertMismatch` | Active | SWIM-advertised certificate does not match a configured pin; recorded only, since the live leaf verified |
 | `TrustMaterialMissing` | Connecting | CA, client certificate, key, server name, or pin policy is absent |
 | `TrustMaterialInvalid` | Connecting | Trust material is malformed, oversized, or uses the deprecated fingerprint format |
 

docs/architecture/operations.md

@@ -523,9 +523,8 @@ The trust bootstrap for a remote site progresses through these steps:
 
 5. **Identity-aware gateway probe passes** — the `GridSite` controller performs
    a bounded mTLS handshake. It verifies the CA chain, DNS SAN, client
-   authentication, canonical pin, and agreement with the SWIM-advertised leaf
-   certificate. Success promotes the site to `Active` with reason
-   `TlsVerified`.
+   authentication, and the canonical pin against the live leaf certificate.
+   Success promotes the site to `Active` with reason `TlsVerified`.
 
    ```yaml
    spec:
@@ -594,7 +593,7 @@ The `GridSite` controller verifies gateway reachability and identity against
 |-----------|---------------|
 | `SWIMReachable` | SWIM membership reports the peer Alive |
 | `GatewayAddressKnown` | `spec.egress.address` is non-empty |
-| `TlsVerified` | Mutual TLS handshake, chain, SAN, pin, and advertised leaf all verify |
+| `TlsVerified` | Mutual TLS handshake, chain, SAN, and live-leaf pin all verify |
 | `IdentityVerificationRequired` | Plaintext endpoint accepts TCP, but remains ineligible because its identity is not verified |
 
 Request-time authorization remains enforced by the provider gateway after the
@@ -870,10 +869,14 @@ arbitrary or ambiguous advertised strings do not become routable endpoints.
 
 **Probe behavior:** In Mutual mode, the `GridSite` controller performs a bounded
 mTLS connection to `spec.egress.address`. It verifies the configured CA,
-`serverName`, canonical live-certificate pin, and that any SWIM-advertised
-certificate matches one of the configured rotation pins. A successful probe
-reports `reason: TlsVerified`. Connection failures move an Active site to
-`Unreachable`; identity or trust failures move it to `Connecting`.
+`serverName`, and the canonical live-certificate pin. A successful probe reports
+`reason: TlsVerified`. Connection failures move an Active site to `Unreachable`;
+identity or trust failures move it to `Connecting`.
+
+A SWIM-advertised certificate that does not match a configured pin is recorded
+as `reason: AdvertisedCertMismatch` and does not change the phase: it arrives
+over gossip and is not trust material. Trust comes from the handshake above,
+where a pin mismatch on the live leaf is `PinMismatch` and still demotes.
 
 Explicit `Plaintext` mode performs only a bounded TCP connection for
 diagnostics. It never promotes a site to `Active` and is never selected as a
@@ -943,8 +946,8 @@ separate steps.
     configured CA/server name.
   - `PinMismatch`: the live leaf certificate does not match either configured
     canonical pin.
-  - `AdvertisedCertMismatch`: wait for certificate gossip to converge or
-    investigate an unexpected live gateway identity.
+  - `AdvertisedCertMismatch`: the gossiped certificate copy does not match a
+    configured pin. Diagnostic, the site stays Active.
   - `HandshakeTimeout` or `TlsProtocolError`: the TCP endpoint answered but did
     not complete the expected TLS protocol.
 

operator/src/controller/grid_site.rs

@@ -238,6 +238,26 @@ async fn evaluate_gateway(site: &GridSite, client: &Client, network: &GridNetwor
     }
 }
 
+/// SWIM-advertised leaf DER, or `None` when absent or unparseable.
+///
+/// Gossiped, so unparseable is ignored: an `Err` would skip the authenticating handshake.
+fn advertised_leaf_der(site: &GridSite) -> Option<Vec<u8>> {
+    site.status
+        .as_ref()
+        .and_then(|s| s.public_cert_pem.as_ref())
+        .and_then(|pem| match first_cert_der_from_pem(pem) {
+            Ok(der) => Some(der),
+            Err(e) => {
+                tracing::warn!(
+                    error = %e,
+                    site = site.metadata.name.as_deref().unwrap_or_default(),
+                    "ignoring unparseable advertised certificate"
+                );
+                None
+            },
+        })
+}
+
 /// Build a `ProbeConfig` by loading trust material from Kubernetes
 /// Secrets referenced by the `GridNetwork`.
 ///
@@ -302,19 +322,14 @@ async fn build_probe_config_from_secrets(
 
     let pins = resolve_pins(site)?;
 
-    let advertised_leaf_der = site
-        .status
-        .as_ref()
-        .and_then(|s| s.public_cert_pem.as_ref())
-        .map(|pem| first_cert_der_from_pem(pem).map_err(|_err| O::TrustMaterialInvalid))
-        .transpose()?;
+    let advertised = advertised_leaf_der(site);
 
     Ok(crate::resources::tls_probe::ProbeConfig {
         address: addr.to_owned(),
         tls_config,
         server_name,
         pins,
-        advertised_leaf_der,
+        advertised_leaf_der: advertised,
     })
 }
 
@@ -525,7 +540,10 @@ async fn update_status(
 /// [`Warning`]: EventType::Warning
 fn event_type_for_reason(reason: &str) -> EventType {
     match reason {
-        "TlsVerified" | "AwaitingDiscovery" | "GatewayAddressKnown" | "Left" => EventType::Normal,
+        // AdvertisedCertMismatch is a success path, so Warning would be false alarm.
+        "TlsVerified" | "AwaitingDiscovery" | "GatewayAddressKnown" | "Left" | "AdvertisedCertMismatch" => {
+            EventType::Normal
+        },
         _ => EventType::Warning,
     }
 }
@@ -977,15 +995,96 @@ mod tests {
         assert_eq!(reason, "PinMismatch");
     }
 
+    /// A site whose only relevant property is its advertised certificate.
+    fn site_with_advertised(pem: Option<&str>) -> GridSite {
+        GridSite {
+            status: Some(GridSiteStatus {
+                public_cert_pem: pem.map(ToOwned::to_owned),
+                ..Default::default()
+            }),
+            ..site_no_egress(None)
+        }
+    }
+
+    /// Absent advertised material yields no DER, and no error.
+    #[test]
+    fn advertised_leaf_absent_is_none() {
+        assert!(advertised_leaf_der(&site_no_egress(None)).is_none(), "no status");
+        assert!(
+            advertised_leaf_der(&site_with_advertised(None)).is_none(),
+            "status, no PEM"
+        );
+    }
+
+    /// A real advertised certificate parses to the expected DER.
+    #[test]
+    fn advertised_leaf_valid_is_parsed() {
+        let ca = certs::generate_ca("t").unwrap_or_else(|_| std::process::abort());
+        let leaf = certs::generate_site_cert(&ca, "peer").unwrap_or_else(|_| std::process::abort());
+        let want = first_cert_der_from_pem(&leaf.cert_pem).unwrap_or_else(|_| std::process::abort());
+        assert_eq!(
+            advertised_leaf_der(&site_with_advertised(Some(&leaf.cert_pem))),
+            Some(want)
+        );
+    }
+
+    /// A chain PEM yields the leaf, not an intermediate.
+    #[test]
+    fn advertised_leaf_of_chain_is_the_leaf() {
+        let ca = certs::generate_ca("t").unwrap_or_else(|_| std::process::abort());
+        let leaf = certs::generate_site_cert(&ca, "peer").unwrap_or_else(|_| std::process::abort());
+        let chain = format!("{}{}", leaf.cert_pem, ca.cert_pem);
+        let want = first_cert_der_from_pem(&leaf.cert_pem).unwrap_or_else(|_| std::process::abort());
+        assert_eq!(advertised_leaf_der(&site_with_advertised(Some(&chain))), Some(want));
+    }
+
+    /// An unparseable advertised PEM is ignored, not surfaced as an error.
+    #[test]
+    fn unparseable_advertised_leaf_is_ignored() {
+        let bad = "-----BEGIN CERTIFICATE-----\nMIIBIjANBgkqhkiG9\n-----END CERTIFICATE-----";
+        assert!(
+            first_cert_der_from_pem(bad).is_err(),
+            "fixture must be the unparseable case this guards"
+        );
+        assert!(
+            advertised_leaf_der(&site_with_advertised(Some(bad))).is_none(),
+            "unparseable advertised material must be ignored, never surfaced as an error"
+        );
+    }
+
     #[test]
-    fn advertised_cert_mismatch_demotes_active_to_connecting() {
+    fn advertised_cert_mismatch_promotes_connecting_to_active() {
+        let site = site_with_egress(Some(GridSitePhase::Connecting), "10.0.0.1:8443");
+        let (phase, reason, _msg) = site_phase_next(
+            &GridSitePhase::Connecting,
+            &site,
+            Some(&GatewayProbeOutcome::AdvertisedCertificateMismatch),
+        );
+        assert_eq!(phase, GridSitePhase::Active);
+        assert_eq!(reason, "AdvertisedCertMismatch");
+    }
+
+    #[test]
+    fn advertised_cert_mismatch_recovers_unreachable_to_active() {
+        let site = site_with_egress(Some(GridSitePhase::Unreachable), "10.0.0.1:8443");
+        let (phase, reason, _msg) = site_phase_next(
+            &GridSitePhase::Unreachable,
+            &site,
+            Some(&GatewayProbeOutcome::AdvertisedCertificateMismatch),
+        );
+        assert_eq!(phase, GridSitePhase::Active);
+        assert_eq!(reason, "AdvertisedCertMismatch");
+    }
+
+    #[test]
+    fn advertised_cert_mismatch_keeps_active() {
         let site = site_with_egress(Some(GridSitePhase::Active), "10.0.0.1:8443");
         let (phase, reason, _msg) = site_phase_next(
             &GridSitePhase::Active,
             &site,
             Some(&GatewayProbeOutcome::AdvertisedCertificateMismatch),
         );
-        assert_eq!(phase, GridSitePhase::Connecting);
+        assert_eq!(phase, GridSitePhase::Active);
         assert_eq!(reason, "AdvertisedCertMismatch");
     }
 
@@ -1374,6 +1473,14 @@ mod tests {
         assert!(matches!(event_type_for_reason("Left"), EventType::Normal));
     }
 
+    #[test]
+    fn event_type_advertised_cert_mismatch_is_normal() {
+        assert!(matches!(
+            event_type_for_reason("AdvertisedCertMismatch"),
+            EventType::Normal
+        ));
+    }
+
     #[test]
     fn event_type_pin_mismatch_is_warning() {
         assert!(matches!(event_type_for_reason("PinMismatch"), EventType::Warning));

operator/src/resources/gateway_probe.rs

@@ -160,9 +160,10 @@ pub(crate) fn probe_transition(current_phase: &GridSitePhase, outcome: &GatewayP
             "PinMismatch",
             "server cert fingerprint does not match any configured pin",
         ),
-        O::AdvertisedCertificateMismatch => trust_failure(
+        // Recorded, not acted on: the live leaf already matched a pin above.
+        O::AdvertisedCertificateMismatch => success(
             "AdvertisedCertMismatch",
-            "SWIM-advertised certificate does not match any configured pin",
+            "SWIM-advertised certificate does not match any configured pin; live leaf verified",
         ),
     }
 }
@@ -470,32 +471,45 @@ mod tests {
     }
 
     #[test]
-    fn advertised_cert_mismatch_demotes_active_to_connecting() {
+    fn advertised_cert_mismatch_keeps_active() {
         let t = probe_transition(
             &GridSitePhase::Active,
             &GatewayProbeOutcome::AdvertisedCertificateMismatch,
         );
         assert_eq!(
             t.phase,
-            GridSitePhase::Connecting,
-            "advertised cert mismatch must demote Active to Connecting"
+            GridSitePhase::Active,
+            "the live leaf already matched a pin; the advertised copy must not demote"
         );
         assert_eq!(
             t.reason, "AdvertisedCertMismatch",
-            "reason must be AdvertisedCertMismatch"
+            "the mismatch is still recorded in the reason"
+        );
+    }
+
+    #[test]
+    fn advertised_cert_mismatch_recovers_from_unreachable() {
+        let t = probe_transition(
+            &GridSitePhase::Unreachable,
+            &GatewayProbeOutcome::AdvertisedCertificateMismatch,
+        );
+        assert_eq!(
+            t.phase,
+            GridSitePhase::Active,
+            "the peer answered and verified, so it is no longer unreachable"
         );
     }
 
     #[test]
-    fn advertised_cert_mismatch_stays_connecting() {
+    fn advertised_cert_mismatch_does_not_block_active() {
         let t = probe_transition(
             &GridSitePhase::Connecting,
             &GatewayProbeOutcome::AdvertisedCertificateMismatch,
         );
         assert_eq!(
             t.phase,
-            GridSitePhase::Connecting,
-            "advertised cert mismatch must keep Connecting"
+            GridSitePhase::Active,
+            "reached only after chain, SAN, and live-leaf pin verified"
         );
         assert_eq!(
             t.reason, "AdvertisedCertMismatch",
@@ -574,7 +588,6 @@ mod tests {
             GatewayProbeOutcome::CertificateExpired,
             GatewayProbeOutcome::CertificateNotYetValid,
             GatewayProbeOutcome::PinMismatch,
-            GatewayProbeOutcome::AdvertisedCertificateMismatch,
         ];
         for outcome in &trust_failures {
             let t = probe_transition(&GridSitePhase::Active, outcome);
@@ -598,7 +611,6 @@ mod tests {
             GatewayProbeOutcome::CertificateExpired,
             GatewayProbeOutcome::CertificateNotYetValid,
             GatewayProbeOutcome::PinMismatch,
-            GatewayProbeOutcome::AdvertisedCertificateMismatch,
         ];
         for outcome in &trust_failures {
             let t = probe_transition(&GridSitePhase::Connecting, outcome);

operator/src/resources/tls_probe.rs

@@ -64,8 +64,8 @@ pub(crate) struct ProbeConfig {
     /// Canonical DER fingerprint pins (1–2 entries).
     pub pins: Vec<CanonicalFingerprint>,
 
-    /// Optional SWIM-advertised leaf cert DER for authorization against the
-    /// configured rotation pins.
+    /// Optional SWIM-advertised leaf cert DER, compared with the configured
+    /// rotation pins for diagnostics only.
     pub advertised_leaf_der: Option<Vec<u8>>,
 }
 
@@ -182,7 +182,8 @@ pub(crate) fn build_tls_config(
 /// 2. TLS handshake under [`PROBE_DEADLINE`] (total, including connect).
 /// 3. Extract peer leaf certificate DER.
 /// 4. Validate canonical fingerprint pin.
-/// 5. If present, verify the SWIM-advertised leaf cert matches an authorized rotation pin.
+/// 5. If present, compare the SWIM-advertised leaf with the pins and record a mismatch without failing the verified
+///    connection.
 ///
 /// Returns a `GatewayProbeOutcome` — never panics, never leaks
 /// private material.

xtask/src/env/mod.rs

@@ -6233,13 +6233,8 @@ fn env_verify_gridsite_rotation(config: &Path, site: Option<&str>) -> Result<(),
             ROTATION_POLL_TIMEOUT,
         )?;
         eprintln!("  [PASS] step 6b: TlsVerified — cert-B matches fp-B in dual-pin");
-        // NOTE: AdvertisedCertificateMismatch (SWIM-advertised cert vs configured
-        // pins) requires modifying the remote operator's TLS broadcast secret
-        // independently of the probe server cert.  This pathway has unit test
-        // coverage (advertised_cert_mismatch_demotes_active_to_connecting,
-        // advertised_cert_mismatch_stays_connecting) but no xtask verifier
-        // intentionally creates that runtime state.  The rotation verifier
-        // focuses on the live TLS handshake lifecycle.
+        // No step for AdvertisedCertificateMismatch: it needs the remote TLS broadcast
+        // secret patched apart from the probe cert, and no longer changes phase.
 
         // ── Step 7: Patch [fp-B] only → still Active/TlsVerified ────────────
         eprintln!("verify-gridsite-rotation: [7] single pin [fp-B]");