diff --git a/input/kube-yaml/service-exposure.md b/input/kube-yaml/service-exposure.md
index c825ddf..18951ea 100644
--- a/input/kube-yaml/service-exposure.md
+++ b/input/kube-yaml/service-exposure.md
@@ -60,10 +60,77 @@ There are many options to consider when creating connectors using YAML, see [Con
NAME STATUS ROUTING-KEY SELECTOR HOST PORT HAS MATCHING LISTENER MESSAGE
backend Pending backend app=backend 8080 false No matching listeners
```
+
+ Understanding connector status:
+
+ | Status | Example message | Conditions | Action |
+ | --- | --- | --- | --- |
+ | Pending | "Not Configured" | - | The Skupper controller hasn't processed the YAML yet. Wait a few seconds. |
+ | Error | "No matches for selector..." | - | Skupper cannot find your application pods. Check that your deployment's labels match your connector's selector. |
+ | Pending | "No matching listeners" | Configured | Your local setup is correct, but Skupper cannot find a remote listener using that routing key. Check the remote site listener configuration. |
+ | Ready | "OK" | Configured, Matched | Traffic can flow from client to server and from server to client. |
+
**📌 NOTE**
By default, the routing key name is set to the name of the connector.
If you want to use a custom routing key, set `spec.routingKey` to your custom value.
+
+## Observing connector lifecycle on Kubernetes sites
+
+
+Monitor how connectors respond to backend pod changes by observing the connector status and controller logs.
+
+On Kubernetes sites, a connector uses a pod selector to discover backend pods dynamically. The Skupper controller watches for pod changes and updates the router configuration accordingly.
+
+Each matching pod gets its own `tcpConnector` entry in the router, named `connector/@`.
+
+**Procedure**
+
+1. Check connector status:
+
+ ```bash
+ kubectl get connector -o yaml
+ ```
+
+ The `Configured` condition in the status reflects whether backend pods are available:
+
+ | Condition | Meaning |
+ | --- | --- |
+ | `Configured=True` | At least one matching pod is running and ready |
+ | `Configured=False`, `message="No matches for selector"` | No pods match the selector, or no pods are running and ready |
+
+2. Observe the controller when pods are added:
+
+ ```bash
+ kubectl logs deploy/skupper-controller -f
+ ```
+
+ With [debug logging](../troubleshooting/index.md#setting-log-levels) enabled, you will see:
+
+ ```
+ component=kube.site.bindings Pod selected for connector pod=
+ ```
+
+ Without debug logging, the controller updates the router configuration silently. You can confirm the router received the update by checking that the connector status transitions to `Configured=True`.
+
+3. Observe the controller when pods are removed:
+
+ When all matching pods are removed (scale to zero, eviction, or crash), the controller removes the `tcpConnector` entries from the router and sets:
+
+ ```
+ Configured=False message="No matches for selector"
+ ```
+
+ With [debug logging](../troubleshooting/index.md#setting-log-levels) enabled:
+
+ ```
+ component=kube.site.bindings No pods available for target selection
+ ```
+
+ **📌 NOTE**
+ The log messages `Pod selected for connector`, `Pod not running for connector`, `Pod not ready for connector`, and `Stopping pod watcher` are all `Debug`-level. They are not visible unless debug logging is explicitly enabled on the controller. See [Setting controller log levels](../troubleshooting/index.md#setting-log-levels) for how to enable debug logging.
+
+
## Creating a listener using YAML
diff --git a/input/system-yaml/service-exposure.md b/input/system-yaml/service-exposure.md
index 9daec51..755d62b 100644
--- a/input/system-yaml/service-exposure.md
+++ b/input/system-yaml/service-exposure.md
@@ -65,6 +65,37 @@ For configuration details, see [Connector resource][connector-resource].
By default, the routing key name is set to the name of the connector.
If you want to use a custom routing key, set `spec.routingKey` to your custom value.
+
+## Observing connector lifecycle on local system sites
+
+
+Monitor static host connectors on local system sites to understand connection behavior.
+
+On local system sites, connectors specify a `host` and `port` directly rather than a pod selector. There is no dynamic pod discovery. The router maintains a persistent TCP connection to the configured host.
+
+**Procedure**
+
+1. Check the connector configuration:
+
+ ```bash
+ skupper connector status
+ ```
+
+ The connector shows the configured host and port.
+
+2. Monitor connection behavior:
+
+ If the host becomes unreachable, the router retries the connection automatically. There is no CR condition equivalent to `Configured=False` for host-based connectors — availability is determined by whether the router can establish a connection to the host.
+
+3. Check router logs for connection errors:
+
+ ```bash
+ podman logs -skupper-router
+ # or
+ docker logs -skupper-router
+ ```
+
+
## Creating a listener using YAML
diff --git a/input/troubleshooting/index.md b/input/troubleshooting/index.md
index b245583..3649b6d 100644
--- a/input/troubleshooting/index.md
+++ b/input/troubleshooting/index.md
@@ -5,8 +5,6 @@
Typically, you can create a network without referencing this troubleshooting guide.
However, this guide provides some tips for situations when the network does not perform as expected.
-See the resolving common problems section if you have encountered a specific issue using the `skupper` CLI.
-
A typical troubleshooting workflow is to check all the sites and create debug tar files.
@@ -152,63 +150,240 @@ This section outlines some advanced options for checking links.
```
-
-## Creating a Skupper debug tar file
+
+## Checking and resolving controller issues
-Create a debug tar file containing diagnostic information about a Skupper site to troubleshoot issues or share with support.
+Check the Skupper controller deployment, logs, and status to diagnose controller-level problems.
-The `skupper debug dump` command creates a compressed tarball (`.tar.gz`) containing logs, configurations, and resource status from a site. The output file is named using the pattern `--.tar.gz`. If no filename is provided, it defaults to `skupper-dump`.
+The controller watches for Skupper custom resources and translates them into the actual Kubernetes infrastructure needed to run the network.
+Typically, the controller does not require debugging.
+The controller deployment is named `skupper-controller`. Its location depends on how Skupper was installed.
-This procedure applies to both Kubernetes and local system sites.
+**Procedure**
+
+1. Find the controller:
+
+ The location depends on the installation scope:
+
+ - **Cluster-scoped install**: Check with the person who installed Skupper. Typically deployed in the `skupper` or `openshift-operators` namespace
+ - **Namespace-scoped install**: deployed in your namespace
+
+2. Check the controller pod:
+
+ ```bash
+ kubectl get pods -l application=skupper-controller -n
+ kubectl describe pod -l application=skupper-controller -n
+ ```
+
+ The controller pod has a single container named `controller`.
+
+3. View logs:
+
+ ```bash
+ kubectl logs -l application=skupper-controller -n -c controller
+ ```
+
+ To include logs from a previously crashed container:
+
+ ```bash
+ kubectl logs -l application=skupper-controller -n -c controller --previous
+ ```
+
+ Look for lines containing `level=ERROR` or `"level":"ERROR"`.
+
+4. Restart the controller:
+
+ ```bash
+ kubectl rollout restart deployment/skupper-controller -n
+ ```
+
+ The controller automatically recovers all existing sites and resources on startup.
+
+
+
+## Checking the router on Kubernetes sites
+
+
+Monitor router pod health and detect when the router becomes unavailable.
+
+The Skupper controller monitors router pods. When no router pod is running and ready, the Site CR status is updated to reflect the router's unavailability.
**Procedure**
-1. Create the debug tar file for a site:
+1. Check router pod readiness directly:
```bash
- skupper debug dump
+ kubectl get pods -l skupper.io/component=router
```
- Or specify a custom filename:
+ A healthy site shows `2/2` ready (both `router` and `kube-adaptor` containers). A problem looks like:
+
+ ```
+ NAME READY STATUS RESTARTS AGE
+ skupper-router-6d9f7b8c4-xk2p9 1/2 Running 0 3m
+ ```
+
+ To see which container is not ready and why:
```bash
- skupper debug dump mysite-debug
+ kubectl describe pod -l skupper.io/component=router
```
- The command creates a file such as `skupper-dump-default-20250526-143022.tar.gz`.
+ Look at the `Conditions` and `Events` sections.
-2. Extract the tar file to examine its contents:
+ **📌 NOTE**
+ If you're running a high availability (HA) configuration, you'll see data for two router pods. See the site configuration documentation for details on HA setup.
+
+2. Check kube-adaptor logs for AMQP connectivity issues:
+
+ The `kube-adaptor` sidecar runs inside the router pod (not `skupper-controller`). It logs errors when it cannot reach the router over AMQP:
```bash
- mkdir skupper-dump
- tar -xzf skupper-dump-default-20250526-143022.tar.gz -C skupper-dump
- cd skupper-dump
+ kubectl logs -l skupper.io/component=router -c kube-adaptor
```
-3. Check the Skupper and platform versions:
+ Look for:
- - `/versions/kubernetes.yaml` - Kubernetes version (on Kubernetes platforms)
- - `/versions/skupper.yaml` - Versions of Skupper components
+ ```
+ component=kube.adaptor.configSync level=ERROR msg="sync failed" error="Could not get management agent : ..."
+ ```
-4. Check the site configuration and ingress:
+3. Observe router pod restarts:
- - `/site-namespace/resources/Site-.yaml` - Site specification and status
- - `/site-namespace/resources/RouterAccess-.yaml` - Ingress and access type configured for the site
+ ```bash
+ kubectl get events -n --field-selector reason=Killing
+ kubectl rollout status deploy/skupper-router
+ ```
-5. Check linking and service configuration:
+ When the router pod restarts and becomes ready, the controller sets `Running=True` and `Ready=True`. The `kube-adaptor` reconnects and re-syncs the bridge configuration.
- - `/site-namespace/resources/Link-.yaml` - Link status between sites
- - `/site-namespace/resources/Accessgrant-.yaml` - Access grants for tokens
- - `/site-namespace/resources/AccessTokens-.yaml` - Token usage information
- - `/site-namespace/resources/Connector-.yaml` - Connector configuration and status
- - `/site-namespace/resources/Listener-.yaml` - Listener configuration and status
+ **📌 NOTE**
+ All existing TCP connections through the router are lost when the router pod is terminated. Clients will see `connection reset` or `EOF`. After the router restarts, new connections can be established, but existing connections are not restored.
-You may notice resources that contain labels prefixed with `internal.skupper.io/`.
+
+## Checking the router on local system sites
+
-**📌 NOTE**
-Labels prefixed with `internal.skupper.io/` are **internal-only**. They are subject to change without notice in future versions of Skupper. Do not modify, delete, or build automation that depends on the state or existence of these labels.
+Monitor router process health on local system sites using heartbeat logs and site status.
+
+On local system sites, the Skupper controller monitors the router using AMQP heartbeats. The `heartbeat.client` component logs state changes when the router becomes unavailable or recovers.
+
+**Procedure**
+
+1. Check if the router container is running:
+
+ ```bash
+ podman ps
+ # or
+ docker ps
+ ```
+
+ Look for a container named `-skupper-router` with status `Up`. If the container is not listed or shows status `Exited`, the router is not running.
+
+2. Monitor controller logs for router heartbeat status:
+
+ ```bash
+ podman logs -skupper-controller -f
+ # or
+ docker logs -skupper-controller -f
+ ```
+
+ Look for:
+
+ ```
+ component=heartbeat.client Router is DOWN reason=
+ component=heartbeat.client Router is UP
+ ```
+
+ `Router is DOWN` is logged when the heartbeat connection to the router is lost (for example, the router process exits or is killed). `Router is UP` is logged when the heartbeat connection is re-established.
+
+3. Check site status:
+
+ ```bash
+ skupper site status
+ ```
+
+ This shows whether the router process is running and healthy.
+
+4. Understand service behavior during router downtime:
+
+ The controller stops all dependent services when the router goes down and restarts them when the router comes back up.
+
+ **📌 NOTE**
+ All existing TCP connections through the router are lost when the router process exits. Clients will see `connection reset` or `EOF`. After the router restarts and the controller logs `Router is UP`, new connections can be established.
+
+
+
+## Understanding TCP client errors when backends fail
+
+
+Diagnose backend pod failures by checking connector status and understanding the client-visible errors.
+
+When backend pods are removed or crash, Skupper detects the change through its pod watcher and removes the corresponding routing entries from the router. This is reflected immediately in the `Connector` status.
+
+**Procedure**
+
+1. Check connector status to understand backend availability:
+
+ ```bash
+ kubectl get connector -o yaml
+ ```
+
+ The `Configured` condition transitions to `False` whenever no pods match the connector's selector:
+
+ ```yaml
+ status:
+ status: Error
+ message: "No matches for selector"
+ selectedPods: []
+ conditions:
+ - type: Configured
+ status: "False"
+ message: "No matches for selector"
+ - type: Ready
+ status: "False"
+ ```
+
+ When at least one backend pod is running and ready, `selectedPods` is populated and `Configured` returns to `True`:
+
+ ```yaml
+ status:
+ status: Ready
+ selectedPods:
+ - name: backend-8485574c8b-254ms
+ ip: 10.244.0.9
+ conditions:
+ - type: Configured
+ status: "True"
+ - type: Ready
+ status: "True"
+ ```
+
+2. (Optional) Monitor connector status changes:
+
+ Watch for the transition using `kubectl wait`:
+
+ ```bash
+ # Wait until a backend becomes available
+ kubectl wait connector/ --for=condition=Configured=True --timeout=60s
+
+ # Or detect loss of backends
+ kubectl wait connector/ --for=condition=Configured=False --timeout=60s
+ ```
+
+3. Understand client-visible errors by scenario:
+
+ | Connector `Configured` | Scenario | Client-visible error |
+ | --- | --- | --- |
+ | Transitions `True` → `False` | Graceful pod termination (SIGTERM, scale-down) | `EOF` or `connection reset` — the pod closes its socket before the router removes the backend |
+ | Transitions `True` → `False` | Pod crash or OOM kill | `connection reset` — the kernel sends a TCP RST |
+ | Transitions `True` → `False` | Router removes backend before pod terminates | `connection reset` or `EOF` depending on timing |
+ | Stays `False` | New connection attempted with no backends | `connection refused` — the router has no target to forward to |
+
+ **📌 NOTE**
+ There is no grace period or connection draining at the Skupper router level. The `Configured=False` condition covers all backend-unavailable scenarios (removal, crash, or error). Clients must implement reconnect logic to recover from these errors.
@@ -269,3 +444,110 @@ By default, the reload type is set to `manual`, meaning resources must be proces
docker logs -skupper-controller | grep -i error
```
+
+
+## Setting controller log levels
+
+
+Change the Skupper controller log level dynamically without restarting the controller.
+
+The controller watches a ConfigMap named `skupper-log-config` in its namespace for live log level changes. You just need to create (or update) that ConfigMap with the key `CONTROLLER_LOG_LEVEL`.
+
+**Procedure**
+
+1. Enable debug logging:
+
+ If the ConfigMap does not exist, create it:
+
+ ```bash
+ kubectl create configmap skupper-log-config \
+ --from-literal=CONTROLLER_LOG_LEVEL=debug \
+ -n
+ ```
+
+ If the ConfigMap already exists:
+
+ ```bash
+ kubectl patch configmap skupper-log-config \
+ -n \
+ --type merge \
+ -p '{"data":{"CONTROLLER_LOG_LEVEL":"debug"}}'
+ ```
+
+ The change is picked up dynamically — no restart needed.
+
+2. Set the desired log level:
+
+ Valid values are:
+ - `debug` - Most verbose, shows all operations
+ - `info` - Default level, shows normal operations
+ - `warn` - Shows warnings and errors only
+ - `error` - Shows errors only
+
+3. Revert to default logging:
+
+ Delete the ConfigMap to revert to `info` level:
+
+ ```bash
+ kubectl delete configmap skupper-log-config -n
+ ```
+
+
+
+## Creating a Skupper debug tar file
+
+
+Create a debug tar file containing diagnostic information about a Skupper site to troubleshoot issues or share with support.
+
+The `skupper debug dump` command creates a compressed tarball (`.tar.gz`) containing logs, configurations, and resource status from a site. The output file is named using the pattern `--.tar.gz`. If no filename is provided, it defaults to `skupper-dump`.
+
+This procedure applies to both Kubernetes and local system sites.
+
+**Procedure**
+
+1. Create the debug tar file for a site:
+
+ ```bash
+ skupper debug dump
+ ```
+
+ Or specify a custom filename:
+
+ ```bash
+ skupper debug dump mysite-debug
+ ```
+
+ The command creates a file such as `skupper-dump-default-20250526-143022.tar.gz`.
+
+2. Extract the tar file to examine its contents:
+
+ ```bash
+ mkdir skupper-dump
+ tar -xzf skupper-dump-default-20250526-143022.tar.gz -C skupper-dump
+ cd skupper-dump
+ ```
+
+3. Check the Skupper and platform versions:
+
+ - `/versions/kubernetes.yaml` - Kubernetes version (on Kubernetes platforms)
+ - `/versions/skupper.yaml` - Versions of Skupper components
+
+4. Check the site configuration and ingress:
+
+ - `/site-namespace/resources/Site-.yaml` - Site specification and status
+ - `/site-namespace/resources/RouterAccess-.yaml` - Ingress and access type configured for the site
+
+5. Check linking and service configuration:
+
+ - `/site-namespace/resources/Link-.yaml` - Link status between sites
+ - `/site-namespace/resources/Accessgrant-.yaml` - Access grants for tokens
+ - `/site-namespace/resources/AccessTokens-.yaml` - Token usage information
+ - `/site-namespace/resources/Connector-.yaml` - Connector configuration and status
+ - `/site-namespace/resources/Listener-.yaml` - Listener configuration and status
+
+
+You may notice resources that contain labels prefixed with `internal.skupper.io/`.
+
+**📌 NOTE**
+Labels prefixed with `internal.skupper.io/` are **internal-only**. They are subject to change without notice in future versions of Skupper. Do not modify, delete, or build automation that depends on the state or existence of these labels.
+