diff --git a/input/kube-yaml/service-exposure.md b/input/kube-yaml/service-exposure.md index c825ddf..18951ea 100644 --- a/input/kube-yaml/service-exposure.md +++ b/input/kube-yaml/service-exposure.md @@ -60,10 +60,77 @@ There are many options to consider when creating connectors using YAML, see [Con NAME STATUS ROUTING-KEY SELECTOR HOST PORT HAS MATCHING LISTENER MESSAGE backend Pending backend app=backend 8080 false No matching listeners ``` + + Understanding connector status: + + | Status | Example message | Conditions | Action | + | --- | --- | --- | --- | + | Pending | "Not Configured" | - | The Skupper controller hasn't processed the YAML yet. Wait a few seconds. | + | Error | "No matches for selector..." | - | Skupper cannot find your application pods. Check that your deployment's labels match your connector's selector. | + | Pending | "No matching listeners" | Configured | Your local setup is correct, but Skupper cannot find a remote listener using that routing key. Check the remote site listener configuration. | + | Ready | "OK" | Configured, Matched | Traffic can flow from client to server and from server to client. | + **📌 NOTE** By default, the routing key name is set to the name of the connector. If you want to use a custom routing key, set `spec.routingKey` to your custom value. + +## Observing connector lifecycle on Kubernetes sites + + +Monitor how connectors respond to backend pod changes by observing the connector status and controller logs. + +On Kubernetes sites, a connector uses a pod selector to discover backend pods dynamically. The Skupper controller watches for pod changes and updates the router configuration accordingly. + +Each matching pod gets its own `tcpConnector` entry in the router, named `connector/@`. + +**Procedure** + +1. Check connector status: + + ```bash + kubectl get connector -o yaml + ``` + + The `Configured` condition in the status reflects whether backend pods are available: + + | Condition | Meaning | + | --- | --- | + | `Configured=True` | At least one matching pod is running and ready | + | `Configured=False`, `message="No matches for selector"` | No pods match the selector, or no pods are running and ready | + +2. Observe the controller when pods are added: + + ```bash + kubectl logs deploy/skupper-controller -f + ``` + + With [debug logging](../troubleshooting/index.md#setting-log-levels) enabled, you will see: + + ``` + component=kube.site.bindings Pod selected for connector pod= + ``` + + Without debug logging, the controller updates the router configuration silently. You can confirm the router received the update by checking that the connector status transitions to `Configured=True`. + +3. Observe the controller when pods are removed: + + When all matching pods are removed (scale to zero, eviction, or crash), the controller removes the `tcpConnector` entries from the router and sets: + + ``` + Configured=False message="No matches for selector" + ``` + + With [debug logging](../troubleshooting/index.md#setting-log-levels) enabled: + + ``` + component=kube.site.bindings No pods available for target selection + ``` + + **📌 NOTE** + The log messages `Pod selected for connector`, `Pod not running for connector`, `Pod not ready for connector`, and `Stopping pod watcher` are all `Debug`-level. They are not visible unless debug logging is explicitly enabled on the controller. See [Setting controller log levels](../troubleshooting/index.md#setting-log-levels) for how to enable debug logging. + + ## Creating a listener using YAML diff --git a/input/system-yaml/service-exposure.md b/input/system-yaml/service-exposure.md index 9daec51..755d62b 100644 --- a/input/system-yaml/service-exposure.md +++ b/input/system-yaml/service-exposure.md @@ -65,6 +65,37 @@ For configuration details, see [Connector resource][connector-resource]. By default, the routing key name is set to the name of the connector. If you want to use a custom routing key, set `spec.routingKey` to your custom value. + +## Observing connector lifecycle on local system sites + + +Monitor static host connectors on local system sites to understand connection behavior. + +On local system sites, connectors specify a `host` and `port` directly rather than a pod selector. There is no dynamic pod discovery. The router maintains a persistent TCP connection to the configured host. + +**Procedure** + +1. Check the connector configuration: + + ```bash + skupper connector status + ``` + + The connector shows the configured host and port. + +2. Monitor connection behavior: + + If the host becomes unreachable, the router retries the connection automatically. There is no CR condition equivalent to `Configured=False` for host-based connectors — availability is determined by whether the router can establish a connection to the host. + +3. Check router logs for connection errors: + + ```bash + podman logs -skupper-router + # or + docker logs -skupper-router + ``` + + ## Creating a listener using YAML diff --git a/input/troubleshooting/index.md b/input/troubleshooting/index.md index b245583..3649b6d 100644 --- a/input/troubleshooting/index.md +++ b/input/troubleshooting/index.md @@ -5,8 +5,6 @@ Typically, you can create a network without referencing this troubleshooting guide. However, this guide provides some tips for situations when the network does not perform as expected. -See the resolving common problems section if you have encountered a specific issue using the `skupper` CLI. - A typical troubleshooting workflow is to check all the sites and create debug tar files. @@ -152,63 +150,240 @@ This section outlines some advanced options for checking links. ``` - -## Creating a Skupper debug tar file + +## Checking and resolving controller issues -Create a debug tar file containing diagnostic information about a Skupper site to troubleshoot issues or share with support. +Check the Skupper controller deployment, logs, and status to diagnose controller-level problems. -The `skupper debug dump` command creates a compressed tarball (`.tar.gz`) containing logs, configurations, and resource status from a site. The output file is named using the pattern `--.tar.gz`. If no filename is provided, it defaults to `skupper-dump`. +The controller watches for Skupper custom resources and translates them into the actual Kubernetes infrastructure needed to run the network. +Typically, the controller does not require debugging. +The controller deployment is named `skupper-controller`. Its location depends on how Skupper was installed. -This procedure applies to both Kubernetes and local system sites. +**Procedure** + +1. Find the controller: + + The location depends on the installation scope: + + - **Cluster-scoped install**: Check with the person who installed Skupper. Typically deployed in the `skupper` or `openshift-operators` namespace + - **Namespace-scoped install**: deployed in your namespace + +2. Check the controller pod: + + ```bash + kubectl get pods -l application=skupper-controller -n + kubectl describe pod -l application=skupper-controller -n + ``` + + The controller pod has a single container named `controller`. + +3. View logs: + + ```bash + kubectl logs -l application=skupper-controller -n -c controller + ``` + + To include logs from a previously crashed container: + + ```bash + kubectl logs -l application=skupper-controller -n -c controller --previous + ``` + + Look for lines containing `level=ERROR` or `"level":"ERROR"`. + +4. Restart the controller: + + ```bash + kubectl rollout restart deployment/skupper-controller -n + ``` + + The controller automatically recovers all existing sites and resources on startup. + + + +## Checking the router on Kubernetes sites + + +Monitor router pod health and detect when the router becomes unavailable. + +The Skupper controller monitors router pods. When no router pod is running and ready, the Site CR status is updated to reflect the router's unavailability. **Procedure** -1. Create the debug tar file for a site: +1. Check router pod readiness directly: ```bash - skupper debug dump + kubectl get pods -l skupper.io/component=router ``` - Or specify a custom filename: + A healthy site shows `2/2` ready (both `router` and `kube-adaptor` containers). A problem looks like: + + ``` + NAME READY STATUS RESTARTS AGE + skupper-router-6d9f7b8c4-xk2p9 1/2 Running 0 3m + ``` + + To see which container is not ready and why: ```bash - skupper debug dump mysite-debug + kubectl describe pod -l skupper.io/component=router ``` - The command creates a file such as `skupper-dump-default-20250526-143022.tar.gz`. + Look at the `Conditions` and `Events` sections. -2. Extract the tar file to examine its contents: + **📌 NOTE** + If you're running a high availability (HA) configuration, you'll see data for two router pods. See the site configuration documentation for details on HA setup. + +2. Check kube-adaptor logs for AMQP connectivity issues: + + The `kube-adaptor` sidecar runs inside the router pod (not `skupper-controller`). It logs errors when it cannot reach the router over AMQP: ```bash - mkdir skupper-dump - tar -xzf skupper-dump-default-20250526-143022.tar.gz -C skupper-dump - cd skupper-dump + kubectl logs -l skupper.io/component=router -c kube-adaptor ``` -3. Check the Skupper and platform versions: + Look for: - - `/versions/kubernetes.yaml` - Kubernetes version (on Kubernetes platforms) - - `/versions/skupper.yaml` - Versions of Skupper components + ``` + component=kube.adaptor.configSync level=ERROR msg="sync failed" error="Could not get management agent : ..." + ``` -4. Check the site configuration and ingress: +3. Observe router pod restarts: - - `/site-namespace/resources/Site-.yaml` - Site specification and status - - `/site-namespace/resources/RouterAccess-.yaml` - Ingress and access type configured for the site + ```bash + kubectl get events -n --field-selector reason=Killing + kubectl rollout status deploy/skupper-router + ``` -5. Check linking and service configuration: + When the router pod restarts and becomes ready, the controller sets `Running=True` and `Ready=True`. The `kube-adaptor` reconnects and re-syncs the bridge configuration. - - `/site-namespace/resources/Link-.yaml` - Link status between sites - - `/site-namespace/resources/Accessgrant-.yaml` - Access grants for tokens - - `/site-namespace/resources/AccessTokens-.yaml` - Token usage information - - `/site-namespace/resources/Connector-.yaml` - Connector configuration and status - - `/site-namespace/resources/Listener-.yaml` - Listener configuration and status + **📌 NOTE** + All existing TCP connections through the router are lost when the router pod is terminated. Clients will see `connection reset` or `EOF`. After the router restarts, new connections can be established, but existing connections are not restored. -You may notice resources that contain labels prefixed with `internal.skupper.io/`. + +## Checking the router on local system sites + -**📌 NOTE** -Labels prefixed with `internal.skupper.io/` are **internal-only**. They are subject to change without notice in future versions of Skupper. Do not modify, delete, or build automation that depends on the state or existence of these labels. +Monitor router process health on local system sites using heartbeat logs and site status. + +On local system sites, the Skupper controller monitors the router using AMQP heartbeats. The `heartbeat.client` component logs state changes when the router becomes unavailable or recovers. + +**Procedure** + +1. Check if the router container is running: + + ```bash + podman ps + # or + docker ps + ``` + + Look for a container named `-skupper-router` with status `Up`. If the container is not listed or shows status `Exited`, the router is not running. + +2. Monitor controller logs for router heartbeat status: + + ```bash + podman logs -skupper-controller -f + # or + docker logs -skupper-controller -f + ``` + + Look for: + + ``` + component=heartbeat.client Router is DOWN reason= + component=heartbeat.client Router is UP + ``` + + `Router is DOWN` is logged when the heartbeat connection to the router is lost (for example, the router process exits or is killed). `Router is UP` is logged when the heartbeat connection is re-established. + +3. Check site status: + + ```bash + skupper site status + ``` + + This shows whether the router process is running and healthy. + +4. Understand service behavior during router downtime: + + The controller stops all dependent services when the router goes down and restarts them when the router comes back up. + + **📌 NOTE** + All existing TCP connections through the router are lost when the router process exits. Clients will see `connection reset` or `EOF`. After the router restarts and the controller logs `Router is UP`, new connections can be established. + + + +## Understanding TCP client errors when backends fail + + +Diagnose backend pod failures by checking connector status and understanding the client-visible errors. + +When backend pods are removed or crash, Skupper detects the change through its pod watcher and removes the corresponding routing entries from the router. This is reflected immediately in the `Connector` status. + +**Procedure** + +1. Check connector status to understand backend availability: + + ```bash + kubectl get connector -o yaml + ``` + + The `Configured` condition transitions to `False` whenever no pods match the connector's selector: + + ```yaml + status: + status: Error + message: "No matches for selector" + selectedPods: [] + conditions: + - type: Configured + status: "False" + message: "No matches for selector" + - type: Ready + status: "False" + ``` + + When at least one backend pod is running and ready, `selectedPods` is populated and `Configured` returns to `True`: + + ```yaml + status: + status: Ready + selectedPods: + - name: backend-8485574c8b-254ms + ip: 10.244.0.9 + conditions: + - type: Configured + status: "True" + - type: Ready + status: "True" + ``` + +2. (Optional) Monitor connector status changes: + + Watch for the transition using `kubectl wait`: + + ```bash + # Wait until a backend becomes available + kubectl wait connector/ --for=condition=Configured=True --timeout=60s + + # Or detect loss of backends + kubectl wait connector/ --for=condition=Configured=False --timeout=60s + ``` + +3. Understand client-visible errors by scenario: + + | Connector `Configured` | Scenario | Client-visible error | + | --- | --- | --- | + | Transitions `True` → `False` | Graceful pod termination (SIGTERM, scale-down) | `EOF` or `connection reset` — the pod closes its socket before the router removes the backend | + | Transitions `True` → `False` | Pod crash or OOM kill | `connection reset` — the kernel sends a TCP RST | + | Transitions `True` → `False` | Router removes backend before pod terminates | `connection reset` or `EOF` depending on timing | + | Stays `False` | New connection attempted with no backends | `connection refused` — the router has no target to forward to | + + **📌 NOTE** + There is no grace period or connection draining at the Skupper router level. The `Configured=False` condition covers all backend-unavailable scenarios (removal, crash, or error). Clients must implement reconnect logic to recover from these errors. @@ -269,3 +444,110 @@ By default, the reload type is set to `manual`, meaning resources must be proces docker logs -skupper-controller | grep -i error ``` + + +## Setting controller log levels + + +Change the Skupper controller log level dynamically without restarting the controller. + +The controller watches a ConfigMap named `skupper-log-config` in its namespace for live log level changes. You just need to create (or update) that ConfigMap with the key `CONTROLLER_LOG_LEVEL`. + +**Procedure** + +1. Enable debug logging: + + If the ConfigMap does not exist, create it: + + ```bash + kubectl create configmap skupper-log-config \ + --from-literal=CONTROLLER_LOG_LEVEL=debug \ + -n + ``` + + If the ConfigMap already exists: + + ```bash + kubectl patch configmap skupper-log-config \ + -n \ + --type merge \ + -p '{"data":{"CONTROLLER_LOG_LEVEL":"debug"}}' + ``` + + The change is picked up dynamically — no restart needed. + +2. Set the desired log level: + + Valid values are: + - `debug` - Most verbose, shows all operations + - `info` - Default level, shows normal operations + - `warn` - Shows warnings and errors only + - `error` - Shows errors only + +3. Revert to default logging: + + Delete the ConfigMap to revert to `info` level: + + ```bash + kubectl delete configmap skupper-log-config -n + ``` + + + +## Creating a Skupper debug tar file + + +Create a debug tar file containing diagnostic information about a Skupper site to troubleshoot issues or share with support. + +The `skupper debug dump` command creates a compressed tarball (`.tar.gz`) containing logs, configurations, and resource status from a site. The output file is named using the pattern `--.tar.gz`. If no filename is provided, it defaults to `skupper-dump`. + +This procedure applies to both Kubernetes and local system sites. + +**Procedure** + +1. Create the debug tar file for a site: + + ```bash + skupper debug dump + ``` + + Or specify a custom filename: + + ```bash + skupper debug dump mysite-debug + ``` + + The command creates a file such as `skupper-dump-default-20250526-143022.tar.gz`. + +2. Extract the tar file to examine its contents: + + ```bash + mkdir skupper-dump + tar -xzf skupper-dump-default-20250526-143022.tar.gz -C skupper-dump + cd skupper-dump + ``` + +3. Check the Skupper and platform versions: + + - `/versions/kubernetes.yaml` - Kubernetes version (on Kubernetes platforms) + - `/versions/skupper.yaml` - Versions of Skupper components + +4. Check the site configuration and ingress: + + - `/site-namespace/resources/Site-.yaml` - Site specification and status + - `/site-namespace/resources/RouterAccess-.yaml` - Ingress and access type configured for the site + +5. Check linking and service configuration: + + - `/site-namespace/resources/Link-.yaml` - Link status between sites + - `/site-namespace/resources/Accessgrant-.yaml` - Access grants for tokens + - `/site-namespace/resources/AccessTokens-.yaml` - Token usage information + - `/site-namespace/resources/Connector-.yaml` - Connector configuration and status + - `/site-namespace/resources/Listener-.yaml` - Listener configuration and status + + +You may notice resources that contain labels prefixed with `internal.skupper.io/`. + +**📌 NOTE** +Labels prefixed with `internal.skupper.io/` are **internal-only**. They are subject to change without notice in future versions of Skupper. Do not modify, delete, or build automation that depends on the state or existence of these labels. +