|
| 1 | +# Kola without Ignition (bootc / COS-3747) – Design Ideas |
| 2 | + |
| 3 | +**Note:** The current POC from the sync meeting (Mar 10, 2026) is **tag tests + Conflux glue** only (see `bootc-kola-poc-action-plan.md`). This document is design ideas for possible future work, not the current scope. |
| 4 | + |
| 5 | +This document outlines ideas for running kola tests without Ignition, reusing kola for bootc base images (QCOW2 with pre-injected SSH key), TMT plans, and sharing tests between Fedora CoreOS and bootc. It is based on the current COSA/kola codebase and the [bootc testing checklist](https://gitlab.com/fedora/bootc/tracker/-/issues/48). |
| 6 | + |
| 7 | +--- |
| 8 | + |
| 9 | +## 1. Running kola without Ignition |
| 10 | + |
| 11 | +### 1.1 Current behavior |
| 12 | + |
| 13 | +- **QEMU platform** already supports **empty userdata**: in `mantle/platform/machine/qemu/cluster.go`, when `conf.IsEmpty()` no Ignition config file is set; in `mantle/platform/qemu.go`, `builder.ConfigFile != ""` is false so no Ignition drive is attached. The VM boots with no first-boot config. |
| 14 | +- **SSH today** is provided by Ignition: `RenderUserData` in `mantle/platform/cluster.go` injects SSH keys into the config unless `NoSSHKeyInUserData` is set. So for “no Ignition” we must **not** inject keys via userdata and instead rely on keys **already in the image** (e.g. injected by bib when building the QCOW2). |
| 15 | + |
| 16 | +### 1.2 Assumptions (from task) |
| 17 | + |
| 18 | +- We get an **SSH key** that can log in as **root**. |
| 19 | +- We get a **QCOW2 image** with that SSH key already injected (e.g. by bib). |
| 20 | + |
| 21 | +### 1.3 Changes in COSA (kola/mantle) |
| 22 | + |
| 23 | +**A. “No-Ignition” / “pre-provisioned” mode** |
| 24 | + |
| 25 | +- Introduce a **runtime or flight option** (e.g. `NoIgnition bool` or `PreProvisionedImage bool`) that: |
| 26 | + - Uses **empty userdata** for the cluster (no Ignition at all). |
| 27 | + - Sets **`NoSSHKeyInUserData: true`** so no keys are added to config (keys are in the image). |
| 28 | +- When this mode is set, the **disk image is required** (e.g. `--qemu-image` / `DiskImage`); kola does not build or assume a COSA build. The same key used to build/inject the QCOW2 (e.g. in TMT) must be the one kola uses to SSH (e.g. from `SSH_AUTH_SOCK` or `--qemu-ssh-key`). |
| 29 | + |
| 30 | +**B. SSH user: root vs core** |
| 31 | + |
| 32 | +- Current code assumes **CoreOS-style** (e.g. `core` user) where keys are in Ignition. For bootc base images, login is often **root** with the injected key in `/root/.ssh/authorized_keys`. |
| 33 | +- Add a way to specify **SSH user** for the platform (e.g. in `RuntimeConfig` or QEMU options), e.g. `SSHUser string` (default `"core"`); when `NoIgnition` (or “bootc”) mode is on, default could be `"root"`. |
| 34 | +- Ensure all SSH/SCP paths (e.g. `platform.InstallFile`, `mach.SSH()`, `cluster.SSHClient`) use this user. |
| 35 | + |
| 36 | +**C. Running tests without Ignition** |
| 37 | + |
| 38 | +Today, **external tests**: |
| 39 | + |
| 40 | +1. Are registered with an **Ignition config** that installs a systemd unit (`kola-runext-<name>.service`) and optionally `data` dir. |
| 41 | +2. After boot, kola **copies** the test binary and (if any) **data** via `setupExternalTest` / `CopyDirToMachine`, and **copies kolet** via `ScpKolet`. |
| 42 | +3. `runExternalTest` runs **`kolet run-test-unit`**, which starts that **pre-installed** unit (so the unit must have been applied by Ignition). |
| 43 | + |
| 44 | +For **no-Ignition** we cannot install that unit via Ignition. Two approaches: |
| 45 | + |
| 46 | +- **Option 1 – Deploy and run over SSH (recommended)** |
| 47 | + - Do **not** register an Ignition config for these tests. |
| 48 | + - After VM is up: SCP kolet, test binary, and `data` (same as today). |
| 49 | + - Instead of starting a pre-installed unit, **run the test binary directly over SSH** (e.g. `mach.SSH(remotepath)` or run a one-shot command that sets `KOLA_EXT_DATA` and executes the binary). Optionally create a transient systemd unit via SSH and start it, then stream output back (simpler: just run the binary and capture stdout/stderr). |
| 50 | + - Reboot/soft-reboot handling can stay similar: script writes to a well-known path or uses the existing autopkgtest-style protocol; kola still waits for reboot and then continues. |
| 51 | + |
| 52 | +- **Option 2 – Install unit over SSH** |
| 53 | + - After SCP of binary and data, **write the same unit file** to e.g. `/etc/systemd/system/` and **start it** via SSH. Then keep using `kolet run-test-unit` as today. This reuses the same execution path but requires writing unit files at runtime and possibly different paths (e.g. root’s home vs `/usr/local/bin`). |
| 54 | + |
| 55 | +Recommendation: **Option 1** for simplicity and to avoid depending on systemd unit layout; only the “run test binary and capture result” part needs to be different when `NoIgnition` is set. |
| 56 | + |
| 57 | +**D. Built-in (native) tests** |
| 58 | + |
| 59 | +- Native tests (Go, in `mantle/kola/tests/`) use **kolet** to run **native functions** on the machine; they also assume the machine was brought up with Ignition (and often with specific configs, e.g. `ignition/*`). |
| 60 | +- For “no-Ignition” mode, restrict to **external tests** and/or **native tests that do not require Ignition** (see “Collect tests that can run without Ignition” below). No change to how native tests run once the machine is up; only the **bring-up** (empty config + pre-injected key) changes. |
| 61 | + |
| 62 | +**E. CheckMachine / IgnitionError** |
| 63 | + |
| 64 | +- `StartMachine` only waits for SSH and starts the journal; it does not require Ignition. So with empty config and pre-injected key, **StartMachine** remains valid. |
| 65 | +- For QEMU, **IgnitionError** is used to detect Ignition failures. When `ConfigFile == ""`, there is no Ignition; either skip IgnitionError checks in this mode or treat “no config” as success. |
| 66 | + |
| 67 | +### 1.4 Splitting kola for COSA vs bootc (optional) |
| 68 | + |
| 69 | +- **Option A – Single binary, mode flag** |
| 70 | + Keep one kola binary; add `--no-ignition` (or `--pre-provisioned-image`) and `--qemu-ssh-user=root`. When set, use empty userdata, no SSH key in userdata, and the “run test over SSH” path for external tests. Easiest to maintain. |
| 71 | + |
| 72 | +- **Option B – Separate entrypoint or package** |
| 73 | + e.g. `kola-bootc` or a subcommand `kola bootc run` that sets these options and maybe only discovers tests tagged `bootc` or `no-ignition`. Keeps COSA and bootc workflows clearly separated. |
| 74 | + |
| 75 | +- **Option C – Shared library, two CLIs** |
| 76 | + Factor a “test runner” that takes (platform, image, SSH key, SSH user, test list); one CLI for COSA (Ignition + core), one for bootc (no Ignition + root). More refactor, clearer separation long term. |
| 77 | + |
| 78 | +Recommendation: start with **Option A** (flags + runtime options); introduce a subcommand or separate binary only if the flag set or behavior diverges too much. |
| 79 | + |
| 80 | +### 1.5 Collecting tests that can run without Ignition |
| 81 | + |
| 82 | +- **External tests** |
| 83 | + - Any test that does **not** rely on `config.ign` / `config.bu` (or that has a no-op/empty config) and only needs SSH + binary + optional `data` can run without Ignition. |
| 84 | + - Add a **tag** (e.g. `bootc` or `no-ignition`) in `kola.json` or inline metadata. When running in no-Ignition mode, either: |
| 85 | + - Only list/run tests with that tag, or |
| 86 | + - Run all tests that are either tagged or have empty config (and no Ignition-only features). |
| 87 | + |
| 88 | +- **Native tests** |
| 89 | + - **Require Ignition**: most of `mantle/kola/tests/ignition/*` (units, mount, passwd, ssh, etc.), and tests that use custom UserData (e.g. upgrade, luks, reprovision). |
| 90 | + - **Can run without Ignition**: e.g. `coretest/basic` (if it only checks basic OS behavior and does not depend on Ignition-applied config), `podman` (if no Ignition), `rpmostree/status`, `metadata/contents`, and similar “generic” tests that only assume a working OS and SSH. |
| 91 | + - Add a tag (e.g. `bootc` or `no-ignition`) to native tests that are safe to run in this mode; in no-Ignition mode, filter to that tag. |
| 92 | + |
| 93 | +- **Concrete steps** |
| 94 | + - Audit `mantle/kola/tests/` and `tests/kola` (and external repos): for each test, decide if it only needs SSH + binary/data or minimal OS (no Ignition). |
| 95 | + - Mark those with a **`bootc`** (or `no-ignition`) tag. |
| 96 | + - Document the list (e.g. in `docs/kola/bootc-kola-no-ignition-design.md` or a `bootc-denylist` / allowlist). |
| 97 | + |
| 98 | +--- |
| 99 | + |
| 100 | +## 2. TMT plan (build container → build QCOW2 → run kola) |
| 101 | + |
| 102 | +Reference: [bootc-workflow-test](https://gitlab.com/fedora/bootc/tests/bootc-workflow-test). |
| 103 | + |
| 104 | +- **Proposed flow** |
| 105 | + 1. **Build container image** (e.g. with bib or existing pipeline). |
| 106 | + 2. **Build QCOW2** using bib, **injecting the SSH public key** for root (so the key used in step 4 is the one in the image). |
| 107 | + 3. **Run kola** with: |
| 108 | + - `--qemu-image <path-to-qcow2>` |
| 109 | + - `--no-ignition` (or equivalent) |
| 110 | + - `--qemu-ssh-user=root` |
| 111 | + - SSH key from the same key used in step 2 (e.g. from TMT’s SSH key or a generated key pair stored as a secret). |
| 112 | + - Pattern or tag to run only bootc-safe tests, e.g. `kola run --tag bootc` or `kola run 'ext.*'`. |
| 113 | + |
| 114 | +- **TMT plan structure** (conceptual) |
| 115 | + - One **plan** with at least: |
| 116 | + - **Discover** phase: discover tests (e.g. from a repo that contains `tests/kola` and/or uses kola’s `--exttest`). |
| 117 | + - **Prepare** phase: build container (if needed), build QCOW2 with bib + SSH key injection, export artifact path. |
| 118 | + - **Execute** phase: run kola with the options above and the built QCOW2. |
| 119 | + - Environment or Beaker-style variables can pass: path to QCOW2, path to SSH private key, kola pattern/tag. |
| 120 | + |
| 121 | +- **SSH key handling** |
| 122 | + - Generate or inject a key pair in the pipeline; store the **public** key for bib when building the QCOW2 and use the **private** key when running kola (e.g. `SSH_AUTH_SOCK` or kola’s `--qemu-ssh-key` if added). |
| 123 | + |
| 124 | +--- |
| 125 | + |
| 126 | +## 3. Adapting Fedora CoreOS tests for bootc |
| 127 | + |
| 128 | +### 3.1 Mark “bootc generic” tests |
| 129 | + |
| 130 | +- In **kola**, add a **tag** (e.g. `bootc`) in test metadata: |
| 131 | + - **Native tests**: in the `register.Test` struct, `Tags: []string{"bootc"}` (and optionally `ExcludePlatforms` / `ExcludeDistros` if needed). |
| 132 | + - **External tests**: in `kola.json` or inline `# kola:` / `## kola:` metadata, add `"tags": "... bootc"` or `tags: bootc`. |
| 133 | +- **Filtering**: when running in bootc/no-Ignition mode, run only tests that have the `bootc` tag (or the allowlist collected in 1.5). When running in normal COSA mode, the tag can be used to run the same subset as “generic” tests. |
| 134 | + |
| 135 | +### 3.2 Run from TMT |
| 136 | + |
| 137 | +- The TMT plan in section 2 runs kola; kola runs the tests tagged `bootc` (or the no-Ignition allowlist). No extra “adaptation” beyond tagging and the no-Ignition execution path. |
| 138 | + |
| 139 | +### 3.3 Copy FCOS tests to bootc base-images repo and inherit via submodule |
| 140 | + |
| 141 | +- **bootc base-images repo** ([gitlab.com/fedora/bootc/base-images](https://gitlab.com/fedora/bootc/base-images)) gets a **copy** of the tests that are “bootc generic” (or the whole `tests/kola` tree and then we maintain an allowlist/tag there). |
| 142 | +- **Fedora CoreOS (e.g. coreos-assembler or FCOS config repo)** **removes** those tests from the tree and **inherits** them via a **git submodule** pointing at the bootc base-images repo (or a dedicated `bootc/tests` repo). That way: |
| 143 | + - Bootc CI (e.g. TMT + GitLab) runs tests from the base-images repo. |
| 144 | + - FCOS still runs the same tests via the submodule (with Ignition) and can run the full suite including Ignition-only tests. |
| 145 | + |
| 146 | +### 3.4 Daily runs (GitLab CI) |
| 147 | + |
| 148 | +- Reference: [bootc-workflow-report](https://gitlab.com/fedora/bootc/tests/bootc-workflow-report). |
| 149 | +- **Trigger**: schedule (e.g. daily) in GitLab CI that: |
| 150 | + - Builds (or fetches) the container and QCOW2 with SSH key (e.g. via bib), |
| 151 | + - Runs the TMT plan (or directly kola with the same options), |
| 152 | + - Publishes results (e.g. to bootc-workflow-report or a dashboard). |
| 153 | + |
| 154 | +--- |
| 155 | + |
| 156 | +## 4. Systemd / container / filesystem note |
| 157 | + |
| 158 | +You mentioned “systemynid - not container mount file system nu can have in fiesystem”. Interpreted as: **systemd** (or the bootc environment) may **not** use a container mount filesystem or may have a different layout (e.g. no `/home/core`, or root-only). That reinforces: |
| 159 | + |
| 160 | +- Using **root** for SSH when in bootc mode. |
| 161 | +- Not relying on **Ignition** or FCOS-specific layout (e.g. `core` user, Zincati, etc.) in tests tagged `bootc`. |
| 162 | +- Keeping no-Ignition tests to **generic** checks (e.g. boot, ssh, podman, rpmostree status, metadata) that don’t assume a specific mount or filesystem layout. |
| 163 | + |
| 164 | +--- |
| 165 | + |
| 166 | +## 5. Summary of COSA changes (concrete) |
| 167 | + |
| 168 | +1. **Platform / runtime options** |
| 169 | + - Add something like `NoIgnition bool` and `SSHUser string` (e.g. in `platform.Options` or `RuntimeConfig`). |
| 170 | + - When `NoIgnition` is true: use `conf.EmptyIgnition()`, set `NoSSHKeyInUserData: true`, require a disk image, and use `SSHUser` (e.g. `root`) for SSH. |
| 171 | + |
| 172 | +2. **QEMU cluster** |
| 173 | + - Already supports empty config; ensure when `ConfigFile == ""` we don’t try to write or attach Ignition and that StartMachine still works (it only needs SSH). |
| 174 | + |
| 175 | +3. **External test execution** |
| 176 | + - When `NoIgnition` is true: after SCP of kolet + test binary + data, **run the test binary directly** over SSH (or via a transient unit created over SSH) instead of starting the Ignition-installed unit. Reuse the same reboot/soft-reboot protocol if needed. |
| 177 | + |
| 178 | +4. **CLI** |
| 179 | + - Add e.g. `--no-ignition` and `--qemu-ssh-user=root`; when `--no-ignition` is set, imply or require `--qemu-image` and (if needed) `--qemu-ssh-key`. |
| 180 | + |
| 181 | +5. **Tags** |
| 182 | + - Add and document tag `bootc` (or `no-ignition`); in no-Ignition mode, filter tests by this tag (and/or by allowlist). |
| 183 | + |
| 184 | +6. **Tests** |
| 185 | + - Audit and tag native and external tests that are safe without Ignition; document the list. |
| 186 | + |
| 187 | +This gives a clear path to “run kola without Ignition” on a pre-built QCOW2 with SSH key, a TMT plan that builds container + QCOW2 and runs kola, and a way to share and run the same tests from both FCOS and bootc repos (submodule + GitLab CI). |
0 commit comments