[WIP] Run virtio-villain tests via petri in CI#4037
Conversation
virtio-villain is a guest-side virtio fault-injection and conformance suite whose static musl init drives the virtio transports itself and feeds out-of-spec virtqueue inputs to the device model, printing a verdict per test on the serial console. This wires it into OpenVMM as an automated test suite so that malformed-descriptor handling and other virtio robustness bugs are caught by CI. The new virtio_villain_tests crate uses petri as a library plus libtest-mimic to expose one nextest case per villain test. Each case boots a single kitchen-sink OpenVMM VM with vv.test=<id> on the kernel command line, waits for it to halt, and reads the verdict from the teed serial log. Villain's initramfs and tests.tsv are supplied via VILLAIN_INITRAMFS/VILLAIN_TSV (later resolved from an openvmm-deps release), and the guest kernel is reused from the existing linux-direct test artifact. Some malformed-descriptor tests send OpenVMM's virtio worker into a non-terminating loop that the VM cannot recover from or even tear down. Rather than paper over that product bug, per-test slow-timeout overrides in nextest terminate such a run and report it as a failure, and the tests known to trigger it are marked ignored so they are skipped in CI but remain runnable locally during fix development. To make results visible on the logview website, the runner writes the per-test petri.jsonl plus the petri.passed/failed markers the uploader counts, and honors TEST_OUTPUT_PATH so CI can publish all logs uniformly. The petri initrd-override and log-source seams are generalized so a caller can supply its own fully-formed initrd and obtain per-test log directories. A resolve_virtio_villain flowey node and a thin `xflowey virtio-villain-run` command download the artifact, build OpenVMM, stage the guest kernel, and run the suite; both stay inert until openvmm-deps cuts a release containing the artifact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7b5df162-05ee-40d7-85c3-ce37b6783805
Wire the virtio-villain conformance suite into checkin-gates as a separate, parallel per-PR job on the x64 Linux KVM pool. It runs alongside the vmm_tests jobs so it doesn't extend the critical path, and publishes JUnit plus per-test petri logs. The logs artifact is named `x64-linux-kvm-virtio-villain-vmm-tests-logs` so it matches the `upload-petri-results` workflow glob (`*-vmm-tests-logs`) and villain results surface on the logview website with no workflow changes. The local-run job is generalized into `run_virtio_villain_tests` with a `nextest_profile` and optional `publish` parameter: the xflowey `virtio-villain-run` pipeline uses the Default profile and no publishing, while CI uses the Ci profile (to emit JUnit) and routes petri's per-test logs through `TEST_OUTPUT_PATH` so they can be published. The publish step is claimed by the report step, so logs upload even when tests fail. The job is inert until openvmm-deps ships the virtio-villain artifact and `OPENVMM_DEPS` is bumped; it is landed in PR CI now so its results can seed the known-failure list once the release exists. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7b5df162-05ee-40d7-85c3-ce37b6783805
Rework the per-PR virtio-villain CI job so the KVM test machines no longer need a Rust toolchain or build dependencies (protoc, mesh, etc.). Following the established vmm_tests pattern, a build machine now produces a nextest archive plus the openvmm binary as artifacts, and the KVM test job simply downloads them and runs cargo-nextest from the archive. A new build-side node (build_nextest_virtio_villain_tests) mirrors the vmm_tests archive build and publishes a NextestVirtioVillainTestsArchive. A new consume/test node stages the reused openvmm binary and guest kernel into a content dir via init_vmm_tests_env so the known-paths resolver finds them without a build, merges the villain guest artifact env, grants KVM access, and runs the archive under the ci nextest profile before publishing JUnit and logs for logview. The former local job reverts to a local-only xflowey virtio-villain-run runner. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7b5df162-05ee-40d7-85c3-ce37b6783805
The openvmm-deps 0.3.0-112 release ships the virtio-villain guest artifact, so bump OPENVMM_DEPS to make the villain resolver (and thus the CI job) live. Also fix the local xflowey virtio-villain-run runner to stage the OpenVMM binary and guest kernel into a content dir via init_vmm_tests_env, exactly as the vmm_tests local and CI runners do, so the known-paths resolver finds them through VMM_TESTS_CONTENT_DIR regardless of the nextest target triple. The previous magicpath-plus-side-effect-build approach left OpenVMM in the native target dir while the test binary built under the triple subdir, so resolution failed. Add a --filter passthrough to run a subset of villain tests locally, and apply the repo nextest config for per-test slow-timeouts. Verified end-to-end on a KVM host: a one-per-device-class subset (18 tests) all pass against the real 0.3.0-112 villain artifact. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7b5df162-05ee-40d7-85c3-ce37b6783805
There was a problem hiding this comment.
Pull request overview
Adds a new “virtio-villain” (guest-side virtio fault-injection / conformance) test suite runner that boots OpenVMM under petri, then wires it into Flowey pipelines and PR CI so results (JUnit + per-test petri logs) can be collected and published.
Changes:
- Introduces
virtio_villain_testscrate: parsestests.tsv, runs one VM per villain test via petri, and surfaces results vialibtest-mimic. - Extends petri logging to support creating additional per-test log roots after global tracing is initialized.
- Adds Flowey nodes/jobs/pipelines + CI wiring to build a nextest archive and run it on KVM runners, publishing logs for logview.
Reviewed changes
Copilot reviewed 24 out of 25 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| vmm_tests/virtio_villain_tests/tests/villain.rs | Standalone libtest-mimic harness that enumerates villain tests and runs each in its own VM with per-test logging. |
| vmm_tests/virtio_villain_tests/src/villain.rs | TSV parsing + serial marker verdict scanning, with unit tests. |
| vmm_tests/virtio_villain_tests/src/run.rs | VM bring-up and “kitchen-sink” virtio device attachment, then serial log scanning. |
| vmm_tests/virtio_villain_tests/src/lib.rs | Public module surface for the new test runner crate. |
| vmm_tests/virtio_villain_tests/src/known_failures.rs | Central list of known-failing villain tests that are marked ignored. |
| vmm_tests/virtio_villain_tests/Cargo.toml | Defines the crate and its custom harness = false libtest-mimic test target. |
| petri/src/vm/openvmm/construct.rs | Allows a provided prebuilt_initrd to override firmware-provided initrd (needed for villain PID1 initramfs). |
| petri/src/tracing.rs | Adds new_log_source() to create additional log roots without reinstalling the global subscriber. |
| flowey/flowey_lib_hvlite/src/resolve_virtio_villain.rs | Flowey node to resolve virtio-villain guest artifacts (local override or GitHub release). |
| flowey/flowey_lib_hvlite/src/lib.rs | Exposes new Flowey nodes for virtio-villain build/resolve. |
| flowey/flowey_lib_hvlite/src/build_nextest_virtio_villain_tests.rs | Builds and archives the virtio-villain nextest suite for toolchain-free execution. |
| flowey/flowey_lib_hvlite/src/_jobs/run_virtio_villain_tests.rs | Local Flowey job to build OpenVMM, resolve artifacts, and run the suite end-to-end. |
| flowey/flowey_lib_hvlite/src/_jobs/mod.rs | Registers new virtio-villain job modules. |
| flowey/flowey_lib_hvlite/src/_jobs/consume_and_test_nextest_virtio_villain_archive.rs | CI job to unpack archive and run tests on KVM test machines, publishing JUnit + logs. |
| flowey/flowey_lib_hvlite/src/_jobs/cfg_versions.rs | Bumps OPENVMM_DEPS and wires resolve_virtio_villain config to that version. |
| flowey/flowey_hvlite/src/pipelines/virtio_villain_run.rs | Adds cargo xflowey virtio-villain-run pipeline entrypoint. |
| flowey/flowey_hvlite/src/pipelines/mod.rs | Registers the new pipeline subcommand. |
| flowey/flowey_hvlite/src/pipelines/checkin_gates.rs | Integrates build+consume split into PR gates, adding a dedicated virtio-villain CI job. |
| ci-flowey/openvmm-pr.yaml | Flowey-generated ADO pipeline wiring for the new archive and test job. |
| Cargo.toml | Adds vmm_tests/virtio_villain_tests to workspace members. |
| Cargo.lock | Adds lockfile entries for the new crate and deps. |
| .github/workflows/openvmm-pr.yaml | Flowey-generated GitHub workflow updates to run the new job and publish artifacts. |
| .github/workflows/openvmm-pr-release.yaml | Flowey-generated release PR workflow updates to include the new job. |
| .github/workflows/openvmm-ci.yaml | Flowey-generated CI workflow updates to include the new job and artifact wiring. |
| .config/nextest.toml | Adds per-package overrides (slow-timeout + threads-required) for virtio-villain tests. |
Comments suppressed due to low confidence (4)
vmm_tests/virtio_villain_tests/src/known_failures.rs:55
- Known-failure reasons currently include a placeholder
(microsoft/openvmm#TODO). This reads like a real issue link but isn't actionable; it’s better to either link a real tracking issue or make the TODO explicit without looking like a PR/issue reference.
KnownFailure {
name: "P0003",
reason: "virtio-blk packed descriptor list exceeding queue size wedges \
the device (microsoft/openvmm#TODO)",
},
vmm_tests/virtio_villain_tests/src/known_failures.rs:60
- Known-failure reason includes placeholder
(microsoft/openvmm#TODO). Replace with a real issue link or an explicit TODO that doesn't look like an issue reference.
KnownFailure {
name: "T0001",
reason: "virtio-blk self-looping descriptor chain hangs the virtio \
worker unrecoverably (microsoft/openvmm#TODO)",
},
vmm_tests/virtio_villain_tests/src/known_failures.rs:65
- Known-failure reason includes placeholder
(microsoft/openvmm#TODO). Replace with a real issue link or an explicit TODO that doesn't look like an issue reference.
KnownFailure {
name: "T0002",
reason: "virtio-blk descriptor chain exceeding queue size hangs the \
virtio worker unrecoverably (microsoft/openvmm#TODO)",
},
vmm_tests/virtio_villain_tests/src/known_failures.rs:70
- Known-failure reason includes placeholder
(microsoft/openvmm#TODO). Replace with a real issue link or an explicit TODO that doesn't look like an issue reference.
KnownFailure {
name: "T0003",
reason: "virtio-blk out-of-bounds descriptor `next` index hangs the \
virtio worker unrecoverably (microsoft/openvmm#TODO)",
},
| KnownFailure { | ||
| name: "B0002", | ||
| reason: "virtio-blk sector*512+data_len 64-bit wrap wedges the device \ | ||
| (microsoft/openvmm#TODO)", | ||
| }, |
Remove the unused petri_artifacts_core and petri_artifacts_vmm_test workspace dependencies flagged by `cargo xtask fmt`'s unused-deps lint, which was failing CI's fmt check. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 7b5df162-05ee-40d7-85c3-ce37b6783805
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 24 out of 25 changed files in this pull request and generated 4 comments.
Comments suppressed due to low confidence (2)
vmm_tests/virtio_villain_tests/src/known_failures.rs:49
- The
reasonstrings inKNOWN_FAILURESstill contain placeholder issue links (microsoft/openvmm#TODO). Since ignored tests are intended to track product bugs, it’s important that each entry links to a real issue so the skip list doesn’t become stale.
pub const KNOWN_FAILURES: &[KnownFailure] = &[
KnownFailure {
name: "B0002",
reason: "virtio-blk sector*512+data_len 64-bit wrap wedges the device \
(microsoft/openvmm#TODO)",
vmm_tests/virtio_villain_tests/src/villain.rs:50
parse_tsvsilently maps a missing/malformeddevice_idcolumn to 0 (unwrap_or("0x0000")+unwrap_or(0)). That can hide issues with a corrupted/incorrecttests.tsvand make debugging harder. Since this is test metadata, failing fast with a line-specific error would be more actionable.
let device_id = cols.next().unwrap_or("0x0000");
let device_id = device_id
.strip_prefix("0x")
.and_then(|h| u16::from_str_radix(h, 16).ok())
.unwrap_or(0);
| CommonProfile::Release => CargoBuildProfile::Release, | ||
| CommonProfile::Debug => CargoBuildProfile::Debug, | ||
| }, | ||
| extra_env: injected_env, |
No description provided.