Skip to content

[WIP] Run virtio-villain tests via petri in CI#4037

Open
jstarks wants to merge 5 commits into
microsoft:mainfrom
jstarks:virtio-villain-tests
Open

[WIP] Run virtio-villain tests via petri in CI#4037
jstarks wants to merge 5 commits into
microsoft:mainfrom
jstarks:virtio-villain-tests

Conversation

@jstarks

@jstarks jstarks commented Jul 26, 2026

Copy link
Copy Markdown
Member

No description provided.

jstarks and others added 4 commits July 26, 2026 06:14
virtio-villain is a guest-side virtio fault-injection and conformance
suite whose static musl init drives the virtio transports itself and
feeds out-of-spec virtqueue inputs to the device model, printing a
verdict per test on the serial console. This wires it into OpenVMM as an
automated test suite so that malformed-descriptor handling and other
virtio robustness bugs are caught by CI.

The new virtio_villain_tests crate uses petri as a library plus
libtest-mimic to expose one nextest case per villain test. Each case
boots a single kitchen-sink OpenVMM VM with vv.test=<id> on the kernel
command line, waits for it to halt, and reads the verdict from the teed
serial log. Villain's initramfs and tests.tsv are supplied via
VILLAIN_INITRAMFS/VILLAIN_TSV (later resolved from an openvmm-deps
release), and the guest kernel is reused from the existing linux-direct
test artifact.

Some malformed-descriptor tests send OpenVMM's virtio worker into a
non-terminating loop that the VM cannot recover from or even tear down.
Rather than paper over that product bug, per-test slow-timeout overrides
in nextest terminate such a run and report it as a failure, and the tests
known to trigger it are marked ignored so they are skipped in CI but
remain runnable locally during fix development.

To make results visible on the logview website, the runner writes the
per-test petri.jsonl plus the petri.passed/failed markers the uploader
counts, and honors TEST_OUTPUT_PATH so CI can publish all logs uniformly.
The petri initrd-override and log-source seams are generalized so a
caller can supply its own fully-formed initrd and obtain per-test log
directories. A resolve_virtio_villain flowey node and a thin
`xflowey virtio-villain-run` command download the artifact, build
OpenVMM, stage the guest kernel, and run the suite; both stay inert until
openvmm-deps cuts a release containing the artifact.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7b5df162-05ee-40d7-85c3-ce37b6783805
Wire the virtio-villain conformance suite into checkin-gates as a
separate, parallel per-PR job on the x64 Linux KVM pool. It runs
alongside the vmm_tests jobs so it doesn't extend the critical path, and
publishes JUnit plus per-test petri logs. The logs artifact is named
`x64-linux-kvm-virtio-villain-vmm-tests-logs` so it matches the
`upload-petri-results` workflow glob (`*-vmm-tests-logs`) and villain
results surface on the logview website with no workflow changes.

The local-run job is generalized into `run_virtio_villain_tests` with a
`nextest_profile` and optional `publish` parameter: the xflowey
`virtio-villain-run` pipeline uses the Default profile and no publishing,
while CI uses the Ci profile (to emit JUnit) and routes petri's per-test
logs through `TEST_OUTPUT_PATH` so they can be published. The publish
step is claimed by the report step, so logs upload even when tests fail.

The job is inert until openvmm-deps ships the virtio-villain artifact and
`OPENVMM_DEPS` is bumped; it is landed in PR CI now so its results can
seed the known-failure list once the release exists.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7b5df162-05ee-40d7-85c3-ce37b6783805
Rework the per-PR virtio-villain CI job so the KVM test machines no longer
need a Rust toolchain or build dependencies (protoc, mesh, etc.). Following
the established vmm_tests pattern, a build machine now produces a nextest
archive plus the openvmm binary as artifacts, and the KVM test job simply
downloads them and runs cargo-nextest from the archive.

A new build-side node (build_nextest_virtio_villain_tests) mirrors the
vmm_tests archive build and publishes a NextestVirtioVillainTestsArchive.
A new consume/test node stages the reused openvmm binary and guest kernel
into a content dir via init_vmm_tests_env so the known-paths resolver finds
them without a build, merges the villain guest artifact env, grants KVM
access, and runs the archive under the ci nextest profile before publishing
JUnit and logs for logview. The former local job reverts to a local-only
xflowey virtio-villain-run runner.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7b5df162-05ee-40d7-85c3-ce37b6783805
The openvmm-deps 0.3.0-112 release ships the virtio-villain guest artifact,
so bump OPENVMM_DEPS to make the villain resolver (and thus the CI job) live.

Also fix the local xflowey virtio-villain-run runner to stage the OpenVMM
binary and guest kernel into a content dir via init_vmm_tests_env, exactly
as the vmm_tests local and CI runners do, so the known-paths resolver finds
them through VMM_TESTS_CONTENT_DIR regardless of the nextest target triple.
The previous magicpath-plus-side-effect-build approach left OpenVMM in the
native target dir while the test binary built under the triple subdir, so
resolution failed. Add a --filter passthrough to run a subset of villain
tests locally, and apply the repo nextest config for per-test slow-timeouts.

Verified end-to-end on a KVM host: a one-per-device-class subset (18 tests)
all pass against the real 0.3.0-112 villain artifact.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7b5df162-05ee-40d7-85c3-ce37b6783805
Copilot AI review requested due to automatic review settings July 26, 2026 21:17
@jstarks
jstarks requested review from a team as code owners July 26, 2026 21:17

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Adds a new “virtio-villain” (guest-side virtio fault-injection / conformance) test suite runner that boots OpenVMM under petri, then wires it into Flowey pipelines and PR CI so results (JUnit + per-test petri logs) can be collected and published.

Changes:

  • Introduces virtio_villain_tests crate: parses tests.tsv, runs one VM per villain test via petri, and surfaces results via libtest-mimic.
  • Extends petri logging to support creating additional per-test log roots after global tracing is initialized.
  • Adds Flowey nodes/jobs/pipelines + CI wiring to build a nextest archive and run it on KVM runners, publishing logs for logview.

Reviewed changes

Copilot reviewed 24 out of 25 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
vmm_tests/virtio_villain_tests/tests/villain.rs Standalone libtest-mimic harness that enumerates villain tests and runs each in its own VM with per-test logging.
vmm_tests/virtio_villain_tests/src/villain.rs TSV parsing + serial marker verdict scanning, with unit tests.
vmm_tests/virtio_villain_tests/src/run.rs VM bring-up and “kitchen-sink” virtio device attachment, then serial log scanning.
vmm_tests/virtio_villain_tests/src/lib.rs Public module surface for the new test runner crate.
vmm_tests/virtio_villain_tests/src/known_failures.rs Central list of known-failing villain tests that are marked ignored.
vmm_tests/virtio_villain_tests/Cargo.toml Defines the crate and its custom harness = false libtest-mimic test target.
petri/src/vm/openvmm/construct.rs Allows a provided prebuilt_initrd to override firmware-provided initrd (needed for villain PID1 initramfs).
petri/src/tracing.rs Adds new_log_source() to create additional log roots without reinstalling the global subscriber.
flowey/flowey_lib_hvlite/src/resolve_virtio_villain.rs Flowey node to resolve virtio-villain guest artifacts (local override or GitHub release).
flowey/flowey_lib_hvlite/src/lib.rs Exposes new Flowey nodes for virtio-villain build/resolve.
flowey/flowey_lib_hvlite/src/build_nextest_virtio_villain_tests.rs Builds and archives the virtio-villain nextest suite for toolchain-free execution.
flowey/flowey_lib_hvlite/src/_jobs/run_virtio_villain_tests.rs Local Flowey job to build OpenVMM, resolve artifacts, and run the suite end-to-end.
flowey/flowey_lib_hvlite/src/_jobs/mod.rs Registers new virtio-villain job modules.
flowey/flowey_lib_hvlite/src/_jobs/consume_and_test_nextest_virtio_villain_archive.rs CI job to unpack archive and run tests on KVM test machines, publishing JUnit + logs.
flowey/flowey_lib_hvlite/src/_jobs/cfg_versions.rs Bumps OPENVMM_DEPS and wires resolve_virtio_villain config to that version.
flowey/flowey_hvlite/src/pipelines/virtio_villain_run.rs Adds cargo xflowey virtio-villain-run pipeline entrypoint.
flowey/flowey_hvlite/src/pipelines/mod.rs Registers the new pipeline subcommand.
flowey/flowey_hvlite/src/pipelines/checkin_gates.rs Integrates build+consume split into PR gates, adding a dedicated virtio-villain CI job.
ci-flowey/openvmm-pr.yaml Flowey-generated ADO pipeline wiring for the new archive and test job.
Cargo.toml Adds vmm_tests/virtio_villain_tests to workspace members.
Cargo.lock Adds lockfile entries for the new crate and deps.
.github/workflows/openvmm-pr.yaml Flowey-generated GitHub workflow updates to run the new job and publish artifacts.
.github/workflows/openvmm-pr-release.yaml Flowey-generated release PR workflow updates to include the new job.
.github/workflows/openvmm-ci.yaml Flowey-generated CI workflow updates to include the new job and artifact wiring.
.config/nextest.toml Adds per-package overrides (slow-timeout + threads-required) for virtio-villain tests.
Comments suppressed due to low confidence (4)

vmm_tests/virtio_villain_tests/src/known_failures.rs:55

  • Known-failure reasons currently include a placeholder (microsoft/openvmm#TODO). This reads like a real issue link but isn't actionable; it’s better to either link a real tracking issue or make the TODO explicit without looking like a PR/issue reference.
    KnownFailure {
        name: "P0003",
        reason: "virtio-blk packed descriptor list exceeding queue size wedges \
                 the device (microsoft/openvmm#TODO)",
    },

vmm_tests/virtio_villain_tests/src/known_failures.rs:60

  • Known-failure reason includes placeholder (microsoft/openvmm#TODO). Replace with a real issue link or an explicit TODO that doesn't look like an issue reference.
    KnownFailure {
        name: "T0001",
        reason: "virtio-blk self-looping descriptor chain hangs the virtio \
                 worker unrecoverably (microsoft/openvmm#TODO)",
    },

vmm_tests/virtio_villain_tests/src/known_failures.rs:65

  • Known-failure reason includes placeholder (microsoft/openvmm#TODO). Replace with a real issue link or an explicit TODO that doesn't look like an issue reference.
    KnownFailure {
        name: "T0002",
        reason: "virtio-blk descriptor chain exceeding queue size hangs the \
                 virtio worker unrecoverably (microsoft/openvmm#TODO)",
    },

vmm_tests/virtio_villain_tests/src/known_failures.rs:70

  • Known-failure reason includes placeholder (microsoft/openvmm#TODO). Replace with a real issue link or an explicit TODO that doesn't look like an issue reference.
    KnownFailure {
        name: "T0003",
        reason: "virtio-blk out-of-bounds descriptor `next` index hangs the \
                 virtio worker unrecoverably (microsoft/openvmm#TODO)",
    },

Comment thread vmm_tests/virtio_villain_tests/src/villain.rs
Comment thread vmm_tests/virtio_villain_tests/tests/villain.rs
Comment on lines +46 to +50
KnownFailure {
name: "B0002",
reason: "virtio-blk sector*512+data_len 64-bit wrap wedges the device \
(microsoft/openvmm#TODO)",
},
Comment thread vmm_tests/virtio_villain_tests/src/run.rs
Remove the unused petri_artifacts_core and petri_artifacts_vmm_test
workspace dependencies flagged by `cargo xtask fmt`'s unused-deps lint,
which was failing CI's fmt check.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 7b5df162-05ee-40d7-85c3-ce37b6783805
Copilot AI review requested due to automatic review settings July 26, 2026 21:29

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 24 out of 25 changed files in this pull request and generated 4 comments.

Comments suppressed due to low confidence (2)

vmm_tests/virtio_villain_tests/src/known_failures.rs:49

  • The reason strings in KNOWN_FAILURES still contain placeholder issue links (microsoft/openvmm#TODO). Since ignored tests are intended to track product bugs, it’s important that each entry links to a real issue so the skip list doesn’t become stale.
pub const KNOWN_FAILURES: &[KnownFailure] = &[
    KnownFailure {
        name: "B0002",
        reason: "virtio-blk sector*512+data_len 64-bit wrap wedges the device \
                 (microsoft/openvmm#TODO)",

vmm_tests/virtio_villain_tests/src/villain.rs:50

  • parse_tsv silently maps a missing/malformed device_id column to 0 (unwrap_or("0x0000") + unwrap_or(0)). That can hide issues with a corrupted/incorrect tests.tsv and make debugging harder. Since this is test metadata, failing fast with a line-specific error would be more actionable.
        let device_id = cols.next().unwrap_or("0x0000");
        let device_id = device_id
            .strip_prefix("0x")
            .and_then(|h| u16::from_str_radix(h, 16).ok())
            .unwrap_or(0);

CommonProfile::Release => CargoBuildProfile::Release,
CommonProfile::Debug => CargoBuildProfile::Debug,
},
extra_env: injected_env,
Comment thread petri/src/vm/openvmm/construct.rs
Comment thread vmm_tests/virtio_villain_tests/Cargo.toml
Comment thread vmm_tests/virtio_villain_tests/src/villain.rs
@github-actions

Copy link
Copy Markdown

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants