Skip to content

DAOS-19248 test: Fix no tag match for Functional VM stages#18597

Open
phender wants to merge 25 commits into
masterfrom
hendersp/DAOS-19247-fix
Open

DAOS-19248 test: Fix no tag match for Functional VM stages#18597
phender wants to merge 25 commits into
masterfrom
hendersp/DAOS-19247-fix

Conversation

@phender

@phender phender commented Jul 1, 2026

Copy link
Copy Markdown
Contributor

Define the stages in the Test stage as scripted stages to enable running the
Functional Test stages run on VM with the getFunctionalTestStage() method.
This will enable using testsInStage() to verify the test tags will result in at
least one test to be run by this stage - fixing a current issue where VM
Functional test stages can fail with no tests matching the tags. To run the
Test RPM and Fault injection testing stages as scripted stages two new
pipeline-lib methods are being utilized: scriptedTestRpmStage() and
scriptedUnitTestStage().

Additional fixes/changes:

  • fix running the Test RPMs on EL 9.6 stage on EL 9.6 instead of 9.7
  • rename Test RPMs on EL 9.6 stage to Test RPMs on EL 9
  • rename Test RPMs on Leap 15.5 to Test RPMs on Leap 15
  • add collecting a log for the Test RPMs stages and log the distro
  • groovyLint cleanup in the Jenkinsfile

Test-tag: test_always_passes_hw
Skip-functional-on-el-8: false
Skip-functional-on-el-9: false
Skip-functional-on-leap-15: false
Skip-functional-on-sles-15: false

Steps for the author:

  • Commit message follows the guidelines.
  • Appropriate Features or Test-tag pragmas were used.
  • Appropriate Functional Test Stages were run.
  • At least two positive code reviews including at least one code owner from each category referenced in the PR.
  • Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

  • Gatekeeper requested (daos-gatekeeper added as a reviewer).

Ensure Functional VM stages are skipped if no test tags match.

Test-tag: test_always_passes_hw
Skip-functional-on-el-8: false
Skip-functional-on-el-9: false
Skip-functional-on-leap-15: false
Skip-functional-on-sles-15: false

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
@phender
phender requested a review from daltonbohning July 1, 2026 18:06
@github-actions

github-actions Bot commented Jul 1, 2026

Copy link
Copy Markdown

Ticket title is 'Update testing branches'
Status is 'Resolved'
https://daosio.atlassian.net/browse/DAOS-19248

Comment thread Jenkinsfile Outdated
Skip-test-hardware: true
Test-tag: test_load_mpi test_setup_vm

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
skip-unit-tests: true
skip-test-hardware: true
test-tag: test_load_mpi test_setup_vm

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-18597/5/display/redirect

@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Functional on EL 9 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-18597/7/display/redirect

phender added 2 commits July 2, 2026 01:24
skip-unit-tests: true
skip-test-hardware: true
test-tag: test_load_mpi test_setup_vm

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
skip-unit-tests: true
skip-test-hardware: true
skip-functional-on-el-8: false
skip-functional-on-leap-15: false
skip-functional-on-sles-15: false
test-tag: test_load_mpi test_setup_vm

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
@daosbuild3

Copy link
Copy Markdown
Collaborator

phender added 2 commits July 2, 2026 17:56
Skip-cancel-previous-builds: true
Skip-functional-test: true
SKip-fault-injection-testing: true
Skip-unit-tests: true

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
skip-unit-tests: true
skip-test-hardware: true
skip-functional-on-el-8: false
skip-functional-on-leap-15: false
skip-functional-on-sles-15: false
test-tag: test_load_mpi test_setup_vm

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
@daosbuild3

Copy link
Copy Markdown
Collaborator

@daosbuild3

Copy link
Copy Markdown
Collaborator

@phender

phender commented Jul 3, 2026

Copy link
Copy Markdown
Contributor Author

Test stage Test RPMs on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/15/execution/node/831/log

In Build 15 the Test RPMs on Leap 15.5 was corrected to install and run on 15.5 - previously (in build 13) it was installing and running on 15.6. Now the stage fails with:

[2026-07-02T23:03:56.240Z] Error: 
[2026-07-02T23:03:56.240Z]  Problem: package daos-server-tests-2.9.100-4.50.g33213bba.suse.lp156.x86_64 requires daos-server = 2.9.100-4.50.g33213bba.suse.lp156, but none of the providers can be installed
[2026-07-02T23:03:56.240Z]   - package daos-server-2.9.100-4.50.g33213bba.suse.lp156.x86_64 requires libpmemobj1 = 2.1.3-2.suse.lp156, but none of the providers can be installed
[2026-07-02T23:03:56.240Z]   - conflicting requests
[2026-07-02T23:03:56.240Z]   - nothing provides libc.so.6(GLIBC_2.38)(64bit) needed by libpmemobj1-2.1.3-2.suse.lp156.x86_64

According to https://daosio.atlassian.net/browse/DAOS-18925, we need pciutils-3.13.0-150300.13.12.1.x86_64 but that's only available in 15.6.

~> sudo dnf provides pciutils
Last metadata expiration check: 0:01:33 ago on Fri Jul  3 03:30:09 2026.
allow_vendor_change is disabled. This option is currently not supported for downgrade and distro-sync commands
pciutils-3.5.6-150300.13.6.1.x86_64 : PCI utilities for the Linux Kernel
Repo        : opensuse-oss-proxy-artifactory
Matched from:
Provide    : pciutils = 3.5.6-150300.13.6.1

pciutils-3.13.0-150300.13.12.1.x86_64 : PCI utilities for the Linux Kernel
Repo        : @System
Matched from:
Provide    : pciutils = 3.13.0-150300.13.12.1

phender added 2 commits July 2, 2026 22:48
skip-unit-tests: true
skip-test-hardware: true
skip-functional-on-el-8: false
skip-functional-on-leap-15: false
skip-functional-on-sles-15: false
test-tag: test_load_mpi test_setup_vm

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
sjip-cancel-previous-builds: true
skip-unit-tests: true
skip-test-hardware: true
skip-functional-on-el-9: true
skip-fault-injection-testing: true

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Test RPMs on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-18597/17/display/redirect

skip-unit-tests: true
skip-test-hardware: true
skip-functional-on-el-9: true
skip-fault-injection-testing: true

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Test RPMs on Leap 15.5 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos/job/PR-18597/18/display/redirect

skip-unit-tests: true
skip-test-hardware: true
skip-functional-on-el-8: false
skip-functional-on-leap-15: false
skip-functional-on-sles-15: false
test-tag: test_load_mpi test_setup_vm

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
@daosbuild3

Copy link
Copy Markdown
Collaborator

skip-unit-tests: true
skip-test-hardware: true
skip-functional-on-el-8: false
skip-functional-on-leap-15: false
skip-functional-on-sles-15: false
test-tag: test_load_mpi test_setup_vm

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
@daosbuild3

Copy link
Copy Markdown
Collaborator

@daosbuild3

Copy link
Copy Markdown
Collaborator

@phender

phender commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

Test stage Functional on EL 8 completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/20/execution/node/1087/log

This failure is expected until https://daosio.atlassian.net/browse/DAOS-15233 has been resolved.

@phender

phender commented Jul 7, 2026

Copy link
Copy Markdown
Contributor Author

Note the improved artifacts for the RPM Test RPM stages:

phender added 2 commits July 9, 2026 01:41
skip-unit-tests: true
skip-test-hardware: true
skip-functional-on-el-8: false
skip-functional-on-leap-15: false
skip-functional-on-sles-15: false
test-tag: test_load_mpi

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
skip-unit-tests: true
skip-test-hardware: true
skip-functional-on-el-8: false
skip-functional-on-leap-15: false
skip-functional-on-sles-15: false
test-tag: test_load_mpi

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
@phender
phender requested a review from grom72 July 9, 2026 16:33
daltonbohning
daltonbohning previously approved these changes Jul 9, 2026
Comment thread Jenkinsfile
imageVersion: 'el9.6',
instRpms: 'mercury-libfabric',
nodeList: env.NODELIST,
alwaysScript: 'ci/rpm/test_daos_post.sh \'Test RPMs on EL 9\'',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What I mean is

  • test_daos.sh
    • lives in daos-stack/daos
    • hardcoded reference in daos-stack/pipeline-lib
      • IMO this is bad design but is pre-existing
  • test_daos_post.sh
    • lives in daos-stack/daos
    • argument reference in daos-stack/daos

@grom72 grom72 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can you adjust PR description - it is hard to follow the goal of this changes.

Comment thread Jenkinsfile
Comment on lines +1315 to +1316
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
}
}
} // parallel
} // script

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The team seems to go back and forth on these types of comments. I've maintained them in the past only to have someone else remove them. When viewing code in a modern IDE, I think we can leave them out.

Comment thread Jenkinsfile
archiveArtifactsArgs: [
artifacts: 'Test RPMs on EL 9/']
),
'Test RPMs on Leap 15': scriptedTestRpmStage(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Something is wrong here:
We have to use the string 'Test RPMs on Leap 15' in 5 places

@phender phender Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some of this is inevitable:

  • one is used for the build parameter name (trying to keep it in sync with the stage name for ease of use)
  • three are used for the stage name - one to define a unique parallel() entry, another to define the stage name in the method linked to said entry, and a third to reference the aforementioned matching build parameter name.

The other two are used as part of the runStage() map. I did try to reduce the number of times the stages are repeated here, but the only clean way to seed the runStage map w/ the stage names from the Jenkins build parameters keys resulted in an unordered list. Having an ordered list is very useful for displayRunStage(), so having one reference in the stageOrder list won out. The other references in updateRunStage() are needed for processing special build params / modes/ commit pragmas - for this stage there is only one when handling quickFunctional().

I did keep maintainability in mind, and at least a simple search and replace should cover most updates.

Comment thread Jenkinsfile
imageVersion: 'leap15.6',
instRpms: 'mercury-libfabric',
nodeList: env.NODELIST,
alwaysScript: 'ci/rpm/test_daos_post.sh \'Test RPMs on Leap 15\'',

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's indicate when this script is used.

Suggested change
alwaysScript: 'ci/rpm/test_daos_post.sh \'Test RPMs on Leap 15\'',
alwaysPostScript: 'ci/rpm/test_daos_post.sh \'Test RPMs on Leap 15\'',

@phender phender Jul 9, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I used alwaysScript as that's the name used for the same type of script in FunctionalTestPost(), functionalTestPostV2(), parseStageInfo(), and unitTest() - converted to a camel-case groovy-style variable name. I think the consistency makes sense.

Comment thread ci/rpm/test_daos.sh
Comment on lines +14 to +16
"bash -lc 'NODE=${nodelist[0]} DAOS_PKG_VERSION=$DAOS_PKG_VERSION PYTHON_VERSION=$PYTHON_VERSION \
bash -s 2>&1 | tee /tmp/test_daos_rpms.log; exit \${PIPESTATUS[0]}'" \
< "$mydir/test_daos_node.sh"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Less bash call easier to follow.

Suggested change
"bash -lc 'NODE=${nodelist[0]} DAOS_PKG_VERSION=$DAOS_PKG_VERSION PYTHON_VERSION=$PYTHON_VERSION \
bash -s 2>&1 | tee /tmp/test_daos_rpms.log; exit \${PIPESTATUS[0]}'" \
< "$mydir/test_daos_node.sh"
"export NODE='${nodelist[0]}' DAOS_PKG_VERSION='$DAOS_PKG_VERSION' PYTHON_VERSION='$PYTHON_VERSION';
bash -s < /dev/stdin |& tee /tmp/test_daos_rpms.log;
exit \${PIPESTATUS[0]}" \
< "$mydir/test_daos_node.sh"

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure I find that to be clearer. It might just be a personal preference thing.

@daosbuild3

Copy link
Copy Markdown
Collaborator

@daosbuild3

Copy link
Copy Markdown
Collaborator

@daosbuild3

Copy link
Copy Markdown
Collaborator

@daosbuild3

Copy link
Copy Markdown
Collaborator

@daosbuild3

Copy link
Copy Markdown
Collaborator

@daosbuild3

Copy link
Copy Markdown
Collaborator

@daosbuild3

Copy link
Copy Markdown
Collaborator

@daosbuild3

Copy link
Copy Markdown
Collaborator

phender added 2 commits July 14, 2026 09:32
skip-unit-tests: true
skip-functional-on-el-8: false
skip-functional-on-leap-15: false
skip-functional-on-sles-15: false
test-tag: test_load_mpi

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
@phender

phender commented Jul 15, 2026

Copy link
Copy Markdown
Contributor Author

Can you adjust PR description - it is hard to follow the goal of this changes.

Description updated.

phender added 3 commits July 15, 2026 15:07
skip-unit-tests: true
skip-functional-on-el-8: false
skip-functional-on-leap-15: false
skip-functional-on-sles-15: false
test-tag: test_load_mpi

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
skip-unit-tests: true
skip-functional-on-el-8: false
skip-functional-on-leap-15: false
skip-functional-on-sles-15: false
test-tag: test_load_mpi

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
skip-unit-tests: true
skip-functional-on-el-8: false
skip-functional-on-leap-15: false
skip-functional-on-sles-15: false
test-tag: test_load_mpi

Signed-off-by: Phil Henderson <phillip.henderson@hpe.com>
@phender
phender requested review from daltonbohning and grom72 July 16, 2026 04:56
@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Functional Cluster Box Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/45/execution/node/1511/log

@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/45/execution/node/1608/log

@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/45/execution/node/1743/log

@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/45/execution/node/1792/log

@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Functional Cluster Box Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/46/execution/node/728/log

@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Large MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/46/execution/node/910/log

@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/46/execution/node/920/log

@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18597/46/execution/node/1007/log

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

4 participants