DAOS-18690 vos: handle DTX commit under space pressure - b28 by Nasf-Fan · Pull Request #18280 · daos-stack/daos

Nasf-Fan · 2026-05-19T05:33:47Z

If we cannot normally allocate space to hold committed DTX table, then release some old DTX entries from some container in the same pool to hold new committed ones.

The patch also uses preallocated space for TX snapshots under the case of space pressure.

Steps for the author:

Commit message follows the guidelines.
Appropriate Features or Test-tag pragmas were used.
Appropriate Functional Test Stages were run.
At least two positive code reviews including at least one code owner from each category referenced in the PR.
Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

Gatekeeper requested (daos-gatekeeper added as a reviewer).

github-actions · 2026-05-19T05:34:03Z

Ticket title is 'Aurora daos_user: SCM single target ran out of space (min:0 B) and not able to finish GC. '
Status is 'In Progress'
Labels: 'request_for_2.8.1'
https://daosio.atlassian.net/browse/DAOS-18690

daosbuild3 · 2026-05-20T04:45:42Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18280/1/testReport/

daosbuild3 · 2026-05-21T14:23:43Z

Test stage Functional Hardware Medium MD on SSD completed with status FAILURE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net//job/daos-stack/job/daos/view/change-requests/job/PR-18280/2/execution/node/1370/log

daosbuild3 · 2026-05-21T15:34:16Z

Test stage Functional Hardware Medium Verbs Provider MD on SSD completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18280/2/testReport/

janekmi · 2026-05-26T06:14:44Z

Is this a (clean) cherry pick of b36dd7b ?

Nasf-Fan · 2026-06-09T11:27:16Z

Is this a (clean) cherry pick of b36dd7b ?

No, we added more in src/tests/suite/daos_base_tx.c for 2.8 version. The other parts are the same.

knard38

Nothing blocking, just ask more details on code change that I am not sure to understand.

knard38 · 2026-06-12T08:53:12Z

+		print_message("Filling DTX committed table for the container " DF_UUID "\n",
+			      DP_UUID(conts[m].uuid));
+
+		for (n = '0'; n < '8'; n++) {


The master branch (i.e. PR #18141) uses '4' here — doubling to '8' on the backport roughly doubles the number of committed DTX entries written during the fill phase. Could you clarify whether this increase is needed because the DTX committed-table threshold or fill rate differs on release/2.8, or whether it reflects a genuine test reliability fix that should also be applied to master?

Either way, replacing the magic '8' (and the '4' on master) with a named constant and/or a short comment explaining the target fill level would help future readers understand why this specific count was chosen.

Thanks for the careful thinking.

During CI test, we found that sometimes, some VOS target had never been written before we injecting fail_loc (DAOS_DTX_NOSPACE_NOREFRESH) to simulate space pressure, as to the committed DTX table on related container shard was empty.

Unfortunately, the subsequent write after fail_loc injection (to simulate space pressure) maybe sent to such VOS target, but because of no committed DTX blob can be reused under space pressure (since the table was empty), then related test failed.

So here, we write more dkeys to generate more transactions on more redundancy groups to avoid empty committed DTX table on all VOS targets. In fact, even if we generated more dkeys, it is still not 100% guaranteed that all container shards can be written. That depends on object layout algorithm, the test targets count, related objects' ID and keys. As you may know, we have ever found some object layout imbalance issue this year and Liang fixed it. On some degree, such issue maybe not completely resolved. Means that it is still possible to hit object shard unbalance issue under some cases.

So replacing original '4' with '8' is just empiric value for test purpose. In the future, we need more controllable test mechanism instead of current full write coverage of all VOS targets.

Thanks for the detailed explanation

knard38 · 2026-06-12T08:54:35Z

+			snprintf(akey, TSIZE, "akey-%c", i);
+			for (j = 'a'; j <= 'm'; j++) {
+				memset(wbuf, j, TSIZE);
+				insert_single(dkey, akey, (j - 'a') * TSIZE, wbuf, TSIZE,


It is not obvious which key is used here as it is the result of the first initialisation loop of the function.
Could help readability to explicitly define the key value to use?

The dkey is the last one that is used in former loop when prepare committed DTX table before fail_loc (DAOS_DTX_NOSPACE_NOREFRESH) injected. In fact, any dkey can be used here, that is no matter. Because the subsequent lookup_single() always use the same dkey for verification.

knard38 · 2026-06-12T09:15:35Z

+	for (m = 0; m < CONTS; m++) {
+		ioreq_fini(&reqs[m]);
+		oids[m] = daos_test_oid_gen(conts[m].coh, OC_RP_3G1, 0, 0, arg->myrank);
+		ioreq_init(&reqs[m], conts[m].coh, oids[m], DAOS_IOD_ARRAY, arg);
+	}


Not sure to understand why this new code is needed and if it should also be changed in the master branch?
A little comment could be helpful ?

We use OC_RP_3GX to prepare committed DTX table on all VOS targets. Then inject DAOS_DTX_NOSPACE_NOREFRESH to simulate space pressure for subsequent DTX commit operation. And then we use OC_RP_3G1 to generate more transactions. Because 3G1 will make transactions to be located on single redundancy group, means that the transactions are more centralized, then more easy to trigger DTX committed blob allocation (for related logic verification).

I plan to port the changes (include above ones) to master when hit related test failure on some master PR.

knard38

LGTM

Nasf-Fan · 2026-06-16T02:58:35Z

Ping @gnailzenh , thanks!

daltonbohning · 2026-06-25T20:05:01Z

Will need to wait for merge approval for 2.8.1

daosbuild3 · 2026-07-15T08:37:16Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18280/4/testReport/

daosbuild3 · 2026-07-15T08:55:36Z

Test stage Unit Test with memcheck completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18280/4/testReport/

If we cannot normally allocate space to hold committed DTX table, then release some old DTX entries from some container in the same pool to hold new committed ones. The patch also uses preallocated space for TX snapshots under the case of space pressure. Signed-off-by: Fan Yong <fan.yong@hpe.com> Co-authored-by: hgichon <hgichon@gmail.com>

daosbuild3 · 2026-07-16T05:26:12Z

Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18280/5/testReport/

daosbuild3 · 2026-07-16T05:34:58Z

Test stage Unit Test with memcheck completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18280/5/testReport/

Nasf-Fan force-pushed the Nasf-Fan/DAOS-18690_b28 branch from a9b514b to e153041 Compare May 20, 2026 08:38

Nasf-Fan force-pushed the Nasf-Fan/DAOS-18690_b28 branch from e153041 to 3dca5b8 Compare May 22, 2026 02:38

Nasf-Fan marked this pull request as ready for review May 25, 2026 02:35

Nasf-Fan requested review from a team as code owners May 25, 2026 02:35

Nasf-Fan requested review from gnailzenh and knard38 June 10, 2026 06:28

knard38 reviewed Jun 12, 2026

View reviewed changes

knard38 previously approved these changes Jun 12, 2026

View reviewed changes

Nasf-Fan requested a review from a team June 22, 2026 16:22

daltonbohning added this to the release-2.8 milestone Jun 23, 2026

daltonbohning added the waiting-for-merge-approval Waiting for merge approval label Jun 25, 2026

daltonbohning removed the request for review from a team June 25, 2026 20:04

daltonbohning removed this from the release-2.8 milestone Jun 25, 2026

Nasf-Fan dismissed knard38’s stale review via c74033e July 15, 2026 07:46

Nasf-Fan force-pushed the Nasf-Fan/DAOS-18690_b28 branch from 3dca5b8 to c74033e Compare July 15, 2026 07:46

Nasf-Fan force-pushed the Nasf-Fan/DAOS-18690_b28 branch from c74033e to bf2d2a3 Compare July 16, 2026 04:22

Uh oh!

Conversation

Nasf-Fan commented May 19, 2026

Steps for the author:

After all prior steps are complete:

Uh oh!

github-actions Bot commented May 19, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

daosbuild3 commented May 20, 2026

Uh oh!

daosbuild3 commented May 21, 2026

Uh oh!

daosbuild3 commented May 21, 2026

Uh oh!

janekmi commented May 26, 2026

Uh oh!

Nasf-Fan commented Jun 9, 2026

Uh oh!

knard38 left a comment

Choose a reason for hiding this comment

Uh oh!

knard38 Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

Nasf-Fan Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

knard38 Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

knard38 Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

Nasf-Fan Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

knard38 Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

Nasf-Fan Jun 12, 2026

Choose a reason for hiding this comment

Uh oh!

knard38 left a comment

Choose a reason for hiding this comment

Uh oh!

Nasf-Fan commented Jun 16, 2026

Uh oh!

daltonbohning commented Jun 25, 2026

Uh oh!

daosbuild3 commented Jul 15, 2026

Uh oh!

daosbuild3 commented Jul 15, 2026

Uh oh!

daosbuild3 commented Jul 16, 2026

Uh oh!

daosbuild3 commented Jul 16, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

5 participants

github-actions Bot commented May 19, 2026 •

edited

Loading