DAOS-18690 vos: fix engine abort in DTX commit blob fallback on ENOSPC by hgichon · Pull Request #18612 · daos-stack/daos

hgichon · 2026-07-04T06:21:29Z

Summary

Regression introduced by PR #18141 (DAOS-18690 "vos: handle DTX commit under space pressure", commit b36dd7b, 2026-05-14) — master only. Release branches (2.6.x/2.8) are not affected: their vos_dtx_commit_internal() returns -DER_NOSPACE directly on blob allocation failure without touching the aborted TX.

When the pool is physically full (ENOSPC), extending the DTX committed table during a punch/update commit crashes the whole engine with SIGABRT.

vos_dtx_extend_cmt_table() allocates a new commit blob with umem_zalloc(). On allocation failure it falls back to vos_dtx_reuse_cmt_blob() inside the same transaction. However, all three allocator backends (pmemobj, dav, dav_v2) abort the transaction automatically when a tx alloc fails (no *_NO_ABORT flag is passed by the umem wrappers). The fallback then calls umem_tx_add_ptr() / dbtree_delete() on the aborted TX, which is a fatal usage error in the allocator:

ERR  pmdk pmdk/src/common/set.c:2260 util_pool_extend() extending the pool by appending parts with headers is not supported!
ERR  pmdk pmdk/tx.c:618 tx_alloc_common() out of memory
CRIT pmdk pmdk/tx.c:1403 pmemobj_tx_add_range_direct() pmemobj_tx_add_range_direct called in invalid stage 3
ERR  *** Process 1551736 (daos_engine) received signal 6 (Aborted) ***

Backtrace (master, commit ~f6b827e3e):

pmemobj_tx_add_range_direct
libdaos_common_pmem.so(+0x2979f)            # umem_tx_add_ptr
libvos_srv.so(+0x9ca53)                     # vos_dtx_* (reuse/commit path)
vos_dtx_commit_internal+0xf91
vos_dtx_prepared+0xb7d
vos_tx_end+0x5fe
vos_obj_punch+0x263
ds_obj_punch_handler+0x840

Why CI doesn't catch it

The reuse-blob fallback is exercised in tests via the DAOS_DTX_NOSPACE_NOREFRESH fail-check, which skips the allocation entirely — the TX stays alive, so the fallback works. A real ENOSPC allocation failure aborts the TX first, which the fail-check path never reproduces.

Reproduction

Small pool (tested: 400 MB tmpfs SCM, 2 targets, master build).
Fill the pool to ENOSPC via POSIX container (dd through dfuse).
Run a create/unlink storm (a few hundred small files) so punch DTX commits force vos_dtx_extend_cmt_table() while the pool is full.
Engine aborts within seconds (invalid stage 3 + SIGABRT). A pool-full production system can be taken down by any client issuing unlinks.

Suggested fix (verified)

Add a UMEM_FLAG_NO_ABORT alloc flag, map it to POBJ_XALLOC_NO_ABORT / DAV_XALLOC_NO_ABORT in the pmem/dav/dav_v2 backends, and use it for the commit-blob allocation in vos_dtx_extend_cmt_table() so the TX survives the failure and the reuse fallback can run as designed:

src/include/daos/mem.h: #define UMEM_FLAG_NO_ABORT (((uint64_t)1) << 2)
src/common/mem.c: pmem_tx_alloc() → POBJ_XALLOC_NO_ABORT, bmem_tx_alloc()/bmem_tx_alloc_v2() → DAV_XALLOC_NO_ABORT
src/vos/vos_dtx.c vos_dtx_extend_cmt_table():
umem_zalloc(umm, DTX_CMT_BLOB_SIZE) → umem_alloc_verb(umm, UMEM_FLAG_ZERO | UMEM_FLAG_NO_ABORT, DTX_CMT_BLOB_SIZE, UMEM_DEFAULT_MBKT_ID)

With this patch the same reproduction survives: unlink storms on a full pool return -DER_NOSPACE cleanly and the engine stays up (verified on tmpfs SCM/pmemobj backend; dav paths compile-verified only).

Note: filed as a PR because GitHub issues are disabled on this repo; happy to file a Jira ticket as well if preferred.

Update (2026-07-04):

UMEM_FLAG_NO_ABORT moved to 1 << 3 to avoid sharing a bit with UMEM_XADD_NO_SNAPSHOT (1 << 2) in the same flags namespace.
Added a umem regression test (UMEM010): a failed UMEM_FLAG_NO_ABORT allocation must leave the TX alive so a subsequent alloc/umem_tx_add_ptr()/commit succeeds — this covers the TX-abort path that the DAOS_DTX_NOSPACE_NOREFRESH fail-check cannot reproduce. Passes 10/10 locally (pmem backend on tmpfs needs PMEMOBJ_CONF=sds.at_create=0).
Clarified in the commit message that the failing allocation is a plain (non-NO_ABORT) alloc inside the regular vos_tx_begin() object transaction, whose failure behavior is left at the default (abort) rather than TX_FAILURE_RETURN.

github-actions · 2026-07-04T06:21:43Z

Ticket title is 'Aurora daos_user: SCM single target ran out of space (min:0 B) and not able to finish GC. '
Status is 'Awaiting backport'
Labels: 'request_for_2.8.1'
https://daosio.atlassian.net/browse/DAOS-18690

When the pool is physically full, vos_dtx_extend_cmt_table() fails to allocate a new commit blob and falls back to vos_dtx_reuse_cmt_blob() inside the same transaction. The failing allocation is a plain (non-NO_ABORT) alloc inside the regular object modification transaction started by vos_tx_begin(), whose failure behavior is left at the default, so the allocator aborts the whole TX on failure. The fallback then calls umem_tx_add_ptr() on the aborted TX, which is a fatal usage error in the allocator: CRIT pmdk tx.c:1403 pmemobj_tx_add_range_direct() called in invalid stage 3 *** Process (daos_engine) received signal 6 (Aborted) *** vos_dtx_commit_internal <- vos_dtx_prepared <- vos_tx_end <- vos_obj_punch <- ds_obj_punch_handler The reuse fallback introduced by daos-stack#18141 is only exercised in tests via the DAOS_DTX_NOSPACE_NOREFRESH fail-check, which skips the allocation entirely and therefore never reproduces the TX abort of a real ENOSPC. Fix: add a UMEM_FLAG_NO_ABORT alloc flag (1 << 3, distinct from UMEM_XADD_NO_SNAPSHOT), map it to POBJ_XALLOC_NO_ABORT / DAV_XALLOC_NO_ABORT in the pmem/dav/dav_v2 backends, and use it for the commit blob allocation so the transaction survives the failure and the reuse fallback can run as designed. Add a umem regression test (UMEM010): a failed NO_ABORT allocation must leave the TX alive so a subsequent alloc/tx_add_ptr/commit succeeds. Verified on a 400 MB tmpfs-SCM pool: fill the pool to ENOSPC, then run a create/unlink storm. Before the fix the engine aborts within seconds with the backtrace above; after the fix it stays up and the operations fail cleanly with -DER_NOSPACE. Signed-off-by: hgichon <hgichon@gmail.com>

hgichon · 2026-07-04T07:10:40Z

Note on backport scope: the Jira bot above shows DAOS-18690 is Awaiting backport with request_for_2.8.1. Since #18141 is what introduces the in-TX reuse fallback, any release branch that picks up #18141 will also pick up this crash — before #18141 the allocation-failure path simply returned -DER_NOSPACE without touching the aborted TX.

So this fix should ride along with the #18141 backport to 2.8.1 (and any other target branches), not just land on master. Happy to prepare the 2.8-based patch as well if useful.

Nasf-Fan

@hgichon, thanks for pointing out and fixing the issue. Please consider the comment.

Nasf-Fan · 2026-07-13T16:38:01Z

 	if (flags & UMEM_FLAG_NO_FLUSH)
 		pflags |= POBJ_FLAG_NO_FLUSH;
+	if (flags & UMEM_FLAG_NO_ABORT)
+		pflags |= POBJ_XALLOC_NO_ABORT;


Can we directly use POBJ_FLAG_TX_NO_ABORT to be consistent with above POBJ_FLAG_NO_FLUSH and POBJ_FLAG_ZERO?

Sure, will switch to POBJ_FLAG_TX_NO_ABORT for consistency with POBJ_FLAG_NO_FLUSH/POBJ_FLAG_ZERO. Thanks.

Nasf-Fan · 2026-07-13T16:38:07Z

 	if (flags & UMEM_FLAG_NO_FLUSH)
 		pflags |= POBJ_FLAG_NO_FLUSH;
+	if (flags & UMEM_FLAG_NO_ABORT)
+		pflags |= POBJ_XALLOC_NO_ABORT;


Can we directly use POBJ_FLAG_TX_NO_ABORT to be consistent with above POBJ_FLAG_NO_FLUSH and POBJ_FLAG_ZERO?

Nasf-Fan · 2026-07-13T16:38:19Z

 	struct umem_instance *umm = &pool->vp_umm;
 	int                   rc;

 	rc = umem_tx_add_ptr(umm, ptr, size);


Just realize that we also need to use umem_tx_xadd_ptr() with NO_ABORT flags instead of directly call umem_tx_add_ptr(). Otherwise, we will hit similar SIGABRT when DER_NOSPACE happened. Would you please to enhance you patch for that? Thanks!

Agreed. I'll route the flag through the add path too: extend pmem_tx_xadd / bmem_tx_xadd(_v2) to map UMEM_FLAG_NO_ABORT -> POBJ_XADD_NO_ABORT / DAV_XADD_NO_ABORT, and call umem_tx_xadd_ptr(..., UMEM_FLAG_NO_ABORT) here so both backends are covered. (On DAV dav_tx_add_range already returns the error instead of aborting, so this mainly closes the PMEM abort while keeping the two backends consistent.)

Nasf-Fan · 2026-07-14T04:07:47Z

Another relative simple solution can be like the followings, then we do not need FLAG_NO_ABORT related changes.

diff --git a/src/vos/vos_dtx.c b/src/vos/vos_dtx.c
index a7e830bff7..ca442b2f6c 100644
--- a/src/vos/vos_dtx.c
+++ b/src/vos/vos_dtx.c
@@ -2574,6 +2574,9 @@ vos_dtx_commit_internal(struct vos_container *cont, struct dtx_id dtis[],
                  "Invalid committed DTX blob slots %d/%d\n",
                  dbd->dbd_cap, dbd->dbd_count);
 
+       if (umem_tx_get_failure_behavior(umm) != TX_FAILURE_RETURN)
+               umem_tx_set_failure_behavior(umm, TX_FAILURE_RETURN);
+
        if (dbd->dbd_cap == dbd->dbd_count)
                goto new_blob;

hgichon · 2026-07-14T10:16:41Z

@Nasf-Fan re: the umem_tx_set_failure_behavior(TX_FAILURE_RETURN) alternative — it only takes effect on the PMEM backend. mo_tx_set_failure_behavior / mo_tx_get_failure_behavior are wired only in pmem_ops; bmem_ops and bmem_v2_ops (DAV / metadata-on-SSD) don't implement them, so umem_tx_set_failure_behavior() is a silent no-op there and umem_tx_get_failure_behavior() returns the default TX_FAILURE_ABORT. Since dav_tx_xalloc() aborts the tx on allocation failure unless DAV_XALLOC_NO_ABORT is set (there is no tx-wide behavior knob for DAV), replacing the per-alloc flag with TX_FAILURE_RETURN would re-introduce the SIGABRT on md-on-SSD pools at the vos_dtx_extend_cmt_table() blob alloc; the vos_dtx_add_ptr() emergency-buffer fallback is likewise gated on behavior == TX_FAILURE_RETURN and never engages on DAV. That's why the patch keeps UMEM_FLAG_NO_ABORT mapped to all three backends.

So my plan is to keep the per-alloc/-add NO_ABORT flag (so DAV stays protected) plus the naming + umem_tx_xadd_ptr cleanups noted inline. If you'd additionally like to gate vos_dtx_commit_internal() on TX_FAILURE_RETURN for the PMEM path (to reuse the emergency-buffer fallback) while keeping the flag for DAV, I'm glad to add that too — just let me know your preference.

Nasf-Fan · 2026-07-14T13:18:14Z

@Nasf-Fan re: the umem_tx_set_failure_behavior(TX_FAILURE_RETURN) alternative — it only takes effect on the PMEM backend. mo_tx_set_failure_behavior / mo_tx_get_failure_behavior are wired only in pmem_ops; bmem_ops and bmem_v2_ops (DAV / metadata-on-SSD) don't implement them, so umem_tx_set_failure_behavior() is a silent no-op there and umem_tx_get_failure_behavior() returns the default TX_FAILURE_ABORT. Since dav_tx_xalloc() aborts the tx on allocation failure unless DAV_XALLOC_NO_ABORT is set (there is no tx-wide behavior knob for DAV), replacing the per-alloc flag with TX_FAILURE_RETURN would re-introduce the SIGABRT on md-on-SSD pools at the vos_dtx_extend_cmt_table() blob alloc; the vos_dtx_add_ptr() emergency-buffer fallback is likewise gated on behavior == TX_FAILURE_RETURN and never engages on DAV. That's why the patch keeps UMEM_FLAG_NO_ABORT mapped to all three backends.

So my plan is to keep the per-alloc/-add NO_ABORT flag (so DAV stays protected) plus the naming + umem_tx_xadd_ptr cleanups noted inline. If you'd additionally like to gate vos_dtx_commit_internal() on TX_FAILURE_RETURN for the PMEM path (to reuse the emergency-buffer fallback) while keeping the flag for DAV, I'm glad to add that too — just let me know your preference.

Currently, tx_{set,get}_failure_behavior() are only available for PMEM mode, that is the short-coming of commit b36dd7b0691. Consider current situation, some clean solution is preferred:

vos_dtx_begin_with_behavior() is useless any longer, replace it with regular umem_tx_begin().
Remove tx_{set,get}_failure_behavior() related logic that were introduced via the commit b36dd7b0691.
Change allocation in vos_dtx_extend_cmt_table() with NO_ABORT flags as your current PR does.
vos_dtx_add_ptr() with NO_ABORT flags and mo_tx_{x}add_ptr() related enhancement.

@hgichon, how do you think? The patch for above changes maybe not small, many of them are for reverting the work from commit b36dd7b0691. Since the trouble was caused by my commit, then I can fix it if that is too much for you. But if you want fix that by yourself, I am also fine.

hgichon · 2026-07-15T02:27:02Z

Thanks @Nasf-Fan — that plan sounds right. Since most of it is unwinding the failure_behavior infrastructure from b36dd7b and you have the most context on the emergency-buffer path, please go ahead and take it; I'm happy to defer to your patch.

A couple of things that might save you time, feel free to lift from this PR:

The 3-backend UMEM_FLAG_NO_ABORT mapping is already worked out (steps 3-4): POBJ_XALLOC_NO_ABORT/DAV_XALLOC_NO_ABORT for the alloc, and the xadd equivalents POBJ_XADD_NO_ABORT/DAV_XADD_NO_ABORT for mo_tx_xadd_ptr() so vos_dtx_add_ptr() stays backend-complete.
There's a small regression test (umem UMEM010: a NO_ABORT alloc failure must leave the TX alive so the in-tx fallback keeps working) that reproduced the original SIGABRT on md-on-SSD — glad to hand it over.

Shall I close this PR in favor of yours, or would you prefer to push onto this branch (maintainer_can_modify is on)? Either works for me. Our downstream already carries the alloc-side NO_ABORT fix, so md-on-SSD is protected in the meantime, and we'll rebase onto your final version once it lands. Thanks for picking this up!

Nasf-Fan · 2026-07-15T07:50:32Z

Thanks @Nasf-Fan — that plan sounds right. Since most of it is unwinding the failure_behavior infrastructure from b36dd7b and you have the most context on the emergency-buffer path, please go ahead and take it; I'm happy to defer to your patch.

A couple of things that might save you time, feel free to lift from this PR:

The 3-backend UMEM_FLAG_NO_ABORT mapping is already worked out (steps 3-4): POBJ_XALLOC_NO_ABORT/DAV_XALLOC_NO_ABORT for the alloc, and the xadd equivalents POBJ_XADD_NO_ABORT/DAV_XADD_NO_ABORT for mo_tx_xadd_ptr() so vos_dtx_add_ptr() stays backend-complete.

There's a small regression test (umem UMEM010: a NO_ABORT alloc failure must leave the TX alive so the in-tx fallback keeps working) that reproduced the original SIGABRT on md-on-SSD — glad to hand it over.

Shall I close this PR in favor of yours, or would you prefer to push onto this branch (maintainer_can_modify is on)? Either works for me. Our downstream already carries the alloc-side NO_ABORT fix, so md-on-SSD is protected in the meantime, and we'll rebase onto your final version once it lands. Thanks for picking this up!

@hgichon , I merged your fixes into the new PR (#18671) against the latest master, and add you as Co-authored-by: hgichon <hgichon@gmail.com> in such patch. Please check. Thanks!

hgichon · 2026-07-15T13:40:37Z

Closing in favor of #18671, which carries these fixes (co-authored) on top of the failure_behavior cleanup against latest master. Thanks @Nasf-Fan!

hgichon requested review from a team as code owners July 4, 2026 06:21

hgichon force-pushed the dtx-cmt-enospc-noabort branch from f395915 to 8e83c8f Compare July 4, 2026 06:53

hgichon force-pushed the dtx-cmt-enospc-noabort branch from 8e83c8f to 6c4dd32 Compare July 4, 2026 06:54

Nasf-Fan reviewed Jul 13, 2026

View reviewed changes

hgichon mentioned this pull request Jul 15, 2026

DAOS-18690 vos: fix engine crash when extend committed DTX table #18671

Draft

6 tasks

hgichon closed this Jul 15, 2026

Uh oh!

Conversation

hgichon commented Jul 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Why CI doesn't catch it

Reproduction

Suggested fix (verified)

Uh oh!

github-actions Bot commented Jul 4, 2026

Uh oh!

hgichon commented Jul 4, 2026

Uh oh!

Nasf-Fan left a comment

Choose a reason for hiding this comment

Uh oh!

Nasf-Fan Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

hgichon Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

Nasf-Fan Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

Nasf-Fan Jul 13, 2026

Choose a reason for hiding this comment

Uh oh!

hgichon Jul 14, 2026

Choose a reason for hiding this comment

Uh oh!

Nasf-Fan commented Jul 14, 2026

Uh oh!

hgichon commented Jul 14, 2026

Uh oh!

Nasf-Fan commented Jul 14, 2026

Uh oh!

hgichon commented Jul 15, 2026

Uh oh!

Nasf-Fan commented Jul 15, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

hgichon commented Jul 15, 2026

Uh oh!

Reviewers

Assignees

Labels

Milestone

Development

Uh oh!

2 participants

hgichon commented Jul 4, 2026 •

edited

Loading

Nasf-Fan commented Jul 15, 2026 •

edited

Loading