Skip to content

DAOS-18690 vos: fix engine crash when extend committed DTX table#18671

Draft
Nasf-Fan wants to merge 1 commit into
masterfrom
Nasf-Fan/DAOS-18690_5
Draft

DAOS-18690 vos: fix engine crash when extend committed DTX table#18671
Nasf-Fan wants to merge 1 commit into
masterfrom
Nasf-Fan/DAOS-18690_5

Conversation

@Nasf-Fan

@Nasf-Fan Nasf-Fan commented Jul 15, 2026

Copy link
Copy Markdown
Contributor

PR#18141 (b36dd7b) introduced some regression:

When the pool is physically full (ENOSPC), extending committed DTX table for punch/update commit may crash the whole engine with SIGABRT. That is because the backend TX logic (such as PMDK) will abort the TX by default if hit failure when allocation. Then subsequent operation (such as tx_add_ptr that is triggered by vos_dtx_extend_cmt_table()) against such aborted backend TX will trigger assertion inside backend TX logic.

The patch replaces umem_zalloc() with umem_alloc_verb() for extending committed DTX table with new UMEM_FLAG_NO_ABORT flag that will notify backend TX logic to return ENOSPC without aborted related TX. Similar enhancement is also added for tx_add_ptr related logic.

New unit tests (UMEM010/BMEM016/BMEM024) for that.

Steps for the author:

  • Commit message follows the guidelines.
  • Appropriate Features or Test-tag pragmas were used.
  • Appropriate Functional Test Stages were run.
  • At least two positive code reviews including at least one code owner from each category referenced in the PR.
  • Testing is complete. If necessary, forced-landing label added and a reason added in a comment.

After all prior steps are complete:

  • Gatekeeper requested (daos-gatekeeper added as a reviewer).

@github-actions

Copy link
Copy Markdown

Ticket title is 'Aurora daos_user: SCM single target ran out of space (min:0 B) and not able to finish GC. '
Status is 'In Progress'
Labels: 'request_for_2.8.1'
https://daosio.atlassian.net/browse/DAOS-18690

@daosbuild3

Copy link
Copy Markdown
Collaborator

@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Unit Test with memcheck completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18671/2/testReport/

@hgichon hgichon left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @Nasf-Fan — cross-checked against #18612 and everything landed: flag consistency, full failure_behavior unwind (no dangling refs), and the commit path (vos_dtx_commit_internal) is fully NO_ABORT-protected, so the crash can't recur. Nice that UMEM010 now covers the xadd_ptr path too. LGTM.

Two non-blocking checks:

  1. vos_dtx_extend_act_table() / vos_dtx_set_flags_one() reverted to plain umem_tx_add_ptr() (abort on failure) — intentional for those non-commit paths?
  2. UMEM010 runs under setup_pmem only; the reported crash was on md-on-SSD (DAV). Same coverage as #18612, just flagging in case a bmem variant is wanted.

@Nasf-Fan

Copy link
Copy Markdown
Contributor Author

Thanks @Nasf-Fan — cross-checked against #18612 and everything landed: flag consistency, full failure_behavior unwind (no dangling refs), and the commit path (vos_dtx_commit_internal) is fully NO_ABORT-protected, so the crash can't recur. Nice that UMEM010 now covers the xadd_ptr path too. LGTM.

Two non-blocking checks:

  1. vos_dtx_extend_act_table() / vos_dtx_set_flags_one() reverted to plain umem_tx_add_ptr() (abort on failure) — intentional for those non-commit paths?

vos_dtx_add_ptr() is used to handle space pressure when we try to release some space (directly or indirectly), such as DTX aggregation or DTX commit. But vos_dtx_extend_act_table() is during update/punch processing that will consume more space, even if we use vos_dtx_add_ptr(), related modification itself will also consume more space and maybe out of space when space pressure. So they are not the candidate users for vos_dtx_add_ptr().

  1. UMEM010 runs under setup_pmem only; the reported crash was on md-on-SSD (DAV). Same coverage as DAOS-18690 vos: fix engine abort in DTX commit blob fallback on ENOSPC #18612, just flagging in case a bmem variant is wanted.

I will add new test cases to cover bmem and bmem_v2.

@Nasf-Fan Nasf-Fan force-pushed the Nasf-Fan/DAOS-18690_5 branch from 71c21bd to 9619ce3 Compare July 16, 2026 04:12
PR#18141 (b36dd7b) introduced some regression:

When the pool is physically full (ENOSPC), extending committed DTX
table for punch/update commit may crash the whole engine with SIGABRT.
That is because the backend TX logic (such as PMDK) will abort the TX
by default if hit failure when allocation. Then subsequent operation
(such as tx_add_ptr that is triggered by vos_dtx_extend_cmt_table())
against such aborted backend TX will trigger assertion inside backend
TX logic.

The patch replaces umem_zalloc() with umem_alloc_verb() for extending
committed DTX table with new UMEM_FLAG_NO_ABORT flag that will notify
backend TX logic to return ENOSPC without aborted related TX. Similar
enhancement is also added for tx_add_ptr related logic.

New unit tests (UMEM010/BMEM016/BMEM024) for that.

Signed-off-by: Fan Yong <fan.yong@hpe.com>
Co-authored-by: hgichon <hgichon@gmail.com>
@Nasf-Fan Nasf-Fan force-pushed the Nasf-Fan/DAOS-18690_5 branch from 9619ce3 to 3b39604 Compare July 16, 2026 04:19
@daosbuild3

Copy link
Copy Markdown
Collaborator

@daosbuild3

Copy link
Copy Markdown
Collaborator

Test stage Unit Test with memcheck completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18671/4/testReport/

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

3 participants