DAOS-18690 vos: fix engine crash when extend committed DTX table#18671
DAOS-18690 vos: fix engine crash when extend committed DTX table#18671Nasf-Fan wants to merge 1 commit into
Conversation
|
Ticket title is 'Aurora daos_user: SCM single target ran out of space (min:0 B) and not able to finish GC. ' |
23eab47 to
71c21bd
Compare
|
Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18671/2/testReport/ |
|
Test stage Unit Test with memcheck completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18671/2/testReport/ |
There was a problem hiding this comment.
Thanks @Nasf-Fan — cross-checked against #18612 and everything landed: flag consistency, full failure_behavior unwind (no dangling refs), and the commit path (vos_dtx_commit_internal) is fully NO_ABORT-protected, so the crash can't recur. Nice that UMEM010 now covers the xadd_ptr path too. LGTM.
Two non-blocking checks:
vos_dtx_extend_act_table()/vos_dtx_set_flags_one()reverted to plainumem_tx_add_ptr()(abort on failure) — intentional for those non-commit paths?- UMEM010 runs under
setup_pmemonly; the reported crash was on md-on-SSD (DAV). Same coverage as #18612, just flagging in case a bmem variant is wanted.
I will add new test cases to cover bmem and bmem_v2. |
71c21bd to
9619ce3
Compare
PR#18141 (b36dd7b) introduced some regression: When the pool is physically full (ENOSPC), extending committed DTX table for punch/update commit may crash the whole engine with SIGABRT. That is because the backend TX logic (such as PMDK) will abort the TX by default if hit failure when allocation. Then subsequent operation (such as tx_add_ptr that is triggered by vos_dtx_extend_cmt_table()) against such aborted backend TX will trigger assertion inside backend TX logic. The patch replaces umem_zalloc() with umem_alloc_verb() for extending committed DTX table with new UMEM_FLAG_NO_ABORT flag that will notify backend TX logic to return ENOSPC without aborted related TX. Similar enhancement is also added for tx_add_ptr related logic. New unit tests (UMEM010/BMEM016/BMEM024) for that. Signed-off-by: Fan Yong <fan.yong@hpe.com> Co-authored-by: hgichon <hgichon@gmail.com>
9619ce3 to
3b39604
Compare
|
Test stage Unit Test completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18671/4/testReport/ |
|
Test stage Unit Test with memcheck completed with status UNSTABLE. https://jenkins-3.daos.hpc.amslabs.hpecorp.net/job/daos-stack/job/daos//view/change-requests/job/PR-18671/4/testReport/ |
PR#18141 (b36dd7b) introduced some regression:
When the pool is physically full (ENOSPC), extending committed DTX table for punch/update commit may crash the whole engine with SIGABRT. That is because the backend TX logic (such as PMDK) will abort the TX by default if hit failure when allocation. Then subsequent operation (such as tx_add_ptr that is triggered by vos_dtx_extend_cmt_table()) against such aborted backend TX will trigger assertion inside backend TX logic.
The patch replaces umem_zalloc() with umem_alloc_verb() for extending committed DTX table with new UMEM_FLAG_NO_ABORT flag that will notify backend TX logic to return ENOSPC without aborted related TX. Similar enhancement is also added for tx_add_ptr related logic.
New unit tests (UMEM010/BMEM016/BMEM024) for that.
Steps for the author:
After all prior steps are complete: