Skip to content

gst: fix aravissrc teardown deadlock under incomplete frames - #1087

Open
MrBlenny wants to merge 1 commit into
AravisProject:mainfrom
greenroom-robotics:greenroom/aravissrc-teardown-deadlock
Open

gst: fix aravissrc teardown deadlock under incomplete frames#1087
MrBlenny wants to merge 1 commit into
AravisProject:mainfrom
greenroom-robotics:greenroom/aravissrc-teardown-deadlock

Conversation

@MrBlenny

@MrBlenny MrBlenny commented Jul 22, 2026

Copy link
Copy Markdown

Problem

Setting a pipeline containing aravissrc to NULL can hang indefinitely. It shows up when complete frames stop arriving - high frame rate, GigE packet loss, bandwidth contention from several cameras, or an unplugged camera and gets worse with more cameras / higher frame rate.

Root cause

gst_aravis_create() waits for a buffer while holding GST_OBJECT_LOCK, and the element registers no GstBaseSrc unlock() vmethod, so the base class has no way to interrupt the wait during a state change. When the stream only yields incomplete (non-SUCCESS) buffers, the do { … } while (status != SUCCESS) loop spins on them forever and never releases the lock, so the PAUSED→READY transition deadlocks waiting for the streaming task to pause / for the lock.

Fix

  • Add unlock() / unlock_stop() that set/clear an atomic flushing flag.
  • create() now takes a ref on the stream and drops GST_OBJECT_LOCK across the buffer wait, polls in short slices (GST_ARAVIS_BUFFER_POLL_SLICE_US) checking the flag and returning GST_FLOW_FLUSHING promptly on unlock.
  • The wait is bounded to buffer_timeout_us total; on continuous failure buffers it now returns GST_FLOW_ERROR instead of spinning.
  • start() clears the flag so a reused element streams again.

Behaviour changes (intentional)

  • Sustained incomplete frames now surface as GST_FLOW_ERROR after buffer_timeout_us rather than an unbounded spin.
  • A missing stream at create() time returns GST_FLOW_FLUSHING instead of hitting an ARV_IS_STREAM g_return_val_if_fail CRITICAL + GST_FLOW_ERROR.

Test

New gstreamer test suite (tests/arvgstteardowntest.c): an in-process fake GV camera with a high gvsp-lost-ratio feeds aravissrc ! fakesink; the test asserts set_state(NULL) completes within a bound. It aborts (deadlock) without the fix and passes with it. Built only when the plugin is enabled; excluded from the valgrind setup like the python suite. subdir('gst') now runs before subdir('tests') so the suite can link the plugin.

Note: I built and ran this against 0.9.3-dev in a container - meson test --suite gstreamer fails (SIGABRT) without the fix and passes with it; a normal 25 fps stream still delivers frames with clean EOS.

gst_aravis_create() waited for a buffer while holding GST_OBJECT_LOCK, and the
element registered no GstBaseSrc unlock() vmethod, so the base class could not
interrupt the wait. Whenever complete frames stopped arriving (high frame-rate,
packet loss, or bandwidth contention from many cameras) the stream delivered a
steady flow of non-SUCCESS buffers and create() spun on them forever, so a state
change to NULL deadlocked trying to take the object lock.

- add unlock()/unlock_stop() that set an atomic "flushing" flag
- create() now refs the stream and drops GST_OBJECT_LOCK across the buffer wait,
  polls in short slices honouring the flushing flag (returning GST_FLOW_FLUSHING),
  and errors out after buffer_timeout_us instead of spinning on failure buffers
- start() clears the flag so a reused element streams again

Add a gstreamer test suite (tests/arvgstteardowntest.c) that tears down an
aravissrc pipeline fed by a lossy in-process fake GV camera; it deadlocks
without the fix. Built only when the plugin is, and excluded from the valgrind
setup like the python suite.
@EmmanuelP

Copy link
Copy Markdown
Contributor

Hi,

Thanks for the patch. Not an issue, but did you use AI to cook this one up ?

The regression test is welcome. But it fails for me:

test:         gstreamer - aravis:gst-teardown                                                                           
start time:   15:31:21                                                                                                  
duration:     19.16s                                                                                                    
result:       killed by signal 6 SIGABRT                                                                                
command:      MSAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1:print_stacktrace=1 UBSAN_OPTIONS=halt_on_error=1:           ->abort_on_error=1:print_summary=1:print_stacktrace=1 LD_LIBRARY_PATH=/home/pacaud/Sources/aravis/build/src:/opt/MVS/lib/64:/opt/MVS/   ->lib/32:/opt/MVS/lib/64:/opt/MVS/lib/32 MESON_TEST_ITERATION=1 GST_PLUGIN_PATH=/home/pacaud/Sources/aravis/build/gst                   ->ASAN_OPTIONS=halt_on_error=1:abort_on_error=1:print_summary=1 MALLOC_PERTURB_=51 /home/pacaud/Sources/aravis/build/tests/arv-gst-     ->teardown-test
----------------------------------- stdout -----------------------------------                                          
TAP version 14                                                                                                          
# random seed: R02Sa7ec2811353a872f4888b1676c32288e                                                                     
# GLib-GIO-DEBUG: _g_io_module_get_default: Found default implementation gvfs (GDaemonVfs) for ‘gio-vfs’                
1..1                                                                                                                    
# Start of gstreamer tests                                                                                              
not ok /gstreamer/teardown-under-packet-loss - ERROR:../tests/arvgstteardowntest.c:77:teardown_under_packet_loss_test: 'torn_down'      ->should be TRUE
Bail out!                                                                                                               
----------------------------------- stderr -----------------------------------                                          
**                                                                                                                      
ERROR:../tests/arvgstteardowntest.c:77:teardown_under_packet_loss_test: 'torn_down' should be TRUE 

While running the test by hand works:

❯ GST_PLUGIN_PATH=./gst/ ./tests/arv-gst-teardown-test
TAP version 14
# random seed: R02S54e1b7b61d88edfc6b13953f4f5322b7
# GLib-GIO-DEBUG: _g_io_module_get_default: Found default implementation gvfs (GDaemonVfs) for ?gio-vfs?
1..1
# Start of gstreamer tests
ok 1 /gstreamer/teardown-under-packet-loss
# slow test /gstreamer/teardown-under-packet-loss executed in 1.05 secs
# End of gstreamer tests

Any idea why ?

@EmmanuelP

EmmanuelP commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

CI pipelines all fail

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants