[BUG] ` FORTIFY: pthread_mutex_lock called on a destroyed mutex (0x<sanitized>)`

[scurkovic]

Hello. I am running onnxruntime with the XNNPACK EP on Android devices. While I can't reproduce the issue on any of my devices, it does seem to be relatively common in the wild.

Here is the stack trace:

*** *** *** *** *** *** *** *** *** *** *** *** *** *** *** ***
pid: 0, tid: 18721 >>> ai.repetitio.app <<<

backtrace:
  #00  pc 0x00000000000950b0  /apex/com.android.runtime/lib64/bionic/libc.so (abort+168)
  #01  pc 0x0000000000096fe4  /apex/com.android.runtime/lib64/bionic/libc.so (__fortify_fatal(char const*, ...)+128)
  #02  pc 0x00000000000fe330  /apex/com.android.runtime/lib64/bionic/libc.so (HandleUsingDestroyedMutex(pthread_mutex_t*, char const*)+68)
  #03  pc 0x00000000000fe1b0  /apex/com.android.runtime/lib64/bionic/libc.so (pthread_mutex_lock+296)
  #04  pc 0x00000000000e04c0  /data/app/~~qeiS50o8ebzSVF21_A1jMg==/ai.repetitio.app-TuQrMOQwSIOlS30pYs5Eag==/split_config.arm64_v8a.apk!libc++_shared.so (std::__ndk1::mutex::lock()+12) (BuildId: e8503e76ded720095755e58f7c0e4b30cf32df8c)
  #05  pc 0x0000000000da8c24  /data/app/~~qeiS50o8ebzSVF21_A1jMg==/ai.repetitio.app-TuQrMOQwSIOlS30pYs5Eag==/split_config.arm64_v8a.apk!librust_lib_repetitio.so (onnxruntime::BFCArena::Free(void*)+33046528) (BuildId: 544bf77442e0c9c9b196440634bb7fa9)
  #06  pc 0x0000000001837b54  /data/app/~~qeiS50o8ebzSVF21_A1jMg==/ai.repetitio.app-TuQrMOQwSIOlS30pYs5Eag==/split_config.arm64_v8a.apk!librust_lib_repetitio.so (xnn_delete_operator+33046528) (BuildId: 544bf77442e0c9c9b196440634bb7fa9)
  #07  pc 0x000000000182f8b8  /data/app/~~qeiS50o8ebzSVF21_A1jMg==/ai.repetitio.app-TuQrMOQwSIOlS30pYs5Eag==/split_config.arm64_v8a.apk!librust_lib_repetitio.so (onnxruntime::xnnpack::Softmax::~Softmax()+33046528) (BuildId: 544bf77442e0c9c9b196440634bb7fa9)
  #08  pc 0x0000000000dcdeb4  /data/app/~~qeiS50o8ebzSVF21_A1jMg==/ai.repetitio.app-TuQrMOQwSIOlS30pYs5Eag==/split_config.arm64_v8a.apk!librust_lib_repetitio.so (onnxruntime::SessionState::~SessionState()+33046528) (BuildId: 544bf77442e0c9c9b196440634bb7fa9)
  #09  pc 0x0000000000ece808  /data/app/~~qeiS50o8ebzSVF21_A1jMg==/ai.repetitio.app-TuQrMOQwSIOlS30pYs5Eag==/split_config.arm64_v8a.apk!librust_lib_repetitio.so (onnxruntime::InferenceSession::~InferenceSession()+33046528) (BuildId: 544bf77442e0c9c9b196440634bb7fa9)
  #10  pc 0x0000000000eceed4  /data/app/~~qeiS50o8ebzSVF21_A1jMg==/ai.repetitio.app-TuQrMOQwSIOlS30pYs5Eag==/split_config.arm64_v8a.apk!librust_lib_repetitio.so (onnxruntime::InferenceSession::~InferenceSession()+33046528) (BuildId: 544bf77442e0c9c9b196440634bb7fa9)
  #11  pc 0x0000000000d170b8  /data/app/~~qeiS50o8ebzSVF21_A1jMg==/ai.repetitio.app-TuQrMOQwSIOlS30pYs5Eag==/split_config.arm64_v8a.apk!librust_lib_repetitio.so (alloc::sync::Arc$LT$T$C$A$GT$::drop_slow::h625daf5b4aba311f+33046528) (BuildId: 544bf77442e0c9c9b196440634bb7fa9)
  #12  pc 0x0000000000bac910  /data/app/~~qeiS50o8ebzSVF21_A1jMg==/ai.repetitio.app-TuQrMOQwSIOlS30pYs5Eag==/split_config.arm64_v8a.apk!librust_lib_repetitio.so (core::ptr::drop_in_place$LT$rust_lib_repetitio..api..extractor..page_detection..PageDetector$GT$::h65c6dfa862d4a137 (.llvm.6585548147502741661)+33046528) (BuildId: 544bf77442e0c9c9b196440634bb7fa9)
  #13  pc 0x0000000000ba9e20  /data/app/~~qeiS50o8ebzSVF21_A1jMg==/ai.repetitio.app-TuQrMOQwSIOlS30pYs5Eag==/split_config.arm64_v8a.apk!librust_lib_repetitio.so (std::sys::backtrace::__rust_begin_short_backtrace::h0829452521d510d1+33046528) (BuildId: 544bf77442e0c9c9b196440634bb7fa9)
  #14  pc 0x0000000000bf4190  /data/app/~~qeiS50o8ebzSVF21_A1jMg==/ai.repetitio.app-TuQrMOQwSIOlS30pYs5Eag==/split_config.arm64_v8a.apk!librust_lib_repetitio.so (core::ops::function::FnOnce::call_once$u7b$$u7b$vtable.shim$u7d$$u7d$::h25926bb59a083456+33046528) (BuildId: 544bf77442e0c9c9b196440634bb7fa9)
  #15  pc 0x00000000018d18fc  /data/app/~~qeiS50o8ebzSVF21_A1jMg==/ai.repetitio.app-TuQrMOQwSIOlS30pYs5Eag==/split_config.arm64_v8a.apk!librust_lib_repetitio.so (std::sys::pal::unix::thread::Thread::new::thread_start::h4f8fd346e5b5de38+33046528) (BuildId: 544bf77442e0c9c9b196440634bb7fa9)
  #16  pc 0x00000000000fd0f4  /apex/com.android.runtime/lib64/bionic/libc.so (__pthread_start(void*)+208)
  #17  pc 0x0000000000096a04  /apex/com.android.runtime/lib64/bionic/libc.so (__start_thread+68)

Aside from this, I really don't know what further information I can give you. I am building onnxruntime version 1.22.1 (commit 89746dc).

[snnn]

I might have a little bit more background on this. It is mainly because std::mutex's destructor might be non-trivial on some platforms. Namely, macOS and Android.
The following static assert might fail on Android:

static_assert(std::is_trivially_destructible_v<std::mutex>,
              "This optimization requires a trivially destructible mutex.");

Most likely it is an ONNX Runtime issue. See microsoft/onnxruntime#24579 . It was related this pull request microsoft/onnxruntime#20413 .