CUDA dlink in cmssw generates duplicate registration entries

[slava77]

observed with CMSSW_16_1_0_pre4:
if I add a simple kernel in RecoTracker/LST/plugins/alpaka/LSTPixelHitsKernels.dev.cc and a module calling it RecoTracker/LST/plugins/alpaka/LSTPixelHitsFromSoAProducer.cc to the existing packages
we end up with duplicate registration symbols __cudaRegisterLinkedBinary_*LSTEvent_dev_cc*
one in pluginRecoTrackerLSTPluginsPortableCudaAsync.so and one in libRecoTrackerLSTCoreCudaAsync.so.
The former also has the new kernel registration in __cudaRegisterLinkedBinary_*_LSTPixelHitsKernels_dev_cc_*.
Due to the LSTEvent_dev_cc symbol duplication the kernel function registration is eventually missed for _LSTPixelHitsKernels_dev_cc_
This eventually leads to a runtime error CUDA_ERROR_INVALID_HANDLE

To reproduce, this branch can be used:
HSF-India-Pune-2026-LST:CMSSW_16_1_0_pre4_LST_pixelDirect-task5-pixsoa2soa
https://github.com/HSF-India-Pune-2026-LST/cmssw/pull/17/changes

Runtime is not necessarily required; after compilation the following indicates a problem

nm -C -l tmp/el9_amd64_gcc13/src/RecoTracker/LSTCore/src/alpaka/RecoTrackerLSTCoreCudaAsync/RecoTrackerLSTCoreCudaAsync_cudadlink.o | grep _LSTEvent_dev_cc_
0000000000000010 T __cudaRegisterLinkedBinary_fe01fecd_15_LSTEvent_dev_cc_75f6ad43_4100264      RecoTrackerLSTCoreCudaAsync_cudadlink.reg.c:2

nm -C -l tmp/el9_amd64_gcc13/src/RecoTracker/LST/plugins/RecoTrackerLSTPluginsPortableCudaAsync/RecoTrackerLSTPluginsPortableCudaAsync_cudadlink.o | grep _LSTEvent_dev_cc_
00000000000000e0 T __cudaRegisterLinkedBinary_fe01fecd_15_LSTEvent_dev_cc_75f6ad43_4100264      RecoTrackerLSTPluginsPortableCudaAsync_cudadlink.reg.c:3

One solution that I could come up with is to drop the present dependency on LSTEvent_dev_cc in the RecoTracker/LST/plugins/alpaka dlink step. Here I manually edited the command from nvcc -dlink ... -lRecoTrackerLSTCoreCudaAsync_nv ...LSTPixelHitsKernels.dev.cc.o -o ...RecoTrackerLSTPluginsPortableCudaAsync_cudadlink.o to drop the -lRecoTrackerLSTCoreCudaAsync_nv.
The first direct evidence that this worked is the __cudaRegisterLinkedBinary_*LSTEvent_dev_cc* does not appear anymore in RecoTrackerLSTPluginsPortableCudaAsync_cudadlink.o
The runtime is also OK.

The question is: do we really need the dependent cudadlink.o to know about dev.cc/fatbins in another library? If not, these extra -lRecoTrackerLSTCoreCudaAsync_nv dependencies during dlink could be safely removed.

more notes in the mattermost

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @slava77.

@Dr15Jones, @ftenchini, @makortel, @mandrenguyen, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[slava77]

... or should this issue be made to https://github.com/cms-sw/cmssw-config instead?

[makortel]

assign core,heterogeneous

[cmsbuild]

New categories assigned: core,heterogeneous

@Dr15Jones,@fwyzard,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

... or should this issue be made to https://github.com/cms-sw/cmssw-config instead?

In my opinion (any) CMSSW-related problem should be reported primarily in https://github.com/cms-sw/cmssw, unless the alternative repository is very clear.

[slava77]

Here I manually edited the command from nvcc -dlink .

the same effect apparently from editing in SCRAM/GMake/Makefile.cuda from
Tool_DependencyCudaDLINK=$(foreach x,$($(1)_LOC_CUDA_DLINK_LIB_ALL),-l$x)
to an empty string
Tool_DependencyCudaDLINK=

My thought though is that these were added here for a reason

[slava77]

In my opinion (any) CMSSW-related problem should be reported primarily in https://github.com/cms-sw/cmssw, unless the alternative repository is very clear.

should I copy-paste this to cms-sw/cmssw and close this one?

[fwyzard]

@slava77 does the approach you are using match the one in https://github.com/cms-sw/cmssw/tree/master/HeterogeneousTest/AlpakaOpaque ? Or one of the other test packages ?

[slava77]

@slava77 does the approach you are using match the one in https://github.com/cms-sw/cmssw/tree/master/HeterogeneousTest/AlpakaOpaque ? Or one of the other test packages ?

I'm not using the AlpakaOpaque pattern (this one seemed small to quickly browse before responding).
I'm not sure how to answer "one of the other test packages" question (I don't know).

Here I just added a dev.cc file with a regular alpaka kernel and called it from a module sitting in a package LST that just happened to have some other modules using functions launching kernels defined in LSTCore.
FWIW, having a dev.cc in the same subdir as the producer was inspired by a random git grep landing on RecoLocalCalo/EcalRecProducers/plugins/alpaka.
And the result was the obscure runtime failure.