Automatically marshal COM outputs as WinRT

[Jevan Saks (jevansaks)]

Summary

• Automatically projects recognized COM IID/void** outputs through C#/WinRT when the returned identity implements IInspectable, with COM fallback on E_NOINTERFACE.
• Enables the behavior by default and adds the comInterop.autoWinRTMarshalling opt-out.
• Supports flat P/Invoke, generated COM managed call paths, built-in COM, null output, and Native AOT.
• Preserves existing CsWinRT/source-generated COM RCW identity through ComWrappers.TryGetComInstance and classic COM RCW identity through Marshal.GetIUnknownForObject.
• Uses generated COM marshalling for [GeneratedComClass] CCWs and C#/WinRT marshalling for ordinary managed CCWs.
• Documents the selected architecture under docs/design.

Validation

• Full warning-as-error build and package creation.
• Full non-hardware and hardware-dependent test suites.
• Focused .NET 9 and .NET 10 identity, interface, and round-trip tests for CsWinRT RCWs, source-generated COM RCWs, classic COM RCWs, WinRT CCWs, and generated COM CCWs.
• Fresh package-consumption Native AOT publish and execution of the adaptive marshaller.

Design: #1770

ADO bug: https://dev.azure.com/microsoft/OS/_workitems/edit/63185687

Generated with Copilot CLI, model: GPT-5.6 Sol.

[Jevan Saks (jevansaks)]

The first hardware run exposed a test-only environment dependency: SIGDN_NORMALDISPLAY returned win on Windows Server (extensions hidden) instead of win.ini. Commit d26630d now asserts the stable SIGDN_FILESYSPATH; the focused test passes in Release on .NET 9 and .NET 10.

[Sergio Pedri (Sergio0694)]

Small note worth pointing out: this will make it so that if you try passing a WinRT-enabled object to a Win32 method, it'll get marshalled via built-in ComWrappers. Then if you later try to pass the same object to a WinRT API, it'll go through the WinRT marshaller. Meaning you'll get two different CCWs for it, with different vtables. I'm not sure how it can be avoided, but just figured we might want to at least explicitly document this.

[Jevan Saks (jevansaks)]

I'm not sure I understand. Does this ComInterfaceMarshaller path cause a new CCW to be created for the WinRT object? Should we use "is" to test the object for being a WinRT object and use a WInRT marshaller instead? We can do that, I just didn't think it was necessary.

[Sergio Pedri (Sergio0694)]

The problem is that if you call ComInterfaceMarshaller<T> to marshal an object that doesn't have a tracked CCW already, it will go through the built-in strategy based ComWrappers logic. Which means that the CCW you get depends on the (non deterministic) order in which calls to different marshallers are done.

Suppose you have a type that implements IDisposable and some generated COM interface. If you pass it through this method first, it'll just fail, because it's not a GeneratedComClass. However if you marshal it through WinRT first, it'll work fine (because the interop generator will detect IDisposable, which is custom-mapped, and also harvest the generated COM interface), and then passing it through here later will also work as it'll just reuse the existing CCW (from CsWinRT). This is just an example, there's lots of subtle ways you can get different marshalling results (in some cases you might even end up with multiple CCWs with different vtables in different places), depending on the order in which you pass stuff.

If you need concrete examples you can repro, I can put together some tests/repros or something.
Also FYI Manodasan Wignarajah (@manodasanW) in case you have thoughts on the above.

[Jevan Saks (jevansaks)]

Ok soooo what's your suggestion for how to fix it?

[Jevan Saks (jevansaks)]

Implemented in 77283ff. The marshaller now preserves existing RCWs before selecting a CCW strategy: ComWrappers.TryGetComInstance preserves ComWrappers-based RCWs (including CsWinRT and source-generated COM), while Marshal.IsComObject/Marshal.GetIUnknownForObject preserves classic COM RCWs. New managed objects use ComInterfaceMarshaller<object> only when the runtime type is directly marked [GeneratedComClass]; all other managed objects use WinRT.MarshalInspectable<object>.

Focused tests now pin canonical IUnknown identity, interface availability, and round-trip behavior for CsWinRT RCWs, source-generated COM RCWs, classic COM RCWs, WinRT CCWs, and generated COM CCWs. This avoids creating a second CCW for an existing RCW and makes the CCW strategy deterministic for new objects.