Add dynamic shape explainer

[miaobin]

This is the initial draft to summarize the discussion on dynamic shapes #883 .

The explainer covers named/unnamed dynamic dimensions, deferred (dispatch-time) shape validation, computeShapes() API, and a new family of shape-as-data (*Dynamic) operators.

Open questions and considered alternatives are called out explicitly in the explainer and feedback on this doc would be very welcome.

[miaobin]

A POC based on the current explainer is WIP. And it has been validated against some real-world Transformer, LLM and image generation models.
With credits to @Honry for the ORT WebNN EP POC and assistance with the explainer, @huningxin and @fdwr for the initial review of this explainer.

[fdwr]

👍 I have some thoughts, but it's 95% 👌.

[fdwr]

I love that computeShapes exists, because even though the primary intention is for returning the shape, it might also offer a means for the backends to precompute needed memory allocations, so that the later dispatch is immediate. Under the hood, model execution has a few key steps between knowing the shape (like planning memory and recompiling shaders) and actually executing, and unfortunately many ML libraries fail to expose that key stage -_- (and deferring all the way to dispatch is bad because then it means the first dispatch will be slow, and later ones will be fast, but also any shape changes will cause stutters).

One performance concern I have is the potential ping-pong of shape vacillation. If the model uses one shape the first iteration, and another shape the next 10 iterations, then every transition might require reallocating and replanning memory again. If however the caller could just say "build this graph, and return me two MLGraphs for these two input shapes, with all weights shared between them", the backends could execute more efficiently (no need to see the future or cache things behind your back).

Additionally, using computeShapes catches errors early rather than more confusingly during dispatch when it's too late to do much about it. So, there are a few good reasons to push shape evaluation earlier than dispatch.

(mostly just thinking aloud - resolve me)

[miaobin]

One performance concern I have is the potential ping-pong of shape vacillation.

This is a point worth considering. I once ran a batch dynamic image classification demo on an OpenVINO CPU EP; when the batch size was between 1 and 16, execution was very fast after the initial run—suggesting that compilation only occurred the first time. However, once the batch size exceeded 16, the subsequent run slowed down abruptly, likely due to recompilation.

I have added a "Shape specialization and preparation" section. I recorded your suggestion as the open direction — build with different sizes , receive several MLGraphs each bound to a set of concrete shapes, weights shared.

[fdwr]

Do you think we even need unnamed dimensions? A uniquely named dynamic dimension and an unnamed dynamic dimension are identical after all, and it's preferable to have debuggable symbols. The only reason to support them would be because existing callers may have them (like ORTWeb calling WebNN), but we could always just synthesize a name on the fly like "inputTensorName2_axis3" 🤔. I mean, if I was debugging and hit a shape inference error, I'd rather see that than just null for a name. Alternately using "" instead of null could be less problematic (no need to check for null first before trying to use/print the string).

Actually, seeing generated names would probably help you too during WebNN/Chromium debugging, seeing where pass-through fails during shape inference.

[miaobin]

Some history, since we did have derived names early on and removed them deliberately. We generated things like "height-3" and "broadcast_1_hight". The names weren't unique and two unrelated conv/slice calls would collide on the same name and the backend would conflate two independent dimensions. We patched that with a per-builder counter. Additionally, when we created the POC on the ORT backend at the begining, we only called OrtApi::SetDimensions. Regardless of the name, any dim that was dynamic was ultimately set to -1. Then we dropped derived names entirely in the rewrite and narrowed the rule to "a name survives only on 1:1 pass-through".

A synthesized name that is unique and implies no relationship is semantically identical to an anonymous dim, and it's strictly better to look at while debugging. I agree with the direction and will take it.

[miaobin]

Done, the explainer now requires every dynamic dimension to carry a name, and the unnamed state is gone. MLInputOperandDescriptor.shape drops its nullable elements entirely (sequence<MLDimension>), so null is no longer accepted at all and the empty string is rejected too.

@huningxin Do you have any thoughts on this point?

[fdwr]

one whose rank is not yet known

Interestingly this has also come up for MLOperandDataType dataType too, not just shape, when you want to infer the type based on the input tensor's type so you can reuse the same mini-model for float16 and float32.
(just comment, no action expected - resolve me)

[fdwr]

optionally declare a minSize and maxSize bound

Another potentially useful constraint is a "multiple of", because models like StableDiffusion will fail if you try to bind an input whose width/height isn't a multiple of 8. (resolve me)

[miaobin]

I investigated various runtime environments and found none that support declarative stride constraints for dimensions:

• Core ML has RangeDim and EnumeratedShapes
• TensorRT has min/opt/max optimization profiles
• OpenVINO has bounded partial shapes
• ORT has symbolic names plus free-dimension overrides
• LiteRT has only -1

models like StableDiffusion will fail if you try to bind an input whose width/height isn't a multiple of 8

The 8-multiple requirement comes from the VAE's downsampling stages. I believe that if the dimensions do not meet the requirements, a shape mismatch error will be reported at a specific operator (such as reshape, concat, or others) during the validation.

[w3cbot]

anssiko marked as non substantive for IPR from ash-nazg.

[anssiko]

I cleared the automatic IPR check for this PR, since explainer documents are consider non-substantive from the W3C Patent Policy point of view.

Details

This IPR check is to ensure normative portions of the specification come from organizations who participate the WG. Furthermore, in this case, I can attest the authors of this PR are affiliated with Intel, and as such, any normative portions are reusable in the spec PR, as appropriate.

[anssiko]

@reillyeon you had good questions comments on our last call for this. Do you have some other Googlers in mind who should review this explainer PR?

When this PR lands, the team will start landing the implementation in smaller chunks for further validation of this approach.