chore: add qwen perf yaml

[tianmu-li]

What does this PR do?

Add performance run .yaml file for Qwen3.6-35B-A3B for agentic inference

Type of change

• Bug fix
• New feature
• Documentation update
• Refactor/cleanup

Related issues

Testing

• Tests added/updated
• All tests pass locally
• Manual testing completed

Checklist

• Code follows project style
• Pre-commit hooks pass
• Documentation updated (if needed)

[github-actions]

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

[gemini-code-assist]

Code Review

This pull request introduces a new benchmark configuration file, qwen_agentic_benchmark.yaml, for Qwen agentic inference. The review feedback highlights missing configuration parameters required to comply with benchmark invariants, specifically recommending the addition of num_trajectories_to_issue and stop_issuing_on_first_user_complete under agentic_inference, as well as the settings.client configuration block.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

To ensure compliance with the benchmark invariants and to make the configuration complete, please explicitly specify num_trajectories_to_issue and stop_issuing_on_first_user_complete under agentic_inference.

    agentic_inference:
      enable_salt: true # do not change.
      inject_tool_delay: true # do not change.
      num_trajectories_to_issue: 990 # Should be integer multiple of dataset trajectory count.
      stop_issuing_on_first_user_complete: false # required benchmark default.

[gemini-code-assist]

The settings.client configuration is missing. For official agentic benchmark runs, the client settings warmup_connections: 0 and max_idle_time: 0.5 are required invariants to ensure consistent and comparable performance results.

  load_pattern:
    type: agentic_inference
    target_concurrency: 8 # Submission-specific concurrency.

  client:
    warmup_connections: 0
    max_idle_time: 0.5

[Copilot]

Pull request overview

Adds a runnable benchmark configuration YAML under examples/10_Agentic_Inference/ for running an online performance benchmark of Qwen/Qwen3.6-35B-A3B using the agentic inference load pattern.

Changes:

• Add qwen_agentic_benchmark.yaml example config for agentic inference performance runs.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[arekay-nv]

Review-council (Codex + Grok + Claude) — notes for the author

This is an example config and it validates cleanly against the schema, so nothing here blocks. The notes below are documentation in case you want to align it with the sibling kimi_agentic_benchmark.yaml in this directory — whose "do not change" / "required benchmark default" fields this config drops. Severities are medium at most given it's an example; all findings were verified against the schema defaults.

The two worth a look: adding the settings.client block (avoids automatic connection warmup) and setting a Qwen thinking-enable flag.

[arekay-nv]

[medium] Consider adding a settings.client block (all three reviewers). The sibling kimi_agentic_benchmark.yaml pins warmup_connections: 0 / max_idle_time: 0.5. Without them this config uses the HTTPClientConfig defaults — warmup_connections: -1 (auto: pre-warms ~50% of the host's ephemeral-port budget → potentially thousands of connections to localhost:30000 before any request is issued) and max_idle_time: 4.0. For an agentic run that can mean a slow/failed startup or distorted endpoint load. Non-blocking for an example, but worth matching the sibling:

Suggested change

	target_concurrency: 8 # Submission-specific concurrency.
	target_concurrency: 8 # Submission-specific concurrency.
	client:
	warmup_connections: 0
	max_idle_time: 0.5

[arekay-nv]

[medium] No Qwen thinking-enable flag (Grok + Claude). This sets preserve_thinking: true but no flag that enables thinking. The sibling (Kimi) uses thinking: true (Kimi-specific); Qwen3's chat template typically uses enable_thinking: true. Without an enable flag, thinking mode depends on the server/template default and may run non-thinking while still preserving reasoning blocks. Confirm the correct key for your served Qwen template and add it under chat_template_kwargs.

[arekay-nv]

[medium] num_trajectories_to_issue / stop_issuing_on_first_user_complete dropped (Grok + Claude). The sibling pins num_trajectories_to_issue: 990. Omitted, it defaults to None → one pass over the dataset (len(base_convs)), so the issued workload becomes whatever the JSONL holds rather than the intended count. stop_issuing_on_first_user_complete defaults to false (same behavior — explicitness only). Consider matching the sibling:

Suggested change

	inject_tool_delay: true # do not change.
	inject_tool_delay: true # do not change.
	num_trajectories_to_issue: 990 # Should be integer multiple of 990.
	stop_issuing_on_first_user_complete: false

[arekay-nv]

[low] Confirm the served model id. model_params.name becomes the OpenAI request model string and must match the endpoint's /v1/models id (the sibling uses the served name /model). Also double-check Qwen/Qwen3.6-35B-A3B itself — unusual vs the widely-known Qwen3-30B-A3B; confirm it's not a typo.

[arekay-nv]

[low] presence_penalty: 1.5 is aggressive for a long multi-turn agentic/thinking workload — it can suppress token reuse that structured / tool-call turns rely on, and it diverges from the locked sibling baseline (temperature / top_p / max_new_tokens only). top_k: 20 is fine for Qwen; repetition_penalty: 1.0 is a no-op. Confirm these are intended.

[arekay-nv]

[low] max_duration_ms: 36000000 (10h) adds a hard cap the sibling doesn't have (unset → unlimited). Agentic completion is trajectory/drain-driven; a hard cap can abort a long tail mid-drain and mark the run incomplete. Confirm it's intended and comfortably above the expected run length.

[arekay-nv]

Added some comments that can be addressed, but looks good overall for an example.