[lts_03_2025] CI: migrate to Microsoft-hosted agents (retired ACR/self-hosted pools break all PRs) - #2747

Status note: build 162518 has been sitting in notStarted for ~22 minutes, queued behind roughly 15 concurrent builds across the azure-iot-sdks org (horton-*, Java, C#, Python gates). That is hosted-agent capacity, not a problem with this change.

One thing already confirmed by this run, though: the stage graph now resolves to Requirements / Setup create_azure_resources / Tests / Cleanup and the jobs are queued for real agents. On build 162517 the old YAML died 14s in at Initialize containers before any job was scheduled. So the ACR blocker is gone; what remains is whether the jobs themselves pass against this branch's older source.

Please check the completed run before merging — as noted above, I could not build or test locally, and a genuine LTS-only failure is plausible given the ~14-month source gap.

First full run completed — 30 jobs green, 6 failures, both root-caused and fixed

Build 162518 got the whole matrix running for the first time on this branch. 30 jobs passed, including every Linux job, all four valgrind/helgrind suites, the E2E jobs and OSX/Xcode. The ACR blocker is gone.

Six jobs failed, from exactly two root causes. Both were my fault, not pre-existing breakage, and both are now fixed.

1. Cross Compile (ARM) and Cross Compile (MIPS32) — ld: cannot find -lc_logging_v2

My first commit took samples/dockerbuilds/myapp/CMakeLists.txt from main along with the pipeline files. That file is not CI infrastructure. main's version links c_logging_v2:

target_link_libraries(myapp ... aziotsharedutil c_logging_v2 parson ...)

main can do that because its root CMakeLists.txt does add_subdirectory(deps/c-logging) and installs the v2 target. This branch has no c-logging wiring in its root CMakeLists.txt at all, so libc_logging_v2 is never built or installed here, and the sample fails at link.

Worth noting: both branches pin deps/c-logging at the same commit (504e7d47). The difference is purely that main wires it into the build and this branch does not — so this is not a submodule problem and does not call for a submodule bump on an LTS branch.

Fixed in d25ef45f5 by restoring this branch's own line. samples/dockerbuilds/myapp/CMakeLists.txt is now byte-identical to lts_03_2025.

2. All four Windows jobs — error C2220 in iothub_client_sample_hsm.c(45,61)

Each of the four Windows jobs reported exactly 1 Error(s), all on the same line. This is a genuine pre-existing bug on this branch, exposed only because the migration moved Windows onto hosted windows-2022 images with a newer MSVC:

if (reason == IOTHUB_CLIENT_CONNECTION_OK && result == IOTHUB_CLIENT_CONFIRMATION_OK)

result is an IOTHUB_CLIENT_CONNECTION_STATUS, but IOTHUB_CLIENT_CONFIRMATION_OK is an IOTHUB_CLIENT_CONFIRMATION_RESULT — a different enum. Newer MSVC flags the mismatched-enum comparison, and this sample builds warnings-as-errors.

main already fixed exactly this in ca8b9bf35 (#2681, "caught by latest VS2022 vs17.14.2"), a one-line change. Cherry-picked here as c52197221.

Verified locally that this is behaviour-preserving. Both constants evaluate to 0, so runtime behaviour is identical and only the type error is corrected:

CONFIRMATION_OK=0 CONNECTION_AUTHENTICATED=0 equal=YES

I also compiled the fixed sample with gcc -Wall -Wextra -fsyntax-only: clean.

This is the one place where this PR now touches non-CI source. It is a single line, it is already on main, and without it no Windows job on this branch can build.

Other things I checked

• The remaining samples/dockerbuilds/*/Dockerfile changes are toolchain modernization only (OpenSSL 1.1.1v→3.0.15, curl 7.60→8.20 with --with-openssl/--disable-ntlm, CMAKE_POLICY_VERSION_MINIMUM=3.5 for CMake 4 in newer base images). None reference c_logging_v2 or anything else main-only.
• The Free memory is lower than 5% messages in the MIPS32 job are warnings emitted alongside the real failure, not the cause; the job died on the link error.

Still to confirm

Build 162560 is running now with both fixes. Please read that run before merging — I have verified the causes locally but cannot run MSVC or the docker cross-compiles here.

✅ Fully green — build 162560 succeeded

All 34 jobs passed, 37/37 GitHub checks green. Both fixes are confirmed by CI:

• Cross Compile (ARM) and Cross Compile (MIPS32) — now pass (were ld: cannot find -lc_logging_v2).
• Windows x86 / x64 (Debug) / x64 (Release) / Dynamic — now pass (were error C2220 in iothub_client_sample_hsm.c(45,61)).

Everything that passed in the previous run still passes, including all valgrind/helgrind suites, the E2E jobs, OSX/Xcode, mbedTLS 2.16/2.28/3.x, WolfSSL, BearSSL and RIOT/DICE. No flakes, no reruns.

This is the first green C-canary run on lts_03_2025. Ready for review.

Recommended: merge this first, then rerun CI on #2746 (which is still red only because its branch predates this pipeline).

Doxyfile reverted; one unrelated infrastructure failure in the new run

Pushed b9a226490 reverting the Doxyfile version bump (thread above). Diff is now 17 files.

Build 162564 has one failure: Ubuntu 24.04 ARM64 (Cross-compile only), in the Install aarch64 cross toolchain and arm64 dev libs step, apt exit 100:

libssl-dev:arm64 : Depends: libssl3t64:arm64 (= 3.0.13-0ubuntu3.15) but it is not going to be installed
libcurl4t64:arm64 : Depends: libssl3t64:arm64 (>= 3.0.0) but it is not going to be installed
E: Unable to correct problems, you have held broken packages.

Not caused by this PR's latest change. The only delta between the fully green 162560 and this run is one line in build_all/docs/Doxyfile, and no job references that file:

$ git diff --stat d25ef45f5 b9a226490
 build_all/docs/Doxyfile | 2 +-

Cause. libssl3t64 is Multi-Arch: same, so libssl3t64:arm64 must be the exact same version as the amd64 copy already on the runner image. The step installs libssl-dev:arm64 unpinned from ports.ubuntu.com. When Ubuntu publishes a new libssl and the runner image has not been refreshed, the two pockets disagree and apt refuses. That is why this passed two hours earlier with identical pipeline content.

This is inherited from main and will recur there too, independent of this PR. 17 jobs have passed so far in this run and it is the only failure.

Suggest a rerun of that job. If it should be hardened instead, the durable fix is to install the arm64 libs pinned to the image's installed libssl3t64 version, which I would rather do on main for the reasons in the thread above. Happy to take either.

Retry result: all 34 build/test jobs green; the ARM64 failure was transient

Build 162564 after retrying the failed job:

• Ubuntu 24.04 ARM64 (Cross-compile only) — passed on attempt 2, unchanged code. Confirms the earlier libssl3t64:arm64 apt failure was transient mirror/image version skew, not a defect.
• All other 33 jobs passed on attempt 1.

34/34 build and test jobs green.

The one remaining red is a retry artifact, not a code or test failure

destroy_azure_resource_group failed on attempt 2:

ERROR: (ResourceGroupNotFound) Resource group 'rg-e2d86b4202cb46289f23ac70b2dfe0f3' could not be found.
##[error]Script failed with exit code: 3

Attempt 1 of that job succeeded and deleted the group. The retry re-ran the whole Cleanup stage, so it tried to delete an already-deleted group. The step is:

set -e
az group delete --name $AZURE_RESOURCE_GROUP --yes --no-wait

az group delete returns 3 when the group is absent, and set -e turns that into a step failure. So Cleanup is not idempotent and any retried build on this branch will end red even when everything passed.

No resources are leaked — the opposite: the group was already removed.

This step is verbatim from main, so main has the same behaviour on any retry. Same reasoning as the other inherited items: better fixed there and allowed to flow down. The one-line fix is to tolerate an absent group, e.g. append || [ $? -eq 3 ], or precede it with an existence check.

State of this PR

Green on content, twice over:

The delta between the two commits is one line in build_all/docs/Doxyfile, which no job reads.

Happy to either queue a fresh run so the current head shows a single clean green, or open the Cleanup idempotency fix against main — say which you prefer.

Addressing the 2 suppressed comments from the latest review

The re-review of b9a226490 generated no new inline comments, but flagged two suppressed ones. Both investigated.

1. build_all/linux/run_tests.sh — flag combinations

Confirmed, and broader than the earlier thread. Simulating the flag logic:

So there are three bad combinations, not one. In the plain branch --e2e-only is never consulted when --e2e is set, and --ut-only suppresses the only ctest call.

Not reachable from this pipeline. All six invocations resolve correctly:

Verbatim from main, so same reasoning as the other inherited items — worth a proper argument-validation fix on main covering all three combinations.

2. build/.vsts-ci.yml:804 — libwolfssl-dev || true

Disagree that this can silently stop exercising wolfSSL. The build cannot succeed without it:

• jenkins/linux_wolfssl.sh configures with -Duse_wolfssl=ON -Duse_openssl=OFF.
• With use_wolfssl set, c-utility compiles adapters/tlsio_wolfssl.c unconditionally, which includes wolfssl/options.h, wolfssl/ssl.h, wolfssl/error-ssl.h and wolfssl/wolfcrypt/types.h.
• That script runs under set -o errexit.

If the package were missing the compile would fail on the missing headers and the Build step would fail. The || true only tolerates a failure in the apt step; the real check is at compile time, and the WolfSSL job passed.

The valid part of the point is that failing at apt time gives a clearer error than failing later at a missing include. That is a readability improvement, not a correctness gap, and the line is verbatim from main.

No change made for either. Both are covered by the follow-up I offered against main.

All 5 review comments fixed in fd35ca602

1. psm1 source — both call sites now fetch from master. That branch never had a PR and was 43 commits behind master; all 3 of its unique commits are superseded there (details in the thread).
2. $HOME — now echo "$HOME".
3. mkdir — now mkdir -p "$WORK_ROOT", idempotent.
4. pushd — stray leading slash removed, quoted.
5. run_tests.sh — three broken flag combinations fixed; the 6 invocations this pipeline uses emit byte-identical ctest commands, verified locally.

One addition beyond the review comments

Made Destroy Azure Resource Group idempotent. az group delete exits 3 when the group is absent and the step runs under set -e, so retrying any job re-ran the Cleanup stage and turned a build red after its first cleanup had already succeeded. That is exactly what happened on the previous run, and it would block a green result whenever a job needs a retry. Now checks for the group first — verified both paths exit 0.

Local verification

• run_tests.sh and linux-setup-raspberry.sh pass bash -n; flag matrix exercised with a ctest stub.
• mkdir idempotency and the cleanup no-op path tested directly.
• YAML parses: 4 stages, 30 jobs, 0 azurecr.io references, 0 references to the old branch.

The MSVC, docker cross-compile and Azure provisioning paths cannot be exercised outside CI, so the pipeline is the check for those.

ARM64 job hardened in 4a07ac04e — it was not a one-off flake

The ARM64 job failed again on the previous run, so my earlier "transient" call was wrong on the recurrence, though right on the cause. Evidence it is environmental rather than tree-related:

Two different demanded versions with no change in between, and a plain retry passing, rules out this branch's contents. main passes the same job, which is why it was not caught earlier.

Cause. libssl3t64, libcurl4t64 and zlib1g are Multi-Arch: same, so the :arm64 copy must be the exact version of the :amd64 copy already on the runner image. ports.ubuntu.com and archive.ubuntu.com are not always in sync; when they drift, apt rejects the whole transaction with held broken packages. Whether a given run passes depends on which mirror snapshot the agent happens to hit.

Fix. Name both arches for those three runtime libraries so apt can settle on a version present in both archives, and pass --allow-downgrades so it can move the amd64 copy back when ports is the lagging side.

This is inherited from main and will keep biting there intermittently; same fix applies.

✅ Fully green — build 162584 succeeded

37/37 checks, all 35 jobs succeeded on first attempt, no retries. Head 4a07ac04e.

Every previously failing job is now green, including the two that were failing for real reasons:

Notably create_azure_resources and all E2E jobs passed while fetching the helper module from master, which confirms the branch pin was not load-bearing.

All 5 review comments are addressed, plus the two robustness fixes needed to make the result reproducible rather than luck-of-the-agent. Ready to merge.

Latest review: no new inline comments; 2 suppressed items

Copilot re-reviewed 4a07ac04e — 17/17 files, no new comments. Both suppressed items are pre-existing lines, no change made:

1. libwolfssl-dev ... || true (line 813) — repeat of an item I answered earlier. The premise that it can silently skip wolfSSL is wrong: jenkins/linux_wolfssl.sh configures with -Duse_wolfssl=ON, which makes c-utility compile adapters/tlsio_wolfssl.c, which includes wolfssl/ssl.h; the script runs under set -o errexit. A missing package fails the build at compile time. Valid only as a fail-faster/clearer-error preference, and the line is verbatim from main.

2. "Stray +" (line 109) — not stray. The comment wraps mid-phrase: "Publish only on the first job+ / stage attempt", i.e. job+stage attempt, which is exactly what the condition checks:

condition: and(always(), eq(variables['System.JobAttempt'], '1'), eq(variables['System.StageAttempt'], '1'))

Both JobAttempt and StageAttempt must be 1. Removing the + would make the sentence say something the code does not do.

Why horton-c-gate shows as skipped

It is filtered out by pipeline configuration, not by anything in this PR. The horton-c-gate pipeline's pull request trigger is:

"branchFilters": ["+main"],
"triggerType": "pullRequest"

Only PRs whose target branch is main match. This PR targets lts_03_2025, so Azure Pipelines reports the check as skipped (GitHub conclusion NEUTRAL). Its last 12 runs are all either refs/heads/main or PRs #2744/#2748, both of which target main. The build-completion trigger is filtered the same way (+public-preview, +main), so it does not chain from C-canary here either.

Enabling the filter alone would not have worked

build/.horton-e2e.yml on this branch is stale in two independently fatal ways:

1. It names the service connection 'GitHub OAuth - az-iot-builder-01'. That endpoint no longer exists in the project — the GitHub connection is now azure-iot-sdk-python-github. Resource resolution fails before any job starts.
2. It declares jobs: with jobs-gate-c.yaml directly and no setup stage. That template is consumed from iot-sdks-e2e-fx@refs/heads/master — a moving ref — and now starts with steps-load-test-config.yaml, which does download: current / artifact: test_config, published by steps-create-azure-resources.yaml. With no setup stage that artifact is never published, so every job fails on the download.

Same root cause as the rest of this PR: this branch's CI config drifted while the shared templates it consumes moved on.

Fixed here

affd1c5d3 takes main's build/.horton-e2e.yml, which wraps the gate in setup / build_and_test / cleanup and uses the endpoint that exists. The file references only e2e-fx templates and no paths inside this repo, so it is branch-agnostic — the SDK is cloned by the gate from Horton.Repo at Horton.Commit. Verified it parses and that main's version passes on PRs #2744 and #2748.

Still needed

Widening the pipeline's PR trigger to ["+main", "+lts_03_2025"] is a pipeline setting, not a repo change. I have requested that edit; it needs an approval I do not hold. Until it is applied the gate will keep reporting skipped here, now for configuration reasons only rather than because the file is broken.

Worth deciding separately whether the build-completion trigger should also include lts_03_2025 so the gate chains after C-canary on that branch post-merge.

Build 162609 green — 37/37, no retries

Head affd1c5d3 (horton yaml port). All 35 jobs succeeded first attempt.

Latest review: 18/18 files, no new inline comments; 1 suppressed item

Setup has dependsOn: [] (line 69) — the claim that this "defeats the submodule-gating goal" is not correct. The stage graph is:

Tests depends on Requirements with the default succeeded() condition, so a submodule mismatch still skips every test job. The gate holds.

The accurate part is the quota point: on a failing-submodule run the resource group is provisioned and then torn down without being used. Cleanup is always(), so nothing leaks. That is the documented tradeoff in the inline comment — running Setup in parallel saves 3-5 minutes on every run, against provisioning briefly on the rare run that fails submodule validation.

Verbatim from main; no change made.

Trigger request widened to +lts_*

Yes, wildcards work in classic branch filters — and this repo already relies on it. C-canary (definition 86, same repo) uses +lts_*, which is exactly why C-canary runs on this PR while horton-c-gate skips:

Five integrate-into-repo-* pipelines use +master,+lts_* too, so the pattern is well established here.

Requested edit is now ["+main", "+lts_*"] instead of pinning one branch. Matches all 10 lts_ branches (lts_01_2021 … lts_08_2025-era, including lts_03_2025), so future LTS branches are covered without another pipeline edit. PR triggers match on the target branch, so topic branches like ci/lts_03_2025-hosted-agents are unaffected.

Two notes:

• This also gates PRs to LTS branches that are past end of support. Harmless — they rarely see PRs — but it is broader than strictly needed.
• The build-completion trigger is still ["+public-preview", "+main"], so the gate will not chain after C-canary on LTS branches post-merge. I left it alone; adding +lts_* there is a one-line change if you want that too, but it starts running live E2E on every LTS merge.

Still needs the pipeline-setting approval; the repo-side fix (affd1c5d3) is already in.

Trigger change not applied — horton-c-gate stays skipped

The pipeline-settings edit to widen horton-c-gate's PR trigger to ["+main", "+lts_*"] was not approved, so definition 204 is unchanged (PR trigger still ["+main"]).

Consequence: horton-c-gate will keep reporting skipped on this PR and any PR targeting an lts_* branch. That is a pipeline configuration decision, not a repo problem, and nothing in this PR can change it.

The repo-side half is still worth having and stays in: affd1c5d3 replaces this branch's stale build/.horton-e2e.yml, which named a service connection that no longer exists ('GitHub OAuth - az-iot-builder-01') and declared jobs: with no setup stage, so it could not have run even with the trigger enabled. If the gate is ever enabled for LTS branches, the file is now correct.

C-canary is unaffected: 37/37 green, all 35 jobs first attempt.

horton-c-gate now runs here — and found a real gate bug, fixed in Azure/iot-sdks-e2e-fx#445

Trigger is applied (definition 204 rev 26, PR filters ["+main", "+lts_*"]). Branch filters are evaluated when a PR event fires, so the already-reported skip did not re-evaluate on its own; empty commit b3d640f33 raised a new event and the gate ran for the first time on an LTS branch.

The ported .horton-e2e.yml works. setup / create_azure_resources succeeded, which confirms both problems it fixed — the missing service connection and the absent test_config artifact. cleanup / destroy_azure_resource_group also passed.

build_and_test / build_linux_amd64 failed, in the gate's own docker build:

/wrapper/build/azure_macro_utils/macro_utils.h:1:10: fatal error:
macro_utils/macro_utils.h: No such file or directory

Cause is in the gate, not this branch. The image is built in two phases: prebuild.sh builds the SDK's default branch to warm the image, then rebuild.sh checks out the commit under test and rebuilds. rebuild.sh clears CMakeCache.txt but not generated headers.

main pins macro-utils-c at 045636ac, which ships inc/macro_utils/, so the SDK generates a forwarder at /azure_macro_utils/macro_utils.h → #include "macro_utils/macro_utils.h". CMAKE_BINARY_DIR is /wrapper/build because the wrapper does add_subdirectory(${C_SDK_ROOT}).

lts_03_2025 pins 5926caf4, which ships inc/azure_macro_utils/ and has no macro_utils/ directory. The phase-1 forwarder survives, sits on the include path ahead of ${C_SDK_ROOT}/deps/azure-macro-utils-c/inc, shadows the real header, and its inner include resolves to nothing.

The existing symlink guard in rebuild.sh covers only the opposite direction and correctly does nothing here.

Fix: Azure/iot-sdks-e2e-fx#445 removes the stale forwarding headers before configuring. Safe both ways — an SDK that needs them regenerates them during the following cmake. Verified by reproducing the shadowing with both layouts at the real include order: stale forwarder reproduces the error exactly, removing it builds clean, default-branch layout unaffected.

This is a pre-existing gate limitation that only surfaced because the gate had never run against an LTS branch before. Nothing to change in this PR.

Sequencing: the gate consumes e2e-fx templates from refs/heads/master, so once #445 merges, a rerun here picks it up automatically. Until then horton-c-gate will fail on this PR for that reason alone. C-canary is unaffected and healthy.

C-canary 162617: 34/35 green; one flake in iothubclient_uploadtoblob_e2e_helgrind

Run Tests (--helgrind --e2e --e2e-only) exited 8 with 17 of 18 tests passing:

94% tests passed, 1 tests failed out of 18
	183 - iothubclient_uploadtoblob_e2e_helgrind (Failed)

Not introduced by this PR. build_all/linux/run_tests.sh on this branch is byte-identical to main in the helgrind section, including the exclusion list, which quarantines only iothubclient_mqtt_dt_e2e.

Intermittent, not a regression. The same job passed on the four previous builds of this branch, all first attempt:

That test is known to be timing-sensitive under helgrind. It was quarantined in #2701 with the note that helgrind instrumentation makes IoTHubClient_Destroy() exceed the test's 30s deadline on the 4-vCPU hosted agents, then deliberately un-quarantined in #2718. So main carries the same exposure.

I have requested a retry of the failed job. If it should be made deterministic instead, the options are re-adding it to the helgrind exclusion list or raising that test's deadline — both decisions belong on main, since main un-quarantined it on purpose. Happy to open that PR.

The retry could not have passed — it ran after resources were deleted

Retrying only the failed job was the wrong remedy, and the 17/18 result is an artifact of that, not new signal.

05:37:02-05:37:10  Cleanup attempt 1 destroyed the resource group
05:49:37           helgrind E2E retry started (12 min later)
06:28:19           retry finished: 17 of 18 failed

The Cleanup stage runs with condition: always(), so when the build first completed as failed it deleted the resource group. The retry then ran every E2E test against Azure resources that no longer existed.

Confirming detail: the only test that passed on the retry was iothub_invalidcert_e2e_helgrind — the one test that does not need a live hub.

So the two attempts say different things:

Structural, inherited from main: "Rerun failed jobs" can never work for E2E in this pipeline, because Cleanup has already torn down the environment. The correct remedy is a full run, which re-executes Setup. Pushed an empty commit to do that.

Worth fixing properly on main — e.g. skipping the teardown when a rerun is possible, or making E2E jobs re-provision. I have not attempted that redesign here; happy to if you want it.

On the underlying flake

iothubclient_uploadtoblob_e2e_helgrind has now failed once in five runs of this branch. It is a known-fragile test: #2701 quarantined it under helgrind with the note that instrumentation makes IoTHubClient_Destroy() exceed the test's 30s deadline on 4-vCPU hosted agents, and #2718 removed that quarantine. main carries the same exposure today.

If it recurs on this run I will re-quarantine it under helgrind here, using #2701's own justification. Holding off for now rather than reducing coverage preemptively on a single data point.

/azp run horton-c-gate

Gate green. The earlier build_linux_amd64 failure was the e2e-fx C image picking up a stale macro-utils forwarding header (Azure/iot-sdks-e2e-fx#445, merged); with that in place the build and all 6 test jobs pass (build 162682).