Stop the metrics check failing on its own pipe, and retry flaky pulls

Two ways CI went red on amd64 without the commit being at fault.

The metrics assertions piped a Prometheus scrape into `grep -q`, which exits
the instant it matches and never drains the rest. Once the scrape outgrows the
pipe buffer that leaves `echo` writing into a closed pipe, and pipefail reports
the EPIPE as the pipeline's status — so the check failed with the series
sitting right there in the payload. Measured on Linux: clean at 82 KB, fails
every time at 123 KB, and a scrape grows with every route and bucket, so it
only bites once a suite is busy enough to cross the line. Both checks now use
herestrings, which take the pipe out of it at any size.

Only these two moved. The same `echo … | grep -q` shape appears throughout both
suites, but every other payload measures well under a kilobyte — /api/sites is
316 bytes, the apex homepage 989 — so none are near the threshold, and a
mechanical sweep is a good way to break a working suite for no gain.

The other failure was Docker Hub answering 502 mid-`make up-auth`. Nothing to
fix in the tree, but the stack pulls six images from Docker Hub anonymously,
from a runner IP shared with every other public repo, so 429s and 502s will
keep arriving now that this one is public. Both bring-ups retry three times
with backoff, tearing down volumes in between. The retry wraps only the pull
and start, never the suite, so a stack that is genuinely broken still fails
three times and reports it.

Worth knowing if this recurs: authenticating to Docker Hub raises the pull
limit far more than retrying works around it, but that needs credentials in
repo secrets, so it is left as a deliberate choice rather than assumed.

Author: vtomasr5

.github/workflows/ci.yml

@@ -83,7 +83,19 @@ jobs:
       - uses: actions/setup-go@v7
         with:
           go-version-file: go.mod
-      - run: make up
+      # Bringing the stack up pulls six images from Docker Hub, anonymously,
+      # from a runner IP shared with every other public repo — so it draws
+      # rate-limit 429s and the occasional 502 that have nothing to do with the
+      # commit. Retry the bring-up rather than making someone re-run the job.
+      - name: Start the stack
+        run: |
+          for attempt in 1 2 3; do
+            make up && exit 0
+            echo "::warning::stack bring-up failed (attempt $attempt/3), retrying"
+            docker compose down -v || true
+            sleep $((attempt * 15))
+          done
+          exit 1
       - run: make smoke
       - name: stack logs
         if: failure()
@@ -112,7 +124,17 @@ jobs:
               || echo '127.0.0.1 dex.click.localhost' | sudo tee -a /etc/hosts >/dev/null
           fi
           getent hosts dex.click.localhost
-      - run: make up-auth
+      # Same transient-registry retry as smoke-open above; the auth overlay adds
+      # three more images (dex, oauth2-proxy, libsql-server) to the pull.
+      - name: Start the auth stack
+        run: |
+          for attempt in 1 2 3; do
+            make up-auth && exit 0
+            echo "::warning::auth stack bring-up failed (attempt $attempt/3), retrying"
+            docker compose -f docker-compose.yml -f docker-compose.auth.yml down -v || true
+            sleep $((attempt * 15))
+          done
+          exit 1
       - run: make smoke-auth
       - name: stack logs
         if: failure()

scripts/smoke.sh

@@ -175,8 +175,17 @@ ok "apex requests can't spoof a site scope (header stripped by nginx)"
 # templates (not concrete paths), so cardinality stays in check. Runs after
 # the database section so /api/db/{collection} traffic exists on a fresh stack.
 METRICS=$(curl -fsS -H "Host: click.localhost" "$BASE/metrics")
-echo "$METRICS" | grep -q '^click_http_requests_total{' || fail "metrics endpoint missing HTTP RED series"
-echo "$METRICS" | grep -q 'route="/api/db/{collection}"' || fail "metrics route label is not a bounded template"
+# Herestrings, not `echo "$METRICS" | grep -q`: `grep -q` exits the instant it
+# matches and never drains the rest, so once a scrape outgrows the pipe buffer
+# `echo` is left writing into a closed pipe. pipefail reports that EPIPE as the
+# pipeline's status, so the check fails with the series sitting right there —
+# and both patterns here match near the top, which is the worst case. Measured
+# on Linux: clean at 82 KB, fails every time at 123 KB, and a scrape grows with
+# every route and bucket, so this only bites once a suite is busy enough. The
+# herestring takes the pipe out of it entirely, at any size. Same hazard the
+# header capture above sidesteps by not piping curl straight into grep.
+grep -q '^click_http_requests_total{' <<<"$METRICS" || fail "metrics endpoint missing HTTP RED series"
+grep -q 'route="/api/db/{collection}"' <<<"$METRICS" || fail "metrics route label is not a bounded template"
 ok "GET /metrics exposes RED metrics with bounded route templates"
 
 echo "== 404 handling =="