fix(orchestrator): restore dpkg-owned dirs under /etc/ssl/certs before cert bundle

[AdaAibaby]

Fixes #3518

What was broken

Any template that installs a JRE (directly or transitively) ships a snapshot where:

• is missing at sandbox runtime
• is in half-configured () state in the dpkg database

causing every subsequent inside those sandboxes to exit with code 100.

Root cause

Three mechanisms interact (full analysis in #3518):

1. is a tmpfs bind-mount inside the guest — dpkg writes from build layers go there and never reach the NBD-backed rootfs.
2. The finalize phase boots a fresh VM; re-seeds the tmpfs from , which never contained , so the directory silently vanishes.
3. then runs , which activates 's dpkg trigger. The trigger's fails with because is missing. swallows the hook failure and exits 0 — the build reports success and the state is baked into the snapshot.

Fix

** — ** (primary fix):

Before :

1. Walk dpkg's file database and recreate any missing directories under
2. Run to resolve pending triggers now that their required paths exist

After :
3. Check for remaining packages and fail the build explicitly — preventing silent broken snapshots going forward

**** (defense in depth):

Seed the tmpfs additively: copy the underlying rootfs directory first, then overlay the tar on top. The tar remains authoritative for cert files; the rootfs copy ensures subdirectories present on the rootfs but absent from an older tar survive the fresh-VM boot.

Testing

Reproduced the issue (template with ) and confirmed:

• Before: → ; → exit 100
• After: → ; → exit 0

/cc @jakubno @dobrac @ValentaTomas @arkamar @tvi @tomassrnka Looking forward to your code review.

[0xez]

Thanks for picking this up — the approach matches the root cause analysis in #3518 exactly (dpkg-owned dir restore + dpkg --configure -a + additive tmpfs seed + explicit build failure on broken state). One blocking issue and one minor suggestion from the issue author:

Blocking: \\ line continuations inside a Go raw string literal

packCertBundleCmd is a backtick (raw) string literal, so \\ is not collapsed by the Go compiler — the guest shell receives two literal backslash characters. The shell then interprets \\<newline> as an escaped backslash (a stray \ argument) followed by an unescaped newline, so the command terminates and the next line starts with |:

sh: syntax error near unexpected token `|'

Since the script runs under set -e (and a syntax error aborts regardless), this would fail the finalize step for every Debian/Ubuntu template, not just the affected ones. The continuations need to be single backslashes:

+	dpkg-query -W -f='${db:Status-Abbrev} ${Package}\n' 2>/dev/null \
+		| awk '/^.i /{print $2}' \

(The \n inside the -f= format is fine as-is — dpkg-query itself interprets that escape, so it should stay literal.)

Easy to verify: go test won't catch it, but extracting the const and running it through sh -n will, and might be worth adding as a unit test alongside the existing template build tests.

Minor: the dpkg -L loop is O(all packages)

Walking every installed package through dpkg -L spawns thousands of processes and does scattered reads on the lazily-fetched NBD rootfs — the exact cost profile this code path was designed to avoid (per the comment above packCertBundleCmd). A single-pass equivalent:

grep -hs '^/etc/ssl/certs/' /var/lib/dpkg/info/*.list | sort -u | while IFS= read -r path; do
	[ -e "$path" ] || mkdir -p "$path"
done

This reads each .list file once and needs no per-package subprocesses. (-s suppresses errors on non-dpkg systems, keeping the existing command -v dpkg-query guard optional.)

Happy to re-test against our reproduction template (from_template("code-interpreter-v1") + apt_install(["default-jre-headless"])) once updated — we have the before/after verification pipeline from #3518 ready to run.

[leonmeijer]

This is superseded by 6f17229 on main. That implementation restores dpkg-owned certificate paths, validates the resulting tree, and includes focused tests without the raw-string continuation bug previously identified here. This PR can be closed as incorporated.