Skip to content

Examples: Support Arm builds, interactive debugging, and proxy configuration #127

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
cjac wants to merge 3 commits into
base: main
Choose a base branch
from
Draft

Examples: Support Arm builds, interactive debugging, and proxy configuration #127

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
a777a38
Support Arm builds, interactive debugging, and proxy configuration
cjac Jun 12, 2026
9d2817b
Support Arm builds, interactive debugging, and proxy configuration
cjac Jun 13, 2026
d7a7d3a
Implement VM retention on failure, increase build disk size, and add …
cjac Jun 15, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -29,3 +29,15 @@ env.json
# failed patches
*.orig
*.rej

# --- Generic ignores for credentials and logs ---
key*.json
*env*.json
env-*.json
*.log
*.bz2
*.zst

# Temporary directories
temp/
tmp/
46 changes: 46 additions & 0 deletions examples/secure-boot/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -231,6 +231,52 @@ To run Dataproc clusters with NVIDIA GPUs and Shielded VM Secure Boot enabled:
3. **Verify module signature:** `sudo modinfo nvidia | grep signer` (Expected: `Cloud Dataproc Custom Image CA`)
4. **Check dmesg:** `dmesg | grep -iE "Secure Boot|NVRM|nvidia"`

## Manual Customization & Interactive Debugging

For complex troubleshooting, hot-patching, or script development, developers can bypass the automated containerized pipeline and run an **idempotent, interactive debugging loop** from their workstation.

This workflow is orchestrated by a single, powerful workstation-side script: **`customize-in-screen.sh`**.

### 1. Configure the Target
In `custom-images/env.json`, configure the target Dataproc version and the script you wish to test/debug:
```json
{
"IMAGE_VERSION": "2.1-debian11",
"CUSTOMIZATION_SCRIPT": "examples/secure-boot/no-customization.sh"
}
```

### 2. Run the Idempotent Customizer
From the `custom-images` directory on your workstation, execute the orchestrator:
```bash
bash examples/secure-boot/bin/customize-in-screen.sh
```

**How it behaves (Idempotency in Action)**:
* **First-Time Run (Cold Start)**: If the debug VM does not exist, it automatically calls `create-debug-vm.sh` to provision a raw, persistent VM (configured with a 24-hour shutdown timer and no automated startup script). It syncs your local code to GCS, triggers a remote background launch of `install-in-screen.sh`, and instantly attaches your terminal to the live screen session.
* **Subsequent Runs (Warm Start / Re-use)**: If the VM is already online, **it bypasses GCE provisioning entirely!** It instantly uploads your latest local edits to GCS, SSHes into the VM, downloads the new scripts, restarts the customization inside a detached `screen` session, and attaches your terminal. **Time to execution is under 5 seconds.**
* **Re-attaching to a Live Build**: If you run the script while a customization build is *already active* on the VM, it detects the running session, bypasses launching, and **instantly attaches your terminal to the live build.**

### 3. Interactive Attachment & Control
Once attached, you are inside a live, interactive `screen` session on the VM:
* **Real-Time Debugging**: You can watch the compilation, press `Ctrl+C` to halt, edit files locally in `/tmp/sources/` on the VM, and manually re-run steps to test fixes.
* **Safe Detachment**: To detach from the screen session and leave it running in the background on the VM (allowing you to close your laptop or disconnect), press:
`Ctrl+A` followed by `D`.
* **Re-attaching**: To re-attach later, simply run `bash examples/secure-boot/bin/customize-in-screen.sh` again from your workstation.

### 4. Run Workstation-Side Diagnostics
While the customization is running (or after a failure), you can audit the VM's active network and proxy state with a single command from your workstation terminal:
```bash
bash examples/secure-boot/audit-image-customizer.sh
```
This remote prober connects via IAP SSH in non-interactive batch mode and prints a pretty-printed JSON **System Audit Report** showing GCS and external network connectivity (verifying if Private Google Access and the SWP proxy are routing correctly).

### 5. Cleanup
Once debugging is complete, delete the GCE VM and clean up the GCS staging assets:
```bash
bash examples/secure-boot/bin/destroy-debug-vm.sh
```

## Key Scripts Involved

* `custom-images/env.json`: Single source of truth for configuration.
Expand Down
27 changes: 19 additions & 8 deletions examples/secure-boot/TESTING.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,15 +50,26 @@ dmesg | grep -iE "Secure Boot|NVRM|nvidia"

---

## Measured Custom Image Build Timing Reference
## Boot and Build Durations

The following table lists the empirical, real-world durations of the various image building and compilation phases observed during sequential and parallel builds inside a standard `us-east4` project:
Comparison of cluster boot times and image creation times.

| Image Build Phase | Customization Script / Action | Typical Duration | Performance & Cache Notes |
### Cluster Boot Times (VM Boot to Dataproc READY)

| Method | Mechanism | Boot Time | Details |
| :--- | :--- | :--- | :--- |
| **Standard Image + Init Action** | `install_gpu_driver.sh` (as Init Action) | **`7m` - `9m`** | Downloads ~4.5 GB of drivers and packages from GCS, compiles kernel modules, and configures YARN/Spark on every boot. |
| **Pre-baked Custom Image** | Pre-installed drivers + deferred systemd config | **`~4m`** | No downloads or installations. Adds ~30s to the first boot for hardware probing and writing configuration files. |

---

### Image Creation Times (Baking)

Image creation times using the Podman pipeline in `us-east4`:

| Phase | Script | Duration | Notes |
| :--- | :--- | :--- | :--- |
| **GCE Base Secure Boot Image** | `examples/secure-boot/no-customization.sh` | `~7m 20s` - `11m 05s` | Boots the unaccelerated VM instance, registers UEFI db public certs, and snapshots the `secure-boot` GCE image. Rocky Linux builds take ~11m, Debian/Ubuntu take ~7m. |
| **Total Baseline Custom Image Build** | `examples/secure-boot/build-and-run-podman.sh` | `~7m` - `11m` | Total OCI/Podman Stage 1 parallel compilation time to generate the UEFI baseline custom images. |
| **GPU/Conda Pre-bake Build (Cold Cache)** | `initialization-actions/gpu/install_gpu_driver.sh` | `~21m` - `24m` | Boots a T4 GPU VM instance, compiles the NVIDIA modules, compiles NCCL, and builds TensorFlow, PyTorch, and RAPIDS Conda environments via Mamba. |
| **GPU/Conda Pre-bake Build (GCS Cache Hit)** | `initialization-actions/gpu/install_gpu_driver.sh` | **`1m 45s`** | Downloads pre-compiled Blackwell drivers and zipped Conda tarballs directly from GCS over Private Google Access routes. |
| **Total Production custom image Build** | `examples/secure-boot/build-and-run-podman.sh` | `~25m` - `35m` | Total end-to-end parallel OCI build duration to generate the final, fully pre-baked production custom images (`-tf`). |
| **GCE Base Secure Boot Image** | `pre-init.sh` (Base Stage) | `~7m` - `11m` | Provisions VM, registers UEFI certs, and snapshots base image. (Rocky: ~11m, Debian/Ubuntu: ~7m). |
| **GPU/Conda Pre-bake Layer** | `install_gpu_driver.sh` (during baking) | `~21m` - `24m` | Compiles NVIDIA modules and builds Conda environments on a GPU VM. |
| **Total Image Suite** | `build-and-run-podman.sh` | `~25m` - `35m` | Total duration to generate the image suite. |

232 changes: 232 additions & 0 deletions examples/secure-boot/audit-image-customizer.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,232 @@
#!/bin/bash
#
# Dual-mode audit and diagnostic tool for GCE image customization.
#
# Workstation Mode: Copies itself to the active GCE builder VM, executes
# remotely, and prints the captured JSON state report.
# Guest Mode: Runs locally on the GCE VM, performs parallel probes,
# and outputs a structured JSON report to stdout.

set -euo pipefail

# --- Environment Detection ---
SCRIPT_DIR=$(dirname "$(readlink -f "$0")")
ENV_JSON="${SCRIPT_DIR}/../../env.json"

# If env.json exists two levels up from the script, we are on the Workstation.
# On the VM, the script is copied to /tmp, so this file will not exist.
if [[ -f "${ENV_JSON}" ]]; then
ROLE="workstation"
else
ROLE="guest"
fi

# ==========================================
# GUEST MODE: Run Parallel Probes on the VM
# ==========================================
if [[ "${ROLE}" == "guest" ]]; then
MDS_PREFIX="http://metadata.google.internal/computeMetadata/v1"
AUDIT_TEMP_DIR=$(mktemp -d)
PIDS=()

# Helper to run a probe in the background and save JSON fragment
run_probe() {
local -r key="$1"
shift
(
"$@" > "${AUDIT_TEMP_DIR}/${key}.json" 2> "${AUDIT_TEMP_DIR}/${key}.err"
) &
PIDS+=( $! )
}

# Probe 1: GCE Metadata Attributes
probe_metadata() {
local -r attributes=(
"http-proxy"
"https-proxy"
"proxy-uri"
"no-proxy"
"dataproc-cluster-name"
"http-proxy-pem-uri"
"universe-domain"
"custom-sources-path"
"project-id"
)
echo -n "{"
local first=true
for attr in "${attributes[@]}"; do
local val
val=$(curl -s -f -H "Metadata-Flavor: Google" "${MDS_PREFIX}/instance/attributes/${attr}" || echo "")
if [[ -z "${val}" ]]; then
val=$(curl -s -f -H "Metadata-Flavor: Google" "${MDS_PREFIX}/project/attributes/${attr}" || echo "")
fi

if [[ -n "${val}" ]]; then
if [[ "${first}" == "true" ]]; then first=false; else echo -n ","; fi
# Escape newlines and quotes in the value for safe JSON
local escaped_val
escaped_val=$(echo -n "${val}" | jq -R .)
echo -n "\"${attr}\": ${escaped_val}"
fi
done
echo -n "}"
}
run_probe "metadata" probe_metadata

# Probe 2: Network Connectivity (Direct vs Proxy)
probe_network() {
local http_proxy_val
http_proxy_val=$(curl -s -f -H "Metadata-Flavor: Google" "${MDS_PREFIX}/instance/attributes/http-proxy" || echo "")
if [[ -z "${http_proxy_val}" ]]; then
http_proxy_val=$(curl -s -f -H "Metadata-Flavor: Google" "${MDS_PREFIX}/instance/attributes/proxy-uri" || echo "")
fi

local direct_gcs="fail"
local proxy_gcs="fail"
local direct_ext="fail"
local proxy_ext="fail"

# Test direct GCS (should succeed if PGA is active, even without proxy/NAT)
if curl -s -f -o /dev/null --connect-timeout 3 "https://storage.googleapis.com" &>/dev/null; then
direct_gcs="success"
fi

# Test direct external (should fail in isolated network)
if curl -s -f -o /dev/null --connect-timeout 3 "https://www.google.com" &>/dev/null; then
direct_ext="success"
fi

# Test via proxy (if proxy metadata exists)
if [[ -n "${http_proxy_val}" ]]; then
if curl -s -f -x "http://${http_proxy_val}" -o /dev/null --connect-timeout 3 "https://storage.googleapis.com" &>/dev/null; then
proxy_gcs="success"
fi
if curl -s -f -x "http://${http_proxy_val}" -o /dev/null --connect-timeout 3 "https://www.google.com" &>/dev/null; then
proxy_ext="success"
fi
fi

cat <<EOF
{
"direct_gcs_connectivity": "${direct_gcs}",
"proxy_gcs_connectivity": "${proxy_gcs}",
"direct_external_connectivity": "${direct_ext}",
"proxy_external_connectivity": "${proxy_ext}"
}
EOF
}
run_probe "network" probe_network

# Probe 3: OS & System State
probe_system() {
local os_id="unknown"
if [[ -f /etc/os-release ]]; then
os_id=$(grep '^ID=' /etc/os-release | cut -d= -f2 | xargs)
fi

local systemd_pid1="false"
if [[ -d /run/systemd/system ]]; then
systemd_pid1="true"
fi

local java_home=""
if [[ -f /etc/environment ]]; then
java_home=$(awk -F= '/^JAVA_HOME=/ {print $2}' /etc/environment || echo "")
fi

cat <<EOF
{
"os_id": "${os_id}",
"systemd_is_pid1": "${systemd_pid1}",
"java_home": "${java_home}"
}
EOF
}
run_probe "system" probe_system

# Wait for all parallel probes to finish
for pid in "${PIDS[@]}"; do
wait "${pid}" || true
done

# Assemble the final JSON report
echo -n "{"
echo -n "\"metadata\": $(cat "${AUDIT_TEMP_DIR}/metadata.json"),"
echo -n "\"network\": $(cat "${AUDIT_TEMP_DIR}/network.json"),"
echo -n "\"system\": $(cat "${AUDIT_TEMP_DIR}/system.json")"
echo "}"

# Cleanup
rm -rf "${AUDIT_TEMP_DIR}"
exit 0
fi

# ==========================================
# HOST MODE: Run on Developer Workstation
# ==========================================
if [[ "${ROLE}" == "workstation" ]]; then
# SCRIPT_DIR and ENV_JSON are defined globally at the top of the script
if [[ ! -f "${ENV_JSON}" ]]; then
echo "ERROR: Configuration file ${ENV_JSON} not found." >&2
exit 1
fi

# 1. Parse GCE Project and Zone from env.json
PROJECT_ID=$(jq -r '.project_id // .PROJECT_ID // empty' "${ENV_JSON}")
ZONE=$(jq -r '.zone // .ZONE // empty' "${ENV_JSON}")

if [[ -z "${PROJECT_ID}" || -z "${ZONE}" ]]; then
echo "ERROR: project_id or zone not defined in env.json." >&2
exit 1
fi

echo "DEBUG: Workstation Mode - Querying GCE for active customization instance..." >&2

# 2. Dynamically discover the active builder VM name
# The builder VM name matches the pattern: dataproc-[version]-[timestamp]-install
VM_NAME=$(gcloud compute instances list --project="${PROJECT_ID}" \
--filter="name:dataproc-*-install AND zone:(${ZONE})" \
--format="value(name)" | head -n 1)

if [[ -z "${VM_NAME}" ]]; then
echo "ERROR: No active GCE customization instance found in project ${PROJECT_ID} (zone ${ZONE})." >&2
exit 1
fi

echo "DEBUG: Found active builder VM: ${VM_NAME}" >&2
echo "DEBUG: Copying audit script to VM..." >&2

# 3. SCP this script to the VM (using non-interactive batch mode)
gcloud compute scp "${SCRIPT_DIR}/audit-image-customizer.sh" "${VM_NAME}:/tmp/audit-image-customizer.sh" \
--project="${PROJECT_ID}" --zone="${ZONE}" --tunnel-through-iap --quiet \
--ssh-flag="-o BatchMode=yes" --ssh-flag="-o ConnectTimeout=5" \
--ssh-flag="-o StrictHostKeyChecking=no" --ssh-flag="-o UserKnownHostsFile=/dev/null" &>/dev/null

echo "DEBUG: Executing audit script remotely on VM..." >&2

# 4. SSH into the VM, run the script, and capture the JSON stdout
set +e
JSON_REPORT=$(gcloud compute ssh "${VM_NAME}" \
--project="${PROJECT_ID}" --zone="${ZONE}" --tunnel-through-iap --quiet \
--ssh-flag="-o BatchMode=yes" --ssh-flag="-o ConnectTimeout=5" \
--ssh-flag="-o StrictHostKeyChecking=no" --ssh-flag="-o UserKnownHostsFile=/dev/null" \
--command="bash /tmp/audit-image-customizer.sh" 2>/dev/null)
RETVAL=$?
set -e

# Clean up the script on the VM
gcloud compute ssh "${VM_NAME}" \
--project="${PROJECT_ID}" --zone="${ZONE}" --tunnel-through-iap --quiet \
--ssh-flag="-o BatchMode=yes" --ssh-flag="-o ConnectTimeout=5" \
--ssh-flag="-o StrictHostKeyChecking=no" --ssh-flag="-o UserKnownHostsFile=/dev/null" \
--command="rm -f /tmp/audit-image-customizer.sh" &>/dev/null || true

if [[ ${RETVAL} -ne 0 || -z "${JSON_REPORT}" ]]; then
echo "ERROR: Failed to execute remote audit on VM." >&2
exit 1
fi

# 5. Output the pretty-printed JSON report to the developer
echo "${JSON_REPORT}" | jq .
exit 0
fi
Loading