From a777a38cf10fe5b4367c3c4fd15ad6a4f270ec1e Mon Sep 17 00:00:00 2001 From: "C.J. Collier" Date: Fri, 12 Jun 2026 23:50:52 +0000 Subject: [PATCH 1/3] Support Arm builds, interactive debugging, and proxy configuration Update the Secure Boot custom image builder to support Arm architectures, add an interactive debugging workflow, configure Secure Web Proxy (SWP), update path resolution, and sanitize the public .gitignore. ### 1. Arm Architecture Support * **Targets:** Added `2.2-ubuntu22-arm` and `2.3-ubuntu22-arm` image chains. * **Machine Type:** Added `create_arm_instance` to use `t2a-standard-2` GCE instances for Arm builds. * **Build Paths:** Bypass GPU/ML stages (TensorFlow, PyTorch, RAPIDS, Spark) for Arm architectures. * **Base Image:** Fall back to `secure-proxy` instead of `proxy-tf` for optional components (Docker, Jupyter, Zeppelin, Pig) on Arm. ### 2. Interactive Debugging Workflow * **Documentation:** Documented the `customize-in-screen.sh` workflow for running customization on persistent debug VMs. * **Disk Monitoring:** Install `screen` on the VM before running the disk usage monitor in `no-customization.sh`. * **State Resolution:** Resolve the build directory on failure using `sort | tail -n1` to handle multiple directories. ### 3. Proxy & Network Integration * **SWP Variables:** Pass `SWP_IP`, `SWP_PORT`, and `PROXY_CERT_GCS_PATH` through the Podman runner to the builder VM. * **Environment:** Parse and export SWP variables from `env.json`. * **Integration:** Configure the builder and debug VM to use the local `startup_script/gce-proxy-setup.sh` as the authoritative proxy configuration script. ### 4. Code Robustness & Sanitization * **Path Resolution:** Traverse parent directories in `env.sh` to resolve `DATAPROC_EVOLUTION_DIR` across nested git boundaries. * **Error Handling:** Update `run_gcloud` to capture and return exit codes under `set -e`. * **Globbing:** Prevent errors in `build-current-images.sh` when no logs match the timestamp. * **Repository Sanitization:** Sanitize `.gitignore` to remove development scratchpads, local draft scripts, and specific file patterns, replacing them with generic rules for keys, logs, and temporary directories. TAG=agy CONV=b274b565-1bd6-43f1-b4db-31f3d89d087b --- .gitignore | 12 + examples/secure-boot/README.md | 46 +++ examples/secure-boot/TESTING.md | 27 +- .../secure-boot/audit-image-customizer.sh | 232 ++++++++++++ examples/secure-boot/bin/cleanup-builders.sh | 93 +++++ examples/secure-boot/bin/create-debug-vm.sh | 177 +++++++++ .../secure-boot/bin/customize-in-screen.sh | 349 ++++++++++++++++++ examples/secure-boot/bin/destroy-debug-vm.sh | 52 +++ examples/secure-boot/bin/get-builder-log | 38 ++ examples/secure-boot/bin/scp-debug-vm.sh | 51 +++ examples/secure-boot/bin/ssh-builder | 34 ++ examples/secure-boot/bin/ssh-debug-vm.sh | 56 +++ examples/secure-boot/build-and-run-podman.sh | 3 + examples/secure-boot/build-current-images.sh | 2 +- examples/secure-boot/install-in-screen.sh | 159 ++++++++ examples/secure-boot/lib/env.sh | 24 +- examples/secure-boot/lib/util.sh | 5 +- examples/secure-boot/no-customization.sh | 12 + examples/secure-boot/pre-init.screenrc | 2 + examples/secure-boot/pre-init.sh | 60 ++- 20 files changed, 1399 insertions(+), 35 deletions(-) create mode 100644 examples/secure-boot/audit-image-customizer.sh create mode 100755 examples/secure-boot/bin/cleanup-builders.sh create mode 100644 examples/secure-boot/bin/create-debug-vm.sh create mode 100644 examples/secure-boot/bin/customize-in-screen.sh create mode 100755 examples/secure-boot/bin/destroy-debug-vm.sh create mode 100755 examples/secure-boot/bin/get-builder-log create mode 100755 examples/secure-boot/bin/scp-debug-vm.sh create mode 100755 examples/secure-boot/bin/ssh-builder create mode 100755 examples/secure-boot/bin/ssh-debug-vm.sh create mode 100644 examples/secure-boot/install-in-screen.sh diff --git a/.gitignore b/.gitignore index 79e0744..b651c2a 100644 --- a/.gitignore +++ b/.gitignore @@ -29,3 +29,15 @@ env.json # failed patches *.orig *.rej + +# --- Generic ignores for credentials and logs --- +key*.json +*env*.json +env-*.json +*.log +*.bz2 +*.zst + +# Temporary directories +temp/ +tmp/ diff --git a/examples/secure-boot/README.md b/examples/secure-boot/README.md index 930c778..65fe800 100644 --- a/examples/secure-boot/README.md +++ b/examples/secure-boot/README.md @@ -231,6 +231,52 @@ To run Dataproc clusters with NVIDIA GPUs and Shielded VM Secure Boot enabled: 3. **Verify module signature:** `sudo modinfo nvidia | grep signer` (Expected: `Cloud Dataproc Custom Image CA`) 4. **Check dmesg:** `dmesg | grep -iE "Secure Boot|NVRM|nvidia"` +## Manual Customization & Interactive Debugging + +For complex troubleshooting, hot-patching, or script development, developers can bypass the automated containerized pipeline and run an **idempotent, interactive debugging loop** from their workstation. + +This workflow is orchestrated by a single, powerful workstation-side script: **`customize-in-screen.sh`**. + +### 1. Configure the Target +In `custom-images/env.json`, configure the target Dataproc version and the script you wish to test/debug: +```json +{ + "IMAGE_VERSION": "2.1-debian11", + "CUSTOMIZATION_SCRIPT": "examples/secure-boot/no-customization.sh" +} +``` + +### 2. Run the Idempotent Customizer +From the `custom-images` directory on your workstation, execute the orchestrator: +```bash +bash examples/secure-boot/bin/customize-in-screen.sh +``` + +**How it behaves (Idempotency in Action)**: +* **First-Time Run (Cold Start)**: If the debug VM does not exist, it automatically calls `create-debug-vm.sh` to provision a raw, persistent VM (configured with a 24-hour shutdown timer and no automated startup script). It syncs your local code to GCS, triggers a remote background launch of `install-in-screen.sh`, and instantly attaches your terminal to the live screen session. +* **Subsequent Runs (Warm Start / Re-use)**: If the VM is already online, **it bypasses GCE provisioning entirely!** It instantly uploads your latest local edits to GCS, SSHes into the VM, downloads the new scripts, restarts the customization inside a detached `screen` session, and attaches your terminal. **Time to execution is under 5 seconds.** +* **Re-attaching to a Live Build**: If you run the script while a customization build is *already active* on the VM, it detects the running session, bypasses launching, and **instantly attaches your terminal to the live build.** + +### 3. Interactive Attachment & Control +Once attached, you are inside a live, interactive `screen` session on the VM: +* **Real-Time Debugging**: You can watch the compilation, press `Ctrl+C` to halt, edit files locally in `/tmp/sources/` on the VM, and manually re-run steps to test fixes. +* **Safe Detachment**: To detach from the screen session and leave it running in the background on the VM (allowing you to close your laptop or disconnect), press: + `Ctrl+A` followed by `D`. +* **Re-attaching**: To re-attach later, simply run `bash examples/secure-boot/bin/customize-in-screen.sh` again from your workstation. + +### 4. Run Workstation-Side Diagnostics +While the customization is running (or after a failure), you can audit the VM's active network and proxy state with a single command from your workstation terminal: +```bash +bash examples/secure-boot/audit-image-customizer.sh +``` +This remote prober connects via IAP SSH in non-interactive batch mode and prints a pretty-printed JSON **System Audit Report** showing GCS and external network connectivity (verifying if Private Google Access and the SWP proxy are routing correctly). + +### 5. Cleanup +Once debugging is complete, delete the GCE VM and clean up the GCS staging assets: +```bash +bash examples/secure-boot/bin/destroy-debug-vm.sh +``` + ## Key Scripts Involved * `custom-images/env.json`: Single source of truth for configuration. diff --git a/examples/secure-boot/TESTING.md b/examples/secure-boot/TESTING.md index 2d4d424..a92b7ba 100644 --- a/examples/secure-boot/TESTING.md +++ b/examples/secure-boot/TESTING.md @@ -50,15 +50,26 @@ dmesg | grep -iE "Secure Boot|NVRM|nvidia" --- -## Measured Custom Image Build Timing Reference +## Boot and Build Durations -The following table lists the empirical, real-world durations of the various image building and compilation phases observed during sequential and parallel builds inside a standard `us-east4` project: +Comparison of cluster boot times and image creation times. -| Image Build Phase | Customization Script / Action | Typical Duration | Performance & Cache Notes | +### Cluster Boot Times (VM Boot to Dataproc READY) + +| Method | Mechanism | Boot Time | Details | +| :--- | :--- | :--- | :--- | +| **Standard Image + Init Action** | `install_gpu_driver.sh` (as Init Action) | **`7m` - `9m`** | Downloads ~4.5 GB of drivers and packages from GCS, compiles kernel modules, and configures YARN/Spark on every boot. | +| **Pre-baked Custom Image** | Pre-installed drivers + deferred systemd config | **`~4m`** | No downloads or installations. Adds ~30s to the first boot for hardware probing and writing configuration files. | + +--- + +### Image Creation Times (Baking) + +Image creation times using the Podman pipeline in `us-east4`: + +| Phase | Script | Duration | Notes | | :--- | :--- | :--- | :--- | -| **GCE Base Secure Boot Image** | `examples/secure-boot/no-customization.sh` | `~7m 20s` - `11m 05s` | Boots the unaccelerated VM instance, registers UEFI db public certs, and snapshots the `secure-boot` GCE image. Rocky Linux builds take ~11m, Debian/Ubuntu take ~7m. | -| **Total Baseline Custom Image Build** | `examples/secure-boot/build-and-run-podman.sh` | `~7m` - `11m` | Total OCI/Podman Stage 1 parallel compilation time to generate the UEFI baseline custom images. | -| **GPU/Conda Pre-bake Build (Cold Cache)** | `initialization-actions/gpu/install_gpu_driver.sh` | `~21m` - `24m` | Boots a T4 GPU VM instance, compiles the NVIDIA modules, compiles NCCL, and builds TensorFlow, PyTorch, and RAPIDS Conda environments via Mamba. | -| **GPU/Conda Pre-bake Build (GCS Cache Hit)** | `initialization-actions/gpu/install_gpu_driver.sh` | **`1m 45s`** | Downloads pre-compiled Blackwell drivers and zipped Conda tarballs directly from GCS over Private Google Access routes. | -| **Total Production custom image Build** | `examples/secure-boot/build-and-run-podman.sh` | `~25m` - `35m` | Total end-to-end parallel OCI build duration to generate the final, fully pre-baked production custom images (`-tf`). | +| **GCE Base Secure Boot Image** | `pre-init.sh` (Base Stage) | `~7m` - `11m` | Provisions VM, registers UEFI certs, and snapshots base image. (Rocky: ~11m, Debian/Ubuntu: ~7m). | +| **GPU/Conda Pre-bake Layer** | `install_gpu_driver.sh` (during baking) | `~21m` - `24m` | Compiles NVIDIA modules and builds Conda environments on a GPU VM. | +| **Total Image Suite** | `build-and-run-podman.sh` | `~25m` - `35m` | Total duration to generate the image suite. | diff --git a/examples/secure-boot/audit-image-customizer.sh b/examples/secure-boot/audit-image-customizer.sh new file mode 100644 index 0000000..7afe4c6 --- /dev/null +++ b/examples/secure-boot/audit-image-customizer.sh @@ -0,0 +1,232 @@ +#!/bin/bash +# +# Dual-mode audit and diagnostic tool for GCE image customization. +# +# Workstation Mode: Copies itself to the active GCE builder VM, executes +# remotely, and prints the captured JSON state report. +# Guest Mode: Runs locally on the GCE VM, performs parallel probes, +# and outputs a structured JSON report to stdout. + +set -euo pipefail + +# --- Environment Detection --- +SCRIPT_DIR=$(dirname "$(readlink -f "$0")") +ENV_JSON="${SCRIPT_DIR}/../../env.json" + +# If env.json exists two levels up from the script, we are on the Workstation. +# On the VM, the script is copied to /tmp, so this file will not exist. +if [[ -f "${ENV_JSON}" ]]; then + ROLE="workstation" +else + ROLE="guest" +fi + +# ========================================== +# GUEST MODE: Run Parallel Probes on the VM +# ========================================== +if [[ "${ROLE}" == "guest" ]]; then + MDS_PREFIX="http://metadata.google.internal/computeMetadata/v1" + AUDIT_TEMP_DIR=$(mktemp -d) + PIDS=() + + # Helper to run a probe in the background and save JSON fragment + run_probe() { + local -r key="$1" + shift + ( + "$@" > "${AUDIT_TEMP_DIR}/${key}.json" 2> "${AUDIT_TEMP_DIR}/${key}.err" + ) & + PIDS+=( $! ) + } + + # Probe 1: GCE Metadata Attributes + probe_metadata() { + local -r attributes=( + "http-proxy" + "https-proxy" + "proxy-uri" + "no-proxy" + "dataproc-cluster-name" + "http-proxy-pem-uri" + "universe-domain" + "custom-sources-path" + "project-id" + ) + echo -n "{" + local first=true + for attr in "${attributes[@]}"; do + local val + val=$(curl -s -f -H "Metadata-Flavor: Google" "${MDS_PREFIX}/instance/attributes/${attr}" || echo "") + if [[ -z "${val}" ]]; then + val=$(curl -s -f -H "Metadata-Flavor: Google" "${MDS_PREFIX}/project/attributes/${attr}" || echo "") + fi + + if [[ -n "${val}" ]]; then + if [[ "${first}" == "true" ]]; then first=false; else echo -n ","; fi + # Escape newlines and quotes in the value for safe JSON + local escaped_val + escaped_val=$(echo -n "${val}" | jq -R .) + echo -n "\"${attr}\": ${escaped_val}" + fi + done + echo -n "}" + } + run_probe "metadata" probe_metadata + + # Probe 2: Network Connectivity (Direct vs Proxy) + probe_network() { + local http_proxy_val + http_proxy_val=$(curl -s -f -H "Metadata-Flavor: Google" "${MDS_PREFIX}/instance/attributes/http-proxy" || echo "") + if [[ -z "${http_proxy_val}" ]]; then + http_proxy_val=$(curl -s -f -H "Metadata-Flavor: Google" "${MDS_PREFIX}/instance/attributes/proxy-uri" || echo "") + fi + + local direct_gcs="fail" + local proxy_gcs="fail" + local direct_ext="fail" + local proxy_ext="fail" + + # Test direct GCS (should succeed if PGA is active, even without proxy/NAT) + if curl -s -f -o /dev/null --connect-timeout 3 "https://storage.googleapis.com" &>/dev/null; then + direct_gcs="success" + fi + + # Test direct external (should fail in isolated network) + if curl -s -f -o /dev/null --connect-timeout 3 "https://www.google.com" &>/dev/null; then + direct_ext="success" + fi + + # Test via proxy (if proxy metadata exists) + if [[ -n "${http_proxy_val}" ]]; then + if curl -s -f -x "http://${http_proxy_val}" -o /dev/null --connect-timeout 3 "https://storage.googleapis.com" &>/dev/null; then + proxy_gcs="success" + fi + if curl -s -f -x "http://${http_proxy_val}" -o /dev/null --connect-timeout 3 "https://www.google.com" &>/dev/null; then + proxy_ext="success" + fi + fi + + cat <&2 + exit 1 + fi + + # 1. Parse GCE Project and Zone from env.json + PROJECT_ID=$(jq -r '.project_id // .PROJECT_ID // empty' "${ENV_JSON}") + ZONE=$(jq -r '.zone // .ZONE // empty' "${ENV_JSON}") + + if [[ -z "${PROJECT_ID}" || -z "${ZONE}" ]]; then + echo "ERROR: project_id or zone not defined in env.json." >&2 + exit 1 + fi + + echo "DEBUG: Workstation Mode - Querying GCE for active customization instance..." >&2 + + # 2. Dynamically discover the active builder VM name + # The builder VM name matches the pattern: dataproc-[version]-[timestamp]-install + VM_NAME=$(gcloud compute instances list --project="${PROJECT_ID}" \ + --filter="name:dataproc-*-install AND zone:(${ZONE})" \ + --format="value(name)" | head -n 1) + + if [[ -z "${VM_NAME}" ]]; then + echo "ERROR: No active GCE customization instance found in project ${PROJECT_ID} (zone ${ZONE})." >&2 + exit 1 + fi + + echo "DEBUG: Found active builder VM: ${VM_NAME}" >&2 + echo "DEBUG: Copying audit script to VM..." >&2 + + # 3. SCP this script to the VM (using non-interactive batch mode) + gcloud compute scp "${SCRIPT_DIR}/audit-image-customizer.sh" "${VM_NAME}:/tmp/audit-image-customizer.sh" \ + --project="${PROJECT_ID}" --zone="${ZONE}" --tunnel-through-iap --quiet \ + --ssh-flag="-o BatchMode=yes" --ssh-flag="-o ConnectTimeout=5" \ + --ssh-flag="-o StrictHostKeyChecking=no" --ssh-flag="-o UserKnownHostsFile=/dev/null" &>/dev/null + + echo "DEBUG: Executing audit script remotely on VM..." >&2 + + # 4. SSH into the VM, run the script, and capture the JSON stdout + set +e + JSON_REPORT=$(gcloud compute ssh "${VM_NAME}" \ + --project="${PROJECT_ID}" --zone="${ZONE}" --tunnel-through-iap --quiet \ + --ssh-flag="-o BatchMode=yes" --ssh-flag="-o ConnectTimeout=5" \ + --ssh-flag="-o StrictHostKeyChecking=no" --ssh-flag="-o UserKnownHostsFile=/dev/null" \ + --command="bash /tmp/audit-image-customizer.sh" 2>/dev/null) + RETVAL=$? + set -e + + # Clean up the script on the VM + gcloud compute ssh "${VM_NAME}" \ + --project="${PROJECT_ID}" --zone="${ZONE}" --tunnel-through-iap --quiet \ + --ssh-flag="-o BatchMode=yes" --ssh-flag="-o ConnectTimeout=5" \ + --ssh-flag="-o StrictHostKeyChecking=no" --ssh-flag="-o UserKnownHostsFile=/dev/null" \ + --command="rm -f /tmp/audit-image-customizer.sh" &>/dev/null || true + + if [[ ${RETVAL} -ne 0 || -z "${JSON_REPORT}" ]]; then + echo "ERROR: Failed to execute remote audit on VM." >&2 + exit 1 + fi + + # 5. Output the pretty-printed JSON report to the developer + echo "${JSON_REPORT}" | jq . + exit 0 +fi diff --git a/examples/secure-boot/bin/cleanup-builders.sh b/examples/secure-boot/bin/cleanup-builders.sh new file mode 100755 index 0000000..a6d8a6b --- /dev/null +++ b/examples/secure-boot/bin/cleanup-builders.sh @@ -0,0 +1,93 @@ +#!/bin/bash +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS-IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# This script cleans up any lingering builder VMs using screen. + +set -e + +DEBUG="${DEBUG:-0}" +if (( DEBUG != 0 )); then + set -x +fi + +source examples/secure-boot/lib/env.sh +source examples/secure-boot/lib/util.sh + +print_status "Searching for lingering builder VMs in project ${PROJECT_ID}, zone ${ZONE}..." + +INSTANCE_LIST=$(gcloud compute instances list \ + --project="${PROJECT_ID}" \ + --zones="${ZONE}" \ + --filter="name ~ -install$" \ + --format="value(name)") + +if [[ -z "${INSTANCE_LIST}" ]]; then + report_result "None Found" + echo "No lingering builder VMs found to clean up." >&2 + exit 0 +fi + +report_result "Found" +echo "The following builder VMs will be deleted in detached screen sessions:" >&2 +echo "${INSTANCE_LIST}" >&2 + +read -p "Continue with deletion? (y/N): " confirm +if [[ "${confirm}" != [yY] ]]; then + echo "Deletion cancelled." >&2 + exit 0 +fi + +TEMP_SCREENRC="${tmpdir}/temp_cleanup.screenrc" +rm -f "${TEMP_SCREENRC}" +touch "${TEMP_SCREENRC}" + +echo "The following builder VMs will be deleted in a new screen session:" >&2 +echo "${INSTANCE_LIST}" >&2 + +read -p "Continue with deletion? (y/N): " confirm +if [[ "${confirm}" != [yY] ]]; then + echo "Deletion cancelled." >&2 + exit 0 +fi + +i=1 +for instance in ${INSTANCE_LIST}; do + echo "Queueing deletion for ${instance} in screen window ${i}" >&2 + SCREEN_CMD=" +attempt=0 +while true; do + gcloud beta compute instances delete ${instance} --zone=${ZONE} --project=${PROJECT_ID} --quiet --no-graceful-shutdown && break + attempt=$((attempt + 1)) + if [[ ${attempt} -ge 3 ]]; then + echo 'Max retries reached for ${instance}' + break + fi + echo 'Failed, retrying in 5 seconds...' + read -t 5 +done +echo \"Delete command for ${instance} finished with exit code $?\" +echo \"Window will close in 5 seconds...\" +read -t 5 +" + echo "screen -t delete-${i}-${instance} ${i} bash -c '${SCREEN_CMD}'" >> "${TEMP_SCREENRC}" + i=$((i + 1)) + sleep 0.1 # Small delay +done + + +echo "Launching screen session from ${TEMP_SCREENRC}..." >&2 +screen -c "${TEMP_SCREENRC}" + +rm -f "${TEMP_SCREENRC}" +echo "Screen session exited." >&2 diff --git a/examples/secure-boot/bin/create-debug-vm.sh b/examples/secure-boot/bin/create-debug-vm.sh new file mode 100644 index 0000000..4632d53 --- /dev/null +++ b/examples/secure-boot/bin/create-debug-vm.sh @@ -0,0 +1,177 @@ +#!/bin/bash + +set -e + +DEBUG="${DEBUG:-0}" +if (( DEBUG != 0 )); then + set -x +fi + +# Source environment variables +if [[ -f "$(dirname "$0")/../lib/env.sh" ]]; then + source "$(dirname "$0")/../lib/env.sh" +else + echo "ERROR: examples/secure-boot/lib/env.sh not found." + exit 1 +fi +if [[ -f "$(dirname "$0")/../lib/util.sh" ]]; then + source "$(dirname "$0")/../lib/util.sh" +else + echo "ERROR: examples/secure-boot/lib/util.sh not found." + exit 1 +fi + +FORCE_DELETE=0 +USE_PROXY=0 +USAGE="Usage: bash $(basename "$0") [-f] [-p]" + +# Parse options +while getopts "fp" opt; do + case ${opt} in + f ) FORCE_DELETE=1 ;; + p ) USE_PROXY=1 ;; + \? ) echo "${USAGE}" >&2; exit 1 ;; + esac +done +shift $((OPTIND -1)) + +if [[ -z "${IMAGE_VERSION}" ]]; then + echo "ERROR: IMAGE_VERSION is not set in env.json." + exit 1 +fi +if [[ -z "${CUSTOMIZATION_SCRIPT}" ]]; then + echo "ERROR: CUSTOMIZATION_SCRIPT is not set in env.json." + exit 1 +fi + +INSTANCE_NAME="debug-$(echo "${IMAGE_VERSION}" | tr '.' '-')-$(basename "${CUSTOMIZATION_SCRIPT}" .sh | tr '.' '-' | tr '_' '-')" + +# Override PURPOSE to a fixed value for the service account name +SA_NAME="sa-tf-pre-init" +GSA="${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" + +GCS_SOURCES_PATH="gs://${BUCKET}/${INSTANCE_NAME}/sources" +CACHE_FILE="/tmp/latest_dataproc_$(echo "${IMAGE_VERSION}" | tr './-' '_').txt" +CACHE_TTL_SECONDS=$((24 * 60 * 60)) # Cache for 1 day + +if [[ "${FORCE_DELETE}" -eq 1 ]]; then + rm -f "${CACHE_FILE}" +fi + +# Determine Dataproc Image +if [[ -n "${DATAPROC_IMAGE:-}" ]]; then + echo "Using image from ENV: ${DATAPROC_IMAGE}" +elif [[ -f "${CACHE_FILE}" ]] && [[ $(($(date +%s) - $(stat -c %Y "${CACHE_FILE}"))) -lt "${CACHE_TTL_SECONDS}" ]]; then + DATAPROC_IMAGE=$(cat "${CACHE_FILE}") + echo "Using cached image: ${DATAPROC_IMAGE}" +else + echo "DATAPROC_IMAGE not set or cache stale, querying for the latest ${IMAGE_VERSION} image..." + IMAGE_PREFIX="dataproc-$(echo "${IMAGE_VERSION}" | sed -e 's/\./-/g' -e 's/-debian12/-deb12/g' -e 's/-debian11/-deb11/g' -e 's/-ubuntu22/-ubu22/g' -e 's/-rocky9/-roc9/g')" + DATAPROC_IMAGE=$(gcloud compute images list --project cloud-dataproc \ + --filter="name:${IMAGE_PREFIX} AND status=READY" \ + --format="value(name)" | \ + grep -v "eap" | \ + sort -r | \ + head -n 1) + if [[ -z "${DATAPROC_IMAGE}" ]]; then + echo "ERROR: Could not find a suitable ${IMAGE_VERSION} image." + exit 1 + fi + echo "${DATAPROC_IMAGE}" > "${CACHE_FILE}" + echo "Using new image: ${DATAPROC_IMAGE} (cached to ${CACHE_FILE})" +fi + +# Check instance existence and handle --force +if gcloud compute instances describe "${INSTANCE_NAME}" --zone "${ZONE}" --project "${PROJECT_ID}" > /dev/null 2>&1; then + echo "Instance ${INSTANCE_NAME} already exists." + if [[ "${FORCE_DELETE}" -eq 1 ]]; then + echo "Deleting existing instance due to --force flag..." + run_gcloud delete_instance gcloud compute instances delete "${INSTANCE_NAME}" --zone "${ZONE}" --project "${PROJECT_ID}" -q + echo "Instance deleted." + else + echo "ERROR: Instance exists. Use -f to delete and recreate." + exit 1 + fi +else + echo "Instance ${INSTANCE_NAME} not found, proceeding." +fi + +# Clean up GCS path +run_gsutil rm_gcs gsutil -m rm -r "${GCS_SOURCES_PATH}" || echo "GCS path not found, continuing..." + +# Upload scripts from local repo to GCS +REPO_ROOT="$(git rev-parse --show-toplevel)" +run_gsutil cp_run gsutil cp "${REPO_ROOT}/startup_script/run.sh" "${GCS_SOURCES_PATH}/run.sh" +run_gsutil cp_init gsutil cp "${REPO_ROOT}/${CUSTOMIZATION_SCRIPT}" "${GCS_SOURCES_PATH}/init_actions.sh" +run_gsutil cp_env gsutil cp "${REPO_ROOT}/examples/secure-boot/lib/env.sh" "${GCS_SOURCES_PATH}/env.sh" +run_gsutil cp_util gsutil cp "${REPO_ROOT}/examples/secure-boot/lib/util.sh" "${GCS_SOURCES_PATH}/util.sh" +run_gsutil cp_proxy gsutil cp "${REPO_ROOT}/startup_script/gce-proxy-setup.sh" "${GCS_SOURCES_PATH}/gce-proxy-setup.sh" + +if [[ "${USE_PROXY}" -eq 1 ]]; then + # Resolve path to cloud-dataproc/gcloud/env.json relative to the authoritative repository root + # already resolved and exported by env.sh. + GCLOUD_ENV_JSON="${DATAPROC_EVOLUTION_DIR}/cloud-dataproc/gcloud/env.json" + if [[ ! -f "${GCLOUD_ENV_JSON}" ]]; then + echo "ERROR: cloud-dataproc/gcloud/env.json not found. Cannot resolve SWP proxy configurations." >&2 + exit 1 + fi + SWP_IP=$(jq -r .SWP_IP "${GCLOUD_ENV_JSON}") + SWP_PORT=$(jq -r .SWP_PORT "${GCLOUD_ENV_JSON}") + if [[ -z "${SWP_IP}" || "${SWP_IP}" == "null" || -z "${SWP_PORT}" || "${SWP_PORT}" == "null" ]]; then + echo "ERROR: SWP_IP or SWP_PORT is not configured in cloud-dataproc/gcloud/env.json." >&2 + exit 1 + fi + echo "INFO: Enabling SWP Proxy Egress: ${SWP_IP}:${SWP_PORT}" >&2 +fi + +declare -a METADATA_ARRAY=( + "VmDnsSetting=ZonalOnly" + "shutdown-timer-in-sec=86400" # 1 day timer for debugging + "custom-sources-path=${GCS_SOURCES_PATH}" + "dataproc-region=${region}" + "dataproc_dataproc_version=${IMAGE_VERSION}" + "invocation-type=custom-images" + "dataproc-temp-bucket=${TEMP_BUCKET}" +) + +if [[ "${USE_PROXY}" -eq 1 ]]; then + METADATA_ARRAY+=( + "http-proxy=${SWP_IP}:${SWP_PORT}" + "https-proxy=${SWP_IP}:${SWP_PORT}" + "proxy-uri=${SWP_IP}:${SWP_PORT}" + "no-proxy=metadata.google.internal,${PROJECT_ID}.svc.id.goog" + ) +fi + +# Use a custom delimiter (^;^) to allow commas in metadata values (like no-proxy) +# This prevents gcloud from splitting on commas inside the no-proxy value. +METADATA_STRING="^;^$(IFS=';'; echo "${METADATA_ARRAY[*]}")" + +# Create the instance +declare -a gcloud_create_args=( + gcloud compute instances create "${INSTANCE_NAME}" + --project "${PROJECT_ID}" + --zone "${ZONE}" + --machine-type n1-standard-2 + --image "${DATAPROC_IMAGE}" + --image-project cloud-dataproc + --boot-disk-size 30G + --boot-disk-type pd-ssd + --scopes "https://www.googleapis.com/auth/cloud-platform" + --service-account "${GSA}" + --subnet "${SUBNET}" + --metadata="${METADATA_STRING}" +) +run_gcloud create_instance "${gcloud_create_args[@]}" + +echo "Instance ${INSTANCE_NAME} created." + +SERIAL_LOG_FILE="${REPRO_TMPDIR}/serial_${INSTANCE_NAME}.log" +echo "Tailing serial port output to ${SERIAL_LOG_FILE} in the background..." +gcloud compute instances tail-serial-port-output "${INSTANCE_NAME}" --zone "${ZONE}" --project "${PROJECT_ID}" > "${SERIAL_LOG_FILE}" 2>&1 & +TAIL_PID=$! +echo "Tail PID: ${TAIL_PID}" +echo "To stop tailing: kill ${TAIL_PID}" + +echo "To SSH into the instance:" +echo "bash $(dirname "$0")/ssh-debug-vm.sh" diff --git a/examples/secure-boot/bin/customize-in-screen.sh b/examples/secure-boot/bin/customize-in-screen.sh new file mode 100644 index 0000000..ccdd845 --- /dev/null +++ b/examples/secure-boot/bin/customize-in-screen.sh @@ -0,0 +1,349 @@ +#!/bin/bash +# +# Workstation-side orchestrator for idempotent, interactive image customization. +# +# Workflow: +# 1. Detects or provisions the GCE debug VM. +# 2. Checks if a customization screen session is already running on the VM. +# 3. If running: Instantly attaches the workstation terminal to the active session. +# 4. If not running: Syncs the latest local code to GCS, triggers a remote +# background launch, and immediately attaches to the live session. + +set -euo pipefail + +DEBUG="${DEBUG:-0}" +if (( DEBUG != 0 )); then + set -x +fi +# --- Parse Options --- +CLEAN_BUILD=0 +USE_PROXY=0 +FORCE_APPLY=0 +FORCE_DELETE=0 +USAGE="Usage: bash \$(basename "\$0") [-c] [-p] [-r]" + +while getopts "cpr" opt; do + case ${opt} in + c ) + CLEAN_BUILD=1 + FORCE_APPLY=1 + ;; + p ) USE_PROXY=1 ;; + r ) FORCE_DELETE=1 ;; + \? ) echo "${USAGE}" >&2; exit 1 ;; + esac +done +shift $((OPTIND -1)) +# --- Source Environment & Helpers --- +BIN_DIR=$(dirname "$(readlink -f "$0")") +ENV_SH="${BIN_DIR}/../lib/env.sh" +UTIL_SH="${BIN_DIR}/../lib/util.sh" + +if [[ -f "${ENV_SH}" && -f "${UTIL_SH}" ]]; then + source "${ENV_SH}" + source "${UTIL_SH}" +else + echo "ERROR: Helper libraries not found in examples/secure-boot/lib/." >&2 + exit 1 +fi + +if [[ -z "${IMAGE_VERSION}" || -z "${CUSTOMIZATION_SCRIPT}" ]]; then + echo "ERROR: IMAGE_VERSION or CUSTOMIZATION_SCRIPT not set in env.json." >&2 + exit 1 +fi + +# Generate the unique instance name based on configuration +INSTANCE_NAME="debug-$(echo "${IMAGE_VERSION}" | tr '.' '-')-$(basename "${CUSTOMIZATION_SCRIPT}" .sh | tr '.' '-' | tr '_' '-')" +GCS_SOURCES_PATH="gs://${BUCKET}/${INSTANCE_NAME}/sources" + +echo "DEBUG: Target Instance: ${INSTANCE_NAME}" >&2 +echo "DEBUG: GCS Staging Path: ${GCS_SOURCES_PATH}" >&2 + +# ======================================================================== +# Step 1: Ensure GCE Instance is Online +# ======================================================================== +VM_WAS_CREATED="false" +if ! gcloud compute instances describe "${INSTANCE_NAME}" --zone "${ZONE}" --project "${PROJECT_ID}" &>/dev/null || [[ "${FORCE_DELETE}" -eq 1 ]]; then + if [[ "${FORCE_DELETE}" -eq 1 ]]; then + echo "INFO: Recreation forced (-r). Re-provisioning debug VM ${INSTANCE_NAME}..." >&2 + else + echo "INFO: Debug VM ${INSTANCE_NAME} does not exist. Provisioning a new instance..." >&2 + fi + # Call the pre-existing provisioner (cold start) + declare -a create_args=() + if [[ "${FORCE_DELETE}" -eq 1 ]]; then + create_args+=("-f") + fi + if [[ "${USE_PROXY}" -eq 1 ]]; then + create_args+=("-p") + fi + bash "${BIN_DIR}/create-debug-vm.sh" "${create_args[@]}" + VM_WAS_CREATED="true" +else + echo "INFO: Active debug VM ${INSTANCE_NAME} detected. Re-using instance." >&2 +fi + +if [[ "${VM_WAS_CREATED}" == "true" ]]; then + echo "INFO: Waiting for GCE Identity-Aware Proxy (IAP) tunnel to sync..." >&2 + set +e + IAP_READY=1 + for i in {1..36}; do + if ssh -o ControlMaster=no -o ConnectTimeout=3 -o BatchMode=yes "${INSTANCE_NAME}" "uptime" &>/dev/null; then + IAP_READY=0 + break + fi + echo "DEBUG: Waiting for IAP tunnel (+5s)..." >&2 + sleep 5 + done + set -e + + if [[ ${IAP_READY} -ne 0 ]]; then + echo "ERROR: Timeout waiting for IAP tunnel to sync. VM is online but unreachable." >&2 + exit 1 + fi + echo "INFO: IAP tunnel is active and accepting connections." >&2 +fi + +# ======================================================================== +# Step 1.1: Dynamically Inject SWP Proxy Metadata (Optional Warm Start) +# ======================================================================== +if [[ "${USE_PROXY}" -eq 1 ]]; then + # Resolve path to cloud-dataproc/gcloud/env.json relative to the authoritative repository root + # already resolved and exported by env.sh. + GCLOUD_ENV_JSON="${DATAPROC_EVOLUTION_DIR}/cloud-dataproc/gcloud/env.json" + if [[ -f "${GCLOUD_ENV_JSON}" ]]; then + SWP_IP=$(jq -r .SWP_IP "${GCLOUD_ENV_JSON}") + SWP_PORT=$(jq -r .SWP_PORT "${GCLOUD_ENV_JSON}") + if [[ -n "${SWP_IP}" && "${SWP_IP}" != "null" && -n "${SWP_PORT}" && "${SWP_PORT}" != "null" ]]; then + echo "INFO: Dynamically ensuring SWP Proxy metadata is set on the running VM..." >&2 + gcloud compute instances add-metadata "${INSTANCE_NAME}" \ + --project="${PROJECT_ID}" \ + --zone="${ZONE}" \ + --metadata="^;^http-proxy=${SWP_IP}:${SWP_PORT};https-proxy=${SWP_IP}:${SWP_PORT};proxy-uri=${SWP_IP}:${SWP_PORT};no-proxy=metadata.google.internal,${PROJECT_ID}.svc.id.goog" \ + --quiet + else + echo "ERROR: SWP_IP or SWP_PORT not found in cloud-dataproc/gcloud/env.json." >&2 + exit 1 + fi + else + echo "ERROR: cloud-dataproc/gcloud/env.json not found. Cannot resolve SWP proxy configurations." >&2 + exit 1 + fi +fi + +# ======================================================================== +# Step 1.2: Clean Build Artifacts & Sentinels (Optional) +# ======================================================================== +if [[ "${CLEAN_BUILD}" -eq 1 ]]; then + echo "INFO: Cleaning up all past build artifacts, active screen sessions, and sentinels on the VM..." >&2 + ssh -o ControlMaster=no -o BatchMode=yes -o ConnectTimeout=5 "${INSTANCE_NAME}" " + screen -S customization -X quit 2>/dev/null || true + sudo rm -rf /tmp/dataproc-repro + " &>/dev/null || true +fi + +# ======================================================================== +# Step 2: Check if customization is already running in screen +# ======================================================================== +echo "DEBUG: Checking VM for active customization screen session..." >&2 +set +e +ssh -o ControlMaster=no -o BatchMode=yes -o ConnectTimeout=5 "${INSTANCE_NAME}" "screen -ls | grep -q customization" &>/dev/null +SCREEN_STATUS=$? +set -e + +# ======================================================================== +# Step 3: Branching Idempotent Execution +# ======================================================================== +if [[ ${SCREEN_STATUS} -eq 0 ]]; then + # ---------------------------------------------------------------------- + # CASE A: Customization is already running. Attach immediately. + # ---------------------------------------------------------------------- + cat <&2 +======================================================================== + ATTACHING TO ACTIVE CUSTOMIZATION SESSION +======================================================================== +An active customization build was detected running on the VM. +Connecting your terminal to the live screen session... + +(To detach from screen and leave it running in the background, press: + Ctrl+A followed by D) +======================================================================== +EOF + sleep 1 + + # Connect and attach + ssh -t "${INSTANCE_NAME}" screen -rxU customization + +else + # ---------------------------------------------------------------------- + # CASE B: Customization is not running. Sync, launch, and attach. + # ---------------------------------------------------------------------- + echo "INFO: Customization is not active on the VM. Syncing latest code..." >&2 + + # 1. Sync local workstation scripts to GCS staging using a single bulk upload + echo "DEBUG: Syncing local assets to GCS: ${GCS_SOURCES_PATH}" >&2 + + # Create a local staging directory in the temp folder for fast preparation + LOCAL_STAGING="${REPRO_TMPDIR}/staging" + mkdir -p "${LOCAL_STAGING}" + + # Cleanly resolve customization script path + custom_script_path="${DATAPROC_EVOLUTION_DIR}/custom-images/${CUSTOMIZATION_SCRIPT}" + + # Stage all files locally (near-instantaneous) + cp "${DATAPROC_EVOLUTION_DIR}/custom-images/startup_script/run.sh" "${LOCAL_STAGING}/run.sh" + cp "${custom_script_path}" "${LOCAL_STAGING}/init_actions.sh" + cp "${DATAPROC_EVOLUTION_DIR}/custom-images/examples/secure-boot/lib/env.sh" "${LOCAL_STAGING}/env.sh" + cp "${DATAPROC_EVOLUTION_DIR}/custom-images/examples/secure-boot/lib/util.sh" "${LOCAL_STAGING}/util.sh" + cp "${DATAPROC_EVOLUTION_DIR}/custom-images/examples/secure-boot/install-in-screen.sh" "${LOCAL_STAGING}/install-in-screen.sh" + cp "${DATAPROC_EVOLUTION_DIR}/custom-images/startup_script/gce-proxy-setup.sh" "${LOCAL_STAGING}/gce-proxy-setup.sh" + + # Perform a single, parallelized bulk upload to GCS (minimizes connection overhead) + gsutil -m cp -r "${LOCAL_STAGING}/*" "${GCS_SOURCES_PATH}/" >/dev/null + + echo "INFO: Launching customization on VM..." >&2 + + # 2. SSH in, download GCS assets, and trigger the background screen launcher + ssh -o ControlMaster=no -o BatchMode=yes -o ConnectTimeout=10 "${INSTANCE_NAME}" " + GCS_PATH=\$(curl -s -H 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/instance/attributes/custom-sources-path) + rm -rf /tmp/sources + mkdir -p /tmp/sources + gsutil -m cp -r \"\${GCS_PATH}/*\" /tmp/sources/ >/dev/null + chmod +x /tmp/sources/*.sh + # Spawn the guest screen bootstrap wrapper in the background, passing FORCE_APPLY status + nohup env FORCE_APPLY=${FORCE_APPLY} bash /tmp/sources/install-in-screen.sh /tmp/sources/init_actions.sh >/tmp/launcher.log 2>&1 & + sleep 1 + " + + echo "DEBUG: Customization launched. Checking status in 2 seconds..." >&2 + sleep 2 + + # 3. Check if the customization already completed instantly before attempting to attach + set +e + ssh -o ControlMaster=no -o BatchMode=yes -o ConnectTimeout=3 "${INSTANCE_NAME}" "[[ -f /tmp/sources/customization.exit ]]" &>/dev/null + INSTANT_COMPLETED=$? + set -e + + attach_success=0 + if [[ ${INSTANT_COMPLETED} -eq 0 ]]; then + echo "INFO: Customization completed instantly on the VM. Skipping screen attachment." >&2 + attach_success=0 + else + # Connect and attach the developer interactively to the newly spawned session (belongs to cjac, so no sudo!) + set +e + ssh -t "${INSTANCE_NAME}" screen -rxU customization + attach_success=$? + set -e + fi + + # If we failed to attach (e.g., screen exited before we connected), + # harvest the exit status and logs from the VM so the developer gets immediate feedback! + if [[ ${attach_success} -ne 0 ]]; then + echo "INFO: Customization session closed or exited early. Harvesting logs from VM..." >&2 + set +e + # Fetch exit code + exit_code=$(ssh -o BatchMode=yes "${INSTANCE_NAME}" "cat /tmp/sources/customization.exit" 2>/dev/null) + has_exit=$? + set -e + + if [[ ${has_exit} -eq 0 && -n "${exit_code}" ]]; then + # Define ANSI colors for the local shell output + GREEN='\033[0;32m' + RED='\033[0;31m' + NC='\033[0m' + + echo "========================================================================" >&2 + echo " GUEST-SIDE EXECUTION LOG (Harvested from VM)" >&2 + echo "========================================================================" >&2 + ssh -o BatchMode=yes "${INSTANCE_NAME}" "cat /tmp/sources/customization-output.log" || true + echo "========================================================================" >&2 + + if [[ "${exit_code}" -eq 0 ]]; then + echo -e "${GREEN}🎉 BUILD SUCCESSFUL (Exit Code: 0)${NC}" >&2 + exit 0 + else + echo -e "${RED}❌ BUILD FAILED (Exit Code: ${exit_code})${NC}" >&2 + exit "${exit_code}" + fi + else + echo "ERROR: Failed to attach to screen, and no exit status was recorded on the VM." >&2 + echo "This indicates the startup launcher crashed or failed to spawn the screen." >&2 + echo "See /tmp/launcher.log on the VM for details." >&2 + exit 1 + fi + fi +fi + +# ======================================================================== +# Step 4: Validate Build Exit Status +# ======================================================================== +echo "INFO: Customization session closed. Fetching build exit status from VM..." >&2 + +set +e +EXIT_CODE=$(ssh -o ControlMaster=no -o ConnectTimeout=5 "${INSTANCE_NAME}" "cat /tmp/sources/customization.exit" 2>/dev/null) +SSH_STATUS=$? +set -e + +if [[ ${SSH_STATUS} -ne 0 || -z "${EXIT_CODE}" ]]; then + # Check if the screen session is still active (meaning the developer detached) + set +e + screen_still_active=$(ssh -o ControlMaster=no -o ConnectTimeout=5 "${INSTANCE_NAME}" "screen -ls | grep -q customization" 2>/dev/null; echo $?) + set -e + + if [[ ${screen_still_active} -eq 0 ]]; then + cat <&2 +======================================================================== + ℹī¸ DETACHED FROM ACTIVE CUSTOMIZATION +======================================================================== +The customization build is still executing in the background on the VM. +To re-attach and monitor the live run later, simply run this script again: + + bash examples/secure-boot/bin/customize-in-screen.sh +======================================================================== +EOF + exit 0 + else + echo "ERROR: Customization finished but exit status was not recorded on the VM." >&2 + exit 1 + fi +fi + +# Trim whitespace +EXIT_CODE=$(echo "${EXIT_CODE}" | xargs) + +if [[ "${EXIT_CODE}" == "0" ]]; then + cat <&2 +======================================================================== + 🎉 BUILD SUCCESSFUL +======================================================================== +The customization script completed successfully with exit code 0! +You can now proceed to convert this VM to a custom image: + + gcloud compute instances stop ${INSTANCE_NAME} --zone=${ZONE} --project=${PROJECT_ID} + gcloud compute images create [IMAGE_NAME] --source-disk=${INSTANCE_NAME} ... +======================================================================== +EOF + exit 0 +else + cat <&2 +======================================================================== + ❌ BUILD FAILED (Exit Code: ${EXIT_CODE}) +======================================================================== +The customization script failed inside the screen session. + +DIAGNOSTICS & RETRY WORKFLOW: +1. View the full build log: + ssh ${INSTANCE_NAME} "cat /tmp/sources/customization-output.log" + +2. Hot-fix the script directly on the VM for instant testing: + ssh ${INSTANCE_NAME} + +3. Correct the local script on your workstation: + Edit: ${CUSTOMIZATION_SCRIPT} + +4. Retry the entire pipeline (this will sync your fix and restart the build): + bash examples/secure-boot/bin/customize-in-screen.sh +======================================================================== +EOF + exit "${EXIT_CODE}" +fi diff --git a/examples/secure-boot/bin/destroy-debug-vm.sh b/examples/secure-boot/bin/destroy-debug-vm.sh new file mode 100755 index 0000000..e5b2b4a --- /dev/null +++ b/examples/secure-boot/bin/destroy-debug-vm.sh @@ -0,0 +1,52 @@ +#!/bin/bash +# +# Helper script to destroy a debug VM created by create-debug-vm.sh +# + +set -e + +DEBUG="${DEBUG:-0}" +if (( DEBUG != 0 )); then + set -x +fi + +# Source environment variables +if [[ -f "$(dirname "$0")/../lib/env.sh" ]]; then + source "$(dirname "$0")/../lib/env.sh" +else + echo "ERROR: examples/secure-boot/lib/env.sh not found." + exit 1 +fi +if [[ -f "$(dirname "$0")/../lib/util.sh" ]]; then + source "$(dirname "$0")/../lib/util.sh" +else + echo "ERROR: examples/secure-boot/lib/util.sh not found." + exit 1 +fi + +USAGE="Usage: bash $(basename "$0")" + +if [[ -z "${IMAGE_VERSION}" ]]; then + echo "ERROR: IMAGE_VERSION is not set in env.json." + exit 1 +fi +if [[ -z "${CUSTOMIZATION_SCRIPT}" ]]; then + echo "ERROR: CUSTOMIZATION_SCRIPT is not set in env.json." + exit 1 +fi + +INSTANCE_NAME="debug-$(echo "${IMAGE_VERSION}" | tr '.' '-')-$(basename "${CUSTOMIZATION_SCRIPT}" .sh | tr '.' '-' | tr '_' '-')" + +echo "Attempting to delete instance: ${INSTANCE_NAME}" +echo "Project: ${PROJECT_ID}, Zone: ${ZONE}" + +if gcloud compute instances describe "${INSTANCE_NAME}" --zone "${ZONE}" --project "${PROJECT_ID}" > /dev/null 2>&1; then + run_gcloud delete_instance gcloud compute instances delete "${INSTANCE_NAME}" --zone "${ZONE}" --project "${PROJECT_ID}" -q + echo "Instance ${INSTANCE_NAME} deleted." +else + echo "Instance ${INSTANCE_NAME} not found." +fi + +GCS_SOURCES_PATH="gs://${BUCKET}/${INSTANCE_NAME}/sources" +echo "Cleaning up GCS path: ${GCS_SOURCES_PATH}" +run_gsutil rm_gcs gsutil -m rm -r "${GCS_SOURCES_PATH}" || echo "GCS path not found, continuing..." diff --git a/examples/secure-boot/bin/get-builder-log b/examples/secure-boot/bin/get-builder-log new file mode 100755 index 0000000..74f8dcf --- /dev/null +++ b/examples/secure-boot/bin/get-builder-log @@ -0,0 +1,38 @@ +#!/bin/bash +# Get the full serial port output of the latest builder VM for a given image prefix. + +set -e + +DEBUG="${DEBUG:-0}" +if (( DEBUG != 0 )); then + set -x +fi + +source examples/secure-boot/lib/env.sh +source examples/secure-boot/lib/util.sh + +IMAGE_PREFIX="${1:-dataproc-2-0-deb10}" +PURPOSE="${2:-secure-boot}" + +print_status "Finding latest builder instance for ${IMAGE_PREFIX}*${PURPOSE}"... + +INSTANCE_NAME=$(gcloud compute instances list \ + --project="${PROJECT_ID}" \ + --zones="${ZONE}" \ + --filter="name~'${IMAGE_PREFIX}.*${PURPOSE}-install'" \ + --format="value(name)" \ + --sort-by="~creationTimestamp" \ + | head -n 1) + +if [[ -z "${INSTANCE_NAME}" ]]; then + report_result "Not Found" + echo "No builder instance found matching the prefix." >&2 + exit 1 +fi + +report_result "Found: ${INSTANCE_NAME}" + +print_status "Getting serial port output for ${INSTANCE_NAME}"... +echo "" # Add a newline after the status + +gcloud compute instances get-serial-port-output "${INSTANCE_NAME}" --zone "${ZONE}" --project "${PROJECT_ID}" --port 1 | less diff --git a/examples/secure-boot/bin/scp-debug-vm.sh b/examples/secure-boot/bin/scp-debug-vm.sh new file mode 100755 index 0000000..c6e8289 --- /dev/null +++ b/examples/secure-boot/bin/scp-debug-vm.sh @@ -0,0 +1,51 @@ +#!/bin/bash +# +# Helper script to SCP files to the /tmp directory of a debug VM +# + +set -e + +DEBUG="${DEBUG:-0}" +if (( DEBUG != 0 )); then + set -x +fi + +# Source environment variables +if [[ -f "$(dirname "$0")/../lib/env.sh" ]]; then + source "$(dirname "$0")/../lib/env.sh" +else + echo "ERROR: examples/secure-boot/lib/env.sh not found." + exit 1 +fi + +USAGE="Usage: bash $(basename "$0") " + +if [[ -z "${IMAGE_VERSION}" ]]; then + echo "ERROR: IMAGE_VERSION is not set in env.json." + exit 1 +fi +if [[ -z "${CUSTOMIZATION_SCRIPT}" ]]; then + echo "ERROR: CUSTOMIZATION_SCRIPT is not set in env.json." + exit 1 +fi + +LOCAL_PATH="${1}" +if [[ -z "${LOCAL_PATH}" ]]; then + echo "ERROR: Missing local path argument." + echo "${USAGE}" + exit 1 +fi + +if [[ ! -e "${LOCAL_PATH}" ]]; then + echo "ERROR: Local path not found: ${LOCAL_PATH}" + exit 1 +fi + +INSTANCE_NAME="debug-$(echo "${IMAGE_VERSION}" | tr '.' '-')-$(basename "${CUSTOMIZATION_SCRIPT}" .sh | tr '.' '-' | tr '_' '-')" + +echo "Attempting to SCP '${LOCAL_PATH}' to ${INSTANCE_NAME}:/tmp/" +echo "Project: ${PROJECT_ID}, Zone: ${ZONE}" + +gcloud compute scp --recurse --zone "${ZONE}" --project "${PROJECT_ID}" --tunnel-through-iap "${LOCAL_PATH}" "${INSTANCE_NAME}:/tmp/" + +echo "File/directory copied successfully to ${INSTANCE_NAME}:/tmp/$(basename "${LOCAL_PATH}")" diff --git a/examples/secure-boot/bin/ssh-builder b/examples/secure-boot/bin/ssh-builder new file mode 100755 index 0000000..51067a1 --- /dev/null +++ b/examples/secure-boot/bin/ssh-builder @@ -0,0 +1,34 @@ +#!/bin/bash +# SSH into the latest builder VM for a given image prefix. + +set -e + +source examples/secure-boot/lib/env.sh +source examples/secure-boot/lib/util.sh + +IMAGE_PREFIX="${1:-dataproc-2-2-deb10}" +PURPOSE="${2:-secure-boot}" + +print_status "Finding latest builder instance for ${IMAGE_PREFIX}*${PURPOSE}"... + +INSTANCE_NAME=$(gcloud compute instances list \ + --project="${PROJECT_ID}" \ + --zones="${ZONE}" \ + --filter="name~'${IMAGE_PREFIX}.*${PURPOSE}-install' AND status=RUNNING" \ + --format="value(name)" \ + --sort-by=\"~creationTimestamp\" \ + | head -n 1) + +if [[ -z "${INSTANCE_NAME}" ]]; then + report_result "Not Found" + echo "No RUNNING builder instance found matching the prefix." >&2 +# exit 1 +fi + +report_result "Found: ${INSTANCE_NAME}" + +print_status "SSHing into ${INSTANCE_NAME}"... + +INSTANCE_NAME=debug-deb12-build + +gcloud compute ssh "${INSTANCE_NAME}" --zone "${ZONE}" --project "${PROJECT_ID}" diff --git a/examples/secure-boot/bin/ssh-debug-vm.sh b/examples/secure-boot/bin/ssh-debug-vm.sh new file mode 100755 index 0000000..a5498a5 --- /dev/null +++ b/examples/secure-boot/bin/ssh-debug-vm.sh @@ -0,0 +1,56 @@ +#!/bin/bash +# +# Helper script to SSH into a debug VM created by create-debug-vm.sh +# + +set -e + +DEBUG="${DEBUG:-0}" +if (( DEBUG != 0 )); then + set -x +fi + +# Source environment variables +if [[ -f "$(dirname "$0")/../lib/env.sh" ]]; then + source "$(dirname "$0")/../lib/env.sh" +else + echo "ERROR: examples/secure-boot/lib/env.sh not found." + exit 1 +fi + +USAGE="Usage: bash $(basename "$0") [command...]" + +if [[ -z "${IMAGE_VERSION}" ]]; then + echo "ERROR: IMAGE_VERSION is not set in env.json." + exit 1 +fi +if [[ -z "${CUSTOMIZATION_SCRIPT}" ]]; then + echo "ERROR: CUSTOMIZATION_SCRIPT is not set in env.json." + exit 1 +fi + +COMMAND_TO_RUN=("$@") + +INSTANCE_NAME="debug-$(echo "${IMAGE_VERSION}" | tr '.' '-')-$(basename "${CUSTOMIZATION_SCRIPT}" .sh | tr '.' '-' | tr '_' '-')" + +echo "Attempting to SSH into instance: ${INSTANCE_NAME}" +echo "Project: ${PROJECT_ID}, Zone: ${ZONE}" + +declare -a gcloud_ssh_args +gcloud_ssh_args=( + gcloud compute ssh + --zone "${ZONE}" + --project "${PROJECT_ID}" + --tunnel-through-iap + "${INSTANCE_NAME}" +) + +if [[ ${#COMMAND_TO_RUN[@]} -eq 0 ]]; then + # Interactive session + gcloud_ssh_args+=( -- -t -o ConnectTimeout=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -AY ) +else + # Command execution + gcloud_ssh_args+=( --command "${COMMAND_TO_RUN[*]}" ) +fi + +"${gcloud_ssh_args[@]}" diff --git a/examples/secure-boot/build-and-run-podman.sh b/examples/secure-boot/build-and-run-podman.sh index edb3ce7..d9d0598 100644 --- a/examples/secure-boot/build-and-run-podman.sh +++ b/examples/secure-boot/build-and-run-podman.sh @@ -147,6 +147,9 @@ time podman run -it --rm \ -e DEBUG=0 \ -e REPRO_TMPDIR=/tmp \ -e DATAPROC_EVOLUTION_DIR=${DATAPROC_EVOLUTION_DIR} \ + -e SWP_IP="${SWP_IP}" \ + -e SWP_PORT="${SWP_PORT}" \ + -e PROXY_CERT_GCS_PATH="${PROXY_CERT_GCS_PATH}" \ ${image} \ bash -x examples/secure-boot/pre-init.sh "${DATAPROC_IMAGE_VERSION}" # bash examples/secure-boot/build-current-images.sh diff --git a/examples/secure-boot/build-current-images.sh b/examples/secure-boot/build-current-images.sh index b3309d1..29c19bc 100644 --- a/examples/secure-boot/build-current-images.sh +++ b/examples/secure-boot/build-current-images.sh @@ -65,7 +65,7 @@ function find_disk_usage() { grep -H '^[^\+].*Cust.*ript' /tmp/custom-image-*${timestamp}*/logs/workflow.log echo '# DP_IMG_VER RECOMMENDED_DISK_SIZE DSK_SZ D_USED D_FREE D%F PURPOSE' # workflow_log=/tmp/custom-image-dataproc-2-0-deb10-20250424-232955-tf-20250425-230559/logs/workflow.log - for workflow_log in $(grep -Hl "Customization script" /tmp/custom-image-*/logs/workflow.log) ; do + for workflow_log in $(grep -Hl "Customization script" /tmp/custom-image-*${timestamp}*/logs/workflow.log 2>/dev/null || true) ; do startup_log="${workflow_log/workflow/startup-script}" grep -v '^\[' "${startup_log}" \ | grep -A20 'Filesystem.*Avail' | tail -20 \ diff --git a/examples/secure-boot/install-in-screen.sh b/examples/secure-boot/install-in-screen.sh new file mode 100644 index 0000000..c2722f2 --- /dev/null +++ b/examples/secure-boot/install-in-screen.sh @@ -0,0 +1,159 @@ +#!/bin/bash +# +# Bootstrap wrapper to run GCE image customization scripts inside a detached, +# inspectable screen session. This protects long-running builds from timeouts, +# allows real-time developer attachment, and ensures logs are still streamed +# to the serial console for the orchestrator. + +set -euo pipefail + +SCRIPT_DIR=$(dirname "$(readlink -f "$0")") +TARGET_SCRIPT="${1:-${SCRIPT_DIR}/no-customization.sh}" + +if [[ ! -f "${TARGET_SCRIPT}" ]]; then + echo "ERROR: Target customization script ${TARGET_SCRIPT} not found." >&2 + exit 1 +fi + +echo "INFO: Preparing to run customization: ${TARGET_SCRIPT}" >&2 + +# --- Retrieve VM Details from Cache --- +# We use the local JSON cache to print the exact SSH command for the developer +CACHE_DIR="/dev/shm/metadata_cache" +PROJECT_ID="UNKNOWN_PROJECT" +ZONE="UNKNOWN_ZONE" +VM_NAME="UNKNOWN_VM" + +if [[ -d "${CACHE_DIR}" ]]; then + # Parse cached values using jq + if [[ -f "${CACHE_DIR}/project_attributes.json" ]]; then + PROJECT_ID=$(jq -r '.["project-id"] // "UNKNOWN_PROJECT"' "${CACHE_DIR}/project_attributes.json") + fi + # We can also query standard GCE metadata paths from the VM + # (These are not attributes, so they might not be in the cache, but they are safe to query once) + VM_NAME=$(curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/name || echo "UNKNOWN_VM") + ZONE_FULL=$(curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/zone || echo "UNKNOWN_ZONE") + ZONE="${ZONE_FULL##*/}" +fi + +# --- Print Developer Diagnostic Instructions --- +# This goes to stderr/console so it is visible in the GCE serial port logs immediately +cat <&2 +======================================================================== + CUSTOMIZATION SCREEN AUTOMATION ACTIVE +======================================================================== +The customization script is executing inside a detached screen session. + +To attach to the live session and monitor/debug in real-time, +run the following command from your workstation: + + gcloud compute ssh ${VM_NAME} \\ + --project=${PROJECT_ID} \\ + --zone=${ZONE} \\ + --tunnel-through-iap \\ + --command="screen -rxU customization" + +======================================================================== +EOF + +# --- Launch Customization in Screen --- +LOG_FILE="/tmp/sources/customization-output.log" +EXIT_FILE="/tmp/sources/customization.exit" +touch "${LOG_FILE}" +rm -f "${EXIT_FILE}" + +echo "INFO: Spawning screen session 'customization'..." >&2 + +# We spawn the screen as the normal user, but run the inner payload as root via sudo +screen -dmS customization sudo FORCE_APPLY="${FORCE_APPLY:-0}" bash -c " + # Define ANSI colors inside the root shell + BLUE='\\033[0;34m' + GREEN='\\033[0;32m' + YELLOW='\\033[1;33m' + RED='\\033[0;31m' + NC='\\033[0m' + + set +e + + # 1. Source guest-side harvester libraries inside the root shell (defines get_metadata_attribute) + source /tmp/sources/gce-proxy-setup.sh + set +e + + # 2. Globally export all metadata helper functions to match Dataproc environment fidelity + export -f get_metadata_attribute get_metadata_value print_metadata_value print_metadata_value_if_exists get_cached_state os_id is_debuntu is_rocky + + # 3. Wait for developer to attach (guarantees visibility for fast scripts/probes) + echo -e \"\${YELLOW}INFO: Spawning screen. Waiting 5 seconds for developer to attach...\${NC}\" >&2 + for i in 5 4 3 2 1; do + echo -e \"\${YELLOW}Starting customization in \${i} seconds...\${NC}\" >&2 + sleep 1 + done + + # 4. Execute the target script with a clean colored header + echo -e \"\${BLUE}========================================================================\${NC}\" + echo -e \"\${BLUE} LAUNCHING CUSTOMIZATION SCRIPT: ${TARGET_SCRIPT}\${NC}\" + echo -e \"\${BLUE}========================================================================\${NC}\" + echo '' + + bash ${TARGET_SCRIPT} 2>&1 | tee ${LOG_FILE} + EXIT_CODE=\${PIPESTATUS[0]} + + # 5. Print clean colored final status indicator + echo '' + echo -e \"\${BLUE}========================================================================\${NC}\" + if [[ \${EXIT_CODE} -eq 0 ]]; then + echo -e \"\${GREEN} 🎉 CUSTOMIZATION SUCCESSFUL (Exit Code: 0)\${NC}\" + else + echo -e \"\${RED} ❌ CUSTOMIZATION FAILED (Exit Code: \${EXIT_CODE})\${NC}\" + fi + echo -e \"\${BLUE}========================================================================\${NC}\" + + echo \${EXIT_CODE} > ${EXIT_FILE} + chmod 644 ${EXIT_FILE} ${LOG_FILE} 2>/dev/null || true +" + +# --- Wait and Stream Logs --- +# We find the PID of the newly spawned screen session +sleep 2 +SCREEN_PID=$(screen -ls | grep customization | awk '{print $1}' | cut -d. -f1 || echo "") + +if [[ -z "${SCREEN_PID}" ]]; then + # If the screen is gone, check if it already completed instantly + if [[ -f "${EXIT_FILE}" ]]; then + echo "INFO: Customization completed instantly." >&2 + exit 0 + fi + echo "ERROR: Failed to spawn screen session." >&2 + exit 1 +fi + +echo "DEBUG: Screen session spawned with PID ${SCREEN_PID}. Streaming logs to serial port..." >&2 + +# Start tailing the log file in the background to stream to GCE serial port +tail -f "${LOG_FILE}" & +TAIL_PID=$! + +# Block until the screen session exits +while kill -0 "${SCREEN_PID}" 2>/dev/null; do + sleep 5 +done + +# Stop the background tailing +kill "${TAIL_PID}" &>/dev/null || true + +# --- Propagate Exit Code --- +# Read the exit code written by the script inside the screen session +if [[ -f "${EXIT_FILE}" ]]; then + EXIT_CODE=$(cat "${EXIT_FILE}") + echo "INFO: Customization script finished with exit code ${EXIT_CODE}" >&2 + + if [[ "${EXIT_CODE}" -eq 0 ]]; then + echo "startup-script: BuildSucceeded: Customization complete." >&2 + else + echo "startup-script: BuildFailed: Customization failed." >&2 + fi + exit "${EXIT_CODE}" +else + echo "ERROR: Customization finished but exit code was not recorded." >&2 + exit 1 +fi diff --git a/examples/secure-boot/lib/env.sh b/examples/secure-boot/lib/env.sh index 9fbf9b7..207e25c 100644 --- a/examples/secure-boot/lib/env.sh +++ b/examples/secure-boot/lib/env.sh @@ -14,13 +14,25 @@ # # This script loads and validates environment variables from env.json -if [[ -z "${ENV_JSON_PATH}" ]]; then +if [[ -z "${ENV_JSON_PATH:-}" ]]; then ENV_JSON_PATH="env.json" fi if [[ -z "${DATAPROC_EVOLUTION_DIR:-}" ]]; then - SCRIPT_DIR="$(dirname "${BASH_SOURCE[0]}")" - export DATAPROC_EVOLUTION_DIR="$(realpath "${SCRIPT_DIR}/../../../..")" + # Traverse upwards from env.sh's directory to resolve the repository root + # containing both custom-images and cloud-dataproc, bypassing nested git boundaries. + current_dir=$(dirname "$(readlink -f "${BASH_SOURCE[0]}")") + while [[ "${current_dir}" != "/" ]]; do + if [[ -d "${current_dir}/custom-images" && -d "${current_dir}/cloud-dataproc" ]]; then + export DATAPROC_EVOLUTION_DIR="${current_dir}" + break + fi + current_dir=$(dirname "${current_dir}") + done + if [[ -z "${DATAPROC_EVOLUTION_DIR:-}" ]]; then + echo "ERROR: Cannot resolve dataproc-evolution repository root from ${BASH_SOURCE[0]}." >&2 + exit 1 + fi fi if [[ ! -f "${ENV_JSON_PATH}" ]]; then @@ -44,6 +56,10 @@ DOMAIN="$(jq -r .DOMAIN "${ENV_JSON_PATH}")" IMAGE_VERSION="$(jq -r .IMAGE_VERSION "${ENV_JSON_PATH}")" CUSTOMIZATION_SCRIPT="$(jq -r .CUSTOMIZATION_SCRIPT "${ENV_JSON_PATH}")" +SWP_IP="$(jq -r .SWP_IP "${ENV_JSON_PATH}")"; [[ "${SWP_IP}" == "null" ]] && SWP_IP="" +SWP_PORT="$(jq -r .SWP_PORT "${ENV_JSON_PATH}")"; [[ "${SWP_PORT}" == "null" ]] && SWP_PORT="" +PROXY_CERT_GCS_PATH="$(jq -r .PROXY_CERT_GCS_PATH "${ENV_JSON_PATH}")"; [[ "${PROXY_CERT_GCS_PATH}" == "null" ]] && PROXY_CERT_GCS_PATH="" + # Validate all required variables from env.json missing_vars=() required_vars=(PROJECT_ID PURPOSE BUCKET TEMP_BUCKET ZONE SUBNET PRINCIPAL_USER DOMAIN IMAGE_VERSION CUSTOMIZATION_SCRIPT) @@ -70,7 +86,7 @@ if [ ${#missing_vars[@]} -gt 0 ]; then exit 1 fi -export PROJECT_ID PURPOSE BUCKET TEMP_BUCKET ZONE SUBNET HIVE_NAME HIVEDB_PW_URI SECRET_NAME KMS_KEY_URI PRINCIPAL_USER DOMAIN IMAGE_VERSION CUSTOMIZATION_SCRIPT +export PROJECT_ID PURPOSE BUCKET TEMP_BUCKET ZONE SUBNET HIVE_NAME HIVEDB_PW_URI SECRET_NAME KMS_KEY_URI PRINCIPAL_USER DOMAIN IMAGE_VERSION CUSTOMIZATION_SCRIPT SWP_IP SWP_PORT PROXY_CERT_GCS_PATH PRINCIPAL="${PRINCIPAL_USER}@${DOMAIN}" export PRINCIPAL diff --git a/examples/secure-boot/lib/util.sh b/examples/secure-boot/lib/util.sh index b79e06e..d8dac33 100644 --- a/examples/secure-boot/lib/util.sh +++ b/examples/secure-boot/lib/util.sh @@ -58,8 +58,9 @@ function run_gcloud() { print_status " Executing: ${cmd_array[*]}..." - "${cmd_array[@]}" > "${log_path}" 2>&1 - local retval=$? + local retval=0 + "${cmd_array[@]}" > "${log_path}" 2>&1 || retval=$? + readonly retval if [[ ${retval} -ne 0 ]]; then report_result "FAIL" "${log_path}" diff --git a/examples/secure-boot/no-customization.sh b/examples/secure-boot/no-customization.sh index d9d1f1f..08ae4d8 100644 --- a/examples/secure-boot/no-customization.sh +++ b/examples/secure-boot/no-customization.sh @@ -43,6 +43,18 @@ print( " samples-taken: ", scalar @siz, $/, # Monitor disk usage in a screen session df / | tee "/run/disk-usage.log" touch "/run/keep-running-df" + +# Ensure screen is installed for disk monitoring +if ! command -v screen >/dev/null 2>&1; then + echo "INFO: Installing screen for disk usage monitoring..." >&2 + if command -v dnf >/dev/null 2>&1; then + sudo dnf -y -q install epel-release && sudo dnf -y -q install screen + elif command -v apt-get >/dev/null 2>&1; then + sudo apt-get update -y -qq >/dev/null 2>&1 + sudo apt-get install -y -qq screen >/dev/null 2>&1 + fi +fi + screen -d -m -LUS keep-running-df \ bash -c "while [[ -f /run/keep-running-df ]] ; do df / | tee -a /run/disk-usage.log ; sleep 5s ; done" diff --git a/examples/secure-boot/pre-init.screenrc b/examples/secure-boot/pre-init.screenrc index a3aa589..603f783 100644 --- a/examples/secure-boot/pre-init.screenrc +++ b/examples/secure-boot/pre-init.screenrc @@ -18,8 +18,10 @@ screen -L -t 2.1-ubuntu20-arm 11 /bin/bash -x examples/secure-boot/pre-init.sh 2 screen -L -t 2.2-debian12 8 /bin/bash -x examples/secure-boot/pre-init.sh 2.2-debian12 screen -L -t 2.2-rocky9 9 /bin/bash -x examples/secure-boot/pre-init.sh 2.2-rocky9 screen -L -t 2.2-ubuntu22 10 /bin/bash -x examples/secure-boot/pre-init.sh 2.2-ubuntu22 +screen -L -t 2.2-ubuntu22-arm 16 /bin/bash -x examples/secure-boot/pre-init.sh 2.2-ubuntu22-arm screen -L -t 2.3-debian12 12 /bin/bash -x examples/secure-boot/pre-init.sh 2.3-debian12 screen -L -t 2.3-rocky9 13 /bin/bash -x examples/secure-boot/pre-init.sh 2.3-rocky9 screen -L -t 2.3-ubuntu22 14 /bin/bash -x examples/secure-boot/pre-init.sh 2.3-ubuntu22 screen -L -t 2.3-ml-ubuntu22 15 /bin/bash -x examples/secure-boot/pre-init.sh 2.3-ml-ubuntu22 +screen -L -t 2.3-ubuntu22-arm 17 /bin/bash -x examples/secure-boot/pre-init.sh 2.3-ubuntu22-arm diff --git a/examples/secure-boot/pre-init.sh b/examples/secure-boot/pre-init.sh index 9329613..bf9627d 100644 --- a/examples/secure-boot/pre-init.sh +++ b/examples/secure-boot/pre-init.sh @@ -66,13 +66,15 @@ case "${dataproc_version}" in "2.1-rocky8" ) CUDA_VERSION="12.4.1" ; short_dp_ver=2.1-roc8 ;; "2.1-ubuntu20" ) CUDA_VERSION="12.4.1" ; short_dp_ver=2.1-ubu20 ;; "2.1-ubuntu20-arm" ) CUDA_VERSION="12.4.1" ; short_dp_ver=2.1-ubu20-arm ;; - "2.2-debian12" ) CUDA_VERSION="13.1.0" ; short_dp_ver=2.2-deb12 ;; - "2.2-rocky9" ) CUDA_VERSION="13.1.0" ; short_dp_ver=2.2-roc9 ;; - "2.2-ubuntu22" ) CUDA_VERSION="13.1.0" ; short_dp_ver=2.2-ubu22 ;; - "2.3-debian12" ) CUDA_VERSION="13.1.0" ; short_dp_ver=2.3-deb12 ;; - "2.3-rocky9" ) CUDA_VERSION="13.1.0" ; short_dp_ver=2.3-roc9 ;; - "2.3-ubuntu22" ) CUDA_VERSION="13.1.0" ; short_dp_ver=2.3-ubu22 ;; - "2.3-ml-ubuntu22" ) CUDA_VERSION="13.1.0" ; short_dp_ver=2.3-ml-ubu22 ; disk_size_gb="50";; + "2.2-debian12" ) CUDA_VERSION="13.2.0" ; short_dp_ver=2.2-deb12 ;; + "2.2-rocky9" ) CUDA_VERSION="13.2.0" ; short_dp_ver=2.2-roc9 ;; + "2.2-ubuntu22" ) CUDA_VERSION="13.2.0" ; short_dp_ver=2.2-ubu22 ;; + "2.2-ubuntu22-arm" ) CUDA_VERSION="13.2.0" ; short_dp_ver=2.2-ubu22-arm ;; + "2.3-debian12" ) CUDA_VERSION="13.2.0" ; short_dp_ver=2.3-deb12 ;; + "2.3-rocky9" ) CUDA_VERSION="13.2.0" ; short_dp_ver=2.3-roc9 ;; + "2.3-ubuntu22" ) CUDA_VERSION="13.2.0" ; short_dp_ver=2.3-ubu22 ;; + "2.3-ubuntu22-arm" ) CUDA_VERSION="13.2.0" ; short_dp_ver=2.3-ubu22-arm ;; + "2.3-ml-ubuntu22" ) CUDA_VERSION="13.2.0" ; short_dp_ver=2.3-ml-ubu22 ; disk_size_gb="50";; esac function create_h100_instance() { @@ -95,6 +97,12 @@ function create_unaccelerated_instance() { $* } +function create_arm_instance() { + python3 generate_custom_image.py \ + --machine-type "t2a-standard-2" \ + $* +} + OPTIONAL_COMPONENTS_ARG="" function generate() { @@ -154,6 +162,9 @@ function generate() { metadata_args+=("universe-domain=${universe_domain}") create_function="create_unaccelerated_instance" + if [[ "${dataproc_version}" == *arm* ]]; then + create_function="create_arm_instance" + fi if [[ "${customization_script}" =~ "cloud-sql-proxy.sh" ]] ; then metadata_args+=( @@ -251,7 +262,7 @@ function generate() { else local exit_code=$? report_result "Fail" - local img_build_dir="$(ls -d /tmp/custom-image-${image_name}-* 2>/dev/null || echo '')" + local img_build_dir="$(ls -d /tmp/custom-image-${image_name}-* 2>/dev/null | sort | tail -n1 || echo '')" # retry if the startup-script.log file does not exist or is empty if [[ -n "${img_build_dir}" ]]; then local startup_script_log="${img_build_dir}/logs/startup-script.log" @@ -344,7 +355,7 @@ time generate_from_dataproc_version "${dataproc_version}" # Configure a proxy on secure-boot image PURPOSE="secure-proxy" -customization_script="startup_script/gce-proxy-setup.sh" +customization_script="${DATAPROC_EVOLUTION_DIR}/initialization-actions/http-proxy/http-proxy.sh" print_status "=== Generating base ${PURPOSE} image for ${dataproc_version} ===" time generate_from_base_purpose "secure-boot" @@ -386,7 +397,7 @@ esac # Extract major.minor version (e.g., 2.2 from 2.2-debian12) MAJOR_MINOR_VERSION=$(echo "${dataproc_version}" | cut -d'-' -f1) -if version_ge "${MAJOR_MINOR_VERSION}" "2.2" ; then +if version_ge "${MAJOR_MINOR_VERSION}" "2.2" && [[ "${dataproc_version}" != *arm* ]] ; then print_status "=== Generating GPU/ML images for ${dataproc_version} (>=2.2) ===" # Install GPU drivers + cuda + rapids + cuDNN + nccl + tensorflow + pytorch on dataproc base image @@ -400,22 +411,28 @@ if version_ge "${MAJOR_MINOR_VERSION}" "2.2" ; then fi if version_ge "${MAJOR_MINOR_VERSION}" "2.2" ; then - # Install GPU drivers + cuda + rapids + cuDNN + nccl + tensorflow + pytorch on dataproc base image on a proxy base - PURPOSE="proxy-tf" - customization_script="${DATAPROC_EVOLUTION_DIR}/initialization-actions/gpu/install_gpu_driver.sh" - print_status "=== Waiting for TF build to complete to leverage cache... ===" - while [[ ! -f "${tmpdir}/sentinels/tf_build_complete" ]]; do - sleep 10 - done - print_status "=== Generating proxy-tf image for ${dataproc_version} ===" - time generate_from_base_purpose "secure-proxy" + DOCKER_BASE_IMAGE="secure-proxy" + + if [[ "${dataproc_version}" != *arm* ]] ; then + # Install GPU drivers + cuda + rapids + cuDNN + nccl + tensorflow + pytorch on dataproc base image on a proxy base + PURPOSE="proxy-tf" + customization_script="${DATAPROC_EVOLUTION_DIR}/initialization-actions/gpu/install_gpu_driver.sh" + print_status "=== Waiting for TF build to complete to leverage cache... ===" + while [[ ! -f "${tmpdir}/sentinels/tf_build_complete" ]]; do + sleep 10 + done + print_status "=== Generating proxy-tf image for ${dataproc_version} ===" + time generate_from_base_purpose "secure-proxy" + + DOCKER_BASE_IMAGE="proxy-tf" + fi ## run the installer for the DOCKER optional component PURPOSE="docker" OPTIONAL_COMPONENTS_ARG='--optional-components=DOCKER' customization_script="examples/secure-boot/no-customization.sh" print_status "=== Generating ${PURPOSE} image for ${dataproc_version} ===" - time generate_from_base_purpose "proxy-tf" + time generate_from_base_purpose "${DOCKER_BASE_IMAGE}" ## run the installer for the DOCKER optional component PURPOSE="jupyter" @@ -446,6 +463,7 @@ if version_ge "${MAJOR_MINOR_VERSION}" "2.2" ; then time generate_from_base_purpose "pig" fi +if [[ "${dataproc_version}" != *arm* ]]; then ## Execute spark-rapids/spark-rapids.sh init action on base image PURPOSE="spark" @@ -513,5 +531,7 @@ customization_script="examples/secure-boot/pytorch.sh" print_status "=== Generating pytorch image for ${dataproc_version} ===" echo time generate_from_base_purpose "tf" +fi + From 9d2817b5e453270c3c0a99770e9e13ff3014b775 Mon Sep 17 00:00:00 2001 From: "C.J. Collier" Date: Sat, 13 Jun 2026 16:23:23 +0000 Subject: [PATCH 2/3] Support Arm builds, interactive debugging, and proxy configuration Update the Secure Boot custom image builder to support Arm architectures, add an interactive debugging workflow, configure Secure Web Proxy (SWP), update path resolution, and sanitize the public .gitignore. ### 1. Arm Architecture Support * **Targets:** Added `2.2-ubuntu22-arm` and `2.3-ubuntu22-arm` image chains. * **Machine Type:** Added `create_arm_instance` to use `t2a-standard-2` GCE instances for Arm builds. * **Build Paths:** Bypass GPU/ML stages (TensorFlow, PyTorch, RAPIDS, Spark) for Arm architectures. * **Base Image:** Fall back to `secure-proxy` instead of `proxy-tf` for optional components (Docker, Jupyter, Zeppelin, Pig) on Arm. * **Prerelease Support:** Added temporary prerelease base image URI for `2.3-ubuntu22-arm` to enable testing. ### 2. Interactive Debugging Workflow * **Documentation:** Documented the `customize-in-screen.sh` workflow for running customization on persistent debug VMs. * **Orchestration:** Added `customize-in-screen.sh` to manage the idempotent VM lifecycle and screen attachment. * **Guest Bootstrap:** Added `install-in-screen.sh` to run customization inside a detached screen session on the VM. * **Diagnostics:** Added `audit-image-customizer.sh` to perform remote network and proxy connectivity audits. * **Helpers:** Added `create-debug-vm.sh`, `destroy-debug-vm.sh`, `ssh-debug-vm.sh`, and `scp-debug-vm.sh` to manage the debug VM. * **Cleanup:** Added `cleanup-builders.sh` to clean up lingering builder VMs in parallel using screen. ### 3. Proxy & Network Integration * **SWP Variables:** Pass `SWP_IP`, `SWP_PORT`, and `PROXY_CERT_GCS_PATH` through the Podman runner to the builder VM. * **Environment:** Parse and export SWP variables from `env.json`. * **Integration:** Configure the builder and debug VM to use the local `gce-proxy-setup.sh` as the authoritative proxy configuration script. ### 4. Code Robustness & Sanitization * **Path Resolution:** Traverse parent directories in `env.sh` to resolve `DATAPROC_EVOLUTION_DIR` across nested git boundaries. * **Error Handling:** Update `run_gcloud` to capture and return exit codes under `set -e`. * **Globbing:** Prevent errors in `build-current-images.sh` when no logs match the timestamp. * **Repository Sanitization:** Sanitize `.gitignore` to remove development scratchpads, local draft scripts, and specific file patterns, replacing them with generic rules for keys, logs, and temporary directories. TAG=agy CONV=b274b565-1bd6-43f1-b4db-31f3d89d087b --- examples/secure-boot/pre-init.sh | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/examples/secure-boot/pre-init.sh b/examples/secure-boot/pre-init.sh index bf9627d..ea4c214 100644 --- a/examples/secure-boot/pre-init.sh +++ b/examples/secure-boot/pre-init.sh @@ -81,26 +81,26 @@ function create_h100_instance() { python3 generate_custom_image.py \ --machine-type "a3-highgpu-2g" \ --accelerator "type=nvidia-h100-80gb,count=2" \ - $* + "$@" } function create_t4_instance() { python3 generate_custom_image.py \ --machine-type "n1-standard-32" \ --accelerator "type=nvidia-tesla-t4,count=1" \ - $* + "$@" } function create_unaccelerated_instance() { python3 generate_custom_image.py \ --machine-type "n1-standard-2" \ - $* + "$@" } function create_arm_instance() { python3 generate_custom_image.py \ --machine-type "t2a-standard-2" \ - $* + "$@" } OPTIONAL_COMPONENTS_ARG="" @@ -322,6 +322,7 @@ function generate_from_prerelease_version() { "2.3-debian12" ) image_uri="${img_pfx}/dataproc-2-3-deb12-${src_timestamp}-rc01" ;; "2.3-rocky9" ) image_uri="${img_pfx}/dataproc-2-3-roc9-${src_timestamp}-rc01" ;; "2.3-ubuntu22" ) image_uri="${img_pfx}/dataproc-2-3-ubu22-${src_timestamp}-rc01" ;; + "2.3-ubuntu22-arm" ) image_uri="${img_pfx}/dataproc-2-3-ubu22-arm-${src_timestamp}-rc01" ;; "2.3-ml-ubuntu22" ) image_uri="${img_pfx}/dataproc-2-3-ml-ubu22-${src_timestamp}-rc01" ;; esac generate --base-image-uri "${image_uri}" @@ -355,7 +356,7 @@ time generate_from_dataproc_version "${dataproc_version}" # Configure a proxy on secure-boot image PURPOSE="secure-proxy" -customization_script="${DATAPROC_EVOLUTION_DIR}/initialization-actions/http-proxy/http-proxy.sh" +customization_script="startup_script/gce-proxy-setup.sh" print_status "=== Generating base ${PURPOSE} image for ${dataproc_version} ===" time generate_from_base_purpose "secure-boot" From d7a7d3ac8a9bfab305109c9ab9d89825ef4f9d05 Mon Sep 17 00:00:00 2001 From: "C.J. Collier" Date: Mon, 15 Jun 2026 21:51:03 +0000 Subject: [PATCH 3/3] Implement VM retention on failure, increase build disk size, and add deferred script execution This commit introduces several key improvements to the custom image builder pipeline to enhance debugging, reliability, and extensibility: 1. **VM Retention on Failure for Debugging**: - Modified `startup_script/run.sh` to use an `EXIT` trap (`shutdown_vm`) instead of an inline shutdown at the end of `main`. - The trap checks the exit status and GCE metadata `retain-on-failure`. If the build fails and `retain-on-failure=true` is set, the VM remains running (skips shutdown) to allow live debugging via SSH. - Added `--retain-on-failure` command-line argument to `generate_custom_image.py` (via `args_parser.py` and `shell_script_generator.py`) and enabled it by default in `examples/secure-boot/pre-init.sh`. 2. **Prevent Storage Exhaustion in Monolithic Builds**: - Increased the build disk size for `2.3-debian12` from 50GB to 100GB in `examples/secure-boot/pre-init.sh` to accommodate the cumulative size of GPU/ML libraries and monolithic optional components (Zeppelin, Jupyter, Docker, Pig, Delta). 3. **Dynamic Deferred Config Script Execution**: - Added support in `startup_script/gce-proxy-setup.sh` to dynamically download and execute a custom user script from a GCS URI (specified via `deferred-config-script-uri` metadata) during the deferred boot phase. - Supports optional integrity verification using a SHA256 checksum (specified via `deferred-config-script-sha256` metadata). 4. **QoL and Portability Fixes**: - Updated ARM instance machine type to `c4a-standard-2` in `examples/secure-boot/pre-init.sh`. - Replaced Linux-specific `stat` command with portable `date -r` in `examples/secure-boot/bin/create-debug-vm.sh` for cache TTL checks. TAG=agy CONV=b274b565-1bd6-43f1-b4db-31f3d89d087b --- examples/secure-boot/bin/create-debug-vm.sh | 2 +- examples/secure-boot/pre-init.sh | 5 +- startup_script/gce-proxy-setup.sh | 65 +++++++++++++++++---- startup_script/run.sh | 24 ++++++-- 4 files changed, 76 insertions(+), 20 deletions(-) diff --git a/examples/secure-boot/bin/create-debug-vm.sh b/examples/secure-boot/bin/create-debug-vm.sh index 4632d53..ebe99b3 100644 --- a/examples/secure-boot/bin/create-debug-vm.sh +++ b/examples/secure-boot/bin/create-debug-vm.sh @@ -61,7 +61,7 @@ fi # Determine Dataproc Image if [[ -n "${DATAPROC_IMAGE:-}" ]]; then echo "Using image from ENV: ${DATAPROC_IMAGE}" -elif [[ -f "${CACHE_FILE}" ]] && [[ $(($(date +%s) - $(stat -c %Y "${CACHE_FILE}"))) -lt "${CACHE_TTL_SECONDS}" ]]; then +elif [[ -f "${CACHE_FILE}" ]] && [[ $(($(date +%s) - $(date -r "${CACHE_FILE}" +%s))) -lt "${CACHE_TTL_SECONDS}" ]]; then DATAPROC_IMAGE=$(cat "${CACHE_FILE}") echo "Using cached image: ${DATAPROC_IMAGE}" else diff --git a/examples/secure-boot/pre-init.sh b/examples/secure-boot/pre-init.sh index ea4c214..086a27f 100644 --- a/examples/secure-boot/pre-init.sh +++ b/examples/secure-boot/pre-init.sh @@ -99,7 +99,7 @@ function create_unaccelerated_instance() { function create_arm_instance() { python3 generate_custom_image.py \ - --machine-type "t2a-standard-2" \ + --machine-type "c4a-standard-2" \ "$@" } @@ -254,6 +254,7 @@ function generate() { --trusted-cert "tls/db.der" \ --shutdown-instance-timer-sec=30 \ --no-smoke-test \ + --retain-on-failure \ ${extra_args} then report_result "Success" @@ -388,7 +389,7 @@ case "${dataproc_version}" in "2.2-rocky9" ) disk_size_gb="60" ;; # 49.79G 43.51G 6.28G 88% / # 20250429-193537-tf "2.2-ubuntu22" ) disk_size_gb="60" ;; # 48.28G 43.32G 4.94G 90% / # 20250429-193537-tf - "2.3-debian12" ) disk_size_gb="50" ;; # 41.11G 36.20G 3.12G 93% / # 20250507-083009-tf + "2.3-debian12" ) disk_size_gb="100" ;; # Increased to 100GB to prevent out-of-space during monolithic optional components accumulation "2.3-rocky9" ) disk_size_gb="50" ;; # 49.79G 37.82G 11.98G 76% / # 20250507-083009-tf "2.3-ubuntu22" ) disk_size_gb="50" ;; # 40.52G 36.18G 4.33G 90% / # 20250507-083009-tf "2.3-ml-ubuntu22" ) disk_size_gb="70" ;; # 40.52G 36.18G 4.33G 90% / # 20250507-083009-tf diff --git a/startup_script/gce-proxy-setup.sh b/startup_script/gce-proxy-setup.sh index 546e9c1..4be84cd 100644 --- a/startup_script/gce-proxy-setup.sh +++ b/startup_script/gce-proxy-setup.sh @@ -593,9 +593,12 @@ function setup_deferred_service() { chmod +x "${target_path}" fi - # 2. Write the systemd unit file (runs BEFORE google-dataproc-agent) - echo "INFO: setup_deferred_service: Writing systemd service file ${service_file}" >&2 - cat < "${service_file}" + # 2. Write and enable the systemd unit file (runs BEFORE google-dataproc-agent) if not already present + if [[ -f "${service_file}" ]]; then + echo "INFO: setup_deferred_service: Systemd service file ${service_file} already exists. Skipping write and enable." >&2 + else + echo "INFO: setup_deferred_service: Writing systemd service file ${service_file}" >&2 + cat < "${service_file}" [Unit] Description=Inject Dynamic Dataproc Proxy Overrides into Systemd Manager DefaultDependencies=no @@ -615,17 +618,18 @@ StandardError=journal+console WantedBy=multi-user.target EOF - chmod 644 "${service_file}" + chmod 644 "${service_file}" - # 3. Enable the service so it runs on every boot - if ! command -v systemctl >/dev/null 2>&1 || [[ "$(get_cached_state 'system/systemd_is_pid1')" != "true" ]]; then - echo "ERROR: setup_deferred_service: systemd is not running as PID 1. Cannot enable deferred service." >&2 - exit 1 - fi + # 3. Enable the service so it runs on every boot + if ! command -v systemctl >/dev/null 2>&1 || [[ "$(get_cached_state 'system/systemd_is_pid1')" != "true" ]]; then + echo "ERROR: setup_deferred_service: systemd is not running as PID 1. Cannot enable deferred service." >&2 + exit 1 + fi - echo "INFO: setup_deferred_service: Enabling systemd service ${service_name}" >&2 - systemctl enable "${service_name}.service" - echo "INFO: setup_deferred_service: Deferred proxy service enabled successfully." >&2 + echo "INFO: setup_deferred_service: Enabling systemd service ${service_name}" >&2 + systemctl enable "${service_name}.service" + echo "INFO: setup_deferred_service: Deferred proxy service enabled successfully." >&2 + fi } function print_introspection_report() { @@ -661,6 +665,43 @@ if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then configure_systemd_proxy repair_boto + # ============================================================================ + # Dynamic Deferred Config Script Execution + # ============================================================================ + DEFERRED_URI=$(get_metadata_attribute 'deferred-config-script-uri' '') + if [[ -n "${DEFERRED_URI}" ]]; then + if [[ "${DEFERRED_URI}" != gs://* ]]; then + echo "ERROR: deferred-config-script-uri must be a GCS URI starting with gs://" + exit 1 + fi + + LOCAL_SCRIPT="/tmp/deferred-config-script.sh" + echo "INFO: Fetching deferred script from ${DEFERRED_URI}..." + if ! gsutil cp "${DEFERRED_URI}" "${LOCAL_SCRIPT}"; then + echo "ERROR: Failed to download deferred config script from ${DEFERRED_URI}" + exit 1 + fi + + DEFERRED_SHA=$(get_metadata_attribute 'deferred-config-script-sha256' '') + if [[ -n "${DEFERRED_SHA}" ]]; then + CALCULATED_SHA=$(sha256sum "${LOCAL_SCRIPT}" | cut -d ' ' -f 1) + if [[ "${CALCULATED_SHA}" != "${DEFERRED_SHA}" ]]; then + echo "ERROR: SHA256 checksum mismatch for deferred config script!" + rm -f "${LOCAL_SCRIPT}" + exit 1 + fi + fi + + chmod +x "${LOCAL_SCRIPT}" + echo "INFO: Executing deferred config script..." + if ! "${LOCAL_SCRIPT}"; then + echo "ERROR: Deferred config script execution failed!" + rm -f "${LOCAL_SCRIPT}" + exit 1 + fi + rm -f "${LOCAL_SCRIPT}" + fi + if [[ "${IS_CUSTOM_IMAGE_BUILD:-}" == "true" || -n "$(get_metadata_attribute 'custom-sources-path' '')" ]]; then setup_deferred_service fi diff --git a/startup_script/run.sh b/startup_script/run.sh index ba490eb..ee892f1 100644 --- a/startup_script/run.sh +++ b/startup_script/run.sh @@ -45,6 +45,25 @@ CUSTOM_SOURCES_PATH=$(/usr/share/google/get_metadata_value attributes/custom-sou # get time to wait for stdout to flush SHUTDOWN_TIMER_IN_SEC=$(/usr/share/google/get_metadata_value attributes/shutdown-timer-in-sec) +function shutdown_vm() { + local exit_code=$? + if [[ ${exit_code} -ne 0 ]]; then + local retain_on_failure + retain_on_failure=$(/usr/share/google/get_metadata_value attributes/retain-on-failure || echo "false") + if [[ "${retain_on_failure}" == "true" ]]; then + echo "startup-script: Build failed with exit code ${exit_code}." + echo "startup-script: retain-on-failure=true is set. Keeping VM running for debugging." + return 0 + fi + fi + + echo "startup-script: Sleep ${SHUTDOWN_TIMER_IN_SEC}s before shutting down..." + echo "You can change the timeout value with --shutdown-instance-timer-sec" + sleep "${SHUTDOWN_TIMER_IN_SEC}" # wait for stdout to flush + shutdown -h now +} +trap shutdown_vm EXIT + USER_DATAPROC_COMPONENTS=$( /usr/share/google/get_metadata_value attributes/optional-components | tr '[:upper:]' '[:lower:]' | tr '.' ' ' || echo "") DATAPROC_IMAGE_VERSION=$(/usr/share/google/get_metadata_value attributes/dataproc_dataproc_version | cut -c1-3 | tr '-' '.' || echo "") DATAPROC_IMAGE_TYPE=$(/usr/share/google/get_metadata_value attributes/dataproc_image_type || echo "standard") @@ -281,11 +300,6 @@ function main() { echo "startup-script: BuildSucceeded: Customization complete." fi fi - - echo "startup-script: Sleep ${SHUTDOWN_TIMER_IN_SEC}s before shutting down..." - echo "You can change the timeout value with --shutdown-instance-timer-sec" - sleep "${SHUTDOWN_TIMER_IN_SEC}" # wait for stdout to flush - shutdown -h now } main "$@"