diff --git a/.gitignore b/.gitignore index 79e0744..b651c2a 100644 --- a/.gitignore +++ b/.gitignore @@ -29,3 +29,15 @@ env.json # failed patches *.orig *.rej + +# --- Generic ignores for credentials and logs --- +key*.json +*env*.json +env-*.json +*.log +*.bz2 +*.zst + +# Temporary directories +temp/ +tmp/ diff --git a/examples/secure-boot/README.md b/examples/secure-boot/README.md index 930c778..65fe800 100644 --- a/examples/secure-boot/README.md +++ b/examples/secure-boot/README.md @@ -231,6 +231,52 @@ To run Dataproc clusters with NVIDIA GPUs and Shielded VM Secure Boot enabled: 3. **Verify module signature:** `sudo modinfo nvidia | grep signer` (Expected: `Cloud Dataproc Custom Image CA`) 4. **Check dmesg:** `dmesg | grep -iE "Secure Boot|NVRM|nvidia"` +## Manual Customization & Interactive Debugging + +For complex troubleshooting, hot-patching, or script development, developers can bypass the automated containerized pipeline and run an **idempotent, interactive debugging loop** from their workstation. + +This workflow is orchestrated by a single, powerful workstation-side script: **`customize-in-screen.sh`**. + +### 1. Configure the Target +In `custom-images/env.json`, configure the target Dataproc version and the script you wish to test/debug: +```json +{ + "IMAGE_VERSION": "2.1-debian11", + "CUSTOMIZATION_SCRIPT": "examples/secure-boot/no-customization.sh" +} +``` + +### 2. Run the Idempotent Customizer +From the `custom-images` directory on your workstation, execute the orchestrator: +```bash +bash examples/secure-boot/bin/customize-in-screen.sh +``` + +**How it behaves (Idempotency in Action)**: +* **First-Time Run (Cold Start)**: If the debug VM does not exist, it automatically calls `create-debug-vm.sh` to provision a raw, persistent VM (configured with a 24-hour shutdown timer and no automated startup script). It syncs your local code to GCS, triggers a remote background launch of `install-in-screen.sh`, and instantly attaches your terminal to the live screen session. +* **Subsequent Runs (Warm Start / Re-use)**: If the VM is already online, **it bypasses GCE provisioning entirely!** It instantly uploads your latest local edits to GCS, SSHes into the VM, downloads the new scripts, restarts the customization inside a detached `screen` session, and attaches your terminal. **Time to execution is under 5 seconds.** +* **Re-attaching to a Live Build**: If you run the script while a customization build is *already active* on the VM, it detects the running session, bypasses launching, and **instantly attaches your terminal to the live build.** + +### 3. Interactive Attachment & Control +Once attached, you are inside a live, interactive `screen` session on the VM: +* **Real-Time Debugging**: You can watch the compilation, press `Ctrl+C` to halt, edit files locally in `/tmp/sources/` on the VM, and manually re-run steps to test fixes. +* **Safe Detachment**: To detach from the screen session and leave it running in the background on the VM (allowing you to close your laptop or disconnect), press: + `Ctrl+A` followed by `D`. +* **Re-attaching**: To re-attach later, simply run `bash examples/secure-boot/bin/customize-in-screen.sh` again from your workstation. + +### 4. Run Workstation-Side Diagnostics +While the customization is running (or after a failure), you can audit the VM's active network and proxy state with a single command from your workstation terminal: +```bash +bash examples/secure-boot/audit-image-customizer.sh +``` +This remote prober connects via IAP SSH in non-interactive batch mode and prints a pretty-printed JSON **System Audit Report** showing GCS and external network connectivity (verifying if Private Google Access and the SWP proxy are routing correctly). + +### 5. Cleanup +Once debugging is complete, delete the GCE VM and clean up the GCS staging assets: +```bash +bash examples/secure-boot/bin/destroy-debug-vm.sh +``` + ## Key Scripts Involved * `custom-images/env.json`: Single source of truth for configuration. diff --git a/examples/secure-boot/TESTING.md b/examples/secure-boot/TESTING.md index 2d4d424..a92b7ba 100644 --- a/examples/secure-boot/TESTING.md +++ b/examples/secure-boot/TESTING.md @@ -50,15 +50,26 @@ dmesg | grep -iE "Secure Boot|NVRM|nvidia" --- -## Measured Custom Image Build Timing Reference +## Boot and Build Durations -The following table lists the empirical, real-world durations of the various image building and compilation phases observed during sequential and parallel builds inside a standard `us-east4` project: +Comparison of cluster boot times and image creation times. -| Image Build Phase | Customization Script / Action | Typical Duration | Performance & Cache Notes | +### Cluster Boot Times (VM Boot to Dataproc READY) + +| Method | Mechanism | Boot Time | Details | +| :--- | :--- | :--- | :--- | +| **Standard Image + Init Action** | `install_gpu_driver.sh` (as Init Action) | **`7m` - `9m`** | Downloads ~4.5 GB of drivers and packages from GCS, compiles kernel modules, and configures YARN/Spark on every boot. | +| **Pre-baked Custom Image** | Pre-installed drivers + deferred systemd config | **`~4m`** | No downloads or installations. Adds ~30s to the first boot for hardware probing and writing configuration files. | + +--- + +### Image Creation Times (Baking) + +Image creation times using the Podman pipeline in `us-east4`: + +| Phase | Script | Duration | Notes | | :--- | :--- | :--- | :--- | -| **GCE Base Secure Boot Image** | `examples/secure-boot/no-customization.sh` | `~7m 20s` - `11m 05s` | Boots the unaccelerated VM instance, registers UEFI db public certs, and snapshots the `secure-boot` GCE image. Rocky Linux builds take ~11m, Debian/Ubuntu take ~7m. | -| **Total Baseline Custom Image Build** | `examples/secure-boot/build-and-run-podman.sh` | `~7m` - `11m` | Total OCI/Podman Stage 1 parallel compilation time to generate the UEFI baseline custom images. | -| **GPU/Conda Pre-bake Build (Cold Cache)** | `initialization-actions/gpu/install_gpu_driver.sh` | `~21m` - `24m` | Boots a T4 GPU VM instance, compiles the NVIDIA modules, compiles NCCL, and builds TensorFlow, PyTorch, and RAPIDS Conda environments via Mamba. | -| **GPU/Conda Pre-bake Build (GCS Cache Hit)** | `initialization-actions/gpu/install_gpu_driver.sh` | **`1m 45s`** | Downloads pre-compiled Blackwell drivers and zipped Conda tarballs directly from GCS over Private Google Access routes. | -| **Total Production custom image Build** | `examples/secure-boot/build-and-run-podman.sh` | `~25m` - `35m` | Total end-to-end parallel OCI build duration to generate the final, fully pre-baked production custom images (`-tf`). | +| **GCE Base Secure Boot Image** | `pre-init.sh` (Base Stage) | `~7m` - `11m` | Provisions VM, registers UEFI certs, and snapshots base image. (Rocky: ~11m, Debian/Ubuntu: ~7m). | +| **GPU/Conda Pre-bake Layer** | `install_gpu_driver.sh` (during baking) | `~21m` - `24m` | Compiles NVIDIA modules and builds Conda environments on a GPU VM. | +| **Total Image Suite** | `build-and-run-podman.sh` | `~25m` - `35m` | Total duration to generate the image suite. | diff --git a/examples/secure-boot/audit-image-customizer.sh b/examples/secure-boot/audit-image-customizer.sh new file mode 100644 index 0000000..7afe4c6 --- /dev/null +++ b/examples/secure-boot/audit-image-customizer.sh @@ -0,0 +1,232 @@ +#!/bin/bash +# +# Dual-mode audit and diagnostic tool for GCE image customization. +# +# Workstation Mode: Copies itself to the active GCE builder VM, executes +# remotely, and prints the captured JSON state report. +# Guest Mode: Runs locally on the GCE VM, performs parallel probes, +# and outputs a structured JSON report to stdout. + +set -euo pipefail + +# --- Environment Detection --- +SCRIPT_DIR=$(dirname "$(readlink -f "$0")") +ENV_JSON="${SCRIPT_DIR}/../../env.json" + +# If env.json exists two levels up from the script, we are on the Workstation. +# On the VM, the script is copied to /tmp, so this file will not exist. +if [[ -f "${ENV_JSON}" ]]; then + ROLE="workstation" +else + ROLE="guest" +fi + +# ========================================== +# GUEST MODE: Run Parallel Probes on the VM +# ========================================== +if [[ "${ROLE}" == "guest" ]]; then + MDS_PREFIX="http://metadata.google.internal/computeMetadata/v1" + AUDIT_TEMP_DIR=$(mktemp -d) + PIDS=() + + # Helper to run a probe in the background and save JSON fragment + run_probe() { + local -r key="$1" + shift + ( + "$@" > "${AUDIT_TEMP_DIR}/${key}.json" 2> "${AUDIT_TEMP_DIR}/${key}.err" + ) & + PIDS+=( $! ) + } + + # Probe 1: GCE Metadata Attributes + probe_metadata() { + local -r attributes=( + "http-proxy" + "https-proxy" + "proxy-uri" + "no-proxy" + "dataproc-cluster-name" + "http-proxy-pem-uri" + "universe-domain" + "custom-sources-path" + "project-id" + ) + echo -n "{" + local first=true + for attr in "${attributes[@]}"; do + local val + val=$(curl -s -f -H "Metadata-Flavor: Google" "${MDS_PREFIX}/instance/attributes/${attr}" || echo "") + if [[ -z "${val}" ]]; then + val=$(curl -s -f -H "Metadata-Flavor: Google" "${MDS_PREFIX}/project/attributes/${attr}" || echo "") + fi + + if [[ -n "${val}" ]]; then + if [[ "${first}" == "true" ]]; then first=false; else echo -n ","; fi + # Escape newlines and quotes in the value for safe JSON + local escaped_val + escaped_val=$(echo -n "${val}" | jq -R .) + echo -n "\"${attr}\": ${escaped_val}" + fi + done + echo -n "}" + } + run_probe "metadata" probe_metadata + + # Probe 2: Network Connectivity (Direct vs Proxy) + probe_network() { + local http_proxy_val + http_proxy_val=$(curl -s -f -H "Metadata-Flavor: Google" "${MDS_PREFIX}/instance/attributes/http-proxy" || echo "") + if [[ -z "${http_proxy_val}" ]]; then + http_proxy_val=$(curl -s -f -H "Metadata-Flavor: Google" "${MDS_PREFIX}/instance/attributes/proxy-uri" || echo "") + fi + + local direct_gcs="fail" + local proxy_gcs="fail" + local direct_ext="fail" + local proxy_ext="fail" + + # Test direct GCS (should succeed if PGA is active, even without proxy/NAT) + if curl -s -f -o /dev/null --connect-timeout 3 "https://storage.googleapis.com" &>/dev/null; then + direct_gcs="success" + fi + + # Test direct external (should fail in isolated network) + if curl -s -f -o /dev/null --connect-timeout 3 "https://www.google.com" &>/dev/null; then + direct_ext="success" + fi + + # Test via proxy (if proxy metadata exists) + if [[ -n "${http_proxy_val}" ]]; then + if curl -s -f -x "http://${http_proxy_val}" -o /dev/null --connect-timeout 3 "https://storage.googleapis.com" &>/dev/null; then + proxy_gcs="success" + fi + if curl -s -f -x "http://${http_proxy_val}" -o /dev/null --connect-timeout 3 "https://www.google.com" &>/dev/null; then + proxy_ext="success" + fi + fi + + cat <&2 + exit 1 + fi + + # 1. Parse GCE Project and Zone from env.json + PROJECT_ID=$(jq -r '.project_id // .PROJECT_ID // empty' "${ENV_JSON}") + ZONE=$(jq -r '.zone // .ZONE // empty' "${ENV_JSON}") + + if [[ -z "${PROJECT_ID}" || -z "${ZONE}" ]]; then + echo "ERROR: project_id or zone not defined in env.json." >&2 + exit 1 + fi + + echo "DEBUG: Workstation Mode - Querying GCE for active customization instance..." >&2 + + # 2. Dynamically discover the active builder VM name + # The builder VM name matches the pattern: dataproc-[version]-[timestamp]-install + VM_NAME=$(gcloud compute instances list --project="${PROJECT_ID}" \ + --filter="name:dataproc-*-install AND zone:(${ZONE})" \ + --format="value(name)" | head -n 1) + + if [[ -z "${VM_NAME}" ]]; then + echo "ERROR: No active GCE customization instance found in project ${PROJECT_ID} (zone ${ZONE})." >&2 + exit 1 + fi + + echo "DEBUG: Found active builder VM: ${VM_NAME}" >&2 + echo "DEBUG: Copying audit script to VM..." >&2 + + # 3. SCP this script to the VM (using non-interactive batch mode) + gcloud compute scp "${SCRIPT_DIR}/audit-image-customizer.sh" "${VM_NAME}:/tmp/audit-image-customizer.sh" \ + --project="${PROJECT_ID}" --zone="${ZONE}" --tunnel-through-iap --quiet \ + --ssh-flag="-o BatchMode=yes" --ssh-flag="-o ConnectTimeout=5" \ + --ssh-flag="-o StrictHostKeyChecking=no" --ssh-flag="-o UserKnownHostsFile=/dev/null" &>/dev/null + + echo "DEBUG: Executing audit script remotely on VM..." >&2 + + # 4. SSH into the VM, run the script, and capture the JSON stdout + set +e + JSON_REPORT=$(gcloud compute ssh "${VM_NAME}" \ + --project="${PROJECT_ID}" --zone="${ZONE}" --tunnel-through-iap --quiet \ + --ssh-flag="-o BatchMode=yes" --ssh-flag="-o ConnectTimeout=5" \ + --ssh-flag="-o StrictHostKeyChecking=no" --ssh-flag="-o UserKnownHostsFile=/dev/null" \ + --command="bash /tmp/audit-image-customizer.sh" 2>/dev/null) + RETVAL=$? + set -e + + # Clean up the script on the VM + gcloud compute ssh "${VM_NAME}" \ + --project="${PROJECT_ID}" --zone="${ZONE}" --tunnel-through-iap --quiet \ + --ssh-flag="-o BatchMode=yes" --ssh-flag="-o ConnectTimeout=5" \ + --ssh-flag="-o StrictHostKeyChecking=no" --ssh-flag="-o UserKnownHostsFile=/dev/null" \ + --command="rm -f /tmp/audit-image-customizer.sh" &>/dev/null || true + + if [[ ${RETVAL} -ne 0 || -z "${JSON_REPORT}" ]]; then + echo "ERROR: Failed to execute remote audit on VM." >&2 + exit 1 + fi + + # 5. Output the pretty-printed JSON report to the developer + echo "${JSON_REPORT}" | jq . + exit 0 +fi diff --git a/examples/secure-boot/bin/cleanup-builders.sh b/examples/secure-boot/bin/cleanup-builders.sh new file mode 100755 index 0000000..a6d8a6b --- /dev/null +++ b/examples/secure-boot/bin/cleanup-builders.sh @@ -0,0 +1,93 @@ +#!/bin/bash +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS-IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +# +# This script cleans up any lingering builder VMs using screen. + +set -e + +DEBUG="${DEBUG:-0}" +if (( DEBUG != 0 )); then + set -x +fi + +source examples/secure-boot/lib/env.sh +source examples/secure-boot/lib/util.sh + +print_status "Searching for lingering builder VMs in project ${PROJECT_ID}, zone ${ZONE}..." + +INSTANCE_LIST=$(gcloud compute instances list \ + --project="${PROJECT_ID}" \ + --zones="${ZONE}" \ + --filter="name ~ -install$" \ + --format="value(name)") + +if [[ -z "${INSTANCE_LIST}" ]]; then + report_result "None Found" + echo "No lingering builder VMs found to clean up." >&2 + exit 0 +fi + +report_result "Found" +echo "The following builder VMs will be deleted in detached screen sessions:" >&2 +echo "${INSTANCE_LIST}" >&2 + +read -p "Continue with deletion? (y/N): " confirm +if [[ "${confirm}" != [yY] ]]; then + echo "Deletion cancelled." >&2 + exit 0 +fi + +TEMP_SCREENRC="${tmpdir}/temp_cleanup.screenrc" +rm -f "${TEMP_SCREENRC}" +touch "${TEMP_SCREENRC}" + +echo "The following builder VMs will be deleted in a new screen session:" >&2 +echo "${INSTANCE_LIST}" >&2 + +read -p "Continue with deletion? (y/N): " confirm +if [[ "${confirm}" != [yY] ]]; then + echo "Deletion cancelled." >&2 + exit 0 +fi + +i=1 +for instance in ${INSTANCE_LIST}; do + echo "Queueing deletion for ${instance} in screen window ${i}" >&2 + SCREEN_CMD=" +attempt=0 +while true; do + gcloud beta compute instances delete ${instance} --zone=${ZONE} --project=${PROJECT_ID} --quiet --no-graceful-shutdown && break + attempt=$((attempt + 1)) + if [[ ${attempt} -ge 3 ]]; then + echo 'Max retries reached for ${instance}' + break + fi + echo 'Failed, retrying in 5 seconds...' + read -t 5 +done +echo \"Delete command for ${instance} finished with exit code $?\" +echo \"Window will close in 5 seconds...\" +read -t 5 +" + echo "screen -t delete-${i}-${instance} ${i} bash -c '${SCREEN_CMD}'" >> "${TEMP_SCREENRC}" + i=$((i + 1)) + sleep 0.1 # Small delay +done + + +echo "Launching screen session from ${TEMP_SCREENRC}..." >&2 +screen -c "${TEMP_SCREENRC}" + +rm -f "${TEMP_SCREENRC}" +echo "Screen session exited." >&2 diff --git a/examples/secure-boot/bin/create-debug-vm.sh b/examples/secure-boot/bin/create-debug-vm.sh new file mode 100644 index 0000000..ebe99b3 --- /dev/null +++ b/examples/secure-boot/bin/create-debug-vm.sh @@ -0,0 +1,177 @@ +#!/bin/bash + +set -e + +DEBUG="${DEBUG:-0}" +if (( DEBUG != 0 )); then + set -x +fi + +# Source environment variables +if [[ -f "$(dirname "$0")/../lib/env.sh" ]]; then + source "$(dirname "$0")/../lib/env.sh" +else + echo "ERROR: examples/secure-boot/lib/env.sh not found." + exit 1 +fi +if [[ -f "$(dirname "$0")/../lib/util.sh" ]]; then + source "$(dirname "$0")/../lib/util.sh" +else + echo "ERROR: examples/secure-boot/lib/util.sh not found." + exit 1 +fi + +FORCE_DELETE=0 +USE_PROXY=0 +USAGE="Usage: bash $(basename "$0") [-f] [-p]" + +# Parse options +while getopts "fp" opt; do + case ${opt} in + f ) FORCE_DELETE=1 ;; + p ) USE_PROXY=1 ;; + \? ) echo "${USAGE}" >&2; exit 1 ;; + esac +done +shift $((OPTIND -1)) + +if [[ -z "${IMAGE_VERSION}" ]]; then + echo "ERROR: IMAGE_VERSION is not set in env.json." + exit 1 +fi +if [[ -z "${CUSTOMIZATION_SCRIPT}" ]]; then + echo "ERROR: CUSTOMIZATION_SCRIPT is not set in env.json." + exit 1 +fi + +INSTANCE_NAME="debug-$(echo "${IMAGE_VERSION}" | tr '.' '-')-$(basename "${CUSTOMIZATION_SCRIPT}" .sh | tr '.' '-' | tr '_' '-')" + +# Override PURPOSE to a fixed value for the service account name +SA_NAME="sa-tf-pre-init" +GSA="${SA_NAME}@${PROJECT_ID}.iam.gserviceaccount.com" + +GCS_SOURCES_PATH="gs://${BUCKET}/${INSTANCE_NAME}/sources" +CACHE_FILE="/tmp/latest_dataproc_$(echo "${IMAGE_VERSION}" | tr './-' '_').txt" +CACHE_TTL_SECONDS=$((24 * 60 * 60)) # Cache for 1 day + +if [[ "${FORCE_DELETE}" -eq 1 ]]; then + rm -f "${CACHE_FILE}" +fi + +# Determine Dataproc Image +if [[ -n "${DATAPROC_IMAGE:-}" ]]; then + echo "Using image from ENV: ${DATAPROC_IMAGE}" +elif [[ -f "${CACHE_FILE}" ]] && [[ $(($(date +%s) - $(date -r "${CACHE_FILE}" +%s))) -lt "${CACHE_TTL_SECONDS}" ]]; then + DATAPROC_IMAGE=$(cat "${CACHE_FILE}") + echo "Using cached image: ${DATAPROC_IMAGE}" +else + echo "DATAPROC_IMAGE not set or cache stale, querying for the latest ${IMAGE_VERSION} image..." + IMAGE_PREFIX="dataproc-$(echo "${IMAGE_VERSION}" | sed -e 's/\./-/g' -e 's/-debian12/-deb12/g' -e 's/-debian11/-deb11/g' -e 's/-ubuntu22/-ubu22/g' -e 's/-rocky9/-roc9/g')" + DATAPROC_IMAGE=$(gcloud compute images list --project cloud-dataproc \ + --filter="name:${IMAGE_PREFIX} AND status=READY" \ + --format="value(name)" | \ + grep -v "eap" | \ + sort -r | \ + head -n 1) + if [[ -z "${DATAPROC_IMAGE}" ]]; then + echo "ERROR: Could not find a suitable ${IMAGE_VERSION} image." + exit 1 + fi + echo "${DATAPROC_IMAGE}" > "${CACHE_FILE}" + echo "Using new image: ${DATAPROC_IMAGE} (cached to ${CACHE_FILE})" +fi + +# Check instance existence and handle --force +if gcloud compute instances describe "${INSTANCE_NAME}" --zone "${ZONE}" --project "${PROJECT_ID}" > /dev/null 2>&1; then + echo "Instance ${INSTANCE_NAME} already exists." + if [[ "${FORCE_DELETE}" -eq 1 ]]; then + echo "Deleting existing instance due to --force flag..." + run_gcloud delete_instance gcloud compute instances delete "${INSTANCE_NAME}" --zone "${ZONE}" --project "${PROJECT_ID}" -q + echo "Instance deleted." + else + echo "ERROR: Instance exists. Use -f to delete and recreate." + exit 1 + fi +else + echo "Instance ${INSTANCE_NAME} not found, proceeding." +fi + +# Clean up GCS path +run_gsutil rm_gcs gsutil -m rm -r "${GCS_SOURCES_PATH}" || echo "GCS path not found, continuing..." + +# Upload scripts from local repo to GCS +REPO_ROOT="$(git rev-parse --show-toplevel)" +run_gsutil cp_run gsutil cp "${REPO_ROOT}/startup_script/run.sh" "${GCS_SOURCES_PATH}/run.sh" +run_gsutil cp_init gsutil cp "${REPO_ROOT}/${CUSTOMIZATION_SCRIPT}" "${GCS_SOURCES_PATH}/init_actions.sh" +run_gsutil cp_env gsutil cp "${REPO_ROOT}/examples/secure-boot/lib/env.sh" "${GCS_SOURCES_PATH}/env.sh" +run_gsutil cp_util gsutil cp "${REPO_ROOT}/examples/secure-boot/lib/util.sh" "${GCS_SOURCES_PATH}/util.sh" +run_gsutil cp_proxy gsutil cp "${REPO_ROOT}/startup_script/gce-proxy-setup.sh" "${GCS_SOURCES_PATH}/gce-proxy-setup.sh" + +if [[ "${USE_PROXY}" -eq 1 ]]; then + # Resolve path to cloud-dataproc/gcloud/env.json relative to the authoritative repository root + # already resolved and exported by env.sh. + GCLOUD_ENV_JSON="${DATAPROC_EVOLUTION_DIR}/cloud-dataproc/gcloud/env.json" + if [[ ! -f "${GCLOUD_ENV_JSON}" ]]; then + echo "ERROR: cloud-dataproc/gcloud/env.json not found. Cannot resolve SWP proxy configurations." >&2 + exit 1 + fi + SWP_IP=$(jq -r .SWP_IP "${GCLOUD_ENV_JSON}") + SWP_PORT=$(jq -r .SWP_PORT "${GCLOUD_ENV_JSON}") + if [[ -z "${SWP_IP}" || "${SWP_IP}" == "null" || -z "${SWP_PORT}" || "${SWP_PORT}" == "null" ]]; then + echo "ERROR: SWP_IP or SWP_PORT is not configured in cloud-dataproc/gcloud/env.json." >&2 + exit 1 + fi + echo "INFO: Enabling SWP Proxy Egress: ${SWP_IP}:${SWP_PORT}" >&2 +fi + +declare -a METADATA_ARRAY=( + "VmDnsSetting=ZonalOnly" + "shutdown-timer-in-sec=86400" # 1 day timer for debugging + "custom-sources-path=${GCS_SOURCES_PATH}" + "dataproc-region=${region}" + "dataproc_dataproc_version=${IMAGE_VERSION}" + "invocation-type=custom-images" + "dataproc-temp-bucket=${TEMP_BUCKET}" +) + +if [[ "${USE_PROXY}" -eq 1 ]]; then + METADATA_ARRAY+=( + "http-proxy=${SWP_IP}:${SWP_PORT}" + "https-proxy=${SWP_IP}:${SWP_PORT}" + "proxy-uri=${SWP_IP}:${SWP_PORT}" + "no-proxy=metadata.google.internal,${PROJECT_ID}.svc.id.goog" + ) +fi + +# Use a custom delimiter (^;^) to allow commas in metadata values (like no-proxy) +# This prevents gcloud from splitting on commas inside the no-proxy value. +METADATA_STRING="^;^$(IFS=';'; echo "${METADATA_ARRAY[*]}")" + +# Create the instance +declare -a gcloud_create_args=( + gcloud compute instances create "${INSTANCE_NAME}" + --project "${PROJECT_ID}" + --zone "${ZONE}" + --machine-type n1-standard-2 + --image "${DATAPROC_IMAGE}" + --image-project cloud-dataproc + --boot-disk-size 30G + --boot-disk-type pd-ssd + --scopes "https://www.googleapis.com/auth/cloud-platform" + --service-account "${GSA}" + --subnet "${SUBNET}" + --metadata="${METADATA_STRING}" +) +run_gcloud create_instance "${gcloud_create_args[@]}" + +echo "Instance ${INSTANCE_NAME} created." + +SERIAL_LOG_FILE="${REPRO_TMPDIR}/serial_${INSTANCE_NAME}.log" +echo "Tailing serial port output to ${SERIAL_LOG_FILE} in the background..." +gcloud compute instances tail-serial-port-output "${INSTANCE_NAME}" --zone "${ZONE}" --project "${PROJECT_ID}" > "${SERIAL_LOG_FILE}" 2>&1 & +TAIL_PID=$! +echo "Tail PID: ${TAIL_PID}" +echo "To stop tailing: kill ${TAIL_PID}" + +echo "To SSH into the instance:" +echo "bash $(dirname "$0")/ssh-debug-vm.sh" diff --git a/examples/secure-boot/bin/customize-in-screen.sh b/examples/secure-boot/bin/customize-in-screen.sh new file mode 100644 index 0000000..ccdd845 --- /dev/null +++ b/examples/secure-boot/bin/customize-in-screen.sh @@ -0,0 +1,349 @@ +#!/bin/bash +# +# Workstation-side orchestrator for idempotent, interactive image customization. +# +# Workflow: +# 1. Detects or provisions the GCE debug VM. +# 2. Checks if a customization screen session is already running on the VM. +# 3. If running: Instantly attaches the workstation terminal to the active session. +# 4. If not running: Syncs the latest local code to GCS, triggers a remote +# background launch, and immediately attaches to the live session. + +set -euo pipefail + +DEBUG="${DEBUG:-0}" +if (( DEBUG != 0 )); then + set -x +fi +# --- Parse Options --- +CLEAN_BUILD=0 +USE_PROXY=0 +FORCE_APPLY=0 +FORCE_DELETE=0 +USAGE="Usage: bash \$(basename "\$0") [-c] [-p] [-r]" + +while getopts "cpr" opt; do + case ${opt} in + c ) + CLEAN_BUILD=1 + FORCE_APPLY=1 + ;; + p ) USE_PROXY=1 ;; + r ) FORCE_DELETE=1 ;; + \? ) echo "${USAGE}" >&2; exit 1 ;; + esac +done +shift $((OPTIND -1)) +# --- Source Environment & Helpers --- +BIN_DIR=$(dirname "$(readlink -f "$0")") +ENV_SH="${BIN_DIR}/../lib/env.sh" +UTIL_SH="${BIN_DIR}/../lib/util.sh" + +if [[ -f "${ENV_SH}" && -f "${UTIL_SH}" ]]; then + source "${ENV_SH}" + source "${UTIL_SH}" +else + echo "ERROR: Helper libraries not found in examples/secure-boot/lib/." >&2 + exit 1 +fi + +if [[ -z "${IMAGE_VERSION}" || -z "${CUSTOMIZATION_SCRIPT}" ]]; then + echo "ERROR: IMAGE_VERSION or CUSTOMIZATION_SCRIPT not set in env.json." >&2 + exit 1 +fi + +# Generate the unique instance name based on configuration +INSTANCE_NAME="debug-$(echo "${IMAGE_VERSION}" | tr '.' '-')-$(basename "${CUSTOMIZATION_SCRIPT}" .sh | tr '.' '-' | tr '_' '-')" +GCS_SOURCES_PATH="gs://${BUCKET}/${INSTANCE_NAME}/sources" + +echo "DEBUG: Target Instance: ${INSTANCE_NAME}" >&2 +echo "DEBUG: GCS Staging Path: ${GCS_SOURCES_PATH}" >&2 + +# ======================================================================== +# Step 1: Ensure GCE Instance is Online +# ======================================================================== +VM_WAS_CREATED="false" +if ! gcloud compute instances describe "${INSTANCE_NAME}" --zone "${ZONE}" --project "${PROJECT_ID}" &>/dev/null || [[ "${FORCE_DELETE}" -eq 1 ]]; then + if [[ "${FORCE_DELETE}" -eq 1 ]]; then + echo "INFO: Recreation forced (-r). Re-provisioning debug VM ${INSTANCE_NAME}..." >&2 + else + echo "INFO: Debug VM ${INSTANCE_NAME} does not exist. Provisioning a new instance..." >&2 + fi + # Call the pre-existing provisioner (cold start) + declare -a create_args=() + if [[ "${FORCE_DELETE}" -eq 1 ]]; then + create_args+=("-f") + fi + if [[ "${USE_PROXY}" -eq 1 ]]; then + create_args+=("-p") + fi + bash "${BIN_DIR}/create-debug-vm.sh" "${create_args[@]}" + VM_WAS_CREATED="true" +else + echo "INFO: Active debug VM ${INSTANCE_NAME} detected. Re-using instance." >&2 +fi + +if [[ "${VM_WAS_CREATED}" == "true" ]]; then + echo "INFO: Waiting for GCE Identity-Aware Proxy (IAP) tunnel to sync..." >&2 + set +e + IAP_READY=1 + for i in {1..36}; do + if ssh -o ControlMaster=no -o ConnectTimeout=3 -o BatchMode=yes "${INSTANCE_NAME}" "uptime" &>/dev/null; then + IAP_READY=0 + break + fi + echo "DEBUG: Waiting for IAP tunnel (+5s)..." >&2 + sleep 5 + done + set -e + + if [[ ${IAP_READY} -ne 0 ]]; then + echo "ERROR: Timeout waiting for IAP tunnel to sync. VM is online but unreachable." >&2 + exit 1 + fi + echo "INFO: IAP tunnel is active and accepting connections." >&2 +fi + +# ======================================================================== +# Step 1.1: Dynamically Inject SWP Proxy Metadata (Optional Warm Start) +# ======================================================================== +if [[ "${USE_PROXY}" -eq 1 ]]; then + # Resolve path to cloud-dataproc/gcloud/env.json relative to the authoritative repository root + # already resolved and exported by env.sh. + GCLOUD_ENV_JSON="${DATAPROC_EVOLUTION_DIR}/cloud-dataproc/gcloud/env.json" + if [[ -f "${GCLOUD_ENV_JSON}" ]]; then + SWP_IP=$(jq -r .SWP_IP "${GCLOUD_ENV_JSON}") + SWP_PORT=$(jq -r .SWP_PORT "${GCLOUD_ENV_JSON}") + if [[ -n "${SWP_IP}" && "${SWP_IP}" != "null" && -n "${SWP_PORT}" && "${SWP_PORT}" != "null" ]]; then + echo "INFO: Dynamically ensuring SWP Proxy metadata is set on the running VM..." >&2 + gcloud compute instances add-metadata "${INSTANCE_NAME}" \ + --project="${PROJECT_ID}" \ + --zone="${ZONE}" \ + --metadata="^;^http-proxy=${SWP_IP}:${SWP_PORT};https-proxy=${SWP_IP}:${SWP_PORT};proxy-uri=${SWP_IP}:${SWP_PORT};no-proxy=metadata.google.internal,${PROJECT_ID}.svc.id.goog" \ + --quiet + else + echo "ERROR: SWP_IP or SWP_PORT not found in cloud-dataproc/gcloud/env.json." >&2 + exit 1 + fi + else + echo "ERROR: cloud-dataproc/gcloud/env.json not found. Cannot resolve SWP proxy configurations." >&2 + exit 1 + fi +fi + +# ======================================================================== +# Step 1.2: Clean Build Artifacts & Sentinels (Optional) +# ======================================================================== +if [[ "${CLEAN_BUILD}" -eq 1 ]]; then + echo "INFO: Cleaning up all past build artifacts, active screen sessions, and sentinels on the VM..." >&2 + ssh -o ControlMaster=no -o BatchMode=yes -o ConnectTimeout=5 "${INSTANCE_NAME}" " + screen -S customization -X quit 2>/dev/null || true + sudo rm -rf /tmp/dataproc-repro + " &>/dev/null || true +fi + +# ======================================================================== +# Step 2: Check if customization is already running in screen +# ======================================================================== +echo "DEBUG: Checking VM for active customization screen session..." >&2 +set +e +ssh -o ControlMaster=no -o BatchMode=yes -o ConnectTimeout=5 "${INSTANCE_NAME}" "screen -ls | grep -q customization" &>/dev/null +SCREEN_STATUS=$? +set -e + +# ======================================================================== +# Step 3: Branching Idempotent Execution +# ======================================================================== +if [[ ${SCREEN_STATUS} -eq 0 ]]; then + # ---------------------------------------------------------------------- + # CASE A: Customization is already running. Attach immediately. + # ---------------------------------------------------------------------- + cat <&2 +======================================================================== + ATTACHING TO ACTIVE CUSTOMIZATION SESSION +======================================================================== +An active customization build was detected running on the VM. +Connecting your terminal to the live screen session... + +(To detach from screen and leave it running in the background, press: + Ctrl+A followed by D) +======================================================================== +EOF + sleep 1 + + # Connect and attach + ssh -t "${INSTANCE_NAME}" screen -rxU customization + +else + # ---------------------------------------------------------------------- + # CASE B: Customization is not running. Sync, launch, and attach. + # ---------------------------------------------------------------------- + echo "INFO: Customization is not active on the VM. Syncing latest code..." >&2 + + # 1. Sync local workstation scripts to GCS staging using a single bulk upload + echo "DEBUG: Syncing local assets to GCS: ${GCS_SOURCES_PATH}" >&2 + + # Create a local staging directory in the temp folder for fast preparation + LOCAL_STAGING="${REPRO_TMPDIR}/staging" + mkdir -p "${LOCAL_STAGING}" + + # Cleanly resolve customization script path + custom_script_path="${DATAPROC_EVOLUTION_DIR}/custom-images/${CUSTOMIZATION_SCRIPT}" + + # Stage all files locally (near-instantaneous) + cp "${DATAPROC_EVOLUTION_DIR}/custom-images/startup_script/run.sh" "${LOCAL_STAGING}/run.sh" + cp "${custom_script_path}" "${LOCAL_STAGING}/init_actions.sh" + cp "${DATAPROC_EVOLUTION_DIR}/custom-images/examples/secure-boot/lib/env.sh" "${LOCAL_STAGING}/env.sh" + cp "${DATAPROC_EVOLUTION_DIR}/custom-images/examples/secure-boot/lib/util.sh" "${LOCAL_STAGING}/util.sh" + cp "${DATAPROC_EVOLUTION_DIR}/custom-images/examples/secure-boot/install-in-screen.sh" "${LOCAL_STAGING}/install-in-screen.sh" + cp "${DATAPROC_EVOLUTION_DIR}/custom-images/startup_script/gce-proxy-setup.sh" "${LOCAL_STAGING}/gce-proxy-setup.sh" + + # Perform a single, parallelized bulk upload to GCS (minimizes connection overhead) + gsutil -m cp -r "${LOCAL_STAGING}/*" "${GCS_SOURCES_PATH}/" >/dev/null + + echo "INFO: Launching customization on VM..." >&2 + + # 2. SSH in, download GCS assets, and trigger the background screen launcher + ssh -o ControlMaster=no -o BatchMode=yes -o ConnectTimeout=10 "${INSTANCE_NAME}" " + GCS_PATH=\$(curl -s -H 'Metadata-Flavor: Google' http://metadata.google.internal/computeMetadata/v1/instance/attributes/custom-sources-path) + rm -rf /tmp/sources + mkdir -p /tmp/sources + gsutil -m cp -r \"\${GCS_PATH}/*\" /tmp/sources/ >/dev/null + chmod +x /tmp/sources/*.sh + # Spawn the guest screen bootstrap wrapper in the background, passing FORCE_APPLY status + nohup env FORCE_APPLY=${FORCE_APPLY} bash /tmp/sources/install-in-screen.sh /tmp/sources/init_actions.sh >/tmp/launcher.log 2>&1 & + sleep 1 + " + + echo "DEBUG: Customization launched. Checking status in 2 seconds..." >&2 + sleep 2 + + # 3. Check if the customization already completed instantly before attempting to attach + set +e + ssh -o ControlMaster=no -o BatchMode=yes -o ConnectTimeout=3 "${INSTANCE_NAME}" "[[ -f /tmp/sources/customization.exit ]]" &>/dev/null + INSTANT_COMPLETED=$? + set -e + + attach_success=0 + if [[ ${INSTANT_COMPLETED} -eq 0 ]]; then + echo "INFO: Customization completed instantly on the VM. Skipping screen attachment." >&2 + attach_success=0 + else + # Connect and attach the developer interactively to the newly spawned session (belongs to cjac, so no sudo!) + set +e + ssh -t "${INSTANCE_NAME}" screen -rxU customization + attach_success=$? + set -e + fi + + # If we failed to attach (e.g., screen exited before we connected), + # harvest the exit status and logs from the VM so the developer gets immediate feedback! + if [[ ${attach_success} -ne 0 ]]; then + echo "INFO: Customization session closed or exited early. Harvesting logs from VM..." >&2 + set +e + # Fetch exit code + exit_code=$(ssh -o BatchMode=yes "${INSTANCE_NAME}" "cat /tmp/sources/customization.exit" 2>/dev/null) + has_exit=$? + set -e + + if [[ ${has_exit} -eq 0 && -n "${exit_code}" ]]; then + # Define ANSI colors for the local shell output + GREEN='\033[0;32m' + RED='\033[0;31m' + NC='\033[0m' + + echo "========================================================================" >&2 + echo " GUEST-SIDE EXECUTION LOG (Harvested from VM)" >&2 + echo "========================================================================" >&2 + ssh -o BatchMode=yes "${INSTANCE_NAME}" "cat /tmp/sources/customization-output.log" || true + echo "========================================================================" >&2 + + if [[ "${exit_code}" -eq 0 ]]; then + echo -e "${GREEN}🎉 BUILD SUCCESSFUL (Exit Code: 0)${NC}" >&2 + exit 0 + else + echo -e "${RED}❌ BUILD FAILED (Exit Code: ${exit_code})${NC}" >&2 + exit "${exit_code}" + fi + else + echo "ERROR: Failed to attach to screen, and no exit status was recorded on the VM." >&2 + echo "This indicates the startup launcher crashed or failed to spawn the screen." >&2 + echo "See /tmp/launcher.log on the VM for details." >&2 + exit 1 + fi + fi +fi + +# ======================================================================== +# Step 4: Validate Build Exit Status +# ======================================================================== +echo "INFO: Customization session closed. Fetching build exit status from VM..." >&2 + +set +e +EXIT_CODE=$(ssh -o ControlMaster=no -o ConnectTimeout=5 "${INSTANCE_NAME}" "cat /tmp/sources/customization.exit" 2>/dev/null) +SSH_STATUS=$? +set -e + +if [[ ${SSH_STATUS} -ne 0 || -z "${EXIT_CODE}" ]]; then + # Check if the screen session is still active (meaning the developer detached) + set +e + screen_still_active=$(ssh -o ControlMaster=no -o ConnectTimeout=5 "${INSTANCE_NAME}" "screen -ls | grep -q customization" 2>/dev/null; echo $?) + set -e + + if [[ ${screen_still_active} -eq 0 ]]; then + cat <&2 +======================================================================== + â„šī¸ DETACHED FROM ACTIVE CUSTOMIZATION +======================================================================== +The customization build is still executing in the background on the VM. +To re-attach and monitor the live run later, simply run this script again: + + bash examples/secure-boot/bin/customize-in-screen.sh +======================================================================== +EOF + exit 0 + else + echo "ERROR: Customization finished but exit status was not recorded on the VM." >&2 + exit 1 + fi +fi + +# Trim whitespace +EXIT_CODE=$(echo "${EXIT_CODE}" | xargs) + +if [[ "${EXIT_CODE}" == "0" ]]; then + cat <&2 +======================================================================== + 🎉 BUILD SUCCESSFUL +======================================================================== +The customization script completed successfully with exit code 0! +You can now proceed to convert this VM to a custom image: + + gcloud compute instances stop ${INSTANCE_NAME} --zone=${ZONE} --project=${PROJECT_ID} + gcloud compute images create [IMAGE_NAME] --source-disk=${INSTANCE_NAME} ... +======================================================================== +EOF + exit 0 +else + cat <&2 +======================================================================== + ❌ BUILD FAILED (Exit Code: ${EXIT_CODE}) +======================================================================== +The customization script failed inside the screen session. + +DIAGNOSTICS & RETRY WORKFLOW: +1. View the full build log: + ssh ${INSTANCE_NAME} "cat /tmp/sources/customization-output.log" + +2. Hot-fix the script directly on the VM for instant testing: + ssh ${INSTANCE_NAME} + +3. Correct the local script on your workstation: + Edit: ${CUSTOMIZATION_SCRIPT} + +4. Retry the entire pipeline (this will sync your fix and restart the build): + bash examples/secure-boot/bin/customize-in-screen.sh +======================================================================== +EOF + exit "${EXIT_CODE}" +fi diff --git a/examples/secure-boot/bin/destroy-debug-vm.sh b/examples/secure-boot/bin/destroy-debug-vm.sh new file mode 100755 index 0000000..e5b2b4a --- /dev/null +++ b/examples/secure-boot/bin/destroy-debug-vm.sh @@ -0,0 +1,52 @@ +#!/bin/bash +# +# Helper script to destroy a debug VM created by create-debug-vm.sh +# + +set -e + +DEBUG="${DEBUG:-0}" +if (( DEBUG != 0 )); then + set -x +fi + +# Source environment variables +if [[ -f "$(dirname "$0")/../lib/env.sh" ]]; then + source "$(dirname "$0")/../lib/env.sh" +else + echo "ERROR: examples/secure-boot/lib/env.sh not found." + exit 1 +fi +if [[ -f "$(dirname "$0")/../lib/util.sh" ]]; then + source "$(dirname "$0")/../lib/util.sh" +else + echo "ERROR: examples/secure-boot/lib/util.sh not found." + exit 1 +fi + +USAGE="Usage: bash $(basename "$0")" + +if [[ -z "${IMAGE_VERSION}" ]]; then + echo "ERROR: IMAGE_VERSION is not set in env.json." + exit 1 +fi +if [[ -z "${CUSTOMIZATION_SCRIPT}" ]]; then + echo "ERROR: CUSTOMIZATION_SCRIPT is not set in env.json." + exit 1 +fi + +INSTANCE_NAME="debug-$(echo "${IMAGE_VERSION}" | tr '.' '-')-$(basename "${CUSTOMIZATION_SCRIPT}" .sh | tr '.' '-' | tr '_' '-')" + +echo "Attempting to delete instance: ${INSTANCE_NAME}" +echo "Project: ${PROJECT_ID}, Zone: ${ZONE}" + +if gcloud compute instances describe "${INSTANCE_NAME}" --zone "${ZONE}" --project "${PROJECT_ID}" > /dev/null 2>&1; then + run_gcloud delete_instance gcloud compute instances delete "${INSTANCE_NAME}" --zone "${ZONE}" --project "${PROJECT_ID}" -q + echo "Instance ${INSTANCE_NAME} deleted." +else + echo "Instance ${INSTANCE_NAME} not found." +fi + +GCS_SOURCES_PATH="gs://${BUCKET}/${INSTANCE_NAME}/sources" +echo "Cleaning up GCS path: ${GCS_SOURCES_PATH}" +run_gsutil rm_gcs gsutil -m rm -r "${GCS_SOURCES_PATH}" || echo "GCS path not found, continuing..." diff --git a/examples/secure-boot/bin/get-builder-log b/examples/secure-boot/bin/get-builder-log new file mode 100755 index 0000000..74f8dcf --- /dev/null +++ b/examples/secure-boot/bin/get-builder-log @@ -0,0 +1,38 @@ +#!/bin/bash +# Get the full serial port output of the latest builder VM for a given image prefix. + +set -e + +DEBUG="${DEBUG:-0}" +if (( DEBUG != 0 )); then + set -x +fi + +source examples/secure-boot/lib/env.sh +source examples/secure-boot/lib/util.sh + +IMAGE_PREFIX="${1:-dataproc-2-0-deb10}" +PURPOSE="${2:-secure-boot}" + +print_status "Finding latest builder instance for ${IMAGE_PREFIX}*${PURPOSE}"... + +INSTANCE_NAME=$(gcloud compute instances list \ + --project="${PROJECT_ID}" \ + --zones="${ZONE}" \ + --filter="name~'${IMAGE_PREFIX}.*${PURPOSE}-install'" \ + --format="value(name)" \ + --sort-by="~creationTimestamp" \ + | head -n 1) + +if [[ -z "${INSTANCE_NAME}" ]]; then + report_result "Not Found" + echo "No builder instance found matching the prefix." >&2 + exit 1 +fi + +report_result "Found: ${INSTANCE_NAME}" + +print_status "Getting serial port output for ${INSTANCE_NAME}"... +echo "" # Add a newline after the status + +gcloud compute instances get-serial-port-output "${INSTANCE_NAME}" --zone "${ZONE}" --project "${PROJECT_ID}" --port 1 | less diff --git a/examples/secure-boot/bin/scp-debug-vm.sh b/examples/secure-boot/bin/scp-debug-vm.sh new file mode 100755 index 0000000..c6e8289 --- /dev/null +++ b/examples/secure-boot/bin/scp-debug-vm.sh @@ -0,0 +1,51 @@ +#!/bin/bash +# +# Helper script to SCP files to the /tmp directory of a debug VM +# + +set -e + +DEBUG="${DEBUG:-0}" +if (( DEBUG != 0 )); then + set -x +fi + +# Source environment variables +if [[ -f "$(dirname "$0")/../lib/env.sh" ]]; then + source "$(dirname "$0")/../lib/env.sh" +else + echo "ERROR: examples/secure-boot/lib/env.sh not found." + exit 1 +fi + +USAGE="Usage: bash $(basename "$0") " + +if [[ -z "${IMAGE_VERSION}" ]]; then + echo "ERROR: IMAGE_VERSION is not set in env.json." + exit 1 +fi +if [[ -z "${CUSTOMIZATION_SCRIPT}" ]]; then + echo "ERROR: CUSTOMIZATION_SCRIPT is not set in env.json." + exit 1 +fi + +LOCAL_PATH="${1}" +if [[ -z "${LOCAL_PATH}" ]]; then + echo "ERROR: Missing local path argument." + echo "${USAGE}" + exit 1 +fi + +if [[ ! -e "${LOCAL_PATH}" ]]; then + echo "ERROR: Local path not found: ${LOCAL_PATH}" + exit 1 +fi + +INSTANCE_NAME="debug-$(echo "${IMAGE_VERSION}" | tr '.' '-')-$(basename "${CUSTOMIZATION_SCRIPT}" .sh | tr '.' '-' | tr '_' '-')" + +echo "Attempting to SCP '${LOCAL_PATH}' to ${INSTANCE_NAME}:/tmp/" +echo "Project: ${PROJECT_ID}, Zone: ${ZONE}" + +gcloud compute scp --recurse --zone "${ZONE}" --project "${PROJECT_ID}" --tunnel-through-iap "${LOCAL_PATH}" "${INSTANCE_NAME}:/tmp/" + +echo "File/directory copied successfully to ${INSTANCE_NAME}:/tmp/$(basename "${LOCAL_PATH}")" diff --git a/examples/secure-boot/bin/ssh-builder b/examples/secure-boot/bin/ssh-builder new file mode 100755 index 0000000..51067a1 --- /dev/null +++ b/examples/secure-boot/bin/ssh-builder @@ -0,0 +1,34 @@ +#!/bin/bash +# SSH into the latest builder VM for a given image prefix. + +set -e + +source examples/secure-boot/lib/env.sh +source examples/secure-boot/lib/util.sh + +IMAGE_PREFIX="${1:-dataproc-2-2-deb10}" +PURPOSE="${2:-secure-boot}" + +print_status "Finding latest builder instance for ${IMAGE_PREFIX}*${PURPOSE}"... + +INSTANCE_NAME=$(gcloud compute instances list \ + --project="${PROJECT_ID}" \ + --zones="${ZONE}" \ + --filter="name~'${IMAGE_PREFIX}.*${PURPOSE}-install' AND status=RUNNING" \ + --format="value(name)" \ + --sort-by=\"~creationTimestamp\" \ + | head -n 1) + +if [[ -z "${INSTANCE_NAME}" ]]; then + report_result "Not Found" + echo "No RUNNING builder instance found matching the prefix." >&2 +# exit 1 +fi + +report_result "Found: ${INSTANCE_NAME}" + +print_status "SSHing into ${INSTANCE_NAME}"... + +INSTANCE_NAME=debug-deb12-build + +gcloud compute ssh "${INSTANCE_NAME}" --zone "${ZONE}" --project "${PROJECT_ID}" diff --git a/examples/secure-boot/bin/ssh-debug-vm.sh b/examples/secure-boot/bin/ssh-debug-vm.sh new file mode 100755 index 0000000..a5498a5 --- /dev/null +++ b/examples/secure-boot/bin/ssh-debug-vm.sh @@ -0,0 +1,56 @@ +#!/bin/bash +# +# Helper script to SSH into a debug VM created by create-debug-vm.sh +# + +set -e + +DEBUG="${DEBUG:-0}" +if (( DEBUG != 0 )); then + set -x +fi + +# Source environment variables +if [[ -f "$(dirname "$0")/../lib/env.sh" ]]; then + source "$(dirname "$0")/../lib/env.sh" +else + echo "ERROR: examples/secure-boot/lib/env.sh not found." + exit 1 +fi + +USAGE="Usage: bash $(basename "$0") [command...]" + +if [[ -z "${IMAGE_VERSION}" ]]; then + echo "ERROR: IMAGE_VERSION is not set in env.json." + exit 1 +fi +if [[ -z "${CUSTOMIZATION_SCRIPT}" ]]; then + echo "ERROR: CUSTOMIZATION_SCRIPT is not set in env.json." + exit 1 +fi + +COMMAND_TO_RUN=("$@") + +INSTANCE_NAME="debug-$(echo "${IMAGE_VERSION}" | tr '.' '-')-$(basename "${CUSTOMIZATION_SCRIPT}" .sh | tr '.' '-' | tr '_' '-')" + +echo "Attempting to SSH into instance: ${INSTANCE_NAME}" +echo "Project: ${PROJECT_ID}, Zone: ${ZONE}" + +declare -a gcloud_ssh_args +gcloud_ssh_args=( + gcloud compute ssh + --zone "${ZONE}" + --project "${PROJECT_ID}" + --tunnel-through-iap + "${INSTANCE_NAME}" +) + +if [[ ${#COMMAND_TO_RUN[@]} -eq 0 ]]; then + # Interactive session + gcloud_ssh_args+=( -- -t -o ConnectTimeout=60 -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -AY ) +else + # Command execution + gcloud_ssh_args+=( --command "${COMMAND_TO_RUN[*]}" ) +fi + +"${gcloud_ssh_args[@]}" diff --git a/examples/secure-boot/build-and-run-podman.sh b/examples/secure-boot/build-and-run-podman.sh index edb3ce7..d9d0598 100644 --- a/examples/secure-boot/build-and-run-podman.sh +++ b/examples/secure-boot/build-and-run-podman.sh @@ -147,6 +147,9 @@ time podman run -it --rm \ -e DEBUG=0 \ -e REPRO_TMPDIR=/tmp \ -e DATAPROC_EVOLUTION_DIR=${DATAPROC_EVOLUTION_DIR} \ + -e SWP_IP="${SWP_IP}" \ + -e SWP_PORT="${SWP_PORT}" \ + -e PROXY_CERT_GCS_PATH="${PROXY_CERT_GCS_PATH}" \ ${image} \ bash -x examples/secure-boot/pre-init.sh "${DATAPROC_IMAGE_VERSION}" # bash examples/secure-boot/build-current-images.sh diff --git a/examples/secure-boot/build-current-images.sh b/examples/secure-boot/build-current-images.sh index b3309d1..29c19bc 100644 --- a/examples/secure-boot/build-current-images.sh +++ b/examples/secure-boot/build-current-images.sh @@ -65,7 +65,7 @@ function find_disk_usage() { grep -H '^[^\+].*Cust.*ript' /tmp/custom-image-*${timestamp}*/logs/workflow.log echo '# DP_IMG_VER RECOMMENDED_DISK_SIZE DSK_SZ D_USED D_FREE D%F PURPOSE' # workflow_log=/tmp/custom-image-dataproc-2-0-deb10-20250424-232955-tf-20250425-230559/logs/workflow.log - for workflow_log in $(grep -Hl "Customization script" /tmp/custom-image-*/logs/workflow.log) ; do + for workflow_log in $(grep -Hl "Customization script" /tmp/custom-image-*${timestamp}*/logs/workflow.log 2>/dev/null || true) ; do startup_log="${workflow_log/workflow/startup-script}" grep -v '^\[' "${startup_log}" \ | grep -A20 'Filesystem.*Avail' | tail -20 \ diff --git a/examples/secure-boot/install-in-screen.sh b/examples/secure-boot/install-in-screen.sh new file mode 100644 index 0000000..c2722f2 --- /dev/null +++ b/examples/secure-boot/install-in-screen.sh @@ -0,0 +1,159 @@ +#!/bin/bash +# +# Bootstrap wrapper to run GCE image customization scripts inside a detached, +# inspectable screen session. This protects long-running builds from timeouts, +# allows real-time developer attachment, and ensures logs are still streamed +# to the serial console for the orchestrator. + +set -euo pipefail + +SCRIPT_DIR=$(dirname "$(readlink -f "$0")") +TARGET_SCRIPT="${1:-${SCRIPT_DIR}/no-customization.sh}" + +if [[ ! -f "${TARGET_SCRIPT}" ]]; then + echo "ERROR: Target customization script ${TARGET_SCRIPT} not found." >&2 + exit 1 +fi + +echo "INFO: Preparing to run customization: ${TARGET_SCRIPT}" >&2 + +# --- Retrieve VM Details from Cache --- +# We use the local JSON cache to print the exact SSH command for the developer +CACHE_DIR="/dev/shm/metadata_cache" +PROJECT_ID="UNKNOWN_PROJECT" +ZONE="UNKNOWN_ZONE" +VM_NAME="UNKNOWN_VM" + +if [[ -d "${CACHE_DIR}" ]]; then + # Parse cached values using jq + if [[ -f "${CACHE_DIR}/project_attributes.json" ]]; then + PROJECT_ID=$(jq -r '.["project-id"] // "UNKNOWN_PROJECT"' "${CACHE_DIR}/project_attributes.json") + fi + # We can also query standard GCE metadata paths from the VM + # (These are not attributes, so they might not be in the cache, but they are safe to query once) + VM_NAME=$(curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/name || echo "UNKNOWN_VM") + ZONE_FULL=$(curl -s -H "Metadata-Flavor: Google" http://metadata.google.internal/computeMetadata/v1/instance/zone || echo "UNKNOWN_ZONE") + ZONE="${ZONE_FULL##*/}" +fi + +# --- Print Developer Diagnostic Instructions --- +# This goes to stderr/console so it is visible in the GCE serial port logs immediately +cat <&2 +======================================================================== + CUSTOMIZATION SCREEN AUTOMATION ACTIVE +======================================================================== +The customization script is executing inside a detached screen session. + +To attach to the live session and monitor/debug in real-time, +run the following command from your workstation: + + gcloud compute ssh ${VM_NAME} \\ + --project=${PROJECT_ID} \\ + --zone=${ZONE} \\ + --tunnel-through-iap \\ + --command="screen -rxU customization" + +======================================================================== +EOF + +# --- Launch Customization in Screen --- +LOG_FILE="/tmp/sources/customization-output.log" +EXIT_FILE="/tmp/sources/customization.exit" +touch "${LOG_FILE}" +rm -f "${EXIT_FILE}" + +echo "INFO: Spawning screen session 'customization'..." >&2 + +# We spawn the screen as the normal user, but run the inner payload as root via sudo +screen -dmS customization sudo FORCE_APPLY="${FORCE_APPLY:-0}" bash -c " + # Define ANSI colors inside the root shell + BLUE='\\033[0;34m' + GREEN='\\033[0;32m' + YELLOW='\\033[1;33m' + RED='\\033[0;31m' + NC='\\033[0m' + + set +e + + # 1. Source guest-side harvester libraries inside the root shell (defines get_metadata_attribute) + source /tmp/sources/gce-proxy-setup.sh + set +e + + # 2. Globally export all metadata helper functions to match Dataproc environment fidelity + export -f get_metadata_attribute get_metadata_value print_metadata_value print_metadata_value_if_exists get_cached_state os_id is_debuntu is_rocky + + # 3. Wait for developer to attach (guarantees visibility for fast scripts/probes) + echo -e \"\${YELLOW}INFO: Spawning screen. Waiting 5 seconds for developer to attach...\${NC}\" >&2 + for i in 5 4 3 2 1; do + echo -e \"\${YELLOW}Starting customization in \${i} seconds...\${NC}\" >&2 + sleep 1 + done + + # 4. Execute the target script with a clean colored header + echo -e \"\${BLUE}========================================================================\${NC}\" + echo -e \"\${BLUE} LAUNCHING CUSTOMIZATION SCRIPT: ${TARGET_SCRIPT}\${NC}\" + echo -e \"\${BLUE}========================================================================\${NC}\" + echo '' + + bash ${TARGET_SCRIPT} 2>&1 | tee ${LOG_FILE} + EXIT_CODE=\${PIPESTATUS[0]} + + # 5. Print clean colored final status indicator + echo '' + echo -e \"\${BLUE}========================================================================\${NC}\" + if [[ \${EXIT_CODE} -eq 0 ]]; then + echo -e \"\${GREEN} 🎉 CUSTOMIZATION SUCCESSFUL (Exit Code: 0)\${NC}\" + else + echo -e \"\${RED} ❌ CUSTOMIZATION FAILED (Exit Code: \${EXIT_CODE})\${NC}\" + fi + echo -e \"\${BLUE}========================================================================\${NC}\" + + echo \${EXIT_CODE} > ${EXIT_FILE} + chmod 644 ${EXIT_FILE} ${LOG_FILE} 2>/dev/null || true +" + +# --- Wait and Stream Logs --- +# We find the PID of the newly spawned screen session +sleep 2 +SCREEN_PID=$(screen -ls | grep customization | awk '{print $1}' | cut -d. -f1 || echo "") + +if [[ -z "${SCREEN_PID}" ]]; then + # If the screen is gone, check if it already completed instantly + if [[ -f "${EXIT_FILE}" ]]; then + echo "INFO: Customization completed instantly." >&2 + exit 0 + fi + echo "ERROR: Failed to spawn screen session." >&2 + exit 1 +fi + +echo "DEBUG: Screen session spawned with PID ${SCREEN_PID}. Streaming logs to serial port..." >&2 + +# Start tailing the log file in the background to stream to GCE serial port +tail -f "${LOG_FILE}" & +TAIL_PID=$! + +# Block until the screen session exits +while kill -0 "${SCREEN_PID}" 2>/dev/null; do + sleep 5 +done + +# Stop the background tailing +kill "${TAIL_PID}" &>/dev/null || true + +# --- Propagate Exit Code --- +# Read the exit code written by the script inside the screen session +if [[ -f "${EXIT_FILE}" ]]; then + EXIT_CODE=$(cat "${EXIT_FILE}") + echo "INFO: Customization script finished with exit code ${EXIT_CODE}" >&2 + + if [[ "${EXIT_CODE}" -eq 0 ]]; then + echo "startup-script: BuildSucceeded: Customization complete." >&2 + else + echo "startup-script: BuildFailed: Customization failed." >&2 + fi + exit "${EXIT_CODE}" +else + echo "ERROR: Customization finished but exit code was not recorded." >&2 + exit 1 +fi diff --git a/examples/secure-boot/lib/env.sh b/examples/secure-boot/lib/env.sh index 9fbf9b7..207e25c 100644 --- a/examples/secure-boot/lib/env.sh +++ b/examples/secure-boot/lib/env.sh @@ -14,13 +14,25 @@ # # This script loads and validates environment variables from env.json -if [[ -z "${ENV_JSON_PATH}" ]]; then +if [[ -z "${ENV_JSON_PATH:-}" ]]; then ENV_JSON_PATH="env.json" fi if [[ -z "${DATAPROC_EVOLUTION_DIR:-}" ]]; then - SCRIPT_DIR="$(dirname "${BASH_SOURCE[0]}")" - export DATAPROC_EVOLUTION_DIR="$(realpath "${SCRIPT_DIR}/../../../..")" + # Traverse upwards from env.sh's directory to resolve the repository root + # containing both custom-images and cloud-dataproc, bypassing nested git boundaries. + current_dir=$(dirname "$(readlink -f "${BASH_SOURCE[0]}")") + while [[ "${current_dir}" != "/" ]]; do + if [[ -d "${current_dir}/custom-images" && -d "${current_dir}/cloud-dataproc" ]]; then + export DATAPROC_EVOLUTION_DIR="${current_dir}" + break + fi + current_dir=$(dirname "${current_dir}") + done + if [[ -z "${DATAPROC_EVOLUTION_DIR:-}" ]]; then + echo "ERROR: Cannot resolve dataproc-evolution repository root from ${BASH_SOURCE[0]}." >&2 + exit 1 + fi fi if [[ ! -f "${ENV_JSON_PATH}" ]]; then @@ -44,6 +56,10 @@ DOMAIN="$(jq -r .DOMAIN "${ENV_JSON_PATH}")" IMAGE_VERSION="$(jq -r .IMAGE_VERSION "${ENV_JSON_PATH}")" CUSTOMIZATION_SCRIPT="$(jq -r .CUSTOMIZATION_SCRIPT "${ENV_JSON_PATH}")" +SWP_IP="$(jq -r .SWP_IP "${ENV_JSON_PATH}")"; [[ "${SWP_IP}" == "null" ]] && SWP_IP="" +SWP_PORT="$(jq -r .SWP_PORT "${ENV_JSON_PATH}")"; [[ "${SWP_PORT}" == "null" ]] && SWP_PORT="" +PROXY_CERT_GCS_PATH="$(jq -r .PROXY_CERT_GCS_PATH "${ENV_JSON_PATH}")"; [[ "${PROXY_CERT_GCS_PATH}" == "null" ]] && PROXY_CERT_GCS_PATH="" + # Validate all required variables from env.json missing_vars=() required_vars=(PROJECT_ID PURPOSE BUCKET TEMP_BUCKET ZONE SUBNET PRINCIPAL_USER DOMAIN IMAGE_VERSION CUSTOMIZATION_SCRIPT) @@ -70,7 +86,7 @@ if [ ${#missing_vars[@]} -gt 0 ]; then exit 1 fi -export PROJECT_ID PURPOSE BUCKET TEMP_BUCKET ZONE SUBNET HIVE_NAME HIVEDB_PW_URI SECRET_NAME KMS_KEY_URI PRINCIPAL_USER DOMAIN IMAGE_VERSION CUSTOMIZATION_SCRIPT +export PROJECT_ID PURPOSE BUCKET TEMP_BUCKET ZONE SUBNET HIVE_NAME HIVEDB_PW_URI SECRET_NAME KMS_KEY_URI PRINCIPAL_USER DOMAIN IMAGE_VERSION CUSTOMIZATION_SCRIPT SWP_IP SWP_PORT PROXY_CERT_GCS_PATH PRINCIPAL="${PRINCIPAL_USER}@${DOMAIN}" export PRINCIPAL diff --git a/examples/secure-boot/lib/util.sh b/examples/secure-boot/lib/util.sh index b79e06e..d8dac33 100644 --- a/examples/secure-boot/lib/util.sh +++ b/examples/secure-boot/lib/util.sh @@ -58,8 +58,9 @@ function run_gcloud() { print_status " Executing: ${cmd_array[*]}..." - "${cmd_array[@]}" > "${log_path}" 2>&1 - local retval=$? + local retval=0 + "${cmd_array[@]}" > "${log_path}" 2>&1 || retval=$? + readonly retval if [[ ${retval} -ne 0 ]]; then report_result "FAIL" "${log_path}" diff --git a/examples/secure-boot/no-customization.sh b/examples/secure-boot/no-customization.sh index d9d1f1f..08ae4d8 100644 --- a/examples/secure-boot/no-customization.sh +++ b/examples/secure-boot/no-customization.sh @@ -43,6 +43,18 @@ print( " samples-taken: ", scalar @siz, $/, # Monitor disk usage in a screen session df / | tee "/run/disk-usage.log" touch "/run/keep-running-df" + +# Ensure screen is installed for disk monitoring +if ! command -v screen >/dev/null 2>&1; then + echo "INFO: Installing screen for disk usage monitoring..." >&2 + if command -v dnf >/dev/null 2>&1; then + sudo dnf -y -q install epel-release && sudo dnf -y -q install screen + elif command -v apt-get >/dev/null 2>&1; then + sudo apt-get update -y -qq >/dev/null 2>&1 + sudo apt-get install -y -qq screen >/dev/null 2>&1 + fi +fi + screen -d -m -LUS keep-running-df \ bash -c "while [[ -f /run/keep-running-df ]] ; do df / | tee -a /run/disk-usage.log ; sleep 5s ; done" diff --git a/examples/secure-boot/pre-init.screenrc b/examples/secure-boot/pre-init.screenrc index a3aa589..603f783 100644 --- a/examples/secure-boot/pre-init.screenrc +++ b/examples/secure-boot/pre-init.screenrc @@ -18,8 +18,10 @@ screen -L -t 2.1-ubuntu20-arm 11 /bin/bash -x examples/secure-boot/pre-init.sh 2 screen -L -t 2.2-debian12 8 /bin/bash -x examples/secure-boot/pre-init.sh 2.2-debian12 screen -L -t 2.2-rocky9 9 /bin/bash -x examples/secure-boot/pre-init.sh 2.2-rocky9 screen -L -t 2.2-ubuntu22 10 /bin/bash -x examples/secure-boot/pre-init.sh 2.2-ubuntu22 +screen -L -t 2.2-ubuntu22-arm 16 /bin/bash -x examples/secure-boot/pre-init.sh 2.2-ubuntu22-arm screen -L -t 2.3-debian12 12 /bin/bash -x examples/secure-boot/pre-init.sh 2.3-debian12 screen -L -t 2.3-rocky9 13 /bin/bash -x examples/secure-boot/pre-init.sh 2.3-rocky9 screen -L -t 2.3-ubuntu22 14 /bin/bash -x examples/secure-boot/pre-init.sh 2.3-ubuntu22 screen -L -t 2.3-ml-ubuntu22 15 /bin/bash -x examples/secure-boot/pre-init.sh 2.3-ml-ubuntu22 +screen -L -t 2.3-ubuntu22-arm 17 /bin/bash -x examples/secure-boot/pre-init.sh 2.3-ubuntu22-arm diff --git a/examples/secure-boot/pre-init.sh b/examples/secure-boot/pre-init.sh index 9329613..086a27f 100644 --- a/examples/secure-boot/pre-init.sh +++ b/examples/secure-boot/pre-init.sh @@ -66,33 +66,41 @@ case "${dataproc_version}" in "2.1-rocky8" ) CUDA_VERSION="12.4.1" ; short_dp_ver=2.1-roc8 ;; "2.1-ubuntu20" ) CUDA_VERSION="12.4.1" ; short_dp_ver=2.1-ubu20 ;; "2.1-ubuntu20-arm" ) CUDA_VERSION="12.4.1" ; short_dp_ver=2.1-ubu20-arm ;; - "2.2-debian12" ) CUDA_VERSION="13.1.0" ; short_dp_ver=2.2-deb12 ;; - "2.2-rocky9" ) CUDA_VERSION="13.1.0" ; short_dp_ver=2.2-roc9 ;; - "2.2-ubuntu22" ) CUDA_VERSION="13.1.0" ; short_dp_ver=2.2-ubu22 ;; - "2.3-debian12" ) CUDA_VERSION="13.1.0" ; short_dp_ver=2.3-deb12 ;; - "2.3-rocky9" ) CUDA_VERSION="13.1.0" ; short_dp_ver=2.3-roc9 ;; - "2.3-ubuntu22" ) CUDA_VERSION="13.1.0" ; short_dp_ver=2.3-ubu22 ;; - "2.3-ml-ubuntu22" ) CUDA_VERSION="13.1.0" ; short_dp_ver=2.3-ml-ubu22 ; disk_size_gb="50";; + "2.2-debian12" ) CUDA_VERSION="13.2.0" ; short_dp_ver=2.2-deb12 ;; + "2.2-rocky9" ) CUDA_VERSION="13.2.0" ; short_dp_ver=2.2-roc9 ;; + "2.2-ubuntu22" ) CUDA_VERSION="13.2.0" ; short_dp_ver=2.2-ubu22 ;; + "2.2-ubuntu22-arm" ) CUDA_VERSION="13.2.0" ; short_dp_ver=2.2-ubu22-arm ;; + "2.3-debian12" ) CUDA_VERSION="13.2.0" ; short_dp_ver=2.3-deb12 ;; + "2.3-rocky9" ) CUDA_VERSION="13.2.0" ; short_dp_ver=2.3-roc9 ;; + "2.3-ubuntu22" ) CUDA_VERSION="13.2.0" ; short_dp_ver=2.3-ubu22 ;; + "2.3-ubuntu22-arm" ) CUDA_VERSION="13.2.0" ; short_dp_ver=2.3-ubu22-arm ;; + "2.3-ml-ubuntu22" ) CUDA_VERSION="13.2.0" ; short_dp_ver=2.3-ml-ubu22 ; disk_size_gb="50";; esac function create_h100_instance() { python3 generate_custom_image.py \ --machine-type "a3-highgpu-2g" \ --accelerator "type=nvidia-h100-80gb,count=2" \ - $* + "$@" } function create_t4_instance() { python3 generate_custom_image.py \ --machine-type "n1-standard-32" \ --accelerator "type=nvidia-tesla-t4,count=1" \ - $* + "$@" } function create_unaccelerated_instance() { python3 generate_custom_image.py \ --machine-type "n1-standard-2" \ - $* + "$@" +} + +function create_arm_instance() { + python3 generate_custom_image.py \ + --machine-type "c4a-standard-2" \ + "$@" } OPTIONAL_COMPONENTS_ARG="" @@ -154,6 +162,9 @@ function generate() { metadata_args+=("universe-domain=${universe_domain}") create_function="create_unaccelerated_instance" + if [[ "${dataproc_version}" == *arm* ]]; then + create_function="create_arm_instance" + fi if [[ "${customization_script}" =~ "cloud-sql-proxy.sh" ]] ; then metadata_args+=( @@ -243,6 +254,7 @@ function generate() { --trusted-cert "tls/db.der" \ --shutdown-instance-timer-sec=30 \ --no-smoke-test \ + --retain-on-failure \ ${extra_args} then report_result "Success" @@ -251,7 +263,7 @@ function generate() { else local exit_code=$? report_result "Fail" - local img_build_dir="$(ls -d /tmp/custom-image-${image_name}-* 2>/dev/null || echo '')" + local img_build_dir="$(ls -d /tmp/custom-image-${image_name}-* 2>/dev/null | sort | tail -n1 || echo '')" # retry if the startup-script.log file does not exist or is empty if [[ -n "${img_build_dir}" ]]; then local startup_script_log="${img_build_dir}/logs/startup-script.log" @@ -311,6 +323,7 @@ function generate_from_prerelease_version() { "2.3-debian12" ) image_uri="${img_pfx}/dataproc-2-3-deb12-${src_timestamp}-rc01" ;; "2.3-rocky9" ) image_uri="${img_pfx}/dataproc-2-3-roc9-${src_timestamp}-rc01" ;; "2.3-ubuntu22" ) image_uri="${img_pfx}/dataproc-2-3-ubu22-${src_timestamp}-rc01" ;; + "2.3-ubuntu22-arm" ) image_uri="${img_pfx}/dataproc-2-3-ubu22-arm-${src_timestamp}-rc01" ;; "2.3-ml-ubuntu22" ) image_uri="${img_pfx}/dataproc-2-3-ml-ubu22-${src_timestamp}-rc01" ;; esac generate --base-image-uri "${image_uri}" @@ -376,7 +389,7 @@ case "${dataproc_version}" in "2.2-rocky9" ) disk_size_gb="60" ;; # 49.79G 43.51G 6.28G 88% / # 20250429-193537-tf "2.2-ubuntu22" ) disk_size_gb="60" ;; # 48.28G 43.32G 4.94G 90% / # 20250429-193537-tf - "2.3-debian12" ) disk_size_gb="50" ;; # 41.11G 36.20G 3.12G 93% / # 20250507-083009-tf + "2.3-debian12" ) disk_size_gb="100" ;; # Increased to 100GB to prevent out-of-space during monolithic optional components accumulation "2.3-rocky9" ) disk_size_gb="50" ;; # 49.79G 37.82G 11.98G 76% / # 20250507-083009-tf "2.3-ubuntu22" ) disk_size_gb="50" ;; # 40.52G 36.18G 4.33G 90% / # 20250507-083009-tf "2.3-ml-ubuntu22" ) disk_size_gb="70" ;; # 40.52G 36.18G 4.33G 90% / # 20250507-083009-tf @@ -386,7 +399,7 @@ esac # Extract major.minor version (e.g., 2.2 from 2.2-debian12) MAJOR_MINOR_VERSION=$(echo "${dataproc_version}" | cut -d'-' -f1) -if version_ge "${MAJOR_MINOR_VERSION}" "2.2" ; then +if version_ge "${MAJOR_MINOR_VERSION}" "2.2" && [[ "${dataproc_version}" != *arm* ]] ; then print_status "=== Generating GPU/ML images for ${dataproc_version} (>=2.2) ===" # Install GPU drivers + cuda + rapids + cuDNN + nccl + tensorflow + pytorch on dataproc base image @@ -400,22 +413,28 @@ if version_ge "${MAJOR_MINOR_VERSION}" "2.2" ; then fi if version_ge "${MAJOR_MINOR_VERSION}" "2.2" ; then - # Install GPU drivers + cuda + rapids + cuDNN + nccl + tensorflow + pytorch on dataproc base image on a proxy base - PURPOSE="proxy-tf" - customization_script="${DATAPROC_EVOLUTION_DIR}/initialization-actions/gpu/install_gpu_driver.sh" - print_status "=== Waiting for TF build to complete to leverage cache... ===" - while [[ ! -f "${tmpdir}/sentinels/tf_build_complete" ]]; do - sleep 10 - done - print_status "=== Generating proxy-tf image for ${dataproc_version} ===" - time generate_from_base_purpose "secure-proxy" + DOCKER_BASE_IMAGE="secure-proxy" + + if [[ "${dataproc_version}" != *arm* ]] ; then + # Install GPU drivers + cuda + rapids + cuDNN + nccl + tensorflow + pytorch on dataproc base image on a proxy base + PURPOSE="proxy-tf" + customization_script="${DATAPROC_EVOLUTION_DIR}/initialization-actions/gpu/install_gpu_driver.sh" + print_status "=== Waiting for TF build to complete to leverage cache... ===" + while [[ ! -f "${tmpdir}/sentinels/tf_build_complete" ]]; do + sleep 10 + done + print_status "=== Generating proxy-tf image for ${dataproc_version} ===" + time generate_from_base_purpose "secure-proxy" + + DOCKER_BASE_IMAGE="proxy-tf" + fi ## run the installer for the DOCKER optional component PURPOSE="docker" OPTIONAL_COMPONENTS_ARG='--optional-components=DOCKER' customization_script="examples/secure-boot/no-customization.sh" print_status "=== Generating ${PURPOSE} image for ${dataproc_version} ===" - time generate_from_base_purpose "proxy-tf" + time generate_from_base_purpose "${DOCKER_BASE_IMAGE}" ## run the installer for the DOCKER optional component PURPOSE="jupyter" @@ -446,6 +465,7 @@ if version_ge "${MAJOR_MINOR_VERSION}" "2.2" ; then time generate_from_base_purpose "pig" fi +if [[ "${dataproc_version}" != *arm* ]]; then ## Execute spark-rapids/spark-rapids.sh init action on base image PURPOSE="spark" @@ -513,5 +533,7 @@ customization_script="examples/secure-boot/pytorch.sh" print_status "=== Generating pytorch image for ${dataproc_version} ===" echo time generate_from_base_purpose "tf" +fi + diff --git a/startup_script/gce-proxy-setup.sh b/startup_script/gce-proxy-setup.sh index 546e9c1..4be84cd 100644 --- a/startup_script/gce-proxy-setup.sh +++ b/startup_script/gce-proxy-setup.sh @@ -593,9 +593,12 @@ function setup_deferred_service() { chmod +x "${target_path}" fi - # 2. Write the systemd unit file (runs BEFORE google-dataproc-agent) - echo "INFO: setup_deferred_service: Writing systemd service file ${service_file}" >&2 - cat < "${service_file}" + # 2. Write and enable the systemd unit file (runs BEFORE google-dataproc-agent) if not already present + if [[ -f "${service_file}" ]]; then + echo "INFO: setup_deferred_service: Systemd service file ${service_file} already exists. Skipping write and enable." >&2 + else + echo "INFO: setup_deferred_service: Writing systemd service file ${service_file}" >&2 + cat < "${service_file}" [Unit] Description=Inject Dynamic Dataproc Proxy Overrides into Systemd Manager DefaultDependencies=no @@ -615,17 +618,18 @@ StandardError=journal+console WantedBy=multi-user.target EOF - chmod 644 "${service_file}" + chmod 644 "${service_file}" - # 3. Enable the service so it runs on every boot - if ! command -v systemctl >/dev/null 2>&1 || [[ "$(get_cached_state 'system/systemd_is_pid1')" != "true" ]]; then - echo "ERROR: setup_deferred_service: systemd is not running as PID 1. Cannot enable deferred service." >&2 - exit 1 - fi + # 3. Enable the service so it runs on every boot + if ! command -v systemctl >/dev/null 2>&1 || [[ "$(get_cached_state 'system/systemd_is_pid1')" != "true" ]]; then + echo "ERROR: setup_deferred_service: systemd is not running as PID 1. Cannot enable deferred service." >&2 + exit 1 + fi - echo "INFO: setup_deferred_service: Enabling systemd service ${service_name}" >&2 - systemctl enable "${service_name}.service" - echo "INFO: setup_deferred_service: Deferred proxy service enabled successfully." >&2 + echo "INFO: setup_deferred_service: Enabling systemd service ${service_name}" >&2 + systemctl enable "${service_name}.service" + echo "INFO: setup_deferred_service: Deferred proxy service enabled successfully." >&2 + fi } function print_introspection_report() { @@ -661,6 +665,43 @@ if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then configure_systemd_proxy repair_boto + # ============================================================================ + # Dynamic Deferred Config Script Execution + # ============================================================================ + DEFERRED_URI=$(get_metadata_attribute 'deferred-config-script-uri' '') + if [[ -n "${DEFERRED_URI}" ]]; then + if [[ "${DEFERRED_URI}" != gs://* ]]; then + echo "ERROR: deferred-config-script-uri must be a GCS URI starting with gs://" + exit 1 + fi + + LOCAL_SCRIPT="/tmp/deferred-config-script.sh" + echo "INFO: Fetching deferred script from ${DEFERRED_URI}..." + if ! gsutil cp "${DEFERRED_URI}" "${LOCAL_SCRIPT}"; then + echo "ERROR: Failed to download deferred config script from ${DEFERRED_URI}" + exit 1 + fi + + DEFERRED_SHA=$(get_metadata_attribute 'deferred-config-script-sha256' '') + if [[ -n "${DEFERRED_SHA}" ]]; then + CALCULATED_SHA=$(sha256sum "${LOCAL_SCRIPT}" | cut -d ' ' -f 1) + if [[ "${CALCULATED_SHA}" != "${DEFERRED_SHA}" ]]; then + echo "ERROR: SHA256 checksum mismatch for deferred config script!" + rm -f "${LOCAL_SCRIPT}" + exit 1 + fi + fi + + chmod +x "${LOCAL_SCRIPT}" + echo "INFO: Executing deferred config script..." + if ! "${LOCAL_SCRIPT}"; then + echo "ERROR: Deferred config script execution failed!" + rm -f "${LOCAL_SCRIPT}" + exit 1 + fi + rm -f "${LOCAL_SCRIPT}" + fi + if [[ "${IS_CUSTOM_IMAGE_BUILD:-}" == "true" || -n "$(get_metadata_attribute 'custom-sources-path' '')" ]]; then setup_deferred_service fi diff --git a/startup_script/run.sh b/startup_script/run.sh index ba490eb..ee892f1 100644 --- a/startup_script/run.sh +++ b/startup_script/run.sh @@ -45,6 +45,25 @@ CUSTOM_SOURCES_PATH=$(/usr/share/google/get_metadata_value attributes/custom-sou # get time to wait for stdout to flush SHUTDOWN_TIMER_IN_SEC=$(/usr/share/google/get_metadata_value attributes/shutdown-timer-in-sec) +function shutdown_vm() { + local exit_code=$? + if [[ ${exit_code} -ne 0 ]]; then + local retain_on_failure + retain_on_failure=$(/usr/share/google/get_metadata_value attributes/retain-on-failure || echo "false") + if [[ "${retain_on_failure}" == "true" ]]; then + echo "startup-script: Build failed with exit code ${exit_code}." + echo "startup-script: retain-on-failure=true is set. Keeping VM running for debugging." + return 0 + fi + fi + + echo "startup-script: Sleep ${SHUTDOWN_TIMER_IN_SEC}s before shutting down..." + echo "You can change the timeout value with --shutdown-instance-timer-sec" + sleep "${SHUTDOWN_TIMER_IN_SEC}" # wait for stdout to flush + shutdown -h now +} +trap shutdown_vm EXIT + USER_DATAPROC_COMPONENTS=$( /usr/share/google/get_metadata_value attributes/optional-components | tr '[:upper:]' '[:lower:]' | tr '.' ' ' || echo "") DATAPROC_IMAGE_VERSION=$(/usr/share/google/get_metadata_value attributes/dataproc_dataproc_version | cut -c1-3 | tr '-' '.' || echo "") DATAPROC_IMAGE_TYPE=$(/usr/share/google/get_metadata_value attributes/dataproc_image_type || echo "standard") @@ -281,11 +300,6 @@ function main() { echo "startup-script: BuildSucceeded: Customization complete." fi fi - - echo "startup-script: Sleep ${SHUTDOWN_TIMER_IN_SEC}s before shutting down..." - echo "You can change the timeout value with --shutdown-instance-timer-sec" - sleep "${SHUTDOWN_TIMER_IN_SEC}" # wait for stdout to flush - shutdown -h now } main "$@"