diff --git a/docs/get-started/prerequisites.mdx b/docs/get-started/prerequisites.mdx index b8b28670bde..ca69d5ba405 100644 --- a/docs/get-started/prerequisites.mdx +++ b/docs/get-started/prerequisites.mdx @@ -41,6 +41,29 @@ If the group change is not active in the current shell, the installer exits with If you choose the native Linux Ollama install path, the onboard wizard also requires `zstd` for Ollama archive extraction. The installer also requires `strings` from `binutils` to verify the OpenShell binary before it continues with OpenShell install work. +On a DGX Station GB300 running the generic Ubuntu 24.04 ARM64 image, accepting express install prepares the host with NVIDIA open driver `610.43.02`, Docker CE `29.6.1` with Buildx, and NVIDIA Container Toolkit `1.19.1`. +DGX OS, NVIDIA BaseOS images, and other Station generations are outside this automatic preparation boundary and stop before host preparation. +On those systems, set `NEMOCLAW_PROVIDER` or `NEMOCLAW_NO_EXPRESS=1` explicitly to continue without Station host automation. +The preparation probes package and runtime state first, reuses exact matches, and installs only missing pinned packages, including the NVIDIA Container Toolkit libraries and `nvidia-ctk` CLI. +It permits only the reviewed factory transition from `dkms` `3.0.11-1ubuntu13` to `1:3.4.0-1ubuntu1`. +After reboot, preparation enables NVIDIA's packaged CDI refresh path and service, requires the `nvidia.com/gpu=all` device, and verifies it with a real container launch. +If the packaged refresh fails or does not produce that device, preparation prints service diagnostics and stops for administrator repair. +It does not bypass the packaged lifecycle with direct CDI generation. +If the `docker --gpus all` acceptance probe fails, preparation registers the NVIDIA Docker runtime only when `docker info` diagnoses that runtime as absent; any other launch failure stops without changing daemon configuration. +If registration or a post-change acceptance probe fails, preparation restores the prior Docker daemon configuration; if restoration fails, it reports the backup path and stops. +After successful registration, the runtime remains configured until the same acceptance probe succeeds through a replacement Docker and NVIDIA runtime integration. +It requires Secure Boot to be disabled, matching headers for the running kernel, at least 20 GiB free on the root filesystem, and no active agent, inference, or Docker workloads. +It also stops when systemd reports a failed unit unless the unit matches an exact, condition-qualified state from the generic Station image: the pinned OEM `cloud-init` telemetry failure, a network-wait failure while current network health is established, masked `fwupd`, or an SSSD socket on a host without SSSD configuration. +Any other failed unit blocks preparation for administrator review. +It does not install a host CUDA toolkit or Docker Compose. +If any other existing prerequisite version differs, preparation stops instead of changing it automatically. +After changing pinned packages, the installer exits with status `10`; reboot, sign in, and run the printed command, which pins the exact accepted NemoClaw commit before resuming express setup. + + +DGX Station remains Deferred. +Full NemoClaw onboarding with this recipe has not completed end-to-end validation on physical DGX Station hardware. + + NemoClaw needs Docker access. On personal Linux development machines, adding your user to the `docker` group is the standard way to run Docker without sudo. diff --git a/docs/get-started/quickstart.mdx b/docs/get-started/quickstart.mdx index 1cfdb813414..5273d8ce94c 100644 --- a/docs/get-started/quickstart.mdx +++ b/docs/get-started/quickstart.mdx @@ -119,8 +119,15 @@ Use these details when your first-run path needs more control. If the installer adds you to the `docker` group, run the printed `newgrp docker` command before you rerun it. On macOS, start Docker Desktop or Colima first. - DGX Spark, DGX Station, and Windows WSL offer an interactive express-install path that chooses a managed local inference option for the platform. - On DGX Station, accepting the express prompt selects the pinned `nemotron-3-ultra-550b-a55b` managed-vLLM recipe and completes onboarding without more provider, model, policy, or sandbox-name choices. + DGX Spark, Station GB300 hosts running the generic Ubuntu 24.04 ARM64 image, and Windows WSL offer an interactive express-install path that chooses a managed local inference option for the platform. + On that Station configuration, accepting the express prompt selects the pinned `nemotron-3-ultra-550b-a55b` managed-vLLM recipe and completes onboarding without more provider, model, policy, or sandbox-name choices. + DGX OS, NVIDIA BaseOS images, and other Station generations stop before host preparation. + Set `NEMOCLAW_PROVIDER` or `NEMOCLAW_NO_EXPRESS=1` explicitly to continue on an unqualified Station without host automation. + It probes the pinned driver, Docker, Buildx, and NVIDIA Container Toolkit versions, reuses exact matches, installs missing pins, and permits only the reviewed factory `dkms` transition. + It establishes NVIDIA CDI through the packaged refresh service, then proves both CDI and `--gpus all` with real container launches. + If the packaged refresh fails or does not advertise `nvidia.com/gpu=all`, preparation prints service diagnostics and stops for administrator repair instead of generating CDI configuration directly. + After it changes pinned packages, the installer exits with status `10` at the required reboot boundary; reboot, sign in, and run the printed exact-commit command to resume the accepted recipe without another prompt. + This automation does not change Station's Deferred support status; physical end-to-end validation remains open. Pass `--station-deepseek` to use DeepSeek V4 Flash for a Station demo instead. Refer to [Platform Support](../reference/platform-support) and [Choose an Inference Provider](../inference/learn-and-choose/choose-inference-provider) for the current platform behavior. @@ -186,10 +193,19 @@ Use these details when your first-run path needs more control. curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash ``` - On DGX Spark, DGX Station, and Windows WSL, interactive installation offers express install after you accept the third-party software notice. + On DGX Spark, Station GB300 hosts running the generic Ubuntu 24.04 ARM64 image, and Windows WSL, interactive installation offers express install after you accept the third-party software notice. Express install switches onboarding to non-interactive mode, allows `sudo` password prompts for required host changes, and selects the managed local inference path for that platform. DGX Spark uses managed vLLM with `qwen3.6-35b-a3b-nvfp4` by default. DGX Station express install explicitly selects `nemotron-3-ultra-550b-a55b` instead of the Station managed-vLLM profile default, `deepseek-v4-flash`, and discloses the approximately `352 GB` model download before confirmation. + Before onboarding, the Station path requires Station GB300 with the generic Ubuntu 24.04 ARM64 image and checks for NVIDIA open driver `610.43.02`, Docker CE `29.6.1` with Buildx, and NVIDIA Container Toolkit `1.19.1`. + DGX OS, NVIDIA BaseOS images, and other Station generations are outside this automatic preparation boundary. + Set `NEMOCLAW_PROVIDER` or `NEMOCLAW_NO_EXPRESS=1` to continue without Station host automation. + Preparation reuses exact versions, installs missing pinned packages, permits only the reviewed `dkms` transition from `3.0.11-1ubuntu13` to `1:3.4.0-1ubuntu1`, and refuses every other mismatched version. + It requires the packaged NVIDIA CDI refresh service to advertise `nvidia.com/gpu=all`, and verifies CDI and `--gpus all` with real container launches. + If the packaged refresh fails or omits that device, preparation prints service diagnostics and stops for administrator repair instead of generating CDI configuration directly. + If NVIDIA Docker runtime registration or a post-change acceptance probe fails, preparation restores the prior Docker daemon configuration. + Changing pinned packages exits with status `10` for a reboot; after you sign in and run the printed exact-commit command, the accepted express recipe resumes without another prompt. + This automation does not change Station's Deferred support status; physical end-to-end validation remains open. To select DeepSeek V4 Flash while retaining the one-confirmation Station express flow, run `curl -fsSL https://www.nvidia.com/nemoclaw.sh | bash -s -- --station-deepseek`. Unless `NEMOCLAW_POLICY_TIER` is set, express install applies policy in `suggested` mode with the `balanced` tier, including the base sandbox policy and supported package, model, web-search, and local-inference presets. Express install uses `my-assistant` as the sandbox name across all platforms unless `NEMOCLAW_SANDBOX_NAME` is set. diff --git a/docs/reference/troubleshooting.mdx b/docs/reference/troubleshooting.mdx index aa13c448016..32368fe9b64 100644 --- a/docs/reference/troubleshooting.mdx +++ b/docs/reference/troubleshooting.mdx @@ -2271,7 +2271,8 @@ When shared gateway cleanup would be unsafe, follow the targeted destroy or gate Recent NVIDIA Container Toolkit installs configure the Docker daemon for Container Device Interface (CDI) device injection, which OpenShell's `gateway start --gpu` then auto-selects. If no `nvidia.com/gpu` CDI spec has been generated on the host yet, gateway start fails with `Docker responded with status code 500: CDI device injection failed: unresolvable CDI devices nvidia.com/gpu=all`. -The standard NemoClaw installer detects this gap before onboarding, first tries to enable the NVIDIA CDI refresh systemd units, and falls back to generating the spec directly with `nvidia-ctk`. +Outside Station Express, the standard NemoClaw installer detects this gap before onboarding, first tries to enable the NVIDIA CDI refresh systemd units, and can fall back to generating the spec directly with `nvidia-ctk`. +Station Express requires the packaged refresh lifecycle to work; if it fails or omits `nvidia.com/gpu=all`, inspect `nvidia-cdi-refresh.service`, repair it, and rerun the printed exact-commit install command. If you run `$$nemoclaw onboard` directly, preflight prints the manual remediation instead. The native Linux fix is the same on Docker hosts whose `docker info` advertises a non-empty `CDISpecDirs`. On WSL with Docker Desktop, Docker may advertise CDI directories even though `--device nvidia.com/gpu=all` is not usable from the WSL distro. @@ -2286,7 +2287,7 @@ nvidia-ctk cdi list $$nemoclaw onboard ``` -If the refresh units are unavailable or do not generate CDI devices, generate the spec directly: +For other native Linux installations, if the refresh units are unavailable or do not generate CDI devices, generate the spec directly: ```bash sudo mkdir -p /etc/cdi diff --git a/install.sh b/install.sh index ab867b4c891..2f705acd337 100755 --- a/install.sh +++ b/install.sh @@ -91,6 +91,10 @@ exec_installer_from_ref() { legacy_script="${source_root}/install.sh" if has_payload_marker "$payload_script"; then + # The public curl|bash boundary deliberately executes from the complete + # selected-ref checkout, not from a standalone payload file. Installer + # helpers beside scripts/install.sh (including DGX Station preparation) + # are therefore staged from the same ref before payload execution. verify_downloaded_script "$payload_script" "versioned installer" NEMOCLAW_INSTALL_REF="$ref" NEMOCLAW_INSTALL_TAG="$ref" NEMOCLAW_BOOTSTRAP_PAYLOAD=1 \ bash "$payload_script" "$@" diff --git a/scripts/checks/vitest-project-overlap.ts b/scripts/checks/vitest-project-overlap.ts index 83dfdf033e0..9e5415c7eed 100644 --- a/scripts/checks/vitest-project-overlap.ts +++ b/scripts/checks/vitest-project-overlap.ts @@ -46,6 +46,7 @@ const INSTALLER_INTEGRATION_TESTS = new Set([ "test/install-openshell-version-check.test.ts", "test/install-preflight-docker-bootstrap.test.ts", "test/install-preflight.test.ts", + "test/install-station-host-preparation.test.ts", ]); function normalizeRepoPath(file: string): string { diff --git a/scripts/install.sh b/scripts/install.sh index b169b61277b..0eb66b34948 100755 --- a/scripts/install.sh +++ b/scripts/install.sh @@ -2866,15 +2866,37 @@ detect_express_platform() { fi case "$model" in *DGX*Spark*) printf "DGX Spark" ;; - *DGX*Station* | *Station*GB300*) printf "DGX Station" ;; + *Station*GB300*) + if [ -e /etc/dgx-release ] || [ -L /etc/dgx-release ]; then + printf "Unsupported DGX Station OS" + else + printf "DGX Station" + fi + ;; + *DGX*Station*) printf "Unsupported DGX Station generation" ;; *) ;; esac } +validate_express_platform_boundary() { + case "${1:-}" in + "Unsupported DGX Station OS") + if [ "${NEMOCLAW_NO_EXPRESS:-}" = "1" ] || [ -n "${NEMOCLAW_PROVIDER:-}" ]; then return 0; fi + error "DGX OS/BaseOS is outside the validated Station express boundary. Use the generic Ubuntu 24.04 ARM64 image." + ;; + "Unsupported DGX Station generation") + if [ "${NEMOCLAW_NO_EXPRESS:-}" = "1" ] || [ -n "${NEMOCLAW_PROVIDER:-}" ]; then return 0; fi + error "This DGX Station generation is outside the validated Station GB300 express boundary." + ;; + esac +} + STATION_ULTRA_VLLM_MODEL="nemotron-3-ultra-550b-a55b" STATION_ULTRA_SERVED_MODEL="nvidia/nemotron-3-ultra-550b-a55b" STATION_DEEPSEEK_VLLM_MODEL="deepseek-v4-flash" STATION_DEEPSEEK_SERVED_MODEL="deepseek-ai/DeepSeek-V4-Flash" +_SELECTED_EXPRESS_PLATFORM="" +_STATION_EXPRESS_RESUME_REVISION="" normalize_station_vllm_model() { printf "%s" "${1:-}" | tr '[:upper:]' '[:lower:]' | sed 's/^[[:space:]]*//; s/[[:space:]]*$//' @@ -2921,6 +2943,7 @@ validate_station_deepseek_override() { preflight_explicit_express_flags() { local platform platform="$(detect_express_platform)" + validate_express_platform_boundary "$platform" validate_station_deepseek_override "$platform" } @@ -2956,6 +2979,178 @@ configure_station_express_model() { fi } +station_express_resume_file() { + local state_dir + state_dir="$(nemoclaw_state_dir)" || return 1 + printf '%s/station-express-resume' "$state_dir" +} + +validate_station_express_resume_model() { + local model="${1:-}" + [[ ${#model} -le 255 && "$model" =~ ^[A-Za-z0-9][A-Za-z0-9._/-]*$ ]] +} + +validate_station_express_resume_revision() { + [[ "${1:-}" =~ ^[0-9a-f]{40}$ ]] +} + +station_installer_revision() { + local revision + revision="$(git -C "${SCRIPT_DIR}/.." rev-parse --verify 'HEAD^{commit}' 2>/dev/null)" \ + || error "Could not resolve the exact NemoClaw revision for DGX Station reboot resume." + validate_station_express_resume_revision "$revision" \ + || error "Resolved NemoClaw revision is invalid: ${revision}" + printf '%s' "$revision" +} + +portable_file_mode() { + stat -c '%a' "$1" 2>/dev/null || stat -f '%Lp' "$1" +} + +assert_station_express_resume_file_safe() { + local state_file=$1 state_dir mode + state_dir="$(dirname "$state_file")" + [[ -d "$state_dir" && -O "$state_dir" ]] \ + || error "DGX Station express resume directory is not owned by the current user: ${state_dir}" + mode="$(portable_file_mode "$state_dir")" \ + || error "Could not inspect DGX Station express resume directory permissions: ${state_dir}" + (((8#$mode & 0077) == 0)) \ + || error "DGX Station express resume directory must not be accessible by group or other users: ${state_dir}" + [[ -f "$state_file" && -O "$state_file" ]] \ + || error "DGX Station express resume state must be a regular file owned by the current user: ${state_file}" + mode="$(portable_file_mode "$state_file")" \ + || error "Could not inspect DGX Station express resume state permissions: ${state_file}" + [[ "$mode" == "600" ]] || error "DGX Station express resume state must have mode 0600: ${state_file}" +} + +load_station_express_resume() { + local state_file revision_line model_line line_count saved_revision current_revision + state_file="$(station_express_resume_file)" || return 1 + assert_nemoclaw_state_path_safe "$state_file" + [[ -e "$state_file" || -L "$state_file" ]] || return 1 + assert_station_express_resume_file_safe "$state_file" + line_count="$(wc -l <"$state_file" | tr -d '[:space:]')" + revision_line="$(sed -n '1p' "$state_file")" + model_line="$(sed -n '2p' "$state_file")" + saved_revision="${revision_line#revision=}" + NEMOCLAW_VLLM_MODEL="${model_line#model=}" + if [[ "$line_count" != "2" || "$revision_line" != "revision=${saved_revision}" || "$model_line" != "model=${NEMOCLAW_VLLM_MODEL}" ]] \ + || ! validate_station_express_resume_revision "$saved_revision" \ + || ! validate_station_express_resume_model "$NEMOCLAW_VLLM_MODEL"; then + error "DGX Station express resume state is invalid. Remove ${state_file} and rerun the installer." + fi + current_revision="$(station_installer_revision)" + if [[ "$current_revision" != "$saved_revision" ]]; then + error "DGX Station express resume requires NemoClaw revision ${saved_revision}, but this installer is ${current_revision}. Rerun with: curl -fsSL https://www.nvidia.com/nemoclaw.sh | NEMOCLAW_INSTALL_TAG=${saved_revision} bash" + fi + export NEMOCLAW_VLLM_MODEL +} + +save_station_express_resume() { + local state_file state_dir temp_file revision model="${NEMOCLAW_VLLM_MODEL:-}" + validate_station_express_resume_model "$model" || error "Cannot save an invalid DGX Station express model selector." + revision="$(station_installer_revision)" + state_file="$(station_express_resume_file)" || error "Could not resolve NemoClaw state for DGX Station express resume." + state_dir="$(ensure_nemoclaw_state_dir)" || error "Could not prepare NemoClaw state for DGX Station express resume." + assert_nemoclaw_state_path_safe "$state_file" + temp_file="$(mktemp "${state_file}.tmp.XXXXXX")" || error "Could not create DGX Station express resume state under ${state_dir}." + chmod 600 "$temp_file" || { + rm -f "$temp_file" + error "Could not secure DGX Station express resume state under ${state_dir}." + } + if ! printf 'revision=%s\nmodel=%s\n' "$revision" "$model" >"$temp_file"; then + rm -f "$temp_file" + error "Could not write DGX Station express resume state under ${state_dir}." + fi + if ! mv -f "$temp_file" "$state_file"; then + rm -f "$temp_file" + error "Could not publish DGX Station express resume state under ${state_dir}." + fi + assert_station_express_resume_file_safe "$state_file" + _STATION_EXPRESS_RESUME_REVISION="$revision" +} + +clear_station_express_resume() { + local state_file + state_file="$(station_express_resume_file)" || return 0 + assert_nemoclaw_state_path_safe "$state_file" + [[ -e "$state_file" || -L "$state_file" ]] || return 0 + [[ -f "$state_file" && -O "$state_file" ]] \ + || error "Refusing to remove invalid DGX Station express resume state: ${state_file}" + rm -f "$state_file" +} + +activate_express_install() { + local platform="$1" + _SELECTED_EXPRESS_PLATFORM="$platform" + NON_INTERACTIVE=1 + export NEMOCLAW_NON_INTERACTIVE=1 + export NEMOCLAW_NON_INTERACTIVE_SUDO_MODE=prompt + export NEMOCLAW_YES=1 + export NEMOCLAW_POLICY_MODE=suggested + case "$platform" in + "DGX Spark") + export NEMOCLAW_SANDBOX_NAME="${NEMOCLAW_SANDBOX_NAME:-my-assistant}" + export NEMOCLAW_PROVIDER=install-vllm + if [ -n "${NEMOCLAW_VLLM_MODEL:-}" ]; then + export NEMOCLAW_VLLM_MODEL + fi + ;; + "DGX Station") + export NEMOCLAW_SANDBOX_NAME="${NEMOCLAW_SANDBOX_NAME:-my-assistant}" + export NEMOCLAW_PROVIDER=install-vllm + configure_station_express_model + ;; + "Windows WSL") + export NEMOCLAW_PROVIDER=install-windows-ollama + ;; + esac +} + +run_station_host_preparation() { + # Public curl|bash starts in the root bootstrap, which clones the complete + # selected ref before executing this payload. Keep the sibling lookup and + # fail-closed check so Station preparation cannot drift from that ref. + local helper="${SCRIPT_DIR}/prepare-dgx-station-host.sh" + [[ -f "$helper" ]] || error "DGX Station host preparation helper is missing: ${helper}" + bash "$helper" --apply +} + +ensure_station_express_host() { + [[ "${_SELECTED_EXPRESS_PLATFORM:-}" == "DGX Station" ]] || return 0 + + info "Checking pinned DGX Station host prerequisites. Exact matches are reused." + local status=0 + run_station_host_preparation || status=$? + case "$status" in + 0) + ok "DGX Station host prerequisites are ready" + ;; + 10) + save_station_express_resume + local revision + revision="${_STATION_EXPRESS_RESUME_REVISION}" + warn "DGX Station host prerequisites were installed and require a reboot." + info "Run: sudo reboot" + info "After signing in again, rerun the accepted revision:" + info "curl -fsSL https://www.nvidia.com/nemoclaw.sh | NEMOCLAW_INSTALL_TAG=${revision} bash" + exit 10 + ;; + *) + error "DGX Station host preparation failed. Review the station-bootstrap log above, correct the reported host state, and rerun the installer." + ;; + esac +} + +prepare_installer_host() { + maybe_offer_express_install + # Intentional ordering: Station preparation owns the reboot boundary before + # generic Docker bootstrap; ensure_station_express_host is a no-op elsewhere. + ensure_station_express_host + ensure_docker + ensure_openshell_build_deps +} + # Prompt the user to opt into express install on supported platforms. Sets the # non-interactive + provider/model env vars when accepted. Skipped when # the user already passed --non-interactive, set NEMOCLAW_PROVIDER, or has @@ -2989,6 +3184,9 @@ describe_express_install() { inference_summary="managed local vLLM with NVIDIA Nemotron 3 Ultra 550B" inference_disclosure="Managed vLLM pulls the pinned Station image and approximately 352 GB model, then runs a local inference container." fi + printf " Station host setup reuses exact prerequisite versions, applies the reviewed factory DKMS transition when present, installs missing pinned driver, Docker, and NVIDIA Container Toolkit packages, and may require one reboot.\n" + printf " Host setup may add this trusted local account to the docker group, which grants root-equivalent control. This flow is only for trusted single-user development hosts; shared or managed hosts require an organization-approved Docker access path.\n" + printf " DGX Station remains Deferred; this recipe has not completed end-to-end validation on physical hardware.\n" sandbox_summary="${NEMOCLAW_SANDBOX_NAME:-my-assistant}" ;; "Windows WSL") @@ -3031,6 +3229,7 @@ describe_express_install() { maybe_offer_express_install() { local platform platform="$(detect_express_platform)" + validate_express_platform_boundary "$platform" validate_station_deepseek_override "$platform" # Not on a platform we have an express recipe for — say nothing. if [ -z "$platform" ]; then @@ -3039,17 +3238,24 @@ maybe_offer_express_install() { # On a supported platform but a skip condition applies — explain why so # the user understands they could have gotten express otherwise. if [ "${NEMOCLAW_NO_EXPRESS:-}" = "1" ]; then + if [ "$platform" = "DGX Station" ]; then clear_station_express_resume; fi info "Detected ${platform}. Skipping express prompt (NEMOCLAW_NO_EXPRESS=1)." return 0 fi - if [ "${NON_INTERACTIVE:-}" = "1" ]; then - info "Detected ${platform}. Skipping express prompt (--non-interactive set)." - return 0 - fi if [ -n "${NEMOCLAW_PROVIDER:-}" ]; then + if [ "$platform" = "DGX Station" ]; then clear_station_express_resume; fi info "Detected ${platform}. Skipping express prompt (NEMOCLAW_PROVIDER=${NEMOCLAW_PROVIDER} already set)." return 0 fi + if [ "$platform" = "DGX Station" ] && load_station_express_resume; then + info "Detected DGX Station. Resuming the accepted express install after host preparation." + activate_express_install "$platform" + return 0 + fi + if [ "${NON_INTERACTIVE:-}" = "1" ]; then + info "Detected ${platform}. Skipping express prompt (--non-interactive set)." + return 0 + fi local reply="" if [ -t 0 ]; then info "Detected ${platform}." @@ -3077,28 +3283,7 @@ maybe_offer_express_install() { case "$reply" in "" | y | yes) info "Using express install for ${platform}." - NON_INTERACTIVE=1 - export NEMOCLAW_NON_INTERACTIVE=1 - export NEMOCLAW_NON_INTERACTIVE_SUDO_MODE=prompt - export NEMOCLAW_YES=1 - export NEMOCLAW_POLICY_MODE=suggested - case "$platform" in - "DGX Spark") - export NEMOCLAW_SANDBOX_NAME="${NEMOCLAW_SANDBOX_NAME:-my-assistant}" - export NEMOCLAW_PROVIDER=install-vllm - if [ -n "${NEMOCLAW_VLLM_MODEL:-}" ]; then - export NEMOCLAW_VLLM_MODEL - fi - ;; - "DGX Station") - export NEMOCLAW_SANDBOX_NAME="${NEMOCLAW_SANDBOX_NAME:-my-assistant}" - export NEMOCLAW_PROVIDER=install-vllm - configure_station_express_model - ;; - "Windows WSL") - export NEMOCLAW_PROVIDER=install-windows-ollama - ;; - esac + activate_express_install "$platform" ;; *) info "Skipping express install. Continuing with interactive flow." @@ -3196,15 +3381,13 @@ main() { # still collect acceptance before Node.js or the CLI are installed. preflight_usage_notice_prompt - ensure_docker - ensure_openshell_build_deps - # Offer express install on supported platforms (DGX Spark / Station / WSL). # Runs AFTER the third-party notice so the user has explicitly accepted the # license before opting into the unattended path. Express only sets the # provider/model/policy + non-interactive vars; license acceptance is - # already recorded by preflight above. - maybe_offer_express_install + # already recorded by preflight above. Station selection runs its pinned + # host prerequisite preparation before the generic Docker bootstrap. + prepare_installer_host _INSTALL_START=$SECONDS bash "${SCRIPT_DIR}/setup-jetson.sh" @@ -3273,6 +3456,9 @@ main() { fi finalize_install + if [[ "${_SELECTED_EXPRESS_PLATFORM:-}" == "DGX Station" ]]; then + clear_station_express_resume + fi } # Print the completion summary, then propagate a fatal/non-zero result when the diff --git a/scripts/prepare-dgx-station-host.sh b/scripts/prepare-dgx-station-host.sh new file mode 100755 index 00000000000..ae14f99bc70 --- /dev/null +++ b/scripts/prepare-dgx-station-host.sh @@ -0,0 +1,982 @@ +#!/usr/bin/env bash +# SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +# SPDX-License-Identifier: Apache-2.0 + +set -Eeuo pipefail +umask 077 + +readonly SCRIPT_VERSION="2026-07-16.5" +readonly REBOOT_REQUIRED_EXIT=10 +readonly MIN_FREE_KIB=$((20 * 1024 * 1024)) +# The qualified generic image currently ships this OEM telemetry bootcmd. Its +# exception disappears automatically when the file changes or the bootcmd +# failure is fixed; update the pin only with a newly audited image. +readonly FACTORY_CLOUD_INIT_TELEMETRY="/etc/cloud/telemetry-bootcmd-event.py" +readonly FACTORY_CLOUD_INIT_RESULT="/run/cloud-init/result.json" +readonly FACTORY_CLOUD_INIT_TELEMETRY_SHA256="09a526c73fcbbe238db56f0ba4ce90a5a0634bab14b5122b016089d581f07275" + +readonly CUDA_KEYRING_URL="https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/sbsa/cuda-keyring_1.1-1_all.deb" +readonly CUDA_KEYRING_SHA256="6ea7d2737648936820e85677177957a0f6521b840d98eb0bbae0a4f003fa7249" +readonly CUDA_KEYRING_PACKAGE_VERSION="1.1-1" +readonly CUDA_KEY_FINGERPRINT="EB693B3035CD5710E231E123A4B469963BF863CC" # gitleaks:allow -- public NVIDIA signing-key fingerprint +readonly DOCKER_KEY_URL="https://download.docker.com/linux/ubuntu/gpg" +readonly DOCKER_KEY_SHA256="1500c1f56fa9e26b9b8f42452a553675796ade0807cdce11975eb98170b3a570" # gitleaks:allow -- public Docker GPG-key integrity pin +readonly DOCKER_KEY_FINGERPRINT="9DC858229FC7DD38854AE2D88D81803C0EBFCD88" + +readonly DRIVER_VERSION="610.43.02" +readonly DOCKER_VERSION="29.6.1" +readonly TOOLKIT_VERSION="1.19.1" +readonly FACTORY_DKMS_VERSION="3.0.11-1ubuntu13" +readonly TARGET_DKMS_VERSION="1:3.4.0-1ubuntu1" +# Keep this as a plain Ubuntu image: NVIDIA Container Toolkit injects the host +# driver utility when CDI or --gpus is requested. This intentionally exercises +# the documented runtime contract instead of relying on a CUDA image payload: +# https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/latest/sample-workload.html +readonly ACCEPTANCE_IMAGE="docker.io/library/ubuntu@sha256:7f622ca8766bccb22f04242ecb6f19f770b2f08827dc4b8c707de5e78a6da7ab" +readonly STATE_DIR="${HOME}/.local/state/station-bootstrap" +readonly INSTALL_BOOT_MARKER="${STATE_DIR}/install-boot-id" + +readonly -a PACKAGE_SPECS=( + "dkms=${TARGET_DKMS_VERSION}" + "nvidia-driver-pinning-610=610-2ubuntu1" + "nvidia-driver-open=610.43.02-1ubuntu1" + "containerd.io=2.2.6-1~ubuntu.24.04~noble" + "docker-buildx-plugin=0.35.0-1~ubuntu.24.04~noble" + "docker-ce=5:29.6.1-1~ubuntu.24.04~noble" + "docker-ce-cli=5:29.6.1-1~ubuntu.24.04~noble" + "libnvidia-container-tools=1.19.1-1" + "libnvidia-container1=1.19.1-1" + "nvidia-container-toolkit=1.19.1-1" + "nvidia-container-toolkit-base=1.19.1-1" +) + +MODE="" +LOG_FILE="" +DOCKER_GROUP_ADDED=0 +CDI_LIFECYCLE_READY=0 +NETWORK_VALIDATED=0 + +info() { + printf '[station-prepare] %s %s\n' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" "$*" +} + +warn() { + printf '[station-prepare] %s WARNING: %s\n' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" "$*" >&2 +} + +fatal() { + printf '[station-prepare] %s ERROR: %s\n' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" "$*" >&2 + exit 1 +} + +on_error() { + local rc=$? + local line=${1:-unknown} + printf '[station-prepare] %s ERROR: command failed at line %s (exit %s)\n' \ + "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" "$line" "$rc" >&2 + exit "$rc" +} + +usage() { + cat <<'EOF' +Usage: prepare-dgx-station-host.sh --check|--apply|--verify + + --check Read-only eligibility and current-state report. + --apply Install exact prerequisites or finish post-reboot runtime setup. + --verify Read-only host verification plus ephemeral GPU container tests. + +Exit 10 from --apply means an operator-controlled reboot is required. After +the reboot, run --apply once more, followed by --verify. +EOF +} + +is_valid_mode() { + case "${1:-}" in + --check | --apply | --verify) return 0 ;; + *) return 1 ;; + esac +} + +is_station_product() { + local product=${1:-} + [[ "$product" == *"Station"* && "$product" == *"GB300"* ]] +} + +is_preparation_critical_unit() { + case "${1:-}" in + containerd.service | docker.service | nvidia-cdi-refresh.service | nvidia-persistenced.service) + return 0 + ;; + *) return 1 ;; + esac +} + +is_driver_transitional_unit() { + [[ "${1:-}" == "nvidia-persistenced.service" ]] +} + +root_owned_file_is_not_writable_by_group_or_other() { + local metadata kind uid gid mode + metadata="$(stat -Lc '%F|%u|%g|%a' "$1" 2>/dev/null)" || return 1 + IFS='|' read -r kind uid gid mode <<<"$metadata" + [[ "$kind" == "regular file" && "$uid" == "0" && "$gid" == "0" && "$mode" =~ ^[0-7]{3,4}$ ]] \ + || return 1 + (((8#$mode & 0022) == 0)) +} + +cloud_init_failure_is_qualified() { + local actual_sha + ((NETWORK_VALIDATED == 1)) || return 1 + root_owned_file_is_not_writable_by_group_or_other "$FACTORY_CLOUD_INIT_TELEMETRY" \ + || return 1 + root_owned_file_is_not_writable_by_group_or_other "$FACTORY_CLOUD_INIT_RESULT" || return 1 + actual_sha="$(sha256sum "$FACTORY_CLOUD_INIT_TELEMETRY" 2>/dev/null | awk '{print $1}')" + [[ "$actual_sha" == "$FACTORY_CLOUD_INIT_TELEMETRY_SHA256" ]] || return 1 + grep -Fq "\"('bootcmd', ProcessExecutionError(" "$FACTORY_CLOUD_INIT_RESULT" +} + +network_wait_failure_is_qualified() { + ((NETWORK_VALIDATED == 1)) \ + && systemctl is-active --quiet NetworkManager.service \ + && systemctl is-active --quiet network-online.target +} + +fwupd_refresh_failure_is_qualified() { + local state + ((NETWORK_VALIDATED == 1)) || return 1 + state="$(systemctl is-enabled fwupd.service 2>/dev/null)" || true + [[ "$state" == "masked" ]] +} + +sssd_socket_failure_is_qualified() { + [[ ! -e /etc/sssd/sssd.conf && ! -L /etc/sssd/sssd.conf ]] +} + +is_qualified_factory_failed_unit() { + case "${1:-}" in + cloud-init.service) cloud_init_failure_is_qualified ;; + NetworkManager-wait-online.service | systemd-networkd-wait-online.service) + network_wait_failure_is_qualified + ;; + fwupd-refresh.service) fwupd_refresh_failure_is_qualified ;; + sssd-autofs.socket | sssd-nss.socket | sssd-pam.socket | sssd-pam-priv.socket) + sssd_socket_failure_is_qualified + ;; + *) return 1 ;; + esac +} + +package_name() { + printf '%s\n' "${1%%=*}" +} + +package_expected_version() { + printf '%s\n' "${1#*=}" +} + +acquire_sudo() { + if sudo -n true >/dev/null 2>&1; then + info "sudo=noninteractive" + return + fi + + info "sudo=interactive_authentication_required" + sudo -v +} + +installed_version() { + dpkg-query -W -f='${Version}' "$1" 2>/dev/null || true +} + +package_is_exact() { + local spec=$1 + local name expected actual + name="$(package_name "$spec")" + expected="$(package_expected_version "$spec")" + actual="$(installed_version "$name")" + [[ "$actual" == "$expected" ]] +} + +package_state() { + local spec=$1 + local name expected actual + name="$(package_name "$spec")" + expected="$(package_expected_version "$spec")" + actual="$(installed_version "$name")" + if [[ -z "$actual" ]]; then + printf 'missing\n' + elif [[ "$actual" == "$expected" ]]; then + printf 'exact\n' + elif [[ "$name" == "dkms" && "$actual" == "$FACTORY_DKMS_VERSION" && "$expected" == "$TARGET_DKMS_VERSION" ]]; then + printf 'approved-transition\n' + else + printf 'mismatch\n' + fi +} + +assert_no_package_mismatches() { + local spec state name expected actual mismatch=0 + for spec in "${PACKAGE_SPECS[@]}"; do + state="$(package_state "$spec")" + if [[ "$state" == "approved-transition" ]]; then + name="$(package_name "$spec")" + expected="$(package_expected_version "$spec")" + actual="$(installed_version "$name")" + info "package=${name} status=approved_transition actual=${actual} expected=${expected}" + continue + fi + [[ "$state" == "mismatch" ]] || continue + name="$(package_name "$spec")" + expected="$(package_expected_version "$spec")" + actual="$(installed_version "$name")" + warn "package=${name} status=mismatch actual=${actual} expected=${expected}" + mismatch=1 + done + ((mismatch == 0)) || fatal "Existing Station prerequisite versions differ from the validated pins or approved factory transition; refusing to change them automatically" +} + +all_packages_exact() { + local spec + for spec in "${PACKAGE_SPECS[@]}"; do + package_is_exact "$spec" || return 1 + done + return 0 +} + +setup_log() { + local log_dir="${HOME}/station-bootstrap-logs" + mkdir -p "$log_dir" + chmod 0700 "$log_dir" + LOG_FILE="${log_dir}/station-prepare-${MODE#--}-$(date -u '+%Y%m%dT%H%M%SZ').log" + exec > >(tee -a "$LOG_FILE") 2>&1 + info "version=${SCRIPT_VERSION} mode=${MODE} log=${LOG_FILE}" +} + +require_command() { + command -v "$1" >/dev/null 2>&1 || fatal "Required command is missing: $1" +} + +file_mode() { + stat -c '%a' "$1" 2>/dev/null || stat -f '%Lp' "$1" +} + +check_platform() { + local arch product + arch="$(uname -m)" + [[ "$arch" == "aarch64" || "$arch" == "arm64" ]] || fatal "Expected ARM64, found ${arch}" + + [[ -r /etc/os-release ]] || fatal "/etc/os-release is unavailable" + # shellcheck disable=SC1091 + source /etc/os-release + [[ "${ID:-}" == "ubuntu" && "${VERSION_ID:-}" == "24.04" ]] \ + || fatal "Expected Ubuntu 24.04, found ${PRETTY_NAME:-unknown}" + [[ ! -e /etc/dgx-release && ! -L /etc/dgx-release ]] \ + || fatal "DGX OS/BaseOS is outside this recipe's validated boundary; use the generic Ubuntu 24.04 ARM64 image" + + product="$(= MIN_FREE_KIB)) || fatal "At least 20 GiB free is required; found $((available / 1024 / 1024)) GiB" + info "root_free_gib=$((available / 1024 / 1024))" +} + +check_network() { + local host + for host in developer.download.nvidia.com download.docker.com registry-1.docker.io; do + getent ahosts "$host" >/dev/null 2>&1 || fatal "DNS resolution failed for ${host}" + done + NETWORK_VALIDATED=1 + info "network=required_vendor_hosts_resolve" +} + +check_package_managers_idle() { + local active + active="$(ps -eo pid=,comm= | awk '$2 ~ /^(apt|apt-get|dpkg|unattended-upgrade)$/ {print}')" + [[ -z "$active" ]] || fatal "A package-manager process is active: ${active}" + info "package_manager=idle" +} + +check_failed_units() { + local unit failed_output blocking=0 + local -a units=() + failed_output="$(systemctl --failed --no-legend --plain 2>/dev/null)" \ + || fatal "Unable to inspect failed system services" + while IFS= read -r unit; do + [[ -n "$unit" ]] && units+=("$unit") + done < <(awk 'NF {print $1}' <<<"$failed_output") + if ((${#units[@]} == 0)); then + info "failed_units=none" + return 0 + fi + for unit in "${units[@]}"; do + if is_driver_transitional_unit "$unit" && all_packages_exact && ! driver_loaded_exact; then + warn "driver unit failure allowed only until post-reboot verification: ${unit}" + elif is_preparation_critical_unit "$unit"; then + warn "failed preparation-critical unit: ${unit}" + blocking=1 + elif is_qualified_factory_failed_unit "$unit"; then + warn "condition-qualified generic-image failed unit: ${unit}" + else + warn "unqualified failed unit: ${unit}" + blocking=1 + fi + done + ((blocking == 0)) || fatal "Unqualified failed system units block Station preparation" +} + +check_no_workloads() { + local processes matches listeners containers="" + processes="$(ps -eo pid=,ppid=,comm=,args=)" + matches="$(awk -v self="$$" -v parent="$PPID" ' + { + pid=$1 + ppid=$2 + comm=tolower($3) + $1=$2=$3="" + args=tolower($0) + if (pid == self || pid == parent) next + if (comm ~ /^(vllm|nemoclaw|openshell)$/ || + args ~ /(^|[[:space:]\/])(vllm|nemoclaw|openshell)([[:space:]:]|\.js([[:space:]]|$)|$)/) print + } + ' <<<"$processes")" + [[ -z "$matches" ]] || fatal "Agent or inference workload is active: ${matches}" + + listeners="$(ss -H -ltn 2>/dev/null | awk '$4 ~ /:8000$/ {print}')" + [[ -z "$listeners" ]] || fatal "Port 8000 is already listening: ${listeners}" + + if command -v docker >/dev/null 2>&1; then + if containers="$(docker ps -aq 2>/dev/null)"; then + : + elif [[ "$MODE" == "--apply" ]] && containers="$(sudo -n docker ps -aq 2>/dev/null)"; then + info "docker_access=sudo_until_group_membership_is_active" + elif systemctl is-active --quiet docker.service; then + fatal "Docker is active but inaccessible to this login; start a new login session with docker-group membership" + else + fatal "Docker is installed but inactive, so existing container state cannot be verified safely; start Docker and rerun preparation" + fi + fi + [[ -z "$containers" ]] || fatal "Existing Docker containers block host preparation: ${containers}" + info "workloads=none port_8000=free" +} + +driver_loaded_exact() { + local loaded + command -v nvidia-smi >/dev/null 2>&1 || return 1 + loaded="$(nvidia-smi --query-gpu=driver_version --format=csv,noheader 2>/dev/null | head -n1 | tr -d '[:space:]')" + [[ "$loaded" == "$DRIVER_VERSION" ]] +} + +assert_station_state_dir_safe() { + local path mode + for path in "${HOME}/.local" "${HOME}/.local/state" "$STATE_DIR"; do + [[ ! -L "$path" ]] || fatal "Refusing symbolic link in Station bootstrap state path: ${path}" + [[ ! -e "$path" || -d "$path" ]] || fatal "Station bootstrap state path is not a directory: ${path}" + if [[ -e "$path" ]]; then + [[ -O "$path" ]] || fatal "Station bootstrap state path is not owned by the current user: ${path}" + mode="$(file_mode "$path")" + (((8#$mode & 0022) == 0)) || fatal "Station bootstrap state path is group- or other-writable: ${path}" + fi + done +} + +assert_install_boot_marker_safe() { + local mode + [[ ! -L "$INSTALL_BOOT_MARKER" ]] || fatal "Refusing symbolic link for Station bootstrap boot marker: ${INSTALL_BOOT_MARKER}" + [[ ! -e "$INSTALL_BOOT_MARKER" || -f "$INSTALL_BOOT_MARKER" ]] \ + || fatal "Station bootstrap boot marker is not a regular file: ${INSTALL_BOOT_MARKER}" + if [[ -e "$INSTALL_BOOT_MARKER" ]]; then + [[ -O "$INSTALL_BOOT_MARKER" ]] || fatal "Station bootstrap boot marker is not owned by the current user" + mode="$(file_mode "$INSTALL_BOOT_MARKER")" + [[ "$mode" == "600" ]] || fatal "Station bootstrap boot marker must have mode 0600" + fi +} + +write_install_boot_marker() { + local temp_file + assert_station_state_dir_safe + mkdir -p "$STATE_DIR" + assert_station_state_dir_safe + chmod 0700 "$STATE_DIR" + assert_install_boot_marker_safe + temp_file="$(mktemp "${INSTALL_BOOT_MARKER}.tmp.XXXXXX")" + chmod 0600 "$temp_file" + tr -d '[:space:]' "$temp_file" + printf '\n' >>"$temp_file" + mv -f "$temp_file" "$INSTALL_BOOT_MARKER" + assert_install_boot_marker_safe +} + +install_boot_marker_matches_current_boot() { + local installed_boot current_boot + assert_station_state_dir_safe + [[ -e "$INSTALL_BOOT_MARKER" || -L "$INSTALL_BOOT_MARKER" ]] || return 1 + assert_install_boot_marker_safe + installed_boot="$(tr -d '[:space:]' <"$INSTALL_BOOT_MARKER")" + current_boot="$(tr -d '[:space:]' /dev/null \ + | awk -F: '$1 == "fpr" {print $10}' \ + | grep -Fxq "$expected" || fatal "Expected signing-key fingerprint ${expected} was not found in ${path}" +} + +root_directory_is_safe() { + local path=$1 metadata uid gid mode + sudo test ! -L "$path" || return 1 + sudo test -d "$path" || return 1 + metadata="$(sudo stat -c '%u %g %a' -- "$path")" || return 1 + read -r uid gid mode <<<"$metadata" + [[ "$uid" == "0" && "$gid" == "0" && "$mode" =~ ^[0-7]{3,4}$ ]] || return 1 + (((8#$mode & 0022) == 0)) +} + +assert_root_directory_safe() { + local path=$1 label=$2 + root_directory_is_safe "$path" \ + || fatal "${label} must be a root-owned directory that is not group- or other-writable: ${path}" +} + +ensure_root_directory_safe() { + local path=$1 parent=$2 mode=$3 label=$4 + assert_root_directory_safe "$parent" "${label} parent" + sudo test ! -L "$path" || fatal "${label} must not be a symbolic link: ${path}" + if ! sudo test -e "$path"; then + sudo install -d -o root -g root -m "$mode" "$path" + fi + assert_root_directory_safe "$path" "$label" +} + +root_regular_file_is_safe() { + local path=$1 expected_mode=${2:-} metadata uid gid mode + sudo test ! -L "$path" || return 1 + sudo test -f "$path" || return 1 + metadata="$(sudo stat -c '%u %g %a' -- "$path")" || return 1 + read -r uid gid mode <<<"$metadata" + [[ "$uid" == "0" && "$gid" == "0" && "$mode" =~ ^[0-7]{3,4}$ ]] || return 1 + (((8#$mode & 0022) == 0)) || return 1 + [[ -z "$expected_mode" || "$mode" == "${expected_mode#0}" ]] +} + +assert_root_regular_file_safe() { + local path=$1 expected_mode=$2 label=$3 + if [[ -n "$expected_mode" ]]; then + root_regular_file_is_safe "$path" "$expected_mode" \ + || fatal "${label} must be a root-owned regular file with mode ${expected_mode}: ${path}" + else + root_regular_file_is_safe "$path" "" \ + || fatal "${label} must be a root-owned regular file that is not group- or other-writable: ${path}" + fi +} + +ensure_cuda_keyring() { + local cuda_deb=$1 actual verification + assert_root_directory_safe /usr/share/keyrings "CUDA repository keyring directory" + actual="$(installed_version cuda-keyring)" + if [[ -z "$actual" ]]; then + curl --fail --silent --show-error --location "$CUDA_KEYRING_URL" --output "$cuda_deb" + verify_file_sha256 "$cuda_deb" "$CUDA_KEYRING_SHA256" + sudo dpkg -i "$cuda_deb" + package_is_exact "cuda-keyring=${CUDA_KEYRING_PACKAGE_VERSION}" \ + || fatal "Installed cuda-keyring does not match ${CUDA_KEYRING_PACKAGE_VERSION}" + elif [[ "$actual" == "$CUDA_KEYRING_PACKAGE_VERSION" ]]; then + verification="$(dpkg -V cuda-keyring 2>&1)" \ + || fatal "Unable to verify the installed cuda-keyring package" + [[ -z "$verification" ]] || fatal "Installed cuda-keyring files differ from the package manifest: ${verification}" + info "cuda_keyring=exact version=${actual}" + else + fatal "Existing cuda-keyring version ${actual} differs from validated pin ${CUDA_KEYRING_PACKAGE_VERSION}; refusing to upgrade or downgrade it automatically" + fi + + assert_root_regular_file_safe /usr/share/keyrings/cuda-archive-keyring.gpg 0644 "CUDA repository keyring" + verify_key_fingerprint /usr/share/keyrings/cuda-archive-keyring.gpg "$CUDA_KEY_FINGERPRINT" +} + +install_exact_file_or_reuse() { + local source=$1 target=$2 mode=$3 label=$4 parent + parent="$(dirname "$target")" + assert_root_directory_safe "$parent" "${label} directory" + sudo test ! -L "$target" || fatal "${label} must not be a symbolic link: ${target}" + if sudo test -e "$target"; then + assert_root_regular_file_safe "$target" "$mode" "$label" + sudo cmp -s "$source" "$target" \ + || fatal "Existing ${label} differs from the validated content; refusing to overwrite ${target}" + info "${label}=exact path=${target}" + return 0 + fi + sudo install -o root -g root -m "$mode" "$source" "$target" + assert_root_regular_file_safe "$target" "$mode" "$label" + info "${label}=installed path=${target}" +} + +configure_repositories() { + local tmp cuda_deb docker_asc docker_gpg docker_list + tmp="$(mktemp -d)" + cuda_deb="${tmp}/cuda-keyring.deb" + docker_asc="${tmp}/docker.asc" + docker_gpg="${tmp}/docker.gpg" + docker_list="${tmp}/docker.list" + + info "Downloading and verifying official repository keys" + ensure_cuda_keyring "$cuda_deb" + + curl --fail --silent --show-error --location "$DOCKER_KEY_URL" --output "$docker_asc" + verify_file_sha256 "$docker_asc" "$DOCKER_KEY_SHA256" + verify_key_fingerprint "$docker_asc" "$DOCKER_KEY_FINGERPRINT" + gpg --batch --yes --dearmor --output "$docker_gpg" "$docker_asc" + ensure_root_directory_safe /etc/apt/keyrings /etc/apt 0755 "Docker repository key directory" + assert_root_directory_safe /etc/apt/sources.list.d "Docker repository source directory" + install_exact_file_or_reuse "$docker_gpg" /etc/apt/keyrings/docker.gpg 0644 docker_repository_key + printf '%s\n' \ + 'deb [arch=arm64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu noble stable' \ + >"$docker_list" + install_exact_file_or_reuse "$docker_list" /etc/apt/sources.list.d/docker.list 0644 docker_repository_source + + rm -rf "$tmp" + info "repository_keys=verified" +} + +validate_package_availability() { + local spec + for spec in "${PACKAGE_SPECS[@]}"; do + apt-cache show "$spec" >/dev/null 2>&1 || fatal "Exact package version is unavailable: ${spec}" + done + info "exact_package_versions=available" +} + +simulate_install() { + local simulation + simulation="$(apt-get -s install --no-install-recommends "${PACKAGE_SPECS[@]}")" \ + || fatal "APT simulation failed" + printf '%s\n' "$simulation" + if grep -Eq '^(Remv |Purg )' <<<"$simulation"; then + fatal "APT simulation proposed a package removal" + fi + info "apt_simulation=no_removals" +} + +install_packages() { + configure_repositories + info "Refreshing package metadata" + sudo apt-get update + validate_package_availability + simulate_install + check_no_workloads + info "Installing pinned Station prerequisites" + sudo env DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \ + "${PACKAGE_SPECS[@]}" + + local spec + for spec in "${PACKAGE_SPECS[@]}"; do + package_is_exact "$spec" || fatal "Installed package does not match ${spec}" + done + info "pinned_packages=installed" +} + +ensure_docker_group() { + local user_name=${SUDO_USER:-$USER} + getent group docker >/dev/null 2>&1 || fatal "Docker group is missing after package installation" + if ! id -nG "$user_name" | tr ' ' '\n' | grep -Fxq docker; then + sudo usermod -aG docker "$user_name" + DOCKER_GROUP_ADDED=1 + info "docker_group=added user=${user_name}; a new login is required" + else + info "docker_group=present user=${user_name}" + fi +} + +ensure_cdi_refresh_lifecycle() { + ((CDI_LIFECYCLE_READY == 0)) || return 0 + check_no_workloads + sudo systemctl enable nvidia-cdi-refresh.path nvidia-cdi-refresh.service \ + || fatal "Could not enable the packaged NVIDIA CDI refresh lifecycle" + sudo systemctl start nvidia-cdi-refresh.path \ + || fatal "Could not activate the packaged NVIDIA CDI refresh path" + CDI_LIFECYCLE_READY=1 + info "cdi_refresh_lifecycle=enabled" +} + +verify_cdi_refresh_lifecycle() { + systemctl is-enabled --quiet nvidia-cdi-refresh.path \ + || fatal "nvidia-cdi-refresh.path is not enabled" + systemctl is-enabled --quiet nvidia-cdi-refresh.service \ + || fatal "nvidia-cdi-refresh.service is not enabled" + systemctl is-active --quiet nvidia-cdi-refresh.path \ + || fatal "nvidia-cdi-refresh.path is not active" + info "cdi_refresh_lifecycle=verified" +} + +refresh_cdi() { + check_no_workloads + ensure_cdi_refresh_lifecycle + if ! sudo systemctl restart nvidia-cdi-refresh.service; then + warn "Packaged CDI refresh failed; collecting diagnostics" + sudo systemctl status nvidia-cdi-refresh.service --no-pager || true + sudo journalctl -u nvidia-cdi-refresh.service --no-pager -n 50 || true + fatal "Packaged CDI refresh failed; repair nvidia-cdi-refresh.service before rerunning preparation" + fi + if ! nvidia-ctk cdi list | grep -Fxq 'nvidia.com/gpu=all'; then + warn "Packaged CDI refresh completed without advertising nvidia.com/gpu=all" + sudo systemctl status nvidia-cdi-refresh.service --no-pager || true + sudo journalctl -u nvidia-cdi-refresh.service --no-pager -n 50 || true + fatal "Packaged CDI refresh did not advertise nvidia.com/gpu=all; direct CDI generation is not permitted" + fi + info "cdi=nvidia.com/gpu=all source=packaged_refresh_service" +} + +ensure_acceptance_image() { + if ! sudo docker image inspect "$ACCEPTANCE_IMAGE" >/dev/null 2>&1; then + info "Pulling digest-pinned ARM64 acceptance image" + sudo docker pull --platform linux/arm64 "$ACCEPTANCE_IMAGE" + fi +} + +run_cdi_test_sudo() { + sudo docker run --rm --device nvidia.com/gpu=all "$ACCEPTANCE_IMAGE" nvidia-smi +} + +run_gpus_test_sudo() { + sudo docker run --rm --gpus all "$ACCEPTANCE_IMAGE" nvidia-smi +} + +docker_has_nvidia_runtime_sudo() { + local runtimes + runtimes="$( + sudo docker info --format '{{range $name, $_ := .Runtimes}}{{println $name}}{{end}}' + )" || fatal "Could not inspect Docker runtimes after the --gpus all probe failed" + grep -Fxq 'nvidia' <<<"$runtimes" +} + +ensure_cdi_runtime() { + ensure_cdi_refresh_lifecycle + if run_cdi_test_sudo; then + info "cdi_contract=pass_without_configuration_change" + return 0 + fi + + warn "CDI GPU launch failed; refreshing the NVIDIA CDI device spec" + refresh_cdi + run_cdi_test_sudo || fatal "CDI Docker GPU test failed after CDI refresh" + info "cdi_contract=pass_after_refresh" +} + +configure_docker_runtime_if_needed() { + local backup_dir previous_daemon=0 + if run_gpus_test_sudo; then + info "docker_gpus_contract=pass_without_configuration_change" + return 0 + fi + + if docker_has_nvidia_runtime_sudo; then + fatal "Docker --gpus all failed even though the NVIDIA runtime is registered; daemon configuration was left unchanged. Inspect the failed container launch and rerun preparation." + fi + + # Persistent registration is the supported repair only for the diagnosed + # missing-runtime state. It remains required until this acceptance probe + # succeeds through a replacement Docker/NVIDIA runtime integration. + warn "Docker --gpus all failed and Docker reports no NVIDIA runtime; applying the reviewed NVIDIA runtime registration" + check_no_workloads + ensure_root_directory_safe /etc/docker /etc 0755 "Docker configuration directory" + ensure_root_directory_safe /var/backups/station-bootstrap /var/backups 0700 "Station bootstrap backup directory" + backup_dir="$(sudo mktemp -d /var/backups/station-bootstrap/docker-runtime.XXXXXXXXXX)" \ + || fatal "Could not create a unique Docker runtime backup directory" + assert_root_directory_safe "$backup_dir" "Docker runtime backup directory" + if sudo test -e /etc/docker/daemon.json || sudo test -L /etc/docker/daemon.json; then + assert_root_regular_file_safe /etc/docker/daemon.json "" "Docker daemon configuration" + sudo cp --archive --no-dereference -- /etc/docker/daemon.json "${backup_dir}/daemon.json" + assert_root_regular_file_safe "${backup_dir}/daemon.json" "" "Docker daemon configuration backup" + previous_daemon=1 + else + sudo touch "${backup_dir}/daemon.json.absent" + sudo chmod 0600 "${backup_dir}/daemon.json.absent" + fi + check_no_workloads + if ! sudo nvidia-ctk runtime configure --runtime=docker; then + fail_after_docker_runtime_rollback "$backup_dir" "$previous_daemon" "NVIDIA runtime registration failed" + fi + if ! root_regular_file_is_safe /etc/docker/daemon.json ""; then + fail_after_docker_runtime_rollback "$backup_dir" "$previous_daemon" "NVIDIA runtime registration produced an unsafe Docker daemon configuration" + fi + if ! (check_no_workloads); then + fail_after_docker_runtime_rollback "$backup_dir" "$previous_daemon" "A workload appeared before Docker restart" 0 + fi + if ! sudo systemctl restart docker.service; then + fail_after_docker_runtime_rollback "$backup_dir" "$previous_daemon" "Docker restart failed after NVIDIA runtime registration" + fi + if ! run_gpus_test_sudo; then + fail_after_docker_runtime_rollback "$backup_dir" "$previous_daemon" "Docker --gpus all still fails after NVIDIA runtime registration" + fi + if ! run_cdi_test_sudo; then + fail_after_docker_runtime_rollback "$backup_dir" "$previous_daemon" "CDI launch regressed after NVIDIA runtime registration" + fi + info "docker_gpus_contract=pass backup=${backup_dir}" +} + +rollback_docker_runtime_config() { + local backup_dir=$1 previous_daemon=$2 restart_after_restore=${3:-1} + warn "Restoring the Docker daemon configuration from ${backup_dir}" + if [[ "$previous_daemon" == "1" ]]; then + root_regular_file_is_safe "${backup_dir}/daemon.json" "" || return 1 + sudo rm -f -- /etc/docker/daemon.json || return 1 + sudo cp --archive --no-dereference -- "${backup_dir}/daemon.json" /etc/docker/daemon.json || return 1 + root_regular_file_is_safe /etc/docker/daemon.json "" || return 1 + else + sudo rm -f -- /etc/docker/daemon.json || return 1 + fi + if [[ "$restart_after_restore" == "1" ]]; then + sudo systemctl restart docker.service + fi +} + +fail_after_docker_runtime_rollback() { + local backup_dir=$1 previous_daemon=$2 reason=$3 restart_after_restore=${4:-1} + if rollback_docker_runtime_config "$backup_dir" "$previous_daemon" "$restart_after_restore"; then + fatal "${reason}; the prior Docker daemon configuration was restored" + fi + fatal "${reason}; automatic Docker daemon rollback failed, restore from ${backup_dir} before retrying" +} + +finish_runtime() { + check_no_workloads + sudo systemctl enable --now containerd.service docker.service + ensure_docker_group + ensure_acceptance_image + ensure_cdi_runtime + configure_docker_runtime_if_needed + [[ -z "$(sudo docker ps -aq)" ]] || fatal "Acceptance tests left a Docker container behind" + info "runtime_setup=complete" +} + +verify_apply_state() { + local spec + for spec in "${PACKAGE_SPECS[@]}"; do + package_is_exact "$spec" || fatal "Package verification failed: ${spec}" + done + verify_gpu + systemctl is-active --quiet nvidia-persistenced.service || fatal "nvidia-persistenced.service is not active" + systemctl is-active --quiet containerd.service || fatal "containerd.service is not active" + systemctl is-active --quiet docker.service || fatal "docker.service is not active" + verify_cdi_refresh_lifecycle + nvidia-ctk cdi list | grep -Fxq 'nvidia.com/gpu=all' || fatal "CDI verification failed" + sudo docker image inspect "$ACCEPTANCE_IMAGE" >/dev/null 2>&1 || fatal "Digest-pinned acceptance image is missing" + [[ -z "$(sudo docker ps -aq)" ]] || fatal "Verification found a leftover Docker container" + info "STATION_HOST_READY" +} + +verify_gpu() { + local row name driver corrected uncorrected + row="$(nvidia-smi \ + --query-gpu=name,driver_version,ecc.errors.corrected.volatile.total,ecc.errors.uncorrected.volatile.total \ + --format=csv,noheader,nounits)" || fatal "nvidia-smi failed" + IFS=',' read -r name driver corrected uncorrected <<<"$row" + name="${name#"${name%%[![:space:]]*}"}" + driver="${driver//[[:space:]]/}" + corrected="${corrected//[[:space:]]/}" + uncorrected="${uncorrected//[[:space:]]/}" + [[ "$name" == *"GB300"* ]] || fatal "Expected NVIDIA GB300, found ${name}" + [[ "$driver" == "$DRIVER_VERSION" ]] || fatal "Expected driver ${DRIVER_VERSION}, found ${driver}" + [[ "$corrected" == "0" && "$uncorrected" == "0" ]] \ + || fatal "ECC must be 0/0, found corrected=${corrected} uncorrected=${uncorrected}" + info "gpu=${name} driver=${driver} ecc_corrected=${corrected} ecc_uncorrected=${uncorrected}" +} + +verify_host() { + local spec user_name=${SUDO_USER:-$USER} + for spec in "${PACKAGE_SPECS[@]}"; do + package_is_exact "$spec" || fatal "Package verification failed: ${spec}" + done + verify_gpu + systemctl is-active --quiet nvidia-persistenced.service || fatal "nvidia-persistenced.service is not active" + systemctl is-active --quiet containerd.service || fatal "containerd.service is not active" + systemctl is-active --quiet docker.service || fatal "docker.service is not active" + verify_cdi_refresh_lifecycle + id -nG "$user_name" | tr ' ' '\n' | grep -Fxq docker || fatal "${user_name} is not in the docker group" + docker info >/dev/null 2>&1 || fatal "${user_name} cannot access Docker; start a new login session" + nvidia-ctk cdi list | grep -Fxq 'nvidia.com/gpu=all' || fatal "CDI verification failed" + docker image inspect "$ACCEPTANCE_IMAGE" >/dev/null 2>&1 || fatal "Digest-pinned acceptance image is missing; run --apply" + docker run --rm --device nvidia.com/gpu=all "$ACCEPTANCE_IMAGE" nvidia-smi >/dev/null + docker run --rm --gpus all "$ACCEPTANCE_IMAGE" nvidia-smi >/dev/null + [[ -z "$(docker ps -aq)" ]] || fatal "Verification left a Docker container behind" + info "docker=$(docker version --format '{{.Server.Version}}') expected_docker=${DOCKER_VERSION} toolkit=$(nvidia-ctk --version | head -n1) expected_toolkit=${TOOLKIT_VERSION}" + info "STATION_HOST_READY" +} + +run_check() { + common_preflight + print_package_status + if all_packages_exact; then + if install_boot_marker_matches_current_boot; then + warn "Package installation completed in the current boot; reboot is required" + info "CHECK_RESULT=REBOOT_REQUIRED" + elif driver_loaded_exact; then + info "CHECK_RESULT=PACKAGES_AND_DRIVER_PRESENT" + else + warn "Exact packages are installed but driver ${DRIVER_VERSION} is not loaded; reboot is required" + info "CHECK_RESULT=REBOOT_REQUIRED" + fi + else + info "CHECK_RESULT=READY_TO_APPLY" + fi +} + +run_apply() { + require_command apt-cache + require_command apt-get + require_command cmp + require_command curl + require_command dpkg + require_command gpg + require_command grep + require_command readlink + require_command sha256sum + require_command sudo + acquire_sudo + common_preflight + + if [[ -e /var/run/reboot-required ]]; then + if all_packages_exact && ! driver_loaded_exact; then + warn "A reboot is required before runtime setup can continue" + exit "$REBOOT_REQUIRED_EXIT" + fi + fatal "An unrelated reboot is already pending" + fi + + if ! all_packages_exact; then + assert_no_package_mismatches + install_packages + ensure_docker_group + check_no_workloads + sudo systemctl enable containerd.service docker.service nvidia-cdi-refresh.path nvidia-cdi-refresh.service + write_install_boot_marker + info "APPLY_RESULT=REBOOT_REQUIRED" + info "Run: sudo reboot" + exit "$REBOOT_REQUIRED_EXIT" + fi + + if install_boot_marker_matches_current_boot; then + warn "Package installation completed in the current boot" + info "APPLY_RESULT=REBOOT_REQUIRED" + info "Run: sudo reboot" + exit "$REBOOT_REQUIRED_EXIT" + fi + + driver_loaded_exact || { + warn "Pinned packages are installed but driver ${DRIVER_VERSION} is not loaded" + info "APPLY_RESULT=REBOOT_REQUIRED" + info "Run: sudo reboot" + exit "$REBOOT_REQUIRED_EXIT" + } + + finish_runtime + verify_apply_state + if ((DOCKER_GROUP_ADDED == 1)); then + warn "Docker group membership was added and requires a new login before onboarding" + info "APPLY_RESULT=REBOOT_REQUIRED" + info "Run: sudo reboot" + exit "$REBOOT_REQUIRED_EXIT" + fi + rm -f "$INSTALL_BOOT_MARKER" + info "APPLY_RESULT=COMPLETE" +} + +run_verify() { + common_preflight + require_command docker + require_command nvidia-ctk + require_command nvidia-smi + all_packages_exact || fatal "Pinned prerequisite packages are incomplete; run --apply" + driver_loaded_exact || fatal "Pinned driver is not loaded; reboot, then run --apply" + verify_host +} + +main() { + if (($# != 1)) || ! is_valid_mode "${1:-}"; then + usage >&2 + exit 2 + fi + MODE=$1 + if [[ "$MODE" == "--apply" ]]; then + setup_log + else + info "version=${SCRIPT_VERSION} mode=${MODE} log=disabled_read_only" + fi + trap 'on_error "$LINENO"' ERR + case "$MODE" in + --check) run_check ;; + --apply) run_apply ;; + --verify) run_verify ;; + esac +} + +if [[ "${BASH_SOURCE[0]}" == "$0" ]]; then + main "$@" +fi diff --git a/test/install-express-prompt.test.ts b/test/install-express-prompt.test.ts index 12224d7c6d1..74f3600ac38 100644 --- a/test/install-express-prompt.test.ts +++ b/test/install-express-prompt.test.ts @@ -253,6 +253,10 @@ detect_express_platform /Express install will configure managed local vLLM with NVIDIA Nemotron 3 Ultra 550B/, ); expect(output).toMatch(/approximately 352 GB model/); + expect(output).toMatch( + /installs missing pinned driver, Docker, and NVIDIA Container Toolkit packages/, + ); + expect(output).toMatch(/DGX Station remains Deferred/); expect(output).toMatch(/Using express install for DGX Station/); expect(output).toMatch( /RESULT NON_INTERACTIVE=1 SUDO_MODE=prompt PROVIDER=install-vllm MODEL=nvidia\/nemotron-3-ultra-550b-a55b VLLM_MODEL=nemotron-3-ultra-550b-a55b POLICY=suggested YES=1 SANDBOX=my-assistant/, @@ -439,6 +443,40 @@ main "$@" expect(output).not.toMatch(/cannot be combined with non-interactive mode/); }); + it.each([ + ["Unsupported DGX Station OS", { NEMOCLAW_NO_EXPRESS: "1" }], + ["Unsupported DGX Station generation", { NEMOCLAW_PROVIDER: "openai" }], + ])("allows an explicit non-express path on %s", (platform, overrides) => { + const result = spawnSync( + "bash", + [ + "--noprofile", + "--norc", + "-c", + ` +source "$INSTALLER_UNDER_TEST" >/dev/null +validate_express_platform_boundary "$EXPRESS_PLATFORM" +printf 'NON_EXPRESS_ALLOWED\n' +`, + ], + { + cwd: path.join(import.meta.dirname, ".."), + encoding: "utf-8", + env: { + HOME: fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-express-platform-override-")), + PATH: TEST_SYSTEM_PATH, + INSTALLER_UNDER_TEST: INSTALLER_PAYLOAD, + EXPRESS_PLATFORM: platform, + ...overrides, + }, + }, + ); + const output = `${result.stdout}${result.stderr}`; + + expect(result.status, output).toBe(0); + expect(output).toContain("NON_EXPRESS_ALLOWED"); + }); + it.each([ ["NEMOCLAW_NO_EXPRESS", "1", /cannot be combined with NEMOCLAW_NO_EXPRESS=1/], // Set directly (bypasses main's flag parsing), so the origin is unknown and @@ -525,6 +563,47 @@ detect_express_platform } }); + it("classifies older DGX Station generations as unsupported", () => { + const result = detectExpressPlatformForProductName("NVIDIA DGX Station A100"); + + expect(result.status, `${result.stdout}${result.stderr}`).toBe(0); + expect(result.stdout).toBe("Unsupported DGX Station generation"); + }); + + it.each([ + "Unsupported DGX Station OS", + "Unsupported DGX Station generation", + ])("rejects %s before the express prompt", (platform) => { + const result = spawnSync( + "bash", + [ + "--noprofile", + "--norc", + "-c", + ` +source "$INSTALLER_UNDER_TEST" >/dev/null +validate_express_platform_boundary "$EXPRESS_PLATFORM" +printf 'PROMPT_REACHED\n' +`, + ], + { + cwd: path.join(import.meta.dirname, ".."), + encoding: "utf-8", + env: { + HOME: fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-express-platform-reject-")), + PATH: TEST_SYSTEM_PATH, + INSTALLER_UNDER_TEST: INSTALLER_PAYLOAD, + EXPRESS_PLATFORM: platform, + }, + }, + ); + const output = `${result.stdout}${result.stderr}`; + + expect(result.status, output).not.toBe(0); + expect(output).toMatch(/outside the validated Station/); + expect(output).not.toContain("PROMPT_REACHED"); + }); + it("maps Windows WSL express install to Windows-host Ollama", () => { const result = runExpressPromptWithTty("\n", "pipe", "Windows WSL"); const output = `${result.stdout}${result.stderr}`; diff --git a/test/install-station-host-preparation.test.ts b/test/install-station-host-preparation.test.ts new file mode 100644 index 00000000000..67228d531e7 --- /dev/null +++ b/test/install-station-host-preparation.test.ts @@ -0,0 +1,1309 @@ +// SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. +// SPDX-License-Identifier: Apache-2.0 + +import { spawnSync } from "node:child_process"; +import fs from "node:fs"; +import os from "node:os"; +import path from "node:path"; +import { describe, expect, it } from "vitest"; +import { INSTALLER_PAYLOAD, TEST_SYSTEM_PATH } from "./helpers/installer-sourced-env"; + +const REPO_ROOT = path.resolve(import.meta.dirname, ".."); +const PUBLIC_BOOTSTRAP = path.join(REPO_ROOT, "install.sh"); +const STATION_PREPARE = path.join(REPO_ROOT, "scripts", "prepare-dgx-station-host.sh"); +const STATION_REVISION = "a".repeat(40); +const STATION_DOCS = [ + path.join(REPO_ROOT, "docs", "get-started", "prerequisites.mdx"), + path.join(REPO_ROOT, "docs", "get-started", "quickstart.mdx"), +]; + +function runSourced(script: string, body: string, extraEnv: Record = {}) { + const home = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-station-host-")); + const result = spawnSync( + "bash", + ["--noprofile", "--norc", "-c", `source "$SCRIPT_UNDER_TEST" >/dev/null\n${body}`], + { + cwd: REPO_ROOT, + encoding: "utf-8", + env: { + HOME: home, + PATH: TEST_SYSTEM_PATH, + SCRIPT_UNDER_TEST: script, + ...extraEnv, + }, + timeout: 15_000, + killSignal: "SIGKILL", + }, + ); + return { home, result, output: `${result.stdout}${result.stderr}` }; +} + +describe("DGX Station host preparation", () => { + it("keeps documented Station pins and Deferred status aligned", () => { + const helper = fs.readFileSync(STATION_PREPARE, "utf-8"); + const docs = STATION_DOCS.map((doc) => fs.readFileSync(doc, "utf-8")); + const pinnedValues = [ + "DRIVER_VERSION", + "DOCKER_VERSION", + "TOOLKIT_VERSION", + "FACTORY_DKMS_VERSION", + "TARGET_DKMS_VERSION", + ].map((name) => { + const value = helper.match(new RegExp(`readonly ${name}="([^"]+)"`))?.[1]; + expect(value, `${name} must remain declared in the Station helper`).toBeTruthy(); + return value as string; + }); + + for (const doc of docs) { + for (const version of pinnedValues) expect(doc).toContain(version); + expect(doc).toMatch(/(?:DGX )?Station(?: remains|'s) Deferred/); + expect(doc).toMatch(/physical (?:DGX Station )?hardware|physical end-to-end validation/); + } + }); + + it("uses the documented plain-Ubuntu driver-injection probe for CDI and --gpus", () => { + const { result, output } = runSourced( + STATION_PREPARE, + ` +sudo() { printf 'SUDO %s\\n' "$*"; } +run_cdi_test_sudo +run_gpus_test_sudo +`, + ); + + const image = + "docker.io/library/ubuntu@sha256:7f622ca8766bccb22f04242ecb6f19f770b2f08827dc4b8c707de5e78a6da7ab"; + expect(result.status, output).toBe(0); + expect(output).toContain( + `SUDO docker run --rm --device nvidia.com/gpu=all ${image} nvidia-smi`, + ); + expect(output).toContain(`SUDO docker run --rm --gpus all ${image} nvidia-smi`); + }); + + it.each([ + ["", "missing"], + ["5:29.6.1-1~ubuntu.24.04~noble", "exact"], + ["5:30.0.0-1~ubuntu.24.04~noble", "mismatch"], + ])("classifies an installed package version as %s -> %s", (actual, expected) => { + const { result, output } = runSourced( + STATION_PREPARE, + ` +installed_version() { + if [[ "$1" == "docker-ce" ]]; then printf '%s' "$PACKAGE_ACTUAL"; fi +} +package_state 'docker-ce=5:29.6.1-1~ubuntu.24.04~noble' +`, + { PACKAGE_ACTUAL: actual }, + ); + + expect(result.status, output).toBe(0); + expect(result.stdout.trim()).toBe(expected); + }); + + it.each([ + ["Dell Pro Max with Station GB300", true], + ["NVIDIA DGX Station GB300", true], + ["NVIDIA DGX Station A100", false], + ["Dell Pro Max with Station GB200", false], + ["Dell Pro Max with GB300", false], + ])("accepts only Station GB300 DMI: %s", (product, accepted) => { + const { result } = runSourced(STATION_PREPARE, `is_station_product "$PRODUCT"`, { + PRODUCT: product, + }); + + expect(result.status === 0).toBe(accepted); + }); + + it("allows only the reviewed factory DKMS transition", () => { + const approved = runSourced( + STATION_PREPARE, + ` +installed_version() { + if [[ "$1" == "dkms" ]]; then printf '%s' "$DKMS_ACTUAL"; fi +} +package_state 'dkms=1:3.4.0-1ubuntu1' +assert_no_package_mismatches +`, + { DKMS_ACTUAL: "3.0.11-1ubuntu13" }, + ); + expect(approved.result.status, approved.output).toBe(0); + expect(approved.output).toContain("approved-transition"); + expect(approved.output).toContain("status=approved_transition"); + + const arbitrary = runSourced( + STATION_PREPARE, + ` +installed_version() { + if [[ "$1" == "dkms" ]]; then printf '%s' "$DKMS_ACTUAL"; fi +} +assert_no_package_mismatches +`, + { DKMS_ACTUAL: "3.2.0-1" }, + ); + expect(arbitrary.result.status, arbitrary.output).not.toBe(0); + expect(arbitrary.output).toMatch(/dkms status=mismatch/); + }); + + it("refuses to change an existing mismatched prerequisite", () => { + const { result, output } = runSourced( + STATION_PREPARE, + ` +installed_version() { + if [[ "$1" == "docker-ce" ]]; then printf '5:30.0.0-1~ubuntu.24.04~noble'; fi +} +assert_no_package_mismatches +`, + ); + + expect(result.status, output).not.toBe(0); + expect(output).toMatch(/docker-ce status=mismatch/); + expect(output).toMatch(/refusing to change them automatically/); + }); + + it("allows only condition-qualified factory failures and blocks other failed units", () => { + const qualified = runSourced( + STATION_PREPARE, + ` +systemctl() { printf 'cloud-init.service loaded failed failed Cloud init\n'; } +cloud_init_failure_is_qualified() { return 0; } +check_failed_units +`, + ); + expect(qualified.result.status, qualified.output).toBe(0); + expect(qualified.output).toMatch( + /condition-qualified generic-image failed unit: cloud-init.service/, + ); + + const unrelated = runSourced( + STATION_PREPARE, + ` +systemctl() { printf 'ssh.service loaded failed failed SSH\n'; } +check_failed_units +`, + ); + expect(unrelated.result.status, unrelated.output).not.toBe(0); + expect(unrelated.output).toMatch(/unqualified failed unit: ssh.service/); + expect(unrelated.output).toMatch(/Unqualified failed system units block Station preparation/); + + const critical = runSourced( + STATION_PREPARE, + ` +systemctl() { printf 'docker.service loaded failed failed Docker\n'; } +check_failed_units +`, + ); + expect(critical.result.status, critical.output).not.toBe(0); + expect(critical.output).toMatch(/failed preparation-critical unit: docker.service/); + }); + + it("qualifies network-wait failures only after current network health is established", () => { + const healthy = runSourced( + STATION_PREPARE, + ` +NETWORK_VALIDATED=1 +systemctl() { + case "$*" in + 'is-active --quiet NetworkManager.service'|'is-active --quiet network-online.target') return 0 ;; + *) return 1 ;; + esac +} +network_wait_failure_is_qualified +`, + ); + expect(healthy.result.status, healthy.output).toBe(0); + + const unvalidated = runSourced( + STATION_PREPARE, + ` +NETWORK_VALIDATED=0 +systemctl() { return 0; } +network_wait_failure_is_qualified +`, + ); + expect(unvalidated.result.status, unvalidated.output).not.toBe(0); + + const inactiveManager = runSourced( + STATION_PREPARE, + ` +NETWORK_VALIDATED=1 +systemctl() { return 1; } +network_wait_failure_is_qualified +`, + ); + expect(inactiveManager.result.status, inactiveManager.output).not.toBe(0); + }); + + it("qualifies only the pinned OEM cloud-init bootcmd failure", () => { + const qualified = runSourced( + STATION_PREPARE, + ` +NETWORK_VALIDATED=1 +stat() { printf 'regular file|0|0|755\n'; } +sha256sum() { printf '%s %s\n' "$FACTORY_CLOUD_INIT_TELEMETRY_SHA256" "$1"; } +grep() { return 0; } +cloud_init_failure_is_qualified +`, + ); + expect(qualified.result.status, qualified.output).toBe(0); + + const changedTelemetry = runSourced( + STATION_PREPARE, + ` +NETWORK_VALIDATED=1 +stat() { printf 'regular file|0|0|755\n'; } +sha256sum() { printf '%064d %s\n' 0 "$1"; } +grep() { return 0; } +cloud_init_failure_is_qualified +`, + ); + expect(changedTelemetry.result.status, changedTelemetry.output).not.toBe(0); + + const unsafeEvidence = runSourced( + STATION_PREPARE, + ` +NETWORK_VALIDATED=1 +stat() { printf 'regular file|0|0|777\n'; } +sha256sum() { printf '%s %s\n' "$FACTORY_CLOUD_INIT_TELEMETRY_SHA256" "$1"; } +grep() { return 0; } +cloud_init_failure_is_qualified +`, + ); + expect(unsafeEvidence.result.status, unsafeEvidence.output).not.toBe(0); + }); + + it("requires exact conditions for auxiliary factory-image failures", () => { + const maskedFwupd = runSourced( + STATION_PREPARE, + ` +NETWORK_VALIDATED=1 +systemctl() { printf 'masked\n'; } +fwupd_refresh_failure_is_qualified +`, + ); + expect(maskedFwupd.result.status, maskedFwupd.output).toBe(0); + + const enabledFwupd = runSourced( + STATION_PREPARE, + ` +NETWORK_VALIDATED=1 +systemctl() { printf 'enabled\n'; } +fwupd_refresh_failure_is_qualified +`, + ); + expect(enabledFwupd.result.status, enabledFwupd.output).not.toBe(0); + + const exactUnits = runSourced( + STATION_PREPARE, + ` +cloud_init_failure_is_qualified() { return 0; } +network_wait_failure_is_qualified() { return 0; } +fwupd_refresh_failure_is_qualified() { return 0; } +sssd_socket_failure_is_qualified() { return 0; } +for unit in \ + cloud-init.service \ + NetworkManager-wait-online.service \ + systemd-networkd-wait-online.service \ + fwupd-refresh.service \ + sssd-autofs.socket \ + sssd-nss.socket \ + sssd-pam.socket \ + sssd-pam-priv.socket; do + is_qualified_factory_failed_unit "$unit" || exit 1 +done +is_qualified_factory_failed_unit ssh.service && exit 1 +exit 0 +`, + ); + expect(exactUnits.result.status, exactUnits.output).toBe(0); + }); + + it("discloses Docker-group root-equivalent access before Station express consent", () => { + const { result, output } = runSourced( + INSTALLER_PAYLOAD, + ` +STATION_DEEPSEEK=0 +NEMOCLAW_VLLM_MODEL='' +describe_express_install 'DGX Station' +`, + ); + + expect(result.status, output).toBe(0); + expect(output).toContain("docker group, which grants root-equivalent control"); + expect(output).toContain("only for trusted single-user development hosts"); + expect(output).toContain( + "shared or managed hosts require an organization-approved Docker access path", + ); + }); + + it("fails closed when failed-service inspection is unavailable", () => { + const { result, output } = runSourced( + STATION_PREPARE, + ` +systemctl() { return 1; } +check_failed_units +`, + ); + + expect(result.status, output).not.toBe(0); + expect(output).toMatch(/Unable to inspect failed system services/); + }); + + it("reuses exact packages and proceeds directly to runtime probes", () => { + const { result, output } = runSourced( + STATION_PREPARE, + ` +common_preflight() { :; } +require_command() { :; } +acquire_sudo() { :; } +all_packages_exact() { return 0; } +install_boot_marker_matches_current_boot() { return 1; } +driver_loaded_exact() { return 0; } +install_packages() { printf 'INSTALL_PACKAGES\n'; } +finish_runtime() { printf 'FINISH_RUNTIME\n'; } +verify_apply_state() { printf 'VERIFY_APPLY_STATE\n'; } +run_apply +`, + ); + + expect(result.status, output).toBe(0); + expect(output).toContain("FINISH_RUNTIME"); + expect(output).toContain("VERIFY_APPLY_STATE"); + expect(output).not.toContain("INSTALL_PACKAGES"); + expect(output).toContain("APPLY_RESULT=COMPLETE"); + }); + + it("applies the reviewed factory DKMS transition and returns the reboot-required contract", () => { + const { result, output } = runSourced( + STATION_PREPARE, + ` +common_preflight() { :; } +require_command() { :; } +acquire_sudo() { :; } +all_packages_exact() { return 1; } +installed_version() { + if [[ "$1" == "dkms" ]]; then printf '3.0.11-1ubuntu13'; fi +} +install_packages() { printf 'INSTALL_PACKAGES\n'; } +ensure_docker_group() { printf 'ENSURE_DOCKER_GROUP\n'; } +check_no_workloads() { printf 'RECHECK_ALL_WORKLOADS\n'; } +write_install_boot_marker() { printf 'WRITE_BOOT_MARKER\n'; } +sudo() { printf 'SUDO %s\n' "$*"; } +run_apply +`, + ); + + expect(result.status, output).toBe(10); + expect(output).toContain("package=dkms status=approved_transition"); + expect(output).toContain("INSTALL_PACKAGES"); + expect(output).toContain("ENSURE_DOCKER_GROUP"); + expect(output).toContain("RECHECK_ALL_WORKLOADS"); + expect(output).toContain("WRITE_BOOT_MARKER"); + expect(output).toContain( + "systemctl enable containerd.service docker.service nvidia-cdi-refresh.path nvidia-cdi-refresh.service", + ); + expect(output).toContain("APPLY_RESULT=REBOOT_REQUIRED"); + }); + + it("installs the exact NVIDIA Container Toolkit package contract", () => { + const { result, output } = runSourced( + STATION_PREPARE, + ` +configure_repositories() { printf 'CONFIGURE_REPOSITORIES\n'; } +validate_package_availability() { printf 'VALIDATE_PACKAGES\n'; } +simulate_install() { printf 'SIMULATE_INSTALL\n'; } +check_no_workloads() { printf 'RECHECK_ALL_WORKLOADS\n'; } +package_is_exact() { return 0; } +sudo() { printf 'SUDO %s\n' "$*"; } +install_packages +`, + ); + + expect(result.status, output).toBe(0); + expect(output).toContain("apt-get update"); + expect(output).toContain("apt-get install -y --no-install-recommends"); + expect(output).toContain("RECHECK_ALL_WORKLOADS"); + for (const spec of [ + "libnvidia-container-tools=1.19.1-1", + "libnvidia-container1=1.19.1-1", + "nvidia-container-toolkit=1.19.1-1", + "nvidia-container-toolkit-base=1.19.1-1", + ]) { + expect(output).toContain(spec); + } + expect(output).toContain("pinned_packages=installed"); + }); + + it("does not refresh CDI when the GPU launch probe already passes", () => { + const { result, output } = runSourced( + STATION_PREPARE, + ` +check_no_workloads() { printf 'RECHECK_ALL_WORKLOADS\n'; } +sudo() { printf 'SUDO %s\n' "$*"; } +run_cdi_test_sudo() { printf 'CDI_TEST\n'; return 0; } +refresh_cdi() { printf 'REFRESH_CDI\n'; } +ensure_cdi_runtime +`, + ); + + expect(result.status, output).toBe(0); + expect(output).toContain("RECHECK_ALL_WORKLOADS"); + expect(output).toContain("systemctl enable nvidia-cdi-refresh.path nvidia-cdi-refresh.service"); + expect(output).toContain("systemctl start nvidia-cdi-refresh.path"); + expect(output).toContain("cdi_contract=pass_without_configuration_change"); + expect(output).not.toContain("REFRESH_CDI"); + }); + + it("refreshes CDI once when the initial GPU launch probe fails", () => { + const { result, output } = runSourced( + STATION_PREPARE, + ` +calls=0 +ensure_cdi_refresh_lifecycle() { printf 'ENSURE_CDI_LIFECYCLE\n'; } +run_cdi_test_sudo() { + calls=$((calls + 1)) + printf 'CDI_TEST_%s\n' "$calls" + [[ "$calls" -gt 1 ]] +} +refresh_cdi() { printf 'REFRESH_CDI\n'; } +ensure_cdi_runtime +`, + ); + + expect(result.status, output).toBe(0); + expect(output).toContain("CDI_TEST_1"); + expect(output).toContain("ENSURE_CDI_LIFECYCLE"); + expect(output).toContain("REFRESH_CDI"); + expect(output).toContain("CDI_TEST_2"); + expect(output).toContain("cdi_contract=pass_after_refresh"); + }); + + it("ignores the installer process while still blocking a real vLLM workload", () => { + const selfOnly = runSourced( + STATION_PREPARE, + ` +ps() { + printf '%s %s bash bash /tmp/NemoClaw/scripts/prepare-dgx-station-host.sh --apply\n' "$$" "$PPID" + printf '%s 1 bash bash /tmp/NemoClaw/scripts/install.sh\n' "$PPID" +} +ss() { :; } +check_no_workloads +`, + ); + expect(selfOnly.result.status, selfOnly.output).toBe(0); + + const active = runSourced( + STATION_PREPARE, + ` +ps() { printf '999 1 python python -m vllm serve model\n'; } +ss() { :; } +check_no_workloads +`, + ); + expect(active.result.status, active.output).not.toBe(0); + expect(active.output).toMatch(/Agent or inference workload is active/); + }); + + it("uses sudo to inspect containers during apply until Docker group access is active", () => { + const { result, output } = runSourced( + STATION_PREPARE, + ` +MODE='--apply' +ps() { printf '%s %s bash bash prepare-dgx-station-host.sh --apply\n' "$$" "$PPID"; } +ss() { :; } +docker() { return 1; } +sudo() { + if [[ "$1" == "-n" ]]; then shift; fi + [[ "$*" == "docker ps -aq" ]] || return 1 +} +systemctl() { return 0; } +check_no_workloads +`, + { PATH: `${path.dirname(process.execPath)}:${TEST_SYSTEM_PATH}` }, + ); + + expect(result.status, output).toBe(0); + expect(output).toContain("docker_access=sudo_until_group_membership_is_active"); + expect(output).toContain("workloads=none"); + }); + + it("fails closed when Docker is installed but its container state cannot be queried", () => { + const { result, output } = runSourced( + STATION_PREPARE, + ` +MODE='--apply' +ps() { printf '%s %s bash bash prepare-dgx-station-host.sh --apply\n' "$$" "$PPID"; } +ss() { :; } +docker() { return 1; } +sudo() { return 1; } +systemctl() { return 1; } +check_no_workloads +`, + { PATH: `${path.dirname(process.execPath)}:${TEST_SYSTEM_PATH}` }, + ); + + expect(result.status, output).not.toBe(0); + expect(output).toMatch(/container state cannot be verified safely/); + }); + + it("refuses an installed CUDA keyring version that differs from the pin", () => { + const { result, output } = runSourced( + STATION_PREPARE, + ` +assert_root_directory_safe() { :; } +installed_version() { printf '2.0-1'; } +ensure_cuda_keyring "$HOME/cuda-keyring.deb" +`, + ); + + expect(result.status, output).not.toBe(0); + expect(output).toMatch(/refusing to upgrade or downgrade it automatically/); + }); + + it("reuses an exact verified CUDA keyring without downloading it again", () => { + const { result, output } = runSourced( + STATION_PREPARE, + ` +assert_root_directory_safe() { :; } +assert_root_regular_file_safe() { :; } +installed_version() { printf '1.1-1'; } +dpkg() { :; } +curl() { printf 'DOWNLOAD\n'; } +sudo() { "$@"; } +verify_key_fingerprint() { printf 'VERIFIED_FINGERPRINT\n'; } +ensure_cuda_keyring "$HOME/cuda-keyring.deb" +`, + ); + + expect(result.status, output).toBe(0); + expect(output).toContain("cuda_keyring=exact version=1.1-1"); + expect(output).toContain("VERIFIED_FINGERPRINT"); + expect(output).not.toContain("DOWNLOAD"); + }); + + it("reuses exact repository files and refuses to overwrite mismatched content", () => { + const { result, output } = runSourced( + STATION_PREPARE, + ` +printf 'validated\n' >"$HOME/source" +cp "$HOME/source" "$HOME/target" +assert_root_directory_safe() { :; } +assert_root_regular_file_safe() { :; } +sudo() { "$@"; } +install_exact_file_or_reuse "$HOME/source" "$HOME/target" 0644 test_repository_file +printf 'modified\n' >"$HOME/target" +install_exact_file_or_reuse "$HOME/source" "$HOME/target" 0644 test_repository_file +`, + ); + + expect(result.status, output).not.toBe(0); + expect(output).toMatch(/test_repository_file=exact/); + expect(output).toMatch(/refusing to overwrite/); + }); + + it("rejects privileged files with unsafe ownership, mode, type, or parent metadata", () => { + for (const metadata of ["1000 0 644", "0 0 666"]) { + const { result, output } = runSourced( + STATION_PREPARE, + ` +sudo() { + if [[ "$1" == "test" ]]; then return 0; fi + if [[ "$1" == "stat" ]]; then printf '%s\n' "$ROOT_METADATA"; return 0; fi + return 1 +} +assert_root_regular_file_safe /etc/example 0644 test_file +`, + { ROOT_METADATA: metadata }, + ); + expect(result.status, output).not.toBe(0); + expect(output).toMatch(/root-owned regular file/); + } + + const unsafeType = runSourced( + STATION_PREPARE, + ` +sudo() { + [[ "$*" == "test ! -L /etc/example" ]] && return 0 + [[ "$*" == "test -f /etc/example" ]] && return 1 + return 1 +} +assert_root_regular_file_safe /etc/example 0644 test_file +`, + ); + expect(unsafeType.result.status, unsafeType.output).not.toBe(0); + expect(unsafeType.output).toMatch(/root-owned regular file/); + + const unsafeParent = runSourced( + STATION_PREPARE, + ` +sudo() { + if [[ "$1" == "test" ]]; then return 0; fi + if [[ "$1" == "stat" ]]; then printf '0 0 777\n'; return 0; fi + return 1 +} +assert_root_directory_safe /etc/apt/keyrings test_directory +`, + ); + expect(unsafeParent.result.status, unsafeParent.output).not.toBe(0); + expect(unsafeParent.output).toMatch(/not group- or other-writable/); + }); + + it("requires a new login after adding Docker group membership", () => { + const { result, output } = runSourced( + STATION_PREPARE, + ` +common_preflight() { :; } +require_command() { :; } +acquire_sudo() { :; } +all_packages_exact() { return 0; } +install_boot_marker_matches_current_boot() { return 1; } +driver_loaded_exact() { return 0; } +finish_runtime() { DOCKER_GROUP_ADDED=1; printf 'FINISH_RUNTIME\n'; } +verify_apply_state() { printf 'VERIFY_APPLY_STATE\n'; } +run_apply +`, + ); + + expect(result.status, output).toBe(10); + expect(output).toContain("VERIFY_APPLY_STATE"); + expect(output).toContain("APPLY_RESULT=REBOOT_REQUIRED"); + expect(output).toMatch(/new login before onboarding/); + }); + + it("fails closed when the packaged CDI refresh service fails", () => { + const { result, output } = runSourced( + STATION_PREPARE, + ` +check_no_workloads() { printf 'RECHECK_ALL_WORKLOADS\n'; } +sudo() { + printf 'SUDO %s\n' "$*" + if [[ "$*" == "systemctl restart nvidia-cdi-refresh.service" ]]; then return 1; fi + return 0 +} +refresh_cdi +`, + ); + + expect(result.status, output).not.toBe(0); + expect(output).toContain("systemctl status nvidia-cdi-refresh.service --no-pager"); + expect(output).toContain("journalctl -u nvidia-cdi-refresh.service --no-pager -n 50"); + expect(output).toContain("RECHECK_ALL_WORKLOADS"); + expect(output).toMatch(/repair nvidia-cdi-refresh\.service/); + expect(output).not.toContain("nvidia-ctk cdi generate"); + }); + + it("fails closed when the packaged CDI refresh produces no GPU device", () => { + const { result, output } = runSourced( + STATION_PREPARE, + ` +check_no_workloads() { printf 'RECHECK_ALL_WORKLOADS\n'; } +sudo() { + printf 'SUDO %s\n' "$*" + return 0 +} +nvidia-ctk() { :; } +refresh_cdi +`, + ); + + expect(result.status, output).not.toBe(0); + expect(output).toContain("RECHECK_ALL_WORKLOADS"); + expect(output).toMatch(/completed without advertising nvidia\.com\/gpu=all/); + expect(output).toContain("systemctl status nvidia-cdi-refresh.service --no-pager"); + expect(output).toContain("journalctl -u nvidia-cdi-refresh.service --no-pager -n 50"); + expect(output).toMatch(/direct CDI generation is not permitted/); + expect(output).not.toContain("nvidia-ctk cdi generate"); + }); + + it("rechecks every workload gate immediately before Docker runtime mutation", () => { + const { result, output } = runSourced( + STATION_PREPARE, + ` +run_gpus_test_sudo() { return 1; } +docker_has_nvidia_runtime_sudo() { return 1; } +sudo() { + [[ "$*" == "docker ps -aq" ]] && return 0 + [[ "$*" == "test -e /etc/docker/daemon.json" ]] && return 1 + printf 'SUDO %s\n' "$*" +} +check_no_workloads() { printf 'RECHECK_ALL_WORKLOADS\n'; return 1; } +configure_docker_runtime_if_needed +`, + ); + + expect(result.status, output).not.toBe(0); + expect(output).toContain("RECHECK_ALL_WORKLOADS"); + expect(output).not.toContain("nvidia-ctk runtime configure"); + expect(output).not.toContain("systemctl restart docker.service"); + }); + + it("leaves Docker unchanged when the NVIDIA runtime is already registered", () => { + const { result, output } = runSourced( + STATION_PREPARE, + ` +run_gpus_test_sudo() { return 1; } +docker_has_nvidia_runtime_sudo() { return 0; } +sudo() { printf 'SUDO %s\n' "$*"; } +configure_docker_runtime_if_needed +`, + ); + + expect(result.status, output).not.toBe(0); + expect(output).toMatch(/NVIDIA runtime is registered/); + expect(output).toMatch(/daemon configuration was left unchanged/); + expect(output).not.toContain("nvidia-ctk runtime configure"); + expect(output).not.toContain("systemctl restart docker.service"); + }); + + it("registers the NVIDIA runtime only when Docker reports it missing", () => { + const { result, output } = runSourced( + STATION_PREPARE, + ` +calls=0 +run_gpus_test_sudo() { + calls=$((calls + 1)) + [[ "$calls" -gt 1 ]] +} +run_cdi_test_sudo() { return 0; } +docker_has_nvidia_runtime_sudo() { return 1; } +check_no_workloads() { printf 'RECHECK_ALL_WORKLOADS\n'; } +ensure_root_directory_safe() { :; } +assert_root_directory_safe() { :; } +assert_root_regular_file_safe() { :; } +root_regular_file_is_safe() { return 0; } +sudo() { + if [[ "$*" == "mktemp -d /var/backups/station-bootstrap/docker-runtime.XXXXXXXXXX" ]]; then + printf '/var/backups/station-bootstrap/docker-runtime.TEST' + return 0 + fi + [[ "$*" == "test -e /etc/docker/daemon.json" || "$*" == "test -L /etc/docker/daemon.json" ]] && return 1 + printf 'SUDO %s\n' "$*" +} +configure_docker_runtime_if_needed +`, + ); + + expect(result.status, output).toBe(0); + expect(output).toContain("Docker reports no NVIDIA runtime"); + expect(output).toContain("nvidia-ctk runtime configure --runtime=docker"); + expect(output).toContain("systemctl restart docker.service"); + expect(output).toContain("docker_gpus_contract=pass"); + }); + + it("restores configuration without restarting Docker when a workload appears at the restart boundary", () => { + const { result, output } = runSourced( + STATION_PREPARE, + ` +runtime_configured=0 +run_gpus_test_sudo() { return 1; } +run_cdi_test_sudo() { return 0; } +docker_has_nvidia_runtime_sudo() { return 1; } +check_no_workloads() { + printf 'RECHECK_ALL_WORKLOADS configured=%s\n' "$runtime_configured" + [[ "$runtime_configured" == "0" ]] +} +ensure_root_directory_safe() { :; } +assert_root_directory_safe() { :; } +assert_root_regular_file_safe() { :; } +root_regular_file_is_safe() { return 0; } +sudo() { + if [[ "$*" == "mktemp -d /var/backups/station-bootstrap/docker-runtime.XXXXXXXXXX" ]]; then + printf '/var/backups/station-bootstrap/docker-runtime.TEST' + return 0 + fi + [[ "$*" == "test -e /etc/docker/daemon.json" || "$*" == "test -L /etc/docker/daemon.json" ]] && return 1 + if [[ "$*" == "nvidia-ctk runtime configure --runtime=docker" ]]; then + runtime_configured=1 + fi + printf 'SUDO %s\n' "$*" +} +configure_docker_runtime_if_needed +`, + ); + + expect(result.status, output).not.toBe(0); + expect(output).toContain("RECHECK_ALL_WORKLOADS configured=1"); + expect(output).toContain("rm -f -- /etc/docker/daemon.json"); + expect(output).toMatch(/A workload appeared before Docker restart/); + expect(output).toMatch(/prior Docker daemon configuration was restored/); + expect(output).not.toContain("systemctl restart docker.service"); + }); + + it("restores the prior Docker configuration when a post-mutation launch probe fails", () => { + const { result, output } = runSourced( + STATION_PREPARE, + ` +run_gpus_test_sudo() { printf 'GPU_PROBE\n'; return 1; } +run_cdi_test_sudo() { return 0; } +docker_has_nvidia_runtime_sudo() { return 1; } +check_no_workloads() { printf 'RECHECK_ALL_WORKLOADS\n'; } +ensure_root_directory_safe() { :; } +assert_root_directory_safe() { :; } +assert_root_regular_file_safe() { :; } +root_regular_file_is_safe() { return 0; } +sudo() { + if [[ "$*" == "mktemp -d /var/backups/station-bootstrap/docker-runtime.XXXXXXXXXX" ]]; then + printf '/var/backups/station-bootstrap/docker-runtime.TEST' + return 0 + fi + [[ "$*" == "test -e /etc/docker/daemon.json" || "$*" == "test -L /etc/docker/daemon.json" ]] && return 1 + printf 'SUDO %s\n' "$*" +} +configure_docker_runtime_if_needed +`, + ); + + expect(result.status, output).not.toBe(0); + expect(output).toContain("Restoring the Docker daemon configuration"); + expect(output).toContain("rm -f -- /etc/docker/daemon.json"); + expect(output).toMatch(/prior Docker daemon configuration was restored/); + }); + + it("accepts a successful packaged CDI refresh", () => { + const { result, output } = runSourced( + STATION_PREPARE, + ` +check_no_workloads() { printf 'RECHECK_ALL_WORKLOADS\n'; } +sudo() { printf 'SUDO %s\n' "$*"; } +nvidia-ctk() { + [[ "$*" == "cdi list" ]] && printf 'nvidia.com/gpu=all\n' +} +refresh_cdi +`, + ); + + expect(result.status, output).toBe(0); + expect(output).toContain("systemctl enable nvidia-cdi-refresh.path nvidia-cdi-refresh.service"); + expect(output).toContain("systemctl start nvidia-cdi-refresh.path"); + expect(output).toContain("systemctl restart nvidia-cdi-refresh.service"); + expect(output).toContain("cdi=nvidia.com/gpu=all source=packaged_refresh_service"); + expect(output).not.toContain("systemctl status"); + expect(output).not.toContain("cdi generate"); + }); + + it("verifies the durable packaged CDI refresh lifecycle", () => { + const { result, output } = runSourced( + STATION_PREPARE, + ` +systemctl() { printf 'SYSTEMCTL %s\n' "$*"; } +verify_cdi_refresh_lifecycle +`, + ); + + expect(result.status, output).toBe(0); + expect(output).toContain("SYSTEMCTL is-enabled --quiet nvidia-cdi-refresh.path"); + expect(output).toContain("SYSTEMCTL is-enabled --quiet nvidia-cdi-refresh.service"); + expect(output).toContain("SYSTEMCTL is-active --quiet nvidia-cdi-refresh.path"); + expect(output).toContain("cdi_refresh_lifecycle=verified"); + }); + + it.each(["--check", "--verify"])("keeps %s read-only under HOME", (mode) => { + const { home, result, output } = runSourced( + STATION_PREPARE, + ` +run_check() { :; } +run_verify() { :; } +main "$READ_MODE" +`, + { READ_MODE: mode }, + ); + + expect(result.status, output).toBe(0); + expect(output).toContain("log=disabled_read_only"); + expect(fs.existsSync(path.join(home, "station-bootstrap-logs"))).toBe(false); + }); + + it("fails verification when exact packages are present but the driver is not loaded", () => { + const { result, output } = runSourced( + STATION_PREPARE, + ` +common_preflight() { :; } +require_command() { :; } +all_packages_exact() { return 0; } +driver_loaded_exact() { return 1; } +run_verify +`, + ); + + expect(result.status, output).not.toBe(0); + expect(output).toMatch(/Pinned driver is not loaded/); + }); + + it("rejects a symlinked Station bootstrap state directory", () => { + const { home, result, output } = runSourced( + STATION_PREPARE, + ` +mkdir -p "$HOME/.local/state" "$HOME/redirect-target" +ln -s "$HOME/redirect-target" "$HOME/.local/state/station-bootstrap" +write_install_boot_marker +`, + ); + + expect(result.status, output).not.toBe(0); + expect(output).toMatch(/Refusing symbolic link in Station bootstrap state path/); + expect(fs.existsSync(path.join(home, "redirect-target", "install-boot-id"))).toBe(false); + }); + + it("rejects a direct boot-marker symlink without modifying its target", () => { + const { home, result, output } = runSourced( + STATION_PREPARE, + ` +mkdir -p "$HOME/.local/state/station-bootstrap" +chmod 0700 "$HOME/.local/state/station-bootstrap" +printf 'preserve-this-target\n' >"$HOME/marker-target" +ln -s "$HOME/marker-target" "$HOME/.local/state/station-bootstrap/install-boot-id" +write_install_boot_marker +`, + ); + + expect(result.status, output).not.toBe(0); + expect(output).toMatch(/Refusing symbolic link for Station bootstrap boot marker/); + expect(fs.readFileSync(path.join(home, "marker-target"), "utf-8")).toBe( + "preserve-this-target\n", + ); + }); +}); + +describe("DGX Station express host integration", () => { + it("ships and invokes Station preparation through the public curl bootstrap", () => { + const tmp = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-station-public-bootstrap-")); + const fakeBin = path.join(tmp, "bin"); + fs.mkdirSync(fakeBin); + fs.writeFileSync( + path.join(fakeBin, "git"), + `#!/usr/bin/env bash +set -euo pipefail +if [ "\${1:-}" = "init" ]; then + target="\${@: -1}" + mkdir -p "$target/scripts" + cat > "$target/scripts/install.sh" <<'PAYLOAD' +#!/usr/bin/env bash +# NEMOCLAW_VERSIONED_INSTALLER_PAYLOAD=1 +set -euo pipefail +source "\${INSTALLER_UNDER_TEST:?}" >/dev/null +SCRIPT_DIR="$(cd "$(dirname "\${BASH_SOURCE[0]}")" && pwd)" +maybe_offer_express_install() { _SELECTED_EXPRESS_PLATFORM='DGX Station'; } +ensure_docker() { printf 'ENSURE_DOCKER\\n'; } +ensure_openshell_build_deps() { printf 'ENSURE_BUILD_DEPS\\n'; } +prepare_installer_host +PAYLOAD + cat > "$target/scripts/prepare-dgx-station-host.sh" <<'HELPER' +#!/usr/bin/env bash +set -euo pipefail +[ "\${1:-}" = "--apply" ] +printf 'PREPARE_STATION\\n' +HELPER + chmod +x "$target/scripts/install.sh" "$target/scripts/prepare-dgx-station-host.sh" + exit 0 +fi +if [ "\${1:-}" = "-C" ]; then shift 2; fi +case "\${1:-}" in + remote|fetch|checkout) exit 0 ;; +esac +exit 0 +`, + { mode: 0o755 }, + ); + + const result = spawnSync("bash", [], { + cwd: tmp, + input: fs.readFileSync(PUBLIC_BOOTSTRAP, "utf-8"), + encoding: "utf-8", + env: { + ...process.env, + HOME: tmp, + PATH: `${fakeBin}:${TEST_SYSTEM_PATH}`, + INSTALLER_UNDER_TEST: INSTALLER_PAYLOAD, + NEMOCLAW_INSTALL_REF: "refs/tags/station-fixture", + }, + timeout: 15_000, + killSignal: "SIGKILL", + }); + const output = `${result.stdout}${result.stderr}`; + + expect(result.status, output).toBe(0); + expect(output).toContain("DGX Station host prerequisites are ready"); + expect(output.indexOf("PREPARE_STATION")).toBeGreaterThanOrEqual(0); + expect(output.indexOf("PREPARE_STATION")).toBeLessThan(output.indexOf("ENSURE_DOCKER")); + expect(output.indexOf("ENSURE_DOCKER")).toBeLessThan(output.indexOf("ENSURE_BUILD_DEPS")); + }); + + it("runs Station preparation before the generic Docker bootstrap", () => { + const { result, output } = runSourced( + INSTALLER_PAYLOAD, + ` +maybe_offer_express_install() { printf 'SELECT_EXPRESS\n'; _SELECTED_EXPRESS_PLATFORM='DGX Station'; } +ensure_station_express_host() { printf 'PREPARE_STATION\n'; } +ensure_docker() { printf 'ENSURE_DOCKER\n'; } +ensure_openshell_build_deps() { printf 'ENSURE_BUILD_DEPS\n'; } +prepare_installer_host +`, + ); + + expect(result.status, output).toBe(0); + expect(result.stdout.trim().split("\n")).toEqual([ + "SELECT_EXPRESS", + "PREPARE_STATION", + "ENSURE_DOCKER", + "ENSURE_BUILD_DEPS", + ]); + }); + + it("skips Station preparation before Docker bootstrap on non-Station platforms", () => { + const { result, output } = runSourced( + INSTALLER_PAYLOAD, + ` +maybe_offer_express_install() { _SELECTED_EXPRESS_PLATFORM='DGX Spark'; } +ensure_station_express_host() { + [[ "$_SELECTED_EXPRESS_PLATFORM" == 'DGX Station' ]] && printf 'PREPARE_STATION\n' + return 0 +} +ensure_docker() { printf 'ENSURE_DOCKER\n'; } +ensure_openshell_build_deps() { printf 'ENSURE_BUILD_DEPS\n'; } +prepare_installer_host +`, + ); + + expect(result.status, output).toBe(0); + expect(output).not.toContain("PREPARE_STATION"); + expect(result.stdout.trim().split("\n")).toEqual(["ENSURE_DOCKER", "ENSURE_BUILD_DEPS"]); + }); + + it("persists the selected model when host preparation requires a reboot", () => { + const { home, result, output } = runSourced( + INSTALLER_PAYLOAD, + ` +_SELECTED_EXPRESS_PLATFORM='DGX Station' +NEMOCLAW_VLLM_MODEL='nemotron-3-ultra-550b-a55b' +station_installer_revision() { printf '${STATION_REVISION}'; } +run_station_host_preparation() { return 10; } +ensure_station_express_host +`, + ); + const stateFile = path.join(home, ".nemoclaw", "station-express-resume"); + + expect(result.status, output).toBe(10); + expect(fs.readFileSync(stateFile, "utf-8")).toBe( + `revision=${STATION_REVISION}\nmodel=nemotron-3-ultra-550b-a55b\n`, + ); + expect(fs.statSync(stateFile).mode & 0o777).toBe(0o600); + expect(output).toContain(`NEMOCLAW_INSTALL_TAG=${STATION_REVISION}`); + }); + + it("rejects a resume-state symlink without loading its target", () => { + const { home, result, output } = runSourced( + INSTALLER_PAYLOAD, + ` +mkdir -p "$HOME/.nemoclaw" +chmod 0700 "$HOME/.nemoclaw" +printf 'deepseek-v4-flash\n' >"$HOME/resume-target" +ln -s "$HOME/resume-target" "$HOME/.nemoclaw/station-express-resume" +load_station_express_resume +`, + ); + const target = path.join(home, "resume-target"); + const stateFile = path.join(home, ".nemoclaw", "station-express-resume"); + + expect(result.status, output).toBe(1); + expect(output).toMatch(/Refusing symbolic link in NemoClaw state path/); + expect(fs.readFileSync(target, "utf-8")).toBe("deepseek-v4-flash\n"); + expect(fs.lstatSync(stateFile).isSymbolicLink()).toBe(true); + }); + + it("rejects a resume-state symlink without modifying its target", () => { + const { home, result, output } = runSourced( + INSTALLER_PAYLOAD, + ` +mkdir -p "$HOME/.nemoclaw" +chmod 0700 "$HOME/.nemoclaw" +printf 'preserve-this-target\n' >"$HOME/resume-target" +ln -s "$HOME/resume-target" "$HOME/.nemoclaw/station-express-resume" +_SELECTED_EXPRESS_PLATFORM='DGX Station' +NEMOCLAW_VLLM_MODEL='nemotron-3-ultra-550b-a55b' +station_installer_revision() { printf '${STATION_REVISION}'; } +run_station_host_preparation() { return 10; } +ensure_station_express_host +`, + ); + const target = path.join(home, "resume-target"); + const stateFile = path.join(home, ".nemoclaw", "station-express-resume"); + + expect(result.status, output).toBe(1); + expect(output).toMatch(/Refusing symbolic link in NemoClaw state path/); + expect(fs.readFileSync(target, "utf-8")).toBe("preserve-this-target\n"); + expect(fs.lstatSync(stateFile).isSymbolicLink()).toBe(true); + }); + + it("resumes the accepted Station recipe without another prompt", () => { + const home = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-station-resume-")); + const stateDir = path.join(home, ".nemoclaw"); + fs.mkdirSync(stateDir, { mode: 0o700 }); + fs.writeFileSync( + path.join(stateDir, "station-express-resume"), + `revision=${STATION_REVISION}\nmodel=nemotron-3-ultra-550b-a55b\n`, + { mode: 0o600 }, + ); + const result = spawnSync( + "bash", + [ + "--noprofile", + "--norc", + "-c", + ` +source "$INSTALLER_UNDER_TEST" >/dev/null +detect_express_platform() { printf 'DGX Station'; } +station_installer_revision() { printf '${STATION_REVISION}'; } +NON_INTERACTIVE='' +NEMOCLAW_PROVIDER='' +NEMOCLAW_NO_EXPRESS='' +maybe_offer_express_install +printf 'RESULT PLATFORM=%s PROVIDER=%s MODEL=%s VLLM_MODEL=%s\n' \ + "$_SELECTED_EXPRESS_PLATFORM" "$NEMOCLAW_PROVIDER" "\${NEMOCLAW_MODEL:-}" "$NEMOCLAW_VLLM_MODEL" +`, + ], + { + cwd: REPO_ROOT, + encoding: "utf-8", + env: { + HOME: home, + PATH: TEST_SYSTEM_PATH, + INSTALLER_UNDER_TEST: INSTALLER_PAYLOAD, + }, + timeout: 15_000, + killSignal: "SIGKILL", + }, + ); + const output = `${result.stdout}${result.stderr}`; + + expect(result.status, output).toBe(0); + expect(output).toMatch(/Resuming the accepted express install/); + expect(output).not.toMatch(/Run express install with these settings/); + expect(output).toMatch( + /RESULT PLATFORM=DGX Station PROVIDER=install-vllm MODEL=nvidia\/nemotron-3-ultra-550b-a55b VLLM_MODEL=nemotron-3-ultra-550b-a55b/, + ); + }); + + it("preserves an explicit provider even when Station resume state exists", () => { + const { home, result, output } = runSourced( + INSTALLER_PAYLOAD, + ` +mkdir -p "$HOME/.nemoclaw" +chmod 0700 "$HOME/.nemoclaw" +printf 'nemotron-3-ultra-550b-a55b\n' >"$HOME/.nemoclaw/station-express-resume" +chmod 0600 "$HOME/.nemoclaw/station-express-resume" +detect_express_platform() { printf 'DGX Station'; } +NON_INTERACTIVE='' +NEMOCLAW_PROVIDER='openai' +NEMOCLAW_NO_EXPRESS='' +maybe_offer_express_install +printf 'RESULT PROVIDER=%s\n' "$NEMOCLAW_PROVIDER" +`, + ); + + expect(result.status, output).toBe(0); + expect(output).toContain("NEMOCLAW_PROVIDER=openai already set"); + expect(output).toContain("RESULT PROVIDER=openai"); + expect(output).not.toContain("Resuming the accepted express install"); + expect(fs.existsSync(path.join(home, ".nemoclaw", "station-express-resume"))).toBe(false); + }); + + it("clears pending Station resume state when express install is explicitly disabled", () => { + const { home, result, output } = runSourced( + INSTALLER_PAYLOAD, + ` +mkdir -p "$HOME/.nemoclaw" +chmod 0700 "$HOME/.nemoclaw" +printf 'revision=${STATION_REVISION}\nmodel=nemotron-3-ultra-550b-a55b\n' >"$HOME/.nemoclaw/station-express-resume" +chmod 0600 "$HOME/.nemoclaw/station-express-resume" +detect_express_platform() { printf 'DGX Station'; } +NON_INTERACTIVE='' +NEMOCLAW_PROVIDER='' +NEMOCLAW_NO_EXPRESS='1' +maybe_offer_express_install +`, + ); + + expect(result.status, output).toBe(0); + expect(output).toContain("NEMOCLAW_NO_EXPRESS=1"); + expect(fs.existsSync(path.join(home, ".nemoclaw", "station-express-resume"))).toBe(false); + }); + + it("does not load Station resume state on DGX Spark", () => { + const { result, output } = runSourced( + INSTALLER_PAYLOAD, + ` +mkdir -p "$HOME/.nemoclaw" +chmod 0700 "$HOME/.nemoclaw" +printf 'nemotron-3-ultra-550b-a55b\n' >"$HOME/.nemoclaw/station-express-resume" +chmod 0600 "$HOME/.nemoclaw/station-express-resume" +detect_express_platform() { printf 'DGX Spark'; } +NON_INTERACTIVE='1' +NEMOCLAW_PROVIDER='' +NEMOCLAW_NO_EXPRESS='' +NEMOCLAW_VLLM_MODEL='' +maybe_offer_express_install +printf 'RESULT MODEL=%s\n' "$NEMOCLAW_VLLM_MODEL" +`, + ); + + expect(result.status, output).toBe(0); + expect(output).toContain("Detected DGX Spark. Skipping express prompt (--non-interactive set)"); + expect(output).toContain("RESULT MODEL="); + expect(output).not.toContain("Resuming the accepted express install"); + expect(output).not.toContain("nemotron-3-ultra-550b-a55b"); + }); + + it("rejects a multi-line Station resume state", () => { + const home = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-station-resume-invalid-")); + const stateDir = path.join(home, ".nemoclaw"); + fs.mkdirSync(stateDir, { mode: 0o700 }); + fs.writeFileSync( + path.join(stateDir, "station-express-resume"), + `revision=${STATION_REVISION}\nmodel=nemotron-3-ultra-550b-a55b\nunexpected\n`, + { mode: 0o600 }, + ); + const result = spawnSync( + "bash", + [ + "--noprofile", + "--norc", + "-c", + `source "$INSTALLER_UNDER_TEST" >/dev/null; load_station_express_resume`, + ], + { + cwd: REPO_ROOT, + encoding: "utf-8", + env: { + HOME: home, + PATH: TEST_SYSTEM_PATH, + INSTALLER_UNDER_TEST: INSTALLER_PAYLOAD, + }, + timeout: 15_000, + killSignal: "SIGKILL", + }, + ); + const output = `${result.stdout}${result.stderr}`; + + expect(result.status, output).not.toBe(0); + expect(output).toMatch(/resume state is invalid/); + }); + + it("rejects resume under a different installer revision with exact rerun guidance", () => { + const savedRevision = "b".repeat(40); + const currentRevision = "c".repeat(40); + const { result, output } = runSourced( + INSTALLER_PAYLOAD, + ` +mkdir -p "$HOME/.nemoclaw" +chmod 0700 "$HOME/.nemoclaw" +printf 'revision=${savedRevision}\nmodel=nemotron-3-ultra-550b-a55b\n' >"$HOME/.nemoclaw/station-express-resume" +chmod 0600 "$HOME/.nemoclaw/station-express-resume" +station_installer_revision() { printf '${currentRevision}'; } +load_station_express_resume +`, + ); + + expect(result.status, output).not.toBe(0); + expect(output).toContain(`requires NemoClaw revision ${savedRevision}`); + expect(output).toContain(`NEMOCLAW_INSTALL_TAG=${savedRevision}`); + }); +}); diff --git a/test/test-boundary-guards.test.ts b/test/test-boundary-guards.test.ts index 3237fbf5b9d..6f0a5115aeb 100644 --- a/test/test-boundary-guards.test.ts +++ b/test/test-boundary-guards.test.ts @@ -696,6 +696,7 @@ describe("Vitest project membership boundary", () => { ["test/install-openshell-version-check.test.ts", "installer-integration"], ["test/install-preflight-docker-bootstrap.test.ts", "installer-integration"], ["test/install-preflight.test.ts", "installer-integration"], + ["test/install-station-host-preparation.test.ts", "installer-integration"], ["test/package-contract/example.test.js", "package-contract"], ["test/e2e/support/example.test.js", "e2e-support"], ["test/e2e/live/example.spec.ts", "e2e-live"], diff --git a/vitest.config.ts b/vitest.config.ts index bd368fa6b0a..6d5509c6b1d 100644 --- a/vitest.config.ts +++ b/vitest.config.ts @@ -124,6 +124,7 @@ export default defineConfig({ "test/install-clone-ref.test.ts", "test/install-preflight.test.ts", "test/install-preflight-docker-bootstrap.test.ts", + "test/install-station-host-preparation.test.ts", "test/install-openshell-version-check.test.ts", ], }, @@ -142,6 +143,7 @@ export default defineConfig({ "test/install-clone-ref.test.ts", "test/install-preflight.test.ts", "test/install-preflight-docker-bootstrap.test.ts", + "test/install-station-host-preparation.test.ts", "test/install-openshell-version-check.test.ts", ], // Slow tests that spawn real bash install.sh processes. Explicit