Skip to content
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions docs/get-started/prerequisites.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,8 @@ On an unqualified system, set `NEMOCLAW_PROVIDER` or `NEMOCLAW_NO_EXPRESS=1` exp
On the generic Ubuntu path, accepting express install prepares the host with NVIDIA open driver `610.43.02`, Docker CE `29.6.1` with Buildx, and NVIDIA Container Toolkit `1.19.1`.
Preparation probes package and runtime state first, reuses exact matches, and installs only missing pinned packages, including the NVIDIA Container Toolkit libraries and `nvidia-ctk` CLI.
It permits only the reviewed factory transition from `dkms` `3.0.11-1ubuntu13` to `1:3.4.0-1ubuntu1`.
An existing Docker APT source is reused only when it exactly matches the reviewed `.gpg` or `.asc` form and its referenced key matches the freshly downloaded, pinned, fingerprint-verified key.
Any other source or key state stops preparation without overwriting it.
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated
After reboot, preparation enables NVIDIA's packaged CDI refresh path and service, requires the `nvidia.com/gpu=all` device, and verifies it with a real container launch.
If the packaged refresh fails or does not produce that device, preparation prints service diagnostics and stops for administrator repair.
It does not bypass the packaged lifecycle with direct CDI generation.
Expand All @@ -64,6 +66,7 @@ After changing pinned packages, the installer exits with status `10`; reboot, si

On a qualifying stock DGX OS image, express install validates the factory stack in place against the local default Docker daemon.
It requires a loaded GB300 driver with zero volatile corrected and uncorrected ECC errors, active Docker and containerd services, working Docker Buildx, `nvidia-ctk`, the `nvidia.com/gpu=all` CDI device, and successful `nvidia-smi` device-visibility probes through both CDI and `--gpus all`.
Driver and ECC validation selects GB300 GPUs by PCI identity and skips auxiliary NVIDIA GPUs.
This path does not install or replace host packages, enable or restart services, generate CDI configuration, or rewrite the Docker runtime.
The preparation helper writes a private audit log under `~/station-bootstrap-logs`.
Aside from the installer's separately disclosed `docker`-group membership grant when the current trusted account needs Docker access, its only persistent stock-runtime effect is that the digest-pinned visibility-test image can remain in the Docker cache.
Expand Down
2 changes: 2 additions & 0 deletions scripts/checks/vitest-project-overlap.ts
Original file line number Diff line number Diff line change
Expand Up @@ -47,7 +47,9 @@ const INSTALLER_INTEGRATION_TESTS = new Set([
"test/install-preflight-docker-bootstrap.test.ts",
"test/install-preflight.test.ts",
"test/install-station-dgx-os.test.ts",
"test/install-station-docker-repository.test.ts",
"test/install-station-host-preparation.test.ts",
"test/install-station-package-transaction.test.ts",
]);

function normalizeRepoPath(file: string): string {
Expand Down
20 changes: 19 additions & 1 deletion scripts/install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3384,7 +3384,25 @@ run_station_host_preparation() {
# fail-closed check so Station preparation cannot drift from that ref.
local helper="${SCRIPT_DIR}/prepare-dgx-station-host.sh"
[[ -f "$helper" ]] || error "DGX Station host preparation helper is missing: ${helper}"
bash "$helper" --apply
bash "$helper" --apply 2>&1 | filter_station_host_preparation_output
}

filter_station_host_preparation_output() {
local line detail
while IFS= read -r line; do
case "$line" in
*" version="*" log="*)
info "DGX Station host preparation log: ${line##* log=}"
;;
*" WARNING: "*)
detail="${line#* WARNING: }"
warn "$detail"
;;
*" ERROR: "*)
printf '%s\n' "$line" >&2
;;
esac
done
}

ensure_station_express_host() {
Expand Down
125 changes: 98 additions & 27 deletions scripts/prepare-dgx-station-host.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,7 @@
set -Eeuo pipefail
umask 077

readonly SCRIPT_VERSION="2026-07-17.3"
readonly SCRIPT_VERSION="2026-07-17.4"
readonly REBOOT_REQUIRED_EXIT=10
readonly MIN_FREE_KIB=$((20 * 1024 * 1024))
readonly GB300_PCI_VENDOR="0x10de"
Expand Down Expand Up @@ -227,19 +227,39 @@ is_station_gb300_product() {
"$product" =~ (^|[^[:alnum:]])[Gg][Bb]300([^[:alnum:]]|$) ]]
}

normalize_nvidia_pci_bus_id() {
local bus_id domain rest
bus_id="$(printf '%s' "$1" | tr '[:upper:]' '[:lower:]')"
bus_id="${bus_id//[[:space:]]/}"
[[ "$bus_id" =~ ^([0-9a-f]{4}|[0-9a-f]{8}):[0-9a-f]{2}:[0-9a-f]{2}\.[0-7]$ ]] || return 1
domain="${bus_id%%:*}"
rest="${bus_id#*:}"
if ((${#domain} == 8)); then
domain="${domain:4}"
fi
printf '%s:%s' "$domain" "$rest"
}

station_pci_device_is_gb300() {
local bus_id=$1 pci_root=${2:-/sys/bus/pci/devices} pci_path vendor device class
bus_id="$(normalize_nvidia_pci_bus_id "$bus_id")" || return 1
pci_path="${pci_root}/${bus_id}"
[[ -d "$pci_path" &&
-r "$pci_path/vendor" &&
-r "$pci_path/device" &&
-r "$pci_path/class" ]] || return 1
IFS= read -r vendor <"$pci_path/vendor" || return 1
IFS= read -r device <"$pci_path/device" || return 1
IFS= read -r class <"$pci_path/class" || return 1
[[ "$vendor" == "$GB300_PCI_VENDOR" &&
"$device" == "$GB300_PCI_DEVICE" &&
"$class" == "${GB300_PCI_CLASS_PREFIX}"* ]]
}

station_has_exact_gb300_pci_gpu() {
local pci_root=${1:-/sys/bus/pci/devices} pci_path vendor device class
local pci_root=${1:-/sys/bus/pci/devices} pci_path
for pci_path in "$pci_root"/*; do
[[ -d "$pci_path" &&
-r "$pci_path/vendor" &&
-r "$pci_path/device" &&
-r "$pci_path/class" ]] || continue
IFS= read -r vendor <"$pci_path/vendor" || continue
IFS= read -r device <"$pci_path/device" || continue
IFS= read -r class <"$pci_path/class" || continue
[[ "$vendor" == "$GB300_PCI_VENDOR" &&
"$device" == "$GB300_PCI_DEVICE" &&
"$class" == "${GB300_PCI_CLASS_PREFIX}"* ]] && return 0
station_pci_device_is_gb300 "${pci_path##*/}" "$pci_root" && return 0
done
return 1
}
Expand Down Expand Up @@ -568,11 +588,19 @@ check_no_workloads() {
}

loaded_driver_version() {
local loaded
local rows row bus_id driver pci_root
command -v nvidia-smi >/dev/null 2>&1 || return 0
loaded="$(nvidia-smi --query-gpu=driver_version --format=csv,noheader 2>/dev/null | head -n1 | tr -d '[:space:]')" \
rows="$(nvidia-smi --query-gpu=pci.bus_id,driver_version --format=csv,noheader 2>/dev/null)" \
|| return 0
printf '%s' "$loaded"
pci_root="$(station_pci_devices_path)"
while IFS= read -r row; do
[[ -n "${row//[[:space:]]/}" ]] || continue
IFS=',' read -r bus_id driver <<<"$row"
bus_id="$(normalize_nvidia_pci_bus_id "$bus_id")" || continue
station_pci_device_is_gb300 "$bus_id" "$pci_root" || continue
printf '%s' "${driver//[[:space:]]/}"
return 0
done <<<"$rows"
}

driver_is_loaded() {
Expand Down Expand Up @@ -780,13 +808,48 @@ install_exact_file_or_reuse() {
info "${label}=installed path=${target}"
}

ensure_docker_repository_source() {
local docker_asc=$1 docker_gpg=$2 docker_gpg_list=$3 docker_asc_list=$4
local source_target=/etc/apt/sources.list.d/docker.list
local gpg_key_target=/etc/apt/keyrings/docker.gpg
local asc_key_target=/etc/apt/keyrings/docker.asc

sudo test ! -L "$source_target" \
|| fatal "Docker repository source must not be a symbolic link: ${source_target}"
if ! sudo test -e "$source_target"; then
install_exact_file_or_reuse "$docker_gpg" "$gpg_key_target" 0644 docker_repository_key
install_exact_file_or_reuse "$docker_gpg_list" "$source_target" 0644 docker_repository_source
return 0
fi

assert_root_regular_file_safe "$source_target" 0644 "Docker repository source"
if sudo cmp -s "$docker_gpg_list" "$source_target"; then
assert_root_regular_file_safe "$gpg_key_target" 0644 "Docker repository key"
sudo cmp -s "$docker_gpg" "$gpg_key_target" \
|| fatal "Existing Docker repository key differs from the verified dearmored key: ${gpg_key_target}"
info "docker_repository_source=exact path=${source_target}"
return 0
fi

if sudo cmp -s "$docker_asc_list" "$source_target"; then
assert_root_regular_file_safe "$asc_key_target" 0644 "Docker repository ASCII key"
sudo cmp -s "$docker_asc" "$asc_key_target" \
|| fatal "Existing Docker repository ASCII key differs from the verified key: ${asc_key_target}"
info "docker_repository_source=verified_compatible path=${source_target}"
return 0
fi

fatal "Existing Docker repository source differs from the validated .gpg and .asc forms; refusing to overwrite ${source_target}"
}

configure_repositories() {
local tmp cuda_deb docker_asc docker_gpg docker_list
local tmp cuda_deb docker_asc docker_gpg docker_gpg_list docker_asc_list
tmp="$(mktemp -d)"
cuda_deb="${tmp}/cuda-keyring.deb"
docker_asc="${tmp}/docker.asc"
docker_gpg="${tmp}/docker.gpg"
docker_list="${tmp}/docker.list"
docker_gpg_list="${tmp}/docker-gpg.list"
docker_asc_list="${tmp}/docker-asc.list"

info "Downloading and verifying official repository keys"
ensure_cuda_keyring "$cuda_deb"
Expand All @@ -797,11 +860,13 @@ configure_repositories() {
gpg --batch --yes --dearmor --output "$docker_gpg" "$docker_asc"
ensure_root_directory_safe /etc/apt/keyrings /etc/apt 0755 "Docker repository key directory"
assert_root_directory_safe /etc/apt/sources.list.d "Docker repository source directory"
install_exact_file_or_reuse "$docker_gpg" /etc/apt/keyrings/docker.gpg 0644 docker_repository_key
printf '%s\n' \
'deb [arch=arm64 signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu noble stable' \
>"$docker_list"
install_exact_file_or_reuse "$docker_list" /etc/apt/sources.list.d/docker.list 0644 docker_repository_source
>"$docker_gpg_list"
printf '%s\n' \
'deb [arch=arm64 signed-by=/etc/apt/keyrings/docker.asc] https://download.docker.com/linux/ubuntu noble stable' \
>"$docker_asc_list"
ensure_docker_repository_source "$docker_asc" "$docker_gpg" "$docker_gpg_list" "$docker_asc_list"

rm -rf "$tmp"
info "repository_keys=verified"
Expand Down Expand Up @@ -1100,29 +1165,35 @@ verify_apply_state() {
}

verify_gpu() {
local rows row name driver corrected uncorrected gpu_count=0
local rows row bus_id name driver corrected uncorrected gb300_count=0 pci_root
rows="$(nvidia-smi \
--query-gpu=name,driver_version,ecc.errors.corrected.volatile.total,ecc.errors.uncorrected.volatile.total \
--query-gpu=pci.bus_id,name,driver_version,ecc.errors.corrected.volatile.total,ecc.errors.uncorrected.volatile.total \
--format=csv,noheader,nounits)" || fatal "nvidia-smi failed"
pci_root="$(station_pci_devices_path)"
while IFS= read -r row; do
[[ -n "${row//[[:space:]]/}" ]] || continue
IFS=',' read -r name driver corrected uncorrected <<<"$row"
IFS=',' read -r bus_id name driver corrected uncorrected <<<"$row"
bus_id="$(normalize_nvidia_pci_bus_id "$bus_id")" \
|| fatal "nvidia-smi returned an invalid PCI bus ID: ${bus_id}"
name="${name#"${name%%[![:space:]]*}"}"
driver="${driver//[[:space:]]/}"
corrected="${corrected//[[:space:]]/}"
uncorrected="${uncorrected//[[:space:]]/}"
[[ "$name" == *"GB300"* ]] || fatal "Expected NVIDIA GB300, found ${name}"
if ! station_pci_device_is_gb300 "$bus_id" "$pci_root"; then
info "gpu_bdf=${bus_id} gpu=${name} role=auxiliary validation=skipped"
continue
fi
[[ -n "$driver" ]] || fatal "NVIDIA driver is not loaded"
if [[ "$STATION_HOST_PROFILE" == "generic-ubuntu" ]]; then
[[ "$driver" == "$DRIVER_VERSION" ]] \
|| fatal "Expected driver ${DRIVER_VERSION}, found ${driver}"
fi
Comment thread
coderabbitai[bot] marked this conversation as resolved.
Outdated
[[ "$corrected" == "0" && "$uncorrected" == "0" ]] \
|| fatal "ECC must be 0/0, found corrected=${corrected} uncorrected=${uncorrected}"
((gpu_count += 1))
info "gpu_index=${gpu_count} gpu=${name} driver=${driver} ecc_corrected=${corrected} ecc_uncorrected=${uncorrected}"
((gb300_count += 1))
info "gpu_bdf=${bus_id} gpu=${name} driver=${driver} ecc_corrected=${corrected} ecc_uncorrected=${uncorrected}"
done <<<"$rows"
((gpu_count > 0)) || fatal "nvidia-smi returned no GPU rows"
((gb300_count > 0)) || fatal "nvidia-smi returned no GB300 GPU matching the PCI identity"
}

verify_host() {
Expand Down
57 changes: 57 additions & 0 deletions test/install-express-prompt.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,24 @@ import { describe, expect, it } from "vitest";
import { INSTALLER_PAYLOAD, TEST_SYSTEM_PATH } from "./helpers/installer-sourced-env";

describe("installer express install prompt (sourced)", () => {
function runInstallerSourced(body: string) {
const home = fs.mkdtempSync(path.join(os.tmpdir(), "nemoclaw-express-sourced-"));
const result = spawnSync(
"bash",
["--noprofile", "--norc", "-c", `source "$INSTALLER_UNDER_TEST" >/dev/null\n${body}`],
{
cwd: path.resolve(import.meta.dirname, ".."),
encoding: "utf-8",
env: {
HOME: home,
PATH: TEST_SYSTEM_PATH,
INSTALLER_UNDER_TEST: INSTALLER_PAYLOAD,
},
},
);
return { result, output: `${result.stdout}${result.stderr}` };
}

function runExpressPromptWithTty(
answer: string,
stdinMode: "pipe" | "tty",
Expand Down Expand Up @@ -317,6 +335,45 @@ detect_express_platform
expect(output).toMatch(/STATION_EXPRESS=1/);
});

it("keeps Station preparation details in the log while showing warnings and errors", () => {
const { result, output } = runInstallerSourced(`
printf '%s\n' \
'[station-prepare] 2026-07-17T07:59:07Z version=2026-07-17.4 mode=--apply log=/tmp/station-prepare.log' \
'[station-prepare] 2026-07-17T07:59:07Z platform=Dell Pro Max with Station GB300 profile=generic-ubuntu' \
'[station-prepare] 2026-07-17T07:59:08Z WARNING: condition-qualified generic-image failed unit: cloud-init.service' \
'NVIDIA-SMI 610.43.02' \
'[station-prepare] 2026-07-17T07:59:20Z ERROR: example failure' \
| filter_station_host_preparation_output
`);

expect(result.status, output).toBe(0);
expect(output).toContain("DGX Station host preparation log: /tmp/station-prepare.log");
expect(output).toContain("condition-qualified generic-image failed unit: cloud-init.service");
expect(output).toContain("ERROR: example failure");
expect(output).not.toMatch(/platform=Dell Pro Max|NVIDIA-SMI/);
});

it("preserves the Station helper exit status while filtering installer output", () => {
const { result, output } = runInstallerSourced(`
bash() {
printf '%s\n' \
'[station-prepare] 2026-07-17T07:59:07Z version=2026-07-17.4 mode=--apply log=/tmp/station-prepare.log' \
'[station-prepare] 2026-07-17T07:59:08Z runtime_setup=complete'
return 10
}
if run_station_host_preparation; then
printf 'STATUS=0\n'
else
printf 'STATUS=%s\n' "$?"
fi
`);

expect(result.status, output).toBe(0);
expect(output).toContain("STATUS=10");
expect(output).toContain("DGX Station host preparation log: /tmp/station-prepare.log");
expect(output).not.toContain("runtime_setup=complete");
});

it("normalizes the canonical Ultra served alias to the registered model slug", () => {
const result = runExpressPromptWithTty("\n", "pipe", "DGX Station", {
NEMOCLAW_VLLM_MODEL: "nvidia/nemotron-3-ultra-550b-a55b",
Expand Down
26 changes: 20 additions & 6 deletions test/install-station-dgx-os.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -323,7 +323,8 @@ run_apply
STATION_PREPARE,
`
STATION_HOST_PROFILE=stock-dgx-os
nvidia-smi() { printf 'NVIDIA GB300, 595.71.05, 0, 0\n'; }
station_pci_device_is_gb300() { return 0; }
nvidia-smi() { printf '00000000:01:00.0, NVIDIA GB300, 595.71.05, 0, 0\n'; }
verify_gpu
`,
);
Expand All @@ -334,7 +335,8 @@ verify_gpu
STATION_PREPARE,
`
STATION_HOST_PROFILE=generic-ubuntu
nvidia-smi() { printf 'NVIDIA GB300, 595.71.05, 0, 0\n'; }
station_pci_device_is_gb300() { return 0; }
nvidia-smi() { printf '00000000:01:00.0, NVIDIA GB300, 595.71.05, 0, 0\n'; }
verify_gpu
`,
);
Expand Down Expand Up @@ -430,18 +432,30 @@ verify_dgx_os_runtime_sudo
});

it.each([
["wrong GPU", "NVIDIA GB200, 595.71.05, 0, 0", /Expected NVIDIA GB300/],
["non-zero volatile ECC", "NVIDIA GB300, 595.71.05, 1, 0", /ECC must be 0\/0/],
[
"a missing GB300 PCI identity",
"00000000:01:00.0, NVIDIA GB300, 595.71.05, 0, 0",
"return 1",
/no GB300 GPU matching the PCI identity/,
],
[
"non-zero volatile ECC",
"00000000:01:00.0, NVIDIA GB300, 595.71.05, 1, 0",
"return 0",
/ECC must be 0\/0/,
],
[
"a failing second GPU row",
"NVIDIA GB300, 595.71.05, 0, 0\nNVIDIA GB300, 595.71.05, 0, 1",
"00000000:01:00.0, NVIDIA GB300, 595.71.05, 0, 0\n00000000:02:00.0, NVIDIA GB300, 595.71.05, 0, 1",
"return 0",
/ECC must be 0\/0/,
],
])("fails stock validation for %s", (_scenario, row, message) => {
])("fails stock validation for %s", (_scenario, row, pciResult, message) => {
const { result, output } = runSourced(
STATION_PREPARE,
`
STATION_HOST_PROFILE=stock-dgx-os
station_pci_device_is_gb300() { ${pciResult}; }
nvidia-smi() { printf '%s\n' "$GPU_ROW"; }
verify_gpu
`,
Expand Down
Loading
Loading