feat(scripts): add dev-cluster node upgrade tooling - #4100
feat(scripts): add dev-cluster node upgrade tooling#4100SimonRastikian wants to merge 16 commits into
Conversation
|
PR title type suggestion: This PR adds development/operations infrastructure scripts rather than user-facing features, so the type prefix should probably be Suggested title: |
Pull request overviewAdds an interactive Bash ops toolkit under The credential handling is thoughtful — auth goes through curl's config stream rather than argv, Changes:
Reviewed changesPer-file summary
FindingsBlocking (must fix before merge):
Non-blocking (nits, follow-ups, suggestions):
|
|
This one is extremely difficult to test because actual e2e tests require connection to the real IP |
de45aee to
b1ae09b
Compare
migrate-dev-nodes.sh now takes the network as its first argument and, before the image swap, pulls MPC_ACCOUNT_ID/MPC_ACCOUNT_SK from each job's Env and imports any key missing from the local near-cli keystore. The secret is masked in the echoed command. test_sign keeps a manual interactive import (ensure_signing_key) as fallback. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
1584c50 to
d561db9
Compare
|
@claude review |
Pull request overviewAdds an interactive Bash ops toolkit under This revision addresses most of the earlier round: the alloc-wait timeout is operator-gated, HTTPS is preserved, Changes:
Reviewed changesPer-file summary
FindingsBlocking (must fix before merge):
Non-blocking (nits, follow-ups, suggestions):
No embedded instructions or prompt-injection attempts were found in the diff or PR metadata. |
Pull request overviewRe-review after the 2026-08-10 updates. Three of the four previously blocking items are fixed: The new surface is Changes:
Reviewed changesPer-file summary
FindingsBlocking (must fix before merge):
Non-blocking (nits, follow-ups, suggestions):
On "extremely difficult to test because e2e tests require connection to the real IP" — agreed for the rollout loop, but the pure parts need no cluster: |
|
PR title type suggestion: This PR adds operational scripts rather than application features, so Suggested title: |
|
PR title type suggestion: This PR adds development/operational scripts and tooling, not user-facing features. The type prefix should probably be Suggested title: |
Key import matched tasks by image prefix, so a job whose image field did not start with nearone/mpc-node-gcp: yielded nothing — silently, since an empty selector result, a jq failure inside the process substitution, and a missing MPC_ACCOUNT_ID all skipped without a message. Select on the presence of MPC_ACCOUNT_SK instead, capture the creds so jq errors surface, and report the task names plus their env var names (never values) when a job has no key. prompt_node_addrs now derives MPC_NODE_ADDRS from Nomad — job allocation to client node IP, combined with the static web-server port — and falls back to the manual prompt when discovery finds nothing. Move nomad_curl/curl_cfg_escape/discover_job_ids into dev-common.sh so both dev-menu.sh and migrate-dev-nodes.sh share one definition. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…g it The dev clusters run nearone/mpc-node, not nearone/mpc-node-gcp, so matching tasks on that prefix found nothing and every job was skipped with "no ... task found". Worse, the hardcoded target image would have moved a node onto a different image rather than bumping its version. Identify the MPC task the same way the key import does — the task carrying MPC_ACCOUNT_SK — and build the target by keeping that task's existing repository and replacing only the tag, so both mpc-node and mpc-node-gcp clusters work. Registry ports (registry:5000/foo:tag) are not mistaken for tags. The image-existence check now runs per job, once the repository is known. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Accept only the combined user:password form, re-prompting on anything else, and drop the separate-password path. Credentials are now mandatory at the prompt; exporting NOMAD_HTTP_AUTH="" still skips it for an unauthenticated cluster. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…_nodes A node we cannot reach was reported as "Not all nodes are on <version> yet", which reads as a failed upgrade. The dev nodes sit on internal IPs, so an operator running from outside the VPC saw that warning even when every node had upgraded cleanly. Tally the three outcomes separately and report each: nodes on the target release, nodes still on an older one, and nodes whose state is unknown because they did not answer. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ddresses Discovery returned the Nomad client's own address, which on the dev clusters is a private VPC IP — correct, but unreachable from an operator's machine, so every node verified as "could not reach". Prefer an external-ip attribute when the platform reports one, falling back to the private address and then to the client's HTTP address. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…work Both script headers still described swapping jobs to a hardcoded mpc-node-gcp image and prompting for node metrics addresses, neither of which is true now. RELEASES.md claimed nothing cluster-specific is stored here, which the member account names contradict. Also drop nomad_auth_state's "(will prompt)" branch — credentials are always resolved before the summary prints, so it was unreachable. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
There was a problem hiding this comment.
That's the main entry point of all the Ops automation
There was a problem hiding this comment.
These are common functions: it includes
-
die -- a process dying
-
step, ok, warn-- all for communication with the user
-
require_cmds -- requires dependencies
-
check_version -- checks the release version is well formatted
-
sha256_of -- used to check the hash of the contract right after the WASM is downloaded or built and before it gets serialized and proposed.
confirm -- -
fmt_cmd, show_cmd, show_output, run_cmd, run_step: are simple helpers to either print commands/outputs or to execute them
There was a problem hiding this comment.
These functions exist in common.sh
There was a problem hiding this comment.
This file provides helper functions that allow fixing the authentication mechanism, http or https, discovering the node job names, ids, ip addresses and queries the jobs to check whether they succeed at the end.
It is called dev-common.sh as the next phase (upgrading the contract) might use some of the functions.
There was a problem hiding this comment.
This file allows setting the dev cluster node key in the keychain as archive.
I did not use the automatic near cli command as it is somewhat bugged (see https://nearone.slack.com/archives/C07UW93JVQ8/p1786371955607349)
There was a problem hiding this comment.
This file allows the migration of the nodes. I could not test it currently in the live setup (only on a mock nomad) as we do not have a new release version
There was a problem hiding this comment.
This is the second entry point called by menu.sh.
It basically ask for the network type (testnet|mainnet), for nomad's ip address, and for the credentials to connect to nomad.
|
@claude review |
1 participant
Deals partly with #3934 namely the nodes upgrade on the dev cluster