[Studio] Add --with-llama-cpp-dir installer flag to reuse a local llama.cpp#6472
Conversation
Users can now pass --with-llama-cpp-dir /path/to/llama.cpp to the installer to skip downloading or building llama.cpp and use a local directory instead. A junction (Windows) or symlink (Linux/macOS) is created at the canonical install location, bypassing both the prebuilt download (Phase 3) and source build (Phase 4) steps in setup.ps1/setup.sh. The path is passed via UNSLOTH_LOCAL_LLAMA_CPP_DIR env var which setup.ps1 and setup.sh read directly. Ported from the idea in unslothai#4384, reimplemented against current Studio architecture.
Cross-checks install.sh, install.ps1, studio/setup.sh and studio/setup.ps1 so the flag's contract (parse -> UNSLOTH_LOCAL_LLAMA_CPP_DIR env var -> link local dir, skip prebuilt download and source build) can't silently regress. Wired into studio-backend-ci.yml alongside the other tests/sh installer tests.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces the --with-llama-cpp-dir flag across both PowerShell and Bash installation and setup scripts, allowing users to link a local llama.cpp directory and bypass prebuilt downloads or source compilation. A static analysis test is also added to ensure consistent wiring of this flag. The review feedback highlights a critical data loss risk in studio/setup.ps1 when using Remove-Item -Recurse on directory junctions, as well as potential path resolution corruption in the Bash scripts due to CDPATH interference.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| if (Test-Path -LiteralPath $LlamaCppDir) { | ||
| Remove-Item -Recurse -Force -LiteralPath $LlamaCppDir -ErrorAction SilentlyContinue | ||
| } |
There was a problem hiding this comment.
Critical Bug: Potential Data Loss via Remove-Item -Recurse on Junctions
In PowerShell (especially 5.1), running Remove-Item -Recurse -Force on a directory junction or symbolic link can traverse the link and delete the contents of the target directory (the user's actual local llama.cpp directory) instead of just deleting the junction/link itself. This can lead to catastrophic data loss for users utilizing the --with-llama-cpp-dir flag.
To prevent this, check if the target is a ReparsePoint (junction/symlink) and delete it safely using the .Delete() method of the DirectoryInfo object, which deletes only the link itself.
if (Test-Path -LiteralPath $LlamaCppDir) {
$item = Get-Item -LiteralPath $LlamaCppDir
if ($item.Attributes -band [System.IO.FileAttributes]::ReparsePoint) {
$item.Delete()
} else {
Remove-Item -Recurse -Force -LiteralPath $LlamaCppDir -ErrorAction SilentlyContinue
}
}
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 35542c6237
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if [ "$_STUDIO_HOME_IS_CUSTOM" = true ]; then | ||
| _assert_studio_owned_or_absent "$LLAMA_CPP_DIR" "llama.cpp install" | ||
| fi | ||
| rm -rf "$LLAMA_CPP_DIR" | ||
| ln -sfn "$_RESOLVED_LOCAL" "$LLAMA_CPP_DIR" |
There was a problem hiding this comment.
For a custom UNSLOTH_STUDIO_HOME, the first --with-llama-cpp-dir run creates $LLAMA_CPP_DIR as a symlink to the user's local directory without adding the Studio-owned marker. On the next run with the same flag, this ownership assertion follows that symlink and rejects the user's directory as unowned before it can simply replace or reuse the link, making the installer non-idempotent for custom-home local llama.cpp installs.
Useful? React with 👍 / 👎.
| STUDIO_LOCAL_INSTALL=1 \ | ||
| STUDIO_LOCAL_REPO="$_REPO_ROOT" \ | ||
| UNSLOTH_NO_TORCH="$SKIP_TORCH" \ | ||
| UNSLOTH_LOCAL_LLAMA_CPP_DIR="$_WITH_LLAMA_CPP_DIR" \ |
There was a problem hiding this comment.
When the POSIX installer is used in the documented piped-install style with UNSLOTH_LOCAL_LLAMA_CPP_DIR=/path sh (or the variable is otherwise exported before invoking install.sh), _WITH_LLAMA_CPP_DIR remains empty unless the CLI flag is used, and this assignment shadows the caller's value with an empty string before setup.sh runs. That makes the env-var path that setup.sh now supports unreachable through install.sh, so web installs silently download/build llama.cpp instead of reusing the requested local directory.
Useful? React with 👍 / 👎.
| _LOCAL_LLAMA_CPP_LINKED=true | ||
| _NEED_LLAMA_SOURCE_BUILD=false | ||
| _SKIP_PREBUILT_INSTALL=true |
There was a problem hiding this comment.
For a normal local llama.cpp source build on Linux/macOS, build/bin/llama-quantize exists but the root-level llama-quantize shim usually does not. Setting _NEED_LLAMA_SOURCE_BUILD=false here skips the existing source-build reuse block that creates that shim, while the setup comments later note GGUF export's check_llama_cpp() looks for it at the root, so export can remain unavailable even though the linked build contains the binary.
Useful? React with 👍 / 👎.
- setup.ps1: delete an existing junction/symlink via DirectoryInfo.Delete() instead of a recursive remove, which can traverse the link and wipe the user's real llama.cpp directory on PowerShell 5.1. - setup.ps1: short-circuit the build chain when a local dir is linked so CMake never runs inside the user's checkout when it lacks a Windows-layout binary. - install.sh / setup.sh: resolve paths with CDPATH= cd -P so a set CDPATH cannot corrupt the resolved path. - install.sh: seed _WITH_LLAMA_CPP_DIR from UNSLOTH_LOCAL_LLAMA_CPP_DIR so an exported env var (piped-install style) is honored instead of being clobbered. - setup.sh: create the root llama-quantize shim when linking a local source build so GGUF export's check_llama_cpp() still finds it. - setup.sh / setup.ps1: drop a stale link before the custom-home ownership assert so re-runs with the flag stay idempotent. - test: pin the new linked-dir build short-circuit.
# Conflicts: # studio/setup.ps1
|
tested on windows 11, the installed a branch build into an isolated studio home pointed at a llama.cpp i already had on disk. what i saw:
how to use it (contributors / testing from a checkout): .\install.ps1 --local --with-llama-cpp-dir "C:\path\to\your\llama.cpp"./install.sh --local --with-llama-cpp-dir /path/to/llama.cppon the released installer you drop one thing worth flagging, not a blocker: the in app "update llama.cpp" button doesnt know about the junction this flag makes. after linking, clicking update swaps a real build over the junction and quietly drops the custom path link. on python 3.13 your real folder is safe since rmtree refuses junctions, but on older pythons that same path used to walk into junctions and could delete the linked folder. might be worth having the updater notice when the install dir is a reparse point. |
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0d16c6a6bd
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if [ "$_STUDIO_HOME_IS_CUSTOM" = true ]; then | ||
| _assert_studio_owned_or_absent "$LLAMA_CPP_DIR" "llama.cpp install" | ||
| fi | ||
| rm -rf "$LLAMA_CPP_DIR" |
There was a problem hiding this comment.
When $HOME contains a symlink, install.sh canonicalizes --with-llama-cpp-dir ~/.unsloth/llama.cpp before forwarding it, but LLAMA_CPP_DIR here is still the textual $HOME/.unsloth/llama.cpp. The self-check can therefore miss the canonical install directory and this rm -rf deletes the very local llama.cpp tree the user asked to reuse, then replaces it with a broken/self-referential symlink. Compare against a resolved canonical form of LLAMA_CPP_DIR before removing it.
Useful? React with 👍 / 👎.
| --no-torch) _NO_TORCH_FLAG=true ;; | ||
| --verbose|-v) _VERBOSE=true ;; | ||
| --shortcuts-only) _SHORTCUTS_ONLY=true ;; | ||
| --with-llama-cpp-dir) _next_is_llama_cpp_dir=true ;; |
There was a problem hiding this comment.
If --with-llama-cpp-dir is the final argument, _next_is_llama_cpp_dir remains true after the loop and no error is raised, so _WITH_LLAMA_CPP_DIR stays empty and the installer silently falls back to the normal prebuilt/source install path. This makes a mistyped invocation behave as if the new flag was not supplied; add a post-loop check for the pending argument state, as the PowerShell installer does.
Useful? React with 👍 / 👎.
| --no-torch) _NO_TORCH_FLAG=true ;; | ||
| --verbose|-v) _VERBOSE=true ;; | ||
| --shortcuts-only) _SHORTCUTS_ONLY=true ;; | ||
| --with-llama-cpp-dir) _next_is_llama_cpp_dir=true ;; |
There was a problem hiding this comment.
This parser records --with-llama-cpp-dir, but the Strix Halo WSL reroute path later reconstructs _rr_args for the Ubuntu-24.04 child install without appending this option or exporting UNSLOTH_LOCAL_LLAMA_CPP_DIR. On those rerouted installs, a user-provided local llama.cpp directory is silently ignored and the child falls back to downloading/building instead of reusing the requested cache.
Useful? React with 👍 / 👎.
|
/gemini review |
There was a problem hiding this comment.
Code Review
This pull request introduces the --with-llama-cpp-dir flag to the installation and setup scripts (install.sh, install.ps1, studio/setup.sh, and studio/setup.ps1), allowing users to link a local llama.cpp directory and skip prebuilt downloads or source builds. It also adds a static analysis test to verify the flag's wiring. The review feedback highlights a few improvements: using -LiteralPath in PowerShell's Copy-Item to avoid wildcard expansion issues, updating the corresponding test assertion, and adding validation in install.sh to handle cases where the flag is provided as the last argument without a path.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
| --no-torch) _NO_TORCH_FLAG=true ;; | ||
| --verbose|-v) _VERBOSE=true ;; | ||
| --shortcuts-only) _SHORTCUTS_ONLY=true ;; | ||
| --with-llama-cpp-dir) _next_is_llama_cpp_dir=true ;; |
There was a problem hiding this comment.
If --with-llama-cpp-dir is passed as the last argument to install.sh, the loop terminates with _next_is_llama_cpp_dir set to true but without assigning any value to _WITH_LLAMA_CPP_DIR. Unlike --package and --python, there is no post-loop validation to catch this, which can lead to silent failures or unexpected behavior.
Consider adding a validation check after the loop (around line 268) to ensure that _next_is_llama_cpp_dir is not left as true.
- install.sh: error when --with-llama-cpp-dir is the final arg with no path, matching the existing --package/--python post-loop guards (was a silent fallback to the normal prebuilt/source install). - studio/setup.sh: canonicalize LLAMA_CPP_DIR before the self-link no-op compare. _RESOLVED_LOCAL is fully resolved while LLAMA_CPP_DIR was textual, so a symlinked $HOME made the guard miss and the rm -rf could wipe the user's real llama.cpp tree. - studio/setup.sh: make the llama-quantize shim non-fatal; it writes through the link into the user's tree, which may be read-only (shared/CI cache), and under set -e a failed ln aborted an otherwise-good reuse. - studio/setup.ps1: detect a broken junction via Get-Item -Force instead of Test-Path so a dangling link from a prior run is removed and mklink can relink to a new valid directory. - studio/setup.ps1: use Copy-Item -LiteralPath so a source path containing [ ] isn't treated as a wildcard in the junction copy fallback. - tests: update the wiring assertions for the LiteralPath copy and the canonicalized compare.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 56b1e3c4a4
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if (Test-Path -LiteralPath $LlamaCppDir) { | ||
| Remove-Item -Recurse -Force -LiteralPath $LlamaCppDir -ErrorAction SilentlyContinue | ||
| } |
There was a problem hiding this comment.
When the existing $LlamaCppDir is in use on Windows, locked files can make this removal fail, but -ErrorAction SilentlyContinue masks the failure and the code proceeds to mklink/Copy-Item with the old directory still present. In that scenario the local runtime is not cleanly linked and the installer can leave a partially deleted or merged llama.cpp tree instead of reporting the active-process condition like the prebuilt path does.
Useful? React with 👍 / 👎.
| step "llama.cpp" "linked local directory: $_RESOLVED_LOCAL" | ||
| _LOCAL_LLAMA_CPP_LINKED=true | ||
| _NEED_LLAMA_SOURCE_BUILD=false | ||
| _SKIP_PREBUILT_INSTALL=true |
There was a problem hiding this comment.
When UNSLOTH_LOCAL_LLAMA_CPP_DIR points at an existing but unbuilt or wrong-platform llama.cpp checkout, this branch still marks it linked and disables both the prebuilt download and the source build. Studio later only discovers llama-server in the managed root/build layouts, so this installer run can report success while leaving GGUF inference/export unusable; fail here unless the linked tree contains a supported llama-server binary.
Useful? React with 👍 / 👎.
| _assert_studio_owned_or_absent "$LLAMA_CPP_DIR" "llama.cpp install" | ||
| fi | ||
| rm -rf "$LLAMA_CPP_DIR" | ||
| ln -sfn "$_RESOLVED_LOCAL" "$LLAMA_CPP_DIR" |
There was a problem hiding this comment.
This creates the managed $LLAMA_CPP_DIR as a symlink to the user's checkout without any marker that it is externally managed. The in-app updater treats a markerless active binary under a parent named llama.cpp as a manageable source build in studio/backend/utils/llama_cpp_update.py::_llama_install_root, so the “Update llama.cpp” action can install a prebuilt into this path and replace the link instead of continuing to use the requested checkout; add a marker or make the updater skip these local-link installs.
Useful? React with 👍 / 👎.
Addresses the second Codex pass on the --with-llama-cpp-dir flag: - Validate the linked tree before disabling installs (setup.sh + setup.ps1): reusing a local dir skips BOTH the prebuilt download and the source build, so the dir must already contain a runnable llama-server (build/bin on Linux/macOS, build\bin\Release\llama-server.exe on Windows). Bail out with a clear message instead of linking an unbuilt/wrong-platform checkout and leaving Studio with no usable binary. - Treat a canonical-path target as already linked when it holds a build (setup.sh + setup.ps1): point the flag at ~/.unsloth/llama.cpp itself and an existing build is reused (skip prebuilt + source) rather than clobbered by the staged prebuilt installer (which uses os.replace()/replace). An empty canonical dir still falls through to the normal in-place install. - Abort when an in-use llama.cpp can't be removed on Windows (setup.ps1): Remove-Item -ErrorAction SilentlyContinue can silently leave a locked tree in place; detect that and stop with the same active-process message + exit 3 the prebuilt path uses, instead of junctioning over a half-present dir. Left as follow-up (already tracked by the PR author as a non-blocker): the in-app "Update llama.cpp" updater does not yet recognize a local-link install as externally managed; that fix belongs in studio/backend/utils/llama_cpp_update.py.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d5e9f5121b
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 43f20829cc
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| _assert_studio_owned_or_absent "$LLAMA_CPP_DIR" "llama.cpp install" | ||
| fi | ||
| rm -rf "$LLAMA_CPP_DIR" | ||
| ln -sfn "$_RESOLVED_LOCAL" "$LLAMA_CPP_DIR" |
There was a problem hiding this comment.
For local-dir installs that create this symlink, the backend's orphan cleanup resolves known install roots before comparing process executables (core/inference/llama_cpp.py lines 7140-7176). If the user already has their own llama-server running from the same checkout, the resolved Studio root becomes $_RESOLVED_LOCAL, so LlamaCppBackend.__init__ classifies that external process as ours and kills it on startup; keep local-link roots out of the orphan allowlist or mark them unmanaged.
Useful? React with 👍 / 👎.
…ation The linked-tree validation only accepted build/bin[/Release]/llama-server, but LlamaCppBackend._layout_candidates() resolves a root-level llama-server first, then build/bin, then build/bin/Release on Windows. A `make` build or a flat release extract (binary at the dir root) was therefore rejected with a hard installer failure even though Studio would have run it. Validate the same candidate set the backend uses in both setup scripts, and add wiring-test assertions so the check can't silently narrow again.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d254f7e035
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| cmd /c "mklink /J `"$LlamaCppDir`" `"$ResolvedLocal`"" 2>&1 | Out-Null | ||
| if ($LASTEXITCODE -ne 0) { | ||
| substep "Could not create directory junction; copying instead..." "Yellow" | ||
| Copy-Item -Recurse -LiteralPath $ResolvedLocal -Destination $LlamaCppDir |
There was a problem hiding this comment.
When UNSLOTH_STUDIO_HOME is custom and mklink /J fails, this fallback creates a real $LlamaCppDir directory but never writes the .unsloth-studio-owned marker. A later run with the same --with-llama-cpp-dir hits the non-reparse path, runs Assert-StudioOwnedOrAbsent before deletion, and rejects the copied directory as unowned, so the documented copy fallback is not idempotent.
Useful? React with 👍 / 👎.
A --with-llama-cpp-dir install junctions/symlinks the canonical llama.cpp dir to the user's own checkout, but two backend paths still treated it as a Studio-owned tree: - The in-app updater (llama_cpp_update) offered and could apply an official prebuilt over the link, writing through it into the user's checkout (or failing) and silently dropping the link the flag created. - Orphan cleanup (LlamaCppBackend._kill_orphaned_servers) resolved the linked root into its kill allowlist, so a llama-server the user launched from the same checkout was classified as ours and killed on startup. Detect the canonical dir being a symlink/junction (reparse point) and treat the install as unmanaged: get_update_status reports unsupported, start_update refuses with reason "local_link", and the linked root is left out of the orphan allowlist. Adds behavioral tests (link vs plain dir, updater refusal, and the spared-vs-killed orphan control).
for more information, see https://pre-commit.ci
The existing tests/sh/test_with_llama_cpp_dir_flag.sh is a static grep of the scripts. This adds a behavioral test that extracts the real link block from studio/setup.sh (by content anchors, with a self-validating extraction) and runs it against hermetic fake dirs, asserting the outcomes that matter: - an external CMake build links and arms neither the prebuilt download nor the source build - a flat / make tree (root-level llama-server, no build/bin) is accepted too - an unbuilt tree is rejected with a non-zero exit and no link left behind - relinking over a stale link preserves the target's contents (no data loss) - pointing at the canonical path is a no-op reuse, not a self-referential link Symlink-identity checks run only where real symlinks exist (skipped on Windows git-bash copy-mode); the link/skip/no-data-loss checks run everywhere. Wired into studio-backend-ci.yml next to the static test.
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: be8a3c38e1
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
The new orphan-cleanup tests import psutil for the process scan, but the Backend CI deps step installed studio.txt plus a fixed extras list that omits it, so the two tests failed with ModuleNotFoundError. Add psutil to both backend pytest dep steps (kept in shared shape), and guard the import with pytest.importorskip so a minimal env without psutil skips these tests instead of erroring.
CI passing here. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 829b3b10eb
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if _is_external_link(root): | ||
| continue |
There was a problem hiding this comment.
When --with-llama-cpp-dir points at a checkout that is also one of the other known roots, e.g. a local/dev install reusing <repo>/llama.cpp, this guard only skips the canonical symlink/junction root. The same target is still added earlier as project_root / "llama.cpp", then resolved into resolved_roots, so an externally launched llama-server from the requested local checkout is still classified as Studio-owned and killed. Fresh evidence beyond the earlier local-link concern is that the new guard filters each root before resolving it but does not remove duplicate resolved targets already present in install_roots.
Useful? React with 👍 / 👎.
| if _is_external_link(parent): | ||
| return True | ||
| if parent.name == "llama.cpp": |
There was a problem hiding this comment.
When the user points --with-llama-cpp-dir at the canonical llama.cpp directory itself, setup now treats it as reused but creates no symlink/junction marker. This updater guard only recognizes actual external links, so that plain canonical source tree later falls through the markerless source-build update path and the in-app Update action can replace the local build the installer explicitly skipped; persist an unmanaged marker for this no-op reuse path or teach the updater to detect it.
Useful? React with 👍 / 👎.
|
Manually re-tested this end-to-end on WSL/Linux using an isolated UNSLOTH_STUDIO_HOME and a real prebuilt llama.cpp directory passed via --with-llama-cpp-dir. Working well, some pending comments may need to be addressed. |
|
Looks like all the relevant comments are addressed, merging. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8d398a2d94
ℹ️ About Codex in GitHub
Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".
| if (-not (Test-Path -LiteralPath $WithLlamaCppDir -PathType Container)) { | ||
| Write-Host "[ERROR] --with-llama-cpp-dir path does not exist: $WithLlamaCppDir" -ForegroundColor Red | ||
| return (Exit-InstallFailure "--with-llama-cpp-dir path does not exist.") |
There was a problem hiding this comment.
When --with-llama-cpp-dir is mistyped in a PowerShell session that already has UNSLOTH_STUDIO_HOME or the installer state env vars set, this validation returns after the code above has removed/overwritten those env vars but before the try/finally that restores them starts. That leaves the caller's session pointed back at the default Studio home (or with stale STUDIO_LOCAL_INSTALL/skip flags), so a subsequent setup in the same shell can install into the wrong root; move this validation before mutating env vars or include it inside the restore finally.
Useful? React with 👍 / 👎.
2 participants
Closes #4379
Supersedes #4384 (credit to @wmeddie for the original implementation and for filing the feature request)
What this does
Adds a
--with-llama-cpp-dir <path>flag to both installers (install.ps1andinstall.sh). Point it at a llama.cpp directory you already have and the installer will use that instead of downloading a prebuilt or building from source:mklink /J) at the canonical~\.unsloth\llama.cpp, and falls back to a plain copy if the junction can't be made.ln -sfn) at~/.unsloth/llama.cpp.The path is handed from the installer to setup through the
UNSLOTH_LOCAL_LLAMA_CPP_DIRenv var, whichstudio/setup.ps1andstudio/setup.shread directly.Why
Issue #4379 asks for a way to reuse an existing llama.cpp build instead of re-downloading or rebuilding it every time. That's handy if you keep your own llama.cpp checkout, you're on an air-gapped box, or your CI already caches a known good binary.
This is the same idea as #4384, rebuilt against the installer as it looks today. A lot has changed in Studio's setup since that PR went up: staged prebuilt installs, per-gfx ROCm bundles, the ownership checks for a custom
STUDIO_HOME, and theinstall.ps1/setup.ps1split. So rather than rebase the old patch, this reimplements the flag to fit the current flow.Behavior notes
UNSLOTH_STUDIO_HOME, the existing "studio owned or absent" assertion still runs before the target is replaced.install.ps1clearsUNSLOTH_LOCAL_LLAMA_CPP_DIRafter the setup run so a later setup in the same shell doesn't pick up a stale link.Tests
Adds
tests/sh/test_with_llama_cpp_dir_flag.sh, a static wiring test that cross-checks all four scripts (install.sh,install.ps1,studio/setup.sh,studio/setup.ps1) so the flag's contract can't quietly regress: parse the flag, forward the env var, link the dir, skip download and build, clean up afterward. It's wired intostudio-backend-ci.ymlnext to the othertests/sh/installer tests and needs no Python, GPU, or network.Manual verification
Tested on Windows with an AMD RX 9060 XT (gfx1200, RDNA4) against a local ROCm gfx120X llama.cpp build:
The installer prints
linked local directorythenalready built, skips the prebuilt install and the source build, and the junction at~\.unsloth\llama.cpppoints at the local directory.llama-server.exe --versionand GGUF inference in Studio both run off the linked binary.