Skip to content

Add shared Xet to HTTP stall fallback (hf_xet_fallback, hf_cache_state) #829

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
danielhanchen wants to merge 82 commits into
base: main
Choose a base branch
from
Open

Add shared Xet to HTTP stall fallback (hf_xet_fallback, hf_cache_state) #829

Show file tree
Hide file tree
Changes from 77 commits
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
9982d77
Add shared Xet to HTTP stall fallback (hf_xet_fallback, hf_cache_state)
danielhanchen Jun 24, 2026
0bba571
Address review: thread cache_dir through watchdog/cleanup; drop unuse…
danielhanchen Jun 24, 2026
8af1b5d
Address review 2: set transport env in parent before spawn; cache_dir…
danielhanchen Jun 24, 2026
6266d08
Harden cleanup, watchdog and result drain (Gemini review)
danielhanchen Jun 24, 2026
2619c82
Fix spawn from notebooks, case-collision deletes, stale symlinks, for…
danielhanchen Jun 25, 2026
dc0fddb
Normalize repo_type=None and make the cache read path case-collision …
danielhanchen Jun 25, 2026
27c2795
Harden process-group kill, status callbacks, and cache_dir expansion …
danielhanchen Jun 25, 2026
e93e669
Let the download child skip unsloth_zoo's heavy GPU init (review 4 P1)
danielhanchen Jun 25, 2026
e2300e5
Address Codex review: spawn from unguarded scripts, partial-cache saf…
danielhanchen Jun 25, 2026
d53ab16
Validate the requested snapshot revision, not just the latest
danielhanchen Jun 25, 2026
a04a983
Merge remote-tracking branch 'origin/main' into hf-xet-fallback-shared
danielhanchen Jun 25, 2026
8050918
Keep the stub-injection gate on _SKIP_GPU_INIT; kill the download chi…
danielhanchen Jun 26, 2026
36d19dd
Harden spawn, cache attribution, offline, and secret scrubbing (Codex…
danielhanchen Jun 26, 2026
cb73a98
Normalize path-like cache dirs and forward subfolder (Codex P2)
danielhanchen Jun 26, 2026
5d898da
Count a fully-sparse blob as 0 bytes, not full size (Codex P2)
danielhanchen Jun 26, 2026
b46f816
Scope the snapshot fast-path validation to the requested revision (Co…
danielhanchen Jun 26, 2026
be5e809
Require weight-level completeness for a cached snapshot (Codex P1 + P2)
danielhanchen Jun 26, 2026
e105537
Reject incomplete child snapshots and spare active sibling partials (…
danielhanchen Jun 26, 2026
66a723a
Gate the snapshot weight check to full model warmups (Codex P2)
danielhanchen Jun 26, 2026
ea8f013
Honor ignore_patterns and tie the purge grace to the stall timeout (C…
danielhanchen Jun 26, 2026
d4bf5e6
Tighten snapshot weight-completeness against index-only and shard-onl…
danielhanchen Jun 26, 2026
cec935b
Scope single-file stall to child partials; exclude trainer state; pro…
danielhanchen Jun 26, 2026
047d8ee
Detect hung resumed single-file downloads; accept string allow_patterns
danielhanchen Jun 27, 2026
e2eba7c
Probe adapter / consolidated weight globs; bump version for the helper
danielhanchen Jun 27, 2026
5d72351
Validate the requested weight; treat no-slash directory globs as weig…
danielhanchen Jun 27, 2026
7bb19ab
Harden weight detection for wildcard parents, shard subsets, and sile…
danielhanchen Jun 27, 2026
2cb8d89
Recognize custom weight globs, bracket globs, and every named exact w…
danielhanchen Jun 27, 2026
8c0fd24
Probe path-qualified weight globs, skip trainer artifacts, retry tran…
danielhanchen Jun 27, 2026
f612061
Harden weight detection for empty filters and subfolders; clarify the…
danielhanchen Jun 27, 2026
69a777e
Treat Diffusers config-only components as weightless; reject checkpoi…
danielhanchen Jun 27, 2026
ae398c6
Require named non-weight files in completeness; honor cancel before t…
danielhanchen Jun 28, 2026
aaf7a2d
Harden the Xet fallback child lifecycle and malformed-index handling
danielhanchen Jun 28, 2026
5ed4779
Scope snapshot completeness to the requested files and preserve Hub e…
danielhanchen Jun 28, 2026
bd92786
Preserve Hub error types across the spawn boundary regardless of cons…
danielhanchen Jun 28, 2026
76e7507
Group weight formats, enforce named files post-download, handle dir p…
danielhanchen Jun 28, 2026
b1e81bf
Scope the weightless broken-symlink check and recognize processor met…
danielhanchen Jun 28, 2026
316fe69
Reject a diffusers pipeline snapshot missing a declared sub-model
danielhanchen Jun 28, 2026
24b03b4
Scope catch-all warms to root weights and treat processor subfolders …
danielhanchen Jun 28, 2026
588c60f
Require root-level weights for root warms and validate pipelines for …
danielhanchen Jun 28, 2026
66cb43e
Validate shard indexes for all root-wide warms and require the sharde…
danielhanchen Jun 28, 2026
74d9179
Make the sharded-index requirement variant-aware
danielhanchen Jun 28, 2026
e625b58
Recognize SentencePiece tokenizer globs and pin the parent transport …
danielhanchen Jun 28, 2026
e130ca7
Recognize variant weight globs, diffusers weight-only warms, and data…
danielhanchen Jun 28, 2026
573b978
Harden the shared completeness check and spawn against the 3-reviewer…
danielhanchen Jun 28, 2026
c288957
Trim the cache-completeness checker to a conservative canonical-only …
danielhanchen Jun 28, 2026
e0f7b43
Fix five review-round findings in the cache-completeness gate and spa…
danielhanchen Jun 29, 2026
ae3b3f0
Merge remote-tracking branch 'origin/main' into hf-xet-fallback-shared
danielhanchen Jun 29, 2026
b6ec9fd
Address second review round: weight-pattern selector, sharded/variant…
danielhanchen Jun 29, 2026
d55d320
Tighten post-download acceptance and HTTP-prep cleanup for the third …
danielhanchen Jun 29, 2026
6fe5acb
Trim comments to a succinct, load-bearing core
danielhanchen Jun 29, 2026
7db059a
Stop misclassifying two weightless request shapes as weight-bearing
danielhanchen Jun 29, 2026
b4ed391
Only canonical model/pytorch_model shard indexes gate the fast path
danielhanchen Jun 29, 2026
2ffeb7f
Clarify get_hf_download_state return-value docstring
danielhanchen Jun 30, 2026
8b58fbe
Defer the pre-download cache skip on a variant load
danielhanchen Jul 1, 2026
98c8ad1
Preserve builtin OSError subclasses across the spawn boundary
danielhanchen Jul 1, 2026
b1628a2
Tighten the post-download completeness check against offline-fallback…
danielhanchen Jul 1, 2026
b8cc019
Honor the variant check for patterned snapshot warms
danielhanchen Jul 1, 2026
18b35d6
Preserve errno across the spawn boundary for OSError subclasses
danielhanchen Jul 1, 2026
8722146
Scope the post-download shard-completeness gate to what the load reads
danielhanchen Jul 1, 2026
e24641d
Tighten the variant-shard and exact-subset post-download checks
danielhanchen Jul 1, 2026
3cf8a06
Defer an incomplete preferred index, and complete patterned-variant s…
danielhanchen Jul 1, 2026
bf24387
Apply ignore filters and request scope consistently in the acceptance…
danielhanchen Jul 1, 2026
428b5c5
Funnel the weight acceptance check through one helper with two invari…
danielhanchen Jul 1, 2026
1b877ee
Complete variant and non-root shard-set checks in the post-download gate
danielhanchen Jul 1, 2026
2757211
Tighten default-load weight presence and either-format / weightless s…
danielhanchen Jul 1, 2026
b8edaf3
Scope default and variant root-weight checks to model, not adapter
danielhanchen Jul 1, 2026
074b808
Match post-download acceptance to what a default load reads (format, …
danielhanchen Jul 1, 2026
b5e5baa
Align canonical acceptance with transformers precedence; scope diffus…
danielhanchen Jul 1, 2026
6b5c3d6
Reject non-canonical variant names, lone shards without an index, and…
danielhanchen Jul 1, 2026
f823d37
Scope diffusers variant presence, order variant single-vs-index, skip…
danielhanchen Jul 2, 2026
c64a868
Harden shard-path validation against Windows drive letters; stop quer…
danielhanchen Jul 2, 2026
a4ef8b2
Scope a malformed shard index by its weight FORMAT, not the .json fil…
danielhanchen Jul 2, 2026
61a629a
Retry a transient LocalEntryNotFoundError over the other transport
danielhanchen Jul 2, 2026
26bd3d5
Defer a bin-only cache from the strict pre-download fast path when sa…
danielhanchen Jul 3, 2026
21fcb55
Exclude variant component weights from the plain diffusers post-downl…
danielhanchen Jul 3, 2026
930d027
Accept from_tf/from_flax weights, fix the no-psutil resumed-partial w…
danielhanchen Jul 3, 2026
b2089bf
Complete the from_tf/from_flax fix for sharded weights, check explici…
danielhanchen Jul 3, 2026
fb80cf9
Handle nested checkpoint subfolders and run the fallback test suite i…
danielhanchen Jul 3, 2026
346ae86
Revert version bump to 2026.6.7
danielhanchen Jul 3, 2026
8ad533b
Trim verbose comments in the shared Xet->HTTP fallback modules
danielhanchen Jul 3, 2026
94d30ff
Reject a diffusers pipeline warm missing a declared component
danielhanchen Jul 3, 2026
8accd3b
Reject a diffusers component dir holding neither config nor weight
danielhanchen Jul 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Loading
Loading