Make a single GCS object usable as Data - #314
Conversation
`Data("gs://bucket/dir/file.h5")` resolved to the wrong thing on both
paths, silently.
Download: `_download_data` listed the prefix `dir/file.h5/`, which
matches no object, so the ref resolved to an empty directory. It now
tries a direct object fetch first and falls back to the prefix listing,
which is what an uploaded local file needs (its ref names the
content-hash directory, not the object). An `is_dir=False` ref that
matches neither raises `FileNotFoundError` naming the URI.
FUSE: GCS FUSE mounts the object's parent, and
`_resolve_fuse_single_file` returned `os.listdir(mount)[0]` — an
arbitrary sibling whenever the parent held more than one object. It now
picks the object out by the name in the ref's URI, and raises instead of
guessing when that name is absent from a populated mount.
Covered against fake-gcs-server, with parent directories holding several
objects, plus docker-tier roundtrips that read the file back inside a
real container.
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request adds support for single-file and single-object Data references in Kinetic, ensuring they resolve directly to file paths rather than directories for both downloaded data and GCS FUSE mounts. It introduces logic to download single objects directly and locate them within FUSE-mounted parent directories, accompanied by comprehensive tests and documentation updates. The review feedback suggests optimizing the _download_object helper by removing the redundant blob.exists() check and instead catching google.cloud.exceptions.NotFound during the download attempt to eliminate an unnecessary network round-trip.
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #314 +/- ##
=======================================
Coverage ? 36.70%
=======================================
Files ? 83
Lines ? 15340
Branches ? 0
=======================================
Hits ? 5631
Misses ? 9709
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Attempt the download and catch NotFound instead of checking exists() first. google-cloud-storage opens the destination before it learns the object is missing, but deletes it again on NotFound (>= 3.10, our floor), so the miss leaves the target directory clean for the fallback prefix listing that uploaded-file refs depend on.
2 participants
Description
Data("gs://bucket/dir/file.h5")resolved to the wrong thing on both data paths, silently.Download:
_download_datalisted the prefixdir/file.h5/, which matches no object, so the ref resolved to an empty directory. It now tries a direct object fetch first and falls back to the prefix listing, which is what an uploaded local file needs (its ref names the content-hash directory, not the object). Anis_dir=Falseref that matches neither raisesFileNotFoundErrornaming the URI.FUSE: GCS FUSE mounts the object's parent, and
_resolve_fuse_single_filereturnedos.listdir(mount)[0]: an arbitrary sibling whenever the parent held more than one object. It now picks the object out by the name in the ref's URI, and raises instead of guessing when that name is absent from a populated mount.Covered against
fake-gcs-server, with parent directories holding several objects, plus docker-tier roundtrips that read the file back inside a real container.Contributor Agreement