Added the ability to open a ZarrTable from a store path.#56
Conversation
I needed this for xarray-sql#69, please let me know if there is a better way to go about implementing this.
|
So I have some changes to how that part works in PR #54, would you rather merge this now and I will rebase and modify once I merge my PR, or wait until I merge and you rebase and make the necessary changes? I'm fine with either way. Also I didn't ask for your review on PR #54 I wasn't sure if you were working on anything in this repo but of course feel free to take a look and let me know if you have any thoughts! And I have a second PR in the works that will allow reading from s3 buckets, is that something you'll want to pick up once I merge (the PR isn't ready yet)? |
|
I'm on a vacation for the next two weeks and have much more down time than normal, so I'm game to review #54 and make other contributions! Let me look at that PR now to see if we should merge this and rebase or wait to land this change. :) I'm excited to get these sources in xarray-sql sometime during this trip. That would be very exciting to me. |
|
I'd be happy to review the S3/Cloud buckets PR, for sure. I just made a PR into your new feature branch here: #57. Happy to merge that one instead of this one. |
|
Okay should we close this one now that you opened #57? |
|
Oh I've been wanting to ask you, are you basically using some of the functionalities here without the datafusion "hookup"? I was briefly discussing with Kyle (on his earlier PR) about if we should have the arrow + zarr logic be completely stand alone or if it's fine to have it fully integrated with datafusion. |
Yes, happy to close this one.
I specifically need something like the ZarrTable or something that implements DataFusion's TableProvider interface that can read subsets of arrays from a group into a table. I've just written up what I'm looking for here: #58. I would be happy to move such logic somewhere else (the "zarrquet" library in xqlsystems/xarray-sql#69 would make me happy!) and use this library for the core scan operations on Zarr. |
I needed this for xarray-sql#69, please let me know if there is a better way to go about implementing this.