Skip to content

Cached open_files readers remain open after context exit #2130

Description

@betacatsling

Exiting with fsspec.open_files(...) leaves the returned readers open when using simplecache or filecache.

Reproduced on master be32e2b2bb775104bc3f2bf291a87bbfedd8a273, Linux, Python 3.10, with only an in-memory source and temporary cache:

import tempfile
import fsspec

memory = fsspec.filesystem("memory")
memory.pipe("/example", b"data")
with tempfile.TemporaryDirectory() as cache:
    with fsspec.open_files("simplecache::memory:///example", cache_storage=cache) as files:
        assert files[0].read() == b"data"
    print(files[0].closed)  # False; expected True
    files[0].close()

This also happens with filecache, a warm cache, and an exception inside the context. OpenFiles.__enter__ returns the handles from open_many, but __exit__ only closes the unpopulated OpenFile wrappers. Keeping the returned list alive therefore retains open file descriptors after the context ends.

AI assistance was used to investigate and run this reproducer.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions