Type hints: PEP 563 annotations on public API + low-risk internals#10
Merged
Conversation
Python 2 and 3.8 support was already removed in py4j#585; this catches up the public-facing claims: * root ``setup.py``: drop ``3.8`` classifier, add ``3.13`` (was missing from the root metadata — py4j-python/setup.py already had it). Add ``python_requires=">=3.9"`` so pip refuses to install on EOL interpreters. * ``py4j-web/install.rst``: section header + body to "3.9-3.13"; Windows path example uses generic ``python3X`` (was ``python27``). * ``py4j-web/download.rst``: "tested with 3.8+" -> "3.9+". * ``py4j-web/contributing.rst`` and root ``CONTRIBUTING.rst``: the baseline-compatibility line goes from 3.8 to 3.9. Note: the README ``shields.io/pypi/pyversions/py4j.svg`` badge reads from PyPI's classifiers — it'll auto-update to the corrected list on the next package publish; no README edit needed here. Co-authored-by: Isaac
…eption chaining)
Mechanical Python 3 syntax modernization across py4j-python/src/py4j/.
No behavior change; all 53 non-JVM unit tests pass, full matrix is the
final validation.
Categories:
* **.format() -> f-strings** (38 conversions): java_gateway.py (21),
java_collections.py (10), clientserver.py (5), protocol.py (2).
Conservative — sites with named placeholders, format specs ({0:5d}),
conversion flags ({0!r}), or escape braces ({{ }}) are deliberately
left as .format(). The remaining ~4 .format() calls are correct
as-is.
* **%-style formatting -> f-strings** (3 conversions): clientserver.py
socket-handshake address/port writes. Also collapsed
"z\n".encode("utf-8") to b"z\n" while in the neighborhood.
* **super(Cls, self) -> super()** (15 conversions): clientserver.py
(9), java_gateway.py (3), protocol.py (3 exception classes).
Py3-only syntax.
* **while(True) -> while True** (1 site in clientserver.py):
PEP 8 / Py3-only style.
* **raise X from e** (3 sites): java_gateway.py
GatewayConnection.send_command, _get_connection, and
CallbackServer.start_callback; clientserver.py
ClientServerConnection.send_command. Preserves the original cause
in tracebacks — improves debug context for PySpark users (and
anyone else) without changing the public API.
Sites NOT converted (deliberately): .format() with conversion flags
({0!r}), escape braces ({{x}}), or multi-line continuations — the
regex converter was tuned conservative to avoid the subtle bugs that
caused the failure on @markjm's py4j#575 review.
Co-authored-by: Isaac
The previous commit's mechanical .format() -> f-string converter
emitted this on java_collections.py:469:
return f"[{", ".join(items)}]"
PEP 701 (nested quotes in f-strings) only landed in Python 3.12, so
this is a SyntaxError on 3.9 / 3.10 / 3.11 — the entire module fails
to import, and any test that uses java_collections crashes at
collection time. CI surfaced this on Python 3.11, Java 8, ubuntu-
latest first; other 3.9-3.11 cells were on the same trajectory.
Refactor the call site to extract the ", ".join() into a local
binding outside the f-string. Equivalent semantics, parses cleanly
on every supported Python.
The original .format()-to-f-string converter has been mentally
patched: if a .format() arg contains a string literal, the resulting
f-string would produce nested quotes — skip such sites or extract
to a local. (No other sites in this PR have that pattern; this was
the only instance.)
Co-authored-by: Isaac
Adds annotations via PEP 563 (from __future__ import annotations) so all annotations are strings at runtime — zero overhead, zero introspection surprises for PySpark or any other consumer. Annotated: * java_gateway.py public API: GatewayParameters / CallbackServerParameters / JavaGateway __init__ signatures; launch_gateway; java_import; get_field / set_field; is_instance_of; exception class constructors (Py4JError, Py4JNetworkError, Py4JAuthenticationError, Py4JJavaError). * protocol.py: encode/decode function signatures (escape_new_line, unescape_new_line, smart_decode, encode_float, encode_bytearray, decode_bytearray, is_python_proxy, get_command_part, get_return_value) plus is_error, is_fatal_error. * signals.py: Signal.connect / disconnect / send. * finalizer.py: ThreadSafeFinalizer.add_finalizer / remove_finalizer / clear_finalizers. Deliberately NOT annotated: * JavaObject / JavaMember / JavaClass / JavaPackage / JVMView — attribute access resolves dynamically via __getattr__ + Java reflection; annotating would either lie about specific types or be pure Any. Honest stance: leave dynamic dispatch unannotated. * java_collections.py (JavaList/JavaMap/JavaSet) — same dynamic dispatch pattern. * GatewayClient / CallbackServer internal methods that touch JavaObject — same reason. mypy is not gated in CI for this PR (deferred per spec); annotations are added for IDE / static-analysis benefit. A future discussion on style / lint tooling (likely covering ruff + black + mypy together) will revisit CI gating. Co-authored-by: Isaac
This was referenced May 21, 2026
Tagar
added a commit
that referenced
this pull request
May 22, 2026
…pe hints (#10) * docs: align Python version claims to 3.9-3.13 Python 2 and 3.8 support was already removed in py4j#585; this catches up the public-facing claims: * root ``setup.py``: drop ``3.8`` classifier, add ``3.13`` (was missing from the root metadata — py4j-python/setup.py already had it). Add ``python_requires=">=3.9"`` so pip refuses to install on EOL interpreters. * ``py4j-web/install.rst``: section header + body to "3.9-3.13"; Windows path example uses generic ``python3X`` (was ``python27``). * ``py4j-web/download.rst``: "tested with 3.8+" -> "3.9+". * ``py4j-web/contributing.rst`` and root ``CONTRIBUTING.rst``: the baseline-compatibility line goes from 3.8 to 3.9. Note: the README ``shields.io/pypi/pyversions/py4j.svg`` badge reads from PyPI's classifiers — it'll auto-update to the corrected list on the next package publish; no README edit needed here. * style: modernize Python 3 syntax (super(), while True, raise from) * ``super(Cls, self) -> super()`` (15 sites): clientserver.py (9), java_gateway.py (3), protocol.py (3 exception classes). Py3-only syntax — no behavior change. * ``while(True) -> while True`` (1 site in clientserver.py): PEP 8 / Py3-only style. * ``raise X from e`` (4 sites): ``java_gateway.py`` GatewayConnection.send_command, _get_connection, and CallbackServer.start_callback; ``clientserver.py`` ClientServerConnection.send_command. Preserves the original cause in tracebacks — better debug context for PySpark users without changing the public API. Sites NOT converted: ``.format()`` calls are left as-is. PySpark itself keeps a mix of ``.format()`` / ``%`` / f-strings in master (``pyspark.sql.types`` and ``pyspark.util`` are still ``%``-style), and a mechanical sweep adds churn without value — per Hyukjin's review feedback. * types: light annotations on public API where they add real signal Added via PEP 563 (``from __future__ import annotations``) so all annotations are strings at runtime — zero overhead, zero introspection surprises for PySpark or any other consumer. Annotated with concrete types only (``str``, ``int``, ``bool``, ``JavaGateway``, ``str | bytes``, ``str | None``, ``Popen``, etc.): * ``java_gateway.py``: ``GatewayParameters`` / ``CallbackServerParameters`` / ``JavaGateway`` constructors, ``launch_gateway``, ``java_import``, ``get_field`` / ``set_field`` (only field_name as str), ``is_instance_of`` (only the JavaGateway arg). * ``protocol.py``: encode/decode function signatures (``escape_new_line``, ``unescape_new_line``, ``smart_decode``, ``encode_float``, ``encode_bytearray``, ``decode_bytearray``, ``is_python_proxy``, ``get_command_part``, ``get_return_value`` — on the parts where concrete types apply), plus ``is_error``, ``is_fatal_error``. Deliberately NOT annotated (would be ``Any``, which is pure noise per Hyukjin's review feedback): * ``finalizer.py`` — internal weak-reference dispatcher; every param would be ``Any``. * ``signals.py`` — Signal class plumbing; receivers / senders / ids are duck-typed. * JavaObject / JavaMember / JavaClass / JavaPackage / JVMView / JavaList / JavaMap / JavaSet — attribute access resolves dynamically via ``__getattr__`` + Java reflection; annotating would either lie about specific types or be pure ``Any``. * GatewayClient / CallbackServer internal methods that touch JavaObject — same reason. * The ``java_object``, ``java_class``, ``value``, ``jvm_view``, ``gateway_client`` parameters on public functions — genuinely opaque dynamic proxies, leaving them unannotated rather than forcing ``Any``. mypy is not gated in CI for this PR (deferred per spec); annotations are added for IDE / static-analysis benefit. A future discussion on style / lint tooling (likely covering ruff + black + mypy together) will revisit CI gating. Co-authored-by: Isaac
Tagar
added a commit
that referenced
this pull request
May 22, 2026
…pe hints (#10) * docs: align Python version claims to 3.9-3.13 Python 2 and 3.8 support was already removed in py4j#585; this catches up the public-facing claims: * root ``setup.py``: drop ``3.8`` classifier, add ``3.13`` (was missing from the root metadata — py4j-python/setup.py already had it). Add ``python_requires=">=3.9"`` so pip refuses to install on EOL interpreters. * ``py4j-web/install.rst``: section header + body to "3.9-3.13"; Windows path example uses generic ``python3X`` (was ``python27``). * ``py4j-web/download.rst``: "tested with 3.8+" -> "3.9+". * ``py4j-web/contributing.rst`` and root ``CONTRIBUTING.rst``: the baseline-compatibility line goes from 3.8 to 3.9. Note: the README ``shields.io/pypi/pyversions/py4j.svg`` badge reads from PyPI's classifiers — it'll auto-update to the corrected list on the next package publish; no README edit needed here. * style: modernize Python 3 syntax (super(), while True, raise from) * ``super(Cls, self) -> super()`` (15 sites): clientserver.py (9), java_gateway.py (3), protocol.py (3 exception classes). Py3-only syntax — no behavior change. * ``while(True) -> while True`` (1 site in clientserver.py): PEP 8 / Py3-only style. * ``raise X from e`` (4 sites): ``java_gateway.py`` GatewayConnection.send_command, _get_connection, and CallbackServer.start_callback; ``clientserver.py`` ClientServerConnection.send_command. Preserves the original cause in tracebacks — better debug context for PySpark users without changing the public API. Sites NOT converted: ``.format()`` calls are left as-is. PySpark itself keeps a mix of ``.format()`` / ``%`` / f-strings in master (``pyspark.sql.types`` and ``pyspark.util`` are still ``%``-style), and a mechanical sweep adds churn without value — per Hyukjin's review feedback. * types: light annotations on public API where they add real signal Added via PEP 563 (``from __future__ import annotations``) so all annotations are strings at runtime — zero overhead, zero introspection surprises for PySpark or any other consumer. Annotated with concrete types only (``str``, ``int``, ``bool``, ``JavaGateway``, ``str | bytes``, ``str | None``, ``Popen``, etc.): * ``java_gateway.py``: ``GatewayParameters`` / ``CallbackServerParameters`` / ``JavaGateway`` constructors, ``launch_gateway``, ``java_import``, ``get_field`` / ``set_field`` (only field_name as str), ``is_instance_of`` (only the JavaGateway arg). * ``protocol.py``: encode/decode function signatures (``escape_new_line``, ``unescape_new_line``, ``smart_decode``, ``encode_float``, ``encode_bytearray``, ``decode_bytearray``, ``is_python_proxy``, ``get_command_part``, ``get_return_value`` — on the parts where concrete types apply), plus ``is_error``, ``is_fatal_error``. Deliberately NOT annotated (would be ``Any``, which is pure noise per Hyukjin's review feedback): * ``finalizer.py`` — internal weak-reference dispatcher; every param would be ``Any``. * ``signals.py`` — Signal class plumbing; receivers / senders / ids are duck-typed. * JavaObject / JavaMember / JavaClass / JavaPackage / JVMView / JavaList / JavaMap / JavaSet — attribute access resolves dynamically via ``__getattr__`` + Java reflection; annotating would either lie about specific types or be pure ``Any``. * GatewayClient / CallbackServer internal methods that touch JavaObject — same reason. * The ``java_object``, ``java_class``, ``value``, ``jvm_view``, ``gateway_client`` parameters on public functions — genuinely opaque dynamic proxies, leaving them unannotated rather than forcing ``Any``. mypy is not gated in CI for this PR (deferred per spec); annotations are added for IDE / static-analysis benefit. A future discussion on style / lint tooling (likely covering ruff + black + mypy together) will revisit CI gating. Co-authored-by: Isaac
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Phase 3 of the post-py2 modernization plan — adds Python type hints via PEP 563 (
from __future__ import annotations) so annotations are strings at runtime: zero overhead, zero introspection surprises for PySpark or any other consumer.Stack note
This PR is stacked on top of #9 (
py3-modernization). It has to merge after #9 — the type annotations are layered on the syntax-modernized source. GitHub renders the PR diff against master, so until #9 merges the diff includes #9's changes too; after #9 merges, this PR's diff naturally shrinks to just the type annotations.Annotated
java_gateway.pypublic API (8 items):GatewayParameters.__init__,CallbackServerParameters.__init__,JavaGateway.__init__launch_gateway,java_import,get_field,set_field,is_instance_ofprotocol.pyinternals (12 items):escape_new_line,unescape_new_line,smart_decode,encode_float,encode_bytearray,decode_bytearray,is_python_proxy,get_command_part,get_return_value,is_error,is_fatal_errorPy4JError,Py4JAuthenticationError,Py4JNetworkError,Py4JJavaErrorsignals.py:Signal.connect,disconnect,send(3 items).finalizer.py:ThreadSafeFinalizer.add_finalizer,remove_finalizer,clear_finalizers(3 items).Deliberately NOT annotated
JavaObject/JavaMember/JavaClass/JavaPackage/JVMView— attributes resolve dynamically via__getattr__+ Java reflection. Annotating would either lie about specific types or be pureAny. Honest stance: leave dynamic dispatch unannotated.java_collections.py(JavaList,JavaMap,JavaSet) — same dynamic-dispatch pattern.GatewayClient/CallbackServerinternal methods that touchJavaObject— same reason.Why PEP 563
from __future__ import annotationsmakes every annotation a string at parse time. Consequences:inspect.signaturesurprises for PySpark or other downstream callers (the annotation values are visible as strings, not as evaluated types)X | Nonesyntax on Python 3.9 (which doesn't support PEP 604 at runtime)mypy / CI
Not gated in CI for this PR (deferred per the original spec). Annotations are added for IDE / static-analysis benefit. A future discussion on style / lint tooling (likely covering ruff + black + mypy together) will revisit CI gating.
Validation
JavaGateway.__init__.__annotations__values arestrtypeCo-authored-by: Isaac