Automatically handle foreign key traversal evaluation in ValuesViewset

docs/backend_architecture/api_patterns.rst

@@ -11,21 +11,24 @@ Overview
 
 ``ValuesViewset`` is the **preferred pattern for all API endpoints in Kolibri** unless there's a compelling reason to use a standard DRF viewset. It uses Django's ``.values()`` queryset method to fetch only needed fields in a single database query, avoiding the overhead of model instantiation and providing better performance.
 
-**Performance benefits:**
+Performance benefits
+^^^^^^^^^^^^^^^^^^^^
 
 - **Avoids N+1 queries** when traversing foreign key lookups (which happens easily with DRF Serializers using method fields)
 - **Reduces memory usage** for large querysets by not instantiating model objects that aren't needed for read operations
 - Single database query with only needed fields (vs. fetching all model fields)
 - Efficient handling of foreign key lookups using ``__`` notation
 
-**When to use ValuesViewset (default):**
+When to use ValuesViewset (default)
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 - Use ValuesViewset for **all API endpoints** as the standard pattern
 - Works for both read and write operations (uses ModelSerializer for write operations)
 - Particularly important for endpoints that traverse foreign key relationships
 - Essential for list endpoints with many objects
 
-**When a standard ModelViewSet might be needed:**
+When a standard ModelViewSet might be needed
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 - Very rare - ValuesViewset should be the default choice
 - Only if there's a specific technical limitation that requires standard DRF patterns
@@ -80,9 +83,9 @@ The viewset introspects the serializer's fields to build the values tuple and fi
    * - ``field = CharField(write_only=True)``
      - Skip (not in read output)
    * - ``nested = NestedSerializer(many=True)``
-     - Flatten nested fields with prefix, auto-consolidate child rows into a list per parent
+     - Auto-defer; fetch children in a batched follow-up query, bucketed per parent
    * - ``nested = NestedSerializer()``
-     - Flatten nested fields with prefix, extract as dict per row
+     - Auto-defer; fetch + merge the related object in a batched follow-up query
    * - Custom field with ``to_representation()``
      - Custom transformation applied automatically
    * - ``field = ValuesMethodField(sources=(...))``
@@ -140,71 +143,30 @@ A plain ``SerializerMethodField`` is rejected at viewset init — the viewset ca
 Nested Serializers
 ~~~~~~~~~~~~~~~~~~
 
-Nested serializers can be handled in two ways: **joined** (default) or **deferred**.
-
-Joined (Default) — Auto-Consolidated
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-When a nested serializer is not listed in ``deferred_fields``, its fields are included in the main ``values()`` call with a prefix. The resulting flat rows are automatically consolidated back into nested structures:
+Every nested serializer relation — forward FK, OneToOne (either direction), reverse FK, M2M — is fetched in a batched follow-up query and assembled onto its parents, rather than joined into the parent ``values()`` call where it would multiply or widen rows. No declaration is needed; see `Automatic Deferral`_.
 
 .. code-block:: python
 
-  class RoleSerializer(serializers.ModelSerializer):
+  class PublisherSerializer(serializers.ModelSerializer):
       class Meta:
-          model = Role
-          fields = ("id", "kind", "collection")
+          model = Publisher
+          fields = ("id", "name")
 
-  class UserSerializer(serializers.ModelSerializer):
-      roles = RoleSerializer(many=True, read_only=True)
+  class BookSerializer(serializers.ModelSerializer):
+      publisher = PublisherSerializer(read_only=True)
 
       class Meta:
-          model = FacilityUser
-          fields = ("id", "username", "roles")
-
-  class UserViewSet(ReadOnlyValuesViewset):
-      serializer_class = UserSerializer
-      queryset = FacilityUser.objects.all()
-
-The viewset fetches ``("id", "username", "roles__id", "roles__kind", "roles__collection")`` and auto-consolidates:
-
-.. code-block:: python
-
-  # Raw values() output (multiple rows per user due to LEFT JOIN):
-  [
-      {"id": "user1", "username": "alice", "roles__id": "r1", "roles__kind": "admin", ...},
-      {"id": "user1", "username": "alice", "roles__id": "r2", "roles__kind": "coach", ...},
-      {"id": "user2", "username": "bob",   "roles__id": "r3", "roles__kind": "learner", ...},
-  ]
-
-  # After auto-consolidation (grouped by primary key):
-  [
-      {"id": "user1", "username": "alice", "roles": [
-          {"id": "r1", "kind": "admin", ...},
-          {"id": "r2", "kind": "coach", ...},
-      ]},
-      {"id": "user2", "username": "bob", "roles": [
-          {"id": "r3", "kind": "learner", ...},
-      ]},
-  ]
-
-Auto-consolidation handles:
-
-- Grouping rows by parent primary key
-- Deduplicating nested items (e.g., from annotation JOINs)
-- NULL handling for LEFT JOINs (null FK → ``None`` for single nested, empty list for ``many=True``)
-- Preserving original queryset ordering
+          model = Book
+          fields = ("id", "title", "publisher")
 
-**Constraints:**
+The viewset fetches ``("id", "title", "publisher_id")`` for the books, then one batched ``Publisher.objects.filter(pk__in=...)`` — each distinct publisher fetched once — and assembles the nested dict. A null FK yields ``None``.
 
-- Only one ``many=True`` nested serializer may be joined per viewset (multiple would create a cartesian product). Additional ``many=True`` fields must be deferred.
-- Deep nesting (nested serializers within nested serializers) is not supported for joined fields. Use ``deferred_fields`` and a custom ``consolidate()`` method instead.
+Any nested serializer shape works without raising at viewset instantiation.
 
-These constraints are checked at viewset instantiation time when ``DEBUG=True``.
+Custom fetch logic with deferred_fields
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
-Deferred — Fetched Separately in consolidate()
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-
-For nested data that should be fetched with separate queries (for performance reasons, to avoid cartesian products, or when the relation is complex), list the field in ``deferred_fields`` and use ``serialize_queryset()`` in ``consolidate()``:
+For a fetch the automatic handling can't express — one needing annotations, filtering, or other custom logic — list the field in ``deferred_fields`` and build it yourself with ``serialize_queryset()`` in ``consolidate()``:
 
 .. code-block:: python
 
@@ -252,6 +214,46 @@ For nested data that should be fetched with separate queries (for performance re
 
 The ``serialize_queryset()`` method applies the values pattern using the nested serializer's field definitions. It accepts a ``group_by`` parameter to return a dict mapping group keys to item lists, which is convenient for mapping back to parent items.
 
+Parents sharing a forward-FK target share **the same nested dict by identity** — one serialized row per target pk. In ``consolidate()``, mutating a nested object in place lands on every parent pointing at that target; replace it instead.
+
+.. _Automatic Deferral:
+
+Automatic Deferral
+^^^^^^^^^^^^^^^^^^
+
+The framework fetches these in follow-up queries on its own — no ``deferred_fields`` declaration or ``consolidate()`` override required: every nested serializer relation (forward FK, OneToOne either direction, reverse FK, M2M), and any field whose source crosses a to-many relation (e.g. ``books.title``).
+
+Explicit ``deferred_fields`` is never auto-fetched. Use it when the fetch needs annotations, soft-delete filtering, or logic beyond a plain queryset.
+
+Each auto-deferral decision emits a ``DEBUG``-level log on ``kolibri.core.utils.values_viewset.introspect``.
+
+Permission scope differs by direction. A reverse-FK / M2M / scalar-many fetch is filtered by the parent pks the parent queryset returned, so it inherits that scope. A forward FK resolves as ``Model._base_manager.filter(pk__in=...)`` — the target's own pk, base manager, no filter backend — as the ``LEFT JOIN`` it replaced also did. Don't rely on a forward nested serializer to hide a target.
+
+Fetch strategy
+""""""""""""""
+
+- Each auto-deferred reverse-FK / M2M field: one batched query per level (``child.objects.filter(<reverse_accessor>__in=parent_pks)``), then bucketed onto parents.
+- Auto-deferred forward-FK fields are keyed on the *target's* own pk:
+
+  - Every reference to a model merges into one query per resolver round, over the union of pks and columns — even across *different* serializers.
+  - Example: ``ContentNode`` and its ``File`` rows both reference ``Language`` → one ``Language`` query, even when each uses a different ``Language`` serializer.
+  - Each reference is then serialized under its own shape (keyed by serializer path), so no fields leak between shapes.
+- Reverse-FK / M2M fetches can't merge across the tree the way forward-FK fetches do: a forward FK is keyed on the *target's* own pk (``Model.objects.filter(pk__in=...)``), so one query serves every reference to that model at any depth; a reverse-FK is keyed on the *parent's* pk through a specific accessor (``child.filter(accessor__in=parent_pks)``), so fields at different positions have different parent sets and accessors and need their own query.
+- Fetched children run through the same pipeline; query count scales with nesting depth, not row count.
+
+Limitations
+"""""""""""
+
+- Deferring a to-one trades a round trip for not repeating the target's columns on every parent row: 0–0.15 ms per request, measured against the pre-auto-defer joined code on a real database.
+- ``--compare-autodefer`` puts that same cost at 30–125%; its viewsets do nothing but the to-one, so read it as an upper bound.
+- A nested list's order is whatever its own query returns, so the child model needs an ``ordering`` in its ``Meta``. Ordering it from the parent queryset (``.order_by("id", "roles__kind")``) no longer reaches the child.
+- ``?ordering=`` takes a forward nested serializer's FK column (``?ordering=collection``), not the child columns a join used to expose (``?ordering=collection__id``). Unknown terms drop silently, so a stale client falls back to the default ordering.
+- Forward-FK targets merge only within one resolver round. A model reached both directly and via a forward→forward chain (depth ≥ 2) is queried once per round it appears in.
+- Rounds advance one forward hop at a time because a target's own forward FKs aren't known until it's fetched, so a static reorder can't pre-merge the deeper pks.
+- Reverse nesting collapses into round 1, so only forward-FK chains trip this.
+- The fetch is shared per model, but serialization is per referencing path: a forward target's own sub-fetches (scalar-many, reverse-FK) run once per path, not deduped across paths.
+- Forward targets of one model share a query only when their SQL-rename annotations agree; positions that alias the same name to a different source column fetch separately.
+
 Dev-Mode Validation
 ~~~~~~~~~~~~~~~~~~~~
 
@@ -399,7 +401,7 @@ Best Practices
 
 2. **Use source for renames**: Use ``source`` on serializer fields rather than ``field_map`` for renaming.
 
-3. **Defer wisely**: Use ``deferred_fields`` for ``many=True`` relations that would create large cartesian products, or for relations that require complex queries. Keep simple FK lookups as joined.
+3. **Rely on auto-deferral for relations**: Use explicit ``deferred_fields`` only when you need ORM annotations or custom filtering in ``consolidate()``.
 
 4. **Batch related queries in consolidate**: Fetch deferred data efficiently using ``serialize_queryset()`` with ``group_by`` and ``__in`` lookups on IDs from already-fetched items.
 
@@ -410,56 +412,8 @@ Best Practices
 Common Pitfalls
 ~~~~~~~~~~~~~~~
 
-**Multiple many=True nested serializers without deferring**
-
-.. code-block:: python
-
-  # Wrong: cartesian product — two many=True JOINs multiply rows
-  class UserSerializer(serializers.ModelSerializer):
-      roles = RoleSerializer(many=True)
-      groups = GroupSerializer(many=True)
-
-      class Meta:
-          model = FacilityUser
-          fields = ("id", "roles", "groups")
-
-  class UserViewSet(ReadOnlyValuesViewset):
-      serializer_class = UserSerializer  # Raises TypeError in DEBUG
-
-  # Correct: defer one of them
-  class UserViewSet(ReadOnlyValuesViewset):
-      serializer_class = UserSerializer
-      deferred_fields = ("groups",)
-
-      def consolidate(self, items, queryset):
-          # Fetch groups separately
-          ...
-
-**Deep nesting without deferring**
-
-.. code-block:: python
-
-  # Wrong: nested serializer within nested serializer
-  class GrandchildSerializer(serializers.ModelSerializer):
-      class Meta:
-          fields = ("id", "name")
-
-  class ChildSerializer(serializers.ModelSerializer):
-      grandchildren = GrandchildSerializer(many=True)
-      class Meta:
-          fields = ("id", "grandchildren")
-
-  class ParentSerializer(serializers.ModelSerializer):
-      children = ChildSerializer(many=True)
-      class Meta:
-          fields = ("id", "children")
-
-  # Correct: defer deeply nested fields
-  class ParentViewSet(ReadOnlyValuesViewset):
-      serializer_class = ParentSerializer
-      deferred_fields = ("children",)  # Fetch children (and grandchildren) in consolidate
-
-**Forgetting to return items from consolidate**
+Forgetting to return items from consolidate
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
 .. code-block:: python
 
@@ -528,7 +482,7 @@ Existing viewsets that use explicit ``values`` tuples and ``field_map`` dicts co
 
 5. **Convert manual consolidation** of nested data:
 
-   - If the viewset manually does ``groupby`` to build nested lists, define a nested serializer with ``many=True`` and let auto-consolidation handle it
+   - If the viewset manually does ``groupby`` to build nested lists, define a nested serializer with ``many=True`` and let auto-deferral handle it
    - If the nested data is fetched separately, add it to ``deferred_fields`` and use ``serialize_queryset()``
 
 6. **Remove** the explicit ``values`` tuple and ``field_map`` dict.