diff --git a/docs/backend_architecture/api_patterns.rst b/docs/backend_architecture/api_patterns.rst index be085cabb22..0ad61facc4d 100644 --- a/docs/backend_architecture/api_patterns.rst +++ b/docs/backend_architecture/api_patterns.rst @@ -11,21 +11,24 @@ Overview ``ValuesViewset`` is the **preferred pattern for all API endpoints in Kolibri** unless there's a compelling reason to use a standard DRF viewset. It uses Django's ``.values()`` queryset method to fetch only needed fields in a single database query, avoiding the overhead of model instantiation and providing better performance. -**Performance benefits:** +Performance benefits +^^^^^^^^^^^^^^^^^^^^ - **Avoids N+1 queries** when traversing foreign key lookups (which happens easily with DRF Serializers using method fields) - **Reduces memory usage** for large querysets by not instantiating model objects that aren't needed for read operations - Single database query with only needed fields (vs. fetching all model fields) - Efficient handling of foreign key lookups using ``__`` notation -**When to use ValuesViewset (default):** +When to use ValuesViewset (default) +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Use ValuesViewset for **all API endpoints** as the standard pattern - Works for both read and write operations (uses ModelSerializer for write operations) - Particularly important for endpoints that traverse foreign key relationships - Essential for list endpoints with many objects -**When a standard ModelViewSet might be needed:** +When a standard ModelViewSet might be needed +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - Very rare - ValuesViewset should be the default choice - Only if there's a specific technical limitation that requires standard DRF patterns @@ -80,9 +83,9 @@ The viewset introspects the serializer's fields to build the values tuple and fi * - ``field = CharField(write_only=True)`` - Skip (not in read output) * - ``nested = NestedSerializer(many=True)`` - - Flatten nested fields with prefix, auto-consolidate child rows into a list per parent + - Auto-defer; fetch children in a batched follow-up query, bucketed per parent * - ``nested = NestedSerializer()`` - - Flatten nested fields with prefix, extract as dict per row + - Auto-defer; fetch + merge the related object in a batched follow-up query * - Custom field with ``to_representation()`` - Custom transformation applied automatically * - ``field = ValuesMethodField(sources=(...))`` @@ -140,71 +143,30 @@ A plain ``SerializerMethodField`` is rejected at viewset init — the viewset ca Nested Serializers ~~~~~~~~~~~~~~~~~~ -Nested serializers can be handled in two ways: **joined** (default) or **deferred**. - -Joined (Default) — Auto-Consolidated -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -When a nested serializer is not listed in ``deferred_fields``, its fields are included in the main ``values()`` call with a prefix. The resulting flat rows are automatically consolidated back into nested structures: +Every nested serializer relation — forward FK, OneToOne (either direction), reverse FK, M2M — is fetched in a batched follow-up query and assembled onto its parents, rather than joined into the parent ``values()`` call where it would multiply or widen rows. No declaration is needed; see `Automatic Deferral`_. .. code-block:: python - class RoleSerializer(serializers.ModelSerializer): + class PublisherSerializer(serializers.ModelSerializer): class Meta: - model = Role - fields = ("id", "kind", "collection") + model = Publisher + fields = ("id", "name") - class UserSerializer(serializers.ModelSerializer): - roles = RoleSerializer(many=True, read_only=True) + class BookSerializer(serializers.ModelSerializer): + publisher = PublisherSerializer(read_only=True) class Meta: - model = FacilityUser - fields = ("id", "username", "roles") - - class UserViewSet(ReadOnlyValuesViewset): - serializer_class = UserSerializer - queryset = FacilityUser.objects.all() - -The viewset fetches ``("id", "username", "roles__id", "roles__kind", "roles__collection")`` and auto-consolidates: - -.. code-block:: python - - # Raw values() output (multiple rows per user due to LEFT JOIN): - [ - {"id": "user1", "username": "alice", "roles__id": "r1", "roles__kind": "admin", ...}, - {"id": "user1", "username": "alice", "roles__id": "r2", "roles__kind": "coach", ...}, - {"id": "user2", "username": "bob", "roles__id": "r3", "roles__kind": "learner", ...}, - ] - - # After auto-consolidation (grouped by primary key): - [ - {"id": "user1", "username": "alice", "roles": [ - {"id": "r1", "kind": "admin", ...}, - {"id": "r2", "kind": "coach", ...}, - ]}, - {"id": "user2", "username": "bob", "roles": [ - {"id": "r3", "kind": "learner", ...}, - ]}, - ] - -Auto-consolidation handles: - -- Grouping rows by parent primary key -- Deduplicating nested items (e.g., from annotation JOINs) -- NULL handling for LEFT JOINs (null FK → ``None`` for single nested, empty list for ``many=True``) -- Preserving original queryset ordering + model = Book + fields = ("id", "title", "publisher") -**Constraints:** +The viewset fetches ``("id", "title", "publisher_id")`` for the books, then one batched ``Publisher.objects.filter(pk__in=...)`` — each distinct publisher fetched once — and assembles the nested dict. A null FK yields ``None``. -- Only one ``many=True`` nested serializer may be joined per viewset (multiple would create a cartesian product). Additional ``many=True`` fields must be deferred. -- Deep nesting (nested serializers within nested serializers) is not supported for joined fields. Use ``deferred_fields`` and a custom ``consolidate()`` method instead. +Any nested serializer shape works without raising at viewset instantiation. -These constraints are checked at viewset instantiation time when ``DEBUG=True``. +Custom fetch logic with deferred_fields +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -Deferred — Fetched Separately in consolidate() -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -For nested data that should be fetched with separate queries (for performance reasons, to avoid cartesian products, or when the relation is complex), list the field in ``deferred_fields`` and use ``serialize_queryset()`` in ``consolidate()``: +For a fetch the automatic handling can't express — one needing annotations, filtering, or other custom logic — list the field in ``deferred_fields`` and build it yourself with ``serialize_queryset()`` in ``consolidate()``: .. code-block:: python @@ -252,6 +214,46 @@ For nested data that should be fetched with separate queries (for performance re The ``serialize_queryset()`` method applies the values pattern using the nested serializer's field definitions. It accepts a ``group_by`` parameter to return a dict mapping group keys to item lists, which is convenient for mapping back to parent items. +Parents sharing a forward-FK target share **the same nested dict by identity** — one serialized row per target pk. In ``consolidate()``, mutating a nested object in place lands on every parent pointing at that target; replace it instead. + +.. _Automatic Deferral: + +Automatic Deferral +^^^^^^^^^^^^^^^^^^ + +The framework fetches these in follow-up queries on its own — no ``deferred_fields`` declaration or ``consolidate()`` override required: every nested serializer relation (forward FK, OneToOne either direction, reverse FK, M2M), and any field whose source crosses a to-many relation (e.g. ``books.title``). + +Explicit ``deferred_fields`` is never auto-fetched. Use it when the fetch needs annotations, soft-delete filtering, or logic beyond a plain queryset. + +Each auto-deferral decision emits a ``DEBUG``-level log on ``kolibri.core.utils.values_viewset.introspect``. + +Permission scope differs by direction. A reverse-FK / M2M / scalar-many fetch is filtered by the parent pks the parent queryset returned, so it inherits that scope. A forward FK resolves as ``Model._base_manager.filter(pk__in=...)`` — the target's own pk, base manager, no filter backend — as the ``LEFT JOIN`` it replaced also did. Don't rely on a forward nested serializer to hide a target. + +Fetch strategy +"""""""""""""" + +- Each auto-deferred reverse-FK / M2M field: one batched query per level (``child.objects.filter(__in=parent_pks)``), then bucketed onto parents. +- Auto-deferred forward-FK fields are keyed on the *target's* own pk: + + - Every reference to a model merges into one query per resolver round, over the union of pks and columns — even across *different* serializers. + - Example: ``ContentNode`` and its ``File`` rows both reference ``Language`` → one ``Language`` query, even when each uses a different ``Language`` serializer. + - Each reference is then serialized under its own shape (keyed by serializer path), so no fields leak between shapes. +- Reverse-FK / M2M fetches can't merge across the tree the way forward-FK fetches do: a forward FK is keyed on the *target's* own pk (``Model.objects.filter(pk__in=...)``), so one query serves every reference to that model at any depth; a reverse-FK is keyed on the *parent's* pk through a specific accessor (``child.filter(accessor__in=parent_pks)``), so fields at different positions have different parent sets and accessors and need their own query. +- Fetched children run through the same pipeline; query count scales with nesting depth, not row count. + +Limitations +""""""""""" + +- Deferring a to-one trades a round trip for not repeating the target's columns on every parent row: 0–0.15 ms per request, measured against the pre-auto-defer joined code on a real database. +- ``--compare-autodefer`` puts that same cost at 30–125%; its viewsets do nothing but the to-one, so read it as an upper bound. +- A nested list's order is whatever its own query returns, so the child model needs an ``ordering`` in its ``Meta``. Ordering it from the parent queryset (``.order_by("id", "roles__kind")``) no longer reaches the child. +- ``?ordering=`` takes a forward nested serializer's FK column (``?ordering=collection``), not the child columns a join used to expose (``?ordering=collection__id``). Unknown terms drop silently, so a stale client falls back to the default ordering. +- Forward-FK targets merge only within one resolver round. A model reached both directly and via a forward→forward chain (depth ≥ 2) is queried once per round it appears in. +- Rounds advance one forward hop at a time because a target's own forward FKs aren't known until it's fetched, so a static reorder can't pre-merge the deeper pks. +- Reverse nesting collapses into round 1, so only forward-FK chains trip this. +- The fetch is shared per model, but serialization is per referencing path: a forward target's own sub-fetches (scalar-many, reverse-FK) run once per path, not deduped across paths. +- Forward targets of one model share a query only when their SQL-rename annotations agree; positions that alias the same name to a different source column fetch separately. + Dev-Mode Validation ~~~~~~~~~~~~~~~~~~~~ @@ -399,7 +401,7 @@ Best Practices 2. **Use source for renames**: Use ``source`` on serializer fields rather than ``field_map`` for renaming. -3. **Defer wisely**: Use ``deferred_fields`` for ``many=True`` relations that would create large cartesian products, or for relations that require complex queries. Keep simple FK lookups as joined. +3. **Rely on auto-deferral for relations**: Use explicit ``deferred_fields`` only when you need ORM annotations or custom filtering in ``consolidate()``. 4. **Batch related queries in consolidate**: Fetch deferred data efficiently using ``serialize_queryset()`` with ``group_by`` and ``__in`` lookups on IDs from already-fetched items. @@ -410,56 +412,8 @@ Best Practices Common Pitfalls ~~~~~~~~~~~~~~~ -**Multiple many=True nested serializers without deferring** - -.. code-block:: python - - # Wrong: cartesian product — two many=True JOINs multiply rows - class UserSerializer(serializers.ModelSerializer): - roles = RoleSerializer(many=True) - groups = GroupSerializer(many=True) - - class Meta: - model = FacilityUser - fields = ("id", "roles", "groups") - - class UserViewSet(ReadOnlyValuesViewset): - serializer_class = UserSerializer # Raises TypeError in DEBUG - - # Correct: defer one of them - class UserViewSet(ReadOnlyValuesViewset): - serializer_class = UserSerializer - deferred_fields = ("groups",) - - def consolidate(self, items, queryset): - # Fetch groups separately - ... - -**Deep nesting without deferring** - -.. code-block:: python - - # Wrong: nested serializer within nested serializer - class GrandchildSerializer(serializers.ModelSerializer): - class Meta: - fields = ("id", "name") - - class ChildSerializer(serializers.ModelSerializer): - grandchildren = GrandchildSerializer(many=True) - class Meta: - fields = ("id", "grandchildren") - - class ParentSerializer(serializers.ModelSerializer): - children = ChildSerializer(many=True) - class Meta: - fields = ("id", "children") - - # Correct: defer deeply nested fields - class ParentViewSet(ReadOnlyValuesViewset): - serializer_class = ParentSerializer - deferred_fields = ("children",) # Fetch children (and grandchildren) in consolidate - -**Forgetting to return items from consolidate** +Forgetting to return items from consolidate +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ .. code-block:: python @@ -528,7 +482,7 @@ Existing viewsets that use explicit ``values`` tuples and ``field_map`` dicts co 5. **Convert manual consolidation** of nested data: - - If the viewset manually does ``groupby`` to build nested lists, define a nested serializer with ``many=True`` and let auto-consolidation handle it + - If the viewset manually does ``groupby`` to build nested lists, define a nested serializer with ``many=True`` and let auto-deferral handle it - If the nested data is fetched separately, add it to ``deferred_fields`` and use ``serialize_queryset()`` 6. **Remove** the explicit ``values`` tuple and ``field_map`` dict. diff --git a/integration_testing/scripts/viewset_serialization_benchmark.py b/integration_testing/scripts/viewset_serialization_benchmark.py index 3223cdb567f..57357f96456 100644 --- a/integration_testing/scripts/viewset_serialization_benchmark.py +++ b/integration_testing/scripts/viewset_serialization_benchmark.py @@ -31,19 +31,25 @@ import sys import time import tracemalloc +import uuid from datetime import datetime -from unittest.mock import MagicMock # Must import kolibri before Django to apply compat patches (e.g. cgi module on Python 3.13+) from kolibri.utils.main import initialize # isort: skip from django.conf import settings from django.db import connection -from rest_framework import serializers as drf_serializers +from django.test.utils import CaptureQueriesContext from rest_framework.request import Request logger = logging.getLogger(__name__) +# Bump whenever a fixture or report field changes shape, so --compare against a +# baseline from before the change warns instead of reporting a bogus delta. +# 2: the synthetic serializer dropped its nested fields — they are auto-deferred +# now, and the mock queryset can't serve the follow-up fetches. +SCHEMA_VERSION = 2 + def parse_args(): parser = argparse.ArgumentParser( @@ -58,8 +64,14 @@ def parse_args(): parser.add_argument( "--synthetic", action="store_true", - help="Run with a synthetic viewset and mock data (no DB needed). " - "Autoscales at sizes 10, 20, 50, 100.", + help="Run with a synthetic viewset over generated fixtures in a " + "throwaway test database. Autoscales at sizes 10, 20, 50, 100.", + ) + parser.add_argument( + "--compare-autodefer", + action="store_true", + help="Compare auto-deferred derived vs explicit values+consolidate() over " + "generated fixtures in a throwaway test database, sweeping reverse-FK fan-out.", ) parser.add_argument( "-o", @@ -197,39 +209,69 @@ def get_queryset_for_viewset(viewset_class): def _build_synthetic_viewset(): - """Build a viewset class with a serializer exercising all serialization paths.""" + """ + Build a viewset covering both halves of the pipeline: the flat paths + (passthrough, source rename, method field over multiple sources) and the + auto-deferred ones (reverse FK, and a forward-FK chain two levels deep). + + Built lazily on the test-only models, so the test app registry and database + must be set up first. + """ + from rest_framework import serializers from kolibri.core.api import BaseValuesViewset from kolibri.core.api import ListModelMixin from kolibri.core.api import ValuesMethodField - - class TagSerializer(drf_serializers.Serializer): - id = drf_serializers.CharField() - label = drf_serializers.CharField() - - class DepartmentSerializer(drf_serializers.Serializer): - id = drf_serializers.CharField() - name = drf_serializers.CharField() - - class SyntheticSerializer(drf_serializers.Serializer): - id = drf_serializers.CharField() - # Flat field with rename (exercises simple_renames path) - display_name = drf_serializers.CharField(source="full_name") - email = drf_serializers.CharField() - score = drf_serializers.IntegerField() - # many=True nested (exercises _auto_consolidate groupby) - tags = TagSerializer(many=True, source="tag_set") - # Single nested FK (exercises _joined_single dict consolidation) - department = DepartmentSerializer(source="dept") - # Method field over multiple sources (exercises _SourcesProxy + - # invoker callable in field_map). - contact_label = ValuesMethodField(sources=("full_name", "email")) + from kolibri.core.test.test_app.models import Author + from kolibri.core.test.test_app.models import Book + from kolibri.core.test.test_app.models import Country + from kolibri.core.test.test_app.models import Publisher + + class CountrySerializer(serializers.ModelSerializer): + class Meta: + model = Country + fields = ("id", "name") + + class PublisherSerializer(serializers.ModelSerializer): + country = CountrySerializer(allow_null=True) + + class Meta: + model = Publisher + fields = ("id", "name", "country") + + class BookSerializer(serializers.ModelSerializer): + class Meta: + model = Book + fields = ("id", "title") + + class SyntheticSerializer(serializers.ModelSerializer): + # rename: exercises the simple_renames path + display_name = serializers.CharField(source="name") + # method field over multiple sources, one of them shared with a rename: + # exercises _SourcesProxy, the field_map invoker, and the refcount that + # keeps a shared column out of rename promotion + contact_label = ValuesMethodField(sources=("name", "email")) + # auto-deferred: reverse FK, and forward FK carrying its own forward FK + books = BookSerializer(many=True) + publisher = PublisherSerializer(allow_null=True) + + class Meta: + model = Author + fields = ( + "id", + "display_name", + "email", + "contact_label", + "books", + "publisher", + ) def get_contact_label(self, row): - return "{} <{}>".format(row.full_name, row.email) + return "{} <{}>".format(row.name, row.email) class SyntheticViewset(BaseValuesViewset, ListModelMixin): serializer_class = SyntheticSerializer + queryset = Author.objects.all().order_by("name") return SyntheticViewset @@ -237,48 +279,6 @@ class SyntheticViewset(BaseValuesViewset, ListModelMixin): SYNTHETIC_SIZES = (10, 20, 50, 100) -def _generate_synthetic_data(n): - """ - Generate n parent records as flat dicts simulating QuerySet.values() output. - - Each parent has 2 tags (many=True join) and 1 department (FK join). - The flat output has n*2 rows because of the tag join expansion. - """ - rows = [] - for i in range(n): - for t in range(2): - rows.append( - { - "id": f"user-{i:04d}", - "full_name": f"User {i}", - "email": f"user{i}@example.com", - "score": 100 + i, - "tag_set__id": f"tag-{i}-{t}", - "tag_set__label": f"label-{i}-{t}", - "dept__id": f"dept-{i % 5:04d}", - "dept__name": f"Department {i % 5}", - } - ) - return rows - - -def _make_synthetic_queryset(flat_items): - """Wrap flat dict list in a mock queryset compatible with serialize().""" - - class StubMeta: - class pk: - name = "id" - - class StubModel: - _meta = StubMeta() - - mock_qs = MagicMock() - mock_qs.model = StubModel - mock_qs.values.side_effect = lambda *a, **kw: [item.copy() for item in flat_items] - mock_qs.count.return_value = len(set(row["id"] for row in flat_items)) - return mock_qs - - def _make_viewset(viewset_class, queryset, user=None): """Create a viewset instance with a DRF Request for standalone use. @@ -474,10 +474,13 @@ def build_report( has_explicit_values = "values" in viewset_class.__dict__ and isinstance( viewset_class.__dict__["values"], tuple ) - has_derived = viewset_class._cached_serializer is not None + has_derived = ( + not has_explicit_values + and getattr(viewset_class, "serializer_class", None) is not None + ) return { - "schema_version": 1, + "schema_version": SCHEMA_VERSION, "metadata": { "viewset_class": dotted_path, "has_explicit_values": has_explicit_values, @@ -524,10 +527,13 @@ def write_report(report, path): def load_report(path): with open(path) as f: report = json.load(f) - if report.get("schema_version") != 1: + if report.get("schema_version") != SCHEMA_VERSION: logger.warning( - "Baseline report has schema_version=%s, expected 1", + "Baseline report has schema_version=%s, expected %s — the fixtures " + "differ, so any timing delta against it is meaningless. Re-record " + "the baseline with this version of the script.", report.get("schema_version"), + SCHEMA_VERSION, ) return report @@ -700,54 +706,31 @@ def row(label, b_val, c_val, diff, verdict_str, detail=""): def _run_synthetic(args): """Run benchmark with synthetic viewset at multiple data sizes.""" - setup_kolibri(inherit_kolibri_home=args.inherit_kolibri_home) + from django.test.utils import setup_test_environment + from django.test.utils import teardown_test_environment - viewset_class = _build_synthetic_viewset() - sizes_report = {} - - for size in SYNTHETIC_SIZES: - if not args.quiet: - logger.info("\n--- Size: %d ---", size) - flat_items = _generate_synthetic_data(size) - mock_qs = _make_synthetic_queryset(flat_items) - - if not args.quiet: - logger.info("Running timing benchmark...") - timing = benchmark_timing(viewset_class, mock_qs, args.iterations, args.warmup) + setup_kolibri(inherit_kolibri_home=args.inherit_kolibri_home) - if not args.quiet: - logger.info("Running memory benchmark...") - memory = benchmark_memory( - viewset_class, mock_qs, args.memory_iterations, args.warmup - ) + # Register the test-only app so its tables are created in the test DB. + import kolibri.core.test # noqa: F401 - if not args.quiet: - logger.info("Capturing data snapshot...") - data_snapshot = capture_data_snapshot(viewset_class, mock_qs) - - sizes_report[str(size)] = build_report( - viewset_class=viewset_class, - dotted_path="", - record_count=size, - iterations=args.iterations, - memory_iterations=args.memory_iterations, - warmup=args.warmup, - timing=timing, - memory=memory, - queries=None, - data_snapshot=data_snapshot, - time_threshold=args.time_threshold, - memory_threshold=args.memory_threshold, - ) + # DB-backed: each iteration hits the DB, so cap well below the in-memory default. + iterations = min(args.iterations, 50) + memory_iterations = min(args.memory_iterations, 20) - if not args.quiet: - r = sizes_report[str(size)] - logger.info(" Time: %.3f ms (mean)", r["timing"]["mean_ms"]) - logger.info(" Memory: %s (mean)", _fmt_bytes(r["memory"]["mean_bytes"])) - logger.info(" Data hash: %s...", r["data"]["output_hash"][:30]) + setup_test_environment() + old_config = connection.creation.create_test_db(verbosity=0, autoclobber=True) + try: + sizes_report = { + str(size): _synthetic_size_report(size, args, iterations, memory_iterations) + for size in SYNTHETIC_SIZES + } + finally: + connection.creation.destroy_test_db(old_config, verbosity=0) + teardown_test_environment() report = { - "schema_version": 1, + "schema_version": SCHEMA_VERSION, "synthetic": True, "sizes": sizes_report, } @@ -765,6 +748,55 @@ def _run_synthetic(args): return 0 +def _synthetic_size_report(size, args, iterations, memory_iterations): + """Benchmark the synthetic viewset over ``size`` authors on a clean DB.""" + from kolibri.core.test.test_app.models import Author + + if not args.quiet: + logger.info("\n--- Size: %d ---", size) + + Author.objects.all().delete() + _build_autodefer_fixtures(size) + + viewset_class = _build_synthetic_viewset() + queryset = viewset_class.queryset + + if not args.quiet: + logger.info("Running timing benchmark...") + timing = benchmark_timing(viewset_class, queryset, iterations, args.warmup) + + if not args.quiet: + logger.info("Running memory benchmark...") + memory = benchmark_memory(viewset_class, queryset, memory_iterations, args.warmup) + + if not args.quiet: + logger.info("Capturing data snapshot...") + data_snapshot = capture_data_snapshot(viewset_class, queryset) + + report = build_report( + viewset_class=viewset_class, + dotted_path="", + record_count=size, + iterations=iterations, + memory_iterations=memory_iterations, + warmup=args.warmup, + timing=timing, + memory=memory, + queries=count_queries(viewset_class, queryset), + data_snapshot=data_snapshot, + time_threshold=args.time_threshold, + memory_threshold=args.memory_threshold, + ) + + if not args.quiet: + logger.info(" Time: %.3f ms (mean)", report["timing"]["mean_ms"]) + logger.info(" Memory: %s (mean)", _fmt_bytes(report["memory"]["mean_bytes"])) + logger.info(" Queries: %s", report["queries"]["count"]) + logger.info(" Data hash: %s...", report["data"]["output_hash"][:30]) + + return report + + def _compare_synthetic(baseline, current, args): """Compare two synthetic reports size-by-size.""" overall_pass = True @@ -813,7 +845,10 @@ def _run_real_viewset(args): has_explicit = "values" in viewset_class.__dict__ and isinstance( viewset_class.__dict__["values"], tuple ) - has_derived = viewset_class._cached_serializer is not None + has_derived = ( + not has_explicit + and getattr(viewset_class, "serializer_class", None) is not None + ) pattern = "derived" if has_derived else ("explicit" if has_explicit else "unknown") if not args.quiet: @@ -886,8 +921,631 @@ def _run_real_viewset(args): return 0 +def _mean_ms(fn, warmup, iterations): + """Mean wall-clock ms over ``iterations`` calls, after ``warmup`` calls.""" + for _ in range(warmup): + fn() + gc.collect() + gc.disable() + try: + times = [] + for _ in range(iterations): + start = time.perf_counter() + fn() + times.append(time.perf_counter() - start) + finally: + gc.enable() + return statistics.mean(times) * 1000 + + +def _query_count(fn): + """Number of DB queries issued by a single ``fn()`` call.""" + with CaptureQueriesContext(connection) as ctx: + fn() + return len(ctx) + + +def _measure_strategy(serialize, warmup, iterations): + return { + "queries": _query_count(serialize), + "mean_ms": _mean_ms(serialize, warmup, iterations), + } + + +def _autodefer_author_queryset(): + from kolibri.core.test.test_app.models import Author + + return Author.objects.all().order_by("name") + + +def _build_autodefer_fixtures(author_count, books_per_author=3, awards_per_author=2): + """Create N authors, each with M books and K awards, sharing one publisher. + + Every third author has no publisher, exercising null forward-FK targets. + Author pks are derived from the index rather than random, so the output hash + is stable across runs and ``--compare`` can detect real output drift. + """ + from kolibri.core.test.test_app.models import Author + from kolibri.core.test.test_app.models import Award + from kolibri.core.test.test_app.models import Book + from kolibri.core.test.test_app.models import Country + from kolibri.core.test.test_app.models import Publisher + + country = Country.objects.create(name="Testland") + publisher = Publisher.objects.create(name="Test House", country=country) + + authors = [] + for i in range(author_count): + pub = publisher if i % 3 != 0 else None + author = Author.objects.create( + id=uuid.UUID(int=i), + name="Author {:03d}".format(i), + email="author{}@example.com".format(i), + publisher=pub, + ) + authors.append(author) + for j in range(books_per_author): + Book.objects.create(author=author, title="Book {}-{}".format(i, j)) + for j in range(awards_per_author): + Award.objects.create(author=author, name="Award {}-{}".format(i, j)) + return authors + + +def _explicit_author_consolidate(items): + """Hand-written reverse/forward-FK assembly — the pre-auto-defer baseline the + derived viewset is benchmarked against.""" + from collections import defaultdict + + from kolibri.core.test.test_app.models import Award + from kolibri.core.test.test_app.models import Book + from kolibri.core.test.test_app.models import Country + from kolibri.core.test.test_app.models import Publisher + + author_pks = [item["id"] for item in items] + + # --- books (reverse FK, one query) --- + book_rows = list( + Book.objects.filter(author_id__in=author_pks).values("id", "title", "author_id") + ) + books_by_author = defaultdict(list) + for row in book_rows: + books_by_author[str(row["author_id"])].append( + {"id": row["id"], "title": row["title"]} + ) + + # --- awards (reverse FK, one query) --- + award_rows = list( + Award.objects.filter(author_id__in=author_pks).values("id", "name", "author_id") + ) + awards_by_author = defaultdict(list) + for row in award_rows: + awards_by_author[str(row["author_id"])].append( + {"id": row["id"], "name": row["name"]} + ) + + # --- publishers + countries (forward FK, one query each) --- + publisher_ids = {item["publisher_id"] for item in items if item["publisher_id"]} + publishers_by_id = _build_publishers_by_id(publisher_ids, Publisher, Country) + + for item in items: + item["books"] = books_by_author.get(str(item["id"]), []) + item["awards"] = awards_by_author.get(str(item["id"]), []) + pid = item.pop("publisher_id") + item["publisher"] = publishers_by_id.get(str(pid)) if pid else None + + return items + + +def _build_publishers_by_id(publisher_ids, Publisher, Country): + """``{publisher_pk: {id, name, country}}`` for the given pks, with country + nested via a single follow-up query.""" + if not publisher_ids: + return {} + + pub_rows = list( + Publisher.objects.filter(pk__in=publisher_ids).values( + "id", "name", "country_id" + ) + ) + country_ids = {row["country_id"] for row in pub_rows if row["country_id"]} + country_rows = Country.objects.filter(pk__in=country_ids).values("id", "name") + countries_by_id = { + str(row["id"]): {"id": row["id"], "name": row["name"]} for row in country_rows + } + + publishers_by_id = {} + for row in pub_rows: + cid = row["country_id"] + publishers_by_id[str(row["id"])] = { + "id": row["id"], + "name": row["name"], + "country": countries_by_id.get(str(cid)) if cid else None, + } + return publishers_by_id + + +def _build_autodefer_viewsets(): + """The derived and explicit Author viewsets, built lazily: their serializer + and queryset class attributes import test-only models, which requires the + test app registry and database to be set up first.""" + from rest_framework import serializers + + from kolibri.core.api import BaseValuesViewset + from kolibri.core.api import ListModelMixin + from kolibri.core.test.test_app.models import Author + from kolibri.core.test.test_app.models import Award + from kolibri.core.test.test_app.models import Book + from kolibri.core.test.test_app.models import Country + from kolibri.core.test.test_app.models import Publisher + + class CountrySerializer(serializers.ModelSerializer): + class Meta: + model = Country + fields = ("id", "name") + + class PublisherSerializer(serializers.ModelSerializer): + country = CountrySerializer(allow_null=True) + + class Meta: + model = Publisher + fields = ("id", "name", "country") + + class AwardSerializer(serializers.ModelSerializer): + class Meta: + model = Award + fields = ("id", "name") + + class BookSerializer(serializers.ModelSerializer): + class Meta: + model = Book + fields = ("id", "title") + + class AuthorSerializer(serializers.ModelSerializer): + """Two many=True relations + a deep forward-FK chain (publisher→country).""" + + books = BookSerializer(many=True) + awards = AwardSerializer(many=True) + publisher = PublisherSerializer(allow_null=True) + + class Meta: + model = Author + fields = ("id", "name", "books", "awards", "publisher") + + class DerivedAuthorViewset(BaseValuesViewset, ListModelMixin): + """Serializer-derived — auto-defers books, awards, publisher, country.""" + + serializer_class = AuthorSerializer + queryset = Author.objects.all().order_by("name") + + class ExplicitAuthorViewset(BaseValuesViewset, ListModelMixin): + """Explicit values tuple + hand-written consolidate() — the pre-auto-defer pattern.""" + + queryset = Author.objects.all().order_by("name") + + # Flat Author columns only — no joins to many-sided relations. + values = ("id", "name", "publisher_id") + + field_map = {} + + def consolidate(self, items, queryset): + return _explicit_author_consolidate(items) + + return DerivedAuthorViewset, ExplicitAuthorViewset + + +def _build_to_one_fixtures(author_count, shared_publisher): + """Authors with a ``publisher → country`` chain and no to-many relations. + + ``shared_publisher`` decides whether every author points at one publisher (the + case batching wins) or each at its own (the case it can't help). + """ + from kolibri.core.test.test_app.models import Author + from kolibri.core.test.test_app.models import Country + from kolibri.core.test.test_app.models import Publisher + + Author.objects.all().delete() + Publisher.objects.all().delete() + Country.objects.all().delete() + + country = Country.objects.create(name="Testland") + shared = ( + Publisher.objects.create(name="Test House", country=country) + if shared_publisher + else None + ) + for i in range(author_count): + publisher = shared or Publisher.objects.create( + name="House {:04d}".format(i), country=country + ) + Author.objects.create( + id=uuid.UUID(int=i), + name="Author {:04d}".format(i), + email="author{}@example.com".format(i), + publisher=publisher, + ) + + +def _nest_joined_to_one(item): + """Reshape one flat joined row into the nested dicts the serializer declares.""" + country_id = item.pop("publisher__country__id") + publisher_id = item.pop("publisher__id") + item["publisher"] = ( + None + if publisher_id is None + else { + "id": publisher_id, + "name": item.pop("publisher__name"), + "country": ( + None + if country_id is None + else {"id": country_id, "name": item.pop("publisher__country__name")} + ), + } + ) + item.pop("publisher__name", None) + item.pop("publisher__country__name", None) + return item + + +def _nest_joined_publisher(item): + """Single-hop variant of ``_nest_joined_to_one``: publisher, no country.""" + publisher_id = item.pop("publisher__id") + name = item.pop("publisher__name") + item["publisher"] = ( + None if publisher_id is None else {"id": publisher_id, "name": name} + ) + return item + + +def _build_to_one_viewsets(): + """The two to-one strategies for the same output: auto-deferred follow-up + queries, versus the joined columns they replaced. + + The main sweep can't show this trade-off — its baseline ``consolidate()`` + defers too. The joined arm is the removed path: the target's columns ride + along on the parent query and are reshaped in Python. + """ + from rest_framework import serializers + + from kolibri.core.api import BaseValuesViewset + from kolibri.core.api import ListModelMixin + from kolibri.core.test.test_app.models import Author + from kolibri.core.test.test_app.models import Country + from kolibri.core.test.test_app.models import Publisher + + class CountrySerializer(serializers.ModelSerializer): + class Meta: + model = Country + fields = ("id", "name") + + class PublisherSerializer(serializers.ModelSerializer): + country = CountrySerializer(allow_null=True) + + class Meta: + model = Publisher + fields = ("id", "name", "country") + + class ChainAuthorSerializer(serializers.ModelSerializer): + """Two to-one hops: author → publisher → country.""" + + publisher = PublisherSerializer(allow_null=True) + + class Meta: + model = Author + fields = ("id", "name", "publisher") + + class FlatPublisherSerializer(serializers.ModelSerializer): + class Meta: + model = Publisher + fields = ("id", "name") + + class SingleHopAuthorSerializer(serializers.ModelSerializer): + """One to-one hop — the common production shape (e.g. ``Lesson.classroom``).""" + + publisher = FlatPublisherSerializer(allow_null=True) + + class Meta: + model = Author + fields = ("id", "name", "publisher") + + class DeferredChainViewset(BaseValuesViewset, ListModelMixin): + serializer_class = ChainAuthorSerializer + queryset = Author.objects.all().order_by("name") + + class JoinedChainViewset(BaseValuesViewset, ListModelMixin): + queryset = Author.objects.all().order_by("name") + + values = ( + "id", + "name", + "publisher__id", + "publisher__name", + "publisher__country__id", + "publisher__country__name", + ) + + field_map = {} + + def consolidate(self, items, queryset): + return [_nest_joined_to_one(item) for item in items] + + class DeferredSingleHopViewset(BaseValuesViewset, ListModelMixin): + serializer_class = SingleHopAuthorSerializer + queryset = Author.objects.all().order_by("name") + + class JoinedSingleHopViewset(BaseValuesViewset, ListModelMixin): + queryset = Author.objects.all().order_by("name") + + values = ("id", "name", "publisher__id", "publisher__name") + + field_map = {} + + def consolidate(self, items, queryset): + return [_nest_joined_publisher(item) for item in items] + + return { + "1 hop": (DeferredSingleHopViewset, JoinedSingleHopViewset), + "2 hops": (DeferredChainViewset, JoinedChainViewset), + } + + +# A whole serialize() call here is well under a millisecond, so one batch cannot +# separate a strategy difference from scheduler noise — an earlier read that +# deferring *won* for shared targets came from a cell whose spread was several +# times its delta. +_TO_ONE_REPEATS = 3 + + +def _repeat_measure(serialize, warmup, iterations): + """Median and half-spread of ``_TO_ONE_REPEATS`` independent measurements.""" + runs = [ + _measure_strategy(serialize, warmup, iterations) for _ in range(_TO_ONE_REPEATS) + ] + times = [run["mean_ms"] for run in runs] + return { + "queries": runs[0]["queries"], + "mean_ms": statistics.median(times), + "spread_ms": (max(times) - min(times)) / 2, + } + + +def _to_one_size_report( + to_one_viewsets, depth, author_count, shared_publisher, warmup, iterations +): + deferred_cls, joined_cls = to_one_viewsets[depth] + _build_to_one_fixtures(author_count, shared_publisher) + qs = _autodefer_author_queryset + deferred = deferred_cls() + joined = joined_cls() + + return { + "depth": depth, + "authors": author_count, + "shared_publisher": shared_publisher, + "output_equal": _normalise(deferred.serialize(qs())) + == _normalise(joined.serialize(qs())), + "deferred": _repeat_measure( + lambda: deferred.serialize(qs()), warmup, iterations + ), + "joined": _repeat_measure(lambda: joined.serialize(qs()), warmup, iterations), + } + + +def _print_to_one_report(sizes): + """Report the isolated cost of deferring a to-one against joining it. + + ``noise`` is the larger arm spread over the delta. Above ~0.5 the two arms + are indistinguishable at that point and the percentage means nothing. + """ + logger.info("\n=== to-one: deferred vs joined (isolated) ===") + for size in sizes: + d, j = size["deferred"], size["joined"] + delta = d["mean_ms"] - j["mean_ms"] + noise = ( + max(d["spread_ms"], j["spread_ms"]) / abs(delta) if delta else float("inf") + ) + logger.info( + "%-7s %4d authors %-8s output_equal=%s queries %d/%d " + "ms %.3f+-%.3f/%.3f+-%.3f deferred %+.1f%% noise %.2f%s", + size["depth"], + size["authors"], + "shared" if size["shared_publisher"] else "distinct", + size["output_equal"], + d["queries"], + j["queries"], + d["mean_ms"], + d["spread_ms"], + j["mean_ms"], + j["spread_ms"], + delta / j["mean_ms"] * 100, + noise, + " (INDISTINGUISHABLE)" if noise > 0.5 else "", + ) + logger.info( + "Isolated cost only — these viewsets do nothing but the to-one, so the " + "round trip is most of the measurement. On real viewsets it costs " + "0-0.15ms; see Automatic Deferral in api_patterns.rst." + ) + + +def _sort_key(obj): + """Stable sort key for a JSON-compatible value (dict/list/scalar).""" + if isinstance(obj, dict): + return repr(sorted((k, _sort_key(v)) for k, v in obj.items())) + if isinstance(obj, list): + return repr([_sort_key(x) for x in obj]) + return repr(obj) + + +def _normalise_item(item): + """Recursively sort list fields so ordering doesn't affect equality. Scalars + are kept typed — both paths read through ``.values()``, so equivalent output + is identically typed; stringifying would mask a real type divergence (e.g. + ``id: 1`` vs ``id: "1"``) between the two strategies.""" + result = {} + for k, v in item.items(): + if isinstance(v, list): + result[k] = sorted( + [_normalise_item(x) if isinstance(x, dict) else x for x in v], + key=_sort_key, + ) + elif isinstance(v, dict): + result[k] = _normalise_item(v) + else: + result[k] = v + return result + + +def _normalise(items): + """Return a sorted, normalised representation of a serialized list.""" + return sorted((_normalise_item(item) for item in items), key=_sort_key) + + +def _autodefer_size_report( + derived_cls, explicit_cls, author_count, books_per_author, warmup, iterations +): + """Compare the auto-deferred derived path to hand-written explicit + consolidate() at one (authors, books-per-author) point. Books-per-author is + the reverse-FK fan-out.""" + from kolibri.core.test.test_app.models import Author + from kolibri.core.test.test_app.models import Country + from kolibri.core.test.test_app.models import Publisher + + # Clean slate so sizes don't accumulate (cascades to books/awards). + Author.objects.all().delete() + Publisher.objects.all().delete() + Country.objects.all().delete() + _build_autodefer_fixtures(author_count, books_per_author=books_per_author) + + qs = _autodefer_author_queryset + derived = derived_cls() + explicit = explicit_cls() + + derived_out = _normalise(derived.serialize(qs())) + explicit_out = _normalise(explicit.serialize(qs())) + + return { + "authors": author_count, + "books_per_author": books_per_author, + "output_equal": derived_out == explicit_out, + "derived": _measure_strategy( + lambda: derived.serialize(qs()), warmup, iterations + ), + "explicit": _measure_strategy( + lambda: explicit.serialize(qs()), warmup, iterations + ), + } + + +def _print_autodefer_report(report): + for size in report["sizes"]: + logger.info( + "\n=== %d authors x %d books ===", + size["authors"], + size["books_per_author"], + ) + d, e = size["derived"], size["explicit"] + logger.info( + "derived vs explicit: output_equal=%s queries %d/%d ms %.3f/%.3f", + size["output_equal"], + d["queries"], + e["queries"], + d["mean_ms"], + e["mean_ms"], + ) + logger.info( + "\nderived query count fixed across fan-out: %s", + report["derived_queries_fixed"], + ) + _print_to_one_report(report["to_one_sizes"]) + + +def _run_autodefer_compare(args): + """In-process derived-vs-explicit comparison over a throwaway test database. + + Confirms the auto-deferred serializer-derived path produces identical output + to a hand-written explicit consolidate(), and that its query count stays + fixed as the reverse-FK fan-out grows — the derived path avoids joins, so it + issues a fixed number of extra queries (an accepted trade-off), never one + that scales with row count. Records each path's timing across the sweep; the + derived path is expected to run more queries than the join-using baseline. + """ + from django.test.utils import setup_test_environment + from django.test.utils import teardown_test_environment + + setup_kolibri(inherit_kolibri_home=args.inherit_kolibri_home) + + # Register the test-only app so its tables are created in the test DB. + import kolibri.core.test # noqa: F401 + + # (authors, books_per_author): sweep reverse-FK fan-out. + sizes = [(100, 3), (100, 10), (100, 30), (100, 100), (100, 300)] + # (authors, shared_publisher): sweep the to-one trade-off over page size and + # target cardinality — deferring is one extra round trip to stop repeating the + # target's columns per parent row, so it should win when targets are shared and + # lose as distinct targets approach the parent count. + to_one_sizes = [ + (depth, n, shared) + for depth in ("1 hop", "2 hops") + for n in (25, 100, 500) + for shared in (True, False) + ] + # DB-backed: each iteration hits the DB, so cap well below the in-memory default. + iterations = min(args.iterations, 50) + + setup_test_environment() + old_config = connection.creation.create_test_db(verbosity=0, autoclobber=True) + try: + derived_cls, explicit_cls = _build_autodefer_viewsets() + to_one_viewsets = _build_to_one_viewsets() + + report = { + "mode": "autodefer-compare", + "sizes": [ + _autodefer_size_report( + derived_cls, explicit_cls, n, books, args.warmup, iterations + ) + for n, books in sizes + ], + "to_one_sizes": [ + _to_one_size_report( + to_one_viewsets, depth, n, shared, args.warmup, iterations + ) + for depth, n, shared in to_one_sizes + ], + } + finally: + connection.creation.destroy_test_db(old_config, verbosity=0) + teardown_test_environment() + + # The derived path's query count must not grow with fan-out. Authors are + # held at 100 while books-per-author sweeps, so a constant derived count + # across sizes is the fixed-not-N-based guarantee. + derived_query_counts = {s["derived"]["queries"] for s in report["sizes"]} + report["derived_queries_fixed"] = len(derived_query_counts) == 1 + report["passed"] = ( + all(s["output_equal"] for s in report["sizes"]) + and all(s["output_equal"] for s in report["to_one_sizes"]) + and report["derived_queries_fixed"] + ) + + if not args.quiet: + _print_autodefer_report(report) + + output_path = args.output or "autodefer_benchmark.json" + write_report(report, output_path) + if not args.quiet: + logger.info("\nReport written to: %s", output_path) + return 0 if report["passed"] else 1 + + def main(): args = parse_args() + + compare_autodefer = getattr(args, "compare_autodefer", False) + if compare_autodefer: + return _run_autodefer_compare(args) + if not args.viewset and not args.synthetic: logger.error("Provide a viewset path or use --synthetic") return 1 diff --git a/kolibri/core/api.py b/kolibri/core/api.py index e26995a3202..7a83539bdfe 100644 --- a/kolibri/core/api.py +++ b/kolibri/core/api.py @@ -10,15 +10,8 @@ For more information, see: docs/backend_architecture/index.rst """ -import operator import threading import uuid -from collections import defaultdict -from contextlib import contextmanager -from itertools import groupby -from typing import Any -from typing import Dict -from typing import List from typing import Optional from django.conf import settings @@ -45,9 +38,8 @@ from kolibri.core.discovery.utils.network.client import NetworkClient from kolibri.core.discovery.utils.network.errors import NetworkClientError from kolibri.core.discovery.utils.network.errors import NetworkLocationResponseFailure -from kolibri.core.utils.serializer_introspection import derive_values_from_serializer -from kolibri.core.utils.serializer_introspection import normalize_field_map -from kolibri.core.utils.serializer_introspection import ValuesMethodField # noqa: F401 +from kolibri.core.utils.values_viewset import ValuesEngine +from kolibri.core.utils.values_viewset import ValuesMethodField # noqa: F401 from kolibri.utils import conf from .utils.portal import registerfacility @@ -136,14 +128,14 @@ def get_default_valid_fields(self, queryset, view, context=None): if context is None: context = {} default_fields = set() - db_columns = view._field_map.db_column_map() + db_columns = view._engine.db_column_map # Invert to get {db_col: api_name} for non-promoted renames whose DB column # appears in _values rather than the client-facing name. db_col_to_api = {v: k for k, v in db_columns.items()} # All the fields of the model model_fields = {f.name for f in queryset.model._meta.get_fields()} # Loop through every value in the view's values tuple - for field in view._values: + for field in view._engine.values: # db_column_map() returns the true DB column for renamed fields; for # SQL-promoted renames _values holds the alias, so we resolve it back. db_source = db_columns.get(field, field) @@ -169,7 +161,7 @@ def remove_invalid_fields(self, queryset, fields, view, request): Modified from https://github.com/encode/django-rest-framework/blob/version-3.12.2/rest_framework/filters.py#L259 to do filtering based on valuesviewset setup """ - db_columns = view._field_map.db_column_map() + db_columns = view._engine.db_column_map valid_fields = [ item[0] for item in self.get_valid_fields(queryset, view, {"request": request}) @@ -190,68 +182,6 @@ def remove_invalid_fields(self, queryset, fields, view, request): return ordering -class _ThreadLocalContext(threading.local): - """ - A dict-like context whose contents are thread-local — writes by one - thread don't leak to others. Used as the ``context`` on a shared cached - serializer so the same instance can safely carry per-request context - (``request``, ``view``, ``format``) on each worker thread without - per-request allocation. - - Inheriting from ``threading.local`` gives every thread its own - ``self.__dict__``; the dict-like protocol proxies straight to it. - - ``BaseValuesViewset.serialize()`` populates this from - ``get_serializer_context()`` before running the pipeline and clears it - on exit. - """ - - def __getitem__(self, key): - return self.__dict__[key] - - def __setitem__(self, key, value): - self.__dict__[key] = value - - def __delitem__(self, key): - del self.__dict__[key] - - def __contains__(self, key): - return key in self.__dict__ - - def __iter__(self): - return iter(self.__dict__) - - def __len__(self): - return len(self.__dict__) - - def __repr__(self): - return repr(self.__dict__) - - def get(self, key, default=None): - return self.__dict__.get(key, default) - - def keys(self): - return self.__dict__.keys() - - def values(self): - return self.__dict__.values() - - def items(self): - return self.__dict__.items() - - def update(self, *args, **kwargs): - self.__dict__.update(*args, **kwargs) - - def setdefault(self, key, default=None): - return self.__dict__.setdefault(key, default) - - def pop(self, key, *args): - return self.__dict__.pop(key, *args) - - def clear(self): - self.__dict__.clear() - - class BaseValuesViewset(viewsets.GenericViewSet): """ A viewset that uses a values call to get all model/queryset data in @@ -282,29 +212,6 @@ class BaseValuesViewset(viewsets.GenericViewSet): # rather than joined in the main query. These fields are handled in consolidate(). deferred_fields = () - # Cached itemgetter for pk_field, used in _auto_consolidate for fast groupby - _pk_getter = None - # Cached many=True nested info for groupby consolidation - _joined_many = () - # Cached serializer instance used for introspection and for invoking - # ``ValuesMethodField`` bound methods. The instance is shared across - # requests; its ``context`` is a ``_ThreadLocalContext`` so per-request - # values don't leak between threads. - _cached_serializer = None - # Thread-local context object attached to ``_cached_serializer.context``. - # ``serialize()`` populates it from ``get_serializer_context()`` and - # clears it on exit. - _serializer_context = None - # Cached derived info for deferred fields, keyed by serializer_path. - # Defaults to None; set per-class by _ensure_initialized. - _nested_derived_cache = None - # Cached validation schema for DEBUG mode: (expected_fields, nested_schemas) - # Built once during _ensure_initialized to avoid per-request recomputation. - _validation_schema = None - # Whether this class derives its values from serializer_class. Legacy - # explicit-values viewsets may pair a write-oriented serializer with a - # different read shape, so DEBUG output validation only applies when True. - _serializer_derived = False # Whether _ensure_initialized has run for this class _initialized = False # Guards _ensure_initialized for this class; each subclass gets its own @@ -348,56 +255,21 @@ def _ensure_initialized(cls): @classmethod def _do_initialize(cls): - cls._serializer_context = _ThreadLocalContext() - has_explicit_values = isinstance(getattr(cls, "values", None), tuple) serializer_class = getattr(cls, "serializer_class", None) - cls._serializer_derived = not has_explicit_values - if has_explicit_values: - cls._values = tuple(cls.values) - if not hasattr(cls, "field_map"): - cls.field_map = {} - # Normalize legacy str/callable entries to canonical entry - # objects (SourceFieldEntry/CallableFieldEntry). Produces a - # fresh _LegacyFieldMap so post-init mutation of cls.field_map - # doesn't leak into instance serialization. - cls._field_map = normalize_field_map(cls.field_map) + # TODO(#14302): remove with the legacy explicit values/field_map path. + cls._engine = ValuesEngine.from_explicit( + cls.values, getattr(cls, "field_map", {}) + ) elif serializer_class is not None: - cls._cached_serializer = serializer_class(context=cls._serializer_context) - ( - cls._values, - cls._field_map, - cls._joined_many, - cls._nested_derived_cache, - ) = derive_values_from_serializer( - cls._cached_serializer, - deferred_fields=cls.deferred_fields, - check_constraints=settings.DEBUG, + cls._engine = ValuesEngine.from_serializer( + serializer_class, cls.deferred_fields ) - # Auto-derived: keep _values/_field_map only on cls. Writing to - # ``cls.values`` here would expose the tuple to subclasses via MRO - # so ``has_explicit_values`` would mis-detect it as user-supplied, - # routing the child into the explicit-values path and skipping - # serializer derivation against its own ``serializer_class``. - cls._values = tuple(cls._values) else: raise TypeError( "Either 'values' tuple or 'serializer_class' must be defined" ) - - # Cache pk itemgetter from queryset - queryset = getattr(cls, "queryset", None) - if queryset is not None and hasattr(queryset, "model"): - cls._pk_getter = operator.itemgetter(queryset.model._meta.pk.name) - - # Cache validation schema for DEBUG mode — serializer-derived - # viewsets only; the serializer is the read contract for them. - if settings.DEBUG and cls._serializer_derived: - cls._validation_schema = cls._build_validation_schema( - cls._cached_serializer - ) - cls._initialized = True def __init__(self, *args, **kwargs): @@ -418,24 +290,6 @@ def get_serializer_context(self): "format": getattr(self, "format_kwarg", None), } - @contextmanager - def _serializer_context_scope(self): - """ - Populate the thread-local serializer context for the duration of a - serialization pipeline, clearing it on exit. Re-entrant: nested - scopes (e.g. ``serialize_queryset`` invoked from ``consolidate``) - are no-ops, so the outer scope's context survives until its own - exit. - """ - already_set = bool(self._serializer_context) - if not already_set: - self._serializer_context.update(self.get_serializer_context()) - try: - yield - finally: - if not already_set: - self._serializer_context.clear() - def generate_serializer(self): queryset = getattr(self, "queryset", None) if queryset is None: @@ -448,10 +302,10 @@ def generate_serializer(self): return Serializer # {source: target} for plain renames, so values can be exposed # under the declared name. - mapped_fields = self._field_map.plain_renames() if self._field_map else {} + mapped_fields = self._engine.plain_renames fields = [] extra_kwargs = {} - for value in self._values: + for value in self._engine.values: try: model._meta.get_field(value) if value in mapped_fields: @@ -508,59 +362,19 @@ def _get_lookup_filter(self): def annotate_queryset(self, queryset): return queryset - def get_nested_serializer(self, path: str): - """ - Resolve a dotted path to a nested serializer. - - Args: - path: Dotted path like 'roles' or 'children__grandchildren' - - Returns: - The nested serializer instance - - Raises: - KeyError: If path doesn't resolve to a valid nested serializer + def consolidate(self, items, queryset): """ - if self._cached_serializer is None: - raise AttributeError( - "get_nested_serializer requires serializer-derived values" - ) - serializer = self._cached_serializer - for part in path.split("__"): - field = serializer.fields[part] - # Handle many=True (ListSerializer wraps the actual serializer) - if hasattr(field, "child"): - serializer = field.child - else: - serializer = field - - return serializer - - def _group_items( - self, items: List[Dict[str, Any]], group_by: str - ) -> Dict[Any, List[Dict[str, Any]]]: + Override point for custom consolidation logic. """ - Group items by a field value. - - Args: - items: List of dictionaries - group_by: Field name to group by + return items - Returns: - Dict mapping group_by values to lists of items - """ - result: Dict[Any, List[Dict[str, Any]]] = defaultdict(list) - for item in items: - result[item.get(group_by)].append(item) - return dict(result) - - @staticmethod - def _serialize_flat(queryset, values, field_map): - """Base serialization: values() call + field mapping.""" - queryset = field_map.annotate_queryset(queryset) - if field_map.is_noop(): - return list(queryset.values(*values)) - return [field_map.map_row(item) for item in queryset.values(*values)] + def serialize(self, queryset): + queryset = self.annotate_queryset(queryset) + items = self._engine.serialize(queryset, context=self.get_serializer_context()) + result = self.consolidate(items, queryset) + if settings.DEBUG: + self._engine.validate_output(result) + return result def serialize_queryset( self, @@ -569,243 +383,22 @@ def serialize_queryset( *, group_by: Optional[str] = None, ): - """ - Serialize any queryset using a serializer's field definitions. - - Args: - queryset: Any Django queryset - serializer_path: Dotted path to nested serializer (e.g., 'roles' - or 'files'); ``None`` selects the viewset's own top-level - serializer. - group_by: Optional field to group results by (returns dict of key -> [items]) - - Returns: - List of serialized items, or Dict {group_key: [items]} if group_by specified - """ - if serializer_path is None: - values = self._values - field_map = self._field_map - joined_many = self._joined_many - pk_getter = self._pk_getter - if pk_getter is None: - # Class-level queryset wasn't resolvable at init time - # (viewset uses get_queryset()); cache it now so subsequent - # calls skip the lookup. - pk_getter = operator.itemgetter(queryset.model._meta.pk.name) - self.__class__._pk_getter = pk_getter - else: - if self._nested_derived_cache is None: - raise AttributeError( - "serialize_queryset requires serializer-derived values" - ) - values, field_map, joined_many = self._nested_derived_cache[serializer_path] - pk_getter = operator.itemgetter(queryset.model._meta.pk.name) - - with self._serializer_context_scope(): - items = self._serialize_flat(queryset, values, field_map) - items = self._auto_consolidate(items, joined_many, pk_getter) + """Serialize any queryset using this viewset's serializer field definitions. - # Group if requested - if group_by is not None: - return self._group_items(items, group_by) + Public seam for custom ``consolidate()`` and composite callers. - return items - - @staticmethod - def _build_validation_schema(serializer): - """ - Build a cached validation schema from a serializer. - - Returns (expected_fields, nested_schemas) where: - - expected_fields: frozenset of field names (excluding write_only) - - nested_schemas: dict mapping field_name to nested schema tuples - """ - expected_fields = set() - nested_schemas = {} - - for field_name, field in serializer.fields.items(): - if getattr(field, "write_only", False): - continue - expected_fields.add(field_name) - if hasattr(field, "child") and isinstance(field.child, Serializer): - nested_schemas[field_name] = BaseValuesViewset._build_validation_schema( - field.child - ) - elif isinstance(field, Serializer): - nested_schemas[field_name] = BaseValuesViewset._build_validation_schema( - field - ) - - return (frozenset(expected_fields), nested_schemas) - - def _validate_output(self, items: List[Dict[str, Any]]) -> None: - """ - Validate serialized output matches serializer contract. - - Only intended for use in DEBUG mode to catch drift between - consolidate() implementation and serializer declarations. - - Uses the cached _validation_schema when available (built during - _ensure_initialized), falling back to building from the serializer. - - Only applies to serializer-derived viewsets — legacy explicit-values - viewsets may pair a write-oriented serializer with a different read - shape, so the serializer is not their read contract. - """ - if not items or not self._serializer_derived: - return - - schema = self._validation_schema - if schema is None: - # Class was initialized under DEBUG=False; build from the - # serializer cached during derivation. - schema = self._build_validation_schema(self._cached_serializer) - - self._validate_items_against_schema(items, schema) - - @staticmethod - def _validate_items_against_schema( - items: List[Dict[str, Any]], - schema, - ) -> None: - """ - Validate items against a cached validation schema. - - Only checks the first item since all rows from values() have - uniform keys — one item is enough to catch schema drift. - Recurses into nested schemas. - """ - if not items: - return - - expected_fields, nested_schemas = schema - item = items[0] - item_keys = set(item.keys()) - - missing = expected_fields - item_keys - if missing: - raise ValueError( - "Missing fields in output: {}. Expected: {}, Got: {}".format( - missing, expected_fields, item_keys - ) - ) - - extra = item_keys - expected_fields - if extra: - raise ValueError( - "Unexpected fields in output: {}. Expected: {}, Got: {}".format( - extra, expected_fields, item_keys - ) - ) - - for field_name, nested_schema in nested_schemas.items(): - nested_value = item.get(field_name) - if nested_value is None: - continue - if isinstance(nested_value, dict): - nested_value = [nested_value] - if isinstance(nested_value, list): - BaseValuesViewset._validate_items_against_schema( - nested_value, nested_schema - ) - - @staticmethod - def _get_nested_child_pk(field_name, nested_pk, val): - """Extract a nested child's PK, raising on missing keys. - - When nested_pk is None the field is a scalar from a one-to-many - relation (e.g. roles__kind); the value itself is the dedup key. + :param serializer_path: dotted-underscore nested-serializer path (e.g. + ``'roles'``, ``'files__metadata'``); ``None`` selects the top-level + serializer. + :param group_by: if set, return a ``{value: [items]}`` dict keyed by that + field instead of a flat list. """ - if nested_pk is None: - return val - try: - return val[nested_pk] - except KeyError: - raise KeyError( - "_auto_consolidate: nested field '{}' has no key " - "'{}' for deduplication. Available keys: {}. " - "Check that _resolve_nested_pk_output_name matches " - "the field_map output.".format(field_name, nested_pk, list(val.keys())) - ) - - def _auto_consolidate( - self, - items: List[Dict[str, Any]], - joined_many, - pk_getter, - ) -> List[Dict[str, Any]]: - """ - Consolidate many=True nested fields using groupby. - - Nested extraction is already done by field_map callables during - _serialize_flat. This method only handles groupby + list collection - for many=True fields (converting per-row dicts to lists and deduplicating). - - Items must be sorted by PK for groupby, but original queryset - ordering is restored afterwards. - - ``joined_many`` and ``pk_getter`` are passed by the caller so the - same routine handles top-level and nested-path consolidation - (``serialize_queryset`` reads them from the cache entry / nested - queryset's model). - """ - if not items or not joined_many: - return items - - # dict.fromkeys deduplicates while preserving insertion order (a set - # would not), so consolidated items can be returned in the original - # queryset order. - original_pk_order = list(dict.fromkeys(pk_getter(item) for item in items)) - # groupby only groups *consecutive* equal keys, so items must be - # sorted by PK first or a custom queryset ordering could split one - # PK's rows into multiple groups. - items = sorted(items, key=pk_getter) - consolidated: Dict[Any, Dict[str, Any]] = {} - - for pk, group in groupby(items, pk_getter): - group_iter = iter(group) - consolidated_item = next(group_iter) - seen_child_pks: Dict[str, set] = {fn: set() for fn, _ in joined_many} - - # Convert per-row nested dicts to lists for the first item - for field_name, nested_pk in joined_many: - val = consolidated_item[field_name] - if val is not None: - child_pk = self._get_nested_child_pk(field_name, nested_pk, val) - seen_child_pks[field_name].add(child_pk) - consolidated_item[field_name] = [val] - else: - consolidated_item[field_name] = [] - - for item in group_iter: - for field_name, nested_pk in joined_many: - val = item[field_name] - if val is not None: - child_pk = self._get_nested_child_pk(field_name, nested_pk, val) - if child_pk not in seen_child_pks[field_name]: - seen_child_pks[field_name].add(child_pk) - consolidated_item[field_name].append(val) - consolidated[pk] = consolidated_item - - return [consolidated[pk] for pk in original_pk_order] - - def consolidate(self, items, queryset): - """ - Override point for custom consolidation logic. - """ - return items - - def serialize(self, queryset): - queryset = self.annotate_queryset(queryset) - with self._serializer_context_scope(): - items = self.serialize_queryset(queryset) - result = self.consolidate(items, queryset) - - # Dev-mode validation: check output matches serializer contract - if settings.DEBUG: - self._validate_output(result) - - return result + return self._engine.serialize( + queryset, + path=serializer_path or "", + group_by=group_by, + context=self.get_serializer_context(), + ) def serialize_object(self, **filter_kwargs): try: diff --git a/kolibri/core/auth/viewsets/facility.py b/kolibri/core/auth/viewsets/facility.py index 3eb1b1f41e8..ee2fbc43a0e 100644 --- a/kolibri/core/auth/viewsets/facility.py +++ b/kolibri/core/auth/viewsets/facility.py @@ -19,7 +19,7 @@ from kolibri.core.api import ValuesViewset from kolibri.core.device.permissions import IsSuperuser from kolibri.core.query import SQCount -from kolibri.core.utils.serializer_introspection import ValuesMethodField +from kolibri.core.utils.values_viewset import ValuesMethodField from ..constants import facility_presets from ..models import Classroom diff --git a/kolibri/core/exams/test/test_exam_api.py b/kolibri/core/exams/test/test_exam_api.py index 4c3a2a36370..0994060d3e6 100644 --- a/kolibri/core/exams/test/test_exam_api.py +++ b/kolibri/core/exams/test/test_exam_api.py @@ -1,5 +1,6 @@ import uuid +from django.test import override_settings from django.urls import reverse from django.utils.timezone import now from le_utils.constants import content_kinds @@ -520,6 +521,55 @@ def test_admin_can_update_instant_report_visibility(self): class ExamAPITestCase(BaseExamTest, APITestCase): class_object = models.Exam + @override_settings(DEBUG=True) + def test_retrieve_serializes_section_missing_optional_fields(self): + # DEBUG-only output-shape validator regression. section_title, description + # and questions are required=False on QuizSectionSerializer, so DRF omits + # them (SkipField) when a stored section lacks them, leaving only the + # defaulted learners_see_fixed_order. Synced/legacy facility data holds + # such sections, and the derived engine raised OutputValidationError on + # them until required_fields was narrowed to required=True fields. Written + # via update() to bypass Exam.save(), which strips questionless sections; + # the bare section is first because the nested validator inspects the + # first element. + exercise_id = uuid.uuid4().hex + question_id = uuid.uuid4().hex + full_section = { + "section_title": "Full Section", + "description": "Every field present", + "learners_see_fixed_order": True, + "questions": [ + { + "exercise_id": exercise_id, + "question_id": question_id, + "title": "Q1", + "counter_in_exercise": 0, + } + ], + } + exam = models.Exam.objects.create( + title="optional-section-fields", + collection=self.classroom, + creator=self.admin, + active=True, + question_sources=[full_section], + ) + models.Exam.objects.filter(pk=exam.id).update( + question_sources=[{"learners_see_fixed_order": False}, full_section] + ) + + self.login_as_admin() + response = self.client.get( + reverse("kolibri:core:exam-detail", kwargs={"pk": exam.id}), + ) + + self.assertEqual(response.status_code, 200) + sections = response.data["question_sources"] + # Absent optionals are omitted, not null-filled. + self.assertEqual(sections[0], {"learners_see_fixed_order": False}) + # A fully populated section round-trips its exact content. + self.assertEqual(sections[1], full_section) + def test_complete_mastery_logs_when_exam_is_closed(self): self.login_as_admin() group = LearnerGroup.objects.create(name="test", parent=self.classroom) diff --git a/kolibri/core/exams/viewsets/exam.py b/kolibri/core/exams/viewsets/exam.py index 9c739ae02ae..f8d663d904e 100644 --- a/kolibri/core/exams/viewsets/exam.py +++ b/kolibri/core/exams/viewsets/exam.py @@ -463,10 +463,12 @@ def get_queryset(self): return Exam.objects.all() def serialize_draft(self, queryset): - # Derive the values to fetch for DraftExam from the serializer-derived _values. + # Derive the values to fetch for DraftExam from the serializer-derived _engine.values. # Exclude Exam-only fields not present on DraftExam, and the assignment_collections # annotation (not available for DraftExam). Add DraftExam-specific JSONFields. - draft_values = tuple(v for v in self._values if v not in _EXAM_ONLY_FIELDS) + ( + draft_values = tuple( + v for v in self._engine.values if v not in _EXAM_ONLY_FIELDS + ) + ( "assignments", "learner_ids", ) diff --git a/kolibri/core/lessons/viewsets/lesson.py b/kolibri/core/lessons/viewsets/lesson.py index 077079cb907..edc5b0625f1 100644 --- a/kolibri/core/lessons/viewsets/lesson.py +++ b/kolibri/core/lessons/viewsets/lesson.py @@ -35,8 +35,8 @@ class ResourceSerializer(Serializer): class ClassroomSerializer(ModelSerializer): - # Use the FK attname column (a UUID) rather than "parent" to avoid - # traversing the relation — ValuesViewset fetches "collection__parent_id" directly. + # Use the FK attname column (a UUID) rather than "parent" to avoid traversing + # the relation, which would auto-defer into a further Collection query. parent = CharField(source="parent_id", read_only=True) class Meta: @@ -56,8 +56,8 @@ class LessonSerializer(ModelSerializer): # Read path: default [] overwritten by consolidate() with adhoc-group member IDs. # Write path: popped in to_internal_value and validated with a temporary field. learner_ids = ValuesMethodField(sources=()) - # source="collection" prefixes child values with "collection__" in the values() query, - # so ClassroomSerializer's parent_id becomes "collection__parent_id". + # Auto-deferred: the lesson query fetches the "collection" FK column, then one + # batched Collection query resolves every distinct classroom. classroom = ClassroomSerializer(source="collection", read_only=True) class Meta: diff --git a/kolibri/core/logger/viewsets/mastery_log.py b/kolibri/core/logger/viewsets/mastery_log.py index a0b196404a1..0e0acd5c822 100644 --- a/kolibri/core/logger/viewsets/mastery_log.py +++ b/kolibri/core/logger/viewsets/mastery_log.py @@ -91,7 +91,7 @@ def diff(self, request, pk=0): tries = list( self.annotate_queryset(self.filter_queryset(self.get_queryset())).values( - *self._values + *self._engine.values )[back : back + 2] ) diff --git a/kolibri/core/mixins.py b/kolibri/core/mixins.py index 2b70c35b87d..6e17419d4a2 100644 --- a/kolibri/core/mixins.py +++ b/kolibri/core/mixins.py @@ -6,7 +6,9 @@ from uuid import UUID from django.core.exceptions import EmptyResultSet +from django.db.models import Field from django.db.models import ForeignKey +from django.db.models import ManyToManyField from django.db.models import QuerySet from django.db.models.fields import CharField from django.db.models.lookups import In @@ -138,6 +140,27 @@ def split_parameter_list_as_sql(self, compiler, connection): ForeignKey.register_lookup(UUIDIn) +class InlineIn(UUIDIn): + """ + ``UUIDIn``'s inline-literal ``IN`` under a name that doesn't claim UUIDs. + + Values go into the statement instead of being bound, so a list of any length + stays one query — Django does not split ``IN`` lists on SQLite, whose + statements cap at ``SQLITE_MAX_VARIABLE_NUMBER``. + + Only safe for values that cannot contain a quote; the caller owns that. + """ + + lookup_name = "inline_in" + + +# Relations need registering separately: ``ForeignObject.get_lookups`` truncates +# the MRO at itself, so a lookup on ``Field`` is invisible to them. +Field.register_lookup(InlineIn) +ForeignKey.register_lookup(InlineIn) +ManyToManyField.register_lookup(InlineIn) + + class UUIDValidationError(Exception): pass diff --git a/kolibri/core/test/test_api.py b/kolibri/core/test/test_api.py index d67ad25e55d..90ee7445fb3 100644 --- a/kolibri/core/test/test_api.py +++ b/kolibri/core/test/test_api.py @@ -1,4 +1,5 @@ import datetime +import uuid from typing import Type from unittest.mock import MagicMock @@ -8,7 +9,6 @@ from django.test import TestCase from django.test.utils import CaptureQueriesContext from django.utils import timezone -from parameterized import parameterized from rest_framework import serializers from kolibri.core.api import BaseValuesViewset @@ -17,13 +17,20 @@ from kolibri.core.api import ValuesViewsetOrderingFilter from kolibri.core.serializers import HexOnlyUUIDField from kolibri.core.test.test_app.models import Author +from kolibri.core.test.test_app.models import Award from kolibri.core.test.test_app.models import Book from kolibri.core.test.test_app.models import Classroom +from kolibri.core.test.test_app.models import Country from kolibri.core.test.test_app.models import DateTimeTzModel from kolibri.core.test.test_app.models import Enrollment +from kolibri.core.test.test_app.models import Hideable +from kolibri.core.test.test_app.models import HideableAccount +from kolibri.core.test.test_app.models import HideableOwner from kolibri.core.test.test_app.models import Profile from kolibri.core.test.test_app.models import Publisher +from kolibri.core.test.test_app.models import Review from kolibri.core.test.test_app.models import Tag +from kolibri.core.utils.values_viewset import OutputValidationError def create_mock_queryset(flat_items, model: Type[Model] = Author): @@ -108,6 +115,15 @@ def make_viewset( ClassroomSerializer = make_serializer( model=Classroom, id=serializers.CharField(), name=serializers.CharField() ) +AwardSerializer = make_serializer( + model=Award, id=serializers.CharField(), name=serializers.CharField() +) +ReviewSerializer = make_serializer( + model=Review, id=serializers.CharField(), rating=serializers.IntegerField() +) +CountrySerializer = make_serializer( + model=Country, id=serializers.CharField(), name=serializers.CharField() +) def author_books_viewset(deferred=False, **extra_author_fields): @@ -132,9 +148,9 @@ def _serialize(viewset, flat_items, **kwargs): def _assert_serialize_raises(test_case, viewset, flat_items, expected_substr): - """Assert that serialize() raises ValueError containing expected_substr.""" + """Assert serialize() raises OutputValidationError containing expected_substr.""" mock_qs = create_mock_queryset(flat_items) - with test_case.assertRaises(ValueError) as ctx: + with test_case.assertRaises(OutputValidationError) as ctx: viewset.serialize(mock_qs) test_case.assertIn(expected_substr, str(ctx.exception)) @@ -151,18 +167,28 @@ class TestDataSerialization(TestCase): A shared ``setUpTestData`` fixture covers every relation type: - - ``alice`` — publisher + profile + 3 books (``book_a3`` has null - description) + 2 classrooms via Enrollment - - ``bob`` — publisher + profile + 1 book, no classrooms - - ``carol`` — orphan: no publisher, no profile, no books, no classrooms - - Tags ``fiction`` + ``classic`` (``book_a1`` has both, ``book_a2`` has - ``fiction``) - - Alice's books × classrooms produces the cartesian needed for dedup tests + - Authors: alice/bob (main_publisher), carol (no publisher). + - Profiles: alice_profile (verified), bob_profile (unverified) — OneToOne rev. + - Books: book_a1/book_a2/book_a3 (alice), book_b1 (bob) — reverse FK many. + - Tags: tag_fiction, tag_classic — M2M fwd (book_a1 has both, book_a2 fiction). + - Classrooms/Enrollments: classroom_101/102, alice in both — M2M-through. + - Awards: award_alice_best/award_alice_honorable (alice), award_bob (bob) — second reverse FK many on Author. + - Reviews: review_a1_1/review_a1_2 on book_a1, review_a2_1 on book_a2, none on book_a3 — grandchild reverse FK many on Book. + - Publisher countries: main_publisher→country_uk (UK), indie_publisher→country_us (US). + - Book publishers: book_a1/book_a2→main_publisher, book_b1→indie_publisher, book_a3→None. """ @classmethod def setUpTestData(cls): - cls.main_publisher = Publisher.objects.create(name="Main House") + cls.country_uk = Country.objects.create(name="UK") + cls.country_us = Country.objects.create(name="US") + + cls.main_publisher = Publisher.objects.create( + name="Main House", country=cls.country_uk + ) + cls.indie_publisher = Publisher.objects.create( + name="Indie", country=cls.country_us + ) cls.alice = Author.objects.create( name="Alice", @@ -187,12 +213,18 @@ def setUpTestData(cls): author=cls.bob, bio="Poet", is_verified=False ) - cls.book_a1 = Book.objects.create(author=cls.alice, title="Alice Book 1") - cls.book_a2 = Book.objects.create(author=cls.alice, title="Alice Book 2") + cls.book_a1 = Book.objects.create( + author=cls.alice, title="Alice Book 1", publisher=cls.main_publisher + ) + cls.book_a2 = Book.objects.create( + author=cls.alice, title="Alice Book 2", publisher=cls.main_publisher + ) cls.book_a3 = Book.objects.create( - author=cls.alice, title="Alice Book 3", description=None + author=cls.alice, title="Alice Book 3", description=None, publisher=None + ) + cls.book_b1 = Book.objects.create( + author=cls.bob, title="Bob Book 1", publisher=cls.indie_publisher ) - cls.book_b1 = Book.objects.create(author=cls.bob, title="Bob Book 1") cls.tag_fiction = Tag.objects.create(name="fiction") cls.tag_classic = Tag.objects.create(name="classic") @@ -204,6 +236,16 @@ def setUpTestData(cls): Enrollment.objects.create(author=cls.alice, classroom=cls.classroom_101) Enrollment.objects.create(author=cls.alice, classroom=cls.classroom_102) + cls.award_alice_best = Award.objects.create(author=cls.alice, name="Best") + cls.award_alice_honorable = Award.objects.create( + author=cls.alice, name="Honorable" + ) + cls.award_bob = Award.objects.create(author=cls.bob, name="Notable") + + cls.review_a1_1 = Review.objects.create(book=cls.book_a1, rating=5) + cls.review_a1_2 = Review.objects.create(book=cls.book_a1, rating=4) + cls.review_a2_1 = Review.objects.create(book=cls.book_a2, rating=3) + def _run(self, viewset): """Run the viewset's own queryset through serialize().""" return viewset.serialize(viewset.get_queryset()) @@ -427,20 +469,6 @@ def test_uuid_field_on_uuid_model_field_passes_through(self): result = self._run(viewset) self.assertEqual(result[0]["author_id"], self.alice.pk) - def test_hex_uuid_field_on_char_model_field_is_passthrough(self): - """HexOnlyUUIDField(format='hex') on a CharField model field (e.g. morango - UUID storage) is a no-op: the DB already returns hex strings, so - to_representation adds no transformation. The field_map.is_noop() - must return True so _serialize_flat can skip dict creation.""" - # Author.email and Author.name are both CharField — simulates morango UUID - viewset = make_viewset( - queryset=Author.objects.none(), - name=serializers.CharField(), - email=HexOnlyUUIDField(), - ) - field_map = viewset._field_map - self.assertTrue(field_map.is_noop()) - def test_plain_serializer_method_field_rejected(self): """Plain ``SerializerMethodField`` is not supported on ValuesViewset; class init raises ``TypeError`` pointing at ``ValuesMethodField``.""" @@ -570,6 +598,7 @@ class MetadataSerializer(serializers.Serializer): def test_fk_single_nested(self): viewset = make_viewset( + model=Book, queryset=Book.objects.filter(pk=self.book_a1.pk), id=serializers.IntegerField(), title=serializers.CharField(), @@ -674,6 +703,113 @@ def test_m2m_through_many_nested(self): names = sorted(c["name"] for c in result[0]["classrooms"]) self.assertEqual(names, ["Room 101", "Room 102"]) + def test_m2m_through_duplicate_rows_yield_one_child(self): + """Two through-rows for the same pair return the child twice from the + join; it must still appear once under its parent.""" + Enrollment.objects.create(author=self.alice, classroom=self.classroom_101) + + viewset = make_viewset( + queryset=Author.objects.filter(pk=self.alice.pk), + id=serializers.UUIDField(), + classrooms=make_nested( + model=Classroom, + many=True, + id=serializers.IntegerField(), + name=serializers.CharField(), + ), + ) + result = self._run(viewset) + names = sorted(c["name"] for c in result[0]["classrooms"]) + self.assertEqual(names, ["Room 101", "Room 102"]) + + # Manager choice: auto-fetch must mirror Django's relation descriptors — + # to-one relations (forward FK, reverse O2O) load through the target's base + # manager, to-many relations (reverse FK, M2M) through its default manager. + + def test_forward_fk_fetches_hidden_target_via_base_manager(self): + """A forward FK is loaded through the base manager (Django's + ForwardManyToOneDescriptor), so a soft-delete-style default-manager + filter on the target model can't null out a referenced row.""" + hidden = Hideable.objects.create(name="secret", hidden=True) + owner = HideableOwner.objects.create(name="owner", featured=hidden) + viewset = make_viewset( + model=HideableOwner, + queryset=HideableOwner.objects.filter(pk=owner.pk), + id=serializers.IntegerField(), + featured=make_nested( + model=Hideable, + allow_null=True, + id=serializers.IntegerField(), + name=serializers.CharField(), + ), + ) + result = self._run(viewset) + self.assertIsNotNone(result[0]["featured"]) + self.assertEqual(result[0]["featured"]["name"], "secret") + + def test_reverse_o2o_fetches_hidden_target_via_base_manager(self): + """Reverse OneToOne loads through the base manager (Django's + ReverseOneToOneDescriptor), so a hidden target still serializes.""" + owner = HideableOwner.objects.create(name="owner") + Hideable.objects.create(name="secret", hidden=True, solo_owner=owner) + viewset = make_viewset( + model=HideableOwner, + queryset=HideableOwner.objects.filter(pk=owner.pk), + id=serializers.IntegerField(), + solo_hideable=make_nested( + model=Hideable, + allow_null=True, + id=serializers.IntegerField(), + name=serializers.CharField(), + ), + ) + result = self._run(viewset) + self.assertIsNotNone(result[0]["solo_hideable"]) + self.assertEqual(result[0]["solo_hideable"]["name"], "secret") + + def test_reverse_fk_many_excludes_hidden_via_default_manager(self): + """Reverse FK many uses the target's default manager (matching the + related manager Django builds), so hidden children are excluded.""" + owner = HideableOwner.objects.create(name="owner") + Hideable.objects.create(name="visible", owner=owner) + Hideable.objects.create(name="secret", hidden=True, owner=owner) + viewset = make_viewset( + model=HideableOwner, + queryset=HideableOwner.objects.filter(pk=owner.pk), + id=serializers.IntegerField(), + hideables=make_nested( + model=Hideable, + many=True, + id=serializers.IntegerField(), + name=serializers.CharField(), + ), + ) + result = self._run(viewset) + names = sorted(h["name"] for h in result[0]["hideables"]) + self.assertEqual(names, ["visible"]) + + def test_m2m_excludes_hidden_via_default_manager(self): + """M2M uses the target's default manager, so hidden children are + excluded (matching Django's M2M related manager).""" + owner = HideableOwner.objects.create(name="owner") + visible = Hideable.objects.create(name="visible") + secret = Hideable.objects.create(name="secret", hidden=True) + owner.tagged.add(visible, secret) + viewset = make_viewset( + model=HideableOwner, + queryset=HideableOwner.objects.filter(pk=owner.pk), + id=serializers.IntegerField(), + tagged=make_nested( + model=Hideable, + many=True, + id=serializers.IntegerField(), + name=serializers.CharField(), + ), + ) + result = self._run(viewset) + names = sorted(h["name"] for h in result[0]["tagged"]) + self.assertEqual(names, ["visible"]) + def test_scalar_many_via_reverse_fk(self): viewset = make_viewset( queryset=Author.objects.filter(pk=self.alice.pk), @@ -706,6 +842,39 @@ def test_scalar_many_via_m2m_through(self): result = self._run(viewset) self.assertEqual(sorted(result[0]["classroom_names"]), ["Room 101", "Room 102"]) + def test_scalar_many_via_to_one_then_to_many(self): + """Scalar source crossing a to-one *then* a to-many + (``publisher.books.title``).""" + viewset = make_viewset( + queryset=Author.objects.filter(pk=self.alice.pk), + id=serializers.UUIDField(), + publisher_book_titles=serializers.CharField(source="publisher.books.title"), + ) + with self.assertNumQueries(2): + result = self._run(viewset) + self.assertEqual( + sorted(result[0]["publisher_book_titles"]), + ["Alice Book 1", "Alice Book 2"], + ) + + def test_scalar_many_via_to_one_then_to_many_excludes_hidden(self): + """Scalar source crossing a to-one then a to-many onto a filtered model + (``owner.hideables.name``): the to-many's default manager applies, so a + hidden row's value drops out. Still one fetch query.""" + owner = HideableOwner.objects.create(name="owner") + Hideable.objects.create(name="visible", owner=owner) + Hideable.objects.create(name="secret", hidden=True, owner=owner) + account = HideableAccount.objects.create(name="acct", owner=owner) + viewset = make_viewset( + model=HideableAccount, + queryset=HideableAccount.objects.filter(pk=account.pk), + id=serializers.IntegerField(), + hideable_names=serializers.CharField(source="owner.hideables.name"), + ) + with self.assertNumQueries(2): + result = self._run(viewset) + self.assertEqual(result[0]["hideable_names"], ["visible"]) + # Consolidation invariants def test_many_rows_same_parent_merge(self): @@ -755,6 +924,31 @@ def test_null_single_fk_produces_null(self): result = self._run(viewset) self.assertIsNone(result[0]["publisher_info"]) + def test_forward_fk_source_column_not_leaked_when_undeclared(self): + """A forward-FK nested serializer whose source column isn't itself a + declared field must not leak that FK column into the output. + + The FK column is fetched only to key the deferred fetch. On the + all-passthrough (noop) path map_row returned the raw row untouched, so + the undeclared source column ("publisher") survived alongside the nested + field ("publisher_info"). + """ + viewset = make_viewset( + queryset=Author.objects.filter(pk=self.alice.pk), + id=serializers.UUIDField(), + name=serializers.CharField(), + publisher_info=make_nested( + model=Publisher, + source="publisher", + id=serializers.IntegerField(), + name=serializers.CharField(), + ), + ) + with self.assertNumQueries(2): + result = self._run(viewset) + self.assertEqual(set(result[0].keys()), {"id", "name", "publisher_info"}) + self.assertEqual(result[0]["publisher_info"]["name"], "Main House") + def test_nullable_first_field_in_nested_not_dropped(self): """Nested row with null first declared field but non-null PK is kept. @@ -778,12 +972,8 @@ def test_nullable_first_field_in_nested_not_dropped(self): self.assertIsNone(book_a3["description"]) def test_duplicate_child_rows_deduplicated(self): - """Cartesian rows (books × classrooms) collapse to actual child counts. - - Alice has 3 books × 2 classrooms = 6 rows via the cartesian, but - the nested list must dedupe to 3 books and the scalar-many to 2 - classroom names. - """ + """A nested many and a scalar-many on the same parent each resolve to + their own child count via independent batched queries.""" viewset = make_viewset( queryset=Author.objects.filter(pk=self.alice.pk), id=serializers.UUIDField(), @@ -813,21 +1003,18 @@ def test_queryset_ordering_preserved(self): ) def test_scalar_many_deduplicates_values(self): - """Duplicates from a cartesian collapse to unique scalar entries. - - Joining both books and classrooms for Alice creates a cartesian - where each book title appears twice (once per classroom). Scalar-many - dedup collapses back to 3 unique titles. - """ + """Repeated scalar values collapse to unique entries.""" + Book.objects.create(author=self.alice, title="Alice Book 1") viewset = make_viewset( queryset=Author.objects.filter(pk=self.alice.pk), id=serializers.UUIDField(), book_titles=serializers.CharField(source="books.title"), - classroom_names=serializers.CharField(source="classrooms.name"), ) result = self._run(viewset) - self.assertEqual(len(result[0]["book_titles"]), 3) - self.assertEqual(len(result[0]["classroom_names"]), 2) + self.assertEqual( + sorted(result[0]["book_titles"]), + ["Alice Book 1", "Alice Book 2", "Alice Book 3"], + ) def test_scalar_many_null_produces_empty_list(self): """Scalar-many with no related rows yields [].""" @@ -839,97 +1026,740 @@ def test_scalar_many_null_produces_empty_list(self): result = self._run(viewset) self.assertEqual(result[0]["book_titles"], []) - def test_method_field_excludes_unshared_source(self): - """A source referenced only by ``ValuesMethodField`` is fetched into - ``values()`` but does not appear in the serialized output row.""" - - class S(serializers.ModelSerializer): - id = serializers.UUIDField() - label = ValuesMethodField(sources=("name",)) + # Auto-defer behaviour tests - def get_label(self, obj): - return "label: {}".format(obj.name) + def test_auto_defer_multi_many_two_reverse_fk(self): + """Two many=True reverse-FK nested serializers auto-defer; parent+books+awards = 3 queries.""" + Ser = make_serializer( + id=serializers.CharField(), + books=BookSerializer(many=True), + awards=AwardSerializer(many=True), + ) + viewset = make_viewset( + serializer_class=Ser, + queryset=Author.objects.filter( + pk__in=[self.alice.pk, self.bob.pk] + ).order_by("name"), + ) + with self.assertNumQueries(3): + result = viewset.serialize(viewset.get_queryset()) + alice, bob = result + self.assertEqual( + sorted(b["title"] for b in alice["books"]), + ["Alice Book 1", "Alice Book 2", "Alice Book 3"], + ) + self.assertEqual( + sorted(a["name"] for a in alice["awards"]), ["Best", "Honorable"] + ) + self.assertEqual([a["name"] for a in bob["awards"]], ["Notable"]) - class Meta: - model = Author - fields = ("id", "label") + def test_auto_defer_multi_many_hex_uuid_parent_pk(self): + """HexOnlyUUIDField parent: bucket on the raw (hyphenated) pk, not the + rendered 32-char hex, or every child comes back []. + parent + books + awards + classrooms = 4 queries.""" + Ser = make_serializer( + id=HexOnlyUUIDField(), + books=BookSerializer(many=True), + awards=AwardSerializer(many=True), + classrooms=ClassroomSerializer(many=True), + ) + viewset = make_viewset( + serializer_class=Ser, + queryset=Author.objects.filter( + pk__in=[self.alice.pk, self.bob.pk] + ).order_by("name"), + ) + with self.assertNumQueries(4): + result = viewset.serialize(viewset.get_queryset()) + alice, bob = result + self.assertEqual( + sorted(b["title"] for b in alice["books"]), + ["Alice Book 1", "Alice Book 2", "Alice Book 3"], + ) + self.assertEqual( + sorted(a["name"] for a in alice["awards"]), ["Best", "Honorable"] + ) + self.assertEqual( + sorted(c["name"] for c in alice["classrooms"]), + ["Room 101", "Room 102"], + ) + self.assertEqual([b["title"] for b in bob["books"]], ["Bob Book 1"]) + self.assertEqual([a["name"] for a in bob["awards"]], ["Notable"]) + self.assertEqual(bob["classrooms"], []) + def test_auto_defer_multi_many_with_m2m(self): + """One reverse-FK + one M2M-through nested serializer auto-defer; parent+books+classrooms = 3 queries.""" + Ser = make_serializer( + id=serializers.CharField(), + books=BookSerializer(many=True), + classrooms=ClassroomSerializer(many=True), + ) viewset = make_viewset( - serializer_class=S, + serializer_class=Ser, queryset=Author.objects.filter(pk=self.alice.pk), ) - result = self._run(viewset) - self.assertEqual(set(result[0].keys()), {"id", "label"}) - self.assertEqual(result[0]["label"], "label: Alice") - - def test_method_field_keeps_shared_source_under_declared_name(self): - """When the method's source is also a declared field, it stays in - output under its declared name.""" - - class S(serializers.ModelSerializer): - id = serializers.UUIDField() - name = serializers.CharField() - label = ValuesMethodField(sources=("name",)) - - def get_label(self, obj): - return "label: {}".format(obj.name) + with self.assertNumQueries(3): + result = viewset.serialize(viewset.get_queryset()) + self.assertEqual(len(result), 1) + alice = result[0] + self.assertEqual( + sorted(b["title"] for b in alice["books"]), + ["Alice Book 1", "Alice Book 2", "Alice Book 3"], + ) + self.assertEqual( + sorted(c["name"] for c in alice["classrooms"]), + ["Room 101", "Room 102"], + ) - class Meta: - model = Author - fields = ("id", "name", "label") + def test_auto_defer_multi_many_without_pk_in_output(self): + """Reverse-FK/M2M children bucket correctly when the serializer omits + the parent pk from its output.""" + Ser = make_serializer( + name=serializers.CharField(), + books=BookSerializer(many=True), + awards=AwardSerializer(many=True), + ) + viewset = make_viewset( + serializer_class=Ser, + queryset=Author.objects.filter( + pk__in=[self.alice.pk, self.bob.pk] + ).order_by("name"), + ) + result = viewset.serialize(viewset.get_queryset()) + alice, bob = result + # pk is not declared output: stripped, but bucketing still works. + self.assertEqual(set(alice), {"name", "books", "awards"}) + self.assertEqual( + sorted(b["title"] for b in alice["books"]), + ["Alice Book 1", "Alice Book 2", "Alice Book 3"], + ) + self.assertEqual( + sorted(a["name"] for a in alice["awards"]), ["Best", "Honorable"] + ) + self.assertEqual([a["name"] for a in bob["awards"]], ["Notable"]) + def test_auto_defer_forward_fk_target_without_pk_in_output(self): + """A deferred forward-FK target serializer that omits its pk resolves.""" + PublisherNoId = make_serializer( + model=Publisher, + name=serializers.CharField(), + country=CountrySerializer(allow_null=True), + ) + Ser = make_serializer( + id=serializers.CharField(), + publisher=PublisherNoId(allow_null=True), + ) viewset = make_viewset( - serializer_class=S, + serializer_class=Ser, queryset=Author.objects.filter(pk=self.alice.pk), ) - result = self._run(viewset) - self.assertEqual(result[0]["name"], "Alice") - self.assertEqual(result[0]["label"], "label: Alice") - - def test_method_field_reads_dotted_source_from_fk(self): - """``sources=('publisher.name',)`` fetches ``publisher__name`` and the - proxy walks it as ``obj.publisher.name``.""" - - class S(serializers.ModelSerializer): - id = serializers.UUIDField() - publisher_label = ValuesMethodField(sources=("publisher.name",)) - - def get_publisher_label(self, obj): - return "pub: {}".format(obj.publisher.name) - - class Meta: - model = Author - fields = ("id", "publisher_label") + result = viewset.serialize(viewset.get_queryset()) + pub = result[0]["publisher"] + self.assertEqual(set(pub), {"name", "country"}) + self.assertEqual(pub["name"], "Main House") + self.assertEqual(pub["country"]["name"], "UK") + def test_auto_defer_forward_fk_leaf_target_without_pk(self): + """A pk-less leaf forward-FK target resolves when reached as a + forward-need target inside a deferred subtree.""" + PublisherLeaf = make_serializer(model=Publisher, name=serializers.CharField()) + BookWithPub = make_serializer( + model=Book, + id=serializers.IntegerField(), + title=serializers.CharField(), + publisher=PublisherLeaf(allow_null=True), + ) + Ser = make_serializer( + id=serializers.CharField(), + books=BookWithPub(many=True), + ) viewset = make_viewset( - serializer_class=S, + serializer_class=Ser, queryset=Author.objects.filter(pk=self.alice.pk), ) - result = self._run(viewset) - self.assertEqual(set(result[0].keys()), {"id", "publisher_label"}) - self.assertEqual(result[0]["publisher_label"], "pub: Main House") - - def test_method_field_reads_context_from_request(self): - """The bound method's ``self.context`` is populated per-request from - ``viewset.get_serializer_context()``.""" - - class S(serializers.ModelSerializer): - id = serializers.UUIDField() - ctx_label = ValuesMethodField(sources=("name",)) - - def get_ctx_label(self, obj): - hint = self.context.get("hint", "missing") - return "{}/{}".format(obj.name, hint) + result = viewset.serialize(viewset.get_queryset()) + books = result[0]["books"] + pub_names = {b["publisher"]["name"] for b in books if b["publisher"]} + self.assertIn("Main House", pub_names) + self.assertTrue( + all(set(b["publisher"]) == {"name"} for b in books if b["publisher"]) + ) - class Meta: - model = Author - fields = ("id", "ctx_label") + def test_auto_defer_reverse_children_without_pk_in_output(self): + """Reverse-FK/M2M child serializers that omit their pk still bucket.""" + TagNoId = make_serializer(model=Tag, name=serializers.CharField()) + ReviewNoId = make_serializer(model=Review, rating=serializers.IntegerField()) + Ser = make_serializer( + model=Book, + id=serializers.IntegerField(), + title=serializers.CharField(), + tags=TagNoId(many=True), + reviews=ReviewNoId(many=True), + ) + viewset = make_viewset( + serializer_class=Ser, + queryset=Book.objects.filter(pk=self.book_a1.pk), + ) + result = viewset.serialize(viewset.get_queryset()) + book = result[0] + self.assertEqual( + sorted(t["name"] for t in book["tags"]), ["classic", "fiction"] + ) + self.assertEqual(sorted(r["rating"] for r in book["reviews"]), [4, 5]) + self.assertTrue(all(set(t) == {"name"} for t in book["tags"])) + def test_auto_defer_many_without_pk_in_output(self): + """An auto-deferred many=True populates even when the parent omits its pk.""" + Ser = make_serializer( + name=serializers.CharField(), + books=BookSerializer(many=True), + ) viewset = make_viewset( - serializer_class=S, + serializer_class=Ser, queryset=Author.objects.filter(pk=self.alice.pk), ) - viewset.get_serializer_context = lambda: {"hint": "yo"} + result = viewset.serialize(viewset.get_queryset()) + alice = result[0] + self.assertEqual(set(alice), {"name", "books"}) + self.assertEqual( + sorted(b["title"] for b in alice["books"]), + ["Alice Book 1", "Alice Book 2", "Alice Book 3"], + ) + + def test_auto_defer_deep_nesting_many_outer(self): + """Deep nesting: books(many){tags(many)} both auto-defer; authors+books+tags = 3 queries.""" + BookWithTagsSer = make_serializer( + model=Book, + id=serializers.IntegerField(), + title=serializers.CharField(), + tags=TagSerializer(many=True), + ) + Ser = make_serializer( + id=serializers.CharField(), + books=BookWithTagsSer(many=True), + ) + viewset = make_viewset( + serializer_class=Ser, + queryset=Author.objects.filter(pk=self.alice.pk), + ) + with self.assertNumQueries(3): + result = viewset.serialize(viewset.get_queryset()) + alice = result[0] + books_by_id = {b["id"]: b for b in alice["books"]} + self.assertEqual( + sorted(t["name"] for t in books_by_id[self.book_a1.pk]["tags"]), + ["classic", "fiction"], + ) + self.assertEqual(books_by_id[self.book_a3.pk]["tags"], []) + + def test_auto_defer_single_fk_deep_nesting(self): + """Forward-FK with nested forward-FK auto-defers; authors+publishers+countries = 3 queries.""" + PublisherWithCountrySer = make_serializer( + model=Publisher, + id=serializers.IntegerField(), + name=serializers.CharField(), + country=CountrySerializer(allow_null=True), + ) + Ser = make_serializer( + id=serializers.CharField(), + publisher=PublisherWithCountrySer(allow_null=True), + ) + viewset = make_viewset( + serializer_class=Ser, + queryset=Author.objects.filter( + pk__in=[self.alice.pk, self.bob.pk, self.carol.pk] + ).order_by("name"), + ) + with self.assertNumQueries(3): + result = viewset.serialize(viewset.get_queryset()) + alice_row = next( + r + for r in result + if r["publisher"] and r["publisher"]["name"] == "Main House" + ) + self.assertEqual(alice_row["publisher"]["country"]["name"], "UK") + carol_row = next(r for r in result if r["publisher"] is None) + self.assertIsNone(carol_row["publisher"]) + + def test_auto_defer_does_not_bind_a_parameter_per_parent(self): + """A deferred fetch must not bind one parameter per parent, or an + unpaginated list blows SQLite's statement variable cap. + + Asserted on parameter counts, not an ``OperationalError``: only a SQLite + older than 3.32 raises at 999. + """ + authors = Author.objects.bulk_create( + Author(name="Bulk {:04d}".format(i), publisher=self.main_publisher) + for i in range(1200) + ) + Book.objects.bulk_create( + Book(author=author, title="Bulk book {}".format(author.name)) + for author in authors + ) + Ser = make_serializer( + id=serializers.CharField(), + books=make_nested( + model=Book, + many=True, + id=serializers.IntegerField(), + title=serializers.CharField(), + ), + publisher=make_nested( + model=Publisher, + allow_null=True, + id=serializers.IntegerField(), + name=serializers.CharField(), + ), + ) + viewset = make_viewset( + serializer_class=Ser, + queryset=Author.objects.filter(name__startswith="Bulk").order_by("name"), + ) + bound = [] + + def record_params(execute, sql, params, many, context): + bound.append(len(params or ())) + return execute(sql, params, many, context) + + with connection.execute_wrapper(record_params): + with self.assertNumQueries(3): + result = viewset.serialize(viewset.get_queryset()) + + self.assertLess(max(bound), 999) + self.assertEqual(len(result), 1200) + self.assertEqual(result[0]["books"][0]["title"], "Bulk book Bulk 0000") + self.assertEqual(result[-1]["publisher"]["name"], "Main House") + + def test_auto_defer_shared_forward_target_merged(self): + """Shared Publisher FK on Author + Book deduplicates to one Publisher query + one Country query = 4 total.""" + PublisherWithCountrySer = make_serializer( + model=Publisher, + id=serializers.IntegerField(), + name=serializers.CharField(), + country=CountrySerializer(allow_null=True), + ) + BookWithPublisherSer = make_serializer( + model=Book, + id=serializers.IntegerField(), + title=serializers.CharField(), + publisher=PublisherWithCountrySer(allow_null=True), + ) + Ser = make_serializer( + id=serializers.CharField(), + publisher=PublisherWithCountrySer(allow_null=True), + books=BookWithPublisherSer(many=True), + ) + viewset = make_viewset( + serializer_class=Ser, + queryset=Author.objects.filter( + pk__in=[self.alice.pk, self.bob.pk] + ).order_by("name"), + ) + with self.assertNumQueries(4): + result = viewset.serialize(viewset.get_queryset()) + alice = result[0] + self.assertEqual(alice["publisher"]["name"], "Main House") + book_a1 = next(b for b in alice["books"] if b["title"] == "Alice Book 1") + self.assertEqual(book_a1["publisher"]["country"]["name"], "UK") + bob = result[1] + self.assertEqual(bob["publisher"]["name"], "Main House") + book_a2 = next(b for b in alice["books"] if b["title"] == "Alice Book 2") + self.assertEqual(book_a2["publisher"]["name"], "Main House") + + def test_auto_defer_same_model_different_shapes_one_fetch_no_leak(self): + """Two forward FKs to one model with different field selections. One + fetch (keyed by model), serialized per shape (keyed by child_path). Rich + (id, name, country) and lean (id) selections don't leak. Budget: 1 + Publisher + 1 Country.""" + RichPublisherSer = make_serializer( + model=Publisher, + id=serializers.IntegerField(), + name=serializers.CharField(), + country=CountrySerializer(allow_null=True), + ) + LeanPublisherSer = make_serializer( + model=Publisher, id=serializers.IntegerField() + ) + BookLeanPublisherSer = make_serializer( + model=Book, + id=serializers.IntegerField(), + title=serializers.CharField(), + publisher=LeanPublisherSer(allow_null=True), + ) + Ser = make_serializer( + id=serializers.CharField(), + publisher=RichPublisherSer(allow_null=True), + books=BookLeanPublisherSer(many=True), + ) + viewset = make_viewset( + serializer_class=Ser, + queryset=Author.objects.filter( + pk__in=[self.alice.pk, self.bob.pk] + ).order_by("name"), + ) + with CaptureQueriesContext(connection) as ctx: + result = viewset.serialize(viewset.get_queryset()) + # One fetch per model, despite the two distinct serializers. + self.assertEqual( + len( + [q for q in ctx.captured_queries if "core_tests_publisher" in q["sql"]] + ), + 1, + ) + self.assertEqual( + len([q for q in ctx.captured_queries if "core_tests_country" in q["sql"]]), + 1, + ) + alice = result[0] + # Author.publisher keeps the rich shape. + self.assertEqual(alice["publisher"]["name"], "Main House") + self.assertEqual(alice["publisher"]["country"]["name"], "UK") + # Book.publisher keeps the lean shape — no name/country leak. + book_with_pub = next(b for b in alice["books"] if b["publisher"]) + self.assertEqual(set(book_with_pub["publisher"].keys()), {"id"}) + + def test_auto_defer_same_model_different_nested_not_mismerged(self): + """Two forward FKs to one model whose serializers differ only in nested + children must not merge. Author.publisher nests books; books' publisher + nests authors. They share flat columns (id, name), so a nesting-blind + merge key would serialize one under the other's shape.""" + AuthorMiniSer = make_serializer( + model=Author, id=serializers.CharField(), name=serializers.CharField() + ) + PublisherWithBooksSer = make_serializer( + model=Publisher, + id=serializers.IntegerField(), + name=serializers.CharField(), + books=BookSerializer(many=True), + ) + PublisherWithAuthorsSer = make_serializer( + model=Publisher, + id=serializers.IntegerField(), + name=serializers.CharField(), + authors=AuthorMiniSer(many=True), + ) + BookWithPublisherSer = make_serializer( + model=Book, + id=serializers.IntegerField(), + title=serializers.CharField(), + publisher=PublisherWithAuthorsSer(allow_null=True), + ) + Ser = make_serializer( + id=serializers.CharField(), + publisher=PublisherWithBooksSer(allow_null=True), + books=BookWithPublisherSer(many=True), + ) + viewset = make_viewset( + serializer_class=Ser, + queryset=Author.objects.filter( + pk__in=[self.alice.pk, self.bob.pk] + ).order_by("name"), + ) + result = viewset.serialize(viewset.get_queryset()) + alice = result[0] + # Author.publisher keeps its books-nesting shape. + self.assertIn("books", alice["publisher"]) + self.assertNotIn("authors", alice["publisher"]) + # Book.publisher keeps its authors-nesting shape — not mismerged to books. + book_with_pub = next(b for b in alice["books"] if b["publisher"]) + self.assertIn("authors", book_with_pub["publisher"]) + self.assertNotIn("books", book_with_pub["publisher"]) + + def test_auto_defer_scalar_subfetch_runs_once_per_path(self): + """Fetch/serialize split: the model fetch is shared, a nested level's own + sub-fetches are not. One Publisher serializer (with a ``book_titles`` + scalar-many) sits at Author.publisher and books' publisher. Publisher is + fetched once (by model), but ``book_titles`` runs per child_path. Budget: + author + books + merged Publisher + 2 book_titles = 5.""" + PublisherWithTitlesSer = make_serializer( + model=Publisher, + id=serializers.IntegerField(), + name=serializers.CharField(), + book_titles=serializers.CharField(source="books.title"), + ) + BookWithPublisherSer = make_serializer( + model=Book, + id=serializers.IntegerField(), + title=serializers.CharField(), + publisher=PublisherWithTitlesSer(allow_null=True), + ) + Ser = make_serializer( + id=serializers.CharField(), + publisher=PublisherWithTitlesSer(allow_null=True), + books=BookWithPublisherSer(many=True), + ) + viewset = make_viewset( + serializer_class=Ser, + queryset=Author.objects.filter( + pk__in=[self.alice.pk, self.bob.pk] + ).order_by("name"), + ) + with CaptureQueriesContext(connection) as ctx: + result = viewset.serialize(viewset.get_queryset()) + self.assertEqual(len(ctx.captured_queries), 5) + # Publisher fetched once (model-keyed)... + self.assertEqual( + len( + [q for q in ctx.captured_queries if "core_tests_publisher" in q["sql"]] + ), + 1, + ) + # ...but the book_titles scalar fetch (filtered on publisher_id) runs + # per path — twice, not deduped across shapes. + self.assertEqual( + len([q for q in ctx.captured_queries if 'publisher_id" IN' in q["sql"]]), + 2, + ) + alice = result[0] + self.assertEqual(alice["publisher"]["name"], "Main House") + self.assertCountEqual( + alice["publisher"]["book_titles"], ["Alice Book 1", "Alice Book 2"] + ) + + def test_auto_defer_explicit_deferred_left_to_dev(self): + """Dev-deferred field is untouched by auto-fetch; auto-deferred authored is fetched = 3 queries.""" + BookWithTagsSer = make_serializer( + model=Book, + id=serializers.IntegerField(), + title=serializers.CharField(), + tags=TagSerializer(many=True), + ) + Ser = make_serializer( + id=serializers.CharField(), + books=BookSerializer(many=True), + authored=BookWithTagsSer(many=True, source="books"), + ) + + class DevDeferViewset(BaseValuesViewset, ListModelMixin): + queryset = Author.objects.filter(pk__in=[self.alice.pk, self.bob.pk]) + serializer_class = Ser + deferred_fields = ("books",) + + def consolidate(self, items, queryset): + for item in items: + item["books"] = ["dev-handled"] + return items + + viewset = DevDeferViewset() + with self.assertNumQueries(3): + result = viewset.serialize(viewset.get_queryset()) + for row in result: + self.assertEqual(row["books"], ["dev-handled"]) + self.assertIsInstance(row["authored"], list) + + def test_auto_defer_emits_debug_log(self): + """Auto-defer engine emits DEBUG logs naming the deferred fields.""" + # Logs fire at viewset construction; build inside the capture block. + with self.assertLogs( + "kolibri.core.utils.values_viewset.introspect", level="DEBUG" + ) as log: + Ser = make_serializer( + id=serializers.CharField(), + books=BookSerializer(many=True), + awards=AwardSerializer(many=True), + ) + viewset = make_viewset( + serializer_class=Ser, + queryset=Author.objects.filter(pk=self.alice.pk), + ) + viewset.serialize(viewset.get_queryset()) + joined = "\n".join(log.output) + self.assertIn("books", joined) + self.assertIn("awards", joined) + + def test_auto_defer_null_forward_target(self): + """No stray query for null forward FK: carol has no publisher, so only the parent query runs.""" + Ser = make_serializer( + id=serializers.CharField(), + publisher=make_nested( + model=Publisher, + allow_null=True, + id=serializers.IntegerField(), + name=serializers.CharField(), + country=make_nested( + model=Country, + allow_null=True, + id=serializers.IntegerField(), + name=serializers.CharField(), + ), + ), + ) + viewset = make_viewset( + serializer_class=Ser, + queryset=Author.objects.filter(pk=self.carol.pk), + ) + with self.assertNumQueries(1): + result = viewset.serialize(viewset.get_queryset()) + self.assertEqual(len(result), 1) + self.assertIsNone(result[0]["publisher"]) + + def test_auto_defer_honours_deep_explicit_defer(self): + """A forward FK explicitly deferred 3 levels deep is NOT auto-fetched. + + Framework auto-fetches author_data + publisher, leaves country to the + dev = 3 queries. If the deep explicit path leaks, country is + auto-fetched (4 queries) and overwrites the dev's value. + """ + Ser = make_serializer( + model=Profile, + id=serializers.IntegerField(), + author_data=make_nested( + model=Author, + source="author", + id=serializers.CharField(), + name=serializers.CharField(), + publisher=make_nested( + model=Publisher, + allow_null=True, + id=serializers.IntegerField(), + name=serializers.CharField(), + country=make_nested( + model=Country, + allow_null=True, + id=serializers.IntegerField(), + name=serializers.CharField(), + ), + ), + ), + ) + + class DeepDeferViewset(BaseValuesViewset, ListModelMixin): + queryset = Profile.objects.filter( + author__in=[self.alice.pk, self.bob.pk] + ).order_by("pk") + serializer_class = Ser + deferred_fields = ("author_data__publisher__country",) + + def consolidate(self, items, queryset): + for item in items: + publisher = item["author_data"]["publisher"] + if publisher is not None: + publisher["country"] = "dev-handled" + return items + + viewset = DeepDeferViewset() + with self.assertNumQueries(3): + result = viewset.serialize(viewset.get_queryset()) + self.assertEqual(len(result), 2) + for row in result: + self.assertEqual(row["author_data"]["publisher"]["name"], "Main House") + self.assertEqual(row["author_data"]["publisher"]["country"], "dev-handled") + + def test_method_field_excludes_unshared_source(self): + """A source referenced only by ``ValuesMethodField`` is fetched into + ``values()`` but does not appear in the serialized output row.""" + + class S(serializers.ModelSerializer): + id = serializers.UUIDField() + label = ValuesMethodField(sources=("name",)) + + def get_label(self, obj): + return "label: {}".format(obj.name) + + class Meta: + model = Author + fields = ("id", "label") + + viewset = make_viewset( + serializer_class=S, + queryset=Author.objects.filter(pk=self.alice.pk), + ) + result = self._run(viewset) + self.assertEqual(set(result[0].keys()), {"id", "label"}) + self.assertEqual(result[0]["label"], "label: Alice") + + def test_method_field_keeps_shared_source_under_declared_name(self): + """When the method's source is also a declared field, it stays in + output under its declared name.""" + + class S(serializers.ModelSerializer): + id = serializers.UUIDField() + name = serializers.CharField() + label = ValuesMethodField(sources=("name",)) + + def get_label(self, obj): + return "label: {}".format(obj.name) + + class Meta: + model = Author + fields = ("id", "name", "label") + + viewset = make_viewset( + serializer_class=S, + queryset=Author.objects.filter(pk=self.alice.pk), + ) + result = self._run(viewset) + self.assertEqual(result[0]["name"], "Alice") + self.assertEqual(result[0]["label"], "label: Alice") + + def test_method_field_source_shared_with_renamed_field(self): + """A source read by both a rename and a method field is not promoted to + a SQL alias: that would drop the column the method still reads.""" + + class S(serializers.ModelSerializer): + id = serializers.UUIDField() + display_name = serializers.CharField(source="name") + label = ValuesMethodField(sources=("name",)) + + def get_label(self, obj): + return "label: {}".format(obj.name) + + class Meta: + model = Author + fields = ("id", "display_name", "label") + + viewset = make_viewset( + serializer_class=S, + queryset=Author.objects.filter(pk=self.alice.pk), + ) + result = self._run(viewset) + self.assertEqual(result[0]["display_name"], "Alice") + self.assertEqual(result[0]["label"], "label: Alice") + + def test_method_field_reads_dotted_source_from_fk(self): + """``sources=('publisher.name',)`` fetches ``publisher__name`` and the + proxy walks it as ``obj.publisher.name``.""" + + class S(serializers.ModelSerializer): + id = serializers.UUIDField() + publisher_label = ValuesMethodField(sources=("publisher.name",)) + + def get_publisher_label(self, obj): + return "pub: {}".format(obj.publisher.name) + + class Meta: + model = Author + fields = ("id", "publisher_label") + + viewset = make_viewset( + serializer_class=S, + queryset=Author.objects.filter(pk=self.alice.pk), + ) + result = self._run(viewset) + self.assertEqual(set(result[0].keys()), {"id", "publisher_label"}) + self.assertEqual(result[0]["publisher_label"], "pub: Main House") + + def test_method_field_reads_context_from_request(self): + """The bound method's ``self.context`` is populated per-request from + ``viewset.get_serializer_context()``.""" + + class S(serializers.ModelSerializer): + id = serializers.UUIDField() + ctx_label = ValuesMethodField(sources=("name",)) + + def get_ctx_label(self, obj): + hint = self.context.get("hint", "missing") + return "{}/{}".format(obj.name, hint) + + class Meta: + model = Author + fields = ("id", "ctx_label") + + viewset = make_viewset( + serializer_class=S, + queryset=Author.objects.filter(pk=self.alice.pk), + ) + viewset.get_serializer_context = lambda: {"hint": "yo"} result = self._run(viewset) self.assertEqual(result[0]["ctx_label"], "Alice/yo") @@ -955,6 +1785,67 @@ def test_serialize_queryset_group_by_returns_dict(self): self.assertEqual(len(result[self.alice.pk]), 3) self.assertEqual(len(result[self.bob.pk]), 1) + def test_serialize_queryset_group_by_unknown_field_raises(self): + """``group_by`` naming no output field raises, not a silent None bucket.""" + viewset = make_viewset( + id=serializers.UUIDField(), + books=make_nested( + model=Book, + many=True, + id=serializers.IntegerField(), + title=serializers.CharField(), + ), + deferred_fields=("books",), + ) + with self.assertRaises(KeyError): + viewset.serialize_queryset(Book.objects.all(), "books", group_by="author") + + def test_auto_defer_two_level_reverse_recursion(self): + """Books auto-defer; inside that deferred subtree the two grandchild + many=True fields — reviews (reverse FK) + tags (M2M) — each auto-defer + into their own batched query instead of cartesian-joining. Budget is + authors + books + reviews + tags = 4, not the 2-query cartesian.""" + BookWithReviewsAndTagsSer = make_serializer( + model=Book, + id=serializers.IntegerField(), + title=serializers.CharField(), + reviews=ReviewSerializer(many=True), + tags=TagSerializer(many=True), + ) + Ser = make_serializer( + id=serializers.CharField(), + books=BookWithReviewsAndTagsSer(many=True), + ) + viewset = make_viewset( + serializer_class=Ser, + queryset=Author.objects.filter(pk=self.alice.pk), + ) + with self.assertNumQueries(4): + result = viewset.serialize(viewset.get_queryset()) + alice = result[0] + books_by_id = {b["id"]: b for b in alice["books"]} + # book_a1: 2 reviews, 2 tags + self.assertEqual( + sorted(r["rating"] for r in books_by_id[self.book_a1.pk]["reviews"]), + [4, 5], + ) + self.assertEqual( + sorted(t["name"] for t in books_by_id[self.book_a1.pk]["tags"]), + ["classic", "fiction"], + ) + # book_a2: 1 review, 1 tag + self.assertEqual( + [r["rating"] for r in books_by_id[self.book_a2.pk]["reviews"]], + [3], + ) + self.assertEqual( + [t["name"] for t in books_by_id[self.book_a2.pk]["tags"]], + ["fiction"], + ) + # book_a3: no reviews, no tags + self.assertEqual(books_by_id[self.book_a3.pk]["reviews"], []) + self.assertEqual(books_by_id[self.book_a3.pk]["tags"], []) + def test_serialize_queryset_consolidates_grand_nested_many(self): """``serialize_queryset`` for a path whose nested serializer itself has a ``many=True`` child must merge the JOIN-multiplied rows into @@ -1000,10 +1891,9 @@ def test_serialize_queryset_consolidates_grand_nested_many(self): def test_serialize_queryset_passes_context_to_nested_method_field(self): """A ``ValuesMethodField`` on a nested serializer must read per-request context via ``self.context`` when reached through - ``serialize_queryset``. The context flows in through the cached - parent's threading-local ``_context``: ``Field.context`` walks - ``self.root._context``, so the nested bound method sees the same - dict the scope manager populated for this request. + ``serialize_queryset``. The per-call ``_MethodContext`` carrier + threads down to the nested ``map_row``, so the method sees this + request's context dict. """ class BookSer(serializers.ModelSerializer): @@ -1033,6 +1923,34 @@ class Meta: ) self.assertEqual(result[0]["title_with_hint"], "Alice Book 1/yo") + def test_method_field_reads_per_request_context_without_leak(self): + """A top-level ``ValuesMethodField`` must not cache request context + between calls on the shared (class-attribute) engine.""" + + class S(serializers.ModelSerializer): + id = serializers.UUIDField() + tagged = ValuesMethodField(sources=("name",)) + + def get_tagged(self, obj): + return "{}/{}".format(obj.name, self.context["request"].tag) + + class Meta: + model = Author + fields = ("id", "tagged") + + viewset = make_viewset( + serializer_class=S, + queryset=Author.objects.filter(pk=self.alice.pk), + ) + + viewset.request = MagicMock(tag="req1") + first = viewset.serialize(viewset.get_queryset()) + viewset.request = MagicMock(tag="req2") + second = viewset.serialize(viewset.get_queryset()) + + self.assertEqual(first[0]["tagged"], "Alice/req1") + self.assertEqual(second[0]["tagged"], "Alice/req2") + def test_nested_path_deferred_with_consolidate(self): """Full pipeline: ``Publisher`` → ``authors`` (deferred at top), ``AuthorSer`` has ``books`` (joined inside the authors query) and @@ -1310,112 +2228,9 @@ class TestDevModeSafeguards(TestCase): identifying info in the error message. """ - @override_settings(DEBUG=True) - def test_multiple_joined_many_nested_raises_error(self): - """Two many=True nested serializers without deferring raise (cartesian product).""" - Ser = make_serializer( - id=serializers.CharField(), - books=BookSerializer(many=True), - classrooms=ClassroomSerializer(many=True), - ) - with self.assertRaises(TypeError) as ctx: - make_viewset(serializer_class=Ser) - self.assertIn("books", str(ctx.exception)) - self.assertIn("classrooms", str(ctx.exception)) - - def test_multiple_joined_many_with_one_deferred_is_fine(self): - """Deferring one of two many-nested serializers avoids the cartesian error.""" - Ser = make_serializer( - id=serializers.CharField(), - books=BookSerializer(many=True), - classrooms=ClassroomSerializer(many=True), - ) - with override_settings(DEBUG=True): - viewset = make_viewset( - serializer_class=Ser, deferred_fields=("classrooms",) - ) - result = _serialize( - viewset, - [{"id": "a1", "books__id": "b1", "books__title": "B1"}], - ) - self.assertEqual(len(result[0]["books"]), 1) - self.assertNotIn("classrooms", result[0]) - - @override_settings(DEBUG=True) - def test_multiple_many_inside_deferred_raises_error(self): - """A deferred nested serializer whose own children include 2+ - un-deferred many=True nested serializers must raise: otherwise - ``serialize_queryset`` for that path would silently emit cartesian - rows (auto-consolidate dedupes but the SQL is over-fetched). - """ - GcA = make_serializer(id=serializers.CharField()) - GcB = make_serializer(id=serializers.CharField()) - InnerSer = make_serializer( - id=serializers.CharField(), - tags=GcA(many=True), - co_authors=GcB(many=True), - ) - Ser = make_serializer( - id=serializers.CharField(), - books=InnerSer(many=True), - ) - with self.assertRaises(TypeError) as ctx: - make_viewset(serializer_class=Ser, deferred_fields=("books",)) - self.assertIn("tags", str(ctx.exception)) - self.assertIn("co_authors", str(ctx.exception)) - - @parameterized.expand( - [ - ("non_many_child_many_gc", False, True, "book"), - ("many_child_many_gc", True, True, "books_outer"), - ("non_many_child_non_many_gc", False, False, "book"), - ("many_child_non_many_gc", True, False, "books_outer"), - ] - ) - @override_settings(DEBUG=True) - def test_deep_nesting_raises_error( - self, _name, child_many, grandchild_many, expected_field - ): - """All deep-nesting shapes (nested-in-nested) raise at viewset instantiation.""" - GC = make_serializer(id=serializers.CharField()) - gc_field = "grandchildren" if grandchild_many else "grandchild" - gc_kwargs = {"many": True} if grandchild_many else {} - ChildSer = make_serializer( - id=serializers.CharField(), **{gc_field: GC(**gc_kwargs)} - ) - child_field = "books_outer" if child_many else "book" - child_kwargs = ( - {"many": True, "source": "books"} if child_many else {"source": "books"} - ) - ParentSer = make_serializer( - id=serializers.CharField(), - **{child_field: ChildSer(**child_kwargs)}, - ) - with self.assertRaises(TypeError) as ctx: - make_viewset(serializer_class=ParentSer) - self.assertIn(expected_field, str(ctx.exception)) - - def test_deep_nesting_with_deferred_avoids_error(self): - """Deferring the deeply-nested field lets derivation succeed.""" - GC = make_serializer(id=serializers.CharField()) - ChildSer = make_serializer( - id=serializers.CharField(), grandchildren=GC(many=True) - ) - ParentSer = make_serializer( - id=serializers.CharField(), - books_outer=ChildSer(many=True, source="books"), - ) - with override_settings(DEBUG=True): - viewset = make_viewset( - serializer_class=ParentSer, - deferred_fields=("books_outer",), - ) - result = _serialize(viewset, [{"id": "a1"}]) - self.assertNotIn("books_outer", result[0]) - @override_settings(DEBUG=True) def test_validate_raises_on_drift_in_flat_output(self): - """consolidate() adding a field not on the serializer raises ValueError.""" + """consolidate() adding a field not on the serializer raises.""" Ser = make_serializer(id=serializers.CharField(), name=serializers.CharField()) class V(BaseValuesViewset, ListModelMixin): @@ -1431,6 +2246,25 @@ def consolidate(self, items, queryset): self, V(), [{"id": "a1", "name": "Alice"}], "unexpected" ) + @override_settings(DEBUG=True) + def test_serialize_object_does_not_mask_drift_as_404(self): + """Drift in a retrieve path propagates, not swallowed into Http404 by + the lookup-error handler.""" + author = Author.objects.create(name="A", email="a@example.com") + Ser = make_serializer(id=serializers.CharField(), name=serializers.CharField()) + + class V(BaseValuesViewset): + queryset = Author.objects.all() + serializer_class = Ser + + def consolidate(self, items, queryset): + for item in items: + item["unexpected"] = "oops" + return items + + with self.assertRaises(OutputValidationError): + V().serialize_object(pk=author.pk) + @override_settings(DEBUG=True) def test_validate_raises_on_drift_in_nested_many_output(self): """consolidate() producing a nested many item missing a field raises.""" @@ -1441,18 +2275,14 @@ def test_validate_raises_on_drift_in_nested_many_output(self): class V(BaseValuesViewset, ListModelMixin): queryset = Author.objects.none() serializer_class = Ser + deferred_fields = ("books",) def consolidate(self, items, queryset): for item in items: item["books"] = [{"id": "b1"}] # missing 'title' return items - _assert_serialize_raises( - self, - V(), - [{"id": "a1", "books__id": "b1", "books__title": "B1"}], - "title", - ) + _assert_serialize_raises(self, V(), [{"id": "a1"}], "title") @override_settings(DEBUG=True) def test_validate_raises_on_drift_in_nested_single_output(self): @@ -1477,10 +2307,7 @@ def consolidate(self, items, queryset): return items _assert_serialize_raises( - self, - V(), - [{"id": "b1", "author__id": "a1", "author__name": "Alice"}], - "name", + self, V(), [{"id": "b1", "author": str(uuid.uuid4())}], "name" ) @override_settings(DEBUG=True) @@ -1499,6 +2326,27 @@ def consolidate(self, items, queryset): _assert_serialize_raises(self, V(), [{"id": "a1", "name": "Alice"}], "name") + @override_settings(DEBUG=True) + def test_validate_catches_consolidate_omitting_read_only_deferred_field(self): + """A ``read_only`` nested field is ``required=False`` in DRF, but + ``consolidate()`` owns filling it — omitting it is drift, not an + optional key. + """ + Ser = make_serializer( + id=serializers.CharField(), + books=BookSerializer(many=True, read_only=True), + ) + + class V(BaseValuesViewset, ListModelMixin): + queryset = Author.objects.none() + serializer_class = Ser + deferred_fields = ("books",) + + def consolidate(self, items, queryset): + return items # never populates "books" + + _assert_serialize_raises(self, V(), [{"id": "a1"}], "books") + @override_settings(DEBUG=True) def test_validate_ignores_write_only_fields(self): """write_only fields missing from output don't trigger validation errors.""" @@ -1509,6 +2357,31 @@ def test_validate_ignores_write_only_fields(self): result = _serialize(viewset, [{"id": "a1"}]) self.assertEqual(result[0], {"id": "a1"}) + @override_settings(DEBUG=True) + def test_validate_allows_absent_optional_nested_fields(self): + """A nested plain ``Serializer`` over a JSON column may declare + ``required=False`` fields. DRF omits them from output (SkipField) when + the stored data lacks them, so the validator must not flag them missing. + """ + + class SectionSerializer(serializers.Serializer): + learners_see_fixed_order = serializers.BooleanField(default=False) + section_title = serializers.CharField(required=False) + description = serializers.CharField(required=False) + + a1 = Author.objects.create( + name="A1", + email="a1@example.com", + metadata=[{"learners_see_fixed_order": True}], + ) + viewset = make_viewset( + queryset=Author.objects.filter(pk=a1.pk), + id=serializers.UUIDField(), + metadata=SectionSerializer(many=True), + ) + result = viewset.serialize(viewset.get_queryset()) + self.assertEqual(result[0]["metadata"], [{"learners_see_fixed_order": True}]) + @override_settings(DEBUG=True) def test_validate_does_not_crash_on_listfield_child(self): """ListField(child=CharField()) in serializer doesn't crash validation. @@ -1584,34 +2457,17 @@ def consolidate(self, items, queryset): @override_settings(DEBUG=True) def test_scalar_many_passes_validation(self): """Scalar-many fields should not trip DEBUG validation (flat list, not dict).""" + a1 = Author.objects.create(name="A1", email="a1@example.com") + Book.objects.create(author=a1, title="B1") + Author.objects.create(name="A2", email="a2@example.com") viewset = make_viewset( - id=serializers.CharField(), + queryset=Author.objects.order_by("name"), + id=serializers.UUIDField(), book_titles=serializers.CharField(source="books.title"), ) - result = _serialize( - viewset, - [ - {"id": "a1", "books__title": "B1"}, - {"id": "a2", "books__title": None}, - ], - ) - self.assertEqual(len(result), 2) - - def test_missing_nested_pk_raises_descriptive_error(self): - """Missing nested-pk key in a row raises with field + key identification.""" - viewset = author_books_viewset() - viewset.__class__._joined_many = (("books", "nonexistent_pk"),) - - flat_items = [ - {"id": "a1", "books__id": "b1", "books__title": "B1"}, - {"id": "a1", "books__id": "b2", "books__title": "B2"}, - ] - with self.assertRaises(KeyError) as ctx: - viewset.serialize(create_mock_queryset(flat_items)) - msg = str(ctx.exception) - self.assertIn("books", msg) - self.assertIn("nonexistent_pk", msg) - self.assertIn("_auto_consolidate", msg) + result = viewset.serialize(viewset.get_queryset()) + self.assertEqual(result[0]["book_titles"], ["B1"]) + self.assertEqual(result[1]["book_titles"], []) def test_missing_source_key_raises_key_error(self): """A field_map entry pointing at a source absent from the row fails fast. @@ -1631,54 +2487,17 @@ class V(BaseValuesViewset, ListModelMixin): class TestAuxiliaryAPIs(TestCase): - """Surfaces beyond ``serialize()``: nested serializer lookup, - separate-queryset serialization (``serialize_queryset``), deferred-field - filtering, and lazy queryset resolution when no class-level ``queryset`` - is defined. + """Surfaces beyond ``serialize()``: separate-queryset serialization + (``serialize_queryset``), deferred-field filtering, and lazy queryset + resolution when no class-level ``queryset`` is defined. """ - def test_get_nested_serializer_direct_path(self): - """Direct nested path returns the corresponding child serializer.""" - viewset = author_books_viewset() - nested = viewset.get_nested_serializer("books") - self.assertIn("id", nested.fields) - self.assertIn("title", nested.fields) - - def test_get_nested_serializer_doubly_nested_path(self): - """Dotted path (e.g., 'books__tags') resolves through deferred nested serializers.""" - TagSer = make_serializer( - model=Tag, - id=serializers.CharField(), - name=serializers.CharField(), - ) - BookSer = make_serializer( - model=Book, - id=serializers.CharField(), - tags=TagSer(many=True), - ) - viewset = make_viewset( - serializer_class=make_serializer( - id=serializers.CharField(), books=BookSer(many=True) - ), - deferred_fields=("books",), - ) - nested = viewset.get_nested_serializer("books__tags") - self.assertEqual(set(nested.fields), {"id", "name"}) - - def test_get_nested_serializer_invalid_path_raises(self): - """A path that doesn't resolve raises KeyError.""" - viewset = make_viewset(id=serializers.CharField()) - with self.assertRaises(KeyError): - viewset.get_nested_serializer("nonexistent") - def test_serialize_queryset_returns_list_of_items(self): """serialize_queryset without group_by returns a flat list.""" viewset = author_books_viewset(deferred=True) - qs = MagicMock() - qs.values.return_value = [ - {"id": "b1", "title": "B1"}, - {"id": "b2", "title": "B2"}, - ] + qs = create_mock_queryset( + [{"id": "b1", "title": "B1"}, {"id": "b2", "title": "B2"}], model=Book + ) result = viewset.serialize_queryset(qs, "books") self.assertEqual(len(result), 2) self.assertEqual(result[0]["title"], "B1") @@ -1695,27 +2514,32 @@ def test_deferred_field_excluded_from_values_call(self): self.assertNotIn("books__title", values_args) self.assertNotIn("books", result[0]) - def test_auto_consolidate_works_without_class_level_queryset(self): - """Viewsets using get_queryset() (no class queryset) still resolve PK lazily.""" + def test_scalar_fetch_resolves_pk_without_class_level_queryset(self): + """A scalar cross-many fetch resolves the parent PK lazily via + get_queryset() when no class-level queryset is defined.""" + author = Author.objects.create(name="A", email="a@example.com") + Book.objects.create(author=author, title="B1") + Book.objects.create(author=author, title="B2") Ser = make_serializer( id=serializers.CharField(), - books=make_nested( - model=Book, - many=True, - id=serializers.CharField(), - title=serializers.CharField(), - ), + book_titles=serializers.CharField(source="books.title"), ) class V(BaseValuesViewset, ListModelMixin): serializer_class = Ser def get_queryset(self): - return Author.objects.none() - - flat_items = [ - {"id": "a1", "books__id": "b1", "books__title": "B1"}, - {"id": "a1", "books__id": "b2", "books__title": "B2"}, - ] - result = _serialize(V(), flat_items) - self.assertEqual(len(result[0]["books"]), 2) + return Author.objects.all() + + result = V().serialize(V().get_queryset()) + self.assertEqual(sorted(result[0]["book_titles"]), ["B1", "B2"]) + + def test_serialize_queryset_raises_on_explicit_values_viewset(self): + """serialize_queryset on an explicit-values engine raises ValueError.""" + + class V(BaseValuesViewset, ListModelMixin): + queryset = Author.objects.none() + values = ("id", "name") + + with self.assertRaises(ValueError): + V().serialize_queryset(Author.objects.none(), "somepath") diff --git a/kolibri/core/test/test_app/models.py b/kolibri/core/test/test_app/models.py index aeb74806a21..f1621525111 100644 --- a/kolibri/core/test/test_app/models.py +++ b/kolibri/core/test/test_app/models.py @@ -44,21 +44,32 @@ class DateTimeTzModel(models.Model): default_timestamp = DateTimeTzField(default=aware_datetime) -# Synthetic relation zoo for test_api.py. -# -# Author is the primary outer model (UUID pk + scalar fields covering the -# types exercised by type-inference tests). The surrounding models provide -# every relation shape the introspection code distinguishes: -# -# - Publisher: FK target (nullable, for flat FK-traversal + null FK tests) -# - Profile: OneToOne to Author (single-nested + reverse 1:1) -# - Book: reverse FK many (via Author.books) + direct M2M (Book.tags) -# - Tag: M2M target (reverse M2M via Tag.books) -# - Enrollment: through-model for Author↔Classroom M2M (Author.classrooms) +# Synthetic relation zoo for test_api.py. Author is the primary outer model +# (UUID pk + scalar fields for type-inference tests); surrounding models cover +# every relation shape introspection distinguishes: +# - Country: FK target for Publisher (nullable, deep FK-traversal tests) +# - Publisher: FK target (nullable, flat FK-traversal + null FK tests) +# - Profile: OneToOne to Author (single-nested + reverse 1:1) +# - Book: reverse FK many via Author.books +# - Tag: M2M target; Book.tags is the forward M2M +# - Enrollment: through-model for Author↔Classroom M2M (Author.classrooms) +# - Award: second reverse-FK many on Author (Author.awards) +# - Review: grandchild reverse-FK many on Book (Book.reviews) + + +class Country(models.Model): + name = models.CharField(max_length=128, default="") class Publisher(models.Model): name = models.CharField(max_length=128, default="") + country = models.ForeignKey( + "Country", + related_name="publishers", + null=True, + blank=True, + on_delete=models.SET_NULL, + ) class Author(models.Model): @@ -105,6 +116,13 @@ class Book(models.Model): title = models.CharField(max_length=128, default="") description = models.CharField(max_length=255, null=True, blank=True) tags = models.ManyToManyField(Tag, related_name="books") + publisher = models.ForeignKey( + Publisher, + related_name="books", + null=True, + blank=True, + on_delete=models.SET_NULL, + ) class Enrollment(models.Model): @@ -114,3 +132,74 @@ class Enrollment(models.Model): classroom = models.ForeignKey( Classroom, related_name="author_enrollments", on_delete=models.CASCADE ) + + +class Award(models.Model): + author = models.ForeignKey(Author, related_name="awards", on_delete=models.CASCADE) + name = models.CharField(max_length=128, default="") + + +class Review(models.Model): + book = models.ForeignKey(Book, related_name="reviews", on_delete=models.CASCADE) + rating = models.IntegerField(default=0) + + +# Manager-semantics fixture: Hideable's default manager filters out hidden rows +# (soft-delete style), reachable from HideableOwner four ways, so auto-fetch's +# base-vs-default manager choice can be pinned per relation direction: +# - featured: forward FK -> base manager (a referenced row is never hidden) +# - solo_hideable: reverse O2O -> base manager +# - hideables: reverse FK -> default manager (hidden rows excluded) +# - tagged: M2M -> default manager + + +class VisibleHideableManager(models.Manager): + def get_queryset(self): + return super().get_queryset().filter(hidden=False) + + +class HideableOwner(models.Model): + name = models.CharField(max_length=128, default="") + featured = models.ForeignKey( + "Hideable", + related_name="featured_for", + null=True, + blank=True, + on_delete=models.SET_NULL, + ) + tagged = models.ManyToManyField("Hideable", related_name="tagged_by") + + +class Hideable(models.Model): + name = models.CharField(max_length=128, default="") + hidden = models.BooleanField(default=False) + owner = models.ForeignKey( + HideableOwner, + related_name="hideables", + null=True, + blank=True, + on_delete=models.CASCADE, + ) + solo_owner = models.OneToOneField( + HideableOwner, + related_name="solo_hideable", + null=True, + blank=True, + on_delete=models.CASCADE, + ) + + objects = VisibleHideableManager() + + +class HideableAccount(models.Model): + # to-one FK onto HideableOwner, so a scalar source can cross a to-one then a + # to-many onto Hideable (``owner.hideables.name``) — exercising the reversed + # relation-path fetch and the to-many's default-manager filtering. + name = models.CharField(max_length=128, default="") + owner = models.ForeignKey( + HideableOwner, + related_name="accounts", + null=True, + blank=True, + on_delete=models.CASCADE, + ) diff --git a/kolibri/core/utils/serializer_introspection.py b/kolibri/core/utils/serializer_introspection.py deleted file mode 100644 index 96bff72f94b..00000000000 --- a/kolibri/core/utils/serializer_introspection.py +++ /dev/null @@ -1,994 +0,0 @@ -from abc import ABCMeta -from abc import abstractmethod -from collections import Counter -from typing import Any -from typing import Callable -from typing import cast -from typing import Dict -from typing import List -from typing import Optional -from typing import Set -from typing import Tuple -from typing import Type -from typing import Union - -from django.core.exceptions import FieldDoesNotExist -from django.db.models import F -from django.db.models import Field -from django.db.models import Model -from django.db.models.fields.related import ForeignObjectRel -from rest_framework import serializers as drf_serializers -from rest_framework.fields import empty -from rest_framework.fields import Field as DrfField -from rest_framework.serializers import ModelSerializer -from rest_framework.serializers import SerializerMethodField -from rest_framework.utils.field_mapping import ClassLookupDict - - -class ValuesMethodField(SerializerMethodField): - """ - ``SerializerMethodField`` variant for ``ValuesViewset``: declares the - row columns the bound method reads. - - ``sources`` paths use DRF dot notation (``"dataset.id"``) and are - translated internally to ORM double-underscore (``"dataset__id"``) for - the ``values()`` query. The bound method receives a proxy over those - declared sources: - - class MySerializer(ModelSerializer): - status = ValuesMethodField(sources=("transfer_status", "last_synced")) - - def get_status(self, obj): - if obj.transfer_status in IN_PROGRESS: - return SYNCING - ... - - Sources that are not also declared as serializer output fields are - fetched but stripped from the final row. - """ - - def __init__(self, *, sources=(), method_name=None, **kwargs): - super().__init__(method_name=method_name, **kwargs) - self.sources = tuple(sources) - - -class _SourcesProxy: - """ - Attribute proxy over a raw ``values()`` row, scoped to the paths - declared in a ``ValuesMethodField``'s ``sources``. - - Supports dotted traversal matching declared paths: for - ``sources=("publisher.name",)``, ``obj.publisher.name`` walks down - and returns ``raw["publisher__name"]``. - - Attribute access outside the declared set raises ``AttributeError`` - with the requested name and the declared sources in the message, so - the boundary is discoverable. - """ - - __slots__ = ("_raw", "_sources", "_prefix") - - def __init__(self, raw, sources, prefix=""): - self._raw = raw - self._sources = sources - self._prefix = prefix - - def __getattr__(self, name): - # Let Python's internal machinery (repr, copy, pickle, etc.) see a - # plain AttributeError for dunder and private attribute lookups - # rather than the ValuesMethodField-framed message below. - if name.startswith("_"): - raise AttributeError(name) - path = "{}__{}".format(self._prefix, name) if self._prefix else name - if path in self._sources: - return self._raw[path] - sep = path + "__" - if any(source.startswith(sep) for source in self._sources): - return _SourcesProxy(self._raw, self._sources, path) - declared = sorted(source.replace("__", ".") for source in self._sources) - raise AttributeError( - "{!r} not declared — ValuesMethodField exposes sources only: " - "{}. Add to sources=, or inline the logic.".format(name, declared) - ) - - -# A row produced by ``queryset.values()`` — dict of flat path → value. -Row = Dict[str, Any] - - -class SourceFieldEntry: - """ - Field map entry for a single-source rename, optionally transformed. - - ``to_repr=None`` means a plain rename. When ``to_repr`` is set and the - raw value is ``None``, ``default`` is substituted — mirrors DRF's - get_attribute fallback for missing relations. - """ - - __slots__ = ("source", "to_repr", "default") - - def __init__( - self, - source: str, - to_repr: Optional[Callable] = None, - default: Any = None, - ): - self.source = source - self.to_repr = to_repr - self.default = default - - def _represent(self, raw: Any) -> Any: - if self.to_repr is None: - return raw - if raw is None: - return self.default - return self.to_repr(raw) - - def extract(self, row: Row) -> Any: - """Produce the output value, reading ``row`` without mutating it.""" - return self._represent(row.get(self.source)) - - def apply(self, key: str, row: Row) -> None: - """Pop the source from ``row`` and write the value under ``key``.""" - row[key] = self._represent(row.pop(self.source)) - - -class CallableFieldEntry: - """ - Field map entry computed by a callable over the whole raw row: nested - extractors, ``ValuesMethodField`` invokers, and legacy user-written - callables. - - ``source`` and ``to_repr`` are ``None`` at the class level: the value - doesn't map to a single DB column, so consumers that introspect sources - (ordering filter, serializer generation) skip these entries. - """ - - __slots__ = ("func",) - - source = None - to_repr = None - - def __init__(self, func: Callable[[Row], Any]): - self.func = func - - def extract(self, row: Row) -> Any: - """Produce the output value, reading ``row`` without mutating it.""" - return self.func(row) - - def apply(self, key: str, row: Row) -> None: - """Write the value under ``key``; legacy callables may mutate ``row`` - themselves (e.g. popping consumed sources).""" - row[key] = self.func(row) - - -# User-written legacy ``field_map = {"x": "source"}`` / ``{"x": callable}`` -# entries are normalized to these classes at ingest by ``normalize_field_map``. -FieldMapEntry = Union[SourceFieldEntry, CallableFieldEntry] -FieldMap = Dict[str, FieldMapEntry] - - -class _BaseFieldMap(dict, metaclass=ABCMeta): - """ - Base field map: a dict of output field name → entry producing its value, - owning row mapping and source introspection over those entries. - """ - - @abstractmethod - def map_row(self, row: Row) -> Row: - """Produce the output row for a raw ``values()`` row.""" - - def source_map(self) -> Dict[str, str]: - """ - ``{target: source}`` pairs for fields backed by a single DB column. - - Callable entries are excluded (``source`` is ``None``) — they may do - arbitrary transformations and don't map cleanly to a single column. - """ - return { - key: entry.source for key, entry in self.items() if entry.source is not None - } - - def plain_renames(self) -> Dict[str, str]: - """ - ``{source: target}`` pairs for plain renames (source set, no - ``to_repr``), so raw values can be exposed under the declared name. - """ - return { - entry.source: key - for key, entry in self.items() - if entry.source is not None and entry.to_repr is None - } - - def is_noop(self) -> bool: - """True if every entry is a trivial passthrough: no transformation - and no renaming. When True, ``_serialize_flat`` can return - ``list(queryset.values(...))`` directly without building new dicts.""" - return all( - isinstance(entry, SourceFieldEntry) - and entry.source == key - and entry.to_repr is None - for key, entry in self.items() - ) - - def db_column_map(self) -> Dict[str, str]: - """ - ``{api_name: db_column}`` for fields where the DB column differs from - the API name. Passthroughs (source == key) are excluded. - - Used by ordering filters to translate API field names to DB columns - and to resolve which DB column backs a given value in ``_values``. - """ - return { - key: entry.source - for key, entry in self.items() - if entry.source is not None and entry.source != key - } - - def annotate_queryset(self, queryset: Any) -> Any: - """No-op default: return queryset unchanged.""" - return queryset - - -class _FieldMap(_BaseFieldMap): - """ - A field map built by serializer introspection, covering every declared - output field (including trivial passthroughs). - - ``map_row`` builds a fresh output dict, reading raw values without - mutating the input row, so the result contains exactly the declared - outputs — method-field sources pulled in for invocation never leak into - output because they're not field_map keys. - - ``_sql_renames`` holds F() aliases precomputed during introspection for - plain source→target renames that can be pushed to SQL. ``annotate_queryset`` - applies them; no further computation is needed at serialization time. - """ - - def __init__(self, *args: Any, **kwargs: Any) -> None: - super().__init__(*args, **kwargs) - self._sql_renames: Dict[str, Any] = {} # {target_key: F(source)} - - def map_row(self, row: Row) -> Row: - return {key: entry.extract(row) for key, entry in self.items()} - - def db_column_map(self) -> Dict[str, str]: - """ - ``{api_name: db_column}`` resolving SQL rename aliases to their - original source columns. - - For SQL-promoted renames the field_map entry was updated to a - passthrough (source == key), so the original column is read from - ``_sql_renames`` instead. Non-promoted renames and passthroughs - fall back to the base implementation. - """ - result = {} - for key, entry in self.items(): - if key in self._sql_renames: - result[key] = self._sql_renames[key].name - elif entry.source is not None and entry.source != key: - result[key] = entry.source - return result - - def annotate_queryset(self, queryset: Any) -> Any: - """Apply pre-computed SQL-level F() aliases to the queryset.""" - if self._sql_renames: - return queryset.annotate(**self._sql_renames) - return queryset - - def promote_renames_to_sql_aliases( - self, values: List[str], model: Type[Model] - ) -> List[str]: - """ - Precompute SQL-level F() aliases for plain source→target renames. - - Promotes a rename when: - - source and target differ (not a trivial passthrough) - - source is referenced by exactly one entry — removing it from the - values() call is safe because no other entry reads that column - - target name does not conflict with a model field (Django raises - ValueError if an annotation shadows a real column) - - Mutates self in place: updates promoted entries to passthroughs and - populates ``_sql_renames`` with ``{target: F(source)}``. - Returns an updated values list with source names replaced by target names - for promoted renames. - """ - model_field_names: Set[str] = {f.name for f in model._meta.get_fields()} # type: ignore[attr-defined] - source_refcount = Counter( - entry.source for entry in self.values() if entry.source is not None - ) - - for target_key, map_entry in list(self.items()): - if ( - isinstance(map_entry, SourceFieldEntry) - and map_entry.source is not None - and map_entry.to_repr is None - and map_entry.source != target_key - and source_refcount[map_entry.source] == 1 - and target_key not in model_field_names - ): - self._sql_renames[target_key] = F(map_entry.source) - self[target_key] = SourceFieldEntry(target_key) - - if self._sql_renames: - promoted_sources = { - f_expr.name: target for target, f_expr in self._sql_renames.items() - } - return [promoted_sources.get(v, v) for v in values] - return values - - -class _LegacyFieldMap(_BaseFieldMap): - """ - A field map normalized from a legacy explicit ``values`` / ``field_map`` - viewset definition. - - ``map_row`` mutates the row in place, popping sources and writing - targets. Required for back-compat with viewsets that rely on - ``values()`` keys passing through to the output when not claimed by a - ``field_map`` entry. - """ - - def map_row(self, row: Row) -> Row: - for key, entry in self.items(): - entry.apply(key, row) - return row - - -# A ``joined_many`` entry: (field_name, nested_pk_output_name) per many=True -# nested serializer, used by _auto_consolidate for dedup. nested_pk_name is -# None for scalar-many fields (dedup by value itself). -JoinedMany = Tuple[Tuple[str, Optional[str]], ...] - -# Nested cache entries keyed by dotted path, built during introspection so -# deferred nested serializers can be serialized separately via the -# ``serialize_queryset`` API. -NestedCacheEntry = Tuple[List[str], _FieldMap, JoinedMany] -NestedCache = Dict[str, NestedCacheEntry] - -# Return shape of the top-level introspection call. -IntrospectionResult = Tuple[List[str], _FieldMap, JoinedMany, NestedCache] - - -def _get_source_path(field: DrfField, field_name: str, prefix: str) -> Optional[str]: - """Extract the source path for a serializer field. - - Converts DRF dot-notation sources (e.g. "parent.name") to Django ORM - double-underscore notation (e.g. "parent__name") for use in values() queries. - """ - source = getattr(field, "source", None) - if source == "*" or isinstance(source, (list, tuple)): - return None - source_path = source if source else field_name - # DRF uses dot notation for relationship traversal (e.g. "parent.name"), - # but Django ORM values() requires double-underscore notation ("parent__name"). - source_path = source_path.replace(".", "__") - return f"{prefix}{source_path}" if prefix else source_path - - -def _is_nested_model_serializer(field: DrfField) -> bool: - """Check if a field is a nested ``ModelSerializer`` (or ``ListSerializer`` - wrapping one). - - Plain ``Serializer`` subclasses (e.g. structural wrappers around a - ``JSONField``) are intentionally excluded — they have no ``Meta.model`` - to introspect and are handled by the regular-field path, where - ``field.to_representation`` runs on the raw value. - """ - if isinstance(field, drf_serializers.ListSerializer): - return isinstance(field.child, ModelSerializer) - return isinstance(field, ModelSerializer) - - -def _source_crosses_many_relation( - model: Optional[Type[Model]], source_path: str -) -> bool: - """ - Check whether a source path crosses a one-to-many or many-to-many relation. - - Used to detect fields like ``roles__kind`` where ``roles`` is a reverse FK, - so the values() query produces multiple rows that need list consolidation. - """ - if model is None: - return False - parts = source_path.split("__") - current_model: Type[Model] = model - for part in parts: - try: - field = current_model._meta.get_field(part) # type: ignore[attr-defined] - except FieldDoesNotExist: - return False - if getattr(field, "one_to_many", False) or getattr( - field, "many_to_many", False - ): - return True - related_model = getattr(field, "related_model", None) - if related_model is None: - return False - current_model = related_model - return False - - -def _get_model_field_for_source( - model: Optional[Type[Model]], source_path: str -) -> Optional[Union[Field, ForeignObjectRel]]: - """ - Walk a source path like 'user__profile__name' to get the final model field. - - Returns the final model field, or None if the path is invalid or doesn't - resolve to a concrete model field. ``_meta.get_field`` returns either a - ``models.Field`` subclass or a ``ForeignObjectRel`` for reverse accessors; - callers only need the ``related_model`` / ``choices`` attributes common - to both. - """ - if model is None: - return None - - parts = source_path.split("__") - current_model: Type[Model] = model - - for i, part in enumerate(parts): - try: - field = current_model._meta.get_field(part) # type: ignore[attr-defined] - except FieldDoesNotExist: - return None - - # If this is the last part, return the field - if i == len(parts) - 1: - return field - - # Otherwise, it must be a relation - get the related model - related_model = getattr(field, "related_model", None) - if related_model is None: - # Not a relation, but we expected more path segments - return None - current_model = related_model - return None - - -def _field_matches_inferred_type( - declared_field: DrfField, - source_path: str, - serializer_class: Type[ModelSerializer], - model: Optional[Type[Model]], -) -> bool: - """ - Check if a declared field matches what DRF's ModelSerializer would auto-generate. - - Returns True if the declared field type is exactly what ModelSerializer - would have inferred for the given model field, meaning we can skip calling - to_representation (it would be effectively a no-op for simple types). - """ - model_field = _get_model_field_for_source(model, source_path) - if model_field is None: - # Field is likely a queryset annotation (no corresponding model field). - # For primitive DRF fields with no explicit default, to_representation - # is a trivial type coercion (int, float, bool) that is identity when the - # DB annotation already returns the correct Python type via output_field. - # Skip to_representation to avoid unnecessary per-row overhead. - if declared_field.default is empty and isinstance( - declared_field, - ( - drf_serializers.IntegerField, - drf_serializers.FloatField, - drf_serializers.BooleanField, - drf_serializers.CharField, - ), - ): - return True - return False - - # For relation fields, check against the serializer's related field class - # (typically PrimaryKeyRelatedField). values() returns the raw FK value - # (e.g. a UUID string), and PrimaryKeyRelatedField.to_representation - # expects a model instance — so when they match, we skip to_representation - # and pass the raw value through, which is already the PK. - # No default check needed here: FK columns always have a value in output; - # any default on the serializer field is for input (deserialization) only. - if getattr(model_field, "related_model", None) is not None: - return type(declared_field) is serializer_class.serializer_related_field - - # Fields with an explicit default need the 3-tuple path so that - # None values (e.g. from a LEFT JOIN miss) get the default substituted. - # The simple rename path doesn't handle None → default. - if declared_field.default is not empty: - return False - - # Special case: fields with choices become ChoiceField - if getattr(model_field, "choices", None): - inferred_class = serializer_class.serializer_choice_field - else: - field_mapping = ClassLookupDict(serializer_class.serializer_field_mapping) - try: - inferred_class = field_mapping[model_field] - except KeyError: - return False - - # UUIDField(format="hex") on a CharField model field: morango's UUIDField - # extends models.CharField (not Django's UUIDField), so DRF's ClassLookupDict - # maps it to CharField via MRO on all backends — including PostgreSQL, even - # though the DB column is native UUID there. morango.UUIDField.from_db_value - # always converts to a 32-char hex string before the value reaches Python, so - # no to_representation transformation is needed regardless of backend. - if ( - inferred_class is drf_serializers.CharField - and isinstance(declared_field, drf_serializers.UUIDField) - and declared_field.uuid_format == "hex" - ): - return True - - # Exact class match only - subclasses may override to_representation - return type(declared_field) is inferred_class - - -def _resolve_nested_null_check_key( - child: ModelSerializer, source: str -) -> Optional[str]: - """ - Resolve the prefixed PK key for a nested serializer's model. - - Used for LEFT JOIN null detection: if the PK is null, the entire nested - object is null (no related row exists). - - Returns the prefixed PK key (e.g., 'child_set__id') or None if the nested - serializer has no model. - """ - child_model = getattr(getattr(type(child), "Meta", None), "model", None) - if child_model is None: - return None - return "{}__{}".format(source, child_model._meta.pk.name) - - -def _resolve_nested_pk_output_name( - null_check_key: str, prefix: str, child_field_map: _FieldMap -) -> str: - """ - Find the output field name of the nested PK, for deduplication in - _auto_consolidate. The null_check_key is the prefixed source name - (e.g. 'child_set__id'); we need to find what child_field_map renames - it to (e.g. 'id' -> 'identifier'), or fall back to the source name. - """ - pk_source = null_check_key[len(prefix) :] - for out_name, map_val in child_field_map.items(): - if map_val.source == pk_source: - return out_name - return pk_source - - -def _make_nested_extractor( - null_check_key: str, - value_pairs: Tuple[Tuple[str, str], ...], - child_field_map: _FieldMap, -) -> Callable[[Row], Optional[Row]]: - """ - Create a field_map callable that extracts a nested item from a raw row. - - Reads all prefixed keys from the raw row without mutating it, builds a - child dict keyed by unprefixed source names, then applies the child's - field_map for renames and transforms. Returns the nested dict or None - if the FK is null. - """ - - def extract(row: Row) -> Optional[Row]: - if row.get(null_check_key) is None: - return None - - nested_item: Row = {} - for source_name, prefixed_key in value_pairs: - nested_item[source_name] = row.get(prefixed_key) - if child_field_map: - nested_item = child_field_map.map_row(nested_item) - return nested_item - - return extract - - -def _deep_nesting_error( - field_name: str, - child: ModelSerializer, - deferred_in_child: Set[str], -) -> Optional[str]: - """Return a deep-nesting error message if ``child`` has further nested - serializers (excluding any deferred at the child level), else ``None``. - """ - if any( - _is_nested_model_serializer(f) and gn not in deferred_in_child - for gn, f in cast(Dict[str, DrfField], child.fields).items() - ): - return ( - "Nested serializer field '{}' contains further nested " - "serializers. Deep nesting is not supported for " - "auto-consolidation. Use deferred_fields to fetch '{}' " - "separately and implement a custom consolidate() " - "method.".format(field_name, field_name) - ) - return None - - -def _check_serializer_constraints( - serializer: ModelSerializer, deferred_fields: Tuple[str, ...] = () -) -> List[str]: - """ - DEBUG-only preflight: validate serializer structure before introspection. - - Returns a list of error messages (empty if no issues). The caller is - responsible for raising; collecting lets a single run surface every - violation across the (possibly recursive) tree at once. - - Checks at this level (and recursively into deferred nested children): - - No deep nesting (nested serializers within nested serializers, - unless the inner one is itself deferred via a nested-path entry) - - At most one many=True nested serializer (multiple cause cartesian products) - """ - unnested_deferred_fields = {p for p in deferred_fields if "__" not in p} - nested_deferred_by_child: Dict[str, List[str]] = {} - for path in deferred_fields: - if "__" in path: - head, tail = path.split("__", 1) - nested_deferred_by_child.setdefault(head, []).append(tail) - - errors: List[str] = [] - many_fields: List[str] = [] - # cast: ``fields`` is a ``cached_property`` that Pyright can't resolve - # to the underlying BindingDict. - for field_name, field in cast(Dict[str, DrfField], serializer.fields).items(): - if not _is_nested_model_serializer(field): - continue - child = cast( - ModelSerializer, - field.child if isinstance(field, drf_serializers.ListSerializer) else field, - ) - if field_name in unnested_deferred_fields: - # Recurse into deferred nested so deep-level violations surface too. - nested_errors = _check_serializer_constraints( - child, tuple(nested_deferred_by_child.get(field_name, ())) - ) - errors.extend("{}: {}".format(field_name, e) for e in nested_errors) - continue - if getattr(field, "write_only", False): - continue - unnested_deferred_in_child = { - p for p in nested_deferred_by_child.get(field_name, ()) if "__" not in p - } - deep_err = _deep_nesting_error(field_name, child, unnested_deferred_in_child) - if deep_err is not None: - errors.append(deep_err) - if isinstance(field, drf_serializers.ListSerializer): - many_fields.append(field_name) - if len(many_fields) > 1: - field_names = ", ".join(sorted(many_fields)) - errors.append( - "Multiple many=True nested serializers cannot be joined in a " - "single query (cartesian product). Found: {}. Move all but one " - "to deferred_fields and handle them in consolidate().".format(field_names) - ) - return errors - - -def _introspect_regular_field( - field_name: str, - field: DrfField, - declared_fields: Dict[str, DrfField], - serializer_class: Type[ModelSerializer], - model: Optional[Type[Model]], -) -> Tuple[Union[str, Tuple[str, ...], None], Optional[FieldMapEntry]]: - """ - Introspect a regular (non-nested) serializer field. - - Returns ``(source_path, entry)``: - - - ``source_path``: value(s) to fetch via ``values()``. ``None`` to skip - the field entirely (e.g. ``source='*'``). For a ``ValuesMethodField``, - a tuple of the declared source paths (caller extends ``values`` with - all of them). - - ``entry``: the field_map entry producing the output value. A - ``CallableFieldEntry`` wrapping an invoker closure for method fields - (wraps the row dict in a ``_SourcesProxy`` and calls the bound - method); a ``SourceFieldEntry`` otherwise, with ``to_repr`` set when - the field transforms its raw value. - """ - if isinstance(field, ValuesMethodField): - source_paths = tuple(source.replace(".", "__") for source in field.sources) - # Capture the bound ``to_representation`` once at introspection time - # so subclass overrides are honoured (and we skip a per-row attribute - # lookup on the field instance). - to_representation = field.to_representation - - def invoke(row: Row) -> Any: - proxy = _SourcesProxy(row, source_paths) - return to_representation(proxy) - - return source_paths, CallableFieldEntry(invoke) - - if isinstance(field, SerializerMethodField): - raise TypeError( - "{}.{}: ValuesViewset does not support plain " - "SerializerMethodField. Use ValuesMethodField(sources=(...)) " - "to declare which row columns the method reads, or a typed " - "field with source= for simple traversals.".format( - serializer_class.__name__, field_name - ) - ) - - source_path = _get_source_path(field, field_name, "") - if source_path is None: - return None, None - - if field_name in declared_fields and not _field_matches_inferred_type( - field, source_path, serializer_class, model - ): - default = field.default if field.default is not empty else None - return source_path, SourceFieldEntry( - source_path, field.to_representation, default - ) - # Trivial passthrough (source == name, matching type) still emits an - # entry so the field_map is a complete spec of output fields. - return source_path, SourceFieldEntry(source_path) - - -def _introspect_deferred_nested( - field_name: str, - child: ModelSerializer, - nested_deferred: Tuple[str, ...] = (), -) -> NestedCache: - """ - Introspect a deferred nested serializer (one listed in deferred_fields). - - Introspects as top-level so the child's own nested serializers are - processed, but does not add values to the parent query. - - Returns a dict of nested_cache entries keyed by path. - """ - ( - child_values, - child_field_map, - child_joined_many, - child_nested, - ) = _introspect_serializer_fields(child, deferred_fields=nested_deferred) - entries: NestedCache = { - field_name: (child_values, child_field_map, child_joined_many), - } - for sub_path, sub_info in child_nested.items(): - entries[f"{field_name}__{sub_path}"] = sub_info - return entries - - -def _introspect_joined_nested( - field_name: str, - field: DrfField, - child: ModelSerializer, - is_many: bool, - nested_deferred: Tuple[str, ...] = (), -) -> Tuple[List[str], FieldMap, NestedCache, List[Tuple[str, Optional[str]]]]: - """ - Introspect a joined (non-deferred) nested serializer. - - Returns ``(prefixed_values, field_map_updates, nested_entries, - joined_many_entries)``. ``joined_many_entries`` is empty for single FK - fields or a one-item list of ``(field_name, nested_pk_name)`` for - many=True fields. - """ - ( - child_values, - child_field_map, - _, - child_nested, - ) = _introspect_serializer_fields( - child, deferred_fields=nested_deferred, _is_nested=True - ) - nested_entries: NestedCache = { - field_name: (child_values, child_field_map, ()), - } - for sub_path, sub_info in child_nested.items(): - nested_entries[f"{field_name}__{sub_path}"] = sub_info - - # Prefix child values for the parent's values() call - source = getattr(field, "source", None) or field_name - prefix = f"{source}__" - prefixed_values = [f"{prefix}{v}" for v in child_values] - - extractor: Optional[Callable[[Row], Optional[Row]]] = None - joined_many_entry: Optional[Tuple[str, Optional[str]]] = None - - if prefixed_values: - value_pairs: Tuple[Tuple[str, str], ...] = tuple( - zip(child_values, prefixed_values) - ) - - null_check_key = _resolve_nested_null_check_key(child, source) - if null_check_key is None: - null_check_key = prefixed_values[0] - - extractor = _make_nested_extractor( - null_check_key, - value_pairs, - child_field_map, - ) - - if is_many: - nested_pk_name = _resolve_nested_pk_output_name( - null_check_key, - prefix, - child_field_map, - ) - joined_many_entry = (field_name, nested_pk_name) - - field_map_updates: FieldMap = {} - if extractor is not None: - field_map_updates[field_name] = CallableFieldEntry(extractor) - joined_many_entries: List[Tuple[str, Optional[str]]] = [] - if joined_many_entry is not None: - joined_many_entries.append(joined_many_entry) - - return prefixed_values, field_map_updates, nested_entries, joined_many_entries - - -def _introspect_serializer_fields( # noqa: C901 - serializer: ModelSerializer, - deferred_fields: Tuple[str, ...] = (), - _is_nested: bool = False, -) -> IntrospectionResult: - """ - Introspect a serializer to derive values tuple and field transformations. - - Args: - serializer: The DRF serializer to introspect - deferred_fields: Field names that should be fetched separately (not joined) - _is_nested: Internal flag; True when recursing into a nested serializer - so that further nested fields are skipped. - - Returns: - - values: Fields to fetch via queryset.values() - - field_map: Transforms for fields to map from values call - - joined_many: many=True nested fields as (field_name, nested_pk_name) for dedup. - nested_pk_name is None for scalar many-relation fields (dedup by value). - - nested_cache: path-keyed dict of (values, field_map, joined_many) for - all nested serializers encountered (deferred and joined alike) - """ - values: List[str] = [] - field_map = _FieldMap() - joined_many_fields: List[Tuple[str, Optional[str]]] = [] - nested_cache: NestedCache = {} - declared_fields: Dict[str, DrfField] = getattr(serializer, "_declared_fields", {}) - - # Get serializer class and model for type inference - serializer_class = type(serializer) - model: Optional[Type[Model]] = getattr( - getattr(serializer_class, "Meta", None), "model", None - ) - - serializer_fields: Dict[str, DrfField] = cast( - Dict[str, DrfField], serializer.fields - ) - for field_name, field in serializer_fields.items(): - if getattr(field, "write_only", False): - continue - - if _is_nested_model_serializer(field): - # --- Nested ModelSerializer fields --- - is_many = isinstance(field, drf_serializers.ListSerializer) - child = cast(ModelSerializer, field.child if is_many else field) - nested_deferred = tuple( - p.split("__", 1)[1] - for p in deferred_fields - if "__" in p and p.split("__", 1)[0] == field_name - ) - - if field_name in deferred_fields: - nested_cache.update( - _introspect_deferred_nested(field_name, child, nested_deferred) - ) - continue - - # Skip further joining when already inside a nested serializer - # (deep JOIN); deferred grand-children are handled above. - if _is_nested: - continue - - # Joined nested field — introspect and build extractor - ( - prefixed, - fm_updates, - nested_entries, - many_entries, - ) = _introspect_joined_nested( - field_name, field, child, is_many, nested_deferred - ) - values.extend(prefixed) - field_map.update(fm_updates) - nested_cache.update(nested_entries) - joined_many_fields.extend(many_entries) - continue - - if field_name in deferred_fields: - continue - - source_path, entry = _introspect_regular_field( - field_name, field, declared_fields, serializer_class, model - ) - if source_path is None: - continue - - # ValuesMethodField returns a tuple of source paths and an - # invoker entry. Extend values() with all of them. - if isinstance(source_path, tuple): - values.extend(source_path) - field_map[field_name] = entry - continue - - # Detect fields whose source crosses a one-to-many relation - # (e.g. roles__kind where roles is a reverse FK). These produce - # multiple rows in values() and need list consolidation — a plain - # rename here, consolidation collects the list. - if _source_crosses_many_relation(model, source_path): - entry = SourceFieldEntry(source_path) - joined_many_fields.append((field_name, None)) - - values.append(source_path) - field_map[field_name] = entry - continue - - # Dedupe values() paths: method-field sources can overlap with declared - # field sources, and the same path only needs to be fetched once. - # Sorted so the column order (and hence the generated SQL) is - # consistent across runs — set iteration order varies per process. - values = sorted(set(values)) - - # Not applied inside inline-joined nested serializers (_is_nested=True): - # their values are fetched with a "{parent}__" prefix and annotation - # names cannot contain "__". - if not _is_nested and model is not None: - values = field_map.promote_renames_to_sql_aliases(values, model) - - return values, field_map, tuple(joined_many_fields), nested_cache - - -def derive_values_from_serializer( - serializer: ModelSerializer, - deferred_fields: Tuple[str, ...] = (), - *, - check_constraints: bool = False, -) -> IntrospectionResult: - """ - Public entry point: derive values tuple and field mappings from a DRF serializer. - - Args: - serializer: The DRF serializer to introspect - deferred_fields: Field names that should be fetched separately (not joined) - check_constraints: When True, runs DEBUG preflight checks before introspection - - Returns: - - values: Fields to fetch via queryset.values() - - field_map: Transforms for fields to map from values call - - joined_many: many=True nested fields as (field_name, nested_pk_name) for dedup - - nested_cache: path-keyed dict of (values, field_map, joined_many) for - all nested serializers encountered during introspection - """ - if check_constraints: - errors = _check_serializer_constraints(serializer, deferred_fields) - if errors: - raise TypeError("\n".join(errors)) - return _introspect_serializer_fields(serializer, deferred_fields=deferred_fields) - - -def normalize_field_map(field_map: Dict[str, Any]) -> _LegacyFieldMap: - """ - Normalize a user-written legacy field_map to canonical entry objects. - - Converts str shorthand entries (``{"name": "source"}``) to - ``SourceFieldEntry`` plain renames and bare callables to - ``CallableFieldEntry``. Returns a new ``_LegacyFieldMap`` with - mutate-in-place ``map_row`` semantics; the input is not mutated. - """ - return _LegacyFieldMap( - ( - key, - ( - SourceFieldEntry(value) - if isinstance(value, str) - else CallableFieldEntry(value) - ), - ) - for key, value in field_map.items() - ) diff --git a/kolibri/core/utils/values_viewset/__init__.py b/kolibri/core/utils/values_viewset/__init__.py new file mode 100644 index 00000000000..a5243215138 --- /dev/null +++ b/kolibri/core/utils/values_viewset/__init__.py @@ -0,0 +1,13 @@ +""" +Serializer-derived ``values()`` querying for Kolibri viewsets. + +``introspect`` reads a serializer into a fetch plan over the ``field_map`` and +``method_fields`` data model, ``fetch`` describes the deferred relations, and +``engine`` runs the plan against a queryset. +""" + +from kolibri.core.utils.values_viewset.engine import OutputValidationError +from kolibri.core.utils.values_viewset.engine import ValuesEngine +from kolibri.core.utils.values_viewset.method_fields import ValuesMethodField + +__all__ = ["OutputValidationError", "ValuesEngine", "ValuesMethodField"] diff --git a/kolibri/core/utils/values_viewset/engine.py b/kolibri/core/utils/values_viewset/engine.py new file mode 100644 index 00000000000..257fed43ab1 --- /dev/null +++ b/kolibri/core/utils/values_viewset/engine.py @@ -0,0 +1,601 @@ +from collections import defaultdict +from functools import partial +from typing import Any +from typing import Callable +from typing import Dict +from typing import List +from typing import NamedTuple +from typing import Optional +from typing import Tuple + +from django.db.models import QuerySet + +from kolibri.core.utils.values_viewset.fetch import AutoFetch +from kolibri.core.utils.values_viewset.fetch import filter_in +from kolibri.core.utils.values_viewset.fetch import ForwardAutoFetch +from kolibri.core.utils.values_viewset.fetch import pk_key +from kolibri.core.utils.values_viewset.fetch import ScalarFetch +from kolibri.core.utils.values_viewset.field_map import MethodFieldEntry +from kolibri.core.utils.values_viewset.field_map import normalize_field_map +from kolibri.core.utils.values_viewset.field_map import Row +from kolibri.core.utils.values_viewset.introspect import derive_values_from_serializer +from kolibri.core.utils.values_viewset.method_fields import MethodContext + + +class ExpandResult(NamedTuple): + """ + One level's expanded rows. ``pks``, ``link_pairs`` and ``raw_by_pk`` are + empty unless the level keys by them — each specialized worker below fills + only the ones its level needs. + """ + + items: List[Row] + pks: List[Any] + # (child_pk, parent_pk) per row, for a reverse-fetched child only. parent_pk + # is read from the child's back-reference FK column (the reverse relation), so + # the fetch can bucket each child onto its parent. Kept index-aligned with + # ``items`` — ReverseAutoFetch zips the two positionally. + link_pairs: List[Any] + raw_by_pk: Dict[Any, Row] + + +LevelExpander = Callable[[Any, Optional["MethodContext"]], ExpandResult] + + +class OutputValidationError(Exception): + """ + DEBUG-only output-shape drift from the serializer's schema. + + Deliberately not a ``ValueError``: callers such as ``serialize_object`` catch + ``ValueError``/``TypeError`` from a failed lookup and raise ``Http404``, so a + ``ValueError`` here would be silently turned into a 404 instead of surfacing + the drift. + """ + + +class DeferredForwardRef(NamedTuple): + """ + A forward FK (or forward OneToOne) whose target is fetched in a later + batched pass, then written back to ``item[field_name]``. Batching lets + same-model targets from across the tree resolve in one query. + """ + + target_model: Any + child_path: str + target_pk: Any + item: Dict[str, Any] + field_name: str + + +class _LevelPlan(NamedTuple): + """A level's serialization compiled to data and a specialized row worker.""" + + fetch_values: Tuple[str, ...] # columns to fetch (declared + keying) + scalar_fetch: Tuple[ScalarFetch, ...] + auto_fetch: Tuple[AutoFetch, ...] + # Bound field_map.annotate_queryset: applies SQL renames (F() aliases) + # before .values() runs. + annotate_queryset: Callable[[QuerySet], QuerySet] + # {target: F(source)} aliases this level pushes to SQL — merged across a + # model's plans when forward fetches coalesce into one query. + sql_renames: Dict[str, Any] + expand_rows: LevelExpander + + +# The row workers below look near-duplicate on purpose. Collapsing them into one +# loop would mean testing "does this level key by pk / carry a link / keep the raw +# row?" on every row. Those facts are fixed per level, not per row, so instead each +# variant is a straight-line loop with only the work its level needs, and +# ``_compile_level_expander`` picks the right one once. The keying decision is made +# at plan-compile time, never inside the per-row serialization hot loop. + + +def _expand_leaf(map_row, extra_cols, raw_rows, mc): + """Leaf level: no keying, so only items are produced.""" + items = [map_row(r, mc, extra_cols) for r in raw_rows] + return ExpandResult(items, [], [], {}) + + +def _expand_pk_only(map_row, extra_cols, raw_pk_name, raw_rows, mc): + """Keys by pk only (has fetches, but no reverse link and no raw row needed).""" + items, pks = [], [] + for r in raw_rows: + items.append(map_row(r, mc, extra_cols)) + pks.append(r[raw_pk_name]) + return ExpandResult(items, pks, [], {}) + + +def _expand_with_link(map_row, extra_cols, raw_pk_name, link, raw_rows, mc): + """ + Reverse-fetched child: keys by the child's own pk and pairs it with the + parent's pk. ``link`` is the child's back-reference FK column (the reverse + relation), so ``r[link]`` is the parent pk each child buckets back onto. + """ + items, pks, link_pairs = [], [], [] + for r in raw_rows: + items.append(map_row(r, mc, extra_cols)) + pk = r[raw_pk_name] + pks.append(pk) + link_pairs.append((pk, r[link])) + return ExpandResult(items, pks, link_pairs, {}) + + +def _expand_with_raw(map_row, extra_cols, raw_pk_name, raw_rows, mc): + """ + Keys by pk and retains the raw row per pk (a forward fetch reads its FK + column out of it). + """ + items, pks, raw_by_pk = [], [], {} + for r in raw_rows: + items.append(map_row(r, mc, extra_cols)) + pk = r[raw_pk_name] + pks.append(pk) + raw_by_pk.setdefault(pk, r) + return ExpandResult(items, pks, [], raw_by_pk) + + +def _expand_with_raw_and_link(map_row, extra_cols, raw_pk_name, link, raw_rows, mc): + """ + Reverse-fetched child that also needs its raw row: keys by pk, pairs each + child with its parent pk via ``link`` (its back-reference FK column), and + retains the raw row per pk. + """ + items, pks, link_pairs, raw_by_pk = [], [], [], {} + for r in raw_rows: + items.append(map_row(r, mc, extra_cols)) + pk = r[raw_pk_name] + pks.append(pk) + raw_by_pk.setdefault(pk, r) + link_pairs.append((pk, r[link])) + return ExpandResult(items, pks, link_pairs, raw_by_pk) + + +def _compile_level_expander(field_map, extra_cols, raw_pk_name, link, needs_raw_by_pk): + """ + Pick the worker matching this level's keying needs and bind its per-level + args once, so the hot per-row loop stays branch-free. The matrix: no pk ⇒ + leaf; otherwise pk plus whichever of (raw_by_pk, link) the level requires. + """ + map_row = field_map.map_row + if raw_pk_name is None: + return partial(_expand_leaf, map_row, extra_cols) + if needs_raw_by_pk and link is not None: + return partial( + _expand_with_raw_and_link, map_row, extra_cols, raw_pk_name, link + ) + if needs_raw_by_pk: + return partial(_expand_with_raw, map_row, extra_cols, raw_pk_name) + if link is not None: + return partial(_expand_with_link, map_row, extra_cols, raw_pk_name, link) + return partial(_expand_pk_only, map_row, extra_cols, raw_pk_name) + + +class ValuesEngine: + """ + Turns a Django queryset into plain dicts via ``values()`` instead of DRF's + per-instance serialization. ``serialize()`` is the entry point. + + Built once per viewset by ``from_serializer()`` (derived) or + ``from_explicit()`` (legacy ``values``/``field_map``) and reused across + requests: instances carry no per-request state. + + Each level fetches its joinable columns in one query and maps the raw rows + to output shape; relations that can't be joined are handed to their fetch + descriptors, which batch one follow-up query per relation. Under ``DEBUG`` + the output is validated against the serializer's schema. + """ + + # Fixed instance shape — the two factories fill these in, nothing else is set. + __slots__ = ( + "_serializer_derived", + "_values", + "_field_map", + "_scalar_fetch", + "_auto_fetch", + "_nested_cache", + "_top_model", + "_validation_schema", + "_has_method_fields", + "_plans", + ) + + def __init__(self): + # Full instance shape in one place; the two factories below fill it in. + self._serializer_derived = False + self._values = () + self._field_map = None + self._scalar_fetch = () + self._auto_fetch = () + self._nested_cache = {} + self._top_model = None + self._validation_schema = None + self._has_method_fields = False + self._plans = {} + + @classmethod + def from_serializer(cls, serializer_class, deferred_fields=()): + self = cls() + self._serializer_derived = True + # Instantiate only for introspection (not retained); the result carries + # everything the engine needs, so the serializer is never re-read. + result = derive_values_from_serializer( + serializer_class(), deferred_fields=deferred_fields + ) + self._values = tuple(result.values) + self._field_map = result.field_map + self._scalar_fetch = result.scalar_fetch + self._auto_fetch = result.auto_fetch + self._nested_cache = result.nested_cache + self._top_model = result.model + self._validation_schema = result.validation_schema + self._has_method_fields = self._check_has_method_fields() + self._plans = self._compile_plans() + return self + + @classmethod + def from_explicit(cls, values, field_map): + # TODO(#14302): remove with the legacy explicit values/field_map path. + self = cls() + self._values = tuple(values) + self._field_map = normalize_field_map(field_map or {}) + self._plans = self._compile_plans() + return self + + def _make_plan( + self, values, field_map, scalar_fetch, auto_fetch, model, link, is_fetched_child + ): + """ + Compile one level's plan. ``is_fetched_child`` is True for a level the + engine fetches as a child (it always keys by pk); ``link`` set ⇒ + reverse-fetched child, and is the child's back-reference FK column. + """ + carry_pk = is_fetched_child or bool(scalar_fetch) or bool(auto_fetch) + raw_pk_name = model._meta.pk.name if (carry_pk and model is not None) else None + fetch_values = list(values) + extra_cols = [] + if link is not None and link not in fetch_values: + fetch_values.append(link) + extra_cols.append(link) + if raw_pk_name is not None and raw_pk_name not in fetch_values: + fetch_values.append(raw_pk_name) + extra_cols.append(raw_pk_name) + # A forward FK's source column is fetched only to key its deferred fetch. + # When the serializer doesn't also declare it as a field, it's a keying + # column, not output — mark it so map_row projects it out instead of + # leaking it (the all-passthrough noop path would otherwise keep it). + for entry in auto_fetch: + if ( + isinstance(entry, ForwardAutoFetch) + and entry.link not in field_map + and entry.link not in extra_cols + ): + extra_cols.append(entry.link) + return _LevelPlan( + fetch_values=tuple(fetch_values), + scalar_fetch=scalar_fetch, + auto_fetch=auto_fetch, + annotate_queryset=field_map.annotate_queryset, + sql_renames=field_map.sql_renames(), + expand_rows=_compile_level_expander( + field_map, + tuple(extra_cols), + raw_pk_name, + link, + needs_raw_by_pk=any(f.needs_raw_by_pk for f in auto_fetch), + ), + ) + + def _compile_plans(self): + """ + One plan per level, keyed by serializer path (``''`` = top). A + nested level's reach (``link``, carry-pk) is fixed by the ``AutoFetch`` + that fetches it. A path deferred explicitly via ``deferred_fields`` is + fetched by the developer through ``serialize_queryset``. + """ + plans = { + "": self._make_plan( + self._values, + self._field_map, + self._scalar_fetch, + self._auto_fetch, + self._top_model, + link=None, + is_fetched_child=False, + ) + } + entries = list(self._auto_fetch) + for nested in self._nested_cache.values(): + entries.extend(nested.auto_fetch) + for entry in entries: + nested = self._nested_cache[entry.child_path] + plans[entry.child_path] = self._make_plan( + nested.values, + nested.field_map, + nested.scalar_fetch, + nested.auto_fetch, + entry.target_model, + entry.child_fetch_link, + is_fetched_child=True, + ) + # Explicit deferred_fields paths, not auto-fetched, get a plan here. + for path, nested in self._nested_cache.items(): + if path not in plans: + plans[path] = self._make_plan( + nested.values, + nested.field_map, + nested.scalar_fetch, + nested.auto_fetch, + nested.model, + link=None, + is_fetched_child=False, + ) + return plans + + def _check_has_method_fields(self): + """ + True if any level (top or nested-cache) has a method field, so + ``serialize()`` only builds the context carrier when one is needed. + """ + field_maps = [self._field_map] + [ + nested.field_map for nested in self._nested_cache.values() + ] + return any( + isinstance(entry, MethodFieldEntry) + for fm in field_maps + for entry in fm.values() + ) + + @property + def values(self): + return self._values + + @property + def db_column_map(self): + return self._field_map.db_column_map() + + @property + def plain_renames(self): + return self._field_map.plain_renames() + + def validate_output(self, items): + """No-op unless serializer-derived; validates output in DEBUG mode.""" + if not items or not self._serializer_derived: + return + self._validate_items_against_schema(items, self._validation_schema) + + def serialize(self, queryset, *, context=None, group_by=None, path=""): + """ + Return ``queryset``'s rows as output dicts, or as + ``{group_by value: [dicts]}`` when ``group_by`` names an output field. + + ``context`` is the serializer context method fields are given; ``path`` + selects which level's plan to serialize under (``""`` = top). + """ + # Per-call, never stored on the shared engine — isolates concurrent requests. + method_context = MethodContext(context) if self._has_method_fields else None + items, _pks, _link_pairs, deferred_forward_refs = self.expand_level( + queryset, path, method_context + ) + self._resolve_deferred_forward_refs(deferred_forward_refs, method_context) + if group_by is not None: + return self._group_items(items, group_by) + return items + + def _group_items( + self, items: List[Dict[str, Any]], group_by: str + ) -> Dict[Any, List[Dict[str, Any]]]: + """ + Group items into ``{group_by value: [items]}``. Strict lookup: a + ``group_by`` naming no output field is a caller error, not a silent + bucket under ``None``. + """ + result: Dict[Any, List[Dict[str, Any]]] = defaultdict(list) + for item in items: + result[item[group_by]].append(item) + return dict(result) + + def expand_level(self, queryset, serializer_path, method_context): + """ + Serialize one level and apply its deferred fetches. + + Returns ``(items, pks, link_pairs, deferred_forward_refs)``: ``pks`` + parallels ``items``; ``link_pairs`` only for a reverse-fetched child; + ``deferred_forward_refs`` gathered here and below. + """ + plan = self._plans.get(serializer_path) + if plan is None: + raise ValueError( + "No serialization plan for path {!r} — serialize_queryset needs " + "a serializer-derived viewset with that nested path.".format( + serializer_path + ) + ) + raw_rows = plan.annotate_queryset(queryset).values(*plan.fetch_values) + return self._expand_plan_rows(raw_rows, plan, method_context) + + def _expand_plan_rows(self, raw_rows, plan, method_context): + """ + Serialize already-materialized ``raw_rows`` under ``plan`` and apply + its deferred fetches. The forward path uses this to serialize each shape + from a model's shared fetch; ``expand_level`` uses it after its own + ``.values()`` call. Same return shape as ``expand_level``. + """ + items, pks, link_pairs, raw_by_pk = plan.expand_rows(raw_rows, method_context) + + # The engine is the sole writer of nested fields: fetches only compute + # values. + deferred_forward_refs = [] + for entry in (*plan.scalar_fetch, *plan.auto_fetch): + values, nested_forward_refs, forward_targets = entry.resolve( + self, pks, raw_by_pk, method_context + ) + for item, value in zip(items, values): + item[entry.field_name] = value + deferred_forward_refs += nested_forward_refs + # Only a forward auto-fetch returns targets (scalar/reverse fetches + # return None here), so entry.child_path is always present below. + if forward_targets is not None: + for item, target_pk in zip(items, forward_targets): + if target_pk is not None: + deferred_forward_refs.append( + DeferredForwardRef( + entry.target_model, + entry.child_path, + target_pk, + item, + entry.field_name, + ) + ) + + return items, pks, link_pairs, deferred_forward_refs + + def _resolve_deferred_forward_refs( + self, deferred_forward_refs, method_context: Optional[MethodContext] + ): + """ + Resolve recorded forward-FK references with a worklist fixpoint, + separating the two concerns the old shape key conflated: + + - Fetch is keyed by model: every reference to a model — whatever the + serializer — is pulled in one query over the union of pks and columns. + Paths whose SQL-rename annotations conflict can't share a ``.values()`` + and fetch separately (see ``_coalesce_fetches``). + - Serialize is keyed by ``child_path``: each path owns one plan, so a + plan is never applied to another shape's rows. Mismerge is impossible + by construction — no signature needed. + + Expanding a target collects its own forward refs into the next round, so + deep chains converge level by level. Items embed live (no copy), so a + later round's mutation of an embedded target lands on the same object. + """ + # Terminates: one nesting level per round; serializers acyclic. + while deferred_forward_refs: + next_refs: List[DeferredForwardRef] = [] + by_path: Dict[str, List[DeferredForwardRef]] = defaultdict(list) + for ref in deferred_forward_refs: + by_path[ref.child_path].append(ref) + + for paths in self._coalesce_fetches(by_path): + next_refs += self._fetch_and_serialize(paths, by_path, method_context) + + deferred_forward_refs = next_refs + + def _coalesce_fetches(self, by_path): + """ + Group the pending child_paths into one fetch each: same target model, + and SQL-rename annotations that don't collide on a target name. A + collision (same alias from a different source) can't share one + ``.values()`` call, so those paths fetch on their own. + """ + by_model: Dict[Any, List[str]] = defaultdict(list) + for path, refs in by_path.items(): + by_model[refs[0].target_model].append(path) + + groups: List[List[str]] = [] + for paths in by_model.values(): + buckets: List[Tuple[Dict[str, Any], List[str]]] = [] + for path in paths: + renames = self._plans[path].sql_renames + for merged, members in buckets: + if all( + merged[k].name == v.name + for k, v in renames.items() + if k in merged + ): + merged.update(renames) + members.append(path) + break + else: + buckets.append((dict(renames), [path])) + groups.extend(members for _merged, members in buckets) + return groups + + def _fetch_and_serialize(self, paths, by_path, method_context): + """ + One DB fetch for ``paths`` (same model, mergeable annotations), then + serialize each path's rows under its own plan and assign to its refs. + + Parents sharing a forward target embed the *same* item dict by identity + (one serialized row per target pk); harmless for the read-only JSON path, + but a ``consolidate()`` override that mutates one in place writes it to + every parent sharing that target. + """ + model = by_path[paths[0]][0].target_model + pk_name = model._meta.pk.name + union_cols = sorted({c for p in paths for c in self._plans[p].fetch_values}) + merged_renames: Dict[str, Any] = {} + for p in paths: + merged_renames.update(self._plans[p].sql_renames) + all_pks = list({ref.target_pk for p in paths for ref in by_path[p]}) + + # Forward FK / OneToOne is always to-one; load through the base manager + # (as Django's ForwardManyToOneDescriptor does) so a target hidden by its + # default manager still resolves rather than becoming a spurious null. + qs = filter_in(model._base_manager, "pk", all_pks) + if merged_renames: + qs = qs.annotate(**merged_renames) + rows_by_pk = {pk_key(row[pk_name]): row for row in qs.values(*union_cols)} + + next_refs: List[DeferredForwardRef] = [] + for path in paths: + plan = self._plans[path] + refs = by_path[path] + keys = {pk_key(ref.target_pk) for ref in refs} + # Project the shared rows to this plan's own columns, so expansion is + # identical to a dedicated .values(*plan.fetch_values) fetch. + rows = [ + {c: rows_by_pk[k][c] for c in plan.fetch_values} + for k in keys + if k in rows_by_pk + ] + items, item_pks, _lp, nested_forward_refs = self._expand_plan_rows( + rows, plan, method_context + ) + next_refs += nested_forward_refs + items_by_pk = {pk_key(pk): item for pk, item in zip(item_pks, items)} + for ref in refs: + ref.item[ref.field_name] = items_by_pk.get(pk_key(ref.target_pk)) + return next_refs + + @staticmethod + def _validate_items_against_schema( + items: List[Dict[str, Any]], + schema, + ) -> None: + """ + Validate items against a cached schema; recurses into nested schemas. + + Checks only the first item — values() rows share uniform keys, so one + catches schema drift. + """ + if not items: + return + + expected_fields, present_fields, nested_schemas = schema + item = items[0] + item_keys = set(item.keys()) + + missing = present_fields - item_keys + if missing: + raise OutputValidationError( + "Missing fields in output: {}. Expected: {}, Got: {}".format( + missing, expected_fields, item_keys + ) + ) + + extra = item_keys - expected_fields + if extra: + raise OutputValidationError( + "Unexpected fields in output: {}. Expected: {}, Got: {}".format( + extra, expected_fields, item_keys + ) + ) + + for field_name, nested_schema in nested_schemas.items(): + nested_value = item.get(field_name) + if nested_value is None: + continue + if isinstance(nested_value, dict): + nested_value = [nested_value] + if isinstance(nested_value, list): + ValuesEngine._validate_items_against_schema(nested_value, nested_schema) diff --git a/kolibri/core/utils/values_viewset/fetch.py b/kolibri/core/utils/values_viewset/fetch.py new file mode 100644 index 00000000000..b8e93f3c069 --- /dev/null +++ b/kolibri/core/utils/values_viewset/fetch.py @@ -0,0 +1,347 @@ +""" +Deferred-fetch descriptors for ``ValuesEngine``. + +``resolve`` is pure: it computes the per-item values to assign (aligned to +``pks``) and never touches the items — the engine owns that write. Reverse +fetches recurse through ``engine.expand_level``. Depends on neither serializer +introspection nor engine internals. +""" + +import logging +from abc import ABC +from abc import abstractmethod +from collections import defaultdict + +from kolibri.core.mixins import InlineIn + +logger = logging.getLogger(__name__) + +# Matches FilterByUUIDQuerysetMixin's threshold for the same mechanism. +_INLINE_WARN_SIZE = 10000 + + +def filter_in(manager, path, pks): + """ + Filter ``manager`` by ``path`` in ``pks``, inlining the pks as SQL literals. + + A deferred fetch's parent-pk list is unbounded where the join it replaced + bound nothing at all, so a parameterized ``IN`` raises ``too many SQL + variables`` on an unpaginated list. See ``InlineIn``. + + Safe here because these pks came straight out of the parent level's query, + never from a request — ``FilterByUUIDQuerysetMixin`` validates its ids for + exactly that reason. A pk carrying a quote falls back to binding. + + Trades the parameter cap for a statement-length one, hence the warning. + """ + if len(pks) > _INLINE_WARN_SIZE: + logger.warning( + "Deferred fetch on %s inlines %d parent pks into one statement; " + "paginate or batch the parent queryset to stay clear of " + "SQLITE_MAX_SQL_LENGTH.", + manager.model.__name__, + len(pks), + ) + lookup = InlineIn.lookup_name if all("'" not in str(pk) for pk in pks) else "in" + return manager.filter(**{"{}__{}".format(path, lookup): pks}) + + +def pk_key(pk): + """ + Normalize a pk to its cross-level bucket/join key. + + A parent's pk and a child's link-back value come from different ``values()`` + columns and can differ in type (UUID instance vs hex string), so every join + keys on ``str(pk)``. Route all keying through here so the sites can't desync. + """ + return str(pk) + + +def pk_keyer(): + """ + A ``pk_key`` that normalizes once per distinct pk instead of once per row. + + A level has far fewer distinct parents than child rows and ``str(UUID)`` + formats a 36-char string, so the per-row call dominates bucketing. Use this + wherever a loop keys many rows; use ``pk_key`` for one-off lookups. + """ + keys = {} + + def key(pk): + normalized = keys.get(pk) + if normalized is None: + normalized = keys[pk] = str(pk) + return normalized + + return key + + +class LevelFetch(ABC): + """ + A relation fetched in a follow-up query and assembled onto a level's + items under ``field_name``. + """ + + __slots__ = () + + # True when resolve() reads the parent raw rows; the engine only + # materializes raw_by_pk for a level that needs it. + needs_raw_by_pk = False + + @abstractmethod + def resolve(self, engine, pks, raw_by_pk, method_context): + """ + Compute this relation's contribution, without mutating the items. + Returns ``(values, nested_forward_refs, forward_targets)``: + + - ``values``: the value to assign to ``field_name``, aligned to ``pks``. + - ``nested_forward_refs``: forward-FK references that a reverse child + recursion surfaced; empty for a fetch with no such recursion. + - ``forward_targets``: ``None`` for a fetch that resolves in place + (scalar, reverse). A forward fetch instead returns a per-parent list + of target pks — ``None`` per entry where the FK is null — which the + engine batch-resolves per model in ``_resolve_deferred_forward_refs``. + """ + + +def _dedupe_scalar_rows(rows, key_col, value_col): + """ + Bucket ``(key_col, value_col)`` rows into ``{pk_key: [distinct values]}``, + dropping nulls. Shared by the two scalar-fetch shapes so their dedup can't + drift. + """ + buckets = defaultdict(list) + seen = defaultdict(set) + to_key = pk_keyer() + for row in rows: + key = to_key(row[key_col]) + value = row[value_col] + if value is not None and value not in seen[key]: + seen[key].add(value) + buckets[key].append(value) + return buckets + + +class ScalarFetch(LevelFetch): + """ + A scalar output field whose source reaches *through* a to-many relation to + a plain column — e.g. ``books.title`` on an author, where one author has many + books. Joining would multiply the parent's rows by its related count, so the + values are instead pulled in a follow-up query and assembled as a + deduplicated list per parent. + + Not an ``AutoFetch``: + + - it produces a flat list of scalars, not a nested serialized object; + - it has no child level, so no reach (``child_fetch_link``) and never raises + forward refs. + + Two shapes fetch differently but bucket identically (see + ``_dedupe_scalar_rows``): ``ScalarFetchEntry`` when the to-many is the first + source segment (reached by its direct reverse accessor); + ``ParentPathScalarFetch`` when a to-one prefix precedes it (reached by the + reversed relation path). Both anchor on the to-many's own model, so its + default manager filters the values. + """ + + __slots__ = () + + +class ScalarFetchEntry(ScalarFetch): + """ + To-many as the first source segment (``books.title``): the parent reaches + the values over a reverse relation, so query the child model directly and + bucket each row back onto the parent by ``link`` — the child's FK column + pointing at the parent. + """ + + __slots__ = ("field_name", "target_model", "link", "leaf_source") + + def __init__(self, field_name, target_model, link, leaf_source): + self.field_name = field_name + self.target_model = target_model # related model holding the values + self.link = link # accessor on the target back to the parent's pk + self.leaf_source = leaf_source # the value column on the target + + def resolve(self, engine, pks, raw_by_pk, method_context): + rows = filter_in(self.target_model._default_manager, self.link, pks).values( + self.link, self.leaf_source + ) + buckets = _dedupe_scalar_rows(rows, self.link, self.leaf_source) + return [buckets.get(pk_key(pk), []) for pk in pks], [], None + + +class ParentPathScalarFetch(ScalarFetch): + """ + To-many crossed *below* the first segment (``publisher.books.title``). + + Anchor on the model owning the first crossed to-many (``Book``) so its + default manager filters the values — a soft-deleted child's value drops out, + matching Django's related manager — and reach parents through the reversed + relation path (``publisher__authors``). To-one hops after the anchor stay + plain joins, i.e. base-manager semantics, matching a forward FK. + """ + + __slots__ = ("field_name", "target_model", "reverse_path", "leaf_source") + + def __init__(self, field_name, target_model, reverse_path, leaf_source): + self.field_name = field_name + self.target_model = target_model # model owning the first crossed to-many + self.reverse_path = reverse_path # ORM path from target back to parent pk + self.leaf_source = leaf_source # value column on / reachable from target + + def resolve(self, engine, pks, raw_by_pk, method_context): + rows = filter_in( + self.target_model._default_manager, self.reverse_path, pks + ).values(self.reverse_path, self.leaf_source) + buckets = _dedupe_scalar_rows(rows, self.reverse_path, self.leaf_source) + return [buckets.get(pk_key(pk), []) for pk in pks], [], None + + +class AutoFetch(LevelFetch): + """ + A deferred nested-serializer relation: a child level the engine fetches + and assembles onto its parents. + """ + + __slots__ = () + + @property + @abstractmethod + def child_fetch_link(self): + """ + The accessor the child level is filtered by, or ``None`` when the + child is fetched by its own pk (a forward target). + """ + + @abstractmethod + def with_child_path(self, child_path): + """A copy of this entry whose child level lives at ``child_path``.""" + + +class ReverseAutoFetch(AutoFetch): + """ + A reverse FK / M2M: one batched child query filtered by the parents' pks, + bucketed back onto each parent (a list when ``is_many``, else one object). + """ + + __slots__ = ( + "field_name", + "target_model", + "link", + "is_many", + "child_path", + "shares_parents", + ) + + def __init__( + self, field_name, target_model, link, is_many, child_path, shares_parents + ): + self.field_name = field_name + self.target_model = target_model + self.link = link # accessor on the child back to the parent + self.is_many = is_many + self.child_path = child_path + # True only for M2M: a child can appear under several parents, so link + # pairs need deduping. A reverse FK/1:1 child has exactly one parent. + self.shares_parents = shares_parents + + @property + def child_fetch_link(self): + return self.link + + def with_child_path(self, child_path): + return ReverseAutoFetch( + self.field_name, + self.target_model, + self.link, + self.is_many, + child_path, + self.shares_parents, + ) + + def resolve(self, engine, pks, raw_by_pk, method_context): + # Mirror Django's relation descriptors: a reverse OneToOne (to-one) loads + # through the base manager, so a target hidden by its default manager + # isn't nulled out; reverse FK / M2M (to-many) use the default manager, + # so a filtered target model drops its hidden children. + manager = ( + self.target_model._default_manager + if self.is_many + else self.target_model._base_manager + ) + child_qs = filter_in(manager, self.link, pks) + child_items, child_pks, link_pairs, nested_forward_refs = engine.expand_level( + child_qs, self.child_path, method_context + ) + buckets = defaultdict(list) + to_key = pk_keyer() + if self.shares_parents: + # An M2M child shared across parents has a link pair per parent; dedup + # so a grandchild fan-out doesn't double-count under one parent. The + # pairs no longer align with child_items, so index the items by pk. + items_by_pk = dict(zip(child_pks, child_items)) + seen_pairs = set() + for child_pk, parent_pk in link_pairs: + if (child_pk, parent_pk) in seen_pairs: + continue + seen_pairs.add((child_pk, parent_pk)) + buckets[to_key(parent_pk)].append(items_by_pk[child_pk]) + else: + # One pair per child row, built in child_items order — pair them + # positionally rather than round-tripping through a pk-keyed dict. + for item, (_child_pk, parent_pk) in zip(child_items, link_pairs): + buckets[to_key(parent_pk)].append(item) + + values = [] + for pk in pks: + bucket = buckets.get(to_key(pk), []) + values.append(bucket if self.is_many else (bucket[0] if bucket else None)) + return values, nested_forward_refs, None + + +class ForwardAutoFetch(AutoFetch): + """ + A forward FK / OneToOne: the parent row carries the target pk. Surfaces a + target pk per parent so same-model targets across the tree merge into one + query, resolved later by ``engine._resolve_deferred_forward_refs``. + """ + + __slots__ = ("field_name", "target_model", "link", "child_path") + + # resolve() reads the parent's FK column out of the raw row. + needs_raw_by_pk = True + + def __init__(self, field_name, target_model, link, child_path): + self.field_name = field_name + self.target_model = target_model + self.link = link # the FK column on the parent holding the target pk + self.child_path = child_path + + @property + def child_fetch_link(self): + return None + + def with_child_path(self, child_path): + return ForwardAutoFetch( + self.field_name, self.target_model, self.link, child_path + ) + + def resolve(self, engine, pks, raw_by_pk, method_context): + # Seed every field to None; the engine raises a need for each non-null + # target and fills it in once the batched forward query runs. Strict: + # _make_plan guarantees the link is fetched, so a missing key is a plan + # bug, not a null FK. + targets = [raw_by_pk[pk][self.link] for pk in pks] + return [None] * len(pks), [], targets + + +def make_auto_fetch( + is_forward, field_name, target_model, link, is_many, child_path, shares_parents +): + """Build the entry class matching the relation direction.""" + if is_forward: + return ForwardAutoFetch(field_name, target_model, link, child_path) + return ReverseAutoFetch( + field_name, target_model, link, is_many, child_path, shares_parents + ) diff --git a/kolibri/core/utils/values_viewset/field_map.py b/kolibri/core/utils/values_viewset/field_map.py new file mode 100644 index 00000000000..d3bc2ab7e99 --- /dev/null +++ b/kolibri/core/utils/values_viewset/field_map.py @@ -0,0 +1,364 @@ +""" +Field-map data model: how one raw ``values()`` row becomes an output dict. + +A ``FieldMapEntry`` produces one output field's value. ``_FieldMap`` (serializer- +derived) and ``_LegacyFieldMap`` (explicit ``values``/``field_map``) own the +row → output mapping. Runtime module — imported per request, no compile-time logic. +""" + +from abc import ABC +from abc import ABCMeta +from abc import abstractmethod +from collections import Counter +from typing import Any +from typing import Callable +from typing import Dict +from typing import List +from typing import Optional +from typing import Set +from typing import Tuple +from typing import Type + +from django.db.models import F +from django.db.models import Model + +from kolibri.core.utils.values_viewset.method_fields import _SourcesProxy +from kolibri.core.utils.values_viewset.method_fields import MethodContext + +# A row produced by ``queryset.values()`` — dict of flat path → value. +Row = Dict[str, Any] + + +class FieldMapEntry(ABC): + """ + Produces one output field's value from a raw ``values()`` row. + + - ``source``: backing DB column, or ``None`` for computed (callable/method) + fields — source-introspecting consumers (ordering filter, serializer + generation) skip ``source is None`` entries. + - ``to_repr``: the raw-value transform, ``None`` for a passthrough. + """ + + __slots__ = () + + source: Optional[str] = None + to_repr: Optional[Callable] = None + + @property + def read_columns(self) -> Tuple[str, ...]: + """ + Raw row columns ``extract`` reads. Rename promotion drops a column from + ``values()`` only when no other entry reads it, so an entry reading more + than its ``source`` must say so here. + """ + return () if self.source is None else (self.source,) + + @abstractmethod + def extract(self, row: Row, method_context: Optional[MethodContext] = None) -> Any: + """Produce the output value, reading ``row`` without mutating it.""" + + def apply( + self, key: str, row: Row, method_context: Optional[MethodContext] = None + ) -> None: + """ + Write ``extract``'s value under ``key``. Legacy maps drop consumed + source columns after mapping (see ``_LegacyFieldMap``); others don't + mutate the row. + """ + row[key] = self.extract(row, method_context) + + +class SourceFieldEntry(FieldMapEntry): + """ + Single-source rename, optionally transformed. + + ``to_repr=None`` is a plain rename. With ``to_repr`` set and a ``None`` raw + value, ``default`` is substituted — mirrors DRF's get_attribute fallback for + missing relations. + """ + + __slots__ = ("source", "to_repr", "default") + + def __init__( + self, + source: str, + to_repr: Optional[Callable] = None, + default: Any = None, + ): + self.source = source + self.to_repr = to_repr + self.default = default + + def _represent(self, raw: Any) -> Any: + if self.to_repr is None: + return raw + if raw is None: + return self.default + return self.to_repr(raw) + + def extract(self, row: Row, method_context: Optional[MethodContext] = None) -> Any: + # Strict lookup — a missing key means a misconfigured map. _represent + # handles None (LEFT JOIN miss). + return self._represent(row[self.source]) + + +class _LegacyCallableFieldEntry(FieldMapEntry): + """ + Legacy ``field_map`` callable reading only the row. + + Invoked 1-arg to avoid a per-row wrapper frame on the hot explicit-values + path. ``source``/``to_repr`` stay ``None``, so source-introspecting consumers + skip it. + + TODO(#14302): remove with the legacy explicit values/field_map path. + """ + + __slots__ = ("func",) + + def __init__(self, func: Callable[[Row], Any]): + self.func = func + + def extract(self, row: Row, method_context: Optional[MethodContext] = None) -> Any: + return self.func(row) + + +class MethodFieldEntry(FieldMapEntry): + """ + Invokes a ``ValuesMethodField``'s unbound ``get_*`` with a per-call + ``MethodContext`` as ``self`` and a ``_SourcesProxy`` over its sources as + ``obj``. + + Threading context per call (not binding a shared serializer) keeps the engine + free of per-request state, so one engine serializes concurrent requests safely. + """ + + __slots__ = ("method_func", "sources") + + def __init__(self, method_func: Callable, sources: Tuple[str, ...]): + self.method_func = method_func + self.sources = sources + + @property + def read_columns(self) -> Tuple[str, ...]: + return self.sources + + def extract(self, row: Row, method_context: Optional[MethodContext] = None) -> Any: + return self.method_func(method_context, _SourcesProxy(row, self.sources)) + + +FieldMap = Dict[str, FieldMapEntry] + + +class _BaseFieldMap(dict, metaclass=ABCMeta): + """Dict of output field name → entry; owns row mapping and source introspection.""" + + # Set by finalize(): True when every entry is a trivial passthrough. + is_noop = False + + def finalize(self) -> None: + """Precompute ``is_noop`` after all entries are added.""" + self.is_noop = all( + isinstance(entry, SourceFieldEntry) + and entry.source == key + and entry.to_repr is None + for key, entry in self.items() + ) + + @abstractmethod + def map_row( + self, + row: Row, + method_context: Optional[MethodContext] = None, + extra_cols: Tuple[str, ...] = (), + ) -> Row: + """ + Produce the output row for a raw ``values()`` row, dropping the + engine's keying columns (``extra_cols``) that aren't declared output. + """ + + def plain_renames(self) -> Dict[str, str]: + """ + ``{source: target}`` for plain renames (source set, no ``to_repr``), + exposing raw values under the declared name. + """ + return { + entry.source: key + for key, entry in self.items() + if entry.source is not None and entry.to_repr is None + } + + def db_column_map(self) -> Dict[str, str]: + """ + ``{api_name: db_column}`` where they differ (passthroughs excluded). + + Ordering filters use it to map API field names to DB columns. + """ + return { + key: entry.source + for key, entry in self.items() + if entry.source is not None and entry.source != key + } + + def annotate_queryset(self, queryset: Any) -> Any: + """No-op default: return queryset unchanged.""" + return queryset + + def sql_renames(self) -> Dict[str, Any]: + """``{target: F(source)}`` aliases pushed to SQL; none by default.""" + return {} + + +class _FieldMap(_BaseFieldMap): + """ + Serializer-introspected field map covering every declared output field. + + ``map_row`` builds a fresh dict without mutating the input row, so method-field + sources pulled in for invocation never leak into output (they aren't keys). + ``_sql_renames`` holds F() aliases for plain renames, applied to SQL by + ``annotate_queryset``. + """ + + def __init__(self, *args: Any, **kwargs: Any) -> None: + super().__init__(*args, **kwargs) + self._sql_renames: Dict[str, Any] = {} # {target_key: F(source)} + + def map_row( + self, + row: Row, + method_context: Optional[MethodContext] = None, + extra_cols: Tuple[str, ...] = (), + ) -> Row: + if self.is_noop: + # Project to declared keys when the engine added keying columns; else + # the raw row is already the output. + return row if not extra_cols else {key: row[key] for key in self} + return {key: entry.extract(row, method_context) for key, entry in self.items()} + + def db_column_map(self) -> Dict[str, str]: + """ + ``{api_name: db_column}`` resolving SQL-rename aliases back to source. + + - A promoted rename's entry is now a passthrough (source == key), so its + original column comes from ``_sql_renames``. + - Other renames and passthroughs fall back to the base. + """ + result = {} + for key, entry in self.items(): + if key in self._sql_renames: + result[key] = self._sql_renames[key].name + elif entry.source is not None and entry.source != key: + result[key] = entry.source + return result + + def annotate_queryset(self, queryset: Any) -> Any: + """Apply pre-computed SQL-level F() aliases to the queryset.""" + if self._sql_renames: + return queryset.annotate(**self._sql_renames) + return queryset + + def sql_renames(self) -> Dict[str, Any]: + return self._sql_renames + + def promote_renames_to_sql_aliases( + self, values: List[str], model: Type[Model] + ) -> List[str]: + """ + Precompute SQL-level F() aliases for plain source→target renames. + + Performance optimization: doing the rename in SQL (an ``F()`` alias) + rather than in Python lets the entry become a passthrough. When every + entry is a passthrough the field map is a no-op and ``map_row`` returns + the raw ``values()`` row as-is, skipping the per-row dict rebuild. + + Promotes a rename when: + - source and target differ (not a passthrough) + - source is read by exactly one entry (safe to drop from values()) + - target doesn't shadow a model field (Django raises ValueError otherwise) + + Mutates self in place (promoted entries → passthroughs, populates + ``_sql_renames``) and returns ``values`` with promoted sources renamed to + targets. + """ + model_field_names: Set[str] = {f.name for f in model._meta.get_fields()} # type: ignore[attr-defined] + source_refcount = Counter( + column for entry in self.values() for column in entry.read_columns + ) + + for target_key, map_entry in list(self.items()): + if ( + isinstance(map_entry, SourceFieldEntry) + and map_entry.source is not None + and map_entry.to_repr is None + and map_entry.source != target_key + and source_refcount[map_entry.source] == 1 + and target_key not in model_field_names + ): + self._sql_renames[target_key] = F(map_entry.source) + self[target_key] = SourceFieldEntry(target_key) + + if self._sql_renames: + promoted_sources = { + f_expr.name: target for target, f_expr in self._sql_renames.items() + } + return [promoted_sources.get(v, v) for v in values] + return values + + +class _LegacyFieldMap(_BaseFieldMap): + """ + Field map normalized from a legacy explicit ``values``/``field_map`` viewset. + + ``map_row`` mutates the row in place — writing each target, then dropping + rename-consumed source columns — so unclaimed ``values()`` keys pass through. + Required for back-compat with viewsets relying on that passthrough. + + TODO(#14302): remove with the legacy explicit values/field_map path. + """ + + # Set by finalize(): rename sources to drop after mapping, minus output keys. + _drop_sources: Tuple[str, ...] = () + + def finalize(self) -> None: + super().finalize() + keys = set(self.keys()) + self._drop_sources = tuple( + { + entry.source + for key, entry in self.items() + if entry.source is not None and entry.source != key + } + - keys + ) + + def map_row( + self, + row: Row, + method_context: Optional[MethodContext] = None, + extra_cols: Tuple[str, ...] = (), + ) -> Row: + # Legacy maps never receive engine keying columns. + for key, entry in self.items(): + entry.apply(key, row, method_context) + for source in self._drop_sources: + row.pop(source, None) + return row + + +def normalize_field_map(field_map: Dict[str, Any]) -> _LegacyFieldMap: + """ + Normalize a user-written legacy field_map to canonical entry objects. + + Str shorthand → ``SourceFieldEntry``; bare callable → ``_LegacyCallableFieldEntry`` + (invoked with ``row`` only). Returns a fresh ``_LegacyFieldMap``; input untouched. + + TODO(#14302): remove with the legacy explicit values/field_map path. + """ + + def wrap(value: Any) -> FieldMapEntry: + if isinstance(value, str): + return SourceFieldEntry(value) + return _LegacyCallableFieldEntry(value) + + result = _LegacyFieldMap((key, wrap(value)) for key, value in field_map.items()) + result.finalize() + return result diff --git a/kolibri/core/utils/values_viewset/introspect.py b/kolibri/core/utils/values_viewset/introspect.py new file mode 100644 index 00000000000..fea99a56035 --- /dev/null +++ b/kolibri/core/utils/values_viewset/introspect.py @@ -0,0 +1,681 @@ +""" +Compile-time introspection: read a DRF serializer into a fetch plan. + +``derive_values_from_serializer`` is the entry point — runs once per viewset +class at engine construction, never per request. +""" + +import logging +from typing import cast +from typing import Dict +from typing import Iterator +from typing import List +from typing import NamedTuple +from typing import Optional +from typing import Tuple +from typing import Type +from typing import Union + +from django.core.exceptions import FieldDoesNotExist +from django.db.models import Field +from django.db.models import Model +from django.db.models.fields.related import ForeignObjectRel +from rest_framework import serializers as drf_serializers +from rest_framework.fields import empty +from rest_framework.fields import Field as DrfField +from rest_framework.serializers import ModelSerializer +from rest_framework.serializers import Serializer +from rest_framework.serializers import SerializerMethodField +from rest_framework.utils.field_mapping import ClassLookupDict + +from kolibri.core.utils.values_viewset.fetch import AutoFetch +from kolibri.core.utils.values_viewset.fetch import make_auto_fetch +from kolibri.core.utils.values_viewset.fetch import ParentPathScalarFetch +from kolibri.core.utils.values_viewset.fetch import ScalarFetch +from kolibri.core.utils.values_viewset.fetch import ScalarFetchEntry +from kolibri.core.utils.values_viewset.field_map import _FieldMap +from kolibri.core.utils.values_viewset.field_map import FieldMapEntry +from kolibri.core.utils.values_viewset.field_map import MethodFieldEntry +from kolibri.core.utils.values_viewset.field_map import SourceFieldEntry +from kolibri.core.utils.values_viewset.method_fields import ValuesMethodField + +logger = logging.getLogger(__name__) + + +class NestedCacheEntry(NamedTuple): + """ + One nested serializer's per-level spec, keyed by dotted path in + ``NestedCache``. + + ``model`` is this nested level's ``Meta.model``. + """ + + values: List[str] + field_map: _FieldMap + scalar_fetch: Tuple[ScalarFetch, ...] + auto_fetch: Tuple[AutoFetch, ...] + model: Optional[Type[Model]] + + +NestedCache = Dict[str, NestedCacheEntry] + + +class IntrospectionResult(NamedTuple): + """ + Everything introspection learns from a serializer; the engine consumes + this alone and never re-reads the serializer. + + ``model`` is the top serializer's ``Meta.model``. ``validation_schema`` is + the DEBUG output-shape schema, ``None`` for nested levels. + """ + + values: List[str] + field_map: _FieldMap + scalar_fetch: Tuple[ScalarFetch, ...] + auto_fetch: Tuple[AutoFetch, ...] + nested_cache: NestedCache + model: Optional[Type[Model]] = None + validation_schema: Optional[Tuple] = None + + +def _get_source_path(field: DrfField, field_name: str, prefix: str) -> Optional[str]: + """ + Source path for a field, converted from DRF dot-notation to Django ORM + ``__`` notation. ``None`` for ``source='*'`` or composite (list/tuple) + sources. + """ + source = getattr(field, "source", None) + if source == "*" or isinstance(source, (list, tuple)): + return None + source_path = source if source else field_name + source_path = source_path.replace(".", "__") + return f"{prefix}{source_path}" if prefix else source_path + + +def _is_nested_model_serializer(field: DrfField) -> bool: + """ + True for a nested ``ModelSerializer`` (or ``ListSerializer`` wrapping one). + + Plain ``Serializer`` subclasses (e.g. ``JSONField`` wrappers) are excluded. + They have no ``Meta.model`` to introspect. The regular-field path handles + them, running ``to_representation`` on the raw value. + """ + if isinstance(field, drf_serializers.ListSerializer): + return isinstance(field.child, ModelSerializer) + return isinstance(field, ModelSerializer) + + +def _walk_source_fields( + model: Optional[Type[Model]], source_path: str +) -> Iterator[Union[Field, ForeignObjectRel]]: + """ + Yield each model field along a ``__``-split source path. + + Stops early at the first segment that doesn't resolve, or at a non-final + segment that isn't a relation. The single place that encodes how a DRF + source maps onto the ORM. + """ + if model is None: + return + current_model: Type[Model] = model + for part in source_path.split("__"): + try: + field = current_model._meta.get_field(part) # type: ignore[attr-defined] + except FieldDoesNotExist: + return + yield field + related_model = getattr(field, "related_model", None) + if related_model is None: + return + current_model = related_model + + +def _source_crosses_many_relation( + model: Optional[Type[Model]], source_path: str +) -> bool: + """ + Whether a source path crosses a one-to-many or many-to-many relation. + + ``roles__kind`` (reverse FK) multiplies rows and needs list consolidation. + ``publisher__name`` (to-one) does not, so it stays a joined column. + """ + return any( + getattr(field, "one_to_many", False) or getattr(field, "many_to_many", False) + for field in _walk_source_fields(model, source_path) + ) + + +def _reverse_link(relation: ForeignObjectRel) -> str: + """ + Return the accessor on a to-many target that links back to its parent. + + Shared by scalar-fetch and auto-fetch resolution so the two can't disagree. + """ + if isinstance(relation, ForeignObjectRel): + return relation.field.name + return relation.remote_field.get_accessor_name() + + +def _resolve_scalar_fetch_info( + parent_model: Type[Model], source_path: str, field_name: str +) -> ScalarFetch: + """ + Resolve a scalar field whose source crosses a to-many into a batched fetch. + + When the crossed relation is the first path segment (a reverse FK or M2M + directly on the parent), anchor on the child via its reverse accessor. When + the to-many is crossed below the first segment (``publisher.books.title``), + anchor on the model owning the first crossed to-many so its default manager + applies, and reach parents by the reversed relation path — inverting each + segment up to and including the to-many, in reverse order. + """ + relation_name, _, leaf_source = source_path.partition("__") + relation = parent_model._meta.get_field(relation_name) + if getattr(relation, "one_to_many", False) or getattr( + relation, "many_to_many", False + ): + return ScalarFetchEntry( + field_name, relation.related_model, _reverse_link(relation), leaf_source + ) + fields = list(_walk_source_fields(parent_model, source_path)) + segments = source_path.split("__") + many_index = next( + index + for index, field in enumerate(fields) + if getattr(field, "one_to_many", False) or getattr(field, "many_to_many", False) + ) + target_model = fields[many_index].related_model + reverse_path = "__".join( + _reverse_link(field) for field in reversed(fields[: many_index + 1]) + ) + leaf_suffix = "__".join(segments[many_index + 1 :]) + return ParentPathScalarFetch(field_name, target_model, reverse_path, leaf_suffix) + + +def _get_model_field_for_source( + model: Optional[Type[Model]], source_path: str +) -> Optional[Union[Field, ForeignObjectRel]]: + """ + Final model field for a source path like ``user__profile__name``, or ``None`` + if the path doesn't fully resolve. + + ``get_field`` returns a ``Field`` or a ``ForeignObjectRel`` (reverse + accessor); callers use only ``related_model`` / ``choices``, common to both. + """ + parts = source_path.split("__") + fields = list(_walk_source_fields(model, source_path)) + # A full-length walk resolved every segment; its last field is the target. + # A short walk hit an invalid or over-deep path. + if len(fields) == len(parts): + return fields[-1] + return None + + +def _field_matches_inferred_type( + declared_field: DrfField, + source_path: str, + serializer_class: Type[ModelSerializer], + model: Optional[Type[Model]], +) -> bool: + """ + True when ``declared_field`` is exactly what ``ModelSerializer`` would infer + for the model field, so ``to_representation`` can be skipped as a no-op. + """ + model_field = _get_model_field_for_source(model, source_path) + if model_field is None: + # No model field → likely a queryset annotation. A primitive DRF field + # with no default does an identity type-coercion (the annotation's + # output_field already returns the right Python type), so skip + # to_representation. + if declared_field.default is empty and isinstance( + declared_field, + ( + drf_serializers.IntegerField, + drf_serializers.FloatField, + drf_serializers.BooleanField, + drf_serializers.CharField, + ), + ): + return True + return False + + # Relation field: values() returns the raw FK value (already the PK), but + # PrimaryKeyRelatedField.to_representation expects a model instance. So when + # the declared field is the plain related field, skip it and pass the raw PK + # through. No default check: an FK column always has a value; a serializer + # default is input-only. + if getattr(model_field, "related_model", None) is not None: + return type(declared_field) is serializer_class.serializer_related_field + + # An explicit default needs the transform path to substitute the default for + # None (e.g. a LEFT JOIN miss), which the plain rename can't. + if declared_field.default is not empty: + return False + + # Special case: fields with choices become ChoiceField + if getattr(model_field, "choices", None): + inferred_class = serializer_class.serializer_choice_field + else: + field_mapping = ClassLookupDict(serializer_class.serializer_field_mapping) + try: + inferred_class = field_mapping[model_field] + except KeyError: + return False + + # morango's UUIDField extends models.CharField (not Django's UUIDField), so + # DRF maps it to CharField via MRO on every backend — even PostgreSQL with a + # native UUID column. from_db_value already yields a 32-char hex string, so no + # to_representation is needed regardless of backend. + if ( + inferred_class is drf_serializers.CharField + and isinstance(declared_field, drf_serializers.UUIDField) + and declared_field.uuid_format == "hex" + ): + return True + + # Exact class match only - subclasses may override to_representation + return type(declared_field) is inferred_class + + +def _introspect_nested_field( + field_name: str, + field: DrfField, + parent_model: Optional[Type[Model]], + explicit_deferred: Tuple[str, ...], +) -> Tuple[NestedCache, Optional[AutoFetch], Optional[str]]: + """ + Introspect one nested ``ModelSerializer`` field — always deferred, never + joined. + + Normally auto-defers. If the dev listed it in ``deferred_fields``, they own + the fetch (via ``consolidate()``) and we record only the nested spec. + + Returns ``(nested_cache, auto_fetch, forward_value)``: + + - ``nested_cache``: subtree specs. + - ``auto_fetch``: the ``AutoFetch`` instance (``None`` when dev-deferred). + - ``forward_value``: parent FK column for a forward FK, else ``None``. + """ + is_many = isinstance(field, drf_serializers.ListSerializer) + child = cast(ModelSerializer, field.child if is_many else field) + child_explicit = tuple(_child_paths(explicit_deferred, field_name)) + nested_cache = _introspect_deferred_nested(field_name, child, child_explicit) + + if field_name in explicit_deferred: + return nested_cache, None, None + + if parent_model is None: + raise TypeError( + "Auto-defer requires a Meta.model on the parent serializer to " + "resolve the fetch relation for '{}'.".format(field_name) + ) + ( + lookup_by_source_pk, + target_model, + link, + fetch_is_many, + shares_parents, + ) = _resolve_auto_fetch_info(parent_model, field, field_name) + auto_fetch = make_auto_fetch( + is_forward=not lookup_by_source_pk, + field_name=field_name, + target_model=target_model, + link=link, + is_many=fetch_is_many, + child_path=field_name, + shares_parents=shares_parents, + ) + # Forward FK — the source row carries the target pk, so fetch that column. + forward_value = link if not lookup_by_source_pk else None + return nested_cache, auto_fetch, forward_value + + +def _resolve_auto_fetch_info( + parent_model: Type[Model], field: DrfField, field_name: str +) -> Tuple[bool, Type[Model], str, bool, bool]: + """ + Resolve how to fetch children for an auto-deferred nested field. + + Returns ``(lookup_by_source_pk, target_model, link, is_many, shares_parents)``. + ``lookup_by_source_pk`` is True for reverse FK/OneToOne and M2M (filter + children by the parent's pk), False for forward FK/OneToOne (the source row + carries the target pk). ``shares_parents`` is True only for M2M, where one + child can appear under several parents. Raises ``TypeError`` when the source + isn't a resolvable relation on ``parent_model``. + """ + source = getattr(field, "source", None) or field_name + try: + relation = parent_model._meta.get_field(source) + except FieldDoesNotExist: + raise TypeError( + "Cannot resolve auto-fetch for nested field '{}': source '{}' " + "is not a relation on {}. Add '{}' to deferred_fields explicitly " + "and implement consolidate() to handle the fetch.".format( + field_name, source, parent_model.__name__, field_name + ) + ) + + if getattr(relation, "one_to_many", False): + # reverse FK + return True, relation.related_model, _reverse_link(relation), True, False + if isinstance(relation, ForeignObjectRel) and getattr( + relation, "one_to_one", False + ): + # reverse OneToOne + return True, relation.related_model, _reverse_link(relation), False, False + if getattr(relation, "many_to_many", False): + # reverse or forward M2M + return True, relation.related_model, _reverse_link(relation), True, True + if getattr(relation, "many_to_one", False) or getattr( + relation, "one_to_one", False + ): + # forward FK / forward OneToOne: link is the source column holding the pk. + return False, relation.related_model, relation.name, False, False + + raise TypeError( + "Cannot auto-defer nested field '{}': source '{}' on {} did not " + "resolve to a supported relation. Add '{}' to deferred_fields " + "explicitly and implement consolidate() to handle the fetch.".format( + field_name, source, parent_model.__name__, field_name + ) + ) + + +def _introspect_regular_field( + field_name: str, + field: DrfField, + declared_fields: Dict[str, DrfField], + serializer_class: Type[ModelSerializer], + model: Optional[Type[Model]], +) -> Tuple[Union[str, Tuple[str, ...], None], Optional[FieldMapEntry]]: + """ + Introspect a regular (non-nested) serializer field. + + Returns ``(source_path, entry)``: + + - ``source_path``: value(s) to fetch via ``values()``. ``None`` to skip + the field entirely (e.g. ``source='*'``). For a ``ValuesMethodField``, + a tuple of the declared source paths (caller extends ``values`` with + all of them). + - ``entry``: the field_map entry producing the output value. A + ``MethodFieldEntry`` capturing the serializer's unbound ``get_*`` method + for method fields; a ``SourceFieldEntry`` otherwise, with ``to_repr`` + set when the field transforms its raw value. + """ + if isinstance(field, ValuesMethodField): + source_paths = tuple(source.replace(".", "__") for source in field.sources) + # field.method_name is populated from the instantiated serializer; + # capture the unbound method off the class. + method_func = getattr(serializer_class, field.method_name) + return source_paths, MethodFieldEntry(method_func, source_paths) + + if isinstance(field, SerializerMethodField): + raise TypeError( + "{}.{}: ValuesViewset does not support plain " + "SerializerMethodField. Use ValuesMethodField(sources=(...)) " + "to declare which row columns the method reads, or a typed " + "field with source= for simple traversals.".format( + serializer_class.__name__, field_name + ) + ) + + source_path = _get_source_path(field, field_name, "") + if source_path is None: + return None, None + + if field_name in declared_fields and not _field_matches_inferred_type( + field, source_path, serializer_class, model + ): + default = field.default if field.default is not empty else None + return source_path, SourceFieldEntry( + source_path, field.to_representation, default + ) + # Trivial passthrough (source == name, matching type) still emits an + # entry so the field_map is a complete spec of output fields. + return source_path, SourceFieldEntry(source_path) + + +def _child_paths(paths, head): + """Sub-paths of `paths` nested under `head`, head segment stripped.""" + return [ + p.split("__", 1)[1] for p in paths if "__" in p and p.split("__", 1)[0] == head + ] + + +def _prefix_auto_fetch_paths(prefix, auto_fetch): + """Prepend ``prefix__`` to each auto-fetch entry's child_path.""" + return tuple( + entry.with_child_path("{}__{}".format(prefix, entry.child_path)) + for entry in auto_fetch + ) + + +def _introspect_deferred_nested( + field_name: str, + child: ModelSerializer, + child_explicit: Tuple[str, ...] = (), +) -> NestedCache: + """ + Recurse into a nested serializer and key its per-level specs by path. + + Introspects ``child`` as a top-level serializer — its own nested fields + auto-defer in turn — then prefixes every resulting path with ``field_name``. + Adds nothing to the parent query. + """ + child_result = _introspect_serializer_fields(child, deferred_fields=child_explicit) + entries: NestedCache = { + field_name: NestedCacheEntry( + child_result.values, + child_result.field_map, + child_result.scalar_fetch, + _prefix_auto_fetch_paths(field_name, child_result.auto_fetch), + child_result.model, + ), + } + for sub_path, sub_info in child_result.nested_cache.items(): + entries[f"{field_name}__{sub_path}"] = NestedCacheEntry( + sub_info.values, + sub_info.field_map, + sub_info.scalar_fetch, + _prefix_auto_fetch_paths(field_name, sub_info.auto_fetch), + sub_info.model, + ) + return entries + + +class _LevelBuilder: + """ + Accumulates one serializer level's introspection results, then assembles the + ``IntrospectionResult``. Field handlers tell it what each field contributes; + it owns the value dedupe, SQL-rename promotion, and field-map finalize. + """ + + def __init__(self, model: Optional[Type[Model]]) -> None: + self._model = model + self._values: List[str] = [] + self._field_map = _FieldMap() + self._scalar_fetch: List[ScalarFetch] = [] + self._auto_fetch: List[AutoFetch] = [] + self._nested_cache: NestedCache = {} + + def add_column( + self, field_name: str, source_path: str, entry: FieldMapEntry + ) -> None: + self._values.append(source_path) + self._field_map[field_name] = entry + + def add_method_field( + self, field_name: str, source_paths: Tuple[str, ...], entry: FieldMapEntry + ) -> None: + self._values.extend(source_paths) + self._field_map[field_name] = entry + + def add_scalar_fetch(self, entry: ScalarFetch) -> None: + self._scalar_fetch.append(entry) + + def add_nested( + self, + nested_cache: NestedCache, + auto_fetch: Optional[AutoFetch], + forward_value: Optional[str], + ) -> None: + self._nested_cache.update(nested_cache) + if auto_fetch is not None: + self._auto_fetch.append(auto_fetch) + if forward_value is not None: + self._values.append(forward_value) + + def build(self) -> IntrospectionResult: + # Dedupe: method-field sources can overlap declared sources. Sorted so + # column (and SQL) order is stable — set iteration order varies per run. + values = sorted(set(self._values)) + if self._model is not None: + values = self._field_map.promote_renames_to_sql_aliases(values, self._model) + self._field_map.finalize() + return IntrospectionResult( + values, + self._field_map, + tuple(self._scalar_fetch), + tuple(self._auto_fetch), + self._nested_cache, + self._model, + ) + + +def _classify_regular_field( + field_name: str, + field: DrfField, + declared_fields: Dict[str, DrfField], + serializer_class: Type[ModelSerializer], + model: Optional[Type[Model]], + builder: _LevelBuilder, +) -> None: + """Route one non-nested field to the right ``builder`` slot.""" + source_path, entry = _introspect_regular_field( + field_name, field, declared_fields, serializer_class, model + ) + if source_path is None: + return + # ValuesMethodField yields a tuple of source paths and an invoker entry. + if isinstance(source_path, tuple): + builder.add_method_field(field_name, source_path, entry) + return + # A source crossing a to-many (e.g. books__title) becomes a batched fetch — + # joining would multiply parent rows by the related count. + if _source_crosses_many_relation(model, source_path): + builder.add_scalar_fetch( + _resolve_scalar_fetch_info(model, source_path, field_name) + ) + return + builder.add_column(field_name, source_path, entry) + + +def _introspect_serializer_fields( + serializer: ModelSerializer, + deferred_fields: Tuple[str, ...] = (), +) -> IntrospectionResult: + """ + Introspect a serializer into a fetch plan for one level, in a single pass: + + - every nested ``ModelSerializer`` defers — auto, unless the dev listed it + in ``deferred_fields``. + - every other field becomes a joined column, a method-field invoker, or a + batched scalar fetch. + + :param serializer: the DRF serializer to introspect. + :param deferred_fields: the dev's explicit fetch-separately paths at this + level. Auto-deferred paths are *not* passed in — they are found here. + :return: an :class:`IntrospectionResult`. + """ + serializer_class = type(serializer) + model: Optional[Type[Model]] = getattr( + getattr(serializer_class, "Meta", None), "model", None + ) + declared_fields: Dict[str, DrfField] = getattr(serializer, "_declared_fields", {}) + builder = _LevelBuilder(model) + + serializer_fields: Dict[str, DrfField] = cast( + Dict[str, DrfField], serializer.fields + ) + for field_name, field in serializer_fields.items(): + if getattr(field, "write_only", False): + continue + if _is_nested_model_serializer(field): + builder.add_nested( + *_introspect_nested_field(field_name, field, model, deferred_fields) + ) + continue + if field_name in deferred_fields: + continue + _classify_regular_field( + field_name, field, declared_fields, serializer_class, model, builder + ) + + return builder.build() + + +def _iter_auto_deferred_paths(result: IntrospectionResult): + """ + Every auto-deferred dotted path in ``result`` — one per ``AutoFetch``, + already prefixed to its full path from the top serializer. + """ + for entry in result.auto_fetch: + yield entry.child_path + for nested in result.nested_cache.values(): + for entry in nested.auto_fetch: + yield entry.child_path + + +def build_validation_schema(serializer: Serializer) -> Tuple: + """ + Build the DEBUG output-shape schema from a serializer. + + Returns ``(expected_fields, present_fields, nested_schemas)``. + + - ``expected_fields``: every read field name; a key not in this set is drift. + - ``present_fields``: the fields whose absence is drift. + - ``nested_schemas``: each nested-serializer field mapped to its own schema. + + A model-backed level is assembled field by field, so every declared field must + be present — ``read_only`` included, since DRF makes those ``required=False`` + and they are exactly the deferred fields ``consolidate()`` must fill. + + A plain ``Serializer`` level (a JSON column's schema) runs DRF's + ``to_representation``, which drops a ``required=False`` field the stored data + lacks, so absence there is legitimate. + """ + expected_fields = set() + required_fields = set() + nested_schemas = {} + for field_name, field in serializer.fields.items(): + if getattr(field, "write_only", False): + continue + expected_fields.add(field_name) + if getattr(field, "required", False): + required_fields.add(field_name) + if hasattr(field, "child") and isinstance(field.child, Serializer): + nested_schemas[field_name] = build_validation_schema(field.child) + elif isinstance(field, Serializer): + nested_schemas[field_name] = build_validation_schema(field) + present_fields = ( + expected_fields if isinstance(serializer, ModelSerializer) else required_fields + ) + return (frozenset(expected_fields), frozenset(present_fields), nested_schemas) + + +def derive_values_from_serializer( + serializer: ModelSerializer, + deferred_fields: Tuple[str, ...] = (), +) -> IntrospectionResult: + """ + Introspect ``serializer`` into a fetch plan, auto-deferring every nested + serializer relation. Logs each auto-defer decision at DEBUG. + """ + result = _introspect_serializer_fields(serializer, deferred_fields=deferred_fields) + result = result._replace(validation_schema=build_validation_schema(serializer)) + serializer_name = type(serializer).__name__ + for path in sorted(_iter_auto_deferred_paths(result)): + logger.debug( + "Auto-deferring nested serializer field '%s' on %s", path, serializer_name + ) + return result diff --git a/kolibri/core/utils/values_viewset/method_fields.py b/kolibri/core/utils/values_viewset/method_fields.py new file mode 100644 index 00000000000..b10ada0c8f1 --- /dev/null +++ b/kolibri/core/utils/values_viewset/method_fields.py @@ -0,0 +1,78 @@ +""" +Method-field support for ``ValuesViewset``: the public ``ValuesMethodField`` +plus the per-call runtime carriers (``MethodContext``, ``_SourcesProxy``) the +engine threads into a field's ``get_*`` method. Leaf module — no package deps. +""" + +from rest_framework.serializers import SerializerMethodField + + +class ValuesMethodField(SerializerMethodField): + """ + ``SerializerMethodField`` for ``ValuesViewset``: declares the row columns + the bound method reads. + + ``sources`` use DRF dot notation (``"dataset.id"``), translated to ORM + double-underscore (``"dataset__id"``) for the ``values()`` query. The method + receives a proxy over the declared sources; sources not also declared as + output fields are fetched then stripped from the final row. + + status = ValuesMethodField(sources=("transfer_status", "last_synced")) + + def get_status(self, obj): + if obj.transfer_status in IN_PROGRESS: + return SYNCING + """ + + def __init__(self, *, sources=(), method_name=None, **kwargs): + super().__init__(method_name=method_name, **kwargs) + self.sources = tuple(sources) + + +class MethodContext: + """ + Context carrier passed as ``self`` to ``ValuesMethodField`` ``get_*()`` + methods. Created per ``serialize()``, never stored on the shared engine, so + concurrent requests don't share context. + """ + + __slots__ = ("context",) + + def __init__(self, context): + self.context = context + + +class _SourcesProxy: + """ + Attribute proxy over a raw ``values()`` row, scoped to a + ``ValuesMethodField``'s declared ``sources``. + + Dotted traversal matches declared paths: for ``sources=("publisher.name",)``, + ``obj.publisher.name`` returns ``raw["publisher__name"]``. Access outside the + declared set raises ``AttributeError`` naming the requested attr and the + declared sources. + """ + + __slots__ = ("_raw", "_sources", "_prefix") + + def __init__(self, raw, sources, prefix=""): + self._raw = raw + self._sources = sources + self._prefix = prefix + + def __getattr__(self, name): + # repr/copy/pickle etc. must see a plain AttributeError for `_`-names, + # not the framed message below. + if name.startswith("_"): + raise AttributeError(name) + path = "{}__{}".format(self._prefix, name) if self._prefix else name + if path in self._sources: + return self._raw[path] + sep = path + "__" + if any(source.startswith(sep) for source in self._sources): + return _SourcesProxy(self._raw, self._sources, path) + declared = sorted(source.replace("__", ".") for source in self._sources) + raise AttributeError( + "{!r} not declared — ValuesMethodField exposes sources only: " + "{}. Add to sources=, or inline the logic.".format(name, declared) + )