allow emulator setup for partners testing DCP

tests/datacommons-integration-tests/README.md

@@ -46,55 +46,84 @@ While running with `--keep-containers`, you can access:
 - **Ingestion Helper**: `localhost:8081`
 - **Website Backend**: `localhost:8082`
 
-## Custom Feature Flags
-
-Local integration tests run with the Spanner multi-entity schema enabled by default in the Mixer backend. This is controlled by mounting [custom_feature_flags.yaml](./custom_feature_flags.yaml) inside the `website` container at `/workspace/deploy/featureflags/custom.yaml`.
-
-The container entrypoint detects the environment variable `RESOLVE_WITH_SPANNER_EMBEDDINGS=true` (defined in `docker-compose.test.yml`) and starts Mixer using these flags.
-
-You can modify the flags inside `custom_feature_flags.yaml` if you need to tweak the Mixer server configuration for local debugging.
+## Spanner Emulator Local Testing & Limitations
+
+The local Docker Compose stack uses the Spanner Emulator for offline testing.
+
+### Emulator Optimizer Bypass
+The emulator is started with `--disable_query_null_filtered_index_check`. Required because the emulator rejects queries on null-filtered indexes (e.g., `NodeEmbeddingIndex`) unless all index keys are explicitly filtered for non-null values.
+
+### Key Drawbacks & Gaps
+* **No Real Vector Similarity (NL Search):** The emulator cannot run semantic vector searches (returns empty results). Also, public/production NLP API endpoints do not index custom locally-seeded variables, and the emulator gateway fails to pass `--disable_query_null_filtered_index_check` to the C++ gRPC engine on port `9010`. Local queries must bypass vector search by appending `disable_feature=use_v2_resolve_for_nl_search_vars` to use the text matcher.
+* **Local UDF Routing only:** The `--remote_functions_host_port` flag can route `ML.PREDICT` to a local mock container on `localhost`, but connections are restricted to localhost.
+* **Performance:** Runs in-memory and does not replicate multi-node Spanner scale or latency.
+* **Heuristic Bypass Limitation:** Appending `disable_feature=use_v2_resolve_for_nl_search_vars` disables database-level vector index queries to bypass the emulator limits. **IMPORTANT:** This is a test-only workaround. To ensure that the actual production database index lookup and vector resolving code paths are verified before release, you must run the GCP sandbox integration tests (`run_gcp_integration_test.py`), which cover the active production vector path against a real GCP Spanner instance.
+
+### Manual Verification of Local NL Search
+1. **Boot containers:**
+   ```bash
+   docker compose -f tests/datacommons-integration-tests/docker-compose.test.yml up -d spanner gcs
+   ```
+2. **Seed Spanner and boot remaining services:**
+   ```bash
+   uv run pytest tests/datacommons-integration-tests/run_local_integration_test.py --keep-containers
+   ```
+3. **Query using the heuristic bypass:**
+   ```bash
+   curl -s -X POST "http://localhost:8082/api/explore/detect-and-fulfill?q=Number+of+frogs+in+United+States+of+America&disable_feature=use_v2_resolve_for_nl_search_vars" \
+     -H "Content-Type: application/json" \
+     -d '{"contextHistory": []}'
+   ```
+
+### Testing Custom NLP Models & Vertex AI
+
+* **Option A: Custom NLP Service URL:** Point the `NL_SERVICE_ROOT_URL` environment variable of the `website` container to your custom model container port or external model API endpoint (and supply any required API keys in the container env).
+* **Option B: Local Real NLP Service Container (CPU/MiniLM):** Run the official model server container locally on CPU using the HuggingFace `all-MiniLM-L6-v2` model:
+  1. Enable the override configuration:
+     ```bash
+     cp tests/datacommons-integration-tests/docker-compose.override.yml.example tests/datacommons-integration-tests/docker-compose.override.yml
+     ```
+  2. Set your GCP project ID in `docker-compose.override.yml`:
+     ```yaml
+     GCP_PROJECT_ID: <YOUR_GCP_PROJECT_ID>
+     ```
+  3. Start the containers using the override:
+     ```bash
+     docker compose -f tests/datacommons-integration-tests/docker-compose.test.yml -f tests/datacommons-integration-tests/docker-compose.override.yml up -d
+     uv run pytest tests/datacommons-integration-tests/run_local_integration_test.py --keep-containers
+     ```
+     *(Note: the automated pytest asserts will fail due to real vs. mock output differences, but the local stack will remain running and fully operational for custom testing)*.
+* **Option C: Spanner UDF Proxy:** To test database-level `ML.PREDICT` calls, run a local proxy container in the Spanner service's network namespace (`network_mode: "service:spanner"`) to forward requests to Vertex AI. Configure the Spanner emulator container with `--remote_functions_host_port=localhost:<proxy_port>`.
 
 ---
 
 ## GCP Sandbox Integration Tests
 
-In addition to local emulated tests, this package includes a script to run automated, end-to-end integration tests on **real GCP sandbox resources**. This validates Terraform scaffolding, GCS transfers, Spanner database seeding, Cloud Workflow executions, and Cloud Run web server APIs.
+This package includes a script to run end-to-end integration tests on real GCP sandbox resources to validate Terraform templates, GCS transfers, Spanner database seeding, and Cloud Run web server APIs.
 
 ### Running the GCP Tests
-
-To run the GCP integration test suite (requires authenticated GCP CLI access):
+Requires authenticated GCP CLI access:
 ```bash
 python3 tests/datacommons-integration-tests/run_gcp_integration_test.py \
     --project-id <YOUR_GCP_PROJECT_ID> \
     --dc-api-key "YOUR_GOOGLE_DATA_COMMONS_API_KEY"
 ```
 
-Alternatively, you can pass the API key via an environment variable:
+Alternatively, pass the API key via an environment variable:
 ```bash
 DC_API_KEY="YOUR_API_KEY" python3 tests/datacommons-integration-tests/run_gcp_integration_test.py \
     --project-id <YOUR_GCP_PROJECT_ID>
 ```
 
 ### Script Arguments
+*   `--project-id`: GCP Project ID (default: `datcom-ci`).
+*   `--dc-api-key`: Data Commons API Key.
+*   `--region`: GCP region (default: `us-central1`).
+*   `--namespace`: Custom resource naming namespace (default: randomized `itest-XXXX`).
+*   `--keep-sandbox`: Do not destroy sandbox GCP resources on completion (useful for debugging).
+*   `--reuse-sandbox`: Reuse existing sandbox resources (requires passing persistent `--namespace` and having run with `--keep-sandbox` previously).
+*   `--tf-git-ref`: Git reference for the Terraform templates repository (default: `main`).
+*   `--services-image`, `--preprocessing-image`, `--helper-image`: Override container images deployed during provisioning.
 
-*   `--project-id`: The Google Cloud Project ID where the sandbox resources should be provisioned (default: `datcom-ci`).
-*   `--dc-api-key`: Google Data Commons API Key needed to authenticate and configure sandbox clients.
-*   `--region`: The GCP region to deploy resources (default: `us-central1`).
-*   `--namespace`: Custom naming namespace for resources (default: randomized `itest-XXXX`).
-*   `--keep-sandbox`: Do not destroy sandbox GCP resources on completion/failure. Useful for debugging active instances.
-*   `--reuse-sandbox`: Reuse existing local workspace and GCP sandbox resources if they exist. Requires passing a persistent, custom `--namespace` (e.g. `--namespace itest-9611`) and having run with `--keep-sandbox` in the previous run.
-*   `--tf-git-ref`: Git reference branch/tag/commit for the GCP Terraform templates repository (default: `main`).
-*   `--services-image`, `--preprocessing-image`, `--helper-image`: Override container image tags deployed during provisioning.
-
-### E2E Verification Stages & Philosophy
-
-Once the GCP sandbox stack is up and seeded with dataset MCFs, the script triggers the entire API verification suite concurrently. This validates three core capabilities of the deployed Data Commons Platform backend:
-
-*   **Stage A: V2 Observations & Custom Data Retrieval**
-    *   *Philosophy:* Validates E2E query routing and data loading for both standard and custom variables. It queries base DC SDG indicators to verify backward compatibility with global schemas, and asserts observations values on custom seeded frog metrics to verify local database mapping and seeding.
-*   **Stage B: V2 Bulk Group Info & Hierarchy Resolution**
-    *   *Philosophy:* Verifies category tree resolution and variable grouping APIs. This validates that the serving container's Mixer component correctly processes unconstrained hierarchy crawls and handles location constraints (e.g. comparing available variables for USA vs Vatican City) without backend lookup errors.
-*   **Stage C: V2 Embeddings & Natural Language Resolution**
-    *   *Philosophy:* Validates custom search index and embeddings generation. This checks that the Ingestion Helper successfully generated vector coordinates from custom MCs, loaded them into Spanner, and that the Resolver can map natural language queries (like "frogs") back to these custom database entities.
 
 

tests/datacommons-integration-tests/clean_orphaned_sandboxes.py

@@ -0,0 +1,232 @@
+#!/usr/bin/env python3
+# Copyright 2026 Google LLC.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""Helper script to list and clean up leaked integration test GCP sandboxes.
+
+It reconstructs local Terraform scaffolding to connect to GCS remote state
+and execute 'terraform destroy' for orphaned namespaces.
+"""
+
+import argparse
+import re
+import shutil
+import subprocess
+import sys
+from pathlib import Path
+
+
+def run_command(
+    args: list[str], cwd: Path | None = None, *, check: bool = True
+) -> subprocess.CompletedProcess:
+    print(f"Running command: {' '.join(args)} (cwd: {cwd or Path.cwd()})", flush=True)
+    try:
+        return subprocess.run(  # noqa: S603
+            args, cwd=cwd, check=check, text=True, capture_output=True
+        )
+    except subprocess.CalledProcessError as e:
+        print(f"--- COMMAND FAILED: {' '.join(args)} ---", file=sys.stderr)
+        if e.stdout:
+            print(f"STDOUT:\n{e.stdout}", file=sys.stderr)
+        if e.stderr:
+            print(f"STDERR:\n{e.stderr}", file=sys.stderr)
+        print("---------------------------------------", file=sys.stderr)
+        raise e
+
+
+def get_active_namespaces(tf_state_bucket: str) -> list[str]:
+    """Lists all prefixes (namespaces) in GCS tf-state-bucket."""
+    print(f"Listing active states in gs://{tf_state_bucket}/terraform/state/...")
+    try:
+        proc = run_command(
+            ["gcloud", "storage", "ls", f"gs://{tf_state_bucket}/terraform/state/"]
+        )
+        namespaces = []
+        # Expected format: gs://bucket-name/terraform/state/itest-XXXX/
+        pattern = rf"gs://{re.escape(tf_state_bucket)}/terraform/state/([^/]+)/"
+        for line in proc.stdout.splitlines():
+            match = re.match(pattern, line.strip())
+            if match:
+                namespaces.append(match.group(1))
+        return sorted(set(namespaces))
+    except Exception as e:  # noqa: BLE001
+        print(f"Error listing state directories: {e}", file=sys.stderr)
+        return []
+
+
+def destroy_sandbox(
+    project_id: str, tf_state_bucket: str, namespace: str, workspace_root: Path
+) -> bool:
+    """Reconstructs local state and executes terraform destroy."""
+    print("\n=====================================================================")
+    print(f" DESTROYING SANDBOX: namespace={namespace} in project={project_id}")
+    print("=====================================================================")
+
+    workspace_dir = Path(f"/tmp/workspace-purge-{namespace}")  # noqa: S108
+    sandbox_dir = workspace_dir / namespace
+
+    try:
+        # Create directories
+        if workspace_dir.exists():
+            shutil.rmtree(workspace_dir)
+        sandbox_dir.mkdir(parents=True)
+
+        # Copy local infra modules & configs
+        local_infra_dir = workspace_root / "infra" / "dcp"
+        shutil.copy(local_infra_dir / "variables.tf", sandbox_dir / "variables.tf")
+        shutil.copy(local_infra_dir / "outputs.tf", sandbox_dir / "outputs.tf")
+        shutil.copy(local_infra_dir / "main.tf", sandbox_dir / "main.tf")
+        shutil.copytree(local_infra_dir / "modules", sandbox_dir / "modules")
+
+        # Construct backend.tf
+        backend_content = f"""terraform {{
+  backend "gcs" {{
+    bucket = "{tf_state_bucket}"
+    prefix = "terraform/state/{namespace}"
+  }}
+}}
+"""
+        (sandbox_dir / "backend.tf").write_text(backend_content, encoding="utf-8")
+
+        # Construct terraform.tfvars
+        tfvars_content = f"""project_id = "{project_id}"
+namespace = "{namespace}"
+spanner_create_instance = false
+spanner_instance_id = "dcp-integration-test-shared-instance"
+spanner_create_database = true
+spanner_create_bigquery_reservation = false
+datacommons_services_min_instances = 1
+datacommons_services_max_instances = 1
+"""
+        (sandbox_dir / "terraform.tfvars").write_text(tfvars_content, encoding="utf-8")
+
+        # Init & Destroy
+        run_command(["terraform", "init"], cwd=sandbox_dir)
+        run_command(["terraform", "destroy", "-auto-approve"], cwd=sandbox_dir)
+
+        # Delete state file in GCS
+        print(f"Removing GCS state files for namespace {namespace}...")
+        run_command(
+            [
+                "gcloud",
+                "storage",
+                "rm",
+                "--recursive",
+                f"gs://{tf_state_bucket}/terraform/state/{namespace}/",
+            ]
+        )
+
+        print(f"Successfully destroyed sandbox resources for namespace: {namespace}")
+        return True
+
+    except Exception as e:  # noqa: BLE001
+        print(f"Failed to destroy sandbox {namespace}: {e}", file=sys.stderr)
+        return False
+
+    finally:
+        if workspace_dir.exists():
+            shutil.rmtree(workspace_dir)
+
+
+def main() -> None:
+    parser = argparse.ArgumentParser(
+        description="Purge orphaned GCP sandbox integration test resources."
+    )
+    parser.add_argument(
+        "--project-id", default="datcom-ci", help="GCP project ID (default: datcom-ci)"
+    )
+    parser.add_argument(
+        "--tf-state-bucket",
+        default="tf-state-dcp-test-datcom-ci",
+        help="GCS bucket for remote Terraform state (default: tf-state-dcp-test-datcom-ci)",
+    )
+    parser.add_argument(
+        "--namespace",
+        default=None,
+        help="Specific namespace to destroy (e.g. itest-1234)",
+    )
+    parser.add_argument(
+        "--list",
+        action="store_true",
+        help="List active namespaces in the state bucket and exit",
+    )
+    parser.add_argument(
+        "--all",
+        action="store_true",
+        help="Destroy all active namespaces found in the state bucket",
+    )
+
+    args = parser.parse_args()
+    workspace_root = Path(__file__).resolve().parent.parent.parent
+
+    # 1. Fetch active namespaces
+    active_namespaces = get_active_namespaces(args.tf_state_bucket)
+    if not active_namespaces:
+        print("No active integration test state namespaces found in GCS.")
+        return
+
+    print(f"Active namespaces found ({len(active_namespaces)}):")
+    for ns in active_namespaces:
+        print(f"  - {ns}")
+
+    if args.list:
+        return
+
+    # 2. Determine target namespaces
+    targets = []
+    if args.namespace:
+        if args.namespace not in active_namespaces:
+            print(
+                f"Error: Namespace '{args.namespace}' was not found in GCS.",
+                file=sys.stderr,
+            )
+            sys.exit(1)
+        targets = [args.namespace]
+    elif args.all:
+        targets = active_namespaces
+    else:
+        print(
+            "\nSpecify either --namespace <name>, --all, or --list. See --help for details."
+        )
+        sys.exit(1)
+
+    # 3. Confirm destroy
+    print(
+        f"\nWARNING: This will permanently destroy all resources associated with the following namespaces: {', '.join(targets)}"
+    )
+    confirm = input("Are you sure you want to proceed? [y/N]: ").strip().lower()
+    if confirm not in ["y", "yes"]:
+        print("Aborted.")
+        sys.exit(0)
+
+    # 4. Run destroys
+    successes = 0
+    failures = 0
+    for ns in targets:
+        if destroy_sandbox(args.project_id, args.tf_state_bucket, ns, workspace_root):
+            successes += 1
+        else:
+            failures += 1
+
+    print("\n=====================================================================")
+    print(" PURGE SUMMARY")
+    print("=====================================================================")
+    print(f"  Successes: {successes}")
+    print(f"  Failures:  {failures}")
+    if failures > 0:
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()

tests/datacommons-integration-tests/docker-compose.override.yml.example

@@ -0,0 +1,27 @@
+# Example docker-compose override file to run real local NLP (CPU-based MiniLM).
+# To use:
+#   cp docker-compose.override.yml.example docker-compose.override.yml
+#
+services:
+  website:
+    environment:
+      - NL_SERVICE_ROOT_URL=http://nl-service:6060
+    depends_on:
+      - spanner
+      - nl-service
+
+  nl-service:
+    image: gcr.io/datcom-ci/datacommons-nl:latest
+    container_name: itest-nl-service
+    volumes:
+      - ~/.config/gcloud:/root/.config/gcloud
+      - ./local_env.yaml:/datacommons/nl/env.yaml
+      - ../../website/deploy/helm_charts/dc_website/nl/catalog.yaml:/datacommons/nl/catalog.yaml
+    environment:
+      - GOOGLE_APPLICATION_CREDENTIALS=/root/.config/gcloud/application_default_credentials.json
+      - GCP_PROJECT_ID=<YOUR_GCP_PROJECT_ID>
+      - NUM_WORKERS=1
+    networks:
+      - test-network
+    extra_hosts:
+      - "metadata.google.internal:127.0.0.1"