Skip to content
Draft
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 7 additions & 1 deletion ee/hogai/context/insight/query_executor.py
Original file line number Diff line number Diff line change
Expand Up @@ -442,7 +442,12 @@ def process_query_dict_with_tags() -> dict | BaseModel:
if query_status.get("error"):
if error_message := query_status.get("error_message"):
raise APIException(error_message)
raise Exception("Query failed")
# The status hides the message text because it's not user-safe, but error_code
# carries the machine-readable cause (a ClickHouse memory limit, an internal
# error, a worker crash). These are transient, so retry instead of dead-ending
# on an opaque "Query failed" that the report pipeline never retries.
error_code = query_status.get("error_code")
raise MaxToolRetryableError(f"Query failed: {error_code}" if error_code else "Query failed")
Comment on lines +445 to +450

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Structured cause is lost before report classification

should_fix bug

Why we think it's a valid issue
  • Checked: the failure path from ee/hogai/context/insight/query_executor.py:449-450 into the AI report pipeline (products/exports/backend/temporal/subscriptions/ai_subscription/report_pipeline.py), the disclosure helpers in products/exports/backend/temporal/subscriptions/types.py, and the delivery UI in products/subscriptions/frontend/scenes/components/SubscriptionAiReportDelivery.tsx.
  • Found: the new error_code lives only inside the exception text. report_pipeline.py:589 sets type_name = type(last_exc).__name__, which is MaxToolRetryableError. report_pipeline.py:590 calls undisclosed_query_error_type(last_exc), which walks __cause__/__context__ for a ClickHouse class; the executor raises from a status dict, so that walk finds nothing. safe_error_message() (types.py:47) matches only ExposedHogQLError/ResolutionError, so human_readable_error stays None. The persisted QueryStepDiagnostic.error_type therefore records MaxToolRetryableError.
  • Found: the same class name reaches copy a person reads. The step placeholder at report_pipeline.py:600-602 renders Query failed to run (MaxToolRetryableError), and GenerateAIReportResult.failure_error() (types.py:288-298) builds the fully-degraded delivery message from those type names.
  • Found: a second effect of the new retry path. report_pipeline.py:575 feeds the fix LLM safe_error_message(exc) or type(exc).__name__, so each retry asks for a rewrite based on the string MaxToolRetryableError. Each failed step now spends up to 2 fix-LLM calls plus 2 ClickHouse reruns with no cause to work from.
  • Found: a naming trap for the follow-up fix. clickhouse_error_type (posthog/errors.py:79-83) emits CHQueryError<Label> names, but UNDISCLOSED_QUERY_ERROR_TYPES (types.py:13) holds ClickHouseQueryMemoryLimitExceeded. A plain plumb-through of the code would pass the memory-limit case straight through the suppression that exists to hide it.
  • Impact: the machine-readable cause never reaches step classification, the persisted delivery diagnostics, or the report text, so the delivery record stays undiagnosable — the surface AI_REPORT_DIAGNOSTICS_KEY exists for (types.py:102-104).
  • Priority: lowered to should_fix. Two claims in the finding overstate the harm. The cause does reach logs and error tracking: report_pipeline.py:591-599 passes exc_info=last_exc and calls capture_exception(last_exc), and both carry the message text. The per-step UI also does not print the class name — SubscriptionAiReportDelivery.tsx:52,61 collapses a step with no human_readable_error to Failed plus a generic internal-error note. The retry half of the change does work, because MaxToolRetryableError is in _RETRYABLE_QUERY_ERRORS (report_pipeline.py:107-111). Nothing regresses and no data leaks today, so this is an incomplete fix rather than a merge blocker.
Issue description

The status provides a machine-readable cause. This code places the cause only in MaxToolRetryableError text. The subscription pipeline classifies failures with type(exc).__name__. Its safe_error_message() also ignores MaxToolRetryableError. The report therefore records and shows MaxToolRetryableError, not the new cause. This leaves the AI subscription problem unresolved.

Suggested fix

Add an error_code attribute to a query-specific retryable exception. Preserve it when the catch block rethrows the exception. Make the subscription pipeline read this field for diagnostics and apply its existing disclosure policy. Add a pipeline test for the final diagnostic and report copy.

Prompt to fix with AI (copy-paste)
## Context
@ee/hogai/context/insight/query_executor.py#L445-450

<issue_description>
The status provides a machine-readable cause. This code places the cause only in `MaxToolRetryableError` text. The subscription pipeline classifies failures with `type(exc).__name__`. Its `safe_error_message()` also ignores `MaxToolRetryableError`. The report therefore records and shows `MaxToolRetryableError`, not the new cause. This leaves the AI subscription problem unresolved.
</issue_description>

<issue_validation>
- **Checked:** the failure path from `ee/hogai/context/insight/query_executor.py:449-450` into the AI report pipeline (`products/exports/backend/temporal/subscriptions/ai_subscription/report_pipeline.py`), the disclosure helpers in `products/exports/backend/temporal/subscriptions/types.py`, and the delivery UI in `products/subscriptions/frontend/scenes/components/SubscriptionAiReportDelivery.tsx`.
- **Found:** the new `error_code` lives only inside the exception text. `report_pipeline.py:589` sets `type_name = type(last_exc).__name__`, which is `MaxToolRetryableError`. `report_pipeline.py:590` calls `undisclosed_query_error_type(last_exc)`, which walks `__cause__`/`__context__` for a ClickHouse class; the executor raises from a status dict, so that walk finds nothing. `safe_error_message()` (`types.py:47`) matches only `ExposedHogQLError`/`ResolutionError`, so `human_readable_error` stays `None`. The persisted `QueryStepDiagnostic.error_type` therefore records `MaxToolRetryableError`.
- **Found:** the same class name reaches copy a person reads. The step placeholder at `report_pipeline.py:600-602` renders `Query failed to run (MaxToolRetryableError)`, and `GenerateAIReportResult.failure_error()` (`types.py:288-298`) builds the fully-degraded delivery message from those type names.
- **Found:** a second effect of the new retry path. `report_pipeline.py:575` feeds the fix LLM `safe_error_message(exc) or type(exc).__name__`, so each retry asks for a rewrite based on the string `MaxToolRetryableError`. Each failed step now spends up to 2 fix-LLM calls plus 2 ClickHouse reruns with no cause to work from.
- **Found:** a naming trap for the follow-up fix. `clickhouse_error_type` (`posthog/errors.py:79-83`) emits `CHQueryError<Label>` names, but `UNDISCLOSED_QUERY_ERROR_TYPES` (`types.py:13`) holds `ClickHouseQueryMemoryLimitExceeded`. A plain plumb-through of the code would pass the memory-limit case straight through the suppression that exists to hide it.
- **Impact:** the machine-readable cause never reaches step classification, the persisted delivery diagnostics, or the report text, so the delivery record stays undiagnosable — the surface `AI_REPORT_DIAGNOSTICS_KEY` exists for (`types.py:102-104`).
- **Priority:** lowered to `should_fix`. Two claims in the finding overstate the harm. The cause does reach logs and error tracking: `report_pipeline.py:591-599` passes `exc_info=last_exc` and calls `capture_exception(last_exc)`, and both carry the message text. The per-step UI also does not print the class name — `SubscriptionAiReportDelivery.tsx:52,61` collapses a step with no `human_readable_error` to `Failed` plus a generic internal-error note. The retry half of the change does work, because `MaxToolRetryableError` is in `_RETRYABLE_QUERY_ERRORS` (`report_pipeline.py:107-111`). Nothing regresses and no data leaks today, so this is an incomplete fix rather than a merge blocker.
</issue_validation>

## Task
Investigate the issue and solve it

<potential_solution>
Add an `error_code` attribute to a query-specific retryable exception. Preserve it when the catch block rethrows the exception. Make the subscription pipeline read this field for diagnostics and apply its existing disclosure policy. Add a pipeline test for the final diagnostic and report copy.
</potential_solution>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a valid, but incomplete, gap and I'm escalating it for a human design decision rather than fixing it unattended.

What I confirmed: the query executor puts the real cause only in the exception's message text (Query failed: <code>) and re-raises without a structured field. The AI subscription pipeline classifies failures by exception class name and by walking the exception's cause chain — but the executor raises from a status dictionary, so there is no cause chain to walk. The result is that the persisted per-step diagnostic records MaxToolRetryableError instead of the real ClickHouse cause, even though that diagnostics surface exists specifically to make a degraded report debuggable after the fact. So the structured cause genuinely does not reach classification. (The cause still reaches logs and error tracking, which carry the message text, and nothing regresses today.)

Why I'm not fixing it here: the correct fix has to carry the code as a structured field from the executor (in ee/hogai) into the subscription pipeline (in products/exports), which crosses a product-isolation boundary and lands in a module deliberately kept import-light for the Temporal sandbox. More importantly, the pipeline's error type also feeds recipient-facing report copy, gated by a suppression list — and the two naming schemes diverge: the executor's code uses the CHQueryError… form, while the suppression list uses the ClickHouseQuery… form. A straight plumb-through would therefore risk showing a subscription recipient an error type that the suppression exists to hide. That is a disclosure decision, and there are a few defensible designs for both the field and the naming reconciliation.

What a human needs to decide: (1) how to carry the code as a structured field through the executor's re-raise; (2) how to reconcile the CHQueryError… vs ClickHouseQuery… naming so the suppression policy still holds before this reaches recipient-facing copy; (3) confirm the pipeline change respects the exports product boundary and Temporal-sandbox import constraints. Then a pipeline test should assert the final diagnostic carries the real cause and that a suppressed type stays suppressed in recipient copy.

Comment on lines +445 to +450

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Infrastructure failures enter the SQL rewrite loop

should_fix performance

Why we think it's a valid issue
  • Checked: the error taxonomy in ee/hogai/tool_errors.py, the report pipeline's retry set and fix loop, and what the pre-change code did with the same failure.
  • Found: the codebase already defines the class this case belongs to, and the diff picks the other one. ee/hogai/tool_errors.py:10-11 states the split: MaxToolTransientError is for "Intermittent issues that can be retried without changes (e.g., rate limits, timeouts)", MaxToolRetryableError is for "Solvable issues that can be fixed with adjusted inputs". MaxToolRetryableError.retry_strategy returns "adjusted" (tool_errors.py:86-93), so the hint appended for the LLM is "You may retry with adjusted inputs." (tool_errors.py:44-46). The diff's own comment at ee/hogai/context/insight/query_executor.py:447 calls these failures transient.
  • Found: the cost figure is right. _MAX_QUERY_FIX_RETRIES = 2 (report_pipeline.py:96) and the loop at report_pipeline.py:523-586 runs up to three attempts, so a step can add two fix-LLM calls (30s timeout each, _FIX_LLM_TIMEOUT_SECONDS) plus two more ClickHouse runs. MaxToolRetryableError is in _RETRYABLE_QUERY_ERRORS (report_pipeline.py:107-111), so the class choice alone decides this.
  • Found: the rewrite is blind. report_pipeline.py:575 passes the fixer safe_error_message(exc) or type(exc).__name__, and safe_error_message matches only ExposedHogQLError/ResolutionError (products/exports/backend/temporal/subscriptions/types.py:47), so the fixer receives the literal string MaxToolRetryableError and no cause.
  • Found: the change contradicts the pipeline's stated policy. The comment above _RETRYABLE_QUERY_ERRORS (report_pipeline.py:104-106) says timeouts, infra failures, and generic exceptions fall through to the placeholder without retrying, "since a different SELECT won't fix a ClickHouse outage or a heartbeat timeout".
  • Found: before the change this path did not retry. The old Exception("Query failed") fell into the catch-all at query_executor.py:474-485 and re-raised a plain Exception, which is absent from _RETRYABLE_QUERY_ERRORS. So the extra LLM calls and reruns for hidden-message failures are new with this diff.
  • Impact: during a ClickHouse degradation, every failed step of an AI report now triples its query attempts and spends two blind fix-LLM calls, and Max chat is told to retry with adjusted inputs for a failure no rewrite addresses.
  • Impact: the fix needs a decision rather than a mechanical swap. MaxToolTransientError is not in _RETRYABLE_QUERY_ERRORS, so switching to it removes the retry the PR wanted, and the pipeline has no retry-without-changes path today.
  • Priority: kept at should_fix rather than raised. The load added is bounded and sits on a low-volume scheduled path, and the same wrapping already happens for user-safe infra errors: query_executor.py:443-444 raises APIException(error_message), which the handler at query_executor.py:455-473 already converts into MaxToolRetryableError. The diff widens an existing pattern; it does not create it.
Issue description

Every hidden async failure becomes MaxToolRetryableError. This class enters the report pipeline's query-rewrite loop. Internal failures cannot be fixed by changing SQL. Each step can add two LLM calls and two ClickHouse runs during an outage. These blind retries can increase system load while dependencies are unhealthy.

Suggested fix

Classify error_code before choosing the exception. Use MaxToolRetryableError only when a SQL rewrite can fix the failure. Use MaxToolTransientError for temporary failures and retry once with backoff. Do not send unknown internal failures through the rewrite loop.

Prompt to fix with AI (copy-paste)
## Context
@ee/hogai/context/insight/query_executor.py#L445-450

<issue_description>
Every hidden async failure becomes MaxToolRetryableError. This class enters the report pipeline's query-rewrite loop. Internal failures cannot be fixed by changing SQL. Each step can add two LLM calls and two ClickHouse runs during an outage. These blind retries can increase system load while dependencies are unhealthy.
</issue_description>

<issue_validation>
- **Checked:** the error taxonomy in `ee/hogai/tool_errors.py`, the report pipeline's retry set and fix loop, and what the pre-change code did with the same failure.
- **Found:** the codebase already defines the class this case belongs to, and the diff picks the other one. `ee/hogai/tool_errors.py:10-11` states the split: `MaxToolTransientError` is for "Intermittent issues that can be retried without changes (e.g., rate limits, timeouts)", `MaxToolRetryableError` is for "Solvable issues that can be fixed with adjusted inputs". `MaxToolRetryableError.retry_strategy` returns `"adjusted"` (`tool_errors.py:86-93`), so the hint appended for the LLM is "You may retry with adjusted inputs." (`tool_errors.py:44-46`). The diff's own comment at `ee/hogai/context/insight/query_executor.py:447` calls these failures transient.
- **Found:** the cost figure is right. `_MAX_QUERY_FIX_RETRIES = 2` (`report_pipeline.py:96`) and the loop at `report_pipeline.py:523-586` runs up to three attempts, so a step can add two fix-LLM calls (30s timeout each, `_FIX_LLM_TIMEOUT_SECONDS`) plus two more ClickHouse runs. `MaxToolRetryableError` is in `_RETRYABLE_QUERY_ERRORS` (`report_pipeline.py:107-111`), so the class choice alone decides this.
- **Found:** the rewrite is blind. `report_pipeline.py:575` passes the fixer `safe_error_message(exc) or type(exc).__name__`, and `safe_error_message` matches only `ExposedHogQLError`/`ResolutionError` (`products/exports/backend/temporal/subscriptions/types.py:47`), so the fixer receives the literal string `MaxToolRetryableError` and no cause.
- **Found:** the change contradicts the pipeline's stated policy. The comment above `_RETRYABLE_QUERY_ERRORS` (`report_pipeline.py:104-106`) says timeouts, infra failures, and generic exceptions fall through to the placeholder without retrying, "since a different SELECT won't fix a ClickHouse outage or a heartbeat timeout".
- **Found:** before the change this path did not retry. The old `Exception("Query failed")` fell into the catch-all at `query_executor.py:474-485` and re-raised a plain `Exception`, which is absent from `_RETRYABLE_QUERY_ERRORS`. So the extra LLM calls and reruns for hidden-message failures are new with this diff.
- **Impact:** during a ClickHouse degradation, every failed step of an AI report now triples its query attempts and spends two blind fix-LLM calls, and Max chat is told to retry with adjusted inputs for a failure no rewrite addresses.
- **Impact:** the fix needs a decision rather than a mechanical swap. `MaxToolTransientError` is not in `_RETRYABLE_QUERY_ERRORS`, so switching to it removes the retry the PR wanted, and the pipeline has no retry-without-changes path today.
- **Priority:** kept at `should_fix` rather than raised. The load added is bounded and sits on a low-volume scheduled path, and the same wrapping already happens for user-safe infra errors: `query_executor.py:443-444` raises `APIException(error_message)`, which the handler at `query_executor.py:455-473` already converts into `MaxToolRetryableError`. The diff widens an existing pattern; it does not create it.
</issue_validation>

## Task
Investigate the issue and solve it

<potential_solution>
Classify error_code before choosing the exception. Use MaxToolRetryableError only when a SQL rewrite can fix the failure. Use MaxToolTransientError for temporary failures and retry once with backoff. Do not send unknown internal failures through the rewrite loop.
</potential_solution>

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a valid finding, and I'm escalating it for a human design decision rather than making an unattended change.

What I confirmed: a hidden-message async failure is now raised as the "retryable" error class, which the report pipeline treats as "the SQL is wrong, ask the LLM to rewrite it." But these are transient infrastructure failures (a ClickHouse outage, a worker crash), and no rewrite can fix them. Because the pipeline can't extract a cause from this error, it feeds the fixer only the literal class name and does a blind rewrite. During a ClickHouse degradation, each failed report step can add two 30-second fix-LLM calls plus two more query reruns, and Max chat is told to "retry with adjusted inputs" for something no input change addresses. That contradicts the pipeline's own stated policy that infra failures should not be retried.

Why this isn't a mechanical swap: the error taxonomy has a class that fits exactly — the transient class, meaning "retry once without changes." But that class is not in the pipeline's retryable set, so simply switching to it would remove the retry this PR was specifically written to add (before the PR, this failure did not retry at all). Reconciling "these should retry" with "don't blindly rewrite them" requires a capability the pipeline doesn't have today: retrying the same query without invoking the fix LLM.

What a human needs to decide: (1) the intended retry policy for hidden-message infra failures — retry the same SQL without a rewrite, or don't retry in the report pipeline at all; (2) if retrying, add the transient error class to the pipeline's retryable set together with a branch in the fix loop that skips the fix-LLM and reruns as-is; (3) how this interacts with Max chat, which consumes the same executor and shows the LLM a different retry hint depending on the class. This is best decided together with the related thread about carrying the structured cause through to classification, since the right class depends on that cause being available.


# Use the completed query results
response_dict = query_status["results"]
Expand All @@ -453,6 +458,7 @@ def process_query_dict_with_tags() -> dict | BaseModel:
HogQLNotImplementedError,
ExposedCHQueryError,
UserAccessControlError,
MaxToolRetryableError,
) as err:
elapsed = time.time() - start_time
# Handle known query execution errors with user-friendly messages
Expand Down
26 changes: 26 additions & 0 deletions ee/hogai/context/insight/test/test_query_executor.py
Original file line number Diff line number Diff line change
Expand Up @@ -396,6 +396,32 @@ async def test_async_query_polling_with_error(self, mock_get_query_status, mock_

self.assertIn("Query failed with error", str(context.exception))

@patch("ee.hogai.context.insight.query_executor.process_query_dict")
@patch("ee.hogai.context.insight.query_executor.get_query_status")
async def test_async_query_polling_error_without_message_is_retryable(
self, mock_get_query_status, mock_process_query
):
# A non-user-safe failure hides the message text but carries error_code. The step must retry
# (MaxToolRetryableError) and surface the cause, not dead-end on an opaque "Query failed".
mock_process_query.return_value = {"query_status": {"id": "test-query-id", "complete": False}}
mock_get_query_status.return_value = Mock(
model_dump=lambda mode: {
"id": "test-query-id",
"complete": True,
"error": True,
"error_message": None,
"error_code": "CHQueryErrorMemoryLimitExceeded",
}
)

query = AssistantTrendsQuery(series=[])

with patch("ee.hogai.context.insight.query_executor.asyncio.sleep"):
with self.assertRaises(MaxToolRetryableError) as context:
await self.query_runner.arun_and_format_query(query)

self.assertIn("CHQueryErrorMemoryLimitExceeded", str(context.exception))

@override_settings(TEST=False)
@patch("ee.hogai.context.insight.query_executor.process_query_dict")
async def test_execution_mode_in_production(self, mock_process_query):
Expand Down
7 changes: 6 additions & 1 deletion posthog/clickhouse/client/execute_async.py
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
from posthog.clickhouse.query_tagging import get_query_tags, tag_queries
from posthog.constants import AvailableFeature
from posthog.direct_query_cancellation import build_direct_query_cancellation_token, request_direct_query_cancellation
from posthog.errors import ExposedCHQueryError
from posthog.errors import ExposedCHQueryError, clickhouse_error_type
from posthog.exceptions import ClickHouseAtCapacity
from posthog.exceptions_capture import capture_exception
from posthog.renderers import SafeJSONRenderer
Expand Down Expand Up @@ -311,6 +311,11 @@ def execute_process_query(
codes = err.get_codes()
if isinstance(codes, str):
query_status.error_code = codes
if not query_status.error_code:
# The message text stays hidden when it's not user-safe, but the exception class is a
# stable, machine-readable cause (e.g. "CHQueryErrorMemoryLimitExceeded"). Carry it so
# callers can classify the failure instead of dead-ending on an opaque "Query failed".
query_status.error_code = clickhouse_error_type(err)
Comment thread
posthog[bot] marked this conversation as resolved.
Comment thread
posthog[bot] marked this conversation as resolved.
logger.exception("Error processing query async", team_id=team_id, query_id=query_id, exc_info=True)
if not is_user_safe_error:
# User-safe errors (e.g. a malformed HogQL query) are already returned to the user as a 400,
Expand Down
17 changes: 17 additions & 0 deletions posthog/clickhouse/client/test/test_execute_async.py
Original file line number Diff line number Diff line change
Expand Up @@ -363,6 +363,23 @@ def test_async_query_user_safe_error_carries_error_code(self):
assert result.error_message
self.assertEqual(result.error_code, ClickHouseQueryMemoryLimitExceeded.default_code)

def test_async_query_non_user_safe_error_carries_error_code_without_message(self):
query = build_query("SELECT * FROM events")
query_id = uuid.uuid4().hex

with patch("posthog.api.services.query.process_query_dict", side_effect=Exception("sensitive detail")):
client.enqueue_process_query_task(
self.team, self.user.id, query, query_id=query_id, _test_only_bypass_celery=True
)

result = client.get_query_status(self.team.id, query_id)
self.assertTrue(result.error)
self.assertTrue(result.complete)
# The message stays hidden for a non-user-safe error, but the class name is carried as a
# machine-readable cause so callers can classify the failure.
self.assertIsNone(result.error_message)
self.assertEqual(result.error_code, "Exception")

def test_async_query_server_errors(self):
query = build_query("SELECT * FROM events")

Expand Down
Loading