Debug the agent: AI agent traces with the OpenSearch Agent Traces - #4206
Debug the agent: AI agent traces with the OpenSearch Agent Traces#4206ps48 wants to merge 4 commits into
Conversation
…gent Traces plugin Signed-off-by: Shenoy Pratik Gurudatt <sgguruda@amazon.com>
|
Thank you for submitting a blog post! The blog post review process is: Submit a PR -> (Optional) Peer review -> Doc review -> Marketing review -> Published. |
ps48
changed the title
- Point the series link to the single-pane-of-glass intro post - Point Observability Stack links to opensearch-project/observability-stack Signed-off-by: Shenoy Pratik Gurudatt <sgguruda@amazon.com>
|
@kolchfa-aws - When you have a moment, please review and provide feedback. |
kolchfa-aws
left a comment
kolchfa-aws
left a comment
There was a problem hiding this comment.
Thank you, @ps48. Please see my comments and let me know if you have any questions. If possible, crop the images or retake them so they have a larger font and readers can follow what is depicted on the images.
|
|
||
| {:class="img-centered"} | ||
|
|
||
| The chart confirms: weather-agent accounts for the majority of errors (both `execute_tool` and `invoke_agent` spans fail), events-agent has a single tool error from a separate fault injection, and travel-planner carries `invoke_agent` errors from orchestrating the failed sub-agents. This is your blast radius: a single MCP tool returning 503 cascades up through the weather-agent and into the orchestrator. |
There was a problem hiding this comment.
The bar chart shows the events-agent bar in green, which the legend maps to invoke_agent, not execute_tool. The chart shows: weather-agent 6 (4 invoke_agent + 2 execute_tool), travel-planner 5, events-agent 1. The text says "the majority of errors", but really it's half of errors.
There was a problem hiding this comment.
"cascades up" is a phrasal verb. Reword using a precise verb.
|
|
||
| ## The multi-agent travel planner | ||
|
|
||
| Before diving into the failure, look at the service topology. Navigate to **Topology Map** in the Observability Stack workspace. The Application Map shows how the travel-planner fans out to weather-agent and events-agent, both of which call the mcp-server for their tools. The weather-agent node shows a red fault indicator (20% fault rate from our injected errors). |
There was a problem hiding this comment.
Navigate to Topology map - is it the same as Application map? What's the actual name of the element in the UI and how exactly does the user navigate to it?
There was a problem hiding this comment.
How is the 20% fault rate calculated? The traces list shows 10 traces/171 spans, the visualization shows 11 traces/205 spans.
|
|
||
| {:class="img-centered"} | ||
|
|
||
| Scan the Output column. Two traces show "Weather info temporarily unavailable" in their response, confirming partial failures. Click the Tokyo trace row to open the detail flyout. |
There was a problem hiding this comment.
The image on line 54 doesn't show the Output column, and all rows show "Success". Where does it show "Weather info temporarily unavailable"? What is the Tokyo trace row?
|
|
||
| ## Opening agent traces and finding the error | ||
|
|
||
| Navigate to **Agent Monitoring > Traces** in the left nav. The Agent Traces plugin shows root-level agent invocations with columns for Kind, Name, Status, Latency, Tokens, Input, and Output. Each row is a complete agent invocation. The metrics bar at the top shows aggregate stats: total traces, total spans, total tokens, and latency percentiles. |
There was a problem hiding this comment.
The Output column is not visible on the image. It does show the Time column, which is not mentioned. If the screenshot is retaken with larger font, it won't be necessary to list out the columns that the user will see in the image.
|
|
||
| The flyout opens with the trace header (Agent badge, status, trace ID, duration, span count, token count) and a left-panel **Trace Tree** showing the full span hierarchy. You can read the execution flow directly: | ||
|
|
||
| ``` |
There was a problem hiding this comment.
This diagram shows the same tree as the image but more succinct. If the image has a large enough font, the diagram is not needed. Alternatively, you can remove the image. Images with a font that is not readable are not helpful for the reader to follow along.
|
|
||
| ### The scenario | ||
|
|
||
| We inject a `tool_error` fault into the weather agent. The MCP tool `get_current_weather` returns a 503, the weather-agent propagates a null response, and the orchestrator produces a partial result ("Weather info temporarily unavailable"). Our goal: find the failing tool call in the trace tree, inspect its error details, and measure how many requests were impacted. |
There was a problem hiding this comment.
Reword by referring to the reader as "you". Assume that the reader is following this example and performing these steps. Use "you" and "your" instead of "we" and "our".
|
|
||
| We inject a `tool_error` fault into the weather agent. The MCP tool `get_current_weather` returns a 503, the weather-agent propagates a null response, and the orchestrator produces a partial result ("Weather info temporarily unavailable"). Our goal: find the failing tool call in the trace tree, inspect its error details, and measure how many requests were impacted. | ||
|
|
||
| ```bash |
There was a problem hiding this comment.
All code blocks must have an intro sentence ending in a colon.
|
|
||
| Before diving into the failure, look at the service topology. Navigate to **Topology Map** in the Observability Stack workspace. The Application Map shows how the travel-planner fans out to weather-agent and events-agent, both of which call the mcp-server for their tools. The weather-agent node shows a red fault indicator (20% fault rate from our injected errors). | ||
|
|
||
| {:class="img-centered"} |
There was a problem hiding this comment.
All images must have an intro sentence ending in a period. Tell the reader what the reader should notice in the image ("....., as shown in the following image").
|
|
||
| {:class="img-centered"} | ||
|
|
||
| The raw span reveals everything you need for RCA: |
There was a problem hiding this comment.
"everything you need" is speculative. Please reword in a more neutral way.
| └── chat summarize (LLM, 60ms, 831 tokens) | ||
| ``` | ||
|
|
||
| The right panel shows the selected span's details. Select `execute_tool get_current_weather` in the tree. The detail panel immediately shows "Error" with status code 2 and the exception event: |
There was a problem hiding this comment.
Does this describe the right panel before telling the reader to select the span? If so, this should be in reverse order.
…enSearch Agent Traces plugin Signed-off-by: Shenoy Pratik Gurudatt <sgguruda@amazon.com>
|
Thanks so much for the review, really appreciate it. I have pushed an update: Reworded per reviewer feedback: removed metaphors and phrasal verbs (blast radius, fans out, cascades up, 30,000-foot view), switched the walkthrough to second person, and added colon lead-ins for code blocks and descriptive lead-ins for every image. Please take another look when you have a moment, and let me know if anything else needs changing. Happy to keep iterating. |
|
@kolchfa-aws - Can you do one more review here. Looks like there are some changes. |
Signed-off-by: Fanit Kolchina <kolchfa@amazon.com>
|
@ps48 Editorial review complete and changes pushed to this PR. Please review my changes for accuracy, and we'll be ready for the final marketing review/publishing. |
3 participants
Description
Adds the blog post Debug the agent: AI agent traces with the OpenSearch Agent Traces plugin, part of the OpenSearch Observability Stack SRE
blog series. The post walks an SRE through a root-cause-analysis scenario using
the Observability Stack, with freshly captured screenshots and verified
configuration.
Issues Resolved
Resolves #4205
Check List
By submitting this pull request, I confirm that my contribution is made under
the terms of the BSD-3-Clause License.