AgentRx: Diagnosing AI Agent Failures from Execution Trajectories

Nov 11, 2026 3 min

Reading notes on AgentRx, constraint validation, and root-cause diagnosis of agent failures.

AgentRx proposes a new attribution framework, but the overall process still has two steps: flag failures, then identify which failure was critical.


AgentRx groups failures into nine categories ($K_1$–$K_9$): instruction or plan adherence failure, intent–plan mismatch, fabrication of new information or hallucination, invalid tool call, misinterpretation of tool output, insufficient specification of user intent, unsupported intent, guardrail triggered, and system failure.

AgentRx takes three inputs: (i) a toolset $\mathcal{T}$ with input and output schemas, (ii) an optional domain policy $\Pi$, and (iii) a failed trajectory $T=\langle s_{1},\dots,s_{n}\rangle$. Each step $s_k$ contains multiple logged events with fields such as agent name, tool name, step index, tool output, or natural-language conversation.

AgentRx normalizes logs from different frameworks into a common intermediate representation (IR). The logs in the appendix show how the authors adapted different frameworks to this representation. A new framework would still need adjustments to its log adapter.

Let $T_{\leq k}$ denote the trajectory prefix through step $k$. AgentRx generates executable constraints from the tool schemas, domain policy, and trajectory prefixes. It evaluates those constraints step by step to produce an auditable validation log $V$. An LLM judge uses $V$ and the taxonomy checklist $K$ to predict the root-cause step $\hat{s}$ and category $\hat{y}$.

The LLM first generates global constraints $\mathcal{C}^{G}$ from the tool schemas and any available domain policy, then puts them in a global store $\mathsf{G}$. At each step, it uses the task instruction $I$, the observed prefix $T_{\leq k}$, and $\mathsf{G}$ as context to generate dynamic constraints $\mathcal{C}^{D}_{k}$, which it puts in a local store $\mathsf{L}_k$. The constraints available at step $k$ are $\mathcal{C}_k:=\mathcal{C}^{G}\cup\mathcal{C}^{D}_k$.

The guard $g_{C,k}=G_C(T_{\leq k},s_k)\in\{0,1\}$ determines whether a constraint applies. If the guard is 0, AgentRx skips the constraint. If it is 1, AgentRx evaluates the assertion $\phi_{C,k}=\Phi_C(T_{\leq k},s_k)\in\{\text{sat},\text{viol}\}$. The assertion can be checked programmatically or by an LLM. The evaluation at step $k$ is

$$ \text{Eval}_{C}(k)=\begin{cases}(\text{skip},\emptyset)&\text{if }g_{C,k}=0,\\ (\phi_{C,k},e)&\text{otherwise.}\end{cases} $$

After evaluating every step, we want to know which constraints were violated, at which steps, and what evidence supports each finding. The validation log $V$ collects these records:

$$ V\;:=\;\{(k,C,e)\mid C\in\mathcal{C}_{k},\ G_{C}(T_{\leq k},s_{k})=1,\ (\text{viol},e)=\text{Eval}_{C}(k)\}. $$

The LLM judge uses the task instruction, trajectory, taxonomy checklist, and validation log to output the critical failure step and root-cause category.


The table below compares AgentRx with the adapted W&W baseline.

Table 1. Failure attribution accuracy across datasets. We compare modified W&W (W&W*) and AgentRx on failed-agent and critical-step accuracy. Mean $\pm$ standard deviation over 3 runs.
Table 1. Failure attribution accuracy across datasets. We compare modified W&W (W&W*) and AgentRx on failed-agent and critical-step accuracy. Mean $\pm$ standard deviation over 3 runs.

AgentRx improves step localization accuracy across all domains and improves failed-agent identification in most settings. The comparison uses the W&W method: the authors convert trajectories from the AgentRx benchmark into the format W&W expects and adjust the prompt so the LLM identifies the first unrecoverable failure step in the log.

Comments