Multi-Agent Failure Attribution Benchmark Notes
Notes on the Who&When and MP-Bench multi-agent failure attribution benchmarks.
Who&When
Published in mid-2025, the paper uses relatively older models. The table shows the performance of OpenAI o1 and DeepSeek R1 on the dataset:

The dataset contains algorithm-generated and hand-crafted agent systems: 127 systems in total, with human annotations. It distinguishes two levels of error: Agent-Level (Who, identifying the responsible agent) and Step-Level (When, identifying the specific step). Step-level performance is lower.
Three evaluation methods are used: All-at-Once, Step-by-Step, and Binary Search. Their performance with GPT-4o is shown below:

Dataset example:

November 8 update: While working with the WW dataset, I found the AI-generated tasks rather unreasonable. This also helps explain why MP-Bench’s approach is reasonable.
For example, one task asks for the penultimate word of the chorus of the fifth song on an album. The Manager assigns the task sequentially to three subagents: a lyrics expert, a linguistic verification expert, and a music historian. At Step 0, the lyrics expert forwards part of the Manager’s message: the Manager assumes the fifth song is F (it is actually A) and asks for confirmation that it is F.
At Step 1, the music historian lists the album’s songs one by one but is biased by the user’s suggestion and concludes that F is fifth. Subsequent work based on this assumption naturally goes wrong.
I think the responsible agent and step should be the music historian and Step 1, since F was explicitly a claim to be verified and the historian did not do so. The actual annotation, however, identifies the lyrics expert and Step 0. From another perspective this answer is understandable: WW selects the earliest step containing the incorrect statement.
I evaluated Deepseek-V4.1-F1017 on 32 examples from Who&When’s eval set and also ran 8 development examples. The overall results are as follows:
Table 3. Comparison of my Who&When evaluation results
| Experiment group | Sample range | Representation | Validity rate | Agent accuracy | Step accuracy | Joint accuracy |
|---|---|---|---|---|---|---|
| Keyword heuristic | eval/16 | raw / canonical | 100% | 25.0% (4/16) | 12.5% (2/16) | 12.5% (2/16) |
dev development set | dev/4 | raw | 100% | 75.0% (3/4) | 50.0% (2/4) | 50.0% (2/4) |
dev development set | dev/4 | canonical | 100% | 100.0% (4/4) | 50.0% (2/4) | 50.0% (2/4) |
eval evaluation set | eval/16 | raw | 100% | 68.75% (11/16) | 50.0% (8/16) | 50.0% (8/16) |
eval evaluation set | eval/16 | canonical | 100% | 56.25% (9/16) | 37.5% (6/16) | 37.5% (6/16) |
The structured representation did not improve accuracy on WW. The LLM tended to attribute errors to the planner or decision-maker when using structured trajectories, whereas with less organized trajectories it leaned toward the part where the actual error appeared. In fact, among the 8/10 questions the LLM answered incorrectly, only one showed the multi-perspective problem I had expected; more often, the model seemed to lose its way in the long context and hallucinate or make an incorrect judgment.
MP-Bench
Different analysts may reasonably attribute a failure to different steps. This non-deterministic, multi-perspective nature arises from the interdependencies inherent in multi-agent systems. Multiple specialized agents can be combined interdependently into diverse, valid execution trajectories aimed at the same goal. Failure attribution therefore depends heavily on an observer’s internal reasoning about what constitutes a “correct” trajectory.
The limitations observed in previous work stem from biased evaluation assumptions that overlook the multi-perspective nature of MAS failures, rather than from insufficient model capabilities.

MP-Bench uses normalized discounted cumulative gain (nDCG@K), which evaluates agreement between the predicted and ground-truth rankings and gives more weight to correctly prioritizing highly salient failure steps.

The paper also evaluates a multi-agent setting:

TrajErrBench
Some existing datasets have the following limitations:
- Trajectories tend to be short: The algorithm-generated subset of Who&When averages only 8.72 steps, making it difficult to test the loss of evidence over long contexts.
- Scenarios can be narrow or toy-like: Many focus on simple multi-agent conversations, web shopping (WebShop), or text-based room games (ALFWorld).
- A lack of realistic complex settings: They offer limited coverage of complex multi-turn tool use with customer-service business rules and very long-horizon repository-level software development.
TrajErrBench consists of two complementary, challenging real-world subsets:
Table 6. Composition of TrajErrBench
| Subset | Trajectories | Average steps | Task setting and characteristics | LLMs generating failed trajectories |
|---|---|---|---|---|
| $\tau^{2}$-Bench subset | 400 | 29.27 | Multi-turn customer service and tool use in airline and retail settings: the agent gathers requirements through dialogue, calls backend APIs, and follows business policies such as change and refund rules, budgets, and seat-class restrictions. | DeepSeek-Reasoner, GPT-5, Qwen3-Max |
| SWE-Bench Pro subset | 86 | 119.70 | Long-horizon repository-level software engineering: the agent locates bugs, reads source files, writes patches, and runs tests in real GitHub repositories. | OpenHands with Claude-Opus-4.6, Kimi-K2.6-Coding, Gemini-3.1-Pro, Grok-4.20-Beta, GPT-5.4 |
| Total | 486 | — | Real tool interaction and long-horizon code repair | 8 frontier models |

The authors observed the following in this dataset:
- Critical errors often occur in the latter half of a trajectory. In previous datasets, agents often make decisive errors within the first few steps. In this dataset, agents spend much of the earlier trajectory analyzing and interpreting before they make a critical error while making key decisions or modifying code.
- Agents often fail during reasoning: this accounts for 60.7% of cases in $\tau^{2}$-Bench and 57.1% in SWE-Bench Pro.
Comments