Multi-Agent Failure Attribution Benchmark Notes

Nov 4, 2026 5 min

Notes on the Who&When and MP-Bench multi-agent failure attribution benchmarks.

Who&When

Published in mid-2025, the paper uses relatively older models. The table shows the performance of OpenAI o1 and DeepSeek R1 on the dataset:

Table 1. Failure attribution performance of OpenAI o1, DeepSeek R1, and GPT-4o
Table 1. Failure attribution performance of OpenAI o1, DeepSeek R1, and GPT-4o

The dataset contains algorithm-generated and hand-crafted agent systems: 127 systems in total, with human annotations. It distinguishes two levels of error: Agent-Level (Who, identifying the responsible agent) and Step-Level (When, identifying the specific step). Step-level performance is lower.

Three evaluation methods are used: All-at-Once, Step-by-Step, and Binary Search. Their performance with GPT-4o is shown below:

Table 2. GPT-4o failure attribution performance with and without ground-truth labels
Table 2. GPT-4o failure attribution performance with and without ground-truth labels

Dataset example:

Figure 1. Examples of failure logs from algorithm-generated and hand-crafted systems in Who&When.
Figure 1. Examples of failure logs from algorithm-generated and hand-crafted systems in Who&When.

November 8 update: While working with the WW dataset, I found the AI-generated tasks rather unreasonable. This also helps explain why MP-Bench’s approach is reasonable.

For example, one task asks for the penultimate word of the chorus of the fifth song on an album. The Manager assigns the task sequentially to three subagents: a lyrics expert, a linguistic verification expert, and a music historian. At Step 0, the lyrics expert forwards part of the Manager’s message: the Manager assumes the fifth song is F (it is actually A) and asks for confirmation that it is F.

At Step 1, the music historian lists the album’s songs one by one but is biased by the user’s suggestion and concludes that F is fifth. Subsequent work based on this assumption naturally goes wrong.

I think the responsible agent and step should be the music historian and Step 1, since F was explicitly a claim to be verified and the historian did not do so. The actual annotation, however, identifies the lyrics expert and Step 0. From another perspective this answer is understandable: WW selects the earliest step containing the incorrect statement.

I evaluated Deepseek-V4.1-F1017 on 32 examples from Who&When’s eval set and also ran 8 development examples. The overall results are as follows:

Table 3. Comparison of my Who&When evaluation results

Experiment groupSample rangeRepresentationValidity rateAgent accuracyStep accuracyJoint accuracy
Keyword heuristiceval/16raw / canonical100%25.0% (4/16)12.5% (2/16)12.5% (2/16)
dev development setdev/4raw100%75.0% (3/4)50.0% (2/4)50.0% (2/4)
dev development setdev/4canonical100%100.0% (4/4)50.0% (2/4)50.0% (2/4)
eval evaluation seteval/16raw100%68.75% (11/16)50.0% (8/16)50.0% (8/16)
eval evaluation seteval/16canonical100%56.25% (9/16)37.5% (6/16)37.5% (6/16)

The structured representation did not improve accuracy on WW. The LLM tended to attribute errors to the planner or decision-maker when using structured trajectories, whereas with less organized trajectories it leaned toward the part where the actual error appeared. In fact, among the 8/10 questions the LLM answered incorrectly, only one showed the multi-perspective problem I had expected; more often, the model seemed to lose its way in the long context and hallucinate or make an incorrect judgment.

MP-Bench

Different analysts may reasonably attribute a failure to different steps. This non-deterministic, multi-perspective nature arises from the interdependencies inherent in multi-agent systems. Multiple specialized agents can be combined interdependently into diverse, valid execution trajectories aimed at the same goal. Failure attribution therefore depends heavily on an observer’s internal reasoning about what constitutes a “correct” trajectory.

The limitations observed in previous work stem from biased evaluation assumptions that overlook the multi-perspective nature of MAS failures, rather than from insufficient model capabilities.

Figure 2. An example illustrating the multi-perspective nature of failure attribution in multi-agent systems.
Figure 2. An example illustrating the multi-perspective nature of failure attribution in multi-agent systems.

MP-Bench uses normalized discounted cumulative gain (nDCG@K), which evaluates agreement between the predicted and ground-truth rankings and gives more weight to correctly prioritizing highly salient failure steps.

Table 4. Failure attribution and attribution reasoning performance on MP-Bench
Table 4. Failure attribution and attribution reasoning performance on MP-Bench

The paper also evaluates a multi-agent setting:

Table 5. Single-LLM and multi-LLM failure attribution systems
Table 5. Single-LLM and multi-LLM failure attribution systems

TrajErrBench

Some existing datasets have the following limitations:

  • Trajectories tend to be short: The algorithm-generated subset of Who&When averages only 8.72 steps, making it difficult to test the loss of evidence over long contexts.
  • Scenarios can be narrow or toy-like: Many focus on simple multi-agent conversations, web shopping (WebShop), or text-based room games (ALFWorld).
  • A lack of realistic complex settings: They offer limited coverage of complex multi-turn tool use with customer-service business rules and very long-horizon repository-level software development.

TrajErrBench consists of two complementary, challenging real-world subsets:

Table 6. Composition of TrajErrBench

SubsetTrajectoriesAverage stepsTask setting and characteristicsLLMs generating failed trajectories
$\tau^{2}$-Bench subset40029.27Multi-turn customer service and tool use in airline and retail settings: the agent gathers requirements through dialogue, calls backend APIs, and follows business policies such as change and refund rules, budgets, and seat-class restrictions.DeepSeek-Reasoner, GPT-5, Qwen3-Max
SWE-Bench Pro subset86119.70Long-horizon repository-level software engineering: the agent locates bugs, reads source files, writes patches, and runs tests in real GitHub repositories.OpenHands with Claude-Opus-4.6, Kimi-K2.6-Coding, Gemini-3.1-Pro, Grok-4.20-Beta, GPT-5.4
Total486—Real tool interaction and long-horizon code repair8 frontier models
Table 7. Main critical-error detection results across multiple benchmarks
Table 7. Main critical-error detection results across multiple benchmarks

The authors observed the following in this dataset:

  1. Critical errors often occur in the latter half of a trajectory. In previous datasets, agents often make decisive errors within the first few steps. In this dataset, agents spend much of the earlier trajectory analyzing and interpreting before they make a critical error while making key decisions or modifying code.
  2. Agents often fail during reasoning: this accounts for 60.7% of cases in $\tau^{2}$-Bench and 57.1% in SWE-Bench Pro.

Comments