acceptodds
Under review as a conference paper at ICLR 2027

RAPO: Responsibility Attribution-Driven Prompt Optimization for LLM-Based Multi-Agent Systems

Abstract

LLM-based multi-agent systems show strong potential for complex collaborative tasks, but their performance depends heavily on the design and optimization of role prompts. Existing optimization methods typically rely on label-free feedback and randomly sampled task cases, which can introduce misaligned signals and agent-irrelevant noise, leading to unstable prompt updates. We propose a Responsibility Attribution-driven Prompt Optimization (RAPO) framework for multi-agent systems. Its two key designs—replacing unreliable LLM self-evaluation with gold-answer guidance and constructing agent-specific sample sets via error attribution—jointly enable stable prompt updates with smaller sample budgets. A formal bias–variance analysis justifies their complementary roles. Extensive experiments on code generation, mathematical reasoning, and scientific QA show that our method outperforms the state-of-the-art MASPO, and ablation studies confirm each component’s contribution. Cross-backbone evaluations further demonstrate the stability and generality of the approach.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Multi-Agent Systems, Prompt Optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Ht2bOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes RAPO, a prompt-optimization framework for fixed-topology LLM-based multi-agent systems. RAPO introduces two main components: gold-answer-guided evaluation of candidate prompt updates, and LLM-based responsibility attribution that assigns failed trajectories to the agents considered responsible for the failure. Each agent is then optimized using a small set of attributed failure cases together with solved examples.

The method is evaluated on mathematical reasoning, scientific QA, and code-generation benchmarks under sequential and hierarchical multi-agent architectures. The paper reports improvements over MASPO, together with ablations on gold supervision, attribution-based sample selection, optimization stability, and cross-backbone generalization.

Critical strengths
  1. The paper addresses a practical issue in multi-agent prompt optimization: failures observed at the system level are not necessarily informative for every agent. Using agent-specific failure samples is a sensible direction.

  2. The experimental section is reasonably broad. The authors evaluate two MAS topologies, six benchmarks, multiple optimizer/generator configurations, and provide useful analyses of optimization stability and sample composition.

Critical weaknesses
  1. The main comparison does not use a matched supervision setting, making the central performance claim difficult to interpret.

RAPO uses gold answers in two important places: to directly evaluate whether candidate prompt updates improve final task correctness, and to provide supervision to the failure-attribution model. In contrast, the primary baseline MASPO is explicitly designed around label-free feedback. Therefore, the comparison RAPO > MASPO simultaneously changes both the optimization algorithm and the amount of supervision available to the optimizer.

The LABEL ablation partially addresses this issue, but I do not think it is sufficient to support the broader claim that RAPO is a stronger multi-agent prompt optimizer. In particular, much of the improvement over MASPO can plausibly come from replacing a noisy proxy objective with direct task supervision, which is expected when gold labels are available. The paper should therefore frame the contribution as a supervised prompt-optimization setting rather than directly positioning the result as a general improvement over label-free methods.

  1. The central responsibility-attribution mechanism is not directly validated.

The paper relies on an LLM to identify which agents are responsible for a failed trajectory, but there is no independent evaluation showing that these responsibility assignments are actually correct. I did not find human annotations, controlled synthetic failures, counterfactual interventions, or another attribution-quality metric.

This is especially important because responsibility attribution is the central novelty of the paper. The current experiments only show that samples selected by the attribution model can sometimes be more useful for prompt optimization. This does not establish that the selected agents are genuinely responsible for the failure.

The existing sample-composition analysis also does not fully isolate this effect. "Attributed Errors Only" improves only modestly over "Random Errors Only", and the most important matched control appears to be missing: Random Errors + Correct Samples under the same k=4 budget. Without this control, it is difficult to separate the benefit of responsibility attribution from the benefit of the overall sample composition and regularization strategy.

More broadly, the theoretical variance argument assumes that attribution reduces the probability of agent-irrelevant failures entering the optimization set. This is essentially the property that the empirical evaluation needs to establish, rather than an independently justified result.

Questions for authors
  1. How accurate is the responsibility-attribution module itself? Do the authors have any independent validation, such as human-labeled responsible agents, synthetic failures with known causes, or counterfactual interventions on the attributed agents?

  2. Could the authors provide or clarify the matched comparison "Random Errors + Correct Samples" with k=4? This seems necessary to isolate the value of attribution from the effect of including solved examples and using a smaller optimization batch.

  3. How should the main comparison with MASPO be interpreted given that RAPO has access to gold answers for candidate evaluation and attribution while MASPO is label-free? In my view, these methods operate under different supervision assumptions.

  4. The theoretical variance result assumes that attribution indeed reduces the probability of irrelevant failures entering the optimization set. Is there empirical evidence in the current experiments that directly estimates or validates this assumption?

Reviewer_TTnXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes RAPO for improving the prompts of individual agents in an LLM-based multi-agent system. It uses reference answers and an LLM to identify which agents contributed to responses that failed, then combines these agent-specific failures with successful examples to generate improvements. Candidate prompts are evaluated for correctness, successor compatibility, and prompt quality, with a regression penalty (for turning an originally correct sample into a wrong). Experiments show higher average accuracy than MASPO while using fewer examples per prompt update (4 to 10)

Critical strengths

In multi agent set ups, One agent may introduce an error, while another may correctly perform its assigned role, hence a wrong final answer does not mean that every agent made a mistake. RAPO uses responsibility attribution to select failures relevant to each agent, providing a sensible way to make more targeted optimization. Therefore, the paper addresses a meaningful problem in multi-agent optimization. In multi agent set up, Improving a prompt on failed examples can sometimes make it worse on examples it previously handled correctly. RAPO combines attributed failures with successful examples and applies a regression penalty when a prompt change turns a correct answer into an incorrect one..so it balances correcting failures with preserving successful behavior An agent’s response may look reasonable on its own but be difficult for the next agents to use. RAPO evaluates candidate prompts using final-answer correctness, successor compatibility, and prompt quality. This recognizes that a useful prompt should help both the individual agent and the overall system. Therefore, the team context is considered in the proposed method

Critical weaknesses

Table 6 does not compare attributed failures plus positive references against randomly selected failures plus positive references. Therefore, it remains unclear how much improvement comes specifically from error attribution.The contribution of responsibility-aware sampling is not fully distinguished An LLM identifies the agents responsible for failed responses, but the paper does not directly evaluate whether these judgments are correct. The sample-selection statistics show which failures are selected, but do not establish that they are relevant to the responsible agent. We cannot confirm reliability. RAPO uses four samples per prompt update, compared with ten for MASPO, but both use the same training set of 50 examples. Table 9 also shows higher average optimization time for RAPO. Evaluation needs better and more clear reporting Variance assumptions impact is not verified in practice, theoretical analysis depends on them

Questions for authors

Have you compared attributed failures plus positive references against randomly selected failures plus positive references, using the same number and proportions of samples? What supports the accuracy of the attribution LLM’s judgments? Have you double checked whether the identified agents actually contributed to the failures? Can you report the standard deviation or individual results across the three random seeds for RAPO, MASPO, and LABEL? What evidence supports the variance assumption?

Reviewer_UksjOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper exmaines how to optimize the role prompt in a multi-agent LLm system with limited labels. They propose RAPO, a framework runs the system on labeled examples, identifies incorrect trajectories, and uses an attribution LLM to assign failures to responsible agents. Each agent’s prompt is optimized using its attributed failures together with successful examples. Experiment results show that RAPO can improve average accuracy over MASPO over various benchmarks.

Critical strengths

The major strength of the paper, to my understanding, is that it can make prompt optimzation more targeted by connecting failure trajectories to specific agent. The system selects failures attributed to the agent being optimized and includes successful examples to preserve useful behavior. This solves a practical problem where an agent might be wrong because it receives incorrect messages.

Critical weaknesses

One weakness I found is that authors do not tested explicitly how realiable is the attribution mechnism. In other words, the experiments, to my understanding, does not establish attribution accuracy against independently annotated responsibility labels or controlled interventions.

Questions for authors

Can we independently test the reliability of the attribution system?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.