- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes RAPO, a prompt-optimization framework for fixed-topology LLM-based multi-agent systems. RAPO introduces two main components: gold-answer-guided evaluation of candidate prompt updates, and LLM-based responsibility attribution that assigns failed trajectories to the agents considered responsible for the failure. Each agent is then optimized using a small set of attributed failure cases together with solved examples.
The method is evaluated on mathematical reasoning, scientific QA, and code-generation benchmarks under sequential and hierarchical multi-agent architectures. The paper reports improvements over MASPO, together with ablations on gold supervision, attribution-based sample selection, optimization stability, and cross-backbone generalization.
Critical strengths
-
The paper addresses a practical issue in multi-agent prompt optimization: failures observed at the system level are not necessarily informative for every agent. Using agent-specific failure samples is a sensible direction.
-
The experimental section is reasonably broad. The authors evaluate two MAS topologies, six benchmarks, multiple optimizer/generator configurations, and provide useful analyses of optimization stability and sample composition.
Critical weaknesses
- The main comparison does not use a matched supervision setting, making the central performance claim difficult to interpret.
RAPO uses gold answers in two important places: to directly evaluate whether candidate prompt updates improve final task correctness, and to provide supervision to the failure-attribution model. In contrast, the primary baseline MASPO is explicitly designed around label-free feedback. Therefore, the comparison RAPO > MASPO simultaneously changes both the optimization algorithm and the amount of supervision available to the optimizer.
The LABEL ablation partially addresses this issue, but I do not think it is sufficient to support the broader claim that RAPO is a stronger multi-agent prompt optimizer. In particular, much of the improvement over MASPO can plausibly come from replacing a noisy proxy objective with direct task supervision, which is expected when gold labels are available. The paper should therefore frame the contribution as a supervised prompt-optimization setting rather than directly positioning the result as a general improvement over label-free methods.
- The central responsibility-attribution mechanism is not directly validated.
The paper relies on an LLM to identify which agents are responsible for a failed trajectory, but there is no independent evaluation showing that these responsibility assignments are actually correct. I did not find human annotations, controlled synthetic failures, counterfactual interventions, or another attribution-quality metric.
This is especially important because responsibility attribution is the central novelty of the paper. The current experiments only show that samples selected by the attribution model can sometimes be more useful for prompt optimization. This does not establish that the selected agents are genuinely responsible for the failure.
The existing sample-composition analysis also does not fully isolate this effect. "Attributed Errors Only" improves only modestly over "Random Errors Only", and the most important matched control appears to be missing: Random Errors + Correct Samples under the same k=4 budget. Without this control, it is difficult to separate the benefit of responsibility attribution from the benefit of the overall sample composition and regularization strategy.
More broadly, the theoretical variance argument assumes that attribution reduces the probability of agent-irrelevant failures entering the optimization set. This is essentially the property that the empirical evaluation needs to establish, rather than an independently justified result.
Questions for authors
-
How accurate is the responsibility-attribution module itself? Do the authors have any independent validation, such as human-labeled responsible agents, synthetic failures with known causes, or counterfactual interventions on the attributed agents?
-
Could the authors provide or clarify the matched comparison "Random Errors + Correct Samples" with k=4? This seems necessary to isolate the value of attribution from the effect of including solved examples and using a smaller optimization batch.
-
How should the main comparison with MASPO be interpreted given that RAPO has access to gold answers for candidate evaluation and attribution while MASPO is label-free? In my view, these methods operate under different supervision assumptions.
-
The theoretical variance result assumes that attribution indeed reduces the probability of irrelevant failures entering the optimization set. Is there empirical evidence in the current experiments that directly estimates or validates this assumption?