- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes MASRubric, a framework for auditing intermediate messages in multi-agent systems to mitigate error propagation. Offline, it automatically distills failed execution trajectories into reusable pitfall criteria with diagnostic definitions, applicability conditions, and inspection directives. At inference time, it retrieves relevant criteria, evaluates each message, and decides whether to pass, revise, or withhold it. Experiments across fixed and dynamic multi-agent frameworks demonstrate improvements in average accuracy on mathematical reasoning and code generation benchmarks.
Critical strengths
The strongest contribution is the conversion of historical failures into reusable, context-specific auditing criteria. The construction, retrieval, and intervention pipeline provides concrete checks without requiring reference answers or manually authored criteria at test time. The transfer experiment further supports the bank’s practical value: criteria mined from a Qwen3-8B system are reused without modification in a Qwen3-14B system, improving average mathematical accuracy from 52.56% to 56.29%.
Critical weaknesses
The distinction from existing iterative feedback-and-refinement methods, particularly the contribution specific to multi-agent information flow, is not sufficiently established. The online procedure follows the familiar generate–evaluate–feedback–revise loop, while criterion retrieval and auditing could also be applied to a single agent’s intermediate reasoning. Consequently, the results do not clearly isolate improvements from more informative feedback versus control of inter-agent communication. The generic-criterion baseline provides partial evidence for the specialized bank, but the paper lacks a single-agent baseline using the same bank under a matched compute budget, as well as an ablation that disables message withholding while retaining revision. These comparisons would clarify the incremental contribution over Self-Refine-style methods and establish the independent value of the information-flow control mechanism.
Questions for authors
None
8 comments
Kind Request for Feedback
Dear Reviewer 7EMW,
Thank you again for your review. As a brief summary of our response, we ran the two experiments you asked for: (i) a single agent equipped with the same bank and auditor, also at a token budget matched to MASRubric, which reaches 50.57 against 53.69 for MASRubric in the MAS (W1); and (ii) an ablation that keeps revision but disables withholding, in which the macro-average falls from 53.69 to 50.66, the level of Dynamic-MAS without auditing (W2). Together they indicate that the gain comes from revision and withholding acting together, and not from a feedback-and-refinement loop alone. We understand that this is a busy period. If time permits, we would be grateful to hear whether these results address your concerns, so that we can use the rest of the discussion period to clarify anything that remains open.
Best regards, The authors
Kind Reminder Before the Discussion Period Closes
Dear Reviewer 7EMW,
As the discussion period closes today, we would be very grateful for a brief note on whether our response resolves your concerns. And if you have any further questions, we would be glad to answer them. Thank you again for your time.
Best regards, The authors
Response to Reviewer 7EMW
We attached MASRubric's online stage unchanged to a single Qwen3-8B agent. The final answer is the agent's last version, since there is no other agent from which a message could be withheld. One pass of this loop uses 0.14× the tokens of MASRubric. To match the budget, we also let the agent repeat the loop for eight rounds per problem (1.02× the tokens of MASRubric): after each round it either addresses the audit feedback or, if its solution passed, is asked to re-check it, and every new version is audited again.
The identical retrieve–audit–revise loop brings no improvement to the single agent (49.93 vs. 49.92). With the budget matched, the single agent reaches 50.57, still below Dynamic-MAS without any auditing (50.86) and 3.1 points below MASRubric. It catches up with MASRubric on the easy tier (81.0 vs. 81.3) but not on the hard tier (26.2 vs. 31.6). In the MAS, the same bank and auditor add +2.83 points.
On the single agent, the audit does detect errors (a solution that fails its first audit is correct in only 22% of the cases, against 77% for one that passes), but revision can repair only part of them, and the agent has to return the solutions that still fail, 81% of them wrong, whereas a MAS withholds such messages from the other agents. MASRubric's gain therefore comes from revision and withholding acting together, which is the information-flow control the reviewer asks about: revision corrects part of the flagged errors, and withholding keeps the rest from propagating. A feedback-and-refinement loop alone does not produce a gain of this size, on a single agent or in the MAS (see W2); Self-Refine, at 2.76× the vanilla budget, even lowers Dynamic-MAS from 50.86 to 48.89 (Table 1). We therefore expect MASRubric to be most useful in larger and more complex systems, where a wrong message can mislead more agents. We will add this experiment to the paper.
We ran Dynamic-MAS + MASRubric with withholding switched off: retrieval, auditing, feedback and up to three revisions are unchanged, but a message that still fails its last audit is broadcast to the other agents instead of being withheld.
The audit itself behaves as in the full method: it requests a revision for 24% of the messages (23% in the Table 1 run), and 1.4% of the messages still fail after their last revision, the same share that the full method withholds. These messages, now delivered to the other agents, are the only difference between the two variants, and 75% of them carry a wrong final answer. Delivering them removes the improvement: the macro-average falls to 50.66, the level of Dynamic-MAS without any auditing (50.86), at the full MASRubric token budget, against 53.69 with withholding. Revision alone therefore does not reproduce the gain, but it remains part of the mechanism: it corrects part of the flagged messages before they are broadcast and leaves only a small residue to be withheld. Together with the single-agent result in W1, this shows that the gain comes from revision and withholding acting together, and that the feedback-and-revision loop alone does not produce a gain of this size. We will add this ablation to the paper.