acceptodds
Under review as a conference paper at ICLR 2027

MASRubric: Auditing Information Flow in Multi-Agent Systems with Failure-Distilled Pitfall Rubrics

Abstract

While multi-agent systems (MAS) excel at complex reasoning, they are vulnerable to errors that intermediate agents introduce and downstream agents build upon. Auditing intermediate messages before they propagate requires an explicit standard, yet evaluation rubrics are typically authored by domain experts or written against a reference answer, neither of which is available for an unseen message at test time. We present MASRubric, a MAS information flow auditing framework with failure-distilled pitfall rubrics. Offline, trajectories on which the MAS has failed are automatically distilled into a reusable bank of pitfall criteria, each describing a recurrent error by its underlying misconception, the reasoning situations in which it arises, and the check that would expose it. Online, the criteria applicable to each intermediate message are retrieved from this off-the-shelf bank and checked one by one, and the resulting satisfaction rate decides whether the message is broadcast, returned to its author with diagnostic feedback for revision, or withheld. Empirical results demonstrate that MASRubric enhances MAS performance on both fixed and dynamic frameworks, achieving average accuracy gains of 2.83 and 1.74 percentage points on math reasoning and code generation benchmarks, respectively. Further analysis shows that the retrieved criteria vary systematically with task types, and that the audit effort tracks task difficulty. Moreover, the bank transfers without re-mining to a system with a stronger backbone, which makes more adaptive and more efficient use of it. Our code and dataset are anonymously released at https://anonymous.4open.science/r/MASRubric-6137.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, LLM-Based Agents, Multi-Agent Systems, Test Time Intervention, Rubric-Based Evaluation, LLM-as-a-Judge

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_7EMWOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes MASRubric, a framework for auditing intermediate messages in multi-agent systems to mitigate error propagation. Offline, it automatically distills failed execution trajectories into reusable pitfall criteria with diagnostic definitions, applicability conditions, and inspection directives. At inference time, it retrieves relevant criteria, evaluates each message, and decides whether to pass, revise, or withhold it. Experiments across fixed and dynamic multi-agent frameworks demonstrate improvements in average accuracy on mathematical reasoning and code generation benchmarks.

Critical strengths

The strongest contribution is the conversion of historical failures into reusable, context-specific auditing criteria. The construction, retrieval, and intervention pipeline provides concrete checks without requiring reference answers or manually authored criteria at test time. The transfer experiment further supports the bank’s practical value: criteria mined from a Qwen3-8B system are reused without modification in a Qwen3-14B system, improving average mathematical accuracy from 52.56% to 56.29%.

Critical weaknesses

The distinction from existing iterative feedback-and-refinement methods, particularly the contribution specific to multi-agent information flow, is not sufficiently established. The online procedure follows the familiar generate–evaluate–feedback–revise loop, while criterion retrieval and auditing could also be applied to a single agent’s intermediate reasoning. Consequently, the results do not clearly isolate improvements from more informative feedback versus control of inter-agent communication. The generic-criterion baseline provides partial evidence for the specialized bank, but the paper lacks a single-agent baseline using the same bank under a matched compute budget, as well as an ablation that disables message withholding while retaining revision. These comparisons would clarify the incremental contribution over Self-Refine-style methods and establish the independent value of the information-flow control mechanism.

Questions for authors

None

8 comments
AuthorsOpenReview ↗

Response to Reviewer 7EMW

W1. The distinction from feedback-and-refinement loops is not established. The paper lacks a single-agent baseline using the same bank under a matched compute budge.

We attached MASRubric's online stage unchanged to a single Qwen3-8B agent. The final answer is the agent's last version, since there is no other agent from which a message could be withheld. One pass of this loop uses 0.14× the tokens of MASRubric. To match the budget, we also let the agent repeat the loop for eight rounds per problem (1.02× the tokens of MASRubric): after each round it either addresses the audit feedback or, if its solution passed, is asked to re-check it, and every new version is audited again.

SystemGSM8KMATH-500AQuAAMC23OlymBAIME24AIME25OlymEOlymHAvg.
Single Agent88.1079.8086.2265.0047.8523.3320.0022.0017.0049.92
Single Agent + MASRubric bank and auditor87.5779.8082.6860.0047.7030.0016.6730.0015.0049.93
Single Agent + MASRubric bank and auditor, matched budget (8 rounds)90.4582.0086.6165.0049.0426.6713.3325.0017.0050.57
Dynamic-MAS (paper)91.6678.0085.4362.5048.1530.0020.0026.0016.0050.86
Dynamic-MAS + MASRubric (paper)91.6679.6083.8670.0052.4430.0026.6732.0017.0053.69

The identical retrieve–audit–revise loop brings no improvement to the single agent (49.93 vs. 49.92). With the budget matched, the single agent reaches 50.57, still below Dynamic-MAS without any auditing (50.86) and 3.1 points below MASRubric. It catches up with MASRubric on the easy tier (81.0 vs. 81.3) but not on the hard tier (26.2 vs. 31.6). In the MAS, the same bank and auditor add +2.83 points.

On the single agent, the audit does detect errors (a solution that fails its first audit is correct in only 22% of the cases, against 77% for one that passes), but revision can repair only part of them, and the agent has to return the solutions that still fail, 81% of them wrong, whereas a MAS withholds such messages from the other agents. MASRubric's gain therefore comes from revision and withholding acting together, which is the information-flow control the reviewer asks about: revision corrects part of the flagged errors, and withholding keeps the rest from propagating. A feedback-and-refinement loop alone does not produce a gain of this size, on a single agent or in the MAS (see W2); Self-Refine, at 2.76× the vanilla budget, even lowers Dynamic-MAS from 50.86 to 48.89 (Table 1). We therefore expect MASRubric to be most useful in larger and more complex systems, where a wrong message can mislead more agents. We will add this experiment to the paper.

W2. The paper also lacks an ablation that disables message withholding while retaining revision.

We ran Dynamic-MAS + MASRubric with withholding switched off: retrieval, auditing, feedback and up to three revisions are unchanged, but a message that still fails its last audit is broadcast to the other agents instead of being withheld.

SystemGSM8KMATH-500AQuAAMC23OlymBAIME24AIME25OlymEOlymHAvg.
Dynamic-MAS (paper)91.6678.0085.4362.5048.1530.0020.0026.0016.0050.86
+ MASRubric, revision only (no withholding)91.1382.2083.0772.5051.7030.0013.3316.0016.0050.66
+ MASRubric (paper)91.6679.6083.8670.0052.4430.0026.6732.0017.0053.69

The audit itself behaves as in the full method: it requests a revision for 24% of the messages (23% in the Table 1 run), and 1.4% of the messages still fail after their last revision, the same share that the full method withholds. These messages, now delivered to the other agents, are the only difference between the two variants, and 75% of them carry a wrong final answer. Delivering them removes the improvement: the macro-average falls to 50.66, the level of Dynamic-MAS without any auditing (50.86), at the full MASRubric token budget, against 53.69 with withholding. Revision alone therefore does not reproduce the gain, but it remains part of the mechanism: it corrects part of the flagged messages before they are broadcast and leaves only a small residue to be withheld. Together with the single-agent result in W1, this shows that the gain comes from revision and withholding acting together, and that the feedback-and-revision loop alone does not produce a gain of this size. We will add this ablation to the paper.

AuthorsOpenReview ↗

Kind Request for Feedback

Dear Reviewer 7EMW,

Thank you again for your review. As a brief summary of our response, we ran the two experiments you asked for: (i) a single agent equipped with the same bank and auditor, also at a token budget matched to MASRubric, which reaches 50.57 against 53.69 for MASRubric in the MAS (W1); and (ii) an ablation that keeps revision but disables withholding, in which the macro-average falls from 53.69 to 50.66, the level of Dynamic-MAS without auditing (W2). Together they indicate that the gain comes from revision and withholding acting together, and not from a feedback-and-refinement loop alone. We understand that this is a busy period. If time permits, we would be grateful to hear whether these results address your concerns, so that we can use the rest of the discussion period to clarify anything that remains open.

Best regards, The authors

AuthorsOpenReview ↗

Kind Reminder Before the Discussion Period Closes

Dear Reviewer 7EMW,

As the discussion period closes today, we would be very grateful for a brief note on whether our response resolves your concerns. And if you have any further questions, we would be glad to answer them. Thank you again for your time.

Best regards, The authors

Reviewer_A3xxOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MASRubric, a framework for auditing intermediate messages in multi-agent systems. It first distills recurring failure patterns from unsuccessful training trajectories into a reusable bank of pitfall criteria. At test time, relevant criteria are retrieved for each agent message and checked by an auditor, which either accepts the message, requests a revision with diagnostic feedback, or withholds it. Experiments on mathematical reasoning and code generation show consistent but moderate accuracy improvements across fixed and dynamic multi-agent frameworks.

Critical strengths
  • The paper addresses the important problem of preventing intermediate errors from propagating through multi-agent workflows.

  • The failure-distilled rubric bank is intuitive and actionable, turning previous failures into reusable diagnostic checks without requiring reference answers at test time.

  • The method is evaluated across two multi-agent frameworks and thirteen benchmarks covering both mathematics and code generation.

  • The paper provides useful ablations, transfer experiments, and detailed token accounting, making the benefits and computational costs relatively transparent.

Critical weaknesses
  • The limited novelty, this paper may like "use stronger model do summarization, retrieval by embedding, llm judge" all of these parts are some existing method.

  • The core claim of this paper is prevent errors pass to following agent, but the main experiments only report the accuracy, which should report whether auditor prevent the middle step error from passing.

  • Author should use human double check to see whether the rubric bank is reliable.

  • The temperature of model in the experiments is 0.7, which has randomness. Author should calculate average acc, pass@n and standard deviation to see the effectiveness of method.

  • The trade-off of cost and accuracy is imbalanced, the token increase from 76M to 200M but the accuracy only improve 3%

Questions for authors

See weaknesses.

4 comments
AuthorsOpenReview ↗

Response to Reviewer A3xx (Part 1/2)

W1. The limited novelty, this paper may like "use stronger model do summarization, retrieval by embedding, llm judge" all of these parts are some existing method.

We agree that each component, taken alone, is a known technique; the contribution lies in what is stored and where it acts. The bank does not store summaries or answers but pitfall criteria with explicit applicability conditions, so a rubric can be assembled for an unseen message without a reference answer. The audit is applied to every inter-agent message before it is broadcast, and its verdict decides whether the message is passed on, revised or withheld. The controls below show that the three ingredients, used the way existing methods use them, do not produce the improvement (average accuracy over the nine math benchmarks; Dynamic-MAS 50.86, MASRubric 53.69):

IngredientControlAvg.
Summaries retrieved by embeddingSame failures, same teacher and same retrieval, given as guidance before generation (AutoGuide-style; Fu et al., NeurIPS 2024)50.76
LLM judgeAudit without the bank: generic criterion / RaR-Gen / Self-Refine52.35 / 51.94 / 48.89
All three, outside the multi-agent message flowSame bank and auditor on a single agent49.93
All three, without withholdingAudit and revision only, failed messages still broadcast50.66

The same supervision delivered as retrieved guidance brings no gain; an LLM judge without failure-mined criteria gains at most about half as much, or loses accuracy; and the full retrieve–audit–revise loop helps neither a single agent nor a MAS in which failed messages are still broadcast. The improvement appears only when failure-mined criteria are used to audit each message and to keep failed ones from the other agents, which is the mechanism we propose against errors that cascade through a MAS. For the details of these controls, please see Tables 3 and 8 of the paper and our replies to Reviewer zweD (W4) and Reviewer 7EMW (W1, W2).

Reference

  • Fu, Yao, et al. "Autoguide: Automated generation and selection of context-aware guidelines for large language model agents." Advances in Neural Information Processing Systems 37 (2024): 119919-119948.

W2. The core claim of this paper is prevent errors pass to following agent, but the main experiments only report the accuracy, which should report whether auditor prevent the middle step error from passing.

We agree that accuracy alone does not show this, so we traced every intermediate message of the MASRubric run behind Table 1 (Dynamic-MAS, Qwen3-8B, nine math benchmarks, 18,292 agent messages). Each message's initial output and each revision were graded against the gold answer, and withheld messages were tracked.

Wrong intermediate messagesCorrect intermediate messages
Initial outputs of the agents4,76713,392
Flagged by the auditor1,679 (35.2%)954 (7.1%)
Corrected by revision / spoiled by revision135 corrected105 spoiled (0.8%)
Withheld from the other agents20543
Delivered to downstream agents4,561 (−4.3%)13,389

So 340 wrong intermediate outputs (135 rewritten into correct ones, 205 withheld) never reached the following agents in their wrong form, at the cost of 148 correct outputs (1.1%). The audit is selective: 64.7% of flagged messages are wrong or have no final answer, against a 26.7% base rate, and 80% of withheld messages are wrong. On GSM8K, 29% of the wrong messages the auditor flags are corrected before they are broadcast. We will report these message-level numbers alongside accuracy in the paper.

W3. Author should use human double check to see whether the rubric bank is reliable.

We carried out this check. Two annotators independently examined 50 criteria sampled from the bank: 12 drawn from the 30 criteria that are selected most often in the run behind Table 1, and 38 drawn uniformly at random from the rest. From the text of each criterion they judged three properties: whether the mathematical or logical content it asks the auditor to check is correct, whether one can decide if a given message violates it, and whether it describes a reusable pitfall rather than a single problem or its answer.

Criteria accepted (of 50)Annotator AAnnotator BBothAgreement
Content is correct48464696%
Violation is decidable47484798%
Reusable, no problem-specific answer48504896%
All three45464494%

Both annotators accept 44 of the 50 criteria (88%) on all three properties, and their overall verdicts agree on 47 (Cohen's κ\kappa = 0.63). The check therefore supports the reliability of the automatically mined bank: each of the three properties is confirmed by both annotators for more than 90% of the criteria, with substantial agreement between them.

AuthorsOpenReview ↗

Response to Reviewer A3xx (Part 2/2)

W4. Author should calculate average acc, pass@n and standard deviation to see the effectiveness of method.

We agree and repeated every Dynamic-MAS configuration three times under the paper's setting (reasoners and selector at temperature 0.7, auditor at temperature 0). The table gives the mean accuracy over the three runs on each benchmark and the average with its standard deviation:

Mean of 3 runsGSM8KMATH-500AQuAAMC23OlymBAIME24AIME25OlymEOlymHAvg. (mean ± s.d.)
Dynamic-MAS91.5379.4084.2570.0047.7027.7823.3323.6718.0051.74 ± 0.78
+ Multi-TAG91.6979.8785.9665.8349.3824.4418.8927.0017.0051.12 ± 0.96
+ Self-Refine92.1781.9385.4365.8349.2823.3312.2222.6715.0049.76 ± 1.11
+ PRM93.7383.2786.8867.5051.6527.7820.0025.3316.0052.46 ± 1.15
+ RaR-Gen91.6481.6784.9171.6750.3725.5625.5628.6713.6752.63 ± 1.22
+ MASRubric91.7979.0085.0472.5051.7528.8926.6730.6717.6753.77 ± 0.28

The +2.0-point average gain of MASRubric over Dynamic-MAS is larger than the sum of the two standard deviations, and MASRubric has the smallest run-to-run variance of all configurations. We report mean ± s.d. rather than pass@n because every configuration produces a single final answer per problem; pass@n over repeated runs would mostly measure the sampling budget rather than the auditing mechanism.

W5. The trade-off of cost and accuracy is imbalanced, the token increase from 76M to 200M but the accuracy only improve 3%

The gain should be read against what the same extra compute buys otherwise. Table 8 lists the ways of adding computation to the same Dynamic-MAS:

Added to Dynamic-MASBudget (× Dynamic-MAS)Avg. gain
Self-Refine2.76×−1.97
PRM2.08×+1.08
RaR-Gen1.90×+1.08
MASRubric2.63×+2.83

Spending more does not by itself buy accuracy: Self-Refine is the most expensive and loses accuracy, and a majority vote over independent vanilla runs at exactly MASRubric's budget reaches 53.06 against 53.69 (please see our reply to Reviewer zweD, W6). Per accuracy point, MASRubric spends 44M additional tokens, against 63M for RaR-Gen and 77M for PRM. The average also understates where the gain lies: the easy benchmarks are close to saturation, and on the hard tier the average rises from 28.03 to 31.62 (Table 1). Finally, all additional tokens are generated by the same local Qwen3-8B model that runs the agents, about 80% of them by the auditor, and no stronger model or API is called at test time.

Reviewer_A3xxOpenReview ↗

Reply of Rebuttal

Thanks for authors' reply, but my concern is not fully solved.

  • For W1, the reply only show the effectiveness of the components, which doesn't show the novelty of them. Please compare with [1], [2], [3].

  • In W5, author only shows that comparing with baseline and shows that it better than the basline method, but they didn't reply my question, although better than other method, whether it is really deserve in the deployment. You budget-matched voting in W5 and experiments in W2 both show that. In W5, increase 1.95 acc need to pay 130M extra tokens. In W2, such huge amount of cost only flag 35.2% wrong intermediate messages and correct 7.1% intermediate messages.

So, I will maintain my scores and I think these are the core flaws, if author can not solve it, I will not raise my rate.

[1] https://aclanthology.org/2026.emnlp-main.2179/ [2] https://arxiv.org/abs/2605.09076 [3] https://arxiv.org/abs/2506.00641

AuthorsOpenReview ↗

Response to Reviewer A3xx

For W1, the reply only show the effectiveness of the components, which doesn't show the novelty of them. Please compare with [1], [2], [3].

We thank the reviewer for the three pointers. Like MASRubric, all three act on messages or agent outputs, but they differ from it in the standard used to judge a message and in what the judgment is used for:

MethodStandard for judging a messageWhere it actsWhat happens to the messageGoal
MessageSieve [1]None: lexical similarity, freshness and channel cues, without an LLM callAt the receiver, when its prompt is assembledKept in full, reduced to evidence spans, or omittedLess redundant context and fewer tokens
SAC [2]None beyond the receiver itself: each agent re-solves the task and gives every received answer one overall scoreAt the receiver, after the answers are exchangedLow-scored answers are dropped locally by the receiverConsensus when some agents are faulty or adversarial
AgentAuditor [3]Recorded interactions with human-annotated safety labels, retrieved as few-shot casesAfter the run, on a recorded agent interactionNothing; the record is labelled safe or unsafeEvaluating agent safety
MASRubricPitfall criteria mined from failed runs of the MAS, selected for each message by their applicability conditionsBefore the message is broadcast to any agentPassed on, or returned to its sender with criterion-level feedback for revision, and withheld if it still failsStopping reasoning errors before they propagate

MessageSieve decides how much of a message a receiver reads, not whether the message is correct; the paper states that semantic verification is outside the scope of its routing cues. SAC does judge correctness, but its only standard is the scoring agent's own solution, so its robustness depends on the agents' base capability, as its authors note. In our setting, an auditor that judges without the bank gains about half as much as MASRubric (the generic-criterion row of Table 3). AgentAuditor is the closest to our offline stage, because it also retrieves stored experience for an LLM judge, but it stores labelled cases rather than reusable criteria, and it evaluates a finished record instead of intervening in a running system.

MASRubric therefore differs from all three in two respects. Its standard is a bank of criteria mined automatically from failed runs of the MAS, each with an explicit applicability condition, so a rubric can be assembled for an unseen message without a reference answer. And its verdict is issued before broadcast and criterion by criterion, so the sender can revise the message, and a message that still fails never reaches the other agents. [1] and [2] act online at the message level, but without an external standard of correctness; [3] has an experience-based standard, but does not act online. None of the three combines the two. The methods are also compatible: MessageSieve removes redundancy and could be applied to the messages that pass our audit. We will add the three works and this comparison to the related work.

For W5, the reply only shows that the method is better than the baselines, not whether it is worth deploying.

We agree that the cost is high for the gain. Token cost is the main limitation of MASRubric, and we will state it in a Limitations section. For tasks with a short answer that can be matched, on which the base system is already accurate, we do not recommend it: voting over independent runs comes within 0.6 points of MASRubric at the same budget (our reply to Reviewer zweD, W6) and is simpler.

Four observations indicate where it is worth deploying. First, on hard problems: about 55% of the extra tokens go to the five hard benchmarks, where the average rises from 28.03 to 31.62, and about 45% to the four easy ones, where it rises by 1.88 points. Second, where outputs cannot be matched for a vote: on the code benchmarks (Table 2), the voting baseline Multi-TAG lowers the average of Dynamic-MAS from 46.63 to 45.61, whereas MASRubric raises it to 48.37. Third, with a stronger backbone: on Qwen3-14B the same bank, without re-mining, gives +3.73 points at 29M tokens per point, against 44M on Qwen3-8B. Fourth, in absolute terms the overhead is about 41k tokens per problem on a locally served 8B model, with no training and no API call.

The audit indeed flags 35.2% of the wrong messages and stops 7.1% of them. This rate is bounded by the auditor, an 8B model that judges without a reference answer, and it bounds the gain in turn. About 40% of the extra tokens go to selecting the criteria rather than to the audit itself, which is the first place to reduce the cost.

Reviewer_zweDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

MASRubric audits messages before they pass between agents in an LLM multi-agent system. Offline, GPT-5.4 uses failed trajectories and reference solutions to derive pitfall criteria specifying a name, diagnosis, applicability condition and check. After deduplication, the bank contains 2,000 math criteria and 2,545 code criteria. At inference time, an auditor retrieves applicable criteria, checks the message, and accepts it, requests revision for up to 3 rounds, or withholds it. Experiments use Qwen3-8B reasoners and auditors in a fixed DAG and AutoGen SelectorGroupChat; the dynamic selector is Qwen3.5-9B. Reported macro-average gains are 1.59 and 2.83 points over nine math benchmarks and 1.74 points over four code benchmarks. Additional studies include ablations and transfer to Qwen3-14B. On the Dynamic-MAS math suite, total token use is 2.63 times the vanilla system's.

Critical strengths
  1. Auditing messages before they enter another agent's context is a useful choice of intervention point, because it need not assume a particular topology. Separating applicability conditions from the actual checks also makes the criteria easier to use. These design choices could carry over to other systems.

  2. The paper provides enough operational detail to scrutinize its claims, including construction and token costs, complete runtime distributions in Tables 6 to 8, and generic and randomly selected criterion controls. This transparency is valuable.

Critical weaknesses
  1. The robustness of the reported gains remains uncertain. The paper does not provide multi-seed uncertainty estimates; the auditor uses temperature 0, while the other model roles use 0.7. All Avg weights a 30-problem AIME set equally with the 1,319-problem GSM8K set. This is a valid macro-average, but the conclusions depend on the aggregation choice. Reconstructing problem-weighted totals from the rounded accuracies in Table 1 and sizes in Table 5 gives gains of 9 problems for Fixed-MAS and 45 out of 3,048 for Dynamic-MAS, including 29 on OlympiadBench. PRM solves more problems under this aggregation: 2,275 versus 2,256 for Fixed and 2,296 versus 2,268 for Dynamic. Both macro- and problem-weighted results should be reported. The hyperparameter-selection protocol also needs clarification.

  2. The contribution of the rubric bank is not fully isolated from teacher supervision and additional inference. GPT-5.4 derives criteria using reference solutions, whereas the compared baselines do not receive equivalent access to this teacher or the same training failures. A self-mined bank and experience-based methods given the same evidence would help identify the source of improvement. The auditor's impact test in Fig. 9 also requires considering whether a correction changes the final answer, and the case study in Fig. 16 supplies decisive solution guidance. The paper does include meaningful compute baselines: Table 8 reports Self-Refine at 2.76 times vanilla, PRM at 2.08 times, and Multi-TAG at 0.91 times. Nevertheless, a budget-matched sampling/voting curve and equal-supervision comparisons would strengthen attribution to the full bank. Existing controls recover substantial gains: the generic criterion yields 1.49 of the 2.83 macro-average points, and random criteria recover 31 of the 45 additional solved problems under problem weighting. The lack of gains on MATH-500 and AQuA for Fixed-MAS, and AQuA for Dynamic-MAS, also merits analysis, although transfer from mining sources need not improve every corresponding test set.

Questions for authors
  1. Could you report mean and standard deviation over at least 3 seeds, problem-level paired comparisons against vanilla MAS and PRM, and both macro- and problem-weighted averages? Please also describe the validation data and procedure used to select hyperparameters.

  2. Beyond the existing Self-Refine, PRM and Multi-TAG controls, could you add (i) criteria mined by Qwen3-8B itself, (ii) a sampling/voting comparison matched to the full method's token budget, and (iii) an experience-based method such as AutoGuide or ExpeL given the same training failures and teacher access?

  3. How frequently do revisions correct a wrong answer or spoil a correct one? What fraction of correct messages does the auditor flag?

  4. The 14B single-agent row (Table 4) and the 8B single-agent row (Table 1) share displayed scores on 5 of 9 benchmarks, including GSM8K with 1,319 problems, and the 14B MATH-500 entry (79.40) equals the 8B +CoT row. Some coincidences are expected on small test sets, but please confirm the run provenance. Table 6 also reports that 8.7% of 14B audits have no criteria in force after selection, leaving Eq. 5 undefined as written. How are these empty audits handled, and what effect does that policy have on the efficiency comparison in Fig. 5b and Table 7?

7 comments
AuthorsOpenReview ↗

Response to Reviewer zweD (Part 1/5)

We split the weaknesses and questions into atomic points, numbered W1–W8 and Q1–Q10 in the order they appear in the review, and quote the relevant sentence before each reply.

W1/Q1. The paper does not provide multi-seed uncertainty estimates.

We repeated every Dynamic-MAS configuration of Table 1 three times under the paper's setting (Qwen3-8B reasoners and auditor, Qwen3.5-9B selector, reasoners/selector at temperature 0.7, auditor at temperature 0; runs differ only through sampling).

MethodMacro-avg. (mean ± s.d.)Problems solved / 3,048 (mean ± s.d.)Problem-weighted acc.
Single Agent49.81 ± 0.342,169 ± 3171.16
+ CoT50.47 ± 1.642,258 ± 3174.08
Dynamic-MAS51.74 ± 0.782,225 ± 373.01
+ Multi-TAG51.12 ± 0.962,244 ± 673.61
+ Self-Refine49.76 ± 1.112,250 ± 1373.81
+ PRM52.46 ± 1.152,305 ± 1975.61
+ RaR-Gen52.63 ± 1.222,259 ± 1774.11
+ MASRubric53.77 ± 0.282,265 ± 474.31

The gain of MASRubric over Dynamic-MAS is stable across runs: +2.03 points (53.77 vs. 51.74), larger than the sum of the two standard deviations. We will report mean ± s.d. over the repeated runs in the revision.

W2/Q3. Both macro- and problem-weighted results should be reported.

The table in our reply to W1 reports both aggregations, as the reviewer suggests. The reviewer is right that they rank PRM and MASRubric differently, and we acknowledge that PRM is ahead under problem weighting: it solves 2,305 problems on average against 2,265 for MASRubric (Dynamic-MAS: 2,225). The per-benchmark means over the three runs show where this comes from (easy and hard tiers as in Table 1; the two tier columns are problem-weighted):

Mean of 3 runsGSM8KMATH-500AQuAAMC23Easy tier avg.OlymBAIME24AIME25OlymEOlymHHard tier avg.
Dynamic-MAS91.5379.4084.2570.0087.3847.7027.7823.3323.6718.0040.53
+ PRM93.7383.2786.8867.5089.9451.6527.7820.0025.3316.0043.24
+ MASRubric91.7979.0085.0472.5087.5851.7528.8926.6730.6717.6744.31

PRM's lead is built on the easy tier, while MASRubric is ahead of PRM on all five benchmarks of the hard tier. To understand why MASRubric gains little on the easy tier, we take the run behind Table 1 and annotate 80 problems that the team failed (40 per tier) and, without seeing the benchmark, categorize the criteria behind 92% of all criterion selections:

Easy tierHard tier
Why the team's answer is wrong
Wrong or incomplete method10%72.5%
Slip in arithmetic or algebra25%12.5%
Problem statement misread27.5%5%
Flawed reference answer, ambiguous problem or answer-form mismatch37.5%10%
What the selected criteria check
That a computation was carried out correctly72%30%
The method or the completeness of the argument17%53%

The pattern follows from difficulty itself. On easy problems Qwen3-8B knows the method, so what remains are failures that knowing the method does not prevent: execution slips, misread statements and flawed items. The criteria retrieved for such short computations merely ask the auditor to redo them, and an auditor that is the same model as the solver tends to repeat its reading and arithmetic. On hard problems the failure is usually a missing or unjustified idea, and the retrieved criteria ask what the solver did not ask itself, for example whether a case analysis is exhaustive or a count is derived rather than assumed; this is where the bank adds information.

Thus, these two methods are complementary, and we will report both aggregations, the per-benchmark table and this analysis in the revision. Moreover, PRM needs a separately trained 7B reward model and three sampled candidates per step, while MASRubric trains no model and mines its criteria only from the training splits of MATH and AQuA, so its hard-tier gain is a transfer to benchmarks that contributed nothing to the bank.

AuthorsOpenReview ↗

Response to Reviewer zweD (Part 2/5)

W3/Q4. The hyperparameter-selection protocol also needs clarification.

The paper's setting (TmaxT_\text{max} = 3, KctxK_\text{ctx} = 5, τpass\tau_\text{pass} = 60%) was chosen as a trade-off between accuracy and audit cost. We ran MASRubric on 600 problems sampled from the GSM8K training split (the bank is mined from MATH and AQuA only), changing one hyperparameter at a time:

AccuracyLower settingDefaultHigher setting
Max revision rounds TmaxT_\text{max}1: 93.00, 2: 93.333: 93.504: 93.83
Criteria in force KctxK_\text{ctx}3: 93.335: 93.507: 93.67
Pass threshold τpass\tau_\text{pass}40%: 92.3360%: 93.50100%: 93.33

Accuracy rises slowly with TmaxT_\text{max} and KctxK_\text{ctx}, and the default threshold is the best of the three. Every setting below the default loses accuracy, and the two above it (TmaxT_\text{max} = 4, KctxK_\text{ctx} = 7) achieve marginal gains while enlarging the audit budget by a fourth revision round or two more criteria per audit. We therefore adopt the setting as demonstrated in the paper.

W4/Q5/Q7. The contribution of the rubric bank is not fully isolated from teacher supervision and additional inference. A self-mined bank and experience-based methods given the same evidence would help identify the source of improvement.

We first let the agents' own backbone mine the bank using the same training failures as our paper employed. Since a model that mines criteria from its own failures and then audits its own outputs is prone to self-evaluation bias, we also built a bank with Gemma-4-12B, a model of similar size from a different family.

GSM8KMATH-500AQuAAMC23OlymBAIME24AIME25OlymEOlymHMacro-avg.
Dynamic-MAS91.6678.0085.4362.5048.1530.0020.0026.0016.0050.86
+ MASRubric, Qwen3-8B self-mined bank91.5881.6084.2570.0050.8130.0020.0029.0016.0052.58
+ MASRubric, Gemma-4-12B bank91.2877.0084.6567.5049.6343.3316.6728.0021.0053.23
+ MASRubric, GPT-5.4 bank (paper)91.6679.6083.8670.0052.4430.0026.6732.0017.0053.69

The self-mined bank already recovers most of the gain (+1.72 points over Dynamic-MAS, against +2.83 for the GPT-5.4 bank). The remaining gap is likely due to this bias, because the Gemma-4-12B bank, mined by an equally small but independent model, comes close to the GPT-5.4 bank (+2.37). The improvement therefore does not mainly depend on the presence of a stronger model. We will add this table to the paper.

For the experience-based comparison, we implemented an AutoGuide-style baseline (Fu et al., 2024) on exactly the evidence behind our bank: the same failed training rollouts with their reference solutions, and the same teacher, GPT-5.4. The teacher turns each failed problem into state-conditioned guidelines ("when you are in situation X, do Y"); at test time the guidelines whose state matches are retrieved with MASRubric's own retrieval and appended to the agent's prompt before it writes. No message is audited, revised or withheld, and everything else is as in Table 1.

GSM8KMATH-500AQuAAMC23OlymBAIME24AIME25OlymEOlymHMacro-avg.
Dynamic-MAS91.6678.0085.4362.5048.1530.0020.0026.0016.0050.86
+ AutoGuide-style91.8177.8085.4367.5048.3023.3326.6720.0016.0050.76
+ MASRubric91.6679.6083.8670.0052.4430.0026.6732.0017.0053.69

With the same failures and the same teacher, guidance given before generation brings no gain over Dynamic-MAS (50.76 vs. 50.86), whereas MASRubric, built from the identical evidence, reaches 53.69. The improvement therefore does not come from the teacher's supervision as such, but from applying that knowledge as an audit of the produced message, which can demand a revision and keep a failed message from the other agents. We will add this baseline to Table 1.

Reference

  • Fu, Yao, et al. "Autoguide: Automated generation and selection of context-aware guidelines for large language model agents." Advances in Neural Information Processing Systems 37 (2024): 119919-119948.
AuthorsOpenReview ↗

Response to Reviewer zweD (Part 3/5)

W5. The auditor's impact test in Fig. 9 also requires considering whether a correction changes the final answer, and the case study in Fig. 16 supplies decisive solution guidance.

At test time the auditor has no access to a reference answer. The impact test of Fig. 9 is a filter against false alarms: a remark counts as a violation only if the auditor expects that fixing it would change the conclusion. It is the auditor's own estimate, not a check against an answer. In the run behind Table 1 it kept 3,257 verdicts that came with a concrete suggestion (6.1% of the verdicts on final message versions) from counting as violations. The impact assessment stays in the audit record; the agent receives the suggestion and the analysis of each violated criterion. In Fig. 16 these do contain the corrected arc term and thereby the answer, so the reviewer's reading of that example is right. We therefore measured how often this happens.

In the run behind Table 1, the auditor rejected a message with a wrong answer 2,890 times, and in only 28 of these cases (1.0%), most of them on GSM8K, did its feedback supply the reference answer. 87% of the revisions that turned a wrong answer into a correct one were made without it. Feedback that hands over the solution, as in Fig. 16, is thus the exception, and the revisions do not rely on the auditor solving the problem for the agent. We will add this analysis to the paper.

W6/Q6. The paper does include meaningful compute baselines: Table 8 reports Self-Refine at 2.76 times vanilla, PRM at 2.08 times, and Multi-TAG at 0.91 times. Nevertheless, a budget-matched sampling/voting curve and equal-supervision comparisons would strengthen attribution to the full bank.

We take a majority vote over the final answers of k independent runs of vanilla Dynamic-MAS; equivalent answers are merged with the benchmark's grader, ties are broken at random, and the results are averaged over the choices of the k runs. One MASRubric pass costs 201.2M tokens, as much as 2.63 vanilla passes (76.4M each), so no integer k matches its budget. To spend exactly the same budget, we vote over 3 runs on 63% of the problems and over 2 runs on the remaining 37%, which gives 0.63 × 3 + 0.37 × 2 = 2.63 passes per problem on average.

Dynamic-MAS (Qwen3-8B)Budget (× MASRubric)Macro-avg.
1 run0.38×51.74 ± 0.78
2-run vote0.76×51.74 ± 0.39
2/3-run vote, exact budget match1.00×53.06
+ MASRubric, 1 run (paper)1.00×53.69

At exactly matched budget, voting over independent vanilla runs reaches 53.06, 0.6 points below MASRubric, and doubling the vanilla budget (2-run vote, 0.76×) brings no gain at all because two votes resolve nothing when they disagree. Self-consistency over whole MAS runs is nevertheless a strong use of the extra compute. The two mechanisms differ in kind: voting needs k complete independent runs and acts only on the final answer, whereas MASRubric spends its budget inside one trajectory and leaves an auditable record of which check failed and how the message was repaired, so the two can in principle be combined, which we leave to future work.

The equal-supervision comparison (an experience-based method given the same training failures and the same teacher) is in our reply to W4.

W7. Existing controls recover substantial gains: the generic criterion yields 1.49 of the 2.83 macro-average points, and random criteria recover 31 of the 45 additional solved problems under problem weighting.

Both controls keep the full pipeline (interception, audit, revision, withholding) and replace only the criteria the auditor holds, so the gain they recover is the gain of auditing a message before it is broadcast, which is itself part of the method. Each of them is close to MASRubric under one weighting only: the generic criterion recovers about half of the macro-average gain but 11 of the 45 additional problems, and random criteria recover 31 of the 45 problems but 0.79 of the 2.83 macro-average points. Converting Block V of Table 3 into problems solved shows where the bank matters (tiers as in Table 1):

Gain over Dynamic-MASMacro-avg.Problems solvedof which easy tierof which hard tier
Random 1–5 criteria+0.79+31+10+21
Generic criterion+1.49+11−8+19
MASRubric+2.83+45+7+38

On the easy tier all three stay within ten problems of Dynamic-MAS, while on the hard tier MASRubric gains about twice as much as either control. This agrees with the analysis in our reply to W2: the criteria selected on easy problems mostly ask the auditor to redo a computation, so replacing them with a generic or random criterion costs little, whereas hard problems fail at the level of the method and need criteria matched to the situation. We will add this breakdown to the ablation study.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.