acceptodds
Under review as a conference paper at ICLR 2027

MiniRep: Robust Reputation-Based Aggregation for Multi-Agent Debate

Abstract

Autonomous agents powered by large language models (LLMs) are rapidly evolving into an open agentic ecosystem. To support trustworthy collaboration, industry initiatives increasingly assess agent reputation from past behavior and provide performance leaderboards. However, reputation derived from past performance may not reliably predict an agent's behavior on new tasks, particularly when malicious agents can adapt their behavior and influence other agents during collaboration. We study reputation in multi-agent debate (MAD), where multiple agents answer the same query, debate to improve their answers, and aggregate them into a final output. We present MiniRep, a reputation-based aggregation system for MAD under malicious agents. To ground our threat model in established research, we construct an attack taxonomy drawing on reputation-system attacks and software-testing mutation operators, covering strategic exploitation of reputation and subtle corruption of agent proposals. Guided by this taxonomy, MiniRep evaluates agents based on both their behavior on the current task and their reputation over time, while preventing groups of agents with highly similar responses from dominating the final decision. We assess MiniRep across diverse tasks, LLM-agent compositions, corruption placements, and attack types drawn from our taxonomy. Our experimental results show that, MiniRep outperforms both conventional MAD aggregation and conventional reputation-based approaches on MATH no matter being attacked or not. Also, under a heterogeneous 10-agent setting on MATH, MiniRep outperforms all baselines in all 28 attack conditions.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Multi-Agent Debate, Reputation Systems, Robust Aggregation, Adversarial Robustness

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_M6LfOpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents MiniRep, a reputation-based aggregation system for multi-agent debate under malicious agents. MiniRep evaluates agents using both current-task proposal quality and historical reputation, applying risk signals before aggregation to prevent high-reputation attackers from manipulating outputs. Evaluated across three datasets and 112 attack conditions, MiniRep outperforms conventional reputation systems, achieving the highest accuracy on MATH in both clean and attacked settings.

Critical strengths

S1. Well-grounded attack taxonomy. The paper constructs a systematic attack taxonomy by combining established reputation-system attacks with software-testing mutation operators (Table 1). Each attack type is linked to real mechanisms reported in prior work, making the evaluation comprehensive rather than purely synthetic.

S2. Clear motivation and intuitive presentation. The paper is well-written and easy to follow. The problem of reputation manipulation in multi-agent debate is clearly motivated, and the running example (Section 3) effectively illustrates why naive reputation-based aggregation fails. The three-module design (Response Analyzer, Aggregator, State Updater) is logically organized and explained in an accessible manner.

Critical weaknesses

W1. Missing critical experimental details undermine reproducibility. The paper does not disclose the specific versions of the seven LLMs used as agents (only product families like "Qwen-Max" and "GLM" are named in Appendix C.2), despite agent capability being a core experimental variable. Similarly, the Qwen2.5-1.5B verifier's training data size and label source (whether gold answers were used as supervision) are not reported (Appendix B.1.2).

W2. Asymmetric access to feedback signals favors MiniRep. All methods share the same verifier feedback, but baselines (Beta, EigenTrust, etc.) only use it after task t to update weights for t+1, while MiniRep uses it before aggregation on task t (Eq. 34, 42). This temporal asymmetry conflates the reputation mechanism with the pre-aggregation screening advantage. An ablation where baselines also access current-task signals before aggregation is needed to isolate the contribution of the reputation design itself.

W3. Evaluation benchmarks are too simple for the system complexity. MiniRep employs a fine-tuned verifier, behavior probes, and multi-signal fusion, yet is evaluated only on GoEmotions, MATH, and HumanEval Pro—relatively simple, short-horizon tasks by 2026 standards. This mismatch raises practicality concerns: modern agentic workflows involve complex, multi-step tool use where such elaborate reputation machinery would be more justified. The authors should demonstrate effectiveness on at least one realistic agentic benchmark (e.g., SWE-bench or multi-step tool-use tasks) or justify why these simple tasks suffice.

Questions for authors

Q1. Which specific models were used as agents, including their sizes and versions?

Q2. Could the authors provide results on at least one long-horizon agentic task?

7 comments

Q1: All methods share the same verifier feedback, but baselines (Beta, EigenTrust, etc.) only use it after task t to update weights for t+1, while MiniRep uses it before aggregation on task t (Eq. 34, 42). This temporal asymmetry conflates the reputation mechanism with the pre-aggregation screening advantage. An ablation where baselines also access current-task signals before aggregation is needed to isolate the contribution of the reputation design itself.

A1: Thank you for highlighting the timing of feedback. MiniRep retains its strongest defense advantage on MATH after the baselines receive current-task feedback; on GoEmotions it limits attack-induced harm, while current-aware baselines remain competitive on HumanEval Pro. Following your suggestion, we give the baselines access to current feedback before aggregation. This comparison preserves the original MATH advantage and clarifies which defense benefits remain on the other tasks. We compare the original baselines with their +Current versions below, reporting clean and attacked performance separately.

Each +Current version preserves its baseline’s historical update and multiplies its historical weight by max(10⁻⁶, 0.75 × semantic support + 0.25 × public support) before pool selection and aggregation. The methods share the same tasks, candidate proposals, configurations, and attacks.

Clean and attacked results. We evaluate 100 tasks per benchmark. Clean averages the three attack-free runs; Attacked averages the 21 attack conditions formed by three settings and seven attacks. Drop is Clean minus Attacked, calculated before rounding. CF-Harm measures the fraction of active-attack tasks whose score falls below the corresponding clean result. All scores and rates below are percentages; Drop is in percentage points.

GoEmotions: sample F1.

MethodCleanAttackedDropCF-Harm ↓
MiniRep34.6835.03−0.362.11
Single-Metric37.3335.501.839.35
Single-Metric+Current38.6336.422.228.08
EigenTrust36.6629.157.5023.83
EigenTrust+Current39.5835.134.4513.01
Beta37.5836.411.176.96
Beta+Current39.1936.442.749.28

MATH: mathematical-equivalence accuracy.

MethodCleanAttackedDropCF-Harm ↓
MiniRep80.0081.38−1.381.95
Single-Metric67.6772.95−5.295.56
Single-Metric+Current73.6771.142.528.49
EigenTrust69.0053.5715.4321.11
EigenTrust+Current74.0064.529.4813.67
Beta70.6773.05−2.385.96
Beta+Current74.3371.103.249.14

HumanEval Pro: Pass@1.

MethodCleanAttackedDropCF-Harm ↓
MiniRep78.6779.57−0.902.65
Single-Metric78.6775.902.768.12
Single-Metric+Current79.6780.29−0.622.83
EigenTrust78.6779.81−1.142.50
EigenTrust+Current80.0080.24−0.242.10
Beta78.6776.002.677.40
Beta+Current81.3380.520.812.89

Current feedback has different effects on clean performance and attack resistance. Adding Current improves the clean MATH accuracy of Single-Metric and Beta, yet their attacked accuracy falls and their CF-Harm rises. For EigenTrust, both clean and attacked accuracy improve. These differences show why clean and attacked results are both needed to assess the benefit of current feedback.

Defense against the original attacks. We also examined Strong, Rand, Div, Arith, Bound, OnOff, and Adapt separately. The results show where the original performance and defense advantages remain:

  • MATH retains the clearest advantage. MiniRep has higher attacked accuracy and lower PayloadHit than every +Current baseline in all 21 attack/configuration conditions. Against the lowest-harm +Current method in each condition, its CF-Harm is lower in 18 conditions and tied in three. The advantage also holds in all seven attack-family averages. Its mean PayloadHit is 18.32%, compared with 34.49% for the best +Current baseline, and NextExcl is 38.41%, compared with 34.31%.
  • GoEmotions benefits mainly through lower attack-induced harm. MiniRep has lower CF-Harm in 19 conditions and ties the lowest-harm +Current method in two; all seven attack-family averages favor MiniRep on CF-Harm. The highest absolute attacked F1 comes from a +Current baseline in six of the seven families. MiniRep’s main advantage here is limiting attack-induced harm, with mean CF-Harm of 2.11% versus 8.08% for the best +Current baseline.
  • HumanEval Pro retains effective payload suppression and subsequent exclusion. MiniRep has zero observed PayloadHit on conditions with an exposed payload and a NextExcl rate of 88.89%, compared with the best +Current values of 0.38% and 67.39%. EigenTrust+Current has lower average CF-Harm, 2.10% versus 2.65%; against the lowest-harm +Current method in each condition, MiniRep records two wins, seven ties, and twelve losses. The results support effective payload suppression and subsequent attacker exclusion, with stronger current-feedback baselines on final task accuracy and harm.

PoolExp, PayloadHit, NextExcl, and CF-Harm use the definitions already given in the paper, with undefined denominators omitted from each rate’s average. GoEmotions retains MiniRep’s proposed-label-set readout and the baselines’ label-wise readout.

Revisions:

  • Section 4.1: We added a pointer to baselines using current feedback before aggregation, identifying the comparison that addresses the feedback-timing concern.
  • Section 4.2: We now separate clean and attacked MATH results and summarize the remaining defense advantages and the HumanEval Pro exception, making the effect of shared current feedback explicit in the main text.
  • Appendix E.3 and Table 16: We retained the full original-versus-+Current comparison of clean scores, attacked scores, drops, and CF-Harm and added an interpretation of how current feedback changes performance under attack.
  • Appendix E.6 and Table 19: We added PoolExp, PayloadHit, and NextExcl for MiniRep and the three +Current baselines, testing whether payload suppression and subsequent attacker exclusion persist after feedback access is shared.
  • Appendix E.6 and Table 20: We reported attacked scores and CF-Harm separately for all seven original attack families, showing where the defense advantage remains and where current-feedback baselines perform better.
  • Appendix E.2 and Table 15: We compare History-only and Current-quality-only variants to clarify the contribution of persistent reputation alongside the matched current-feedback baselines.

Q2: The paper does not disclose the specific versions of the seven LLMs used as agents (only product families like "Qwen-Max" and "GLM" are named in Appendix C.2), despite agent capability being a core experimental variable. Similarly, the Qwen2.5-1.5B verifier training data size and label source (whether gold answers were used as supervision) are not reported (Appendix B.1.2).

A2: Thank you for pointing out these omissions. Appendix E.1 now provides the exact agent identifiers, verifier training counts, label sources, and training configuration. These details make the model pool and learned feedback behind the reported results explicit, providing the information needed to assess and reproduce the paper’s empirical findings.

Agent models and generation settings. The seven agent identifiers recorded in the experiments are:

Name in the paperAPI model identifier
Qwen-Maxqwen3.7-max
Qwen-Plusqwen3.7-plus
Qwen-Flashqwen3.7-flash
GLMglm-5.2
DeepSeekdeepseek-v4-flash
Kimikimi/kimi-k3
MiniMaxMiniMax/MiniMax-M3

All seven are accessed through Bailian. Generation uses temperature 0.2, with Kimi set to 0.6 in the added debate and task-sample experiments. Output limits are 1,024 tokens for MATH and GoEmotions and 4,096 for HumanEval Pro.

Training data and supervision. The semantic verifier is based on Qwen/Qwen2.5-1.5B-Instruct. The pipeline first collects API responses from clean and attacked bootstrap runs on a calibration task pool separate from the aggregation evaluation; the default construction also adds task-grounded synthetic cohorts. Gold answers and hidden tests provide the correctness labels: exact label-set agreement for GoEmotions, answer equivalence for MATH, and hidden-test outcomes for HumanEval Pro. Correct responses receive label 1; incorrect or invalid responses receive label 0. Gold information is excluded from verifier inputs. The behavior probe has a separate label indicating an active attack payload, so an honest error receives a negative attack label.

Training size and configuration. The recorded data partitions are:

Semantic-verifier partitionResponse examples
Training10,470: 3,876 correct and 6,594 incorrect
Validation1,120
Held-out-task test977

These partitions are grouped by canonical task, keeping response and attack variants of the same task together. Attack-family generalization is reported separately in Appendix D.1.

The training manifest records class-weighted focal cross-entropy with gamma 2, soft-ECE regularization with weight 0.05, and validation temperature scaling with T=0.7590. The released training recipe uses 4-bit LoRA, rank 16, scaling 32, two epochs, learning rate 2×10⁻⁴, batch size 1, gradient accumulation 16, and maximum sequence length 1,024.

Revisions:

  • Appendix E.1 and Table 14: We listed the seven recorded API model identifiers and their generation settings, resolving the missing model-version information.
  • Appendix E.1, “Training data and supervision”: We described the API-response collection pipeline and benchmark-derived correctness labels, explicitly stating how gold answers and hidden tests supply supervision and are excluded from verifier inputs.
  • Appendix E.1, “Training size and configuration”: We added training, validation, and held-out-task counts, the training label balance, the recorded objective, and the released training recipe to make verifier training reproducible.
  • Appendix B.1.2: We clarified that training, validation, and held-out-task partitions are grouped by canonical task, explaining how responses and attack variants from the same task are kept together.
Reviewer_WJsbOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MiniRep, a reputation-aware aggregation procedure for multi-agent debate settings with persistent agent identities and a known mapping from agents to underlying model clone groups. MiniRep combines current-task proposal quality and behavioral-risk signals with historical reputation, applies clone-group caps during selection and weighting, and updates persistent quality and risk records after each task. The paper evaluates the system on GoEmotions, MATH, and HumanEval Pro under seven synthetic attack types, four agent compositions, and four corruption placements.

Critical strengths
  1. The paper targets a meaningful security problem: historical performance is an insufficient trust signal when agents can behave honestly to accumulate reputation and subsequently submit adversarial proposals.

  2. The MATH results are substantively promising within the stated experimental setting. In Table 3, MiniRep reaches 66.75% clean MATH accuracy and 61.95% attacked accuracy, compared with 59.25% and 54.37% for the respective best evaluated baselines.

Critical weaknesses
  1. The central empirical setting is not actually multi-agent debate in the usual interactive sense. The paper evaluates every reported main result with R=0, as stated in Section 4.1 and Appendix A.2. Thus agents generate independent cached proposals and never observe, challenge, revise, or propagate one another’s reasoning. This is an aggregation experiment over a fixed panel of responses, not an evaluation of robustness under debate dynamics. That distinction matters because several attacks motivating the paper, such as persuasion, conformity, fabricated consensus, instruction override, and cross-round behavior changes, become more consequential only when messages are exchanged. The paper should substantially narrow claims from robust MAD to robust proposal aggregation, or evaluate at least one genuine multi-round protocol with visible debate histories.

  2. The evaluation is too small and too deterministic to support broad robustness claims. Each benchmark uses only 100 fixed tasks, cached proposals are reused across methods, and the 112 attack conditions are not independent task samples or generation-seed repetitions.

  3. The comparison does not isolate whether the gains come from reputation, current-task verification, learned attack detection, clone caps, or task-specific aggregation rules. MiniRep contains a fine-tuned semantic verifier, an MLP behavior probe, a fused blocker, current-task quality weighting, historical risk penalties, cooldowns, clone-group constraints, and selection rules. Yet there is no matched ablation removing each component or comparing against a strong current-task-quality-only approach.

Questions for authors

See Weaknesses

4 comments

Q1: The central empirical setting is not actually multi-agent debate in the usual interactive sense. The paper evaluates every reported main result with R=0, as stated in Section 4.1 and Appendix A.2. The paper should substantially narrow claims from robust MAD to robust proposal aggregation, or evaluate at least one genuine multi-round protocol with visible debate histories.

A1: Thank you for highlighting the need for an interactive evaluation. MiniRep attains higher final-answer accuracy with two debate rounds than with no debate, while attack-induced harm increases. The added debates extend the original advantage in attacked MATH accuracy to interactive runs and identify weaknesses in the defense, especially against the two peer-only communication attacks.

Comparison with no debate. The following table compares MiniRep on the same 100 MATH problems and three composition/corruption settings. Clean scores average the three attack-free runs; attacked scores and defense metrics average the 21 conditions formed by the seven original attacks.

RoundsCleanAttackedDropCF-HarmPoolExpPayloadHitNextExcl
R=080.0081.38−1.381.9562.6718.3238.41
R=183.6781.572.107.8740.8313.2562.26
R=294.0087.196.8112.5741.1213.4361.56

Scores and rates are percentages; Drop is Clean minus Attacked in percentage points. Lower CF-Harm, PoolExp, and PayloadHit are better; higher NextExcl is better. All MATH scores use mathematical-equivalence evaluation.

The comparison gives three findings relevant to the original robustness claim.

1. The original advantage in attacked accuracy persists during debate. Relative to R=0, attacked accuracy increases by 0.19 points at R=1 and 5.81 points at R=2. MiniRep retains the highest attacked accuracy for each of the seven original attack families at both R=1 and R=2, averaged over the three configurations. Relative to R=0, one round improves attacked accuracy for Div and Adapt, while reducing it for random Sybil, strongest Sybil, Arith, Bound, and OnOff; two rounds improve attacked accuracy for all seven. These results extend the original aggregation finding to answers revised through interaction.

Kimi’s temperature changes from 0.2 to 0.6, the random Byzantine identities vary across round counts, and the historical-update source changes from proposal feedback to combined proposal/debate feedback. These settings accompany the reported comparison.

2. Higher answer accuracy accompanies greater attack damage. At R=1, MiniRep has the lowest CF-Harm for every original attack family. CF-Harm nevertheless increases for every family relative to R=0 at both round counts. Overall CF-Harm rises from 1.95% at R=0 to 7.87% and 12.57% at R=1 and R=2. At R=2, Beta has lower CF-Harm than MiniRep (11.37% versus 12.57%) and a smaller drop (6.10 versus 6.81 points). Adaptive attacks expose the largest weakness: MiniRep’s CF-Harm reaches 18.64%, compared with 7.64% for TrueSkill. The results support partial protection against the original attacks, with clear weaknesses at R=2.

The lower PoolExp and PayloadHit and higher NextExcl relative to R=0 indicate that selection and payload suppression remain active, although these gains do not prevent the higher overall harm. We therefore distinguish the original advantage in attacked accuracy from the degree of protection against degradation. Appendix E.6, Table 21 gives the individual attack comparisons.

3. Peer-only feedback provides insufficient protection against the new communication attacks. Prompt injection and echo-chamber attacks are evaluated in a separate peer-only suite, with the learned verifier and behavior probe disabled and an independent clean reference for each round count.

AttackRoundsMiniRep CleanMiniRep AttackedDropCF-HarmBest baseline: Attacked
Prompt injection184.6752.3332.3339.02UMaj: 70.00
Echo chamber184.6747.0037.6745.38URand-k: 65.33
Prompt injection290.3359.6730.6739.43UMaj: 76.33
Echo chamber290.3356.6733.6741.77URand-k: 71.33

Both attacks substantially degrade MiniRep, and the strongest uniform baseline performs better in every row. We compare these attacks with the corresponding clean runs and same-round baselines. We have not evaluated a peer-only R=0 run or these two attacks with learned assessment enabled.

When reputation changes. Persistent reputation is updated once after final aggregation and remains fixed within a task’s debate rounds. Quality and risk feedback is recomputed after the initial proposals and after every round; the both policy combines these observations for the history used on the next task. All ten agents participate in every round and receive the complete preceding public transcript, including malicious messages. Screening and weighting are applied to the final-round answers, and k=7 specifies the final aggregation pool. There is no filtering of participants or transcript messages between rounds. This timing allows malicious messages to influence other agents before the final screen.

Revisions:

  • Abstract and Section 1: We identify shared-answer aggregation as the evaluated setting and distinguish the original zero-debate sweep from the added interactive experiments, aligning the scope with the evidence.
  • Section 4.1: We distinguish the original R=0 proposal-aggregation setting from the added interactive debate, making the no-debate reference explicit.
  • Section 4.2: We added the R=0/R=1/R=2 performance and harm comparison and clarified the timing of reputation updates, addressing both the effect of debate and the mechanism used during interaction.
  • Appendix E.4 and Table 17: We report clean, attacked, and defense metrics across all three round counts and describe the full-transcript protocol and post-task reputation update.
  • Appendix E.6 and Table 21: We report the seven original attacks separately across R=0/R=1/R=2, showing where protection is retained and where attack-induced harm grows.
  • Appendix E.6 and Table 22: We separate prompt injection from echo-chamber results and compare each with its clean reference and strongest same-round baseline, making the communication-attack failures explicit.
  • Section 6: We align the concluding evaluation scope with shared-answer aggregation and interactive debate, avoiding a broader claim about coordinated agentic execution.

Q2: The comparison does not isolate whether the gains come from reputation, current-task verification, learned attack detection, clone caps, or task-specific aggregation rules. MiniRep contains a fine-tuned semantic verifier, an MLP behavior probe, a fused blocker, current-task quality weighting, historical risk penalties, cooldowns, clone-group constraints, and selection rules. Yet there is no matched ablation removing each component or comparing against a strong current-task-quality-only approach.

A2: Thank you for asking for component-level comparisons. Semantic-quality weighting produces the most consistent benefit in the component study; historical feedback also helps, while risk and clone controls have smaller, task-dependent effects. These ablations examine the original explanation of the gains and show which parts of the design contribute most. We added the following eight variants using the same 100 tasks per benchmark, three configurations, and seven attacks.

VariantGoEmotions F1MATH accuracyHumanEval Pro Pass@1
Full MiniRep35.0381.3879.57
History-only35.0382.9578.62
Current-quality-only34.7863.3877.24
History + current quality35.2783.0579.81
w/o semantic quality32.5867.5258.10
w/o probe risk35.4283.3379.19
w/o clone mass cap35.0281.3379.62
w/o cooldown / historical risk34.9481.7679.95

Entries are attacked task scores (%). All variants share cached base proposals. Non-adaptive attack realizations are fixed across variants; the adaptive attacker responds to each variant’s evolving state.

The results clarify the original component explanation in three respects.

1. Semantic quality contributes across all three benchmarks. Removing semantic-quality weighting lowers attacked performance on every benchmark in the table, supporting the original emphasis on assessing proposal quality.

2. Historical feedback contributes beyond current-response quality. History-only exceeds Current-quality-only on MATH, supporting the use of accumulated feedback in the original design. History-only also exceeds Full on MATH accuracy, although Full has lower CF-Harm (1.95% versus 3.14%). This comparison explains why we report both accuracy and attack-induced harm when assessing the full method.

3. Risk and clone controls have smaller, task-dependent effects. Removing clone-mass normalization changes attacked scores by approximately 0.05 points or less. We revised the component discussion to reflect these different effects, with historical feedback and semantic quality accounting for the clearest gains in this study.

What the ablations change. The semantic-quality ablation replaces semantic support with public support in current and historical quality weights; the shared verifier and fused detector remain available. Probe-risk removal retains semantic-outlier checks and temporal quality signals. The clone ablation removes mass normalization. History-only retains post-task semantic feedback, while Current-quality-only fixes historical weights to one. Appendix E.2 defines these switches explicitly.

For GoEmotions, MiniRep aggregates proposed label sets, while the baselines use label-wise aggregation. The revised text identifies this as a comparison of complete aggregation methods. Appendix E.3 separately examines baselines with access to current-task feedback before aggregation.

Revisions:

  • Section 4.1: We clarified that the comparisons include each method’s screening and GoEmotions readout, making the role of task-specific aggregation rules explicit.
  • Section 4.2: We added a component-level interpretation that identifies the large effects of semantic quality and historical feedback and the setting-dependent effects of risk and clone controls.
  • Appendix E.2 and Table 15: We specified exactly which signals and restrictions each variant removes and reported all eight variants, allowing the component claims to be assessed against the implemented changes.
  • Appendix E.3 and Table 16: We added baselines with current feedback before aggregation, examining whether the observed advantage persists when those baselines receive current-task signals.

Q3: The evaluation is too small and too deterministic to support broad robustness claims. Each benchmark uses only 100 fixed tasks, cached proposals are reused across methods, and the 112 attack conditions are not independent task samples or generation-seed repetitions.

A3: We agree that broader task coverage is needed to assess the original MATH finding. The expanded evaluation covers 290 distinct problems across three 100-problem samples, and the results show that method rankings vary across samples. The added samples support the value of historical feedback and semantic quality, while showing that the original MATH accuracy lead varies with the task sample.

We added task samples selected with seeds 2028 and 2029 and generated fresh proposals for each. Methods share proposals within each sample and configuration.

VariantOriginal sampleSample 2028Sample 2029Mean ± SD
Full MiniRep81.3884.4378.2981.37 ± 3.07
History-only82.9588.0581.1084.03 ± 3.60
Current-quality-only63.3879.5275.2472.71 ± 8.36
History + current quality83.0585.7679.8182.87 ± 2.98
w/o semantic quality67.5279.8673.6273.67 ± 6.17

Entries are attacked accuracy (%) under mathematical-equivalence scoring. Each score averages the same 21 attack conditions. Mean and sample standard deviation are computed over the three sample-level scores.

Two findings connect these results to the original conclusions.

1. Historical feedback and semantic quality remain useful across samples. Full MiniRep exceeds Current-quality-only and the semantic-quality ablation on every sample. This supports the original emphasis on agent history and proposal quality beyond the first task sample.

2. The MATH accuracy ranking changes with the sample. History-only exceeds Full on all three, and Beta reaches 87.33% and 80.19% on the two added samples, also exceeding Full. The full risk policy provides no consistent additional accuracy gain. We have therefore made the sample dependence of the original MATH advantage explicit in the revised discussion.

We describe the 112 conditions as an evaluation grid and report variability across the three task samples with their fresh proposals. Appendix E.1 records implementation and decoding settings, including Kimi’s temperature change from 0.2 to 0.6 in the added samples.

Revisions:

  • Section 4.1: We clarified that the original 112 conditions reuse fixed tasks and identified the two additional 100-task MATH samples, distinguishing condition coverage from task coverage.
  • Section 4.2: We added the total of 290 distinct problems and the three-sample statistics for Full and History-only, extending the main-text evidence beyond the original task sample.
  • Appendix E.5 and Table 18: We reported the additional samples, task overlaps, sample-level means and standard deviations, and changes in method rankings, directly addressing task coverage and variability.
  • Appendix E.1: We documented implementation and decoding settings, including Kimi’s temperature change, so readers can identify the factors that vary across the sample runs.
Reviewer_7ra8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces MiniRep, a reputation-based aggregation system for Multi Agent Debate. It aims to protect against attacks from malicious agents that are trying to manipulate the Multi Agent Debate output away from the correct answers.

Critical strengths
  • The proposed method is novel and tackles a relevant problem in agentic systems. Combining current-response assessment with historical reputation is a useful approach to this problem.

  • Results show that MiniRep maintains strong performance in both clean and attacked settings.

Critical weaknesses
  • There is limited ablation analysis to identify which components of MiniRep are contributing to the improvements. MiniRep introduces historical reputation, current response assessment and a reputation aware aggregation. The experiments do not isolate whether the gains come from persistent reputation or current response assessment.

  • Aside from the MATH benchmark, the performance gain over the best baseline is mixed and often small. Higher attacked accuracy should also be evaluated separately from greater resistance to degradation under attack. A method can have higher accuracy under attack but show a larger drop from clean to attack setting.

Questions for authors

Please refer to the Weaknesses section above.

3 comments

Q1: There is limited ablation analysis to identify which components of MiniRep are contributing to the improvements.

A1: Thank you for suggesting a component-level comparison. Historical feedback contributes substantially on MATH, and semantic-quality weighting is useful across all three benchmarks. These findings support our original claim in the paper that agent history and proposal quality contribute to the gains. The additional experiemnts clarify the role of the remaining safeguards. We added eight variants, and the table below highlights the comparison between historical and current feedback.

VariantGoEmotions F1MATH accuracyHumanEval Pro Pass@1
Full MiniRep35.0381.3879.57
History-only35.0382.9578.62
Current-quality-only34.7863.3877.24
History + current quality35.2783.0579.81
w/o semantic quality32.5867.5258.10

Scores are attacked task performance (%). Each benchmark uses 100 tasks, three agent-composition/corruption settings, and seven attacks. Methods share the same cached base proposals.

The results provide two main findings:

  • Historical feedback contributes beyond current-response assessment. Full MiniRep exceeds Current-quality-only by 18.00 percentage points on MATH. History-only also performs well, reaching 82.95%; it continues to update historical quality from semantic feedback after each task. This supports the paper’s use of information accumulated across tasks.
  • Semantic quality provides a consistent benefit across tasks. Removing semantic quality lowers performance by 2.45, 13.86, and 21.48 points on GoEmotions, MATH, and HumanEval Pro, respectively. This supports the original design choice to assess proposal quality before aggregation and use this feedback to update agent history.

Together, these comparisons explain which parts of the original design account for the observed gains. The accuracy benefit of the full risk policy varies across tasks. We have incorporated this distinction into the component analysis in Section 4.2 and Appendix E.2.

Revisions:

  • Section 4.2, “Component ablations”: We added an analysis identifying historical feedback and semantic quality as the main sources of improvement, directly addressing the attribution of the reported gains.
  • Appendix E.2 and Table 15: We defined the eight variants and reported their clean scores, attacked scores, drops, and CF-Harm, providing the requested comparison of historical and current-response assessment.

Q2: Aside from the MATH benchmark, the performance gain over the best baseline is mixed and often small. Higher attacked accuracy should also be evaluated separately from greater resistance to degradation under attack. A method can have higher accuracy under attack but show a larger drop from clean to attack setting.

A2: Thank you for raising this distinction. MiniRep’s advantage is strongest on MATH, and its ranking by attacked score differs from its ranking by performance degradation. The added comparisons retain the original finding of a clear MATH advantage and mixed results on the other tasks. Following your suggestion, we now examine clean-to-attack losses alongside attacked scores, as shown below.

EvaluationMethodCleanAttackedDrop (pp)
GoEmotions F1, R=0MiniRep34.6835.03−0.36
GoEmotions F1, R=0Beta+Current39.1936.442.74
HumanEval Pro Pass@1, R=0MiniRep78.6779.57−0.90
HumanEval Pro Pass@1, R=0Beta+Current81.3380.520.81
MATH accuracy, R=2MiniRep94.0087.196.81
MATH accuracy, R=2Beta91.3385.246.10

Scores are percentages. Drop is Clean minus Attacked, in percentage points, calculated before rounding. The R=0 rows come from the current-feedback study; the MATH rows come from the regular debate suite.

Two observations explain how these results relate to the paper’s conclusions:

  • The MATH accuracy advantage persists, while degradation exposes a different ranking. In the two-round debate comparison, MiniRep has higher attacked accuracy, while Beta has a smaller drop and lower CF-Harm (11.37% versus 12.57%). The no-debate comparison shows the same distinction: MiniRep’s attacked accuracy rises from 81.38% at R=0 to 87.19% at R=2, while CF-Harm rises from 1.95% to 12.57%. These results support the accuracy finding and qualify how we describe the defense under debate.
  • The other tasks retain the mixed performance pattern reported in the original paper. On GoEmotions and HumanEval Pro, Beta+Current has higher attacked scores, while MiniRep has smaller clean-to-attack drops. Reporting both makes the performance tradeoff visible.

We have added this analysis to Section 4.2 and the detailed comparisons to Appendices E.3 and E.4. Appendix E.6 separates the seven original attacks and the two communication attacks to show where the defense is effective and where greater losses occur.

Revisions:

  • Section 4.2, “Attack-induced degradation”: We added a paragraph comparing attacked scores with clean-to-attack losses, separating absolute task performance from resistance to degradation as requested.
  • Appendix E.3 and Table 16: We added clean, attacked, drop, and CF-Harm results for current-feedback baselines, showing the mixed performance advantages on GoEmotions and HumanEval Pro.
  • Appendix E.4 and Table 17: We reported debate accuracy alongside drop and CF-Harm, including the R=2 case where MiniRep has higher attacked accuracy and Beta has a smaller performance loss.
  • Appendix E.6 and Tables 21–22: We separate the original attacks from prompt injection and echo chamber, showing the attack-specific losses that an aggregate attacked score can conceal.

Thank you for the detailed response and additional experiments. The new ablations clarify the contributions of historical feedback and current-response assessment, and the added analysis distinguishes attacked performance from attack-induced degradation.

However, the history-plus-current-quality variant achieves higher attacked performance than the full method across all three benchmarks, raising questions about the added value of the full safeguards. The mixed gains outside MATH and the tradeoff between accuracy and resistance to degradation also make it difficult to establish a consistent overall advantage.

I appreciate your efforts to address these concerns. Given these remaining reservations, I will maintain my score at 2.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.