- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents MiniRep, a reputation-based aggregation system for multi-agent debate under malicious agents. MiniRep evaluates agents using both current-task proposal quality and historical reputation, applying risk signals before aggregation to prevent high-reputation attackers from manipulating outputs. Evaluated across three datasets and 112 attack conditions, MiniRep outperforms conventional reputation systems, achieving the highest accuracy on MATH in both clean and attacked settings.
Critical strengths
S1. Well-grounded attack taxonomy. The paper constructs a systematic attack taxonomy by combining established reputation-system attacks with software-testing mutation operators (Table 1). Each attack type is linked to real mechanisms reported in prior work, making the evaluation comprehensive rather than purely synthetic.
S2. Clear motivation and intuitive presentation. The paper is well-written and easy to follow. The problem of reputation manipulation in multi-agent debate is clearly motivated, and the running example (Section 3) effectively illustrates why naive reputation-based aggregation fails. The three-module design (Response Analyzer, Aggregator, State Updater) is logically organized and explained in an accessible manner.
Critical weaknesses
W1. Missing critical experimental details undermine reproducibility. The paper does not disclose the specific versions of the seven LLMs used as agents (only product families like "Qwen-Max" and "GLM" are named in Appendix C.2), despite agent capability being a core experimental variable. Similarly, the Qwen2.5-1.5B verifier's training data size and label source (whether gold answers were used as supervision) are not reported (Appendix B.1.2).
W2. Asymmetric access to feedback signals favors MiniRep. All methods share the same verifier feedback, but baselines (Beta, EigenTrust, etc.) only use it after task t to update weights for t+1, while MiniRep uses it before aggregation on task t (Eq. 34, 42). This temporal asymmetry conflates the reputation mechanism with the pre-aggregation screening advantage. An ablation where baselines also access current-task signals before aggregation is needed to isolate the contribution of the reputation design itself.
W3. Evaluation benchmarks are too simple for the system complexity. MiniRep employs a fine-tuned verifier, behavior probes, and multi-signal fusion, yet is evaluated only on GoEmotions, MATH, and HumanEval Pro—relatively simple, short-horizon tasks by 2026 standards. This mismatch raises practicality concerns: modern agentic workflows involve complex, multi-step tool use where such elaborate reputation machinery would be more justified. The authors should demonstrate effectiveness on at least one realistic agentic benchmark (e.g., SWE-bench or multi-step tool-use tasks) or justify why these simple tasks suffice.
Questions for authors
Q1. Which specific models were used as agents, including their sizes and versions?
Q2. Could the authors provide results on at least one long-horizon agentic task?
7 comments
Part 2
- MATH retains the clearest advantage. MiniRep has higher attacked accuracy and lower PayloadHit than every +Current baseline in all 21 attack/configuration conditions. Against the lowest-harm +Current method in each condition, its CF-Harm is lower in 18 conditions and tied in three. The advantage also holds in all seven attack-family averages. Its mean PayloadHit is 18.32%, compared with 34.49% for the best +Current baseline, and NextExcl is 38.41%, compared with 34.31%.
- GoEmotions benefits mainly through lower attack-induced harm. MiniRep has lower CF-Harm in 19 conditions and ties the lowest-harm +Current method in two; all seven attack-family averages favor MiniRep on CF-Harm. The highest absolute attacked F1 comes from a +Current baseline in six of the seven families. MiniRep’s main advantage here is limiting attack-induced harm, with mean CF-Harm of 2.11% versus 8.08% for the best +Current baseline.
- HumanEval Pro retains effective payload suppression and subsequent exclusion. MiniRep has zero observed PayloadHit on conditions with an exposed payload and a NextExcl rate of 88.89%, compared with the best +Current values of 0.38% and 67.39%. EigenTrust+Current has lower average CF-Harm, 2.10% versus 2.65%; against the lowest-harm +Current method in each condition, MiniRep records two wins, seven ties, and twelve losses. The results support effective payload suppression and subsequent attacker exclusion, with stronger current-feedback baselines on final task accuracy and harm.
PoolExp, PayloadHit, NextExcl, and CF-Harm use the definitions already given in the paper, with undefined denominators omitted from each rate’s average. GoEmotions retains MiniRep’s proposed-label-set readout and the baselines’ label-wise readout.
Revisions:
- Section 4.1: We added a pointer to baselines using current feedback before aggregation, identifying the comparison that addresses the feedback-timing concern.
- Section 4.2: We now separate clean and attacked MATH results and summarize the remaining defense advantages and the HumanEval Pro exception, making the effect of shared current feedback explicit in the main text.
- Appendix E.3 and Table 16: We retained the full original-versus-+Current comparison of clean scores, attacked scores, drops, and CF-Harm and added an interpretation of how current feedback changes performance under attack.
- Appendix E.6 and Table 19: We added PoolExp, PayloadHit, and NextExcl for MiniRep and the three +Current baselines, testing whether payload suppression and subsequent attacker exclusion persist after feedback access is shared.
- Appendix E.6 and Table 20: We reported attacked scores and CF-Harm separately for all seven original attack families, showing where the defense advantage remains and where current-feedback baselines perform better.
- Appendix E.2 and Table 15: We compare History-only and Current-quality-only variants to clarify the contribution of persistent reputation alongside the matched current-feedback baselines.
Part 3
Q2: The paper does not disclose the specific versions of the seven LLMs used as agents (only product families like "Qwen-Max" and "GLM" are named in Appendix C.2), despite agent capability being a core experimental variable. Similarly, the Qwen2.5-1.5B verifier training data size and label source (whether gold answers were used as supervision) are not reported (Appendix B.1.2).
A2: Thank you for pointing out these omissions. Appendix E.1 now provides the exact agent identifiers, verifier training counts, label sources, and training configuration. These details make the model pool and learned feedback behind the reported results explicit, providing the information needed to assess and reproduce the paper’s empirical findings.
Agent models and generation settings. The seven agent identifiers recorded in the experiments are:
| Name in the paper | API model identifier |
|---|---|
| Qwen-Max | qwen3.7-max |
| Qwen-Plus | qwen3.7-plus |
| Qwen-Flash | qwen3.7-flash |
| GLM | glm-5.2 |
| DeepSeek | deepseek-v4-flash |
| Kimi | kimi/kimi-k3 |
| MiniMax | MiniMax/MiniMax-M3 |
All seven are accessed through Bailian. Generation uses temperature 0.2, with Kimi set to 0.6 in the added debate and task-sample experiments. Output limits are 1,024 tokens for MATH and GoEmotions and 4,096 for HumanEval Pro.
Training data and supervision. The semantic verifier is based on Qwen/Qwen2.5-1.5B-Instruct. The pipeline first collects API responses from clean and attacked bootstrap runs on a calibration task pool separate from the aggregation evaluation; the default construction also adds task-grounded synthetic cohorts. Gold answers and hidden tests provide the correctness labels: exact label-set agreement for GoEmotions, answer equivalence for MATH, and hidden-test outcomes for HumanEval Pro. Correct responses receive label 1; incorrect or invalid responses receive label 0. Gold information is excluded from verifier inputs. The behavior probe has a separate label indicating an active attack payload, so an honest error receives a negative attack label.
Training size and configuration. The recorded data partitions are:
| Semantic-verifier partition | Response examples |
|---|---|
| Training | 10,470: 3,876 correct and 6,594 incorrect |
| Validation | 1,120 |
| Held-out-task test | 977 |
These partitions are grouped by canonical task, keeping response and attack variants of the same task together. Attack-family generalization is reported separately in Appendix D.1.
The training manifest records class-weighted focal cross-entropy with gamma 2, soft-ECE regularization with weight 0.05, and validation temperature scaling with T=0.7590. The released training recipe uses 4-bit LoRA, rank 16, scaling 32, two epochs, learning rate 2×10⁻⁴, batch size 1, gradient accumulation 16, and maximum sequence length 1,024.
Revisions:
- Appendix E.1 and Table 14: We listed the seven recorded API model identifiers and their generation settings, resolving the missing model-version information.
- Appendix E.1, “Training data and supervision”: We described the API-response collection pipeline and benchmark-derived correctness labels, explicitly stating how gold answers and hidden tests supply supervision and are excluded from verifier inputs.
- Appendix E.1, “Training size and configuration”: We added training, validation, and held-out-task counts, the training label balance, the recorded objective, and the released training recipe to make verifier training reproducible.
- Appendix B.1.2: We clarified that training, validation, and held-out-task partitions are grouped by canonical task, explaining how responses and attack variants from the same task are kept together.
Part 1
Q1: All methods share the same verifier feedback, but baselines (Beta, EigenTrust, etc.) only use it after task t to update weights for t+1, while MiniRep uses it before aggregation on task t (Eq. 34, 42). This temporal asymmetry conflates the reputation mechanism with the pre-aggregation screening advantage. An ablation where baselines also access current-task signals before aggregation is needed to isolate the contribution of the reputation design itself.
A1: Thank you for highlighting the timing of feedback. MiniRep retains its strongest defense advantage on MATH after the baselines receive current-task feedback; on GoEmotions it limits attack-induced harm, while current-aware baselines remain competitive on HumanEval Pro. Following your suggestion, we give the baselines access to current feedback before aggregation. This comparison preserves the original MATH advantage and clarifies which defense benefits remain on the other tasks. We compare the original baselines with their +Current versions below, reporting clean and attacked performance separately.
Each +Current version preserves its baseline’s historical update and multiplies its historical weight by max(10⁻⁶, 0.75 × semantic support + 0.25 × public support) before pool selection and aggregation. The methods share the same tasks, candidate proposals, configurations, and attacks.
Clean and attacked results. We evaluate 100 tasks per benchmark. Clean averages the three attack-free runs; Attacked averages the 21 attack conditions formed by three settings and seven attacks. Drop is Clean minus Attacked, calculated before rounding. CF-Harm measures the fraction of active-attack tasks whose score falls below the corresponding clean result. All scores and rates below are percentages; Drop is in percentage points.
GoEmotions: sample F1.
MATH: mathematical-equivalence accuracy.
HumanEval Pro: Pass@1.
Current feedback has different effects on clean performance and attack resistance. Adding Current improves the clean MATH accuracy of Single-Metric and Beta, yet their attacked accuracy falls and their CF-Harm rises. For EigenTrust, both clean and attacked accuracy improve. These differences show why clean and attacked results are both needed to assess the benefit of current feedback.
Defense against the original attacks. We also examined Strong, Rand, Div, Arith, Bound, OnOff, and Adapt separately. The results show where the original performance and defense advantages remain: