- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces BaRe-Mem, an online Bayesian reliability memory mechanism for multi-agent consultation that estimates advisor reliability from the central model's internal beliefs and verified interactions. It addresses two core challenges: how much to weight advisor responses, and when to consult versus reason autonomously. Validated across nine benchmarks, BaRe-Mem shows strong robustness against misleading information and significantly improves task completion efficiency in realistic agent team routing scenarios.
Critical strengths
-
BaRe-Mem's use of Bayesian linear regression with Kalman gain updates provides mathematically justified uncertainty quantification, adaptive weighting, and exact order-independent online updates. Appendix A.7 connects mathematical properties to behavioral predictions, providing an explanation why stronger autonomous models require more reliable evidence before consultation becomes beneficial.
-
The experiments are convincing across both capability regimes, showing BaRe-Mem successfully maintains performance by adaptively shifting to autonomous reasoning (e.g., dropping consultation to 21% under 100% misleading info in challenging regimes). Beyond synthetic corruption, the method shows tangible real-world utility in the MuSiQue agent routing application, improving task completion by 13-34% with fewer worker calls. The results support the method's applicability beyond response-level consultation.
Critical weaknesses
-
The paper lacks critical bounds on when the method fails, such as characterizing performance degradation below the 1% sparse verification mark. It assumes IID corruption and ignores adversarial scenarios such as structured misleading information explicitly targeting the central model's uncertainty, or "sleeper" advisors that build false reliability on easy tasks before failing critical ones. There is also no analysis of how quickly the system detects newly unreliable advisors or recovers when they improve, making it hard for practitioners to assess deployment risks.
-
Several crucial algorithmic decisions are presented without principled justification or targeted ablations to isolate their contributions. The paper does not explain why frozen pre-consultation representations are optimal compared to post-consultation states, why the attention modulation uses arbitrary logarithmic scaling and max-normalization rather than standard softmax, whether the strict linear assumption between representation and correctness holds in practice, or how sensitive the final system is to miscalibration of the decision threshold parameters ().
Questions for authors
-
What is the minimum verification rate (e.g., <1%) needed for stable performance, and how quickly does the method degrade below this critical threshold across different capability regimes?
-
How does BaRe-Mem perform under adversarial corruption patterns, such as strategically providing misleading information when the central model is uncertain, or poisoning attacks where advisors build false reliability on easy questions?
-
What is the theoretical or empirical rationale for using pre-consultation frozen representations rather than post-consultation states (which reflect advisor influence) for reliability estimation?
-
Can you provide ablations justifying the reliability-guided attention design, specifically comparing the logarithmic scaling and max-normalization against alternatives like linear scaling or standard softmax?
-
Given the strict linear assumption for Bayesian updates, have you explored the performance vs. computation trade-off of using non-linear models (e.g., kernel methods or neural predictors) to capture more complex reliability patterns?
7 comments
Response to Weakness 1, Questions 1 and Question 2: (2/2)
2. Robustness Under Adversarial Corruption Patterns
W1 & Q2: Adversarial Corruption and Sleeper Advisors. Evaluating BaRe-Mem when advisors provide misleading information targeting the central model’s uncertainty, act as sleeper advisors that build false reliability before failing, or form misleading majorities.
In Table R3, we evaluate four structured patterns on the first 6,000 questions of the capability-challenging regime with Qwen3-14B: uncertainty-targeted, where all advisors mislead where the central model is least sure of its own answer; oracle-targeted, where all advisors mislead on the exact questions where the central model is wrong on its own; sleeper advisor, where the highest-rated advisor is honest for the first half of the stream and then misleads where the central model is wrong; and misleading majority, where 4 or 5 of the 6 advisors mislead on every question.
Table R3: Accuracy (%) under adversarial corruption patterns on Qwen3-14B, where Harm % denotes questions where BaRe-Mem consults and loses a correct autonomous answer.
| Corruption pattern | No consultation | Question + Peers | Advisors + memory | BaRe-Mem | Consult % | Harm % |
|---|---|---|---|---|---|---|
| Random corruption, 100% | 78.9 | 54.9 | 56.0 | 79.6 | 7.8 | 0.2 |
| Oracle-targeted | 78.9 | 74.5 | 76.9 | 77.5 | 78.6 | 2.2 |
| Uncertainty-targeted | 78.9 | 68.3 | 70.6 | 80.3 | 64.8 | 0.5 |
| Sleeper advisor | 78.9 | 77.3 | 79.9 | 80.2 | 91.9 | 3.8 |
| 4/6 advisors misleading | 78.9 | 65.8 | 78.0 | 80.9 | 79.0 | 2.1 |
| 5/6 advisors misleading | 78.9 | 60.4 | 74.8 | 80.6 | 65.7 | 1.8 |
- Gains under uncertainty-targeted, sleeper, and majority attacks: BaRe-Mem stays strictly above
No consultationat78.9%under the uncertainty-targeted attack by+1.4%at80.3%, the sleeper attack by+1.3%at80.2%, and the 4/6 and 5/6 misleading majorities by+2.0%and+1.7%at80.9%and80.6%, whereasQuestion + Peersdrops to60.4%–77.3%, losing1.6%to18.5%relative toNo consultation. - Boundary under oracle-targeted corruption: The oracle-targeted pattern marks the structural limit: because advisors mislead on every question where consultation could rescue an autonomous error, consultation has almost nothing to rescue by construction. Even here, BaRe-Mem limits harm to
2.2%and stays within1.4%ofNo consultationat77.5%, compared with a4.4%drop forQuestion + Peersat74.5%.
3. Detection and Recovery Under Mid-Stream Advisor Shifts
W1: Detection and Recovery Dynamics. Measuring how quickly BaRe-Mem detects a newly unreliable advisor or recovers when an advisor improves.
In Table R4, the most trusted advisor switches behavior mid-stream on the same 6,000-question stream, with forgetting factor . Detection delay counts verified questions until its reliability estimate crosses the median of the other advisors; estimate after is the mean over the next 1,000 questions; and accuracy after change is measured over the first 500 questions after the switch.
Table R4: Mid-stream advisor shift dynamics and accuracy (%) on Qwen3-14B.
| Advisor change | Forgetting | Detection delay | Estimate before | Estimate after | BaRe-Mem after change | No consultation | Stable-period accuracy |
|---|---|---|---|---|---|---|---|
| Honest → misleading | 1 | 313 | 0.58 | 0.30 | 79.4 | 79.6 | 81.3 |
| Honest → misleading | 0.999 | 277 | 0.56 | 0.28 | 80.6 | 79.6 | 80.5 |
| Honest → misleading | 0.99 | 103 | 0.52 | 0.34 | 79.2 | 79.6 | 80.0 |
| Misleading → honest | 1 | > 3,000 | 0.22 | 0.48 | 82.2 | 79.6 | 80.1 |
| Misleading → honest | 0.999 | > 3,000 | 0.27 | 0.53 | 81.6 | 79.6 | 79.1 |
| Misleading → honest | 0.99 | > 3,000 | 0.34 | 0.52 | 81.2 | 79.6 | 78.4 |
- Fast detection and conservative trust restoration: A newly misleading advisor is detected within 313 verified questions at , or within 103 verified questions at with
1.3%lower stable-period accuracy before the switch at80.0%versus81.3%. When a misleading advisor becomes honest, its estimate rises from0.22to0.48within 1,000 questions without crossing the peer median before the stream ends, showing that trust is withdrawn faster than it is restored. In both directions, accuracy after the change stays within0.4%ofNo consultationor exceeds it by up to+2.6%.
Weakness 2 + Questions 3, 4, 5: (1/2)
We thank the reviewer for examining these algorithmic choices. Each design choice follows directly from the requirements of online consultation: pre-consultation frozen representations already encode all advisor answers while preserving causal ordering and an exact posterior; logarithmic max-normalization performs exact multiplicative reweighting inside softmax and outperforms alternative attention rules; the linear Bayesian model in representation space surpasses kernel and nearest-neighbor predictors with sub-2 ms updates; and the consultation gate tolerates perturbations to of up to within 1.5%.
1. Rationale for Frozen Pre-Consultation Representations
W2 & Q3: Pre-Consultation Frozen Representations vs. Post-Consultation States. Explaining the theoretical and empirical rationale for estimating reliability from frozen pre-consultation representations rather than post-consultation states.
Our pre-consultation belief representations already reflect advisor influence: as described in Section 3.1, the frozen central model encodes each candidate response in the context of the question after reading all advisor answers. Extracting these representations from the frozen model before generating the consulted answer satisfies three requirements:
- Causal ordering: Reliability estimates must exist before consultation generation so they can modulate advisor attention in Equation 7 and gate consultation in Equation 12. By contrast, post-consultation states exist only after generation and only on questions where the gate chooses to consult.
- Unconfounded targets: Post-consultation hidden states are already altered by the memory’s own attention bias , confounding reliability estimation with the current steering weights.
- Exact, order-invariant posterior: Keeping the feature extractor frozen maintains a fixed feature space across the stream, ensuring that the Gaussian posterior in Appendix A.4 remains exact and invariant to question arrival order.
2. Justification and Ablations of Reliability-Guided Attention
W2 & Q4: Logarithmic Scaling and Max-Normalization. Justifying the logarithmic scaling and max-normalization in reliability-guided attention and comparing against linear scaling and standard softmax.
Adding to pre-softmax attention logits in Equation 7 is equivalent to multiplicative power reweighting inside standard attention softmax:
Here, max-normalization sets so the highest-rated advisor retains its natural attention weight, and reduces to zero modification across all tokens whenever advisor estimates are equal, such as at initialization.
Table R5: Comparison of reliability-guided attention designs in the capability-challenging regime, reported as Advisors + memory / BaRe-Mem accuracy (%).
| Attention design | Qwen3-14B 0% | 100% | Phi-4 0% | 100% |
|---|---|---|---|---|
| Log-ratio + max-normalization, , ours | 75.7 / 75.9 | 55.1 / 68.1 | 72.3 / 71.9 | 52.7 / 64.9 |
| Without max-normalization | 73.9 / 74.2 | 54.9 / 67.5 | 67.6 / 67.5 | 54.2 / 63.5 |
| Linear scaling | 73.3 / 73.7 | 50.9 / 67.1 | 70.8 / 70.7 | 48.9 / 64.2 |
| Softmax over advisors | 75.2 / 75.3 | 56.6 / 68.1 | 69.3 / 69.7 | 54.4 / 64.2 |
| Log-ratio, | 74.5 / 74.8 | 52.4 / 67.0 | 70.4 / 70.0 | 50.3 / 63.7 |
| Log-ratio, | 75.3 / 75.7 | 54.2 / 67.8 | 71.3 / 70.9 | 51.8 / 63.8 |
| Log-ratio, | 75.3 / 75.9 | 56.0 / 68.8 | 73.2 / 73.1 | 54.6 / 65.6 |
- Consistent gains over alternative attention rules: At , logarithmic scaling with max-normalization achieves the highest BaRe-Mem accuracy in every column, outperforming removing max-normalization by up to
+4.4%on Phi-4 at 0% misleading information, linear scaling by up to+2.2%on BaRe-Mem and+4.2%onAdvisors + memory, and softmax over advisors by up to+2.2%on Phi-4 at 0% misleading information. Accuracy also increases smoothly from to , where exceeds our fixed default by up to1.2%, confirming that our results do not rely on hyperparameter tuning. Appendix D with Table 15 of the revised paper reports the sensitivity to and in blue.
Response to Weakness 1, Questions 1 and Question 2: (1/2)
We thank the reviewer for asking where the boundaries of our method lie. Evaluating these settings directly establishes three findings: BaRe-Mem degrades smoothly down to 0.1% verification without a critical threshold, stays above autonomous reasoning under uncertainty-targeted, sleeper, and majority attacks, and detects a newly unreliable advisor within 103 to 313 verified questions.
1. Sparse Verification Limits Below 1% Across Capability Regimes
We extend Figure 5 down to
0%,0.1%,0.25%, and0.5%verification across both capability regimes and three misleading information ratios in Tables R1 and R2. Both the reliability memory and the consultation gate are masked under the same sparse verification budget, evaluated on every 12th question, corresponding to 1,451 questions in the capability-challenging regime and 360 questions in the capability-supported regime.Table R1: BaRe-Mem accuracy (%) under sparse verification in the capability-challenging regime.
0%verification reduces to unadapted consultation as a lower limit, every verification rate from0.1%to100%improves accuracy over0%verification by+3.7%to+10.9%at 50% misleading information and by+9.0%to+17.5%at 100% misleading information, and the verification rate needed to matchNo consultationgrows smoothly with the misleading information ratio (0%under honest advisors,5%to10%at 50% misleading information, and100%at 100% misleading information).Table R2: BaRe-Mem accuracy (%) under sparse verification in the capability-supported regime.
No consultationat75.3%on Qwen3-14B and70.8%on Phi-4 across nearly all settings down.