- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper introduces Epistemic Context Learning (ECL), a two-stage framework that enables individual LLM agents to assess peer reliability from historical interactions and use this information in subsequent reasoning. The authors introduce a Peer Recognition Reward (PRR) to supervise trust estimation and present experimental results on MMLU-Pro and GPQA across eight models under natural and adversarial peer settings.
Overall, the paper addresses an important problem in LLM-based multi-agent reasoning. The motivation is clear, and the diagnostic experiments provide useful insights into how agents assess and use peer information.
However, the experimental evaluation is not sufficiently comprehensive, and several methodological details need further clarification.
Critical strengths
-
Addresses an important and relatively underexplored problem: how an individual LLM agent should assess peer reliability and selectively incorporate peer information.
-
The diagnostic experiments identify meaningful failure modes, including limited use of historical reliability signals and over-reliance on incorrect peer consensus.
-
The evaluation covers multiple model families, natural and adversarial settings, and analyses of peer counts, history lengths, and inference overhead.
Critical weaknesses
-
It is not explicitly clear whether the two stages use the same LLM with shared parameters or separate models. Although the appendix provides prompts, the main text gives limited explanation of how historical interactions are transformed into trust representations and how the resulting representation is used by Stage 2.
-
The paper emphasizes the selective utilization of peer information. However, the training objective of Stage 1 only rewards identifying the single most reliable peer. This reward design is overly simplistic and does not adequately capture the varying reliability of different peers or how their information should be selectively utilized.
-
The authors claim that ECL can be applied on top of any existing multi-agent protocol. However, the experiments are limited to aggregation-based settings, without evaluating ECL in combination with existing multi-agent frameworks or topology optimization methods such as G-Designer.
-
The authors only report the additional inference time overhead of ECL compared to conventional methods, without providing a comparison of inference-time token consumption token consumption, which is important for evaluating the computational efficiency of the proposed framework.
Questions for authors
-
Are Stage 1 and Stage 2 implemented using the same LLM with shared parameters, or are they two separately trained models? How are the two stages optimized using their respective rewards?
-
Could the authors provide additional experimental results to demonstrate whether ECL remains effective when integrated with existing multi-agent frameworks or topology optimization methods?
-
Could the authors provide a comparison of inference-time token consumption, including both input and output tokens, between ECL and existing methods to better evaluate its computational efficiency?
6 comments
Rebuttal to Reviewer Jb5t (2/2)
W3 & Q2: Could the authors provide additional experimental results to demonstrate whether ECL remains effective when integrated with existing multi-agent frameworks or topology optimization methods?
A3: Thank you for pointing out this ambiguity. When stating that ECL can be applied on top of existing multi-agent protocols, our intended meaning was architectural orthogonality: because topology design and communication protocols govern who communicates with whom at the system level, whereas ECL governs how an individual receiving agent processes incoming peer responses at the local level, ECL is interface-compatible with any protocol that supplies peer outputs, without implying that empirical gains have already been established across all dynamic topologies.
We agree that our current experiments focus on single-round aggregation rather than end-to-end multi-round graph topologies (such as G-Designer), which would require reconstructing multi-round interaction histories for each dynamic graph structure. Within our aggregation setting, we have already investigated its robustness across different multi-agent configurations in Section 6.3:
- Number of peers (Table 11):
- History length (Table 12):
ECL shows consistent gains over AG across all these structural variations. To prevent any over-interpretation of our claims, we have revised Section 1 to explicitly distinguish architectural orthogonality from our current empirical scope (single-round aggregation), and highlighted end-to-end integration with dynamic topology optimization as an important direction for future work in Section 7.
W4 & Q3: Could the authors provide a comparison of inference-time token consumption, including both input and output tokens, between ECL and existing methods to better evaluate its computational efficiency?
A4: Thank you for your interest in the token efficiency of our method. We compare the inference-time input, output, and total token consumption of ECL against AG and 1S below.
Mean input tokens
| Method | GPQA (Nat.) | GPQA (Adv.) | MMLU-Pro (Nat.) | MMLU-Pro (Adv.) | Avg. |
|---|---|---|---|---|---|
| AG | 3914.6 | 1855.2 | 2227.7 | 1306.1 | 2325.9 |
| 1S | 5319.5 | 3312.5 | 3756.2 | 2775.4 | 3790.9 |
| ECL (E) | 5528.3 | 3572.5 | 3901.8 | 3021.0 | 4005.9 |
Mean output tokens (including reasoning)
| Method | GPQA (Nat.) | GPQA (Adv.) | MMLU-Pro (Nat.) | MMLU-Pro (Adv.) | Avg. |
|---|---|---|---|---|---|
| AG | 3349.1 | 6257.2 | 1466.5 | 1855.9 | 3232.2 |
| 1S | 4320.6 | 7179.0 | 2044.8 | 1605.9 | 3787.6 |
| ECL (E) | 11292.0 | 15786.4 | 4310.1 | 5241.9 | 9157.6 |
Mean total tokens
| Method | GPQA (Nat.) | GPQA (Adv.) | MMLU-Pro (Nat.) | MMLU-Pro (Adv.) | Avg. |
|---|---|---|---|---|---|
| AG | 7263.6 | 8112.3 | 3694.3 | 3161.9 | 5558.0 |
| 1S | 9640.1 | 10491.6 | 5801.0 | 4381.3 | 7578.5 |
| ECL (E) | 16820.3 | 19358.9 | 8211.9 | 8262.9 | 13163.5 |
Analysis of Token Overhead: Compared to the history-aware 1S baseline, ECL increases input tokens by only 5.7%, because the distilled trust summary passed from Stage 1 to Stage 2 is highly compact (consistent with Appendix A.7). The 2.42$\times$ increase in output tokens (resulting in 1.74$\times$ total tokens) stems primarily from the internal CoT generated during Stage 1 to verify historical peer accuracy without ground-truth labels, alongside Stage 2's final reasoning.
Notably, this extra test-time compute is substantially more effective than alternative inference-scaling strategies: as shown in Table 10 (formerly Table 9), 3-round self-reflection incurs roughly 3$\times$ the compute of single-pass inference yet yields markedly smaller accuracy gains than ECL (e.g., 86.7% vs. 98.9% for Gemini 3 Flash on MMLU-Pro). Meanwhile, ECL incurs only 1.40$\times$ wall-clock latency over AG (Appendix A.7), representing a favorable efficiency-effectiveness trade-off. We have incorporated this token consumption analysis into Appendix A.7 (Tables 20–22) in the revised manuscript.
Sincerely Looking Forward to Your Further Feedback
Dear Reviewer Jb5t,
Thanks again for your insightful and helpful comments! We have submitted our response and we hope it has addressed all your concerns and questions. Since the end of discussion period is coming soon, we sincerely look forward to your further suggestions to our response, and we are open to any discussion to improve our paper.
Best regards,
Authors
Rebuttal to Reviewer Jb5t (1/2)
Thank you for your helpful review and for recognizing the importance of our problem formulation, the diagnostic failure modes we identify, and our comprehensive evaluation across model families and configurations. We address your concerns and questions one by one as follows:
W1 & Q1: Are Stage 1 and Stage 2 implemented using the same LLM with shared parameters, or are they two separately trained models? How are the two stages optimized using their respective rewards?
A1: Thank you for raising your question and we apologize for your confusion. Stage 1 and Stage 2 are implemented using the same LLM with shared parameters. Regarding RL optimization:
On large models that are not trained with RL, the difference between ECL (I) and ECL (E) is only the different requirement of stage 1 summary (and correspondingly, stage 2 input) in prompts.
The generated contents outside the reasoning block will replace {output_stage1} in the stage 2 prompt, i.e., the free-form summary in ECL (I) and the conclusion in ECL (E).
We have added this explicit description of parameter sharing and inter-stage representation passing to the revised manuscript.
W2: PRR is overly simplistic and does not adequately capture the varying reliability of different peers or how their information should be selectively utilized.
A2: Thank you for this comment. We agree that identifying the single most reliable peer does not fully capture the complexity of trust modeling. However, PRR is intentionally designed as a minimal proof-of-concept for explicitly supervising trust estimation, rather than a comprehensive solution to modeling peer reliability.
This design is motivated by two observations in our diagnostic experiments (Section 4):
More importantly, ECL does not restrict trust modeling or information utilization to a single peer, as our ECL (I) variant allows free-form reliability profiles capturing the varying reliability of different peers. We therefore view PRR as a minimal but effective implementation that establishes the value of explicit trust supervision. Designing richer rewards for more fine-grained trust modeling is a promising extension, but is not necessary for validating our central hypothesis.