acceptodds
Under review as a conference paper at ICLR 2027

Evidence and Position-Aware Chain-of-Agents for Long-Context Reasoning

Abstract

Large Language Models frequently encounter challenges when processing long-context tasks, often neglecting significant information that may be dispersed throughout extensive inputs. Traditional multi-agent frameworks, such as Chain-of-Agents, attempt to address this issue by partitioning long inputs among multiple agents. However, these approaches fail to account for the relative importance and positional vulnerability of the evidence when delegating tasks to agents. To overcome these limitations, we propose a framework that strategically assigns long-context evidence to appropriate agents based on its significance, susceptibility to positional displacement, and agent capabilities. Furthermore, a worker summary index provides non-sequential access to earlier evidence, while a scoped, non-regressive escalation mechanism corrects intermediate errors without degrading prior results. The effectiveness of the framework is assessed through rigorous evaluations on long-context question-answering and summarization tasks, utilizing established benchmarks including Qasper, QuALITY, HotpotQA, NarrativeQA, and GovReport. Performance analysis compares the proposed framework with baseline methods, such as Vanilla, CoA, Tree-of-Agents, G-Designer, and AgentRouter. Additionally, ablation studies are conducted to assess the impact of evidence importance and positional awareness on overall performance. Our findings indicate that the proposed framework provides a cohesive solution for evidence processing in long-context reasoning, significantly enhancing the retention of critical information and fostering improved multi-agent reasoning capabilities.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language models, Long-context tasks, Chain-of-agents, Multi-agent frameworks, Evidence processing, Positional vulnerability, Worker summary index, Multi-agent reasoning.

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_UGZdOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes EP-CoA for long-context reasoning. It combines evidence scoring, position-aware assignment, a worker summary index, and gated correction. Experiments cover five benchmarks and three language models. Results show competitive performance, while the clearest component gain comes from correction gating. Evidence placement itself does not show a clear benefit.

Critical strengths

The paper studies a useful problem and reports results across different tasks and model families. I appreciate the comparison with a strong single-agent baseline. The authors also report negative assignment results and examine serving variation. The correction analysis shows when rewriting helps or damages answers, which is useful for practice.

Critical weaknesses

The main evidence-placement idea is not supported by the reported ablations. Joint assignment performs worse when documents have several chunks (Table 7). The quality proxy and its implementation are not clearly specified. Also, two or four repeated runs on only two tasks cannot support the broad claims about reliable improvements.

Questions for authors

How is the quality proxy computed, and does it use any reference answers? Can you report paired confidence intervals and repeat all tasks? Why do Equations 4 and 5 use different importance terms? Can placement be tested without changing reading order? Please provide scorer prompts, thresholds, and all parameter settings.

Reviewer_qjzEOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

Previous work shows that language models often miss evidence when they deal with long inputs, especially important information in the middle of long text. To resolve this, prior methods like Chain of Agents split a long document into chunks and use separate workers to read the chunks and pass information through the worker chain. In these methods, all chunks are treated the same, which is not true in real cases. Instead, EP-CoA gives each chunk two scores. The first is how important it is, and the second is how likely it is to be missed. These two scores are multiplied into a risk score, and high-risk chunks are moved to safer places in the chain.

On five LongBench tasks with three 7–12B models and five baselines, EP-CoA is best in 6 of 15 settings. The rewrite step adds +0.50 ROUGE-L on GovReport with about 2.5% more calls.

Critical strengths

S1. The motivation of this paper is clear. Models often miss evidence in the middle of long inputs, and although earlier methods like Chain of Agents try to fix this by splitting up the context, they still treat every chunk the same. Giving each chunk its own score makes a lot of sense.

Critical weaknesses

W1. The main idea of this paper is giving different chunks different scores based on importance and position. However, the usefulness of this idea is not supported by the results. On HotpotQA and GovReport, documents fit in one chunk at the reported setting, so moving chunks does nothing. And when documents are split, it actually hurts: −5.37 on HotpotQA and −0.44 on GovReport. Appendix D also says it does not beat treating all chunks the same. The position score curve is also very ambiguous in the paper: the text first says that far slots are riskier, which makes sense, but in Eq. 2 the curve peaks in the middle, and γ is never reported.

W2. Most of the gain in the paper's results doesn't come from the proposed new methods. On LLaMA, the plain agent chain with the new chunker (V0) beats the full system on HotpotQA (61.30 vs. 59.28) and NarrativeQA (23.50 vs. 23.08). Table 4 also shows that the gain may actually come from chunking, not the proposed method. Running Chain of Agents with the same chunker would help clarify this.

W3. Some results look unreliable or inconsistent:

  • AgentRouter gets the same scores on all three models (Table 4).
  • Sec 5 says scoring needs no extra model calls, but Appendix B says it uses one LLM call per segment.
  • Appendix D says V0 and V1 are the same at the reported setting, but Table 5 gives different scores.
  • Sec 5 says extra calls are 48–62%, but Table 5 shows +64% on GovReport.

Overall, this paper seems unfinished and needs more polish.

Questions for authors

see weakness

Reviewer_35B2OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces EP-CoA, an extension of the Chain-of-Agents (CoA) framework for long-context reasoning. It incorporates evidence importance and positional vulnerability scoring, capacity-constrained assignment, a Worker Summary Index (WSI), and scoped error correction with an Escalation Non-Regression (ENR) gate. The paper also proposes Capacity-Aware Sentence Packing (CASP) to prevent information loss during text chunking. Experiments are conducted on five LongBench tasks using three 7B–12B backbone models, with 80 test samples per task. EP-CoA achieves the highest score in 6 out of 15 settings. However, ablation results show that the proposed joint assignment fails to outperform uniform assignment in the reported multi-chunk experiments, while ENR shows some potential benefits.

Critical strengths
  1. ENR addresses the important problem of performance degradation caused by self-correction in multi-agent pipelines. On GovReport, it reduces the regression rate from 48.6% to 31.0% and improves ROUGE-L by 0.50, with only approximately 2.5% more model calls relative to V2.
  2. Appendix A identifies a practical engineering issue: silent text loss caused by inconsistent token budget calculations, offering useful insights for implementing long-context multi-agent systems. Appendix D also transparently reports the negative results of the proposed joint assignment mechanism.
Critical weaknesses
  1. In Table 5, V0 achieves 61.30 on LLaMA HotpotQA, outperforming the full system V3 (59.28) and Vanilla (54.62). The performance gains therefore cannot be clearly attributed to the proposed mechanisms, and the Worker Summary Index (WSI) lacks an independent ablation. Appendix D also states that V0 and V1 should be identical in single-chunk settings, yet Table 5 reports a difference of approximately 2.98 points.
  2. Appendix D reports an average of only 1.00 chunk for HotpotQA and GovReport under the main configuration, leaving no meaningful role for joint assignment. When multiple chunks are used, joint assignment underperforms uniform assignment by 5.37 and 0.44 points, respectively. The paper provides no positive experimental evidence for its core evidence and position-aware assignment mechanism, making the corresponding contribution claims insufficiently supported.
  3. The quality proxy Q used by ENR is not defined, and the denominators for Benefit/Regr. are unclear. The number of escalation triggers differs between V2 and V3, preventing a controlled assessment of the gating mechanism alone. The regression rate also increases on NarrativeQA after adding ENR. These results do not sufficiently support the claimed reliability of ENR.
  4. The positional vulnerability function in Equation (2) is inconsistent with its textual description, while Equations (4) and (5) use different optimization objectives. Several key parameters remain undefined. In Algorithm 2, sentences removed during backtracking appear not to be preserved, potentially causing text loss and contradicting the stated purpose of CASP. These issues undermine reproducibility.
  5. AgentRouter reports identical scores across all five tasks for three different backbone models, raising concerns about the baseline implementation or evaluation. The effective token budgets and tuning configurations for CoA and EP-CoA are insufficiently specified. Ablations cover only three tasks on one backbone, with limited analysis of chunk counts, truncation rates, and longer context window settings.
  6. Additional diagrams illustrating key mechanisms and improvements to figure clarity and resolution would enhance the presentation.
Questions for authors
  1. Why does V0 outperform the full system V3 on HotpotQA in Table 5, and why do V0 and V1 produce different scores despite being described as identical by construction in the single-chunk setting?
  2. Could you provide controlled experimental evidence demonstrating a positive contribution from the proposed evidence and position-aware assignment mechanism in multi-chunk settings?
  3. How is the quality proxy Q for ENR defined, and can the independent effect of the ENR gate be evaluated using a fixed set of candidate rewrites?
  4. Could you provide paired bootstrap confidence intervals for the main experimental results and ablation studies?
  5. Why does AgentRouter report identical scores across all three backbone models?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.