acceptodds
Under review as a conference paper at ICLR 2027

LongHarness: Advancing Agentic Reasoning over 100M-Token Corpora via Stateful Evidence Management

10 views

Abstract

Corpus-level long-context reasoning requires agents to acquire and integrate evidence across document collections that exceed their context windows. Existing agent harnesses enable incremental access to these corpora, but acquired evidence and evolving conclusions often remain entangled in a growing interaction history. This implicit state management can lead to two coupled failure modes: unbounded exploration, where retrieval expands the interaction history without resolving decision-relevant information gaps, and evidence-state drift, where previously acquired evidence or subsequent corrections fail to inform the current conclusion. We present LongHarness, a model-agnostic harness that makes the interaction between acquired evidence and evolving conclusions explicit. Its core mechanism, Evidence–Conclusion Coevolution, uses unresolved conclusions to guide exploration toward missing information, while newly acquired evidence strengthens or revises the conclusions that directed the search. LongHarness couples this process with Evidence Lifecycle Management, which keeps accumulated evidence current and traceable throughout the interaction. Together, these mechanisms focus retrieval on remaining information needs and keep conclusions aligned with current evidence. We evaluate LongHarness on OfficeQA Pro V1 and V2, AA-LCR, and LongBench v2 using GLM-5.3-Flash and Qwen3.8-Flash. Compared with direct reasoning using the same backbone, LongHarness improves GLM-5.3-Flash exact match on OfficeQA Pro from 15.8% to 60.2% on V1 and from 4.4% to 61.1% on V2. Against state-of-the-art agent harnesses, it surpasses Claude Code by 12.0 percentage points on V1 and Codex by 18.9 points on V2. With Qwen3.8-Flash, LongHarness reaches 61.7% on V1, outperforming Claude Code by 8.3 points and Codex by 12.0 points.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long Context Reasoning, Agent Harness

Reviews

Among:
Shape:
1234this paper, Top 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.67 over 3 reviews, above 99% of the others.
Reviewer_m1ZcOpenReview ↗
Rating
4
Confidence
3(+1)
Informativeness
3(+1)
Presentation
3
Soundness
3
Read review
Paper summary
  • identifies 2 failures: unbounded exploration, evidence-state drift
  • Evidence–Conclusion: Coevolution and Evidence Lifecycle Management: model-agnostic harness: interactions between evedence and conclusions
Critical strengths
  • propose model agnostic harness, and evaluates its against huge corpus
  • idea is clean and sensible: keep a source-linked evidence board, and let open questions drive the search
  • tested on OfficeQA Pro
  • more efficient on harder V2 tasks
Critical weaknesses
  • a baseline would be to give these LLM models with the same tools and see how that performs (whatif gains coming from tools rather than notebook structure)
  • Claude Code and Codex are tuned for Claude and GPT models. instead running them on GLM and Qwen likely handicaps them (missing frontier model results) which shrinks the claim of model agnostic with just mid sized models being tested
  • there needs to analysis on why LongHarness with glm scores higher on the "harder" V2 > V1, while Qwen drops instead. (discuss deeper on why)
  • token use, cost and other details should be added
  • how you tackle data contamination as facts could be memorized
Questions for authors
3 comments
AuthorsOpenReview ↗

Response to Reviewer m1Zc [Part1]

We thank Reviewer m1Zc for the comments on tool fairness, model adaptability, computational overhead, and data contamination. We provide comparisons using the LongHarness tool set, native frontier-model comparisons, and token-usage statistics. We also analyze the interaction trajectories of GLM-5.3-Flash and Qwen-3.8-Flash and examine data-contamination concerns through the evaluation protocol and trajectory evidence.

Q1: Should baselines receive the same tools to distinguish the contributions of the tool set and the harness's state structure?

A1: Thank you for this suggestion. To directly test gains beyond the tool set, we provide controlled experiments using the same basic tools. Claude Code and Codex are given LongHarness tools, and we include a LongHarness variant that uses only these tools without evidence–conclusion coevolution or evidence lifecycle management. This tests the contributions of state structure and management in the full method. LongHarness-tools-only denotes the variant using only the basic LongHarness tool set. All these tool-alignment experiments use GLM-5.3-Flash as the base model. The reported metric is Exact accuracy on OfficeQA V1 and V2.

MethodOfficeQA V1OfficeQA V2
LongHarness60.2%61.1%
LongHarness-tools-only52.6%46.7%
Claude Code48.1%35.6%
Claude Code + LongHarness tools51.9%40.0%
Codex46.6%42.2%
Codex + LongHarness tools51.9%35.6%

LongHarness tools improve Claude Code on both V1 and V2 and improve Codex on V1, while Codex declines on V2. Tools designed for corpus-level reasoning can therefore improve the fit of general-purpose file and shell operations to the task, although gains also depend on the harness and task. A suitable tool set with a basic interaction loop already achieves performance comparable to or higher than more complex harnesses such as Codex and Claude Code, demonstrating that appropriate tools can provide effective evidence access and processing for complex corpus-level reasoning. Building on these tools, full LongHarness introduces evidence–conclusion coevolution and evidence lifecycle management, exceeding LongHarness-tools-only by 7.6 and 14.4 percentage points on V1 and V2. This reflects the additional benefit of managing evidence and conclusions across turns and supports the design of LongHarness, which strengthens long-horizon reasoning through state management on top of task-adapted tools and a simple interaction loop. We have added the tool-set comparisons in Appendix C.3 and the definition and results of LongHarness-tools-only in Section 3.3 and Table 2.

Q2: Does the lack of native frontier-model results for Claude Code and Codex weaken the model-agnostic claim?

A2: Thank you for pointing out the limited coverage of native frontier models in the original experiments. Our goal is to compare harnesses rather than model capabilities. We therefore held the backbone fixed and minimized harness-specific adaptation to examine plug-and-play performance. Claude and GPT models often undergo substantial adaptation to their corresponding harnesses, so evaluation in their native frameworks may combine model-adaptation benefits with the effects of the harness itself. GLM-5.3-Flash and Qwen-3.8-Flash have less comparable harness-specific adaptation and are therefore useful for controlled harness comparisons. We nevertheless agree that coverage limited to mid-sized models is insufficient. We therefore provide native and cross-harness combinations using Claude Opus 5.5 and GPT-6.1-Sol on OfficeQA Pro V1:

HarnessClaude Opus 5.5GPT-6.1-Sol
Claude Code65.41%64.66%
Codex62.41%66.17%
LongHarness71.43%72.93%

LongHarness achieves 71.43% with Claude Opus 5.5 and 72.93% with GPT-6.1-Sol, exceeding the corresponding native Claude Code and Codex combinations by 6.02 and 6.76 percentage points, respectively. These results extend the controlled comparisons with GLM-5.3-Flash and Qwen-3.8-Flash to native Claude and GPT frontier models. LongHarness achieves higher accuracy across both model families and their native harness combinations, indicating that its effectiveness is not confined to a particular model family or a single adaptation setting.

AuthorsOpenReview ↗

Response to Reviewer m1Zc [Part2]

Q3: Why does LongHarness perform better on the harder OfficeQA Pro V2 with GLM-5.3-Flash, but worse with Qwen-3.8-Flash?

A3: Thank you for this question. We further analyze interaction trajectories to understand why GLM-5.3-Flash improves slightly on OfficeQA V2 while Qwen-3.8-Flash declines. The difference is associated with how the two backbones use the harness's persistent evidence states.

The gap is concentrated in tasks requiring integration across multiple documents. Questions requiring at least 5 gold documents account for only 3% of V1 but 58% of V2. Among the 38 V2 questions requiring at most 4 documents, GLM-5.3-Flash and Qwen-3.8-Flash achieve 60.5% and 57.9%, respectively. Among the 52 questions requiring at least 5 documents, they achieve 61.5% and 46.2%, widening the gap to 15.3 percentage points. On these 52 questions, both models have a median retrieval count of 12.5 and cover 8 documents, but GLM-5.3-Flash records 145 numerical values per question in the Evidence Bank, compared with 96.5 for Qwen-3.8-Flash. As document requirements increase from at most 4 to at least 5, the former's number of recorded values per read document remains stable, changing from 18.5 to 18.2, whereas the latter's falls from 14.7 to 10.5. This suggests that GLM-5.3-Flash continues to convert intermediate results into persistent evidence that can be reused across turns as document requirements grow. Qwen-3.8-Flash instead records less evidence per document and relies more on the interaction context to retain intermediate results, making it harder to accumulate and reuse information reliably across multiple documents.

We therefore interpret GLM-5.3-Flash as adapting more actively to the harness: it makes fuller use of persistent evidence states to support subsequent reasoning, whereas Qwen-3.8-Flash more often reuses intermediate results directly from context. As V2 substantially increases the demand for multi-document integration, the latter approach becomes less reliable at maintaining and reusing information, which we view as an important contributor to Qwen-3.8-Flash's decline. In future work, we will explore incorporating persistent-state use into post-training objectives so that different backbones can benefit more consistently from the harness.

Q4: Should token usage, costs, and other computational details be reported?

A4: Thank you for this suggestion. We have added mean cumulative input, output, and total tokens per question for each method, together with an analysis of structured-state overhead, in Appendix C.4. In LongHarness requests, the Evidence State occupies approximately 748 tokens and the Conclusion State approximately 195 tokens per turn, totaling approximately 944 tokens per turn. Repeated injection across requests accounts for approximately 18,880 tokens per question, or 1.5% of cumulative input tokens. Structured states therefore account for only a small fraction of cumulative context overhead. The table reports mean token consumption per question for each harness on OfficeQA V1 with GLM-5.3-Flash as the base model.

MethodExact (%)Mean cumulative input tokensMean cumulative output tokensMean cumulative total tokens
LongHarness60.21,296,94449,9021,346,845
LongHarness (Last 5-Interactions)58.7607,11754,987662,104
Claude Code48.1515,94417,754533,698
Codex46.62,048,3367,0682,055,405

To examine performance at comparable computational cost, we additionally retain only the last 5 complete interactions, including chain-of-thought, ordinary responses, tool calls, and tool results. This setting uses an average of 662,104 total tokens per question, of the same order as Claude Code's 533,698 and below Codex's 2,055,405. It achieves 58.7% Exact, exceeding Claude Code's 48.1% and Codex's 46.6% by 10.6 and 12.1 percentage points, respectively. Even without the full interaction history, the model can maintain evidence and conclusions across turns through the Evidence Bank and Conclusion State and complete the reasoning task. This indicates that structured persistent states can support long-horizon reasoning with lower context cost.

AuthorsOpenReview ↗

Response to Reviewer m1Zc [Part3]

Q5: How do you address possible data contamination from facts memorized during pretraining?

A5: Thank you for raising this concern. We distinguish answer leakage during evaluation from potential memorization during pretraining. Regarding the former, all harnesses receive only the question and the complete corpus, without gold documents, reference answers, or source annotations. Benchmark answers and annotations are not embedded in the harness. LongHarness retains its advantage over the baseline harnesses when using the same backbone, questions, corpus, and scorer.

Regarding pretraining memorization, complex corpus-level reasoning requires more than retrieving facts. It also requires locating and aligning data across documents and then performing computations. With GLM-5.3-Flash as the backbone, LongHarness correctly answers 80 and 55 questions on OfficeQA Pro V1 and V2, respectively. Among these successful trajectories, 86% and 91% of final answers cite at least two different documents, with a median of 3. Computation occurs before submission in 98% and 95% of cases, and the final numerical value does not appear verbatim in any cited passage in 91% and 95% of cases. Even if individual values appeared in pretraining data, LongHarness still needs to locate the correct tables, years, statistical definitions, and versions among 697 or 1,435 documents and then integrate the evidence and perform the required computations. Taken together, the controlled comparisons with the same backbone and this trajectory evidence lead us to attribute the performance advantage of LongHarness primarily to effective organization of documentary evidence and cross-document reasoning, rather than answer leakage or direct recall of pretrained facts.

Reviewer_uRRyOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies corpus-level long-context reasoning, where LLM agents need to retrieve and integrate information from document collections that exceed their context windows. The authors identify two main limitations of existing agent harnesses: (1) unbounded exploration, where agents repeatedly search and read documents without resolving important information gaps, and (2) evidence-state drift, where previously retrieved evidence or subsequent corrections are not properly incorporated into final conclusions. The paper aims to address these issues by improving how agents manage accumulated evidence and evolving conclusions throughout the reasoning process.

To achieve this, the authors propose LongHarness, a model-agnostic agent harness that features two main mechanisms. (1) Evidence–Conclusion Coevolution maintains an explicit Conclusion State that tracks established claims, supported hypotheses, and unresolved information gaps. These gaps guide subsequent retrieval, while newly acquired evidence is used to update existing conclusions. (2) Evidence Lifecycle Management introduces an Evidence Bank that stores source-linked evidence and supports adding, revising, and retracting evidence as new information becomes available. Together, these mechanisms create a feedback loop between evidence acquisition and conclusion refinement. The whole sisyte aims to reduce redundant exploration and keep conclusions consistent with the latest evidence.

The authors evaluate LongHarness on 4 benchmarks (OfficeQA Pro V1/V2, AA-LCR, and LongBench v2) using two backbone models, comparing against direct LLM reasoning and three strong agent harness baselines (Codex, Claude Code, and RLM). LongHarness achieves the best results across all four benchmarks with both backbones. In particular, with GLM-5.3-Flash, it improves exact-match accuracy to 60.2% on OfficeQA Pro V1 and 61.1% on V2, outperforming the strongest competing harnesses by 12.0 and 18.9 percentage points, respectively. Ablation studies and trajectory analyses further support the effectiveness of explicit state management, conclusion-guided retrieval, and evidence revision in improving reasoning accuracy and reducing unnecessary exploration.

Critical strengths
  • The paper is well written and easy to follow. The motivation is clear, and the proposed framework is explained with intuitive examples and illustrations. The diagram and charts are informative and easy to understand.

  • The experimental evaluation is comprehensive. The authors evaluate the LongHarness on 4 benchmarks with 2 backbone models and compare against competitive agent harnesses, including Codex, Claude Code, and RLM. The ablation studies also cover different components of the framework, and the trajectory analysis and case studies provide further insights into the results.

  • The proposed method is simple and intuitive, yet achieves strong empirical results. The improvements are consistent across different benchmarks and backbone models. The results in Table 3 also show that LongHarness improves multiple stages of the reasoning process, rather than only the final answer accuracy.

Critical weaknesses
  • The technical novelty of the proposed method seems somewhat limited. The main ideas, including persistent evidence storage, explicit conclusion tracking, gap-guided retrieval, and evidence revision, are relatively straightforward and closely related to existing agent memory and retrieval approaches. While combining these components into a unified framework is useful, many key decisions, such as when to stop exploration and how to revise evidence, are still largely left to the backbone LLM. I would like to see a more systematic design for these decisions, rather than relying primarily on the LLM's own judgment.

  • The computational cost of LongHarness is not sufficiently evaluated. The method achieves impressive accuracy improvements, but it is unclear how much additional computation is required to obtain these gains. Figure 3 reports the number of model turns, but this alone does not reflect the actual inference cost, token consumption, or latency. In particular, LongHarness retains the full interaction history while maintaining additional evidence and conclusion states. I would like to know whether the substantial accuracy improvements come at the expense of significantly higher computational costs, and how LongHarness compares with other harnesses under comparable compute or cost budgets.

  • The definition of unresolved gaps and the stopping criteria for exploration are not sufficiently clear. LongHarness relies on the LLM to identify information gaps and determine when enough evidence has been collected, but there is no explicit mechanism to ensure that important gaps have not been overlooked. For example, an agent may conclude that a supplier cannot meet a delivery deadline based on one document, while another unexplored document contains a more recent correction. The agent may then stop searching without realizing that its conclusion is incomplete. I would like to see more discussion or analysis of how LongHarness balances avoiding unnecessary exploration with the risk of prematurely stopping before finding important evidence.

Questions for authors
  • How does LongHarness handle conflicting evidence? When newly retrieved evidence contradicts an existing statement, how does the agent decide whether to replace, retract, or retain the original evidence?

  • How does LongHarness handle temporally evolving evidence? For example, two financial reports may contain different values because they refer to different periods, or because a later report corrects an earlier one. Does the Evidence Bank explicitly account for such temporal or version relationships? While Table 3 evaluates overall evidence editing success, it would be interesting to understand how the method handles these specific cases.

6 comments
AuthorsOpenReview ↗

Response to Reviewer uRRy [Part1]

We thank Reviewer uRRy for the constructive comments. We provide further explanations of state maintenance and operational constraints, token statistics, trajectory analyses, and concrete cases involving conflicting and evolving evidence to address the technical mechanisms, computational cost, exploration stopping, and evidence revision in LongHarness.

Q1: Is the technical novelty of LongHarness limited, and do exploration stopping and evidence revision depend too heavily on the backbone LLM?

A1: Thank you for this comment. We agree that persistent evidence storage, explicit conclusion tracking, gap-guided retrieval, and evidence revision are individually simple and related to existing memory and retrieval approaches. LongHarness, however, targets more complex corpus-level reasoning. During long-horizon agentic interactions, existing ways of organizing evidence can be overwhelmed by the accumulating context and become difficult to maintain, leading to unbounded exploration and evidence-state drift. Our contribution is to organize evidence and conclusions as two persistent, coevolving states, making their interaction explicit and controllable in corpus-level reasoning and maintaining it through evidence lifecycle management.

Key decisions, including when to stop exploring and how to revise evidence, still require the backbone LLM to consider task semantics and current evidence and cannot readily be delegated entirely to fixed quantitative criteria. LongHarness reduces the model's memory and state-management burden through systematic state maintenance and operational constraints, allowing it to focus its reasoning budget on these decisions. (i) Evidence revision. To prevent provenance loss or inadvertent overwriting through unrestricted editing, revisions are constrained to source-referenced upsert/revise operations. Superseded entries are marked superseded, and the controller validates references and rejects invalid operations. The model determines whether evidence conflicts or requires correction without having to maintain all version relationships itself. (ii) Exploration stopping. To avoid relying solely on an expanding interaction history to reconstruct what is known and unknown, each turn explicitly presents evidence, conclusions, hypotheses, and unresolved gaps. Before submission, the agent checks for evidence or gaps not yet reflected in its conclusion, and the final state is recorded. These structured states and checks support judgments about whether the evidence is sufficient and make the basis for stopping inspectable. By making the evidence, states, and constraints required for these decisions explicit, LongHarness helps the backbone LLM revise evidence and stop exploration more consistently and in a way that can be inspected. We have added the exploration stopping conditions in Section 2.2, clarifying their relationship to unresolved gaps and available evidence.

AuthorsOpenReview ↗

Response to Reviewer uRRy [Part2]

Q2: What are the computational cost and token consumption of LongHarness, and how does it compare with other harnesses at comparable cost?

A2: Thank you for this suggestion. We agree that the number of model turns alone does not fully characterize inference cost. We therefore report cumulative input and output tokens for each method and analyze both the context overhead of structured states and the accuracy and token consumption obtained by restricting the interaction-history window.

The Evidence Bank and Conclusion State retain the key evidence, sources, conclusions, hypotheses, and unresolved gaps needed for current reasoning. Statistics over the requests in LongHarness trajectories indicate that the Evidence State occupies approximately 748 tokens and the Conclusion State approximately 195 tokens per turn, totaling approximately 944 tokens per turn. Repeated injection across requests accounts for an average of approximately 18,880 tokens per question, or 1.5% of cumulative input tokens. The direct overhead of structured states therefore accounts for only a small fraction of cumulative input.

The table reports mean token consumption per question for each harness on OfficeQA V1 with GLM-5.3-Flash as the base model.

MethodExact (%)Mean cumulative input tokensMean cumulative output tokensMean cumulative total tokens
LongHarness60.21,296,94449,9021,346,845
LongHarness (Last 5-Interactions)58.7607,11754,987662,104
Claude Code48.1515,94417,754533,698
Codex46.62,048,3367,0682,055,405

Compared with Codex, full LongHarness reduces cumulative input tokens by 36.7%, improves Exact accuracy by 13.6 percentage points, and reduces total input and output tokens by 34.5%. To address the question of performance at comparable computational cost, we additionally retain only the last 5 complete interactions on OfficeQA V1, reducing the overhead of historical context. This setting uses an average of 662,104 total tokens per question, of the same order as Claude Code's 533,698 and below Codex's 2,055,405. It achieves 58.7% Exact, exceeding Claude Code and Codex by 10.6 and 12.1 percentage points, respectively, while remaining only 1.5 points below full LongHarness. Even without the full interaction history, the model can maintain evidence and conclusions across turns through the Evidence Bank and Conclusion State and complete the reasoning task. This indicates that structured persistent states can support long-horizon reasoning with lower context cost. We have added the token-usage table, state overhead, and cost–accuracy analysis for the last-five-interaction setting in Appendix C.4.

AuthorsOpenReview ↗

Response to Reviewer uRRy [Part3]

Q3: How are unresolved gaps defined, and how does LongHarness balance avoiding unnecessary exploration against premature stopping?

A3: Thank you for highlighting the need to clarify unresolved gaps and exploration stopping criteria. Unresolved gaps are information not yet established by evidence that could change the final answer, including unexplained disagreements or corrections across sources. LongHarness explicitly maintains these gaps in the Conclusion State and updates them in conjunction with the Evidence Bank. Gaps that could change the answer guide further retrieval, and exploration stops when the identified gaps no longer affect the final answer. This provides an inspectable basis for judging evidence sufficiency, reducing the need to repeatedly reconstruct known and unknown information from a long history and thereby helping avoid unnecessary exploration and premature stopping. We have clarified the definition of unresolved gaps and exploration stopping conditions in the Conclusion-guided exploration paragraph of Section 2.2.

To examine their practical effects, we use DeepSeek-V4.1-Flash-1017 to audit each question's trajectories with method names hidden. All audited trajectories were generated by harnesses using GLM-5.3-Flash as the backbone. Premature stopping denotes cases where conflicting, missing, or unchecked clues have already appeared in the trajectory, but the model submits without addressing them and produces an incorrect answer. Unbounded exploration denotes continued retrieval until budget exhaustion after retrieval has ceased to target gaps that could change the answer. The table reports each failure type as a percentage of all questions, where lower is better.

MethodPremature Stopping (OfficeQA Pro V1)Premature Stopping (OfficeQA Pro V2)Unbounded Exploration (OfficeQA Pro V1)Unbounded Exploration (OfficeQA Pro V2)
LongHarness21.8%20.0%0.8%2.2%
Claude Code30.8%34.4%3.0%7.8%
Codex40.6%43.3%10.5%16.7%

LongHarness has the lowest rates of both failure types on OfficeQA Pro V1 and V2. Relative to Claude Code, premature stopping decreases by 9.0 and 14.4 percentage points, while unbounded exploration decreases from 3.0% and 7.8% to 0.8% and 2.2%, respectively. These results support the role of explicit gap management in continuing verification while decision-relevant information remains unresolved and reducing exploration that drifts away from the answer.

A concrete example from OfficeQA Pro V1 asks for the coefficient of variation of logarithmic growth rates in U.S. national defense expenditure over fiscal years 1933–1941, with emergency shipbuilding expenditures included in the 1941 value. The corpus contains two readings: 6080 in an earlier bulletin and 6301 under the revised accounting used from February 1942 onward. LongHarness initially records whether the 1941 value includes emergency shipbuilding expenditures as an unresolved gap. After finding 6301, it revisits 6080 and refines the gap to identify which reading matches the question's requirements. It then checks several later issues that also report 6301, concludes that the revised figure includes the expenditure, closes the gap, and submits the correct answer of 1.56. Claude Code also retrieves 6301 but does not resolve its discrepancy with 6080, ultimately using 6080 and producing the incorrect answer of 1.54. This case shows that finding relevant evidence does not necessarily resolve the key uncertainty. LongHarness retains source disagreements as explicit gaps that continue to influence exploration and stopping decisions, reducing the risk of overlooking a conflict and submitting prematurely once a superficially usable answer is available.

Reviewer_YaXkOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
1(−1)
Read review
Paper summary

Agents that answer questions over a very large document corpus must search over many steps and read massive documents at each step. This leads to two challenges: the agent might keep exploring without really resolving the gaps that matter, and some evidence or corrections may actually be found but never make it into the final answer.

To resolve this, LongHarness proposes two things. The first is an Evidence Bank: a list of short facts linked to their source text. These facts can be updated, replaced, or removed. The second is a Conclusion State: a list of claims, competing guesses, and the current open gaps. The open gaps can guide the agent on what it should search and read next, and then the agent can use new evidence to update the conclusions. The final answer must cite valid evidence.

On OfficeQA Pro, LongHarness with GLM-5.3-Flash gets 60.2% and 61.1% exact match on V1 and V2. In comparison, Claude Code gets 48.1% on V1, and Codex gets 42.2% on V2. It also has the best average across four benchmarks with both models. Removing any of its parts lowers accuracy in all five ablations.

Critical strengths

S1. The paper clearly describes two important challenges in long-horizon agents: they can explore without end, and their evidence can drift over time. It then proposes a fix for each challenge. The paper uses open gaps to guide the search, and evidence can be changed or removed while keeping its source.

S2. The gains are large on a hard benchmark. Answering directly gets at most 23.2% (V1) and 5.6% (V2), while LongHarness gets 60.2% and 61.1% with the same GLM-5.3-Flash model. It also has the best average across four benchmarks with both models (70.3 and 69.2; Table 1).

S3. In addition to just showing accuracy, the paper also studies where agents fail and gives some other detailed ablation studies.

Critical weaknesses

W1. The main test mixes too many things. LongHarness gets its own search tool, short reads, and a large reading budget. However, Codex and Claude Code's tools are limited to the shell with 200 turns. In the experiments, it seems that just by keeping these tools, the agent still gets 55.6 on V1 and 53.3 on V2. That is already better than Claude Code (48.1) and Codex (42.2). So it is not clear whether the improvement mainly comes from a better tool set.

W2. The results need more clarification. In Table 1, the numbers are from one run, while the paper actually did three runs, and for the V1 row, the paper compares LongHarness's best run to Claude Code's worst run. When looking at the average of three runs, the gaps get smaller: 10.5 on V1 and 18.5 on V2. Some numbers also do not fit a whole number of right answers, like 23.2% of 133 questions and 51.7% of 90 questions.

W3. The novelty of the idea is limited. The paper cites several works that also keep evidence over time, like MemGPT, ReadAgent, GraphReader, ReContext, and LedgerMind. It seems that the main new contribution of the paper is the list of open gaps that tells the agent where to look next. However, the experimental results don't show that this part of LongHarness brings a significant improvement.

Questions for authors

see weakness

6 comments
AuthorsOpenReview ↗

Response to Reviewer YaXk [Part2]

Q2: How should the single-run main results, repeated-run means, and accuracy denominators be interpreted?

A2: (1) Repeated runs and selection of the main-table run. We first completed one run under a consistent configuration. To assess stability, we subsequently ran each agentic harness using GLM-5.3-Flash two more times, yielding three runs in total, and reported individual scores, means, and standard deviations in the appendix. Computational constraints limited repeated experiments to this configuration. The main table uses the first run of each setting, rather than selecting runs by score. For example, Claude Code's 35.6% on OfficeQA V2 is the highest of its three scores (27.78 / 34.44 / 35.56), while Codex's 42.2% is the middle score (36.67 / 42.22 / 44.44). Codex's 46.6% on V1 is also the middle score (45.86 / 46.62 / 47.37), with a range of only 1.51 percentage points across the three runs. According to the three-run statistics in the appendix, Codex, Claude Code, and LongHarness achieve 46.62 ± 0.76, 49.37 ± 1.57, and 59.90 ± 0.43 on V1, and 41.11 ± 4.00, 32.59 ± 4.21, and 59.63 ± 1.28 on V2, respectively. LongHarness retains a substantial advantage when comparing mean performance, exceeding the strongest baseline by 10.53 and 18.52 percentage points on V1 and V2, respectively.

(2) Percentages that do not correspond to an integer number of correct answers over the full benchmark. Direct LLM baselines use a 1800-second timeout per request and allow at most four attempts. Questions that still do not return successfully are excluded, so the denominator is the number of completed questions. Such cases are rare for agentic harnesses, but the same accounting convention applies. For example, 51.7% on OfficeQA V2 corresponds to 46/89, with one question failing to return successfully. We have added the timeout, retry settings, and accuracy-denominator convention to Implementation details in Section 3.1.

AuthorsOpenReview ↗

Response to Reviewer YaXk [Part3]

Q3: Is the novelty of LongHarness mainly the introduction of open gaps, and does this mechanism provide substantive gains?

A3: Thank you for this comment. We clarify the novelty of LongHarness through its core mechanism and the ablation evidence.

(1) Our main contribution goes beyond using a list of unresolved information needs to tell the agent where to search next. We organize evidence and conclusions as two continuously maintained, mutually revising states, and use evidence lifecycle management to preserve the provenance, validity, and supersession relationships of evidence. This makes evidence–conclusion coevolution an explicit and controllable process within the harness. Existing approaches such as MemGPT, ReadAgent, GraphReader, ReContext, and LedgerMind primarily focus on accessing, compressing, organizing, or tracing information. They typically strengthen L1 retrieval and L2 editing, while the transition from accumulated evidence to evolving conclusions remains embedded in conversation history, free-form notes, or transient reasoning. LongHarness targets reasoning over much larger corpora. OfficeQA Pro V1 contains approximately 100M tokens, and V2 roughly doubles this scale, requiring multiple rounds of search and reading to locate, integrate, and revise evidence. In this setting, unbounded exploration and evidence-state drift can reinforce one another and are difficult to resolve through a single retrieval or compression step. LongHarness therefore uses the Evidence Bank to manage the evidence lifecycle and the Conclusion State to maintain conclusions, hypotheses, and unresolved gaps. Gaps guide subsequent retrieval, new evidence drives conclusion revision, and outdated evidence is retracted or superseded. Evidence maintenance, conclusion updates, and exploration decisions form a continuous feedback loop, grounding subsequent actions in the current evidence and conclusions. Open gaps are thus a control signal within this coevolution process. The contribution of LongHarness is the coordinated mechanism connecting evidence, conclusions, and exploration, beyond the introduction of open gaps alone.

(2) Ablation results support the role of explicit gap guidance, with larger benefits on questions requiring longer exploration. These grouped ablations all use GLM-5.3-Flash as the base model. In Table 2, w/o Explicit Conclusion-Guided Exploration retains the Evidence Bank and Conclusion State, including open gaps, and removes only the explicit instruction to guide exploration using conclusion gaps. Accuracy decreases from 60.2% to 55.6% on OfficeQA V1 and from 61.1% to 53.3% on V2, corresponding to drops of 4.5 and 7.8 percentage points. Because both states are retained, this comparison directly tests the benefit of explicitly using gaps to guide exploration. The w/o Persistent States and Naive Evidence Accumulation variants further examine the contributions of persistent states and evidence lifecycle management. We also conduct an exploratory analysis that groups questions at a threshold of 20 turns, using the average number of model-call turns for full LongHarness and w/o Explicit Conclusion-Guided Exploration on each question.

MethodV1 ≤20 turnsV1 >20 turnsV2 ≤20 turnsV2 >20 turnsCombined ≤20 turnsCombined >20 turns
LongHarness (full)63.8655.1060.0063.0163.2759.84
w/o Explicit Conclusion-Guided Exploration62.6544.9053.3354.7961.2250.82
Gain (percentage points)+1.20+10.20+6.67+8.22+2.04+9.02

On questions requiring longer exploration (>20 turns), the full method improves accuracy by 10.20 and 8.22 percentage points on V1 and V2, respectively, compared with 1.20 and 6.67 points on questions requiring ≤20 turns. Across both versions, the gain in the >20-turn group is 9.02 points, approximately 4.4 times the gain in the ≤20-turn group (2.04 points). This is consistent with our mechanistic explanation: explicitly maintaining and investigating unresolved gaps helps exploration converge toward evidence-supported conclusions on tasks that require sustained exploration and evidence integration across turns. We have clarified the definition of unresolved gaps and their role in guiding exploration and stopping decisions in Section 2.2.

AuthorsOpenReview ↗

A Kind Reminder Regarding Our Rebuttal

Dear Reviewer YaXk, 

Thank you again for your thoughtful comments and constructive suggestions. In response to your feedback, we have provided additional experiments and clarifications in our rebuttal. We hope these provide useful evidence for your consideration. 

With less than 48 hours remaining in the discussion period, we would greatly appreciate it if you could take a look at our responses when you have a chance. Please let us know if any concerns remain, we would be happy to clarify them. 

Thank you for your time and consideration. 

Best regards,

The Authors of Submission 42128

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.