acceptodds
Under review as a conference paper at ICLR 2027

LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning

Abstract

Language-model (LM) harnesses enable LMs to operate effectively over long contexts using additional compute. However, existing long-context evaluations are insufficient for distinguishing modern harnesses, reflected by saturated accuracy across harnesses and largely similar evaluation costs. In this paper, we introduce a benchmark for evaluating both the effectiveness and efficiency of long-context harnesses. Our tasks require diverse retrieval strategies, including lexical search and semantic matching, together with strategic and adaptive reasoning over global and local context. Much of the context is semantically relevant but only a small subset is useful at each step, creating both a challenging search problem and different accuracy–cost tradeoffs across processing strategies. For example, one task requires identifying every person satisfying several conditions using evidence scattered across documents; strategically checking the most selective condition first can narrow the search before verifying the remaining conditions. We evaluate multiple families of frontier language models with four state-of-the-art harnesses. Our benchmarks remain challenging even for strong model–harness combinations: the best reaches 68% macro-average accuracy across four evaluation suites. More importantly, we find that the same underlying model can exhibit markedly different efficiency under different harnesses. Our results establish efficiency as an important axis for long-context evaluation and provide a testbed for developing harnesses that process context strategically rather than exhaustively.

Primary area:
datasets and benchmarks
Keywords:
Long Context, Agentic, Harnesses, Evaluation, Efficiency

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_uLX4OpenReview ↗
Rating
3(+1)
Confidence
3(−1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces LongHarness Bench, a benchmark evaluating language-model harnesses rather than base models alone. It comprises four synthetic long-context tasks: Constraint Solving Search, Equivalent Program Pair Search, Program Execution Tracing, and Outlier Memo Detection. These tasks require retrieval, multi-step reasoning, and adaptive evidence selection, with different strategies involving different computational costs.

Critical strengths

S1. Existing benchmarks primarily evaluate model performance, often under a fixed harness, making it difficult to assess the impact of different harness designs. This paper explicitly examines harness quality by holding the underlying model and context fixed while varying the harness configuration (Table 2). This is a useful distinction from existing benchmarks. The comparison in Table 1 also provides reasonable support for the claim that existing suites, including OOLONG, LongProc, LongBench-v2, and HELMET, do not directly address this setting. The experimental results further suggest that the proposed tasks are more challenging and better at distinguishing between harness configurations. No single configuration consistently performs best across all four tasks, and macro accuracy ranges from 0% to 68%, compared with the much narrower 60–68% range reported for LongBench-v2. The results provide a reasonable case for evaluating harness design separately from the underlying model.

S2. The tasks are generated from hidden structured specifications, with distractors introduced through controlled perturbations rather than random modifications. The authors also employ a private solver and an independent rule-based validator to recompute the ground-truth answers and verify the intended roles of required items and distractors (Appendix B, Table 5). Several task-specific design choices are particularly useful. In Constraint Solving Search, the prevalence constraints (≤10% and ≥60%) create meaningful differences in the cost of alternative search orders. In Program Execution Tracing, decoy cells are retained only when they agree on the displayed call but differ on a separate contract probe. In Equivalent Program Pair Search, true matching pairs are required to rank below at least twelve mutants under the similarity measure. These constraints make the task difficulty more convincing, particularly where different search strategies incur different costs. I also appreciate that the authors explicitly acknowledge the limitations of their ground-truth construction, particularly that program equivalence is defined operationally through testing rather than established through formal verification.

Critical weaknesses

W1. This is my main concern, as cost efficiency is central to the paper's contribution. Using the token counts in Table 4b and the rates in Table 6, I cannot reproduce several costs reported in Table 2. The discrepancies are particularly large for agentic harnesses. For example, GPT-5.6-sol with OpenCode on Constraint Solving Search costs \1.72accordingtoTable2,butapproximately1.72 according to Table 2, but approximately \\7.32 based on the reported tokens and rates. Prompt caching does not fully explain these differences. Since these costs underpin the accuracy–cost comparisons, the authors should clarify their accounting methodology, including cached tokens and sub-call costs, and verify the reported results. The paper should report token counts and model calls alongside dollar costs so that these comparisons can be checked.

W2. The paper does not clearly report the number of evaluation instances per task, confidence intervals, or run-to-run variability. Without uncertainty estimates, I am not convinced that the smaller accuracy differences are meaningful. For example, several comparisons in Section 4.3 differ by only a few percentage points, yet are used to support broader conclusions about differences between model families. I would like to see the sample sizes and uncertainty estimates reported explicitly. It would also help to distinguish incorrect answers from formatting failures, timeouts, and budget exhaustion, since these failure modes have different implications for evaluating harness effectiveness.

Questions for authors

Q1. What exactly is included in the per-instance cost calculation? Please distinguish uncached input, cached input, output tokens, and any sub-calls made by agentic harnesses. Were the token counts in Table 4b collected from the same runs used for Table 2?

Q2. How many instances are evaluated per task, and are the reported accuracy differences statistically significant? Could the authors also report cost distributions, uncertainty estimates, and budget-exhaustion rates? Mean costs alone may conceal substantial variation across instances.

Q3. Were the harnesses given comparable prompt-tuning efforts, interaction limits, and retry budgets? Could the authors clarify whether the observed performance differences are robust to prompt variations and whether the proposed mechanism-level explanations are supported by component ablations?

Q4. What evidence shows that the synthetic tasks measure the intended harness capabilities rather than generation or formatting artifacts? Additionally, since the comparisons with existing benchmarks use only GPT-5.6-sol, would the claimed advantages remain consistent across other model families?

3 comments

We thank the reviewer for your comments. We address your questions as follow:

What exactly is included in the per-instance cost calculation? Please distinguish uncached input, cached input, output tokens, and any sub-calls made by agentic harnesses. Were the token counts in Table 4b collected from the same runs used for Table 2?

Our cost calculation follow the following formula: cost=(I−C−W)rinput+Crcached+Wrwrite+Oroutput\text{cost} = (I - C - W) r_{\text{input}} + C r_{\text{cached}} + W r_{\text{write}} + O r_{\text{output}}

Here, I is total input, C cached reads, W cache writes, O output including reasoning and r denotes corresponding API pricing . And sub-calls made by harnesses are also counted toward this budget. We have updated Table 4 in the appendix to show our cost breakdown. The token counts in Table 4b were collected from the exact same run used in Table 2.

For example, GPT-5.6-sol with OpenCode on Constraint Solving Search has the following mean usage per instance:

Token categoryMean tokens (M)Rate ($/M)Cost contribution ($)
Uncached input0.0007504.000.0030
Cache read1.6441120.400.6576
Cache write0.1924375.000.9622
Output0.00595320.000.1191

The reconstructed cost is 4(0.000750)+0.4(1.644112)+5(0.192437)+20(0.005953)=$1.7418898≈$1.744(0.000750) + 0.4(1.644112) + 5(0.192437) + 20(0.005953) = \$1.7418898 \approx \$1.74 matching Table 2. Table 4(c) additionally identifies tokens subject to higher request-length rates where applicable.

How many instances are evaluated per task, and are the reported accuracy differences statistically significant? Could the authors also report cost distributions, uncertainty estimates, and budget-exhaustion rates? Mean costs alone may conceal substantial variation across instances

Each of our reported tasks has 50 samples totalling a 200 instance test suite. We have reported the 95% Wilson confidence interval in our updated paper (Table 4). Cost distribution and number of budget exhaustions are also added (Appendix E).

For example, several comparisons in Section 4.3 differ by only a few percentage points, yet are used to support broader conclusions about differences between model families

Section 4.3 uses small numerical differences to describe similar observed performance across configurations. Our central conclusions are supported by substantial accuracy differences. For GPT-5.6-sol, mini-swe-agent improves Outlier Memo Detection accuracy from 44% to 100%. This is a 56-percentage-point gain, with a paired 95% bootstrap confidence interval of [42, 70] points. On Equivalent Program Pair Search, RLM reduces accuracy from 44% to 2%. This is a 42-percentage-point drop, with a corresponding confidence interval for the drop of [28, 56] points. Both intervals exclude zero, supporting the observed directions of change.

Our efficiency findings also concern substantial differences. On Outlier Memo Detection, mini-swe-agent achieves 100% accuracy at \0.583perinstance,whileRLMachieves920.583 per instance, while RLM achieves 92% at \7.23, approximately 12.4 times the cost. These central findings do not depend on resolving small accuracy differences.

Were the harnesses given comparable prompt-tuning efforts, interaction limits, and retry budgets? Could the authors clarify whether the observed performance differences are robust to prompt variations?

We guarantee that all harnesses receive the same vanilla prompt describing the task requirement, retry budget for each tool call, interaction limits are also set so that they can only work on the sample files without accessing gold answer or browsing the internet. For our results robustness, because of budget constraint, we additionally report the 95% Wilson confidence interval.

...whether the proposed mechanism-level explanations are supported by component ablations? What evidence shows that the synthetic tasks measure the intended harness capabilities rather than generation or formatting artifacts?

We provide additional behavioral analysis on 800 GPT-5.6-sol agent runs, covering four harnesses and four tasks to quantify the harness behaviors discussed in our section 4.4. In general, our tasks are capable of showing substantially different costs between context processing strategies. Some notable processing example include:

Program Execution Tracing (Appendix D.4): RLM uses two primary delegation strategies:

  1. Step-based filtering (22/50 samples): The parent agent parses the query into 8 steps, filters relevant program cells for each, and assigns one step per child agent to identify candidates (one-to-many mapping). Average cost: $0.935/sample.

  2. Context-based partitioning (21/50 samples): The parent agent splits the cells evenly and sends the full 8-step query to each child agent (many-to-many mapping). Average cost: $2.08/sample

Outlier Memo Detection (Appendix D.2):

  • mini-swe-agent (50/50 samples): Parses entity names to group memos by entity pairs, then uses code and templates to extract entity-relationship entries into a fact table. It programmatically detects conflicting claims and identifies outliers based on supporting fact counts. Average cost: $0.583/sample.

  • RLM: Despite having programming capabilities, RLM chunks the memos and delegates fact-table generation to child models. This repeated LLM-based parsing across chunks drives up costs to $7.23/sample.

For the more detailed analysis, please refer to our full appendix D.

would the claimed advantages to other datasets remain consistent across other model families other than GPT-5.6-sol?

Relative to existing benchmarks, LongHarness (1) separates harnesses more clearly and (2) gives a more comprehensive view of harness behavior. We run Sonnet-5.5 as a second model family, and both points hold for both models.

Table R1. Accuracy (avg. USD per instance). Each cell: GPT-5.6-sol / Sonnet-5.5. LongHarness shows the macro average over its four tasks.

HarnessLongHarnessOOLONG-SynthLongBench-v2
Direct60.0% (\0.823)/28.50.823) / 28.5% (\0.706)64% (\0.686)/500.686) / 50% (\0.394)62% (\0.532)/520.532) / 52% (\0.403)
mini-swe-agent68.5% (\0.816)/79.50.816) / 79.5% (\0.367)86% (\0.217)/800.217) / 80% (\0.106)68% (\0.190)/600.190) / 60% (\0.086)
OpenCode41.0% (\2.91)/73.02.91) / 73.0% (\0.498)74% (\2.08)/742.08) / 74% (\0.150)60% (\0.463)/620.463) / 62% (\0.102)
RLM53.5% (\4.51)/44.14.51) / 44.1% (\0.649)78% (\3.70)/743.70) / 74% (\0.231)62% (\0.641)/600.641) / 60% (\0.222)
ReAct43.0% (\1.93)/25.51.93) / 25.5% (\0.742)70% (\0.751)/700.751) / 70% (\0.500)68% (\0.354)/520.354) / 52% (\0.105)

(1) Clearer separation. Across five configurations, LongHarness accuracy spans 41.0–68.5% for GPT-5.6-sol and 25.5–79.5% for Sonnet-5.5, versus 64–86% / 50–80% on OOLONG-Synth and 60–68% / 52–62% on LongBench-v2. In addition, while on both existing benchmarks, every harness costing more than direct inference is at least as accurate, on LongHarness, three GPT-5.6-sol harnesses (OpenCode, RLM, ReAct) cost 2.3–5.5× more yet score lower.

(2) A more comprehensive view. On OOLONG-Synth, both models agree that mini-swe-agent is the best harness and direct inference the worst. LongHarness's four tasks ranks harness differently: For GPT-5.6-sol, mini-swe-agent leads on Memos, OpenCode on Tracing, and ReAct on Pairs, while direct inference ties for best on Constraint. The performance of the same harness can vary differently across tasks and models. For instance, with GPT-5.6-sol, Opencode scores 96% on Tracing but 0% on Pairs; on Memos, it scores 100% with Sonnet-5.5 versus 10% with GPT-5.6-sol.

On the scope of the motivation. Our central claim is that existing benchmarks do not adequately distinguish modern harnesses. Table 3 supports this: LongBench-v2 shows little separation, while OOLONG-Synth separates only in one direction, with the same best and worst configurations for both models. We will make the abstract’s description of this gap (“saturated accuracy”, “similar costs”) more precise.

We will add these results, including the per-task Sonnet-5.5 numbers, to Tables 2–3 and Section 4.5, and clarify the intent of this comparison in the abstract.

The authors' response has addressed most of my concerns. I have decided to raise my score to 3.

Reviewer_Sk4tOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces LongHarnessBench, comprising four retrieval and reasoning tasks for evaluating the accuracy and cost of language models and their harnesses. Experiments with five models and four harnesses show that harness benefits depend on the model and task, and that additional inference expenditure does not necessarily improve accuracy.

Critical strengths
  1. Evaluating both accuracy and cost, while comparing harnesses around the same model, reveals trade-offs that accuracy-only evaluation would miss.
  2. The tasks use similar distractors, distributed evidence, and dependent steps to assess capabilities beyond lexical retrieval. Structured answer generation and programmatic checks support reproducible evaluation.
Critical weaknesses
  1. Limited controlled comparisons of model capabilities. The five models differ in training and tool-use capabilities, making performance differences difficult to interpret. Evaluating different sizes or capability levels within the same model family, under a fixed harness and budget, would help test whether the benchmark reflects expected capability differences.
  2. Insufficient analysis of near-zero scores. Kimi-K2.6 achieves near-zero accuracy under most configurations without targeted diagnosis. The authors should distinguish reasoning errors, partially correct answers, parsing failures, tool errors, and budget exhaustion, and manually inspect outputs to rule out evaluation-pipeline issues.
Questions for authors

The text in Figure 1 is too small. Further recommendations are included in the weaknesses above.

2 comments

We thank the reviewer for your comments. We address your questions as follow:

Limited controlled comparisons of model capabilities

Our primary objective is to compare harnesses while holding the underlying model fixed, rather than to isolate differences in model capability.

However, we have additionally run gpt-5.6-luna, a similar family less capable variation of gpt-5.6-sol, with high thinking effort.

ModelHarnessConstraintMemosTracingPairsMacro avg.
GPT-5.6-solDirect58% ($0.690)44% ($1.17)94% ($0.648)44% ($0.780)60.0% ($0.823)
GPT-5.6-solRLM44% ($0.659)92% ($7.23)76% ($1.47)2% ($8.88)53.5% ($4.56)
GPT-5.6-solOpenCode58% ($1.74)10% ($3.11)96% ($1.36)0% ($5.41)41.0% ($2.91)
GPT-5.6-solmini-swe-agent44% ($0.321)100% ($0.583)88% ($1.02)42% ($1.34)68.5% ($0.816)
GPT-5.6-solReAct32% ($1.40)6% ($2.77)82% ($0.866)52% ($2.68)43.0% ($1.93)
GPT-5.6-lunaDirect0% ($0.039)0% ($0.078)6% ($0.038)0% ($0.040)1.5% ($0.049)
GPT-5.6-lunaRLM14% ($0.050)0% ($0.507)4% ($0.141)0% ($0.515)4.5% ($0.303)
GPT-5.6-lunaOpenCode8% ($0.095)0% ($0.131)16% ($0.087)0% ($0.153)6.0% ($0.117)
GPT-5.6-lunamini-swe-agent8% ($0.028)4% ($0.103)6% ($0.067)0% ($0.096)4.5% ($0.074)
GPT-5.6-lunaReAct4% ($0.092)0% ($0.154)6% ($0.077)0% ($0.123)2.5% ($0.111)

From the table, luna shows a substantially different performance from sol on our test suite.

Error analysis for near-zero strict accuracy of Kimi.

Kimi’s near-zero strict accuracy in our runs primarily results from budget exhaustion (68%), followed by completed but incorrect or incomplete outputs (31.5%). Under direct inference, almost all of the traces exhausted their output limit without producing an answer, and almost 60% of the agentic runs reached either the 3M token limit or the 128K token context-limit of the model served by our provider, tinker API.

However, our tasks do not inherently require exhausting resources. For instance, using mini-swe-agent, GPT-5.6-sol solves all 50 Outlier Memo Detection instances using an average of 0.38M tokens (12.7% of the 3M allowance). Similarly, GLM-5.3 achieves 66% accuracy on Constraint Solving Search using 1.11M tokens per instance (37% of the allowance, including ~22.7K output tokens).

To further investigate why the budget exhaustion happen, we looked into runs with mini-swe-agent and figured out that on more than 80% of the traces in each task the model shows brute-force behavior such as repeatedly checking every single candidate by verbal reasoning rather than writing programs or doing bulk reading to compare many candidates together. Since this Kimi model was served on Tinker with the tag “moonshotai/Kimi-K2.6:peft:131072”, it indicates that the model might have gone through a parameter-efficient fine-tuning (PEFT) technique, which could lead to reduced capability.

Thank you for your response. However, my main concerns regarding evaluation soundness and presentation remain unresolved, so I will maintain my original score.

Reviewer_LcqaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper propose LongHarness Bench, which introduces four tasks for evaluating long-context language model harnesses: Constraint Solving Search, Equivalent Program Pair Search, Program Execution Tracing, and Outlier Memo Detection. The tasks combine semantically confusable evidence with opportunities for selective retrieval, verification, and reuse. Five models are evaluated with direct inference and four agentic harnesses using exact task success, token usage, and estimated API cost. The best configuration achieves 68% macro accuracy. Comparisons with OOLONG-Synth and LongBench-v2 assess whether the proposed suite better distinguishes harness behavior.

Critical strengths
  1. The task designs give a concrete reason to study selective verification. Constraint Solving Search controls predicate selectivity and includes candidates that satisfy all but one condition. Equivalent Program Pair Search asks the model to find all matching pairs among 200 programs, even when behavior-changing mutants resemble a target more closely than its true mate. Compared with the supplied-pair classification setting in EquiBench [1], this tests search and verification together. The structured generation procedures and deterministic checks in Appendix B provide a useful foundation, and the authors clearly acknowledge the limits of test-based equivalence.

  2. The most useful results, in my view, are the reversals across tasks. Mini-swe-agent raises GPT-5.6-sol’s Outlier Memo Detection accuracy from 44% to 100%, yet lowers its Constraint Solving Search accuracy from 58% to 42%. OpenCode takes the same model from 44% to 0% on Equivalent Program Pair Search. These differences in Table 2 matter in practice and show why a macro score alone is not enough to judge a harness. Comparing several models and harnesses makes the suite potentially useful for identifying these mismatches.

Critical weaknesses
  1. The token budget is hard to reconcile with the reported usage. For Kimi-K2.6 on Program Execution Tracing, OpenCode averages 5.83M input plus 0.0987M output tokens, and mini-swe-agent averages 4.30M plus 0.0242M, according to Table 4. Qwen3.8-27B/OpenCode and GLM-5.3/mini-swe-agent also exceed the 3M cumulative input-and-output limit specified in Section 4.1. These are averages, not isolated outliers. It is therefore unclear what budget was actually enforced and whether all configurations followed the same stopping rule. More detail is also needed to understand the dollar costs. GPT-5.6-sol/mini-swe-agent uses approximately 3.5 times as many cumulative tokens as direct inference on Outlier Memo Detection while costing roughly half as much. This could be a legitimate benefit of caching, but the cache rates in Table 6 are not accompanied by the usage breakdown needed to distinguish that benefit from changes in processing strategy. The RLM root/child model configuration also needs to be stated, since this choice matters for both capability and cost in the original RLM setup [2].

  2. I am not yet convinced that the experiments establish the claimed diagnostic advantage over existing benchmarks. The abstract describes existing evaluations as having saturated accuracy and largely similar costs. Yet the paper’s own OOLONG-Synth results span 70–86% accuracy across harnesses and $0.217–$3.70 per instance, approximately a 17-fold cost difference. LongBench-v2’s 60–68% accuracy range indicates limited separation, rather than saturation. These results in Table 3 support a narrower motivation than the one presented. The main question is what the new suite tells us about processing strategy. Selective search, reusable fact tables, and ineffective decomposition could explain the outcomes discussed in Section 4.4. Without systematic trajectory measurements or matched strategy comparisons, though, it is hard to tell how much these behaviors account for the observed differences. Earlier work already examines task-specific processing strategies and their costs [3]. To establish the added value here, the analysis needs to show what the new task design reveals beyond configuration-level performance differences. It currently does not isolate how much of the additional discrimination comes from adaptive retrieval rather than semantic subproblem difficulty or complete-set scoring.

[1] Anjiang Wei et al. EquiBench: Benchmarking Large Language Models’ Reasoning about Program Semantics via Equivalence Checking. EMNLP, 2025.

[2] Alex L. Zhang, Tim Kraska, and Omar Khattab. Recursive Language Models. arXiv:2512.24601, 2025; revised 2026.

[3] Weili Cao, Xunjian Yin, Bhuwan Dhingra, and Shuyan Zhou. Coding Agents are Effective Long-Context Processors. arXiv:2603.20432, 2026.

Questions for authors
  1. How was the 3M-token budget enforced across cached tokens, recursive calls, retries, and concurrent requests? What explains the above-budget means, and were all reported accuracies obtained under the same stopping rule?

  2. Could you provide the uncached-input, cache-read/write, and output-token breakdowns needed to reconstruct the reported costs, together with the RLM root/child model identities and recursion settings?

  3. Do the existing trajectories provide quantitative support for the strategy explanations in Section 4.4? For example, is selective-first search or evidence reuse associated with lower cost at comparable accuracy beyond the illustrative behaviors described?

4 comments

We thank the reviewer for your comments. We address your questions as follow:

How was the 3M-token budget enforced across cached tokens, recursive calls, retries, and concurrent requests? What explains the above-budget means, and were all reported accuracies obtained under the same stopping rule?

Thank you for noting this out. We indeed did enforce a 3M token (input + output), recursive calls, retries and concurrent requests were all included in this limit via a thread-safe central recorder which will reject a call if it goes over the total budget. And the same rules are applied for all of our experiments.

There were a few initial exploration data points where we tried to accommodate slightly over the budget. We have updated these few datapoints results by tracing back these experiments and marking submitted answers after exceeding the limit as wrong as well as recalculating the average cost with respect to the 3M limit. That being said, our major claims remain unchanged.

provide the uncached-input, cache-read/write, and output-token breakdowns needed to reconstruct the reported costs

Yes please refer to our updated table 4.

together with the RLM root/child model identities and recursion settings

For RLM, we use the same model for both root and child models with recursion depth = 1.

Do the existing trajectories provide quantitative support for the strategy explanations in Section 4.4?

We did additional analysis on 800 GPT-5.6-sol agent runs, covering four harnesses and four tasks. In general, our tasks are capable of showing substantially different costs between context processing strategies. Some notable processing example include:

Program Execution Tracing (Appendix D.4): RLM uses two primary delegation strategies:

  1. Step-based filtering (22/50 samples): The parent agent parses the query into 8 steps, filters relevant program cells for each, and assigns one step per child agent to identify candidates (one-to-many mapping). Average cost: $0.935/sample.

  2. Context-based partitioning (21/50 samples): The parent agent splits the cells evenly and sends the full 8-step query to each child agent (many-to-many mapping). Average cost: $2.08/sample

Outlier Memo Detection (Appendix D.2):

  • mini-swe-agent (50/50 samples): Parses entity names to group memos by entity pairs, then uses code and templates to extract entity-relationship entries into a fact table. It programmatically detects conflicting claims and identifies outliers based on supporting fact counts. Average cost: $0.583/sample.

  • RLM: Despite having programming capabilities, RLM chunks the memos and delegates fact-table generation to child models. This repeated LLM-based parsing across chunks drives up costs to $7.23/sample.

For the more detailed analysis, please refer to our full appendix D.

On diagnostic advantage of LongHarness over existing benchmarks

We thank the reviewer for the careful reading. Our motivation is comparative. Relative to existing benchmarks, LongHarness (1) separates harnesses more clearly and (2) gives a more comprehensive view of harness behavior. We run Sonnet-5.5 as a second model family, and both points hold for both models.

Table R1. Accuracy (avg. USD per instance). Each cell: GPT-5.6-sol / Sonnet-5.5. LongHarness shows the macro average over its four tasks.

HarnessLongHarnessOOLONG-SynthLongBench-v2
Direct60.0% (\0.823)/28.50.823) / 28.5% (\0.706)64% (\0.686)/500.686) / 50% (\0.394)62% (\0.532)/520.532) / 52% (\0.403)
mini-swe-agent68.5% (\0.816)/79.50.816) / 79.5% (\0.367)86% (\0.217)/800.217) / 80% (\0.106)68% (\0.190)/600.190) / 60% (\0.086)
OpenCode41.0% (\2.91)/73.02.91) / 73.0% (\0.498)74% (\2.08)/742.08) / 74% (\0.150)60% (\0.463)/620.463) / 62% (\0.102)
RLM53.5% (\4.51)/44.14.51) / 44.1% (\0.649)78% (\3.70)/743.70) / 74% (\0.231)62% (\0.641)/600.641) / 60% (\0.222)
ReAct43.0% (\1.93)/25.51.93) / 25.5% (\0.742)70% (\0.751)/700.751) / 70% (\0.500)68% (\0.354)/520.354) / 52% (\0.105)

(1) Clearer separation. Across five configurations, LongHarness accuracy spans 41.0–68.5% for GPT-5.6-sol and 25.5–79.5% for Sonnet-5.5, versus 64–86% / 50–80% on OOLONG-Synth and 60–68% / 52–62% on LongBench-v2. In addition, while on both existing benchmarks, every harness costing more than direct inference is at least as accurate, on LongHarness, three GPT-5.6-sol harnesses (OpenCode, RLM, ReAct) cost 2.3–5.5× more yet score lower.

(2) A more comprehensive view. On OOLONG-Synth, both models agree that mini-swe-agent is the best harness and direct inference the worst. LongHarness's four tasks ranks harness differently: For GPT-5.6-sol, mini-swe-agent leads on Memos, OpenCode on Tracing, and ReAct on Pairs, while direct inference ties for best on Constraint. The performance of the same harness can vary differently across tasks and models. For instance, with GPT-5.6-sol, Opencode scores 96% on Tracing but 0% on Pairs; on Memos, it scores 100% with Sonnet-5.5 versus 10% with GPT-5.6-sol.

On the scope of the motivation. Our central claim is that existing benchmarks do not adequately distinguish modern harnesses. Table 3 supports this: LongBench-v2 shows little separation, while OOLONG-Synth separates only in one direction, with the same best and worst configurations for both models. We will make the abstract’s description of this gap (“saturated accuracy”, “similar costs”) more precise.

We will add these results, including the per-task Sonnet-5.5 numbers, to Tables 2–3 and Section 4.5, and clarify the intent of this comparison in the abstract.

I appreciate the added analyses. The cost breakdown and the RLM configuration address several of my concerns, and the trajectory analysis adds solid empirical support. One question on the budget correction remains: which runs were impacted, and how exactly was the 3M-token cutoff enforced for both answer-level and call-level costs? GPT-5.6-sol's Constraint accuracy improves from 56% to 58% on OpenCode and from 42% to 44% on mini-swe-agent. Since flagging over-budget answers as incorrect cannot account for these gains by itself, were any other scoring or data adjustments made? I will keep my current rating for now.

Thank you for your quick response. We appreciate your effort in reviewing our paper.

which runs were impacted

Specifically these runs were impacted after our retrospective cutoff at a total 3M budget. These changes are also reflected in our cost accounting.

TaskModel / harnessCorrect/ overbudget answersMean tokens spentAccuracy before/ after
Program Execution TracingGLM-5.3 / OpenCode17/184.94M84% / 50%
Program Execution TracingGLM-5.3 / mini-swe-agent3/204.01M46% / 40%
Program Execution TracingQwen3.8-27B / OpenCode25/284.40M92% / 42%
Program Execution TracingKimi-K2.6 / OpenCode3/339.59M6% / 0%
Program Execution TracingKimi-K2.6 / mini-swe-agent1/494.39M2% / 0%

how exactly was the 3M-token cutoff enforced for both answer-level and call-level costs?

The 3M-token cutoff in our experiments is an answer-level threshold cutoff. For call-level cutoff, we set an universal threshold of 16K token per each API call made by a harness, across all harnesses. Thank you for pointing this out. We will incorporate this clarification in any further revision.

GPT-5.6-sol's Constraint accuracy improves from 56% to 58% on OpenCode and from 42% to 44% on mini-swe-agent…Were any other scoring or data adjustments made?

During our budget examination, we found two samples affected by network infrastructure failures during execution, which prevented the agents from producing answers (one with GPT-5.6-Sol + OpenCode and the other with GPT-5.6-Sol + mini-swe-agent). We reran the experiments and found that they succeeded on these examples hence accuracies are gained by 2% (1/50) for each of these experiments.

Upon further inspection we found that there were 8 samples in gpt-5.6-sol with RLM on Equivalent Program Pair Search subjected to these network failures. We reran them and found that the final accuracy across all samples stayed the same at 2% while the recalculated mean cost is \8.88insteadof8.88 instead of \\7.86. Similarly for Qwen-3.8-27B with RLM on Equivalent Program Pair Search, we found 7 samples affected, rerunning them resulted in no accuracy change (0%) and a recalculated cost of \7.96/sampleinsteadof7.96/sample instead of \\7.13 as initially reported.

We highly appreciate your effort spent on our paper!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.