acceptodds
Under review as a conference paper at ICLR 2027

Implementation-Specific Effects in Language-Model Harnesses

Abstract

Runtime code determines which context a language model receives, how often it is called, and when its answers are revised. HarnessBench studies these choices through factorial ablations of context management, memory, feedback, and planning. Each component has a shared null and two concrete active implementations. A complete-output replication contains 4,608 evaluations across three fixed model identifiers, two task domains, 16 configurations, and both implementation variants. Direct paired contrasts show differences in marginal effects between global implementation bundles, including Planning on math and code and Context and Feedback on code. These contrasts jointly change every active component and do not isolate a single component implementation. The direction and statistical conclusion also vary across models, domains, and endpoints. Near-call-budget controls, repeated runs, and a 500-task MATH-500 validation show that these treatments are implementation bundles whose effects include inference cost and operational reliability. On two local models, 3,200 planning episodes show no clear accuracy recovery after summary truncation is removed, despite substantially higher input-token costs. An isolated 1,600-condition feedback experiment shares first answers across revision prompts: self-checking outperforms a same-call neutral revision on Llama but not Qwen. HarnessBench therefore supports implementation-level diagnosis; two implementations per component do not estimate population-level implementation variance or identify a universal best harness.

Primary area:
datasets and benchmarks
Keywords:
language model agents, harness evaluation, factorial experiments, inference cost, code generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_1XsPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes HarnessBench to study how concrete implementations of LLM runtime components affect task performance. The authors divide the system into four components: context, memory, feedback, and planning. Each component has two active implementations and a shared OFF baseline. The 24 full factorial experiment is conducted within each of two global implementation variants. The main experiment covers three fixed models and two task domains, mathematics and code, examining average component ON–OFF effects and differences in these effects between the two global variants. Follow-up experiments examine inference-call costs, run-to-run repeatability, external task sets, and the effects of summary truncation and feedback prompts. The authors emphasize that component effects should be interpreted in relation to the specific implementation, model, task, and resource costs, rather than assuming general benefits from component names alone.

Critical strengths
  1. The paper points out that abstract component names can obscure substantial differences between concrete implementations. It introduces two sets of implementations and evaluates all ON/OFF combinations of the four components within each set, providing a relatively systematic diagnosis of these implementations.

  2. The B-versus-neutral experiment holds the initial answer, visible prefix, and number of additional calls fixed, helping distinguish the benefit of an opportunity to revise from that of a specific self-check prompt.

Critical weaknesses
  1. The global A/B comparison changes the implementations of multiple components simultaneously, so the observed difference cannot be attributed solely to replacing the target component. The rationale for selecting the two sets of implementations and their contribution beyond existing work on component attribution are also unclear. The current results do not establish sufficient insights beyond these specific cases.

  2. The specific scoring rules for Ytmv(c,z), the execution flow in Algorithm 1, and the actual roles of Memory and Context are not clearly explained. Terminology is inconsistent and insufficiently defined. The original feedback checker has known issues, but its corrected version has not been evaluated for performance, limiting assessment of the reliability of the results and the factors underlying them.

Questions for authors
  1. What new methods or findings does this paper offer beyond component attribution work such as ShapleyFlow and AgentSHAP? Why were Context, Memory, Feedback, and Planning selected, along with these two sets of implementations? What insights does the paper provide beyond these two specific sets of implementations?

  2. The global A/B comparison also changes the implementations of other enabled components. How can the observed differences be attributed to the target component rather than interactions among components? Please report all 24 isolated contrasts mentioned in Section 8 and clarify whether comparisons were conducted that replace only the target component with A or B while holding the other components fixed. Please also explain together what questions the marginal contrast, implementation contrast, and isolated-component contrast each answer.

  3. What are the specific scoring rules for Ytmv(c,z)? Please explain how answers are extracted and judged on HB-Math, how test results are converted into scores on HB-Code, and how parsing failures, execution failures, and incomplete outputs are handled. What exactly were the parsing failures for Gemini, and were the other models checked for similar issues? Did task-type misclassification and incorrect verified=True flags in the original Feedback-A affect output selection or final scores? Do the accuracy and costs of the corrected implementation still support the original conclusions?

  4. Please provide the execution flows for A/B/Null in Algorithm 1, including what create_plan() returns when Planning is OFF, how Planning-B uses only one model call, and how the final output is selected. Do Memory and history store and pass on the same content? Can a single-step task use information newly written to Memory? If Planning-A has only 2–5 steps and history grows by one entry per step, how is the rule that compression starts only above six entries triggered?

  5. In Section 5.4, is a summary a truncated intermediate output or a separately generated summary? Since A-trunc already retains the full conversation, which inputs does A-full actually change, and is this consistent with the 200-character limit in the main experiment? The cost controls for Planning and Feedback match or approximately match call counts, but not token consumption. What comparisons between methods do these results support, and do the conclusions hold under a common resource budget?

  6. The main experiment contains only 30 mathematics tasks and 18 coding tasks, and the pooled Code Feedback-B effect is influenced by the large gain on Claude. Are the main conclusions robust to task selection and model weighting?

  7. Do harness, runtime code/system/infrastructure, workflow, pipeline, and scaffold refer to the same concept in this paper? Please use these terms consistently or distinguish them clearly, along with bare/null/neutral and episode/condition.

4 comments

Thank you for requesting isolated replacements, explicit scoring definitions, and corrected-checker validation. We have completed these analyses and additional execution diagnostics.

Separating the comparison questions.

ComparisonQuestion answered
Marginal ON-OFF contrastWhat is the average component effect across tested coalitions?
Isolated-component contrastWhat changes when this component is added to the null harness?
Fixed-background A/B replacementWhat changes when only this implementation is replaced?

We recomputed all 24 isolated contrasts and completed 528 local-model conditions covering replacements against both A and B backgrounds, using 12 MATH-500 and 12 MBPP tasks. Another 264 conditions replicate the design through a fixed OpenRouter/DeepInfra provider. These mixed-implementation interventions add outcomes that cannot be recovered by attributing scores from the original global A/B records alone.

Replacement and budget findings. On MBPP, with Context, Memory, and Feedback fixed to B:

Execution settingPlanning-A, six callsPlanning-B, six callsPlanning-A, twelve calls
Qwen, local0/127/124/12
Llama, local0/127/123/12
Llama, fixed DeepInfra provider0/125/12—

All conditions retain the 16,384-token ceiling. The twelve-call results come from the additional 96-condition diagnostic. These experiments identify both the planning replacement effect and its interaction with the call budget.

Corrected checker. Using the same frozen first answers and a maximum of two revision calls:

ModelOriginal A accuracyCorrected A accuracy
Qwen49.5%51.5%
Llama29.0%31.5%

The corrected checker also improves output-contract compliance on both models. Task accuracy is evaluated independently of the checker's verification label.

Scoring and execution flow. HB-Code scores represent the fraction of tests passed; all-tests-passed accuracy is a separate endpoint. We will explicitly distinguish these measures and provide the isolated estimates with task-level uncertainty.

History receives two messages per completed step, so compression can activate at the fifth context preparation. Our new traces confirm 73 compression events and 772 context preparations with nonempty memory. Memory written after a step becomes available to subsequent steps, and both history and memory are reset between tasks.

Planning-OFF executes the task as a single step. Planning-B places its reasoning instruction in a single generation prompt, while Planning-A makes a separate planning call followed by step executions. These definitions and the measured traces make the roles of planning, context, memory, and feedback explicit. We will use consistent terminology throughout the revision and organize the findings around the implementation change, execution mechanism, and measured outcome.

We thank the authors for their response during the rebuttal phase. However, several critical questions remain unaddressed, and the newly provided data introduce a few new concerns that require further clarification.

1. Unaddressed or Incompletely Addressed Questions

①. Core Contribution & Rationale: The authors have not yet clarified the novel contributions of this work compared to existing component attribution studies (e.g., ShapleyFlow, AgentSHAP). The rationale for selecting these specific four components and the two concrete implementations remains unexplained. ②. Truncation Logic & Token Costs: Please clarify whether the summary in Section 5.4 is a "truncated intermediate output" or a "separately generated summary." Additionally, since token consumption is not matched in the Planning and Feedback comparisons despite matching call counts, do the original conclusions hold under a strict common resource budget? ③. Dataset Size & Stability: The main experiment relies on very small datasets (30 math, 18 code tasks). Please explain whether Claude's exceptionally large gain on Feedback-B dominates the overall component conclusion. Are the main findings robust to task sampling variations? ④. Reproducibility Details & Summary Table: The response omitted model versions, generation parameters, random seeds, details on Gemini's parsing failures, a clear definition of "endpoint," and how model-specific variance is distinguished from provider routing variance. Please provide a single summary table of all experiment configurations and main results, along with these metadata.

2. New Questions Raised by the Author's Response

①. Call Budget Bottleneck: The new data show Planning-A scoring much higher with 12 calls (Qwen 4/12, Llama 3/12) than with 6 calls (0/12). Does this imply that Planning-A's poor performance is primarily an artifact of the artificial budget cap rather than an inherent flaw in its mechanism? Please explicitly discuss this resource sensitivity in the manuscript. ②. Impact of the Corrected Checker: Given that the corrected Feedback checker improved accuracy and compliance on both Qwen and Llama, do the negative or non-significant statistical conclusions derived from the originally flawed checker change?

Thank you for the follow-up. We address the remaining questions directly below.

Contribution and component selection. ShapleyFlow and AgentSHAP attribute performance across component or tool coalitions under already specified implementations. HarnessBench asks a different intervention question: what changes when one concrete implementation is replaced while the other components are fixed? Our 528 local and 264 fixed-provider conditions execute mixed A/B configurations that are absent from the original matrices and therefore cannot be recovered from attribution over the saved outcomes. We selected Context, Memory, Feedback, and Planning because they control four distinct runtime decisions: visible state, retained state, answer revision, and execution flow. The A/B pairs contrast engineered multi-step mechanisms with simpler or model-native alternatives.

Truncation and resource sensitivity. The summary in Section 5.4 is not separately generated. A-trunc directly limits the intermediate output passed forward to 200 characters, while A-full removes this limit. The replacement experiments use common ceilings of 6 calls, 16384 total input-plus-output tokens, and 1024 output tokens per call. Increasing only the call ceiling to 12 raises Planning-A on MBPP from 0/12 to 4/12 for Qwen and from 0/12 to 3/12 for Llama. This shows that call-budget exhaustion explains part of the loss. It does not explain the full gap: Planning-B reaches 7/12 for both models with only 6 calls. We will state the conclusion as an implementation-by-budget interaction rather than an inherent defect or an equal-realized-cost ranking.

Robustness without Claude. Claude increases the magnitude of the pooled Feedback-B result, but it does not determine the positive conclusion. The model-specific HB-Code score effects are +11.57 points for GPT-4o, +8.56 for DeepSeek, and +68.75 for Claude. After excluding Claude, the pooled effect remains +10.07 points, with a task-bootstrap 95% interval of [5.21, 15.63] and paired sign-flip p = 0.0010. Thus the result remains positive and passes the Holm threshold of 0.0125 without Claude.

Corrected checker. On the same 200 frozen MBPP tasks per model, corrected Feedback-A improves accuracy by 2.0 points for Qwen and 2.5 points for Llama. The corrected checker also improves output-contract compliance. The corrected-versus-original differences do not cross the Holm-adjusted threshold (adjusted p = 0.500 and 0.539), so the correction does not reverse the statistical conclusion in this matched audit. We will report the repaired checker as a separate intervention while retaining the historical factorial result as the measurement of the original implementation.

Reproducibility details. The main factorial uses the November 2024 version of GPT-4o, DeepSeek Chat V3.1, and Claude Sonnet 4.6, with temperature 0 and a 4096-token per-call cap. No generation seed was specified. Default OpenRouter routing was not pinned, so the original records do not separate model from routing variation; the new replication addresses this by fixing Llama 3.1 8B Instruct to DeepInfra with fallback disabled. The local replacements use Qwen2.5 7B Instruct and Llama 3.1 8B Instruct with greedy decoding; task selections were frozen before generation and their checksums were recorded. Gemini 2.5 Flash was excluded before analysis because its returned format did not map reliably to the evaluator's expected answer schema, and no Gemini record enters the 4608 evaluations. The revision will consolidate these settings and results in one protocol table.

Endpoint means the retained failure-inclusive outcome after the harness stops. Math uses exact correctness. HB-Code uses fraction of tests passed as its primary score and reports all-tests-passed accuracy separately; failed and incomplete episodes remain in the denominator.

Thank you for the additional experiments and detailed clarifications. I remain concerned about the scope of the contribution. The original global A/B comparisons change multiple component implementations simultaneously. The new mixed-implementation experiments provide evidence for individual replacements, but the reported findings remain closely tied to the selected implementations, component backgrounds, and resource budgets. I am still not convinced that the results establish sufficiently generalizable insights or a sufficiently substantial contribution beyond existing component attribution work.

The limited scale of the main evaluation also remains a concern. I acknowledge that the Feedback-B effect remains positive and statistically significant after excluding Claude. However, that analysis covers only two models and 18 coding tasks. The reported uncertainty supports an effect within this tested setting, while providing limited evidence of robustness across a broader range of tasks and models.

Overall, my concerns about the contribution and empirical scope remain. I therefore maintain my score. I hope these comments will be helpful as you further revise and strengthen the paper.

Reviewer_kLs3OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper investigates how concrete implementations of language-model harness components affect downstream task performance. The authors introduce HarnessBench, a modular factorial evaluation framework covering four components: Context, Memory, Feedback, and Planning. Each component has a shared null implementation and two active implementations, organized into two global variants.

Critical strengths

The paper provides a structured factorial framework for evaluating harness components and their interactions. Evaluating all 16 configurations per implementation variant enables paired comparisons across component settings, which is more informative than comparing only a few arbitrarily selected complete harnesses.

Critical weaknesses

W1. Limited methodological novelty and insufficiently differentiated contribution. The paper primarily presents factorial ablations of existing harness components rather than introducing a novel methodology or algorithm. The central finding that harness effectiveness depends on implementation choices is relatively intuitive. The authors do not sufficiently differentiate their contributions from existing work on harness optimization and component attribution.

W2. Limited benchmark coverage and generalizability. The primary factorial experiments include only 30 math and 18 coding tasks. Although additional experiments on MATH-500 and MBPP are provided, they mainly focus on Planning and Feedback. The lack of diverse, long-horizon, interactive agentic benchmarks limits the generalizability of the findings, particularly for Context and Memory components.

W3. Limited actionable insights for harness design. While the paper demonstrates that implementation choices affect performance, it offers limited practical guidance for designing better harnesses. Most findings identify performance differences without sufficiently explaining the underlying mechanisms or deriving actionable design principles. For example, the substantial degradation caused by Planning-A does not translate into clear recommendations for improving planning implementations.

W4. Lack of grounding in established agent frameworks and insufficient task difficulty. The evaluated harness implementations are largely simplified, manually designed mechanisms rather than components integrated into established agent frameworks such as SWE-agent or mini-swe-agent. Furthermore, the evaluated tasks are relatively easy for the selected models, with the bare baseline achieving 94.8% accuracy on MATH-500. Such settings may not adequately expose the benefits of sophisticated harness components, limiting the practical relevance of the conclusions.

W5. Poor writing and presentation. The paper's organization makes it difficult to follow the central research questions and findings. Multiple experimental settings and comparisons are introduced without a sufficiently clear overarching narrative, while extensive implementation details and methodological qualifications obscure the main takeaways. A more coherent experimental overview and clearer synthesis of the results would substantially improve readability.

Questions for authors

See weakness.

2 comments

Thank you for asking for stronger component-level evidence and actionable explanations. We have completed 888 additional conditions:

Additional experimentConditionsPurpose
Local fixed-background replacements528Isolate individual implementation changes
Fixed-provider API replication264Evaluate replacements with pinned provider routing
Call-budget controls96Diagnose planning-feedback budget interactions

Actionable implementation diagnosis. The new replacement experiments use 12 MATH-500 and 12 MBPP tasks. On MBPP, fixing Context, Memory, and Feedback to B and replacing only the planning implementation gives:

Execution settingPlanning-APlanning-B
Qwen, local0/127/12
Llama, local0/127/12
Llama, fixed DeepInfra provider0/125/12

Entries are tasks passing all tests under a common six-call limit and 16,384-token ceiling. The local models are Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct; the API model is Llama-3.1-8B-Instruct.

The budget controls explain an important part of this difference. Increasing the call allowance to twelve, without increasing the token ceiling, restores Planning-A to 4/12 on Qwen and 3/12 on Llama. This yields concrete design guidance: budget the complete planning-and-feedback execution path, including revision calls, rather than planning steps alone.

Practical repairs. The corrected Feedback-A audit improves task accuracy from 49.5% to 51.5% on Qwen and from 29.0% to 31.5% on Llama. Output-contract compliance also improves, from 82.0% to 84.5% and from 82.0% to 86.0%, respectively.

Together, the replacement experiments and repair controls connect implementation changes to measurable execution behavior. We will foreground this evidence chain in the revision: fixed-background replacement, execution diagnosis, targeted intervention, and measured outcome. The practical contribution is a diagnosis of which implementation to change, what failure mechanism to address, and how to verify the resulting improvement.

Thank you for your response. Most of my concerns remain unaddressed. I will keep my original score.

Reviewer_CTVsOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes HarnessBench, which uses full-factorial experiments to examine the context, memory, feedback and planning components within language model execution frameworks. The study set up two global implementation versions, each evaluating all 16 component configurations, and conducted a total of 4,608 evaluations across three API model identifiers, 30 HB-Math tasks and 18 HB-Code tasks. Subsequent experiments examined reproducibility, inference cost, MATH-500 performance, truncation of intermediate result summaries, and comparisons of feedback prompts that share the first answer. The results indicate that component effectiveness is implementation-dependent, including performance degradation caused by the multi-step planners tested, and that the benefits of self-checking relative to neutral revisions vary across models. The paper also diagnoses flaws in the legacy output checker and describes a fix, though the benchmark performance of the revised version has not yet been evaluated. The authors explicitly limit their conclusions to the tested implementations and settings.

Critical strengths

S1. The feedback comparison using a shared initial response provides valuable evidence for distinguishing the effect of the self-check prompt from that of simply adding an extra model invocation.

Section 6 compares the self-check with a neutral revision using the same initial response, the same prefix length constraint for the initial response, and a single additional invocation, whilst not providing evaluator feedback to the model. The difference between B and the neutral revision on Llama was +16.0 percentage points, with a reported 95% interval of [10.0, 22.5]; on Qwen, it was –0.5 percentage points, with an interval of [–4.0, 2.5]. This supports the notion that the revision prompts tested yield model-dependent benefits. The authors rightly distinguish this comparison from that of ‘with feedback versus no revision’, and also acknowledge that an equal number of calls does not imply the same output token cost.

S2. Reliability controls and cautious interpretation of results enhance the credibility of the empirical audit.

Self-consistency controls with similar call budgets, API re-runs, validation on 500 MATH-500 tasks, and 3,200 local model planning runs all provide substantial evidence beyond small-scale full-factorial task sets. Local evaluations retained failed and incomplete runs, and the paper also examined saved MBPP outputs in isolated evaluators. The authors distinguish between task-level uncertainty and fluctuations in repeat runs, between pointwise confidence intervals and significance judgements corrected for multiple testing, and between whether outputs meet agreed requirements and semantic correctness. The paper explicitly treats the global A/B switch as a combined intervention involving multiple concurrent changes across implementations, making it easier to provide credible explanations for negative and mixed results.

Critical weaknesses

W1. The practical implications of the identified feedback verification flaws, as well as the specific operational causes of the performance decline in Planning-A, remain inadequately explained.

Section 7 identifies specific issues with the older version of the checker: heuristic rules targeting mathematical keywords may be applied to programming tasks, and once the number of retries is exhausted, verified=True may be returned without a final check. The comparison of Feedback B minus A for HB-Code in Figure 1 (p = 0.0002) includes this older version of Feedback-A, and the shared first-answer experiment in Section 6 also continues to use it. The paper does not report benchmark evaluations of the revised, task-aware checker. The measurements from the older version remain of reference value, and the independent task scores are distinct from the verifier’s verification labels. However, the paper has not yet demonstrated how the comparison results would change after the known defects have been rectified; consequently, its diagnostic contribution in guiding practical improvements remains limited. The comparison between B and the neutral revision nevertheless constitutes independent positive evidence.

Q1—Targeted validation: Could the authors incorporate the revised version of Feedback-A into the existing frozen-task, shared-first-answer comparison, whilst retaining the retry quota of the original A, and report the task accuracy, whether the final output meets the agreed requirements, as well as the actual number of invocations and token consumption? Please clarify which conclusions remain applicable only to the older version of the checker. As a related clarification, reporting first-answer format/syntax failures separately from semantic errors would also help to understand Claude’s larger gain from Feedback-B (+0.688) on HB-Code: although all three models showed positive effects, Claude contributed the majority of the overall average gain of +0.296.

The call counts for Planning-A also require more precise clarification. Table 3 reports an average of 6.17 calls per HB-Math task, whilst the planning prompt requests the generation of 2–5 steps. This prompt requirement does not establish a hard call limit for the API experiments; the six-call limit explicitly stated in Section 5.4 applies to a separate set of experiments involving local models. On MATH-500, Planning-A made 5.75 calls per question but completed only 1.76 steps; the paper does not define what constitutes ‘completing a step’. The truncation control for local models is valuable, but the specific failure modes leading to these API results have not yet been identified.

Q2 – Clarification of existing runs: Please explain the number of calls in the context of the actual distribution of parsed steps, define what constitutes a ‘completed step’, and summarise failed or incomplete steps, as well as any retries or auxiliary calls included in the API call statistics. This will help to identify which specific run behaviours led to the performance degradation reported, whilst continuing to retain performance metrics that include failed runs.

W2. The scope of component-level diagnostics and the reporting of results remain incomplete, particularly regarding Context and Memory.

Context-A only performs compression when the history exceeds six entries. Algorithm 1 appends one output for every planning step executed; Planning-A requests the generation of 2–5 steps, whilst both Planning-B and the ‘Planning disabled’ condition utilise a single prompt. If each task begins with an empty memory and empty history, and follows the context preparation loop illustrated, then all 16 Variant-B configurations, as well as the eight Variant-A configurations with ‘Planning disabled’, will prepare context before any cross-step history or memory has been accumulated. Memory-A’s five-entry working buffer and fifty-entry episodic store, as well as Memory-B’s ten-entry sliding window, also suggest that capacity pressure may be limited in short plans that meet the requested number of steps. These are conditional inferences that require confirmation via run records. The prompt format of Context-B, as well as other retrieval and filtering operations, may still influence model inputs. In the absence of evidence that mechanisms were actually triggered, it is difficult to determine which implementation features the Context and Memory comparisons specifically tested; larger-scale follow-up experiments primarily covered Planning and Feedback.

Q3 – Clarification of component coverage: Please confirm how states are initialised, whether they are persisted across steps or tasks, and how history is handled during retries, and briefly summarise the actual size of history/memory as well as compression, retention and retrieval behaviour. If certain mechanisms are not triggered, please clarify this boundary and explain what the measured comparisons represent.

Section 8 states that 24 model/domain/component comparisons were estimated when all other components were disabled, but no specific figures are presented. It is also unclear how the score values used for analysis in HB-Math and HB-Code are defined. Assuming binary scoring and an equally weighted average across 18 tasks and eight configurations of the other components, as described in the text, a component’s main effect for a single model on HB-Code should be an integer multiple of 1/144. The reported values of 0.116 and 0.086 do not conform to this granularity, even after rounding to three decimal places. Partial scores or different aggregation methods may explain this discrepancy; therefore, the scoring definitions must be clarified in order to account for these values.

Q4—Clarification of reported results: Please define the score values and their weights used for the analysis, including whether partial scores were employed, and report the scores when all components are disabled, as well as the isolated-component comparisons with task-level uncertainty. Providing the exact original text of the Feedback-B self-check prompts and neutral revision prompts would also assist in reproducing the comparison of prompt wording. Please also indicate the access points for the saved evaluation logs, the frozen task selections and the audit scripts: the PDF provided mentions these materials but does not supply access links to the artefacts.

Overall evaluation. I am inclined towards a weak rejection, close to the acceptance threshold. The controlled feedback comparisons and reliability studies are valuable; however, the diagnosis would be more persuasive if the actual results following corrections to the checker could be verified, and evidence provided as to which mechanisms were actually triggered by the four-component design. The absence of scoring definitions, single-component estimates and exact revision prompts also makes it more difficult to understand and evaluate the paper. The statistical design described in the paper is generally prudent; the numerical issues mentioned above constitute requests for clarification rather than confirmed arithmetic errors. Targeted verification and clearer run reports could substantially alter my assessment.

Questions for authors

See the ‘Weaknesses’ section.

2 comments

Thank you for the detailed suggestions. We have completed the corrected-checker audit, fixed-background replacements, and targeted execution diagnostics.

Corrected Feedback-A. We verified 800 repaired-checker conditions using the same frozen first answers and a maximum of two revision calls. Task correctness is measured by independent tests, and output-contract compliance by explicit public-interface checks.

ModelOriginal A accuracyCorrected A accuracyOriginal A contract passCorrected A contract pass
Qwen49.5%51.5%82.0%84.5%
Llama29.0%31.5%82.0%86.0%

Each rate uses 200 tasks. The corrected checker improves both task accuracy and output-contract compliance. Actual costs, including the shared first answer, are:

Model / checkerMean callsMean input tokensMean output tokens
Qwen / original A1.25188.81490.80
Qwen / corrected A1.34252.08546.71
Llama / original A1.28227.17575.26
Llama / corrected A1.31238.86600.76

We also separated first-answer contract failures from contract-valid answers that fail task tests: the respective counts are 37 and 72 for Qwen, and 32 and 114 for Llama. This distinguishes interface and syntax problems from failures on the task tests.

Planning failure mechanisms. We completed 96 additional conditions that increase the call limit from six to twelve while retaining the same 16,384-token ceiling. With Context, Memory, and Feedback fixed to B, Planning-A's MBPP results are:

ModelSix-call limitTwelve-call limit
Qwen0/124/12
Llama0/123/12

This intervention identifies call-budget exhaustion from combining multi-step planning with per-step feedback as an operational failure mechanism. A completed step is one whose generation and feedback processing finish and whose output is stored before the plan advances. Call accounting includes the planning invocation, step generations, and feedback revisions.

Context and Memory activation. The runner appends two history messages per completed step: the step prompt and its response. The fifth context preparation can therefore receive eight messages and trigger compression. The new 528-condition trace audit records 73 compression events and 772 context preparations with nonempty memory. Memory written after a step is available to subsequent steps; history and memory are reset between tasks.

Scoring and isolated comparisons. We recomputed all 24 isolated-component comparisons with task-level uncertainty. HB-Code's analyzed score is the fraction of tests passed, explaining values that are not multiples of 1/144; all-tests-passed accuracy is reported separately. We will include both endpoints, null-baseline scores, the isolated comparisons, and the exact revision prompts in the revision.

Reviewer_CTVsOpenReview ↗

Follow-up to the author’s response

Thank you to the author for their response. The revised Feedback-A results and scoring definitions address the main request in my Q1 and the 1/144 issue. Remaining issues:

  1. Whether the mechanism was triggered (Q3). It remains unclear whether compression, memory retrieval and Feedback-A retries were triggered during the original 4,608 evaluations. The 73 compression events originated from new local experiments; whilst in the 24 single-step configurations, both history and memory were empty when the context was prepared.
  2. Planning-A failures (Q2). The failures in the original experiments remain unexplained: 6.17 calls per question in Table 3; 5.75 calls on MATH-500 resulted in only 1.76 steps completed. The 12-question budget experiment used a different setup (Feedback-B applied at every step, with call limits of 6 and 12 respectively).
  3. Claude’s results on HB-Code. The breakdown between “format/syntax failures” and “test failures” only provided data for the local model on MBPP; no data was provided for Claude on HB-Code.
  4. Evaluation criteria. HB-Code scores based on the proportion of tests passed, whilst MBPP scores based on whether all tests were passed. The results for HB-Code under the “all-pass” criterion were not provided in either the paper or the response.
  5. Please provide:
    • 24 isolated comparisons;
    • scores when all components are disabled (all-OFF) in the main experiment;
    • the original text of the Feedback-B prompt and the neutral-revision prompt;
    • the complete table of replacement experiments;
    • the revised PDF.

Minor issues.

  • Line 10 of Algorithm 1 contradicts the statement that two historical messages are appended after each step.
  • Both 37 + 72 and 32 + 114 exceed the number of failures implied in Table 5.
  • Describing these differences as “improving task accuracy” is an overstatement: the corrected p-values of 0.500 and 0.539 are not statistically significant.
Reviewer_GwDaOpenReview ↗
Rating
3
Confidence
1
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies whether the effect of an LLM harness component depends on its concrete implementation. HarnessBench defines context, memory, feedback, and planning components, each with an inactive baseline and two active implementations. A complete factorial evaluation covers all 16 configurations, three model identifiers, and small math and coding task sets. The global contrasts differ for planning in both domains and for context and feedback in coding. Because each contrast also changes other active implementations, it estimates a bundle-level effect. Follow-up controls test several narrower explanations. Planning-A remains substantially less accurate than bare inference on MATH-500 despite using more calls and tokens. Removing summary truncation leaves its accuracy unchanged on local models. In a matched feedback experiment, self-check prompting gains 16 percentage points over neutral revision for Llama and loses 0.5 points for Qwen. These results show that harness effects depend on the implementation, model, domain, cost, and failure behavior.

Critical strengths
  • Careful estimands. Section 4 keeps marginal factorial effects, isolated additions to a null harness, and matched revision comparisons distinct.
  • Tables 3–5 report failure-inclusive accuracy together with calls, tokens, and incomplete episodes, making operational reliability part of the evaluated procedure.
Critical weaknesses
  • The main B-A contrast changes every active implementation in each coalition and identifies a bundle-level difference. A one-component replacement design is needed for component-level attribution.
  • HB-Math and HB-Code contain only 30 and 18 tasks, respectively. HB-Math lacks contamination control, and provider routing was unpinned.
  • Two hand-designed implementations cover too little of the implementation space to support a deployment ranking.
Questions for authors
  1. Can you replace one component at a time with the other three active implementations fixed? Please run the replacement against both A and B backgrounds. This would separate the named implementation effect from changed interactions in the global bundles.
  2. Do the conclusions persist on established agent benchmarks with pinned providers and a common token or monetary budget, particularly for the large Planning-A and Feedback-B effects?
  3. What additional conclusion does the dual-implementation design provide beyond applying ShapleyFlow or AgentSHAP to the same saved records?
1 comment

Thank you for suggesting one-component replacements against both fixed backgrounds. We have completed these experiments and a fixed-provider API replication.

Single-component attribution. We evaluated 528 local-model conditions on 12 MATH-500 and 12 MBPP tasks. Each replacement changes only one component, while the other three implementations remain fixed to either A or B. All conditions use a 16,384-token episode ceiling and a six-call limit.

On MBPP, replacing only Planning-A with Planning-B yields:

Model / executionFixed backgroundPlanning-APlanning-B
Qwen, localA1/126/12
Llama, localA0/123/12
Qwen, localB0/127/12
Llama, localB0/127/12
Llama, OpenRouter / DeepInfraB0/125/12

Entries are tasks passing all tests. The fixed background specifies the implementations of Context, Memory, and Feedback. The local models are Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct; the API model is Llama-3.1-8B-Instruct.

These comparisons directly isolate the planning replacement. The additional 264-condition API replication fixes both the model identifier and provider, with fallback routing disabled.

What the implementation comparison adds. The replacement experiments measure mixed-implementation configurations absent from the original global A/B matrices. Attribution computed from the original saved records cannot recover these unobserved outcomes. Our contribution is the combination of measured replacement effects, execution traces, and targeted controls that explain why implementation choices change performance.

The budget controls provide a concrete example: increasing the call allowance from six to twelve, while retaining the same token ceiling, restores Planning-A's MBPP accuracy from 0/12 to 4/12 on Qwen and from 0/12 to 3/12 on Llama against the B background. This identifies the interaction between multi-step planning, per-step feedback, and the available call budget.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.