- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper proposes HarnessBench to study how concrete implementations of LLM runtime components affect task performance. The authors divide the system into four components: context, memory, feedback, and planning. Each component has two active implementations and a shared OFF baseline. The 24 full factorial experiment is conducted within each of two global implementation variants. The main experiment covers three fixed models and two task domains, mathematics and code, examining average component ON–OFF effects and differences in these effects between the two global variants. Follow-up experiments examine inference-call costs, run-to-run repeatability, external task sets, and the effects of summary truncation and feedback prompts. The authors emphasize that component effects should be interpreted in relation to the specific implementation, model, task, and resource costs, rather than assuming general benefits from component names alone.
Critical strengths
-
The paper points out that abstract component names can obscure substantial differences between concrete implementations. It introduces two sets of implementations and evaluates all ON/OFF combinations of the four components within each set, providing a relatively systematic diagnosis of these implementations.
-
The B-versus-neutral experiment holds the initial answer, visible prefix, and number of additional calls fixed, helping distinguish the benefit of an opportunity to revise from that of a specific self-check prompt.
Critical weaknesses
-
The global A/B comparison changes the implementations of multiple components simultaneously, so the observed difference cannot be attributed solely to replacing the target component. The rationale for selecting the two sets of implementations and their contribution beyond existing work on component attribution are also unclear. The current results do not establish sufficient insights beyond these specific cases.
-
The specific scoring rules for Ytmv(c,z), the execution flow in Algorithm 1, and the actual roles of Memory and Context are not clearly explained. Terminology is inconsistent and insufficiently defined. The original feedback checker has known issues, but its corrected version has not been evaluated for performance, limiting assessment of the reliability of the results and the factors underlying them.
Questions for authors
-
What new methods or findings does this paper offer beyond component attribution work such as ShapleyFlow and AgentSHAP? Why were Context, Memory, Feedback, and Planning selected, along with these two sets of implementations? What insights does the paper provide beyond these two specific sets of implementations?
-
The global A/B comparison also changes the implementations of other enabled components. How can the observed differences be attributed to the target component rather than interactions among components? Please report all 24 isolated contrasts mentioned in Section 8 and clarify whether comparisons were conducted that replace only the target component with A or B while holding the other components fixed. Please also explain together what questions the marginal contrast, implementation contrast, and isolated-component contrast each answer.
-
What are the specific scoring rules for Ytmv(c,z)? Please explain how answers are extracted and judged on HB-Math, how test results are converted into scores on HB-Code, and how parsing failures, execution failures, and incomplete outputs are handled. What exactly were the parsing failures for Gemini, and were the other models checked for similar issues? Did task-type misclassification and incorrect verified=True flags in the original Feedback-A affect output selection or final scores? Do the accuracy and costs of the corrected implementation still support the original conclusions?
-
Please provide the execution flows for A/B/Null in Algorithm 1, including what create_plan() returns when Planning is OFF, how Planning-B uses only one model call, and how the final output is selected. Do Memory and history store and pass on the same content? Can a single-step task use information newly written to Memory? If Planning-A has only 2–5 steps and history grows by one entry per step, how is the rule that compression starts only above six entries triggered?
-
In Section 5.4, is a summary a truncated intermediate output or a separately generated summary? Since A-trunc already retains the full conversation, which inputs does A-full actually change, and is this consistent with the 200-character limit in the main experiment? The cost controls for Planning and Feedback match or approximately match call counts, but not token consumption. What comparisons between methods do these results support, and do the conclusions hold under a common resource budget?
-
The main experiment contains only 30 mathematics tasks and 18 coding tasks, and the pooled Code Feedback-B effect is influenced by the large gain on Claude. Are the main conclusions robust to task selection and model weighting?
-
Do harness, runtime code/system/infrastructure, workflow, pipeline, and scaffold refer to the same concept in this paper? Please use these terms consistently or distinguish them clearly, along with bare/null/neutral and episode/condition.
4 comments
We thank the authors for their response during the rebuttal phase. However, several critical questions remain unaddressed, and the newly provided data introduce a few new concerns that require further clarification.
1. Unaddressed or Incompletely Addressed Questions
①. Core Contribution & Rationale: The authors have not yet clarified the novel contributions of this work compared to existing component attribution studies (e.g., ShapleyFlow, AgentSHAP). The rationale for selecting these specific four components and the two concrete implementations remains unexplained. ②. Truncation Logic & Token Costs: Please clarify whether the summary in Section 5.4 is a "truncated intermediate output" or a "separately generated summary." Additionally, since token consumption is not matched in the Planning and Feedback comparisons despite matching call counts, do the original conclusions hold under a strict common resource budget? ③. Dataset Size & Stability: The main experiment relies on very small datasets (30 math, 18 code tasks). Please explain whether Claude's exceptionally large gain on Feedback-B dominates the overall component conclusion. Are the main findings robust to task sampling variations? ④. Reproducibility Details & Summary Table: The response omitted model versions, generation parameters, random seeds, details on Gemini's parsing failures, a clear definition of "endpoint," and how model-specific variance is distinguished from provider routing variance. Please provide a single summary table of all experiment configurations and main results, along with these metadata.
2. New Questions Raised by the Author's Response
①. Call Budget Bottleneck: The new data show Planning-A scoring much higher with 12 calls (Qwen 4/12, Llama 3/12) than with 6 calls (0/12). Does this imply that Planning-A's poor performance is primarily an artifact of the artificial budget cap rather than an inherent flaw in its mechanism? Please explicitly discuss this resource sensitivity in the manuscript. ②. Impact of the Corrected Checker: Given that the corrected Feedback checker improved accuracy and compliance on both Qwen and Llama, do the negative or non-significant statistical conclusions derived from the originally flawed checker change?
Thank you for the follow-up. We address the remaining questions directly below.
Contribution and component selection. ShapleyFlow and AgentSHAP attribute performance across component or tool coalitions under already specified implementations. HarnessBench asks a different intervention question: what changes when one concrete implementation is replaced while the other components are fixed? Our 528 local and 264 fixed-provider conditions execute mixed A/B configurations that are absent from the original matrices and therefore cannot be recovered from attribution over the saved outcomes. We selected Context, Memory, Feedback, and Planning because they control four distinct runtime decisions: visible state, retained state, answer revision, and execution flow. The A/B pairs contrast engineered multi-step mechanisms with simpler or model-native alternatives.
Truncation and resource sensitivity. The summary in Section 5.4 is not separately generated. A-trunc directly limits the intermediate output passed forward to 200 characters, while A-full removes this limit. The replacement experiments use common ceilings of 6 calls, 16384 total input-plus-output tokens, and 1024 output tokens per call. Increasing only the call ceiling to 12 raises Planning-A on MBPP from 0/12 to 4/12 for Qwen and from 0/12 to 3/12 for Llama. This shows that call-budget exhaustion explains part of the loss. It does not explain the full gap: Planning-B reaches 7/12 for both models with only 6 calls. We will state the conclusion as an implementation-by-budget interaction rather than an inherent defect or an equal-realized-cost ranking.
Robustness without Claude. Claude increases the magnitude of the pooled Feedback-B result, but it does not determine the positive conclusion. The model-specific HB-Code score effects are +11.57 points for GPT-4o, +8.56 for DeepSeek, and +68.75 for Claude. After excluding Claude, the pooled effect remains +10.07 points, with a task-bootstrap 95% interval of [5.21, 15.63] and paired sign-flip p = 0.0010. Thus the result remains positive and passes the Holm threshold of 0.0125 without Claude.
Corrected checker. On the same 200 frozen MBPP tasks per model, corrected Feedback-A improves accuracy by 2.0 points for Qwen and 2.5 points for Llama. The corrected checker also improves output-contract compliance. The corrected-versus-original differences do not cross the Holm-adjusted threshold (adjusted p = 0.500 and 0.539), so the correction does not reverse the statistical conclusion in this matched audit. We will report the repaired checker as a separate intervention while retaining the historical factorial result as the measurement of the original implementation.
Reproducibility details. The main factorial uses the November 2024 version of GPT-4o, DeepSeek Chat V3.1, and Claude Sonnet 4.6, with temperature 0 and a 4096-token per-call cap. No generation seed was specified. Default OpenRouter routing was not pinned, so the original records do not separate model from routing variation; the new replication addresses this by fixing Llama 3.1 8B Instruct to DeepInfra with fallback disabled. The local replacements use Qwen2.5 7B Instruct and Llama 3.1 8B Instruct with greedy decoding; task selections were frozen before generation and their checksums were recorded. Gemini 2.5 Flash was excluded before analysis because its returned format did not map reliably to the evaluator's expected answer schema, and no Gemini record enters the 4608 evaluations. The revision will consolidate these settings and results in one protocol table.
Endpoint means the retained failure-inclusive outcome after the harness stops. Math uses exact correctness. HB-Code uses fraction of tests passed as its primary score and reports all-tests-passed accuracy separately; failed and incomplete episodes remain in the denominator.
Thank you for the additional experiments and detailed clarifications. I remain concerned about the scope of the contribution. The original global A/B comparisons change multiple component implementations simultaneously. The new mixed-implementation experiments provide evidence for individual replacements, but the reported findings remain closely tied to the selected implementations, component backgrounds, and resource budgets. I am still not convinced that the results establish sufficiently generalizable insights or a sufficiently substantial contribution beyond existing component attribution work.
The limited scale of the main evaluation also remains a concern. I acknowledge that the Feedback-B effect remains positive and statistically significant after excluding Claude. However, that analysis covers only two models and 18 coding tasks. The reported uncertainty supports an effect within this tested setting, while providing limited evidence of robustness across a broader range of tasks and models.
Overall, my concerns about the contribution and empirical scope remain. I therefore maintain my score. I hope these comments will be helpful as you further revise and strengthen the paper.
Thank you for requesting isolated replacements, explicit scoring definitions, and corrected-checker validation. We have completed these analyses and additional execution diagnostics.
Separating the comparison questions.
We recomputed all 24 isolated contrasts and completed 528 local-model conditions covering replacements against both A and B backgrounds, using 12 MATH-500 and 12 MBPP tasks. Another 264 conditions replicate the design through a fixed OpenRouter/DeepInfra provider. These mixed-implementation interventions add outcomes that cannot be recovered by attributing scores from the original global A/B records alone.
Replacement and budget findings. On MBPP, with Context, Memory, and Feedback fixed to B:
All conditions retain the 16,384-token ceiling. The twelve-call results come from the additional 96-condition diagnostic. These experiments identify both the planning replacement effect and its interaction with the call budget.
Corrected checker. Using the same frozen first answers and a maximum of two revision calls:
The corrected checker also improves output-contract compliance on both models. Task accuracy is evaluated independently of the checker's verification label.
Scoring and execution flow. HB-Code scores represent the fraction of tests passed; all-tests-passed accuracy is a separate endpoint. We will explicitly distinguish these measures and provide the isolated estimates with task-level uncertainty.
History receives two messages per completed step, so compression can activate at the fifth context preparation. Our new traces confirm 73 compression events and 772 context preparations with nonempty memory. Memory written after a step becomes available to subsequent steps, and both history and memory are reset between tasks.
Planning-OFF executes the task as a single step. Planning-B places its reasoning instruction in a single generation prompt, while Planning-A makes a separate planning call followed by step executions. These definitions and the measured traces make the roles of planning, context, memory, and feedback explicit. We will use consistent terminology throughout the revision and organize the findings around the implementation change, execution mechanism, and measured outcome.