- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how model–harness pairings affect LLM agent performance on benchmarks. The authors introduce Valid Completion Rate (VCR) to measure the proportion of runs that terminate normally with a valid evaluator handoff. They construct a Joint Score by averaging the product of each attempt's valid-completion indicator and official task score, retaining credit only from valid completions. Model–Harness Fit and Cross-Model Fit are then defined using this score. The experiments cross DeepAgents, OpenCode, Pi CLI, and Codex CLI with multiple models on six benchmarks covering code repair, terminal interaction, long-context tasks, and artifact handoff. The results show that task performance, valid completion, and resource use all depend on the specific combination of model, harness, and task.
Critical strengths
- The paper clearly distinguishes agents' task performance from operational validity. VCR measures the proportion of runs that terminate normally with a valid evaluator handoff, while Joint Score captures task credit earned through these valid completions. Reporting both metrics alongside official task scores provides a more informative account of how performance is achieved than task success rates alone. It also helps distinguish task-solving failures from execution and handoff failures.
Critical weaknesses
-
Appendix C.3 separates failure stages, and Appendix F.4 examines specific runtime errors for Kimi K2.7 on LOCA-Bench. However, comparable analyses are missing for other pairings with high failure rates. For example, MiniMax M3–DeepAgents achieves a VCR of only 32.33% on LOCA-Bench. Although the paper shows that failures primarily occur before normal termination, it does not report a breakdown of error types for this pairing. Without such root-cause analysis, it is difficult to determine how much the low performance reflects poor model–harness compatibility rather than failures specific to the execution conditions.
-
The paper reports extensive results across models, harnesses, and benchmarks, together with comparisons across multiple metrics and robustness checks. However, its conclusions largely describe performance differences among configurations. The paper does not systematically establish which task characteristics or interaction demands favor particular model–harness pairings, or whether these findings can predict pairing performance on new tasks.
Questions for authors
The paper shows multiple model–harness ranking reversals, as well as differing rankings under task scores and VCR. Although the aggregate results establish that these differences exist, they do not sufficiently explain how they arise. Could the authors provide a fuller explanation of these differences, drawing on the experimental results and building on the existing analyses?
5 comments
Weakness 2
The paper reports extensive results across models, harnesses, and benchmarks, together with comparisons across multiple metrics and robustness checks. However, its conclusions largely describe performance differences among configurations. The paper does not systematically establish which task characteristics or interaction demands favor particular model–harness pairings, or whether these findings can predict pairing performance on new tasks.
Our goal is to provide a framework for evaluating model–harness pairings on a given task and selecting among them according to the task’s requirements. We compare candidates under matched conditions and distinguish two criteria that can favor different choices: how reliably a pairing completes execution and how much task credit it earns through valid completion. We apply the same framework across six benchmarks with substantially different interaction demands and scoring rules. Each benchmark retains its own tasks, evaluator, and success criteria. What transfers across benchmarks is the method for comparing and selecting pairings.
Our AgentSkillOS stress test provides a concrete example of how interaction demands affect pairing selection. We evaluate the same 30 tasks with 200- and 900-skill catalogs, using four repetitions per pairing in each condition. For MiniMax M3, OpenCode leads Pi CLI in official Task Score under 200 skills (93.49 vs. 85.39), whereas Pi CLI leads under 900 skills (89.98 vs. 64.08). Qwen3.6-35B-A3B shows the same reversal, while Pi CLI has the highest Task Score among the crossed harnesses for Kimi K2.7 in both conditions (Appendix F.6, Table 25). These results show that the relative advantage of a harness depends on both the model and the candidate skill catalog presented during execution. They identify a concrete interaction condition relevant to pairing selection and illustrate how our framework supports comparisons as interaction requirements change.
Prior work provides a clear reason to make this selection conditional on the task. In τ²-Bench, agents were less successful when they had to coordinate actions with a user than when they could execute the actions themselves [1]. Harness-Bench found that harness dependence varied across workflow categories, with larger cross-harness variation in data analysis, workspace/tool use, and software engineering [2]. These findings motivate comparing candidate pairings directly under the conditions relevant to the intended application.
Our contribution is a transferable measurement framework. Given a new benchmark, researchers can use it to test the pairings relevant to their task and select among them using observed results. A comprehensive audit of all current model–harness combinations and the prediction of a winning pairing for an unseen task before execution are separate research objectives, outside the scope of this study.
Question
The paper shows multiple model–harness ranking reversals, as well as differing rankings under task scores and VCR. Although the aggregate results establish that these differences exist, they do not sufficiently explain how they arise. Could the authors provide a fuller explanation of these differences, drawing on the experimental results and building on the existing analyses?
A model–harness pairing’s performance depends on how well the two components meet the demands of the task. As we mentioned in Seciton 3, the model must interpret the task and decide what to do. The harness must translate those decisions into actions, execute them in the environment, and return feedback the model can use. The task also defines what must be delivered and how it will be scored. Changing tasks can therefore change which pairing performs better. Even on the same task, rankings can reverse when the evaluation criterion changes: the official score measures what the agent accomplished, VCR measures whether it terminated normally with a valid handoff, and Joint Score measures how much task credit was earned through those valid completions.
Section 6.2 and Appendix D give a concrete reversal whose source we can identify. On SWE-bench-45, MiniMax M3 with DeepAgents recorded slightly more official successes than the same model with Codex CLI. Ten DeepAgents attempts produced patches that passed the official tests even though execution terminated abnormally. Every successful Codex attempt, in contrast, also completed validly. Those ten attempts explain the reversal: DeepAgents leads when all successful patches count, but Codex leads when success also requires valid completion.
These results clarify how we interpret ranking reversals. Across tasks, a reversal shows that a pairing’s advantage depends on the reasoning, actions, and deliverables the task requires. Within a task, a reversal across metrics shows which aspect of performance accounts for that advantage. In the SWE-bench-45 example, both rankings use the same runs and patches. The official task score counts all patches that pass the tests, giving DeepAgents a slight lead with 56 successful attempts versus 55 for Codex CLI. However, ten of those DeepAgents successes come from abnormally terminated runs, whereas all 55 Codex successes also complete validly. Requiring valid completion therefore changes the credited successes to 46/180 for DeepAgents and 55/180 for Codex CLI, reversing their order. Our framework makes the source of this reversal explicit: the ten DeepAgents attempts contribute to the official task score but not to the Joint Score because they achieve task success without valid completion.
By tracing a ranking reversal to the runs that change the ordering, our analysis framework gives a task-level audit a concrete starting point: did the model choose an appropriate action, did the harness execute it as intended, and did the resulting work meet the task requirements? The answers can guide model training on action selection and the use of execution feedback, or inform changes to how a harness executes actions, handles errors, and verifies the final handoff.
References
[1] Victor Barres et al. τ²-Bench: Evaluating Conversational Agents in a Dual-Control Environment. 2025.
[2] Yilun Yao et al. Harness-Bench: Measuring Harness Effects across Models in Realistic Agent Workflows. 2026. Appendix C, Figure 4. arXiv:2605.27922v1.
Dear Reviewer T2Cx,
We sincerely thank you for recognizing our work and the substantial effort invested in reviewing our manuscript. We greatly appreciate the insightful comments and valuable suggestions provided, which have helped us further refine our paper. In this rebuttal, we address each of your concerns and suggestions point-by-point.
Weakness 1
Our framework evaluates how well a model and a harness work together under a benchmark’s task and execution requirements. We match task instances and external conditions, including tool access and resource limits, while retaining each harness’s native execution loop (Section 3). Completing execution under these requirements is part of the pairing’s performance, which we measure alongside task achievement.
The MiniMax M3–DeepAgents example is informative in the context of these crossed comparisons. On the same 75 LOCA-Bench tasks with four repetitions per pairing, MiniMax M3 achieves a VCR of 32.33% with DeepAgents and 98.00% with Codex CLI. For DeepSeek V4 Flash Preview, the corresponding VCRs are 95.33% and 98.67% (Appendix F.4, Table 19). The difference between these harnesses therefore depends strongly on the model. MiniMax–DeepAgents’ low VCR identifies a practical weakness in meeting LOCA-Bench’s execution requirements, which is directly relevant to selecting a pairing for this setting.
Cross-benchmark measurement and trajectory auditing answer complementary questions. VCR, official task scores, and Joint Score distinguish execution reliability, task achievement, and the credit earned through valid completion. The failure-stage decomposition in Appendix C.3 further separates abnormal termination from normal termination without a valid evaluator handoff. These measurements reveal how the pairings differ in outcomes and where losses occur in the execution protocol. A trajectory audit then connects the observed differences to the actions, execution events, and error-handling behavior recorded in individual runs.
Appendix F.4 demonstrates runtime-error analysis for Kimi K2.7 on LOCA-Bench. Revised Section 6.2 and Appendix C.4 further examine execution after recorded tool errors and the use of context compaction. These analyses show how the framework guides investigation of particular interactions while preserving the benchmark’s tasks, evaluator, and success criteria. Our contribution is a transferable method for measuring model–harness interaction and using those measurements to compare pairings and guide detailed failure analysis.