- Rating
- 3(+1)
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
ReHarness learns a reusable harness adaptation strategy through cross-task optimization. A learned adapter modifies task-specific worker harnesses using five support examples. The paper reports consistent improvements across four models, but insufficient runtime disclosure and unusually weak baselines make the significance of these gains difficult to assess.
Critical strengths
- Separating reusable adaptation strategies from task-specific harnesses is well motivated, and the support/query formulation encourages generalization.
- Improvements are consistent across the reported model-benchmark combinations.
- The five-shot, initial-adapter, and history ablations provide useful evidence for the contribution of offline optimization.
Critical weaknesses
- My primary concern is the unexplained baseline underperformance. DeepSeek-V4-Flash scores 26.7% initially and 33.3% after ReHarness on SWE-bench Verified, compared with 73.7–79.0% in its official model card; similar things happen to other models, such as GLM 5.3 Flash and GPT 5.6 Sol. In fact, most of the frontier models score around 80% on Swe-Bench verified. Although protocols may differ, the paper does not disclose evaluation settings to reconcile this gap. The gains may reflect recovery from an inadequate initial harness; no specific misconfiguration can be established from the available information.
- The initialization ablation only considers harnesses constrained to remain simple and general. The method’s advantage over competitive agent systems is unclear.
- The diversity ablation increases source-task count without matching total data and computation. So it does not isolate the benefit of task diversity. An adapter trained with single task under matched resources would also help establish whether cross-task learning is necessary.
- The paper does not clearly show what the adapter learns. Several changes in the case study involve answer extraction and formatting, but their contribution to the gains is not measured. The authors should separate these changes from improvements in reasoning and task execution.
Questions for authors
- Can you provide the complete initial and evolved harnesses, prompts, tools, reasoning settings, token budgets, execution limits?
- What explains the baseline performance gap?
- Does cross-task diversity still help when total data and computation are fixed, including a single-task trained adapter control?
7 comments
Response to Weakness 1 & Weakness 2 & Question 2
To Weakness 1 & Weakness 2 & Question 2 (Reasons for the Gap and Additional Experiments):
Thank you for your comments. In the following, we would like to discuss the reasons for the performance gap and provide experiments under the mainstream SWE-bench evaluation setting to verify the method's advantage over competitive agent systems. Specifically, we have revised three parts of the main text in our rebuttal revision with changes highlighted in $ \textcolor{blue}{blue}$: (1) the first paragraph of Section 4, (2) a new paragraph in Section 4.1 (Main Results), and (3) Table 3. The detailed experimental configuration is provided in the newly added Appendix D.10.
Reasons for the performance gap. As discussed in our response to Weakness 1 & Question 1, we use a general worker harness with the necessary adaptation interfaces, following the design of Meta-Harness, rather than the mini-SWE-agent configuration used in the official SWE-bench evaluation, to measure the adapter agent's ability to modify the worker harness more directly. Our setting therefore differs from the model-card setting in agent implementation. This is the main source of the performance gap.
Evaluation under the official mini-SWE-agent configuration. To test whether the method remains effective on a stronger and widely used agent system, we additionally evaluate it on the mini-SWE-agent protocol. This setting uses the upstream DefaultAgent, LiteLLM model, and Swe-bench agent prompt, one fresh E2B sandbox per instance, the same SWE-bench Verified data and official E2B grader, 250 maximum interaction steps, 98304 maximum generated tokens, a 60-second per-command timeout, a 2400-second E2B sandbox timeout, and patch extraction from git diff. To adapt our method to this protocol, we retain the tool loop, prompt, and execution environment unchanged, and attach the general worker harness as an additive layer to the agent's task and context construction. Then we re-evaluate all four models used in our experiments under this official mini-SWE-agent configuration. The results are shown in the table below.
| Method | DeepSeek-V4-Flash | GLM-5.3-Flash | Qwen3.8-Flash | GPT-5.6 Sol | Avg. |
|---|---|---|---|---|---|
| Baseline | 72.7 | 83.0 | 77.0 | 84.9 | 79.4 |
| ReHarness | 78.2 (+5.5) | 85.5 (+2.5) | 80.6 (+3.6) | 88.5 (+3.6) | 83.2 (+3.8) |
Under this official protocol, the baseline reaches 72.7%, 83.0%, 77.0%, and 84.9% on the four models, which recovers to almost the same range as the scores reported in the model cards and public leaderboards. This confirms that the earlier performance gap is mainly due to different evaluation protocols. As shown in the table above, ReHarness improves over this stronger baseline by an average gain of 3.8%. These results show that ReHarness remains effective on a strong and widely used agent system.
Response to Weakness 3 & Question 3
To Weakness 3 & Question 3 (Task Diversity Experiments under Fixed Data Budget):
Thank you for your comments. To isolate the effect of task diversity from that of the data budget and computation, we add an ablation in which both the total amount of offline data and the total number of task-level adaptations are fixed. All the analyses and experiments have been added in a new paragraph following Appendix C.4 in our rebuttal revision, with changes highlighted in $ \textcolor{blue}{blue}$. We keep the support and query configuration within each batch unchanged and keep sequential proposals per batch, and vary only the number of given tasks and the number of batches per task. In this ablation, the multi-task settings use the same task types as our earlier diversity analysis, while the single-task control is trained separately on SpreadsheetBench and BFCL V4.
| Given tasks | Batches | Size of support set | Size of query set | Total data |
|---|---|---|---|---|
| 1 | 12 | 5 | 15 | 12 1 (5 + 15) = 240 |
| 2 | 6 | 5 | 15 | 6 2 (5 + 15) = 240 |
| 3 | 4 | 5 | 15 | 4 3 (5 + 15) = 240 |
| 4 | 3 | 5 | 15 | 3 4 (5 + 15) = 240 |
This keeps the total offline data constant. Under this setting, we evaluate them on three new tasks not used in the offline phase, GAIA, -Bench, and OfficeQA-Pro. The results are shown in the table below.
| Setting | GAIA | -Bench | OfficeQA-Pro | Avg. |
|---|---|---|---|---|
| Baseline | 28.7 | 79.5 | 36.7 | 48.3 |
| 1 task (SpreadsheetBench) | 21.6 | 76.2 | 11.2 | 36.3 |
| 1 task (BFCL V4) | 29.8 | 76.8 | 32.1 | 46.2 |
| 2 tasks | 30.6 | 85.5 | 37.3 | 51.1 |
| 3 tasks | 33.6 | 83.5 | 41.4 | 52.8 |
| 4 tasks | 34.1 | 86.2 | 40.4 | 53.6 |
With the total data budget fixed, increasing the number of given tasks still improves the average performance on new tasks. GAIA improves monotonically across all settings, while -Bench and OfficeQA-Pro show non-monotonic variations, consistent with the observation in Appendix C.4. The gain from 2 to 4 tasks is smaller than in the original Appendix C.4, indicating that part of the gain reported in Appendix C.4 came from the larger data budget. However, the direction of the task diversity effect remains the same, as more given tasks still lead to better average performance when the total data budget is held constant.
More importantly, an adapter harness evolved on a single task under the same budget performs worse than both the baseline and all multi-task settings. A single task provides too narrow a signal to capture a reusable evolution strategy, so the resulting adapter harness overfits to task-specific behavior and transfers poorly. This negative transfer is highly dependent on the nature of the source task. The adapter harness optimized on SpreadsheetBench reaches only 11.2% on OfficeQA-Pro, as it learns spreadsheet-specific calculation rules that mislead on document-based numerical QA. In contrast, the adapter harness optimized on BFCL V4 reaches 29.8% on GAIA, slightly above the baseline, likely because BFCL V4 involves tool-calling behaviors that remain useful for GAIA's search-tool use. These disparities indicate that a single-task adapter captures task-specific characteristics rather than a general reusable strategy, further supporting the necessity of cross-task evolution.
Response to Weakness 1 & Question 1
Thank you for your comments. We would like to provide our detailed experimental configuration on SWE-bench Verified evaluation. And we have incorporated all these details into Appendix D.9 in our rebuttal revision, with changes highlighted in $ \textcolor{blue}{blue}$.
Initial and evolved harnesses. The initial worker harness is a minimal and general worker scaffold with six functions: repository inspection, shell command execution, observation parsing and message construction, file modification through shell commands, focused test execution, and final patch extraction from the workspace. The patch is collected from git diff against the initial repository HEAD. We use this minimal and general harness to measure the adapter agent's ability to modify the worker harness more directly. The evolved harness preserves the same interface and execution protocol, but changes the worker agent’s behavior in four ways. It adds an explicit workflow of inspection, reproduction, editing, and testing, instructs the worker agent to make the smallest repository change and rerun focused tests, requires the final response to contain an applicable unified diff, and normalizes and extracts the diff before submission.
Prompts. The worker harness prompt is built from our general worker prompt, with the official SWE-bench prompt content appended. The general part describes the basic worker agent’s role and the interaction protocol. The appended official content specifies the SWE-bench task statement, the testbed working directory, the expected patch output, and the prohibition on committing or resetting the repository. The detailed prompts have been provided in Appendix D.9.
Tools. The worker agent receives a shell interface inside the E2B sandbox. Repository reading, searching, editing, and test execution are all performed through this interface.
Reasoning settings. We use a medium reasoning effort for GPT-5.6-Sol and Qwen3.8-Flash, and a high reasoning effort for DeepSeek-V4-Flash and GLM-5.3-Flash. All four settings are within each model’s supported range. The model endpoint and exact model revision are fixed by the evaluation configuration and recorded in the run artifacts. For sampling, each model runs with its API-native configuration. GPT-5.6 Sol uses its API-default sampling behavior because it doesn’t accept temperature or top-p. DeepSeek-V4-Flash runs in thinking mode with temperature ignored and top-p at its default of 1.0. Qwen3.8-Flash and GLM-5.3-Flash run in thinking mode with temperature=1.0 and top-p=0.95.
Token budget. Each model call has a maximum generation budget of 98304 tokens. The budget includes the complete response generated for one worker agent’s step.
Execution limits. Each instance runs in an isolated E2B sandbox with 8 CPUs and 8 GB memory. The worker agent’s loop allows at most 250 interaction steps. Individual shell commands are limited to 120 seconds, the worker sandbox wall-clock timeout is 3600 seconds, and official patch grading uses a separate 600-second timeout, mainly following the recommended configuration in AgentCompass’s SWE-bench evaluation.
If you would like any further details, please let us know and we will be glad to provide them.