- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper divides verification in the agent harness into four layers, initial instructions, reminders during execution, a gate at completion, and independent review, and compares 17 designs. The experiments use seven programming models, report accuracy changes and relative cost on task subsets of Terminal-Bench and SWE-bench Verified, and also include an internal VisualBench of 18 tasks. The average gain from independent review is about 5.9 to 9.5 points on TB and 3.9 to 6.6 points on SWE, at a cost of about 1.00 to 1.25 times the baseline.
Critical strengths
Tables 2 and 3 list accuracy changes for 17 designs and seven models on two programming benchmarks, and negative cells are kept in the tables, for example Kimi on the L0 user setting is −8.3. Cost is priced separately by model and by reviewer, with the reject-style gate at L2 reaching about 1.5 to 1.8 times the baseline and L3 at about 1.00 to 1.25 times. L0 and L1 also have cells where accuracy rises and output tokens do not rise. Costs across the four layers and per-model results can be read directly from the tables.
Critical weaknesses
The tasks used to recommend harness designs were kept after looking at these models' baselines. Design recommendations need to be based on tasks fixed before the runs started. The 31 Terminal-Bench tasks come from an 89-task registry, and the difficulty band used the pilot pass rates of four models in the table. SWE first selected a batch of tasks by repair time, then adjusted them after the baselines were seen, so as to keep both tasks with room to improve and tasks already solved that are used to observe regressions. VisualBench is another 18 internal tasks. The positive averages on the main table are computed on this set of tasks whose baseline behavior had already been seen.
The reviewer is fixed as Qwen3.7-Plus, and the paper does not manually check whether the review comments are reliable. All seven programming models are reviewed by this one model, with new context, sandbox checks, and one rework. The experiments do not include a control that gives the original agent the same extra tokens and the same one rework without introducing a second model, and they do not replace the reviewer model. The human evaluation in the paper evaluates finished VisualBench outputs and does not check whether the review comments point out real defects, whether later edits adopt them, or how this relates to whether the outcome finally passes. The same L3 design is negative on some models, for example on SWE the result for claim-guided changes is −1.1 for Qwen3.7-Max and −3.4 for DeepSeek V4-Flash, and the result for task-first trajectory is −3.4 for GLM-5.2. Coloring is a secondary issue. Figure 2 uses four highly saturated colors to distinguish the four layers and overlays seventeen numbers, and the legend of Figure 3 is blue, orange, green, red, purple, brown, pink, and gray in that order. When reading relative positions, color draws the eye more than the coordinates, and this point does not by itself determine the score.
Questions for authors
- Were the TB task list and the SWE task list used in the report frozen before any L0 to L3 run? Did the pilot pass rates of the four models, and the task adjustment after seeing the SWE baselines, take part in deciding which tasks were kept?
- Is there a control that gives the original agent the same extra tokens and the same one rework without introducing a second model? Was the reviewer changed from Qwen3.7-Plus to another model?
- Please provide a manual check of reviewer reliability. It needs to state whether the review comments point out real defects, whether later edits adopt them, and how this relates to whether the final tests pass. The sample size and the checking method need to be enough to support the average L3 gain. If the paper did not include this check before, please add it now and write the corresponding content into the paper.