- Rating
- 3(+2)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2(+1)
Read review
Paper summary
The paper focuses on identifying the redundancies in harness modules and presents a framework for pruning them to minimize tokens while preserving performance. The components to be pruned are identified by ablating them individually to observe the impact on token cost and performance. Through their experiments over many harnesses, they find that many existing harnesses have redundant components.
Critical strengths
S1: The problem is very relevant as development efforts quickly lead to bloated features in the agentic harnesses
S2: The broad evaluation and qualitative analysis act as very useful datapoints for future works on harness optimization
Critical weaknesses
W1: My fundamental concern is that the work conflates harness redundancy with performance on a specific evaluation dataset. Harness components may serve particular functional purposes like clarification, error recovery etc. that may be marked as redundant if the evaluation dataset does not include inputs that trigger these. Pruning solely on the benchmark performance may compromise capability and usage in niche settings the harness may be designed for. Often, the theoretical reasoning for a component can be more reliable than an imperfect and incomplete benchmark at evaluating its utility.
W2: Single-module ablations do not capture interactions between components. E.g. Component B is only effective once component A triggers, but if B is marked as being impactful on performance before A is pruned, and A is later identified as a redundant component, then pruning it makes B redundant but it will not be pruned OR two modules with overlapping functionality do not show performance difference with single ablations, but removing both impacts performance. I do not see how these type of complex interactions can be captured with the current setup
W3: The performance efficiency tradeoff is difficult to control and developers can only set the number of modules to be removed, not the desired token reduction or acceptable performance degradation.
W4: Previous works (Beyond the Model: Demystifying Harness Effects in Software Engineering Agents by Hu et al. 2026) show that harness effectiveness depends on model backbone. Since all experiments are run on only one model, this creates a question how effective the approach is across different models?
Questions for authors
Q1: I would appreciate a discussion on how W1 is tackled by the current approach.
Q2: How does harness pruning change for different model backbones from small to large models?
5 comments
Response to Reviewer fZ7B -2
W2. Single-module ablations miss dependencies and overlapping functionality that can change module importance during pruning. How does SHarP address these interactions?
We agree that dependencies and overlapping functionality can change a module’s importance after other modules are removed. SHarP uses single-module ablations to estimate importance at a limited evaluation cost. Although these estimates do not explicitly model interactions, they may still provide useful guidance for pruning. We conducted an additional random-drop experiment to examine whether modules identified as important by SHarP remain relatively important when multiple modules are removed together.
We sampled 60 configurations on LIFE-harness retail, independently retaining each module with probability 0.7, and evaluated each on the development set. For each module, we compared success rates and mean token costs between its enabled and disabled groups using one-sided Welch tests. Because other modules vary within both groups, these comparisons measure contributions across different module combinations, including interactions present in those combinations. We ranked modules separately for each metric in ascending order of p-value, with smaller p-values indicating stronger evidence that retention improves success rate or reduces token cost.
For token efficiency, the seven modules originally identified as important have a mean random-drop rank of 16.9, versus 25.8 for the remaining modules. Five of seven remain in the top half, suggesting that SHarP's efficiency estimates retain useful information for pruning. The table shows all seven modules, with rank 1 indicating the strongest evidence for retention.
| Module | Original efficiency rank | Random-drop efficiency rank |
|---|---|---|
| Exchange item existence check | 1 | 28 |
| Order action guidance | 2 | 2 |
| Multi-item exchange skill | 3 | 7 |
| Category-scope skill | 4 | 12 |
| Item multiplicity check | 5 | 23 |
| Variant price extrema | 6 | 42 |
| Confirmed-action skill | 7 | 4 |
For success rate, only 1 of the 48 modules shows a significant benefit from retention under random drop (p ≤ 0.05). The 13 originally protected modules have a mean rank of 26.9, near the middle of the ranking. However, the largely inconclusive random-drop results do not rule out the performance risks identified by the original screening. In efficiency-oriented pruning, SHarP takes a conservative approach by retaining these modules to reduce those risks, even if this gives up some potential token saving.
Although random drop allows module importance to be assessed across different combinations of other modules, it requires repeated sampling to populate each module’s enabled and disabled groups. Obtaining sufficient observations in both groups for every module usually requires more configurations than the number of modules. SHarP instead requires only one single-module ablation per module, giving it a fixed screening budget determined by the number of modules. Overall, this simple screening can still provide useful pruning guidance in an efficient way.
W3. How can users choose a pruning configuration for a desired token reduction and acceptable performance loss, rather than only specifying a module count?
We thank the reviewer for suggesting more direct control over token savings and performance loss. SHarP determines a pruning path. Although our original procedure does not explicitly take a desired token reduction or acceptable performance loss as inputs, these preferences can guide where to stop along that path. We illustrate this on LIFE-Harness retail using the original protected set and saliency order. We follow the pruning order on the development set, removing three modules at a time. When the token target is reached, we test removing only the first one or two modules from the last group to see whether less pruning would suffice. We choose the stopping point and configuration using only the development results, then evaluate the selected configuration on the held-out validation set.
With a 10% mean-token reduction target and a 15 percentage-point performance-loss tolerance, this selects the 37-module configuration. In single-run validation on held-out tasks, it reduces mean token cost by 13.7%, while success rate increases from 67.5% to 80.0%. A 20% target yields no qualifying configuration along the path evaluated on the development set. This illustrates how user preferences can guide the choice of pruning degree within SHarP’s existing pipeline.
Response to Reviewer fZ7B -3
W4/Q2. How effective is SHarP across model backbones, and how does pruning change from smaller to larger models?
We thank the reviewer for suggesting experiments across model sizes. We have added the appendix subsection “A.7 Harness pruning across model sizes,” marked with a blue title, containing detailed experimental results and discussion. The experiments run the full SHarP pipeline separately with Qwen3.5-35B-A3B and Qwen3.5-397B-A17B on LIFE-harness retail and JIT agentfold shopping, with one validation run per configuration. We summarize the main observations below.
Pruning provides larger benefits on 35B in these experiments, with configurations that use fewer tokens and achieve higher observed performance on both benchmarks. On 397B, shopping still offers substantial token savings at some performance cost, while retail offers limited savings. One possible explanation is that stronger models can use existing harness components more effectively, whereas weaker models benefit more from simplifying a harness whose components they use less effectively. The contrast between retail and shopping on 397B also shows that pruning opportunities depend on the task and harness.
Response to Reviewer fZ7B -1
We thank reviewer fZ7B for your valuable comments.
W1/Q1. Can benchmark-based pruning mistake rarely triggered functions, such as clarification or error recovery, for redundancy? How does SHarP account for their functional importance?
SHarP targets pruning for a specific task distribution or application scenario. A component’s functional purpose explains when it may help, but its practical value also depends on how often it is needed, what other mechanisms remain available, and its recurring cost. A rarely needed component can therefore present a meaningful performance–cost trade-off. Our evaluation examines these trade-offs within the target scenario; its conclusions do not imply that a pruned component is unnecessary in every setting.
We also tested whether functional judgments would identify better pruning configurations on retail. We asked an LLM to review all modules and identify those requiring protection based on task requirements and module descriptions, without access to ablation results. It identified two such modules. One of the two modules also narrowly misses SHarP’s original performance-protection threshold (p = 0.0515), showing that its potential importance is already reflected in the saliency results. We retained these alongside SHarP’s original 13 protected modules, then evaluated the resulting pruning path on the validation set. The table shows the non-dominated configurations from each path, using median token cost and success rate.
Every configuration on the additional-protection path is matched or exceeded in success rate by an original SHarP configuration that uses fewer median tokens. For example, original SHarP reaches 82.5% success with 19 modules and 84.5k tokens, compared with 33 modules and 88.6k tokens after adding protection. Thus, the additional functional judgments do not yield a better observed trade-off.