- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SHIFT, which builds an agent harness for each query by running MCTS with a small learned architect that predicts which harness will work and how much it costs, so no candidate needs to be executed during search. On six benchmarks with a Gemini 3.5 Flash executor, the paper reports about 80% mean accuracy, 7.2 points above the strongest baseline.
Critical strengths
- The problem is practical and interesting. The best harness really depends on the query, and the paper shows this clearly in Figure2.
- Moving execution out of the search loop is a nice idea, and building a harness only takes under one second.
- The gains on long, tool-heavy tasks are large, e.g., 59.3% vs. 29.3% for Trace on OfficeQA.
Critical weaknesses
-
The mean accuracy depends on how the benchmarks are weighted. Each benchmark gets the same weight, although their test sets differ greatly in size. Weighting by question gives approximately 74.7% for Trace and 73.9% for SHIFT-search. I would like to see both averages reported.
-
Most of the 7.2-point improvement comes from OfficeQA and GAIA, which have only 41 and 51 test questions. The paper reports significance tests, but it is unclear how stable these gains would be across different data splits.
-
Figure 1 could be clearer. Some icons are blurry, and the fonts are inconsistent.
-
SHIFT-search is also expensive: its normalized token cost is 84.3, compared with 40.8 for Trace. Training costs are reported separately, but it is unclear how they compare with the baselines’ optimization costs.
-
The action library includes domain-specific instructions, such as checking Excel formulas and distinguishing millions from billions. It is unclear whether the baselines receive the same hints. This makes it difficult to separate the benefit of SHIFT from that of the manually designed instructions.
-
MaAS is discussed but not compared experimentally. Since it also constructs query-dependent agent systems, it would be a relevant baseline.
-
The model name is inconsistent. The Figure 3 caption says Gemma-2-2B, but Section 3.2 says Gemma 4 E2B.
Questions for authors
- What does the mean accuracy look like when weighted by question instead of by benchmark?
- What are the optimization budgets of Trace, GEPA and AFlow compared with SHIFT's training?
- Why do some directives in the action library target specific benchmarks, and what happens without them?
- Why is MaAS not included as a baseline?
- Which model is the architect, Gemma-2-2B or Gemma 4 E2B?
8 comments
3. Inference efficiency and offline optimization costs (W4,Q2)
SHIFT provides two deployment choices: SHIFT-value improves observed macro accuracy over Trace at a lower normalized executor-token cost, while SHIFT-search achieves higher accuracy at greater execution cost. We will distinguish these deployment costs from investment in training the architect.
| Method | Six-benchmark macro accuracy (↑) | Normalized executor-token cost (↓) |
|---|---|---|
| Trace | 72.7% | 40.8 |
| SHIFT-value | 74.5% | 27.7 |
| SHIFT-search | 79.9% | 84.3 |
SHIFT-value achieves a 32% lower normalized executor-token cost index than Trace, alongside accuracy improvements of 9.7 percentage points on OfficeQA and 11.1 points on GAIA. Both SHIFT variants use exactly the same trained architect, so switching between these operating points requires no additional training.
We agree that deployment efficiency should be considered alongside the offline investment required to obtain it. As Reviewer 54zd also highlighted, Table 2 compares decoding strategies using the same trained architect, allowing us to isolate what inference-time search contributes. MCTS reduces the executor-token cost index from 120.2 to 84.3 a 29.9% reduction at nearly unchanged macro accuracy, 79.85% versus 79.87%. The paired analysis places this reduction between 26.0% and 33.5% at 95% confidence.
Offline optimization and reuse. SHIFT learns a query-conditioned construction policy and utility predictor over four training epochs. After accounting for cache reuse, this averages 3.2–4.6 fresh candidate-workflow evaluations per training question per epoch, excluding validation. The resulting architect is reused across subsequent queries within the benchmark and executor setting, and the same checkpoint supports both SHIFT-value and SHIFT-search without further training.
For comparison, Trace’s retained optimization records contain 20 training and 40 validation workflow evaluations per benchmark. AFlow uses 24–84 workflow evaluations, including candidate validation. GEPA’s recorded main-program compilation usage ranges from 1.26-44.11M tokens across the six benchmarks, excluding its separate reflection model. The original experiments used method-specific optimization budgets rather than a matched offline-compute budget. Table 3 also provides complementary controlled evidence by holding candidate pools fixed and comparing selection rules. We will expand Appendix A.3 with the benchmark-level accounting for all four methods, separating training, validation, cache reuse, and the scope of recorded token usage.
4. Are the improvements attributable to domain-specific directives or unequal baseline access? (W5,Q3)
Our comparisons include tool-equipped baselines with explicit checking instructions, as well as optimizers that can develop instructions beyond SHIFT’s fixed catalogue. Tool-enabled baseline implementations use the same executor and domain tool pool K_d. Our DSPy programs, including those optimized by MIPROv2 and GEPA, support tool use during optimization and evaluation. MIPROv2, GEPA, Trace, and AFlow optimize free-form instructions, allowing them to express guidance from SHIFT’s catalogue and develop additional instructions.
All 15 directives in Table 6 are available across all six benchmarks, without per-benchmark filtering. The architect learns which directives to select for each query. Three benchmark-level comparisons are particularly relevant:
- OfficeQA: unit-checking guidance is already present in a strong baseline. ReAct+Refine explicitly instructs its Critic to “check units, magnitudes, constraints” and equips its coding agents with file-reading, file-writing, calculator, and Python tools. It also revises the answer following critique. Using the same Gemini 3.5 Flash executor on the same 41 questions with three execution repeats, SHIFT-search achieves 59.35%, compared with 46.34% for ReAct+Refine a 13.01 percentage-point improvement. The OfficeQA advantage therefore persists against a tool-equipped baseline that already receives unit-checking guidance.
- SpreadsheetBench: the tool-enabled baseline exceeds SHIFT. GEPA achieves 95.3%, compared with SHIFT-search’s 87.8%, even though SHIFT’s catalogue includes an Excel directive. Our implementation gives GEPA the file and code-execution capabilities needed to produce and evaluate modified workbooks.
- GAIA: improvements extend to tasks without benchmark-specific directives. The catalogue contains no GAIA-specific directives, yet SHIFT-search improves over Trace by 15.0 percentage points.
These results provide concrete evidence that SHIFT’s gains extend beyond comparisons against baselines lacking tools or basic checking instructions. We will expand Appendix C to document each baseline’s tool access and instruction interface, highlight the existing OfficeQA unit-checking comparison, and clarify that the complete directive catalogue is shared across benchmarks.
5. Comparison with MaAS (W6,Q4)
Thank you for highlighting MaAS. Incorporating it into our evaluation required adapting its operator interface to the shared benchmark tools, file-based outputs, and grading framework. MaAS constructs query-dependent systems from composite operators that can themselves use tools, whereas SHIFT additionally treats individual tool permissions as explicit construction decisions. We now report the adapted comparison below and will document the integration choices alongside the results. We now report the following comparison using the same Gemini 3.5 Flash executor and test cohorts:
| Benchmark | MaAS adaptation | SHIFT-search |
|---|---|---|
| GSM8K | 98.46% | 98.21% |
| MBPP | 97.02% | 96.63% |
| HotpotQA | 68.09% | 68.65% |
| MaAS | 65.36% | 68.63% |
| OfficeQA | 9.76%+ | 59.35% |
| SpreadsheetBench | 0.00%* | 87.78% |
*SpreadsheetBench requires editing a workbook and saving the correct result. MaAS constructed workflows with agents for reviewing and testing, but omitted the coding agent equipped to carry out those edits. None of the resulting workbooks passed all required checks.
+For OfficeQA the adapted MaAS controller selected the same workflow for all 41 questions: two agents generating candidate answers, followed by answer aggregation. These workflows used only a calculator and omitted file-reading and code-execution tools. Consequently, the selected harnesses lacked a document-retrieval stage for inspecting the Treasury reports which is an important limitation for OfficeQA’s document-grounded questions.
6. Architect model identification and Figure 1 readability (W3,W7,Q5)
Thank you for identifying these presentation issues. The architect used in the reported experiments is Gemma 4 E2B, as specified in Section 3.2 and Table 8. The Gemma-2-2B reference in the Figure 3 caption is an outdated label from an earlier experimental version. We will correct the caption to consistently identify the architect.
We have also prepared a revised vector version of Figure 1, with consistent typography and larger, clearer role and tool labels. We will include it in the revised paper to improve legibility at normal viewing size.
We thank the reviewer again for their careful assessment and constructive feedback. We hope the additional analyses, baseline comparisons, and clarifications address the concerns raised and make SHIFT’s benefits, costs, and scope clearer. We will incorporate these findings and revisions into the paper.
We thank the reviewer for recognizing the practical importance of query-specific harness construction, the benefit of moving execution outside the search loop, and the gains on tool-intensive tasks. We appreciate their careful questions about aggregation, experimental controls, and cost accounting. Below, we address every concern in turn, presenting additional statistical analyses and baseline comparisons, clarifying aspects of the evaluation that were insufficiently explained in the submission, and detailing the corresponding revisions to the paper. These additions make the evidence, scope, and practical tradeoffs of SHIFT clearer.
1. Benchmark-weighted and question-weighted accuracy (W1,Q1)
We thank the reviewer for suggesting this additional analysis. We now report both benchmark-weighted (macro) and question-weighted (micro) accuracy. SHIFT-search achieves the highest six-benchmark macro-average, and its substantial improvements on longer-horizon tasks hold under both weighting schemes. Recomputing from the unrounded results gives:
Shorter-horizon benchmarks: GSM8K, HotpotQA, and MBPP. Longer-horizon benchmarks: SpreadsheetBench, OfficeQA, and GAIA.
Our choice of macro-averaging reflects the evaluation’s coverage of six distinct task types: each benchmark contributes equally, regardless of its size. The pooled micro-average instead weights task types according to their numbers of test questions. Here, HotpotQA alone contributes 80.6% of all questions, while the three longer-horizon benchmarks together contribute only 2.3%. Consequently, the overall micro-average primarily describes performance on the shorter-horizon portion of the evaluation. The horizon-specific breakdowns provide a more informative basis for assessing SHIFT’s applicability across different settings.
In particular, SHIFT-search’s longer-horizon advantage persists when every question receives equal weight: +8.96 percentage points under micro-averaging, alongside +14.76 points under macro-averaging. The improvement in this setting therefore does not depend on giving each benchmark equal weight. These gains directly support the motivation for query-specific harness construction: improving performance on complex tasks requiring extensive tool use and coordinated execution. On shorter-horizon tasks, SHIFT remains competitive, within 1.03 points of Trace.
We will make these distinctions explicit in the revision and report both overall averages, and emphasize the horizon-specific results when discussing applicability. This makes the central empirical contribution clear: substantial gains on complex, tool-intensive tasks, supported by both aggregation schemes, alongside competitive shorter-horizon performance.
2. Reliability of gains on the smaller benchmarks (W2)
We examined whether the improvements on OfficeQA and GAIA depend on a few favorable test questions. We average the three execution repeats within each question and compute paired bootstrap intervals by resampling questions, keeping SHIFT-search and Trace paired:
Both intervals remain above zero. The gains also survive removing any individual question: the difference ranges from +28.33 to +32.50 points on OfficeQA and +13.33 to +16.00 points on GAIA.
The improvements extend across multiple questions. SHIFT-search outperforms Trace on 24 OfficeQA questions, loses on four, and ties on 13. On GAIA, the corresponding counts are 19 wins, seven losses, and 25 ties, using accuracy averaged across execution repeats. The positive bootstrap intervals and leave-one-out results show that the gains extend across the evaluated questions and do not depend on a single favorable example. These analyses keep the trained models fixed. Testing whether the gains also hold after changing the training and validation splits would require retraining both methods.
We will add the paired intervals and question-level analyses to the revised paper. They support the conclusion that the observed gains are spread across the evaluated questions and remain positive after removing any single example. We will distinguish this evidence from robustness across new training partitions, which requires a separate retraining study.