acceptodds
Under review as a conference paper at ICLR 2027

Dynamic Harness Search: Building Multi-Agent Systems Per-Query via Prediction

Abstract

Agent harnesses specify the roles, instructions, tools, and communication structure used to solve a task, and the right harness depends on the query. Because the value of each design choice is observable only through execution, tailoring a harness to each query has required either executing alternatives at inference time or costly manual design. We introduce SHIFT, which moves execution out of the per-query search loop. A local LLM architect learns a policy over harness-building actions from search, and a value function that predicts, from measured executions, a utility balancing accuracy against execution cost. For each query, Monte Carlo tree search uses these predictions to construct a harness. Across 9,193 tasks in six benchmarks, from math to document and general-assistant tasks, with a Gemini 3.5 Flash executor, SHIFT attains the highest mean accuracy, about 80%, outperforming 17 baselines that span prompting, prompt optimization, and workflow search, and exceeding the strongest baseline by 7.2 percentage points. A cheaper mode of SHIFT also attains a higher mean accuracy than every baseline while using 32% fewer execution tokens than the strongest baseline. We further show that choosing structure, instructions, and tools jointly beats choosing only instructions or only tools by up to 9.1 percentage points, and that learned value selection identifies more accurate harnesses with lower execution cost from candidate pools.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Multi-agent systems, Learning-guided search

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_AHosOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SHIFT, which builds an agent harness for each query by running MCTS with a small learned architect that predicts which harness will work and how much it costs, so no candidate needs to be executed during search. On six benchmarks with a Gemini 3.5 Flash executor, the paper reports about 80% mean accuracy, 7.2 points above the strongest baseline.

Critical strengths
  1. The problem is practical and interesting. The best harness really depends on the query, and the paper shows this clearly in Figure2.
  2. Moving execution out of the search loop is a nice idea, and building a harness only takes under one second.
  3. The gains on long, tool-heavy tasks are large, e.g., 59.3% vs. 29.3% for Trace on OfficeQA.
Critical weaknesses
  1. The mean accuracy depends on how the benchmarks are weighted. Each benchmark gets the same weight, although their test sets differ greatly in size. Weighting by question gives approximately 74.7% for Trace and 73.9% for SHIFT-search. I would like to see both averages reported.

  2. Most of the 7.2-point improvement comes from OfficeQA and GAIA, which have only 41 and 51 test questions. The paper reports significance tests, but it is unclear how stable these gains would be across different data splits.

  3. Figure 1 could be clearer. Some icons are blurry, and the fonts are inconsistent.

  4. SHIFT-search is also expensive: its normalized token cost is 84.3, compared with 40.8 for Trace. Training costs are reported separately, but it is unclear how they compare with the baselines’ optimization costs.

  5. The action library includes domain-specific instructions, such as checking Excel formulas and distinguishing millions from billions. It is unclear whether the baselines receive the same hints. This makes it difficult to separate the benefit of SHIFT from that of the manually designed instructions.

  6. MaAS is discussed but not compared experimentally. Since it also constructs query-dependent agent systems, it would be a relevant baseline.

  7. The model name is inconsistent. The Figure 3 caption says Gemma-2-2B, but Section 3.2 says Gemma 4 E2B.

Questions for authors
  1. What does the mean accuracy look like when weighted by question instead of by benchmark?
  2. What are the optimization budgets of Trace, GEPA and AFlow compared with SHIFT's training?
  3. Why do some directives in the action library target specific benchmarks, and what happens without them?
  4. Why is MaAS not included as a baseline?
  5. Which model is the architect, Gemma-2-2B or Gemma 4 E2B?
8 comments

We thank the reviewer for recognizing the practical importance of query-specific harness construction, the benefit of moving execution outside the search loop, and the gains on tool-intensive tasks. We appreciate their careful questions about aggregation, experimental controls, and cost accounting. Below, we address every concern in turn, presenting additional statistical analyses and baseline comparisons, clarifying aspects of the evaluation that were insufficiently explained in the submission, and detailing the corresponding revisions to the paper. These additions make the evidence, scope, and practical tradeoffs of SHIFT clearer.

1. Benchmark-weighted and question-weighted accuracy (W1,Q1)

We thank the reviewer for suggesting this additional analysis. We now report both benchmark-weighted (macro) and question-weighted (micro) accuracy. SHIFT-search achieves the highest six-benchmark macro-average, and its substantial improvements on longer-horizon tasks hold under both weighting schemes. Recomputing from the unrounded results gives:

Evaluation summaryTraceSHIFT-searchDifference
Six-benchmark macro-average72.73%79.87%+7.15 pp
Overall micro-average, N = 9,19374.68%73.88%−0.80 pp
Shorter-horizon micro-average, N = 8,98174.82%73.79%−1.03 pp
Longer-horizon micro-average, N = 21268.71%77.67%+8.96 pp
Longer-horizon macro-average57.16%71.92%+14.76 pp

Shorter-horizon benchmarks: GSM8K, HotpotQA, and MBPP. Longer-horizon benchmarks: SpreadsheetBench, OfficeQA, and GAIA.

Our choice of macro-averaging reflects the evaluation’s coverage of six distinct task types: each benchmark contributes equally, regardless of its size. The pooled micro-average instead weights task types according to their numbers of test questions. Here, HotpotQA alone contributes 80.6% of all questions, while the three longer-horizon benchmarks together contribute only 2.3%. Consequently, the overall micro-average primarily describes performance on the shorter-horizon portion of the evaluation. The horizon-specific breakdowns provide a more informative basis for assessing SHIFT’s applicability across different settings.

In particular, SHIFT-search’s longer-horizon advantage persists when every question receives equal weight: +8.96 percentage points under micro-averaging, alongside +14.76 points under macro-averaging. The improvement in this setting therefore does not depend on giving each benchmark equal weight. These gains directly support the motivation for query-specific harness construction: improving performance on complex tasks requiring extensive tool use and coordinated execution. On shorter-horizon tasks, SHIFT remains competitive, within 1.03 points of Trace.

We will make these distinctions explicit in the revision and report both overall averages, and emphasize the horizon-specific results when discussing applicability. This makes the central empirical contribution clear: substantial gains on complex, tool-intensive tasks, supported by both aggregation schemes, alongside competitive shorter-horizon performance.

2. Reliability of gains on the smaller benchmarks (W2)

We examined whether the improvements on OfficeQA and GAIA depend on a few favorable test questions. We average the three execution repeats within each question and compute paired bootstrap intervals by resampling questions, keeping SHIFT-search and Trace paired:

BenchmarkTest questionsSHIFT-search − Trace95% paired bootstrap interval
OfficeQA41+30.08 pp[+17.07, +43.09]
GAIA51+15.03 pp[+5.88, +24.84]

Both intervals remain above zero. The gains also survive removing any individual question: the difference ranges from +28.33 to +32.50 points on OfficeQA and +13.33 to +16.00 points on GAIA.

The improvements extend across multiple questions. SHIFT-search outperforms Trace on 24 OfficeQA questions, loses on four, and ties on 13. On GAIA, the corresponding counts are 19 wins, seven losses, and 25 ties, using accuracy averaged across execution repeats. The positive bootstrap intervals and leave-one-out results show that the gains extend across the evaluated questions and do not depend on a single favorable example. These analyses keep the trained models fixed. Testing whether the gains also hold after changing the training and validation splits would require retraining both methods.

We will add the paired intervals and question-level analyses to the revised paper. They support the conclusion that the observed gains are spread across the evaluated questions and remain positive after removing any single example. We will distinguish this evidence from robustness across new training partitions, which requires a separate retraining study.

3. Inference efficiency and offline optimization costs (W4,Q2)

SHIFT provides two deployment choices: SHIFT-value improves observed macro accuracy over Trace at a lower normalized executor-token cost, while SHIFT-search achieves higher accuracy at greater execution cost. We will distinguish these deployment costs from investment in training the architect.

MethodSix-benchmark macro accuracy (↑)Normalized executor-token cost (↓)
Trace72.7%40.8
SHIFT-value74.5%27.7
SHIFT-search79.9%84.3

SHIFT-value achieves a 32% lower normalized executor-token cost index than Trace, alongside accuracy improvements of 9.7 percentage points on OfficeQA and 11.1 points on GAIA. Both SHIFT variants use exactly the same trained architect, so switching between these operating points requires no additional training.

We agree that deployment efficiency should be considered alongside the offline investment required to obtain it. As Reviewer 54zd also highlighted, Table 2 compares decoding strategies using the same trained architect, allowing us to isolate what inference-time search contributes. MCTS reduces the executor-token cost index from 120.2 to 84.3 a 29.9% reduction at nearly unchanged macro accuracy, 79.85% versus 79.87%. The paired analysis places this reduction between 26.0% and 33.5% at 95% confidence.

Offline optimization and reuse. SHIFT learns a query-conditioned construction policy and utility predictor over four training epochs. After accounting for cache reuse, this averages 3.2–4.6 fresh candidate-workflow evaluations per training question per epoch, excluding validation. The resulting architect is reused across subsequent queries within the benchmark and executor setting, and the same checkpoint supports both SHIFT-value and SHIFT-search without further training.

For comparison, Trace’s retained optimization records contain 20 training and 40 validation workflow evaluations per benchmark. AFlow uses 24–84 workflow evaluations, including candidate validation. GEPA’s recorded main-program compilation usage ranges from 1.26-44.11M tokens across the six benchmarks, excluding its separate reflection model. The original experiments used method-specific optimization budgets rather than a matched offline-compute budget. Table 3 also provides complementary controlled evidence by holding candidate pools fixed and comparing selection rules. We will expand Appendix A.3 with the benchmark-level accounting for all four methods, separating training, validation, cache reuse, and the scope of recorded token usage.

4. Are the improvements attributable to domain-specific directives or unequal baseline access? (W5,Q3)

Our comparisons include tool-equipped baselines with explicit checking instructions, as well as optimizers that can develop instructions beyond SHIFT’s fixed catalogue. Tool-enabled baseline implementations use the same executor and domain tool pool K_d. Our DSPy programs, including those optimized by MIPROv2 and GEPA, support tool use during optimization and evaluation. MIPROv2, GEPA, Trace, and AFlow optimize free-form instructions, allowing them to express guidance from SHIFT’s catalogue and develop additional instructions.

All 15 directives in Table 6 are available across all six benchmarks, without per-benchmark filtering. The architect learns which directives to select for each query. Three benchmark-level comparisons are particularly relevant:

  1. OfficeQA: unit-checking guidance is already present in a strong baseline. ReAct+Refine explicitly instructs its Critic to “check units, magnitudes, constraints” and equips its coding agents with file-reading, file-writing, calculator, and Python tools. It also revises the answer following critique. Using the same Gemini 3.5 Flash executor on the same 41 questions with three execution repeats, SHIFT-search achieves 59.35%, compared with 46.34% for ReAct+Refine a 13.01 percentage-point improvement. The OfficeQA advantage therefore persists against a tool-equipped baseline that already receives unit-checking guidance.
  2. SpreadsheetBench: the tool-enabled baseline exceeds SHIFT. GEPA achieves 95.3%, compared with SHIFT-search’s 87.8%, even though SHIFT’s catalogue includes an Excel directive. Our implementation gives GEPA the file and code-execution capabilities needed to produce and evaluate modified workbooks.
  3. GAIA: improvements extend to tasks without benchmark-specific directives. The catalogue contains no GAIA-specific directives, yet SHIFT-search improves over Trace by 15.0 percentage points.

These results provide concrete evidence that SHIFT’s gains extend beyond comparisons against baselines lacking tools or basic checking instructions. We will expand Appendix C to document each baseline’s tool access and instruction interface, highlight the existing OfficeQA unit-checking comparison, and clarify that the complete directive catalogue is shared across benchmarks.

5. Comparison with MaAS (W6,Q4)

Thank you for highlighting MaAS. Incorporating it into our evaluation required adapting its operator interface to the shared benchmark tools, file-based outputs, and grading framework. MaAS constructs query-dependent systems from composite operators that can themselves use tools, whereas SHIFT additionally treats individual tool permissions as explicit construction decisions. We now report the adapted comparison below and will document the integration choices alongside the results. We now report the following comparison using the same Gemini 3.5 Flash executor and test cohorts:

BenchmarkMaAS adaptationSHIFT-search
GSM8K98.46%98.21%
MBPP97.02%96.63%
HotpotQA68.09%68.65%
MaAS65.36%68.63%
OfficeQA9.76%+59.35%
SpreadsheetBench0.00%*87.78%

*SpreadsheetBench requires editing a workbook and saving the correct result. MaAS constructed workflows with agents for reviewing and testing, but omitted the coding agent equipped to carry out those edits. None of the resulting workbooks passed all required checks.

+For OfficeQA the adapted MaAS controller selected the same workflow for all 41 questions: two agents generating candidate answers, followed by answer aggregation. These workflows used only a calculator and omitted file-reading and code-execution tools. Consequently, the selected harnesses lacked a document-retrieval stage for inspecting the Treasury reports which is an important limitation for OfficeQA’s document-grounded questions.

6. Architect model identification and Figure 1 readability (W3,W7,Q5)

Thank you for identifying these presentation issues. The architect used in the reported experiments is Gemma 4 E2B, as specified in Section 3.2 and Table 8. The Gemma-2-2B reference in the Figure 3 caption is an outdated label from an earlier experimental version. We will correct the caption to consistently identify the architect.

We have also prepared a revised vector version of Figure 1, with consistent typography and larger, clearer role and tool labels. We will include it in the revised paper to improve legibility at normal viewing size.

We thank the reviewer again for their careful assessment and constructive feedback. We hope the additional analyses, baseline comparisons, and clarifications address the concerns raised and make SHIFT’s benefits, costs, and scope clearer. We will incorporate these findings and revisions into the paper.

Reviewer_54zdOpenReview ↗
Rating
4
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SHIFT, which learns the cost-sensitive utility of query–harness pairs to guide tree search over agents, instructions, and tools. Candidate harnesses are not executed during inference-time search; only the selected harness is run, shifting execution-based evaluation to the training stage.

Critical strengths
  1. A practically relevant problem with a well-aligned methodological design. Rather than merely selecting from existing workflows, SHIFT incrementally constructs a harness for each query. Its integration of policy prediction, utility estimation, and lookahead search forms a coherent framework for accounting for dependencies among roles, instructions, and tools.

  2. Convincing empirical results.

  3. Targeted analyses beyond overall performance comparisons. Table 2 compares decoding strategies using the same trained architect, supporting the benefit of search in reducing execution cost while maintaining accuracy. Table 3 compares selection rules over identical candidate pools, providing further evidence that the learned value predictor helps identify better configurations. These controlled comparisons are a particular strength.

Critical weaknesses

The relationship to capability-specification approaches deserves substantive discussion. The related work should discuss [1,2]. These approaches first construct capability specifications for models or agents and then use specification matching to select suitable existing candidates; the latter also supports adding new agents without retraining a global selector. In contrast, SHIFT learns the utility of query–harness pairs and uses it to search for new harnesses. Please clarify the differences in what is characterized, how component interactions are modeled, and what adaptation is required when the available models or tools change. This is primarily a request for clearer positioning, rather than additional experimental comparisons under mismatched settings.

[1] Constructive Specification for Plug-and-Play Learnware Agents. [2] Learnware of Language Models: Specialized Small Language Models Can Do Big

Questions for authors

Compared to using a single frontier LLM, what scenarios will be the killer application of building multi-agent systems per-query? It is only about saving cost, or it may improve the limit of intelligent capability?

3 comments

We thank the reviewer for their support of our work and thoughtful feedback. We especially appreciate their recognition of SHIFT’s practical relevance, coherent methodological design, and controlled analyses. Their suggestions help us sharpen the paper’s positioning and clarify where query-specific harness construction is most valuable.

1. Relationship to capability-specification approaches. (W1) We agree that [1], [2] are closely relevant. Both use capability specifications to identify suitable existing models or agents, whereas SHIFT learns the utility of a proposed query harness pair and uses that estimate to guide construction. We will make the following distinctions explicit:

DimensionCapability-specification approaches [1,2]SHIFT
What is characterizedCapabilities of existing models or agents, matched against user requirements.Cost-aware utility Vθ(q,s)V_\theta(q,s) of a query paired with a partial or complete harness graph.
Component interactionsMatches each existing model or agent as a whole; the selection step does not construct new combinations of roles, instructions, and tools.Scores combinations of roles, instructions, tools, and connections, allowing a tool’s predicted benefit to depend on which agent receives it and how that agent participates in the workflow.
Changes to available componentsNew candidates can be added through their specifications; [1] explicitly supports onboarding without retraining a global selector.Supported tools can be enabled or disabled through feasible actions. Unfamiliar tools require extending the action representation, while changed executor or tool behavior may require fresh execution feedback and architect fine-tuning.

More specifically, Constructive Specification [1] builds capability representations from limited profiling of black-box agents, while Learnware of Language Models [2] constructs parameter-based specifications for specialized language models. These specifications support reuse of existing capabilities. SHIFT addresses the complementary problem of deciding how to organize capabilities for a particular query. We will also acknowledge the adaptation advantage of specification matching. SHIFT currently learns within a predefined action vocabulary and execution environment; it does not provide the same plug-and-play onboarding guarantee.

We thank the reviewer for highlighting this connection. We will add a dedicated subsection on capability-specification approaches to Related Work, discussing both papers along these three dimensions. We will also discuss how conditioning SHIFT’s architect on capability specifications could combine plug-and-play component discovery with query-specific harness construction.

2. When does query-specific construction help beyond saving cost?

We view query-specific construction as a way to improve what a given model can accomplish through its execution workflow. Its strongest applications involve several dependent operations gathering evidence, reconciling sources, performing calculations, and checking the answer whose requirements vary across queries. For example, answering a financial question across multiple documents may require separate retrieval and computation stages, whereas a straightforward lookup may need only one agent. SHIFT learns to construct a harness for these different requirements.

Our results show benefits beyond cost savings with the executor held fixed. The following comparison uses Gemini 3.5 Flash throughout and covers all 51 GAIA questions, with three execution repeats per question:

MethodAccuracyQuestions solved in all three repeatsMean executor tokens per query
Single-agent ReAct33.3%11/51182k
SHIFT-value64.7%24/511,014k
SHIFT-search68.6%26/511,108k

SHIFT-search improves observed accuracy by 35.3 percentage points over the evaluated single-agent ReAct implementation. Comparing successful repeats question by question, SHIFT-search improves on 31 questions and ties on the remaining 20. Here, the benefit is improved task completion at greater execution cost. The following provide concrete illustrations:

  1. Cross-source counting with a historical cutoff. A question asks for the difference between the numbers of unique Survivor and American Idol winners at a specified date. SHIFT distinguishes 44 Survivor seasons from 42 unique winners, accounts for repeat winners, and compares this with 21 American Idol winners. Its scripts deduplicate the winner list and calculate the correct difference, 21. ReAct ends while still attempting to retrieve or parse the relevant lists.
  2. Landmark identification followed by numerical conversion. A question asks for the height, in yards, of a landmark appearing on a magazine cover. SHIFT identifies the Washington Monument, and its calculation script checks that alternative reported height measurements round to 185 yards. ReAct remains in the retrieval stage and does not produce the requested numerical answer.
  3. Cross-document financial analysis (OfficeQA). A question requires comparing two Treasury bulletins and counting investor categories whose holdings exceed a specified threshold. SHIFT selects a Planner → Coder → Critic → Aggregator harness, with a feedback connection from the Critic to the Coder. The Coder receives file and computation tools, while the Critic receives file-reading access and instructions to check units and bounds; saved telemetry confirms tool use by both. SHIFT returns the correct count in 3/3 repeats, compared with 0/3 for single-agent ReAct, which reaches its 28-step limit.

SHIFT succeeds in 3/3 repeats on each, compared with 0/3 for ReAct. They illustrate the benefit of the evaluated workflow and computation allocation together; they do not isolate coordination from additional computation.

We appreciate the reviewer’s question, which helped us clarify the practical value of query-specific construction beyond cost savings. Our results point to document-grounded analytical tasks, such as answering financial questions across reports, where evidence gathering, calculation, and checking must work together. In these settings, a query-specific workflow and computation allocation can help a fixed executor complete substantially more tasks. The benefit is therefore greater task-solving capability when additional computation is worthwhile.

Reviewer_54zdOpenReview ↗

Thank authors for the response.

Thank authors for the response. They basically addressed my concerns, so I will maintain my positive score.

Reviewer_kCL6OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a framework called SHIFT for constructing query-specific agent harnesses. It represents a harness as an executable graph and uses a learned policy-value architect to guide monte carlo tree search over agent structures, instructions, and tool permissions. The architect is trained using search trajectories and execution feedback, and this makes SHIFT to evaluate candidate harnesses without executing them during inference.

Experiments on 6 benchmarks compare SHIFT against 17 baselines including prompt optimization, workflow optimization and agent routing methods. In the main experiment, they use Gemini 3.5 Flash as the executor. And SHIFT-search achieves best aggregated accuracy.

Critical strengths
  1. This paper focuses on how to adapt agent harness to each query without repeatedly executing each candidate harness. This is an interesting and important problem, as realistic tasks are more complex and it would be helpful if we could adapt the harness to each query. The proposed method combines a learned policy-value architect and MCTS which provides an interesting approach to this problem.

  2. The proposed method SHIFT achieves strong improvements on complex agentic tasks including OfficeQA and GAIA. The results support the author's hypothesis that harness structure matters more for tasks requiring complex tool use.

Critical weaknesses
  1. The contributions of MCTS and joint harness optimization are not clearly isolated in the ablation studies. One of the main contributions of SHIFT is using learned policy and value preditions to guide MCTS for harness construction.

However, Table 2 shows that greedy construction achieves almost the same accuracy as SHIFT-search (79.8% vs. 79.9%). It seems that the learned policy may account for most of accuracy gains, while MCTS mainly helps reduce execution costs.

I also have concerns about the action-space ablation in Figure 6. When evaluating the instruction-only and tool-only variants, the authors restrict the action space of the architect already trained on the full action space, rather than retraining each method. It is unclear whether joint optimization is better than separately trained methods.

Questions for authors
  1. What is the motivation for including short-horizon tasks in the evaluation? On GSM8K, HotpotQA, MBPP, and SpreadsheetBench, SHIFT does not outperform the strongest existing methods, while its execution cost is substantially higher than Direct. This suggests that SHIFT may be more suitable for long-horizon, tool-intensive tasks. Could the authors clarify what they aim to demonstrate through the short-horizon benchmarks?
3 comments

We thank the reviewer for their thoughtful review and positive assessment of SHIFT. We especially appreciate their recognition of the importance of query-specific harness construction and its strong performance on complex, tool-intensive tasks. The reviewer’s feedback led us to run additional experiments to clarify what search and joint optimization each contribute to SHIFT and helped us clarify where adaptive harness construction offers the greatest benefit. We address each concern in detail below.

1. What does MCTS contribute beyond the learned policy? (W1)

Table 2 shows that inference-time MCTS reduces the normalized executor-token cost index by 29.9% at nearly unchanged aggregate accuracy, while improving OfficeQA accuracy by 6.5 percentage points. Using the same trained architect, search reduces the cost index from 120.2 to 84.3, with six-benchmark macro accuracy of 79.87% versus 79.85% for greedy decoding. On OfficeQA, accuracy increases from 52.8% to 59.3%. Both decoding strategies use a policy trained on MCTS visit distributions (Eq. 4), so this comparison isolates the additional contribution of inference-time search.

To separately examine training-time search, we trained architects using either MCTS-generated targets or direct policy rollouts with reward-based updates, then evaluated each with greedy decoding and MCTS. For this supplementary study, we used Qwen3-235B-A22B, a SOTA open-weight executor. The executor is held fixed across all four conditions.

The GAIA comparison uses three training seeds, matched initialization within each seed, identical training questions, action vocabulary, and reward definition, and 360 training workflow executions per architect. We evaluate the fixed final checkpoints on the same 51 test questions, with two execution repeats, giving 306 evaluations per condition.

Architect trainingInference decodingAccuracy (%)
Direct policy rolloutsGreedy21.57%
Direct policy rolloutsMCTS25.16%
MCTS-generated targetsGreedy22.55%
MCTS-generated targetsMCTS31.37% (+9.8)

With greedy decoding, the architect trained on MCTS-generated targets achieves 22.55% accuracy, compared with 21.57% for the architect trained on direct policy rollouts. This small difference suggests comparable performance without inference-time search. When combined with MCTS at inference, the MCTS-trained architect reaches 31.37% accuracy an improvement of 8.82 percentage points over greedy decoding with the same architect and 9.80 percentage points over the direct-rollout-trained architect with greedy decoding. We will add this ablation alongside Table 2 and clarify how MCTS affects accuracy and execution cost during training and inference, including how these effects vary across executor settings.

2. Does joint optimization outperform separately trained restricted variants? (W1: action-space ablation)

The reviewer is correct: Figure 6 measures the sensitivity of a full-action architect to restrictions imposed at inference. It does not establish superiority over architects trained specifically for those restricted spaces. To address this, we separately trained full, directives-only, and tools-only architects on HotpotQA. All three variants use the same Llama-Scout executor, matched initialization within each training seed, and the same task splits and training schedule: 472 training questions, 128 validation questions, and four training epochs. We selected checkpoints using validation and evaluated them on 1,000 shared test questions, with three training seeds and two execution repeats per question.

Training action spaceHotpotQA Test accuracy (↑)GAIA Test accuracy (↑)
Full58.63 ± 0.35%18.63 ± 1.39%
Instruction-only58.53 ± 0.58%13.24 ± 0.69%
Tools-only55.65 ± 0.15%13.73 ± 0.00%

On HotpotQA, which mostly needs retrieval, instructions carry most of the benefit which is why see similar accuracy with Full and instruction-only action spaces; joint optimization matters more on tasks that need tools and multiple agents together, which we see in GAIA.

These results complement Figure 6: the original experiment restricts actions during inference, while the new experiment trains each architect specifically for its action space. We will add this comparison to the paper and revise the discussion to distinguish these settings and report where joint optimization helps and where restricted variants perform better.

3. Why include shorter-horizon tasks? (Q1)

We agree that SHIFT’s strongest benefits are on complex, long-horizon tasks requiring extensive tool use. We include shorter-horizon tasks to assess whether query-specific construction preserves strong performance where simple methods already work and whether it avoids building unnecessarily elaborate harnesses. These benchmarks help establish when additional orchestration is useful and when its overhead outweighs the benefit.

This also motivates our cost-aware value function, which rewards task success while penalizing execution cost and additional agents and tools. The objective encourages the architect to learn when additional complexity is worthwhile. Appendix B.1, Figure 13(b), provides evidence of adaptive harness size: SHIFT-value selects one or two agents for almost all GSM8K and MBPP queries, at approximately one-tenth of SHIFT-search’s executor-token cost, while selecting four or more agents on GAIA and SpreadsheetBench.

Our existing candidate-selection experiment provides complementary evidence. This is a separate controlled ranking study using Gemma 3 27B on 400 HotpotQA and 200 GSM8K questions. Holding each question’s candidate pool fixed, selecting by predicted utility improves accuracy while reducing execution tokens relative to uniform selection:

BenchmarkUniform: accuracyValue: accuracyUniform: tokensValue: tokens
GSM8K89.7 ± 0.7%92.6 ± 0.9%8.8k1.8k
HotpotQA39.1 ± 0.3%49.6 ± 0.4%19.5k9.5k

These results show that the learned value function can identify smaller, more economical harnesses on simpler tasks. The comparison is against uniform candidate selection; Direct remains an important lower-cost reference in the main evaluation. We will clarify this motivation in the introduction and organize the experimental discussion around the benefits on complex, long-horizon tasks and the accuracy–overhead tradeoff on shorter-horizon tasks. We will retain the complete six-benchmark comparison and give shorter-horizon results a dedicated discussion, making clear where query-specific construction offers the greatest practical benefit.

Thank you for the detailed response and additional experiments. The new ablations help address my main concerns. The explanation for including short-horizon tasks is also reasonable. I encourage the authors to include these experiments and the corresponding analysis in the revised manuscript, as they help clarify the contributions of SHIFT. Based on the current discussion, I am inclined to maintain my rating of 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.