acceptodds
Under review as a conference paper at ICLR 2027

Learning When and What to Teach: Self-Evolving Privileged Search Agents

Abstract

Long-horizon search agents are difficult to optimize because rewards arrive only after many interdependent actions. On hard tasks, all sampled trajectories may fail, eliminating group-relative learning signals and leaving useful recovery actions outside the policy’s support. We observe that search answers are often entities or relations that can serve as training-time anchors for reasoning backward toward better queries. Direct answer conditioning, however, risks privilege leakage rather than transferable search behavior. We introduce MIST-OPD, a single-backbone self-evolution framework requiring no external teacher, language-model judge, or reward model. An answer-blind Student and a privileged Guide are initialized as two trainable policies from the same base model; their asymmetry is information, not capacity. At each public state, the Guide transforms the answer into backward hints at six ordered granularities, ranging from general search strategy to an explicit multi-step route that still withholds the answer. Repeated common-seed rollouts compare no intervention, each candidate granularity, and a source-matched placebo at the same granularity. These matched counterfactuals determine whether guidance is causally useful and select the minimum sufficient intervention for each state. Training alternates updates to both roles. The Guide learns when to abstain, how much information to reveal, and what to teach. The Student learns when to request help and which next action to take through causal token-level OPD, private-token projection, and behavior replay. A lagged Guide stabilizes distillation, and the updated pair generates the next interaction round. At deployment, privileged information and the Guide are removed, leaving only the evolved answer-blind Student. Results show that MIST-OPD achieves state-of-the-art performance, demonstrating its effectiveness as a self-evolving framework for converting privileged answers into transferable long-horizon search behavior without external model supervision.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Self-Evolving, On Policy Distillation, Long-horizon Agent, Post-Training

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_ZaG5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes MIST-OPD, a framework for training search agents using privileged information (PI). A Guide proposes graded interventions based on reference answers, and the accepted interventions are distilled into the Student. Experiments on Qwen3.5 show improvements over several baselines on search and code benchmarks.

Critical strengths
  1. The paper explores an interesting direction for using PI to improve search agents.
  2. The paper provides experiments across multiple benchmarks and model scales, with several component ablations.
Critical weaknesses
  1. Unclear contribution of the limited supervision and self-evolution.

The method achieves a large improvement over the untrained agent using only 197 accepted states, along with replay supervision, with a learning rate of 2×10−72\times 10^{-7}. Moreover, Round 1 already achieves most of the final performance. The authors should better explain these results and the additional contribution of self-evolution.

  1. Insufficient justification for the complicated method design.

The method introduces many components but fails to provide clear justification for them. The current ablations are insufficient to establish the usefulness of each component. More detailed ablations and hyperparameter sensitivity studies are needed.

  1. Unclear experimental and implementation details.

The paper does not specify the evaluation budget or thinking-mode configuration, making it difficult to assess the fairness of the comparisons with other methods. It is also unclear how intermediate public states are selected from Student rollouts for Guide intervention. These details are important for understanding MIST-OPD.

  1. Missing training-cost comparison.

MIST-OPD requires substantial additional computation for Guide proposals, continuations, and Guide updates. The provided BoN control matches continuation calls only, not proposal generation or Guide updates. Wall-clock time and a compute-matched baseline should be provided.

  1. Unclear presentation.

The training procedure is difficult to follow. A clear end-to-end algorithm description would improve readability.

Questions for authors
  1. Could the authors explain the large gains from relatively few accepted states?
  2. Could the authors provide more detailed loss ablations and hyperparameter sensitivity results?
  3. Could the authors clarify the experimental details, particularly the evaluation budget, thinking-mode configuration, and intermediate state selection?
  4. Could the authors report wall-clock time for MIST-OPD and the baselines?
11 comments
AuthorsOpenReview ↗

Response to Reviewer ZaG5

Thank you for asking us to clarify supervision volume, the incremental value of self-evolution, and the role of the individual components. We address these with matched controls that separate repair selection, replay, and Guide adaptation, alongside the complete compute accounting.

Definitions and evaluation scope. Results concern Qwen3.5-9B unless stated otherwise. Macro accuracy is the unweighted mean over BrowseComp, BrowseComp-Plus, FRAMES, GAIA-text, HLE-text, and HotpotQA; pp denotes absolute percentage points. π-Distill is the privileged-distillation baseline. The Frozen Guide control keeps the teaching policy fixed while continuing Student training. R is the number of matched construction continuations per arm. The frozen-curriculum controls share fixed construction artifacts and are distinct from the submitted online system. Scoring and deployment settings are specified in W3/Q3 below.

W1 / Q1. How can 197 accepted states yield substantial gains, and what does self-evolution add after Round 1?

Response. We clarify the supervision associated with the 197 accepted repairs. Across four rounds, these records yield 197 Student updates. Each update combines one accepted repaired trajectory with eight source-balanced replay records, totaling 1,576 replay-record uses.

Two supervision scopes are involved. Trajectory-level cross-entropy (CE) supervises assistant output tokens throughout each accepted repaired trajectory, while repair-specific supervision is restricted to the local intervention span, capped at 256 tokens. This trajectory-level CE, with weight 0.2, was already part of the reported full-method training. The 8,192-token limit applies to each serialized training sequence, including conditioning context; question and tool-observation tokens are not CE targets. Replay CE is capped at 2,048 target tokens per replay record. The 41,736 repair-span positions and 36,918 repair-local KL positions are counts of token prediction positions accumulated over the 197 repair records. The span count measures the bounded repair segments; the KL count measures positions contributing to the local distillation loss. They are not disjoint token pools to be added together, and neither is the total number of trajectory-level CE targets. We will align the objective definition in Sec. 2.3, the training algorithm in Appendix A, the local-target description in Appendix C, the optimization settings in Appendix E.2 and Table 12, and the baseline budget descriptions with these supervision scopes.

QuantitySubmitted training run
Accepted repaired trajectories / Student optimizer steps197 / 197
Trajectory-level CE supervisionAssistant outputs throughout each accepted repaired trajectory
Maximum serialized training sequence length8,192 tokens, including conditioning context
Local repair-span cap256 target tokens per repair
Repair-span target-token positions, summed over 197 repairs41,736
Token positions contributing to repair-local KL, summed over 197 repairs36,918
Replay records / CE-supervised tokens1,576 / 1,842,116
Replay-supervision cap2,048 tokens per record
ParametersFull-parameter Qwen3.5-9B Student and Guide; no LoRA
Optimizer / learning ratesAdamW; Student 2×10⁻⁷, Guide 1×10⁻⁷; norm clip 1.0
PrecisionFP32 parameters and AdamW states; BF16 autocast

The contribution of training on accepted repaired records is measured with the existing controlled curriculum:

Frozen-curriculum controlMacro accuracy (%)
Replay-only48.1
No-gate + Full-OPD49.6
Full repair teaching + replay52.4
Full − Replay-only+4.3 pp

Full exceeds Replay-only by 4.3 points and No-gate by 2.8 points. The first contrast measures the contribution of the repaired-record training branch beyond replay, including trajectory CE and repair-specific teaching. The No-gate contrast retains the objective structure and update schedule while changing outcome-based state/tier selection. It measures the value of screened training records. The matching is in initialization, objective structure, replay, and update schedule; the repair-span figures are local supervision statistics. The submitted online result of 52.8% is separate; the two controlled contrasts consistently use Full = 52.4%.

For self-evolution, the evolving and Frozen Guide arms give:

CheckpointEvolving Guide: Student macro (%)Frozen Guide: Student macro (%)
Shared initialization42.742.7
Round 150.450.3
Round 251.350.8
Round 352.051.1
Round 452.851.3
Round 4 − Round 1+2.4+1.0

After Round 1, the evolving arm gains 2.4 points while Frozen gains 1.0 point. Thus, later adaptation contributes beyond continued Student training, even though the first round produces a substantial initial gain.

The fixed-state control provides complementary evidence. Starting from the same Round-3 Student on 228 Round-4 states, guided continuation succeeds on 49/228 states without guidance, 56/228 with Frozen Guide, and 71/228 with Updated Guide. Updating the Student using each Guide's accepted set gives 51.9% versus 52.5% guide-free macro accuracy. This downstream comparison measures the end-to-end value of the accepted sets, including differences in their size; the common-state continuation comparison directly measures guidance quality. Together, these results support the incremental value of adapting the Guide to the evolving Student.


W2 / Q2. Do the ablations justify the components and isolate their contribution?

Response. The matched-objective control shows a 2.8-point benefit from repair selection while keeping controlled distillation and replay fixed. It complements the 4.3-point Full-versus-Replay-only comparison and the submitted component ablations.

Controlled OPD (on-policy distillation) constructs an answer-blind Student target from the lagged Guide’s gold-answer, placebo, and public-only views, using an anchored residual and private-token projection. Trajectory-level CE supervises assistant outputs in the repaired record, alongside the bounded repair-specific objectives and replay. KL denotes the distributional distillation loss. No-gate + Full-OPD retains the objective structure and changes outcome-based repair selection.

No-gate + Full-OPD samples the Guide's predicted legal tier within each source×round stratum without consulting outcome-based acceptance. It preserves residual projection, KL/CE weights, replay, initialization, and the 197-update schedule, as well as format and explicit-answer checks. Full and No-gate contain 41.7k/42.1k target-token positions in their bounded repair spans and 36.9k/37.4k positions contributing to repair-local KL; all three arms use 1.84M replay CE targets. These statistics distinguish repair-local loss positions from replay supervision. Trajectory-level CE additionally supervises assistant outputs across each repaired record. The controlled comparison holds initialization, objective structure, replay, and the update schedule fixed; these local-span figures do not establish equality of total trajectory-level CE target counts.

Shared frozen-curriculum conditionOutcome-based selectionControlled OPDMacro accuracy (%)
Replay-onlyNot usedDisabled48.1
No-gate + Full-OPDDisabledEnabled49.6
FullEnabledEnabled52.4

48.1 → 49.6 → 52.4 separates improvement beyond replay from the further benefit of screened repairs. The shared curriculum isolates repair selection while preserving the distillation objective, replay, and update schedule. The +2.8-point contrast measures the value of selecting which states and tiers provide the teaching signal.

Submitted component comparisonFull (%)Ablated (%)
No placebo52.850.4
No hierarchy (binary ABSTAIN/HELP variant)52.850.2
Gold-only anchored distillation52.851.9
Frozen Guide52.851.3

No-gate and Replay-only isolate screening and the joint repair objective. They do not isolate trajectory-level CE from KL, and changing R does not replace a loss-term ablation. The submitted no-hierarchy ablation is the binary ABSTAIN/HELP variant; it is distinct from the Always-L5 training control. The CE and KL knockouts keep the Full arm’s 197 accepted states, 197 updates, replay, and remaining weights. Always-L5 replaces the repair targets on those same states. No-gate and Replay-only are repeated below as reference controls: No-gate matches the source×round allocation and update count while changing state/tier selection; Replay-only removes the repair objectives.

Frozen-curriculum knockoutDisabled termMacro accuracy (%)Full − ablated (pp)
Full—52.4—
No trajectory-level CETrajectory-level CE term51.2+1.2
No KLDKL(qi∥Sθ)D_{\mathrm{KL}}(q_i\|S_\theta)50.6+1.8
Always-L5 trainingSelected-tier target (replaced by L5)50.8+1.6
No-gate + Full-OPDOutcome-based selection49.6+2.8
Replay-onlyAll repair objectives48.1+4.3

Removing KL costs 1.8 points and removing trajectory-level CE costs 1.2 points; both remain above No-gate and Replay-only. Selected-tier Full has a 1.6-point advantage over Always-L5 in this matched frozen-curriculum comparison.

KL weight λKD\lambda_{\mathrm{KD}}SettingMacro accuracy (%)
0.1250.5×0.5\times submitted51.9
0.25Submitted52.4
0.502×2\times submitted52.1

The trajectory-level CE, replay, hierarchy, and public-help weights remain fixed at 0.2, 1.0, 0.5, and 0.1, respectively. The submitted KL coefficient has the highest observed macro among these three settings; halving or doubling it changes macro by at most 0.5 points. The matched controls support the contributions of screening, trajectory-level CE, and KL, while the local coefficient sweep tests sensitivity to the KL weight.


Reviewer_88E4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper tackles the problem that search-based agents often struggle to recover from errors during execution. The authors propose initializing both a "Student" and a "Guide" from the same base model. During training, the Guide uses ground-truth answers to propose tiered interventions, which are then filtered by comparing rollout outcomes starting from the same state. Finally, both models are iteratively updated via controlled distillation. The method is evaluated across several search benchmarks and further extended to smaller models and code generation tasks.

Critical strengths

Using ground-truth answers to automatically discover recovery actions (and learning when and how much to intervene) is a neat and practical idea. It provides a viable way to train search agents without relying on a much stronger external teacher model. The overall pipeline makes intuitive sense. I also appreciate the thorough baselines, especially the fixed-Guide ablation, the rollout-matched best-of-N comparison, and the independent rollout audits, which make the empirical results quite convincing.

Critical weaknesses
  1. Even with the rollout audits and sensitivity analyses, the filtering step still relies heavily on sampling. More importantly, because the Guide has access to the ground-truth answer, it is hard to tell whether the accepted interventions capture genuine, transferable recovery strategies or just subtle semantic shortcuts/hints leaked from the ground truth.
  2. While the paper matches the number of rollouts for baselines like best-of-N, the proposed pipeline introduces non-trivial extra overhead (generating candidate interventions, multi-condition scoring, and updating the Guide). It is not clear if the method still holds a net advantage under a strictly matched total training compute budget.
  3. Several important setup details are missing or hard to find, including the exact search backend, query/call budgets during inference, decoding hyperparameters, and exact metric implementations. The description of the training data construction and train/eval splits for the code generation tasks is also too brief to ensure reproducibility.
Questions for authors
  1. Can you provide a more direct semantic audit (or qualitative examples) of the accepted interventions to show how many generalizable recovery behaviors versus answer-specific hints? Also, how sensitive is training stability across multiple random seeds?
  2. What is the exact computational overhead of the full training loop (including candidate generation, scoring, and Guide updates) compared to the baselines? How does the cost-performance trade-off look when plotted against total training FLOPs or wall-clock time?
  3. Could you clarify the exact search environment, query budget, and evaluation protocol for each benchmark? Please also elaborate on how the coding dataset was constructed and how you prevented data contamination between the training and evaluation splits.
5 comments
AuthorsOpenReview ↗

Response to Reviewer 88E4

Thank you for recognizing the practical motivation and the value of the fixed-Guide, best-of-N, and independent-audit controls. We address the remaining concerns with a complete semantic audit, an occupied-compute-matched comparison, and consolidated implementation details.

Definitions and evaluation scope. Unless stated otherwise, results concern Qwen3.5-9B. Macro accuracy averages BrowseComp, BrowseComp-Plus, FRAMES, GAIA-text, HLE-text, and HotpotQA equally; pp denotes absolute percentage points. π-Distill is the privileged-distillation baseline. R counts matched construction continuations per arm, rather than independent training runs. Occupied GPU-hours sum assigned devices × stage elapsed time, including API wait while devices remain assigned; reserved device-hours measure the full GPU reservation. Scoring and the shared deployment settings are specified in W3/Q3 below.

W1 / Q1. Do accepted interventions teach transferable recovery strategies or answer-specific shortcuts? How stable are the findings?

Response. The semantic audit finds 157/197 interventions (79.7%) publicly grounded or derivable from public evidence, with no detected direct answer/alias leakage in the accepted Guide hints and rendered repairs. Transfer evaluation independently retains a 4.3-point advantage on answer-disjoint questions and a 3.6-point advantage after additionally excluding explicit teaching entities.

Two authors independently label all accepted Guide interventions and rendered repair actions, first from the public state and then with the privileged answer. Primary labels are mutually exclusive; direct leakage, unsupported cues, and uncertainty take precedence over public grounding. Disagreements are adjudicated by a third author. Exact primary-label agreement is 169/197 (85.8%).

Primary categoryInterventionsShare (%)
Publicly grounded recovery11055.8
Answer-related cue derivable from public evidence4723.9
Answer-specific cue without sufficient public support168.1
Uncertain2412.2
Direct answer or alias leakage00.0
Total197100

The audit follows each teaching signal from Guide text to the rendered repair target. It assesses privileged teaching content, separately from ordinary final-answer outputs in trajectory-level cross-entropy (CE). A representative Round-2 MuSiQue L2 repair corrects the tested relation while its query uses only entities already present in the public observation. Gold-alias scans find no accepted Guide hint or rendered repair containing an answer alias, and the filter excludes homonym overlaps identified in unused drafts. The full category counts above retain the audit denominator of 197.

The deployment evaluation below tests whether the audited teaching signal transfers to disjoint answers and teaching entities.

The two frozen subsets are evaluated head-to-head below. Each entry gives correct predictions / evaluated questions; macro accuracy is the unweighted mean of the six unrounded rates. The answer-disjoint subset excludes overlap with construction and replay answer aliases. The stricter subset additionally excludes question/gold entities overlapping explicit teaching bridge or route entities, using normalized entity/alias matching.

Answer-disjointnBaseπ-DistillMIST-OPDFull − π-Distill (pp)
BrowseComp1,184241 / 1,184291 / 1,184343 / 1,184+4.4
BrowseComp-Plus792235 / 792300 / 792332 / 792+4.0
FRAMES776469 / 776528 / 776560 / 776+4.1
GAIA-text12160 / 12164 / 12170 / 121+5.0
HLE-text2,094503 / 2,094565 / 2,094651 / 2,094+4.1
HotpotQA4,1202,834 / 4,1203,152 / 4,1203,316 / 4,120+4.0
Macro9,08742.147.852.1+4.3
Answer + teaching-entity-disjointnBaseπ-DistillMIST-OPDFull − π-Distill (pp)
BrowseComp1,022202 / 1,022246 / 1,022290 / 1,022+4.3
BrowseComp-Plus661191 / 661246 / 661268 / 661+3.3
FRAMES640380 / 640429 / 640450 / 640+3.3
GAIA-text9848 / 9851 / 9855 / 98+4.1
HLE-text1,731406 / 1,731456 / 1,731524 / 1,731+3.9
HotpotQA2,2601,536 / 2,2601,730 / 2,2601,790 / 2,260+2.7
Macro6,41241.447.250.8+3.6

The advantage is positive on every benchmark in both subsets: +4.3 pp macro on 9,087 answer-disjoint questions and +3.6 pp on 6,412 answer-plus-teaching-entity-disjoint questions. These results support transfer beyond overlap with the training answers and explicit teaching entities under the specified matching rules.

For rollout stability, independent selected-tier success is 35.6%/34.7%/33.9% at construction R=2/5/10 on the Round-1 45/36/31 accepted states. Separately, the supervision-rebuilding and retraining comparison in Appendix D, Table 8 reports downstream macro accuracy of 52.8%/52.4%/52.1%. Those macros are not continuation-audit rates. Each training condition is represented by one run; the independent continuation audit evaluates rollout-level repeatability.


W2 / Q2. Does the gain persist when candidate generation, scoring, and Guide updates are included in the compute budget?

Response. Yes under matched occupied GPU compute: MIST-OPD reaches 52.8% versus 50.4% for extended π-Distill at 412.4 occupied H200-hours, a 2.4-point advantage. The comparison charges construction and Guide updates to MIST-OPD and allows π-Distill to spend its additional compute on its normal data generation and Student optimization.

The complete ledger includes state collection, candidate/placebo generation, failed and rejected continuations, teacher-target computation, replay, and Student and Guide updates. Occupied GPU-hours are assigned device count × stage elapsed time, including API wait while those devices remain assigned. The non-overlapping intervals assigned to the five stage categories below sum to 86.4 wall-clock hours and 412.4 occupied H200-hours. Within optimization, Student precedes live Guide, followed by the Guide EMA copy. Continuation execution includes 41.6 hours of API wait. Reserved device-hours are reported alongside the matched unit.

ComponentMIST-OPD devices × stage elapsed hoursMIST-OPD occupied H200-hoursπ-Distill occupied H200-hours
State/data collection and candidate generation4 × 12.1548.662.0
Continuation execution and scoring4 × 57.55230.2198.6
Teacher-target computation8 × 3.9031.241.8
Student optimization, including replay8 × 8.62569.0110.0
Guide optimization8 × 4.17533.4Not applicable
Occupied total—412.4412.4

Peak hardware is eight H200 141GB GPUs. The matched occupied-compute budget is B=412.4 H200-hours. Initialization, the 12,000-question 2WikiQA/MuSiQue inventory (6,000 each), teacher-target answer access, and deployment settings are shared. Checkpoints are selected by a prespecified occupied-budget rule, not by test-set performance.

MethodOccupied H200-hoursReserved device-hoursWall-clockStudent updatesMacro accuracy (%)
π-Distill, original allocation118.6—24.8 h19748.3
π-Distill, occupied-budget matched412.4632.879.1 h48650.4
MIST-OPD412.4691.286.4 h19752.8

Extending π-Distill improves it by 2.1 points but leaves the 2.4-point gap. Its B/4, B/2, and B scores are 48.6%/49.5%/50.4%; MIST-OPD's round checkpoints score 50.4%/51.3%/52.0%/52.8% at 158.2/256.1/334.7/412.4 occupied H200-hours. These are discrete checkpoint measurements.

The improvement persists after including the complete construction and optimization pipeline in the occupied-GPU-hour budget. Resource accounting additionally reports 691.2 versus 632.8 reserved device-hours, 1.84M versus 1.62M search/API calls, and 41.6 versus 28.4 API-wait hours. Reserved time includes the eight-GPU allocation during waiting and stages using fewer devices. Thus, the comparison fixes occupied GPU-hours and explicitly documents the accompanying resource costs. Deployment uses the same answer-blind Student interface and requires zero Guide calls.


W3 / Q3. What are the search environment, evaluation protocol, and code-data isolation rules?

Response. All compared methods use Serper as the search backend under the same evaluation harness. As specified in Appendix C, the model-facing search interface provides search(query) and batch_search(queries). These calls use Serper, and returned search results are formatted into public tool observations using the same rules across methods. Each query is limited to 16 words; each search returns six results, with snippets capped at 700 characters each and the resulting observation capped at 4,200 characters. The search-cache identifier is serper-c0-2026-08-19.

Thinking is enabled in the main evaluation, including the reported 52.8% macro accuracy of MIST-OPD. Compared methods share the reasoning configuration, model-facing search interface, observation formatting, context limit, model-call budget, and final-answer grading protocol. Sec. 3.1 specifies the common evaluation protocol; we consolidate the settings below. The paper's 8,192-token sequence length is a training setting. Local repair and replay supervision are capped at 256 and 2,048 target tokens, respectively; trajectory-level CE additionally supervises assistant outputs throughout each accepted repaired record. At inference, the total context window is 32,768 tokens, including the prompt and current completion, and each call generates at most 2,048 new tokens. Thinking tokens count toward this generation cap. Training sequence length, inference context capacity, and per-call generation are separate limits.

SettingEvaluation configuration
Search backendSerper
HarnessShared search-agent harness with fixed search-result formatting
Final-answer judgeGPT-5 mini (LLM-as-judge)
Search cacheserper-c0-2026-08-19
Search outputSix results; 4,200-character observations; 700-character snippets
Query / model-call limits16-word queries; 30 model calls
Training sequence length8,192 tokens per training sequence
Inference context window32,768 tokens total, including prompt, tool schemas, observations, retained history, and current completion
Per-call generation capAt most 2,048 new tokens
Thinking-token accountingThinking tokens share the 2,048-token generation cap with tool-call or answer tokens; no additional thinking allowance
Thinking modeEnabled in the main evaluation
Student decodingTemperature 0.2; final-answer temperature 0.0
Failure handlingTimeouts, malformed outputs, tool failures, and exhausted budgets remain in the denominator and score incorrect
BenchmarkEvaluation questionsScoring in the shared harness
BrowseComp1,266GPT-5 mini LLM judge
BrowseComp-Plus830Same
FRAMES824Same
GAIA-text127Same; 38 attachment-dependent examples excluded
HLE-text2,158Same; 342 multimodal examples excluded
HotpotQA7,405Same

For final-answer evaluation, we use GPT-5 mini as an LLM judge. Within each benchmark, all compared methods use the same judge and grading protocol. Correctness is assessed from the final answer after the complete agent trajectory, and macro accuracy is the unweighted mean over the six benchmarks. Sec. 3.1 already specifies the common evaluation protocol; we additionally identify the evaluation judge here and will make this detail explicit in the revised manuscript.

Code-data itemSetting
Construction sourceSWE-smith (Yang et al., 2025), Hugging Face SWE-bench/SWE-smith, revision 9f2a104651948c44939327e9649a4c2c2dd57c91; arXiv:2504.21798
Split unitRepository plus issue identifier
Retained training tasks1,842 after decontamination
Evaluation500 SWE-bench Verified and 300 SWE-bench Multilingual issues
Training feedbackPublic issue text, repository state, and public construction-scaffold tests
Evaluation testsHidden tests used only after the trajectory is frozen
Overlap checksExact issue ID, patch hash, issue-text near-duplicates, and repository/issue matching
Removal38 construction tasks removed; evaluation splits retained in full

Evaluation tests are unavailable during training, and construction/replay overlap is checked before training. These checks document construction/evaluation separation and keep held-out evaluation tests outside the training feedback path.


Reviewer_tDt5OpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MIST-OPD, a framework for improving search agents using training-time reference answers without relying on a stronger external teacher. The method initializes a Student and a privileged Guide from the same base model. At states visited by the current Student, the Guide uses the reference answer to propose six increasingly specific interventions that withhold the answer itself. Matched continuations compare each intervention against no guidance, the adjacent weaker intervention, and a placebo generated from an unrelated answer, retaining the first level that passes all applicable controls. Accepted interventions train the Student through action supervision and controlled distillation, while also teaching the Guide when and how to intervene. Both models are updated over successive rounds, and only the Student is deployed.

Critical strengths
  1. Effective use of privileged answers to discover recovery behavior. Instead of directly teaching the Student to reproduce the answer, the same-backbone Guide uses training-time answers to propose recovery actions that the current Student policy has not explored, providing useful learning signals in all-failure and sparse-reward settings.

  2. Careful control of teaching signals through graded interventions and matched public execution. The intervention ladder seeks the least informative yet effective hint, while the no-guidance, adjacent-weaker-tier, and placebo controls require each Guide proposal to demonstrate utility under the same public execution environment. This is more principled than directly distilling an answer-conditioned Guide.

  3. A clear separation between training-time privilege and deployment-time behavior. The Guide, reference answers, intervention labels, and teacher logits are used only during training, while deployment retains only the answer-blind Student. Alternating Guide-Student updates also allow the system to generate new repair signals for failures encountered by the evolving Student.

Critical weaknesses
  1. The relatively low same-tier agreement under larger repeat counts suggests that the selected intervention tier is sensitive to rollout noise. Since the accepted tier directly determines the teaching signal, R = 2 may be insufficient to reliably identify stable first-passing boundaries. While larger R increases construction cost and reduces the number of accepted states, it should provide a more reliable estimate of whether an intervention consistently improves success. Can the authors further justify the reliability-cost trade-off behind using R = 2?

  2. The paper compares the selected first-passing tier with L5 on a small held-out audit set, but it does not fully establish how success changes across stronger tiers. How often do stronger tiers outperform, tie, or underperform the selected boundary? If stronger tiers are not consistently more successful, what does the ordinal ladder mean operationally, and how should the minimum-intervention claim be interpreted?

  3. The paper reports a small audit distribution over 45 accepted states and a decreasing trend in L5 proposals, but it does not provide the full accepted-tier distribution across all training rounds. Can the authors report the distribution of accepted intervention levels across all accepted boundaries, rounds, and datasets? If a large fraction of accepted interventions are concentrated at L3-L5, this would raise concerns that the method relies on high-specificity privileged hints rather than general recovery behavior.

  4. Does the trained Student tend to generate recovery behaviors corresponding to high-specificity tiers such as L3-L5 in its rollouts? Since the deployment Student does not output tier labels, can the authors annotate or classify Student actions under the ladder taxonomy and compare the distribution against base and distillation baselines? Are these bridge/route-like patterns still beneficial on answer/entity-disjoint subsets? Such an analysis would help distinguish general recovery behavior from possible internalization of privileged answer-conditioned patterns.

Questions for authors

See the weaknesses.

6 comments
AuthorsOpenReview ↗

Response to Reviewer tDt5

Thank you for recognizing the value of same-backbone repair discovery and matched public execution. We address your concerns with independent continuation audits, the complete accepted-tier distribution, and deployment-time behavior and transfer analyses.

Definitions and evaluation scope. R is the number of matched construction continuations per arm. An accepted boundary is a public state together with its selected first-passing intervention. No guidance continues without a Guide intervention; the same-tier placebo is a matched control generated using an unrelated answer. Unless stated otherwise, model results concern Qwen3.5-9B. Macro accuracy is the unweighted mean of accuracies on BrowseComp, BrowseComp-Plus, FRAMES, GAIA-text, HLE-text, and HotpotQA; pp denotes absolute percentage points. π-Distill is the privileged-distillation baseline. Final-answer accuracy is evaluated using GPT-5 mini as an LLM judge under a common grading protocol across the compared methods.

W1. Is R=2 sufficient to identify reliable first-passing interventions?

Response. R=2, 5, and 10 each screen interventions that remain useful on independent continuations. Selected-tier success exceeds no guidance and the same-tier placebo in every row. On states jointly accepted by R=2 and each larger-R setting, exact tier agreement is 28/32 (87.5%) at R=5 and 26/29 (89.7%) at R=10. R=2 obtains this screening at 26 continuation episodes per state, versus 130 at R=10.

The comparison freezes the same 296 technically valid Round-1 public states, Guide checkpoints, candidate ladders, and execution budgets. The gate accepts 45, 36, and 31 states at R=2, 5, and 10. Evaluation uses two matched continuations per accepted state, disjoint from the continuations used for selection. Thus, the reported utility is measured after selection on independent executions. Each construction repeat schedules 13 arms: six candidate tiers, six same-tier placebos, and no guidance. The 26/65/130 counts below are continuation episodes per state, not total end-to-end training costs.

Construction repeatsAccepted / validConstruction episodes / stateHeld-out selected-tier successSelected − no guidance (pp)Selected − same-tier placebo (pp)
R=245 / 2962632 / 90 (35.6%)+23.3 (32/90 − 11/90)+20.0 (32/90 − 14/90)
R=536 / 2966525 / 72 (34.7%)+22.2 (25/72 − 9/72)+18.1 (25/72 − 12/72)
R=1031 / 29613021 / 62 (33.9%)+21.0 (21/62 − 8/62)+16.1 (21/62 − 11/62)

Because the accepted sets differ in both size and membership, their aggregate success rates compare different selected populations. We therefore also fix state membership. On the 29 states accepted by all three settings, success is 21/58 (36.2%), 21/58 (36.2%), and 20/58 (34.5%), respectively. A separate supplement of 80 accepted states from Rounds 2–4 gives 56/160 successes (35.0%), versus 21/160 without guidance and 26/160 with placebo: improvements of 21.9 and 18.8 points.

We separate changes in which states are accepted from changes in the selected tier on jointly accepted states. Accepted-set Jaccard is intersection divided by union. The submitted same-tier/union statistic counts a state as agreeing only when both settings accept it at the same tier; it therefore includes both sources of variation.

Comparison with R=2Accepted-set intersection / unionSame tier / unionSame tier / jointly accepted states
R=532/49 (65.3%)28/49 (57.1%)28/32 (87.5%)
R=1029/47 (61.7%)26/47 (55.3%)26/29 (89.7%)

Among jointly accepted states, exact tier agreement is 87.5%–89.7%. The union-based figures are lower because increasing R also changes which states are admitted; accepted sets are not nested, and the training data therefore change with R. Held-out selected-tier success is 35.6%/34.7%/33.9%; these rates concern different accepted-state populations. They are Round-1 continuation-audit scores on the 45/36/31 accepted states, not downstream training scores.

The downstream sensitivity results are already reported in Appendix D, Table 8: rebuilding supervision and retraining gives macro accuracies of 52.8%, 52.4%, and 52.1% at R=2, 5, and 10. These are downstream Student scores, separate from the held-out continuation rates above. Repeating the gate changes the accepted supervision, so this comparison evaluates the resulting training configurations rather than an isolated effect of R on an unchanged training set.

R=2 combines lower construction cost, more retained repairs on the frozen Round-1 slice, and the highest reported downstream accuracy. The gate uses the same minimum-win and zero-reversal conditions for every R; increasing R creates more opportunities to observe either a win or a disqualifying reversal. On this slice, R=2 admits 45 repairs, compared with 36 and 31 at R=5 and R=10. With one pass over the accepted set, these retained repairs also provide more Student update opportunities. The downstream sensitivity results are 52.8%, 52.4%, and 52.1%, respectively. The practical advantage of R=2 thus includes both its lower screening cost and the supervision retained under that screening procedure. This comparison evaluates the resulting training configurations, including changes in accepted states and update opportunities.

R=2 screens effective repairs at one-fifth of the R=10 construction-continuation cost. Independent continuation gains and high conditional tier agreement support this cost–utility trade-off.


W2. How often do stronger tiers outperform, tie, or underperform the selected tier? What does the ladder mean operationally?

Response. The selected tier provides effective guided recovery and stronger downstream learning than always using maximum-specificity hints in the matched training control. Replacing the selected tier with its weaker reference reduces guided terminal correctness from 35.6% to 17.8%; replacing it with L5 raises guided terminal correctness to 38.9%. These are guided continuation outcomes, distinct from the unguided Student’s performance after training. On the same 45-state, two-continuation audit:

Intervention-time armHeld-out success
Adjacent weaker control (unguided for L0)16 / 90 (17.8%)
Selected first-passing tier32 / 90 (35.6%), +17.8 pp vs adjacent weaker
Always L535 / 90 (38.9%), +3.3 pp vs selected

The selected intervention is below L5 on 38 of 45 states. Round 1 contains one accepted L0 state. The gate requires Lk−1L_{k-1} only for k>0k>0. For that L0 state the weaker control is unguided, and those two held-out trials fail (0/2); the selected L0 repair succeeds on both (2/2). The other 44 states use Lk−1L_{k-1} and give 16/88 (18.2%). Combining them recovers the submitted 16/90 (17.8%). On the 44 states with a defined adjacent tier Lk−1L_{k-1}, selected-tier success is 30/88 (34.1%) versus 16/88 (18.2%), an improvement of 15.9 pp. Thus, the selected-tier advantage also holds when the L0 state is excluded. The 16/90 denominator stays 45×245\times 2, matching the other arms.

The ladder orders permitted information specificity: L0 diagnoses the failed branch; L1 specifies a search operation; L2 identifies a relation or entity type; L3 supplies a distinguishing attribute; L4 provides a bridge; and L5 gives a route. This ordering does not assume monotone success probabilities. A candidate is tested against no guidance, the adjacent weaker tier where applicable, and its same-tier placebo at the same public state. The method selects the first passing tier, subject to the stronger-tier reversal check. To characterize stronger tiers directly, we compare every available stronger tier with the selected tier on the same two held-out continuations. A stronger tier wins when it succeeds and the selected tier fails, loses in the reverse case, and ties when both outcomes agree.

Stronger tier relative to selected tierPaired comparisonsStronger winsTieStronger losesStronger − selected success (pp)
One level above767645+2.6
Two levels above648515+4.7
Three or more levels above729567+2.8
All available stronger tiers2122417117+3.3

Each difference is 100 × (wins − losses) / paired comparisons. States selected at L5 have no stronger-tier comparison. Outcomes agree in 171/212 comparisons (80.7%); stronger tiers win 24 times and lose 17 times. The 16/90 → 32/90 → 35/90 sequence shows that the selected tier has already captured most of the immediate success available from maximum guidance.

The Always-L5 training control keeps the same 197 accepted states, the same 197 updates, the same replay set, and the same loss weights, replaces every selected repair with L5, and evaluates the unguided Student. It is distinct from the submitted no-hierarchy (binary ABSTAIN/HELP) ablation.

Frozen-curriculum training armTeaching targetUnguided Student macro (%)
Replay-onlyNone48.1
Always-L5L5 on every accepted state50.8
Selected first-passing (Full)First-passing tier52.4

Selected-tier Full achieves the highest observed unguided Student macro in this matched training comparison: 52.4%, versus 50.8% for Always-L5 and 48.1% for Replay-only. The two evaluations answer different questions: L5 helps the currently guided policy finish more of the audited continuations, whereas first-passing supervision trains a stronger unguided Student in the matched curriculum. Immediate guided success therefore does not by itself rank the value of a training target. The selected curriculum gains 1.6 points over Always-L5 while using less specific guidance on 38 of the 45 audited states. The operational claim is “first-passing intervention under the specified ladder and verification protocol.”


W3. What is the full accepted-tier distribution across rounds and datasets?

Response. The accepted training signal is distributed across the ladder: L0–L2 provide 107/197 boundaries (54.3%), while L5 provides 22/197 (11.2%). The complete distribution below reports accepted boundaries, rather than proposed levels. The 45-state audit is the Round-1 row.

RoundL0L1L2L3L4L5TotalL3–L5 share (%)
1111119674548.9
22121311765147.1
33131211755145.1
43141211645042.0
All rounds9504842262219745.7

Construction uses 12,000 questions from 2WikiQA and MuSiQue (6,000 each). HotpotQA is used for evaluation, not for construction or replay. The source-and-round accepted-tier breakdown is:

RoundSourceL0L1L2L3L4L5Total
12WikiQA06553322
1MuSiQue15643423
22WikiQA16764327
2MuSiQue16653324
32WikiQA17653224
3MuSiQue26664327
42WikiQA27653225
4MuSiQue17663225
All2WikiQA4262421131098
AllMuSiQue5242421131299

Accepted L5 decreases from 7/45 (15.6%) in Round 1 to 4/50 (8.0%) in Round 4, while the number of accepted boundaries remains 45–51 per round. L3–L5 share also decreases from 48.9% to 42.0%. The two sources have similar aggregate L3–L5 shares: 44/98 for 2WikiQA and 46/99 for MuSiQue.

These counts show that training does not collapse onto the maximally specific route. Whether a concrete bridge or route is answer-specific is assessed separately through public-evidence annotation and disjoint transfer evaluation below.


From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.