- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes MIST-OPD, a framework for training search agents using privileged information (PI). A Guide proposes graded interventions based on reference answers, and the accepted interventions are distilled into the Student. Experiments on Qwen3.5 show improvements over several baselines on search and code benchmarks.
Critical strengths
- The paper explores an interesting direction for using PI to improve search agents.
- The paper provides experiments across multiple benchmarks and model scales, with several component ablations.
Critical weaknesses
- Unclear contribution of the limited supervision and self-evolution.
The method achieves a large improvement over the untrained agent using only 197 accepted states, along with replay supervision, with a learning rate of . Moreover, Round 1 already achieves most of the final performance. The authors should better explain these results and the additional contribution of self-evolution.
- Insufficient justification for the complicated method design.
The method introduces many components but fails to provide clear justification for them. The current ablations are insufficient to establish the usefulness of each component. More detailed ablations and hyperparameter sensitivity studies are needed.
- Unclear experimental and implementation details.
The paper does not specify the evaluation budget or thinking-mode configuration, making it difficult to assess the fairness of the comparisons with other methods. It is also unclear how intermediate public states are selected from Student rollouts for Guide intervention. These details are important for understanding MIST-OPD.
- Missing training-cost comparison.
MIST-OPD requires substantial additional computation for Guide proposals, continuations, and Guide updates. The provided BoN control matches continuation calls only, not proposal generation or Guide updates. Wall-clock time and a compute-matched baseline should be provided.
- Unclear presentation.
The training procedure is difficult to follow. A clear end-to-end algorithm description would improve readability.
Questions for authors
- Could the authors explain the large gains from relatively few accepted states?
- Could the authors provide more detailed loss ablations and hyperparameter sensitivity results?
- Could the authors clarify the experimental details, particularly the evaluation budget, thinking-mode configuration, and intermediate state selection?
- Could the authors report wall-clock time for MIST-OPD and the baselines?
11 comments
W1 / Q1. How can 197 accepted states yield substantial gains, and what does self-evolution add after Round 1?
Response. We clarify the supervision associated with the 197 accepted repairs. Across four rounds, these records yield 197 Student updates. Each update combines one accepted repaired trajectory with eight source-balanced replay records, totaling 1,576 replay-record uses.
Two supervision scopes are involved. Trajectory-level cross-entropy (CE) supervises assistant output tokens throughout each accepted repaired trajectory, while repair-specific supervision is restricted to the local intervention span, capped at 256 tokens. This trajectory-level CE, with weight 0.2, was already part of the reported full-method training. The 8,192-token limit applies to each serialized training sequence, including conditioning context; question and tool-observation tokens are not CE targets. Replay CE is capped at 2,048 target tokens per replay record. The 41,736 repair-span positions and 36,918 repair-local KL positions are counts of token prediction positions accumulated over the 197 repair records. The span count measures the bounded repair segments; the KL count measures positions contributing to the local distillation loss. They are not disjoint token pools to be added together, and neither is the total number of trajectory-level CE targets. We will align the objective definition in Sec. 2.3, the training algorithm in Appendix A, the local-target description in Appendix C, the optimization settings in Appendix E.2 and Table 12, and the baseline budget descriptions with these supervision scopes.
| Quantity | Submitted training run |
|---|---|
| Accepted repaired trajectories / Student optimizer steps | 197 / 197 |
| Trajectory-level CE supervision | Assistant outputs throughout each accepted repaired trajectory |
| Maximum serialized training sequence length | 8,192 tokens, including conditioning context |
| Local repair-span cap | 256 target tokens per repair |
| Repair-span target-token positions, summed over 197 repairs | 41,736 |
| Token positions contributing to repair-local KL, summed over 197 repairs | 36,918 |
| Replay records / CE-supervised tokens | 1,576 / 1,842,116 |
| Replay-supervision cap | 2,048 tokens per record |
| Parameters | Full-parameter Qwen3.5-9B Student and Guide; no LoRA |
| Optimizer / learning rates | AdamW; Student 2×10⁻⁷, Guide 1×10⁻⁷; norm clip 1.0 |
| Precision | FP32 parameters and AdamW states; BF16 autocast |
The contribution of training on accepted repaired records is measured with the existing controlled curriculum:
| Frozen-curriculum control | Macro accuracy (%) |
|---|---|
| Replay-only | 48.1 |
| No-gate + Full-OPD | 49.6 |
| Full repair teaching + replay | 52.4 |
| Full − Replay-only | +4.3 pp |
Full exceeds Replay-only by 4.3 points and No-gate by 2.8 points. The first contrast measures the contribution of the repaired-record training branch beyond replay, including trajectory CE and repair-specific teaching. The No-gate contrast retains the objective structure and update schedule while changing outcome-based state/tier selection. It measures the value of screened training records. The matching is in initialization, objective structure, replay, and update schedule; the repair-span figures are local supervision statistics. The submitted online result of 52.8% is separate; the two controlled contrasts consistently use Full = 52.4%.
For self-evolution, the evolving and Frozen Guide arms give:
| Checkpoint | Evolving Guide: Student macro (%) | Frozen Guide: Student macro (%) |
|---|---|---|
| Shared initialization | 42.7 | 42.7 |
| Round 1 | 50.4 | 50.3 |
| Round 2 | 51.3 | 50.8 |
| Round 3 | 52.0 | 51.1 |
| Round 4 | 52.8 | 51.3 |
| Round 4 − Round 1 | +2.4 | +1.0 |
After Round 1, the evolving arm gains 2.4 points while Frozen gains 1.0 point. Thus, later adaptation contributes beyond continued Student training, even though the first round produces a substantial initial gain.
The fixed-state control provides complementary evidence. Starting from the same Round-3 Student on 228 Round-4 states, guided continuation succeeds on 49/228 states without guidance, 56/228 with Frozen Guide, and 71/228 with Updated Guide. Updating the Student using each Guide's accepted set gives 51.9% versus 52.5% guide-free macro accuracy. This downstream comparison measures the end-to-end value of the accepted sets, including differences in their size; the common-state continuation comparison directly measures guidance quality. Together, these results support the incremental value of adapting the Guide to the evolving Student.
W2 / Q2. Do the ablations justify the components and isolate their contribution?
Response. The matched-objective control shows a 2.8-point benefit from repair selection while keeping controlled distillation and replay fixed. It complements the 4.3-point Full-versus-Replay-only comparison and the submitted component ablations.
Controlled OPD (on-policy distillation) constructs an answer-blind Student target from the lagged Guide’s gold-answer, placebo, and public-only views, using an anchored residual and private-token projection. Trajectory-level CE supervises assistant outputs in the repaired record, alongside the bounded repair-specific objectives and replay. KL denotes the distributional distillation loss. No-gate + Full-OPD retains the objective structure and changes outcome-based repair selection.
No-gate + Full-OPD samples the Guide's predicted legal tier within each source×round stratum without consulting outcome-based acceptance. It preserves residual projection, KL/CE weights, replay, initialization, and the 197-update schedule, as well as format and explicit-answer checks. Full and No-gate contain 41.7k/42.1k target-token positions in their bounded repair spans and 36.9k/37.4k positions contributing to repair-local KL; all three arms use 1.84M replay CE targets. These statistics distinguish repair-local loss positions from replay supervision. Trajectory-level CE additionally supervises assistant outputs across each repaired record. The controlled comparison holds initialization, objective structure, replay, and the update schedule fixed; these local-span figures do not establish equality of total trajectory-level CE target counts.
| Shared frozen-curriculum condition | Outcome-based selection | Controlled OPD | Macro accuracy (%) |
|---|---|---|---|
| Replay-only | Not used | Disabled | 48.1 |
| No-gate + Full-OPD | Disabled | Enabled | 49.6 |
| Full | Enabled | Enabled | 52.4 |
48.1 → 49.6 → 52.4 separates improvement beyond replay from the further benefit of screened repairs. The shared curriculum isolates repair selection while preserving the distillation objective, replay, and update schedule. The +2.8-point contrast measures the value of selecting which states and tiers provide the teaching signal.
| Submitted component comparison | Full (%) | Ablated (%) |
|---|---|---|
| No placebo | 52.8 | 50.4 |
| No hierarchy (binary ABSTAIN/HELP variant) | 52.8 | 50.2 |
| Gold-only anchored distillation | 52.8 | 51.9 |
| Frozen Guide | 52.8 | 51.3 |
No-gate and Replay-only isolate screening and the joint repair objective. They do not isolate trajectory-level CE from KL, and changing R does not replace a loss-term ablation. The submitted no-hierarchy ablation is the binary ABSTAIN/HELP variant; it is distinct from the Always-L5 training control. The CE and KL knockouts keep the Full arm’s 197 accepted states, 197 updates, replay, and remaining weights. Always-L5 replaces the repair targets on those same states. No-gate and Replay-only are repeated below as reference controls: No-gate matches the source×round allocation and update count while changing state/tier selection; Replay-only removes the repair objectives.
| Frozen-curriculum knockout | Disabled term | Macro accuracy (%) | Full − ablated (pp) |
|---|---|---|---|
| Full | — | 52.4 | — |
| No trajectory-level CE | Trajectory-level CE term | 51.2 | +1.2 |
| No KL | 50.6 | +1.8 | |
| Always-L5 training | Selected-tier target (replaced by L5) | 50.8 | +1.6 |
| No-gate + Full-OPD | Outcome-based selection | 49.6 | +2.8 |
| Replay-only | All repair objectives | 48.1 | +4.3 |
Removing KL costs 1.8 points and removing trajectory-level CE costs 1.2 points; both remain above No-gate and Replay-only. Selected-tier Full has a 1.6-point advantage over Always-L5 in this matched frozen-curriculum comparison.
| KL weight | Setting | Macro accuracy (%) |
|---|---|---|
| 0.125 | submitted | 51.9 |
| 0.25 | Submitted | 52.4 |
| 0.50 | submitted | 52.1 |
The trajectory-level CE, replay, hierarchy, and public-help weights remain fixed at 0.2, 1.0, 0.5, and 0.1, respectively. The submitted KL coefficient has the highest observed macro among these three settings; halving or doubling it changes macro by at most 0.5 points. The matched controls support the contributions of screening, trajectory-level CE, and KL, while the local coefficient sweep tests sensitivity to the KL weight.
Response to Reviewer ZaG5
Thank you for asking us to clarify supervision volume, the incremental value of self-evolution, and the role of the individual components. We address these with matched controls that separate repair selection, replay, and Guide adaptation, alongside the complete compute accounting.
Definitions and evaluation scope. Results concern Qwen3.5-9B unless stated otherwise. Macro accuracy is the unweighted mean over BrowseComp, BrowseComp-Plus, FRAMES, GAIA-text, HLE-text, and HotpotQA; pp denotes absolute percentage points. π-Distill is the privileged-distillation baseline. The Frozen Guide control keeps the teaching policy fixed while continuing Student training. R is the number of matched construction continuations per arm. The frozen-curriculum controls share fixed construction artifacts and are distinct from the submitted online system. Scoring and deployment settings are specified in W3/Q3 below.