- Rating
- 3(+1)
- Confidence
- 4(+2)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces RIVO, a plug-in controller designed for long-horizon deep-research agents. The core idea is to keep the main Host model frozen while using a lightweight 0.6B controller. This controller leverages confidence, evidence attention, and attention convergence to dynamically choose between STOP, VERIFY, CONTINUE, and REROUTE actions. When a reroute is triggered, RIVO applies a sparse Top-K logit correction to steer the Host toward an alternative search direction. Additionally, the authors propose BrowseCore, a compact dataset paired with a retrieval corpus. Experiments conducted across two Host models and three benchmarks demonstrate consistent improvements over existing trajectory-control baselines.
Critical strengths
- The collaboration between a small controller and a larger, frozen generator is a very appealing direction for deep-research systems. Decoupling trajectory control from language generation is practical and could allow various Host models to benefit without requiring full, expensive retraining.
- RIVO shows consistent improvements across two different Host models and three benchmarks. I also appreciate the ablation studies, which clearly show that both deciding when to reroute and learning how to reroute contribute meaningfully to the final performance.
Critical weaknesses
- Potential test-set leakage in BrowseCore evaluation is my primary concern. The same 359 questions are referred to as a "test set" in the main text but as a "development set" in the appendix. More problematically, these exact 359 questions are used to select hyperparameters (β, Top-K, λ, and τ), and the resulting score (59.30) is reported as a main result. Without a strictly held-out test set, the current BrowseCore results are highly inflated and unconvincing.
- RIVO's pipeline is actually quite heavy: it requires generating provisional answers, extracting attention from the final four Host layers, running an auxiliary 0.6B model, and prompting the Host to rewrite rerouting hints. Yet, there is no analysis of latency, throughput, memory footprint, or total training cost. Furthermore, while the aggregate ablations are nice, they don't explain when or why the system succeeds or fails at different stages. There is a noticeable lack of trajectory-level case studies and intervention statistics.
Questions for authors
- Can the authors clarify whether the 359-question split is a dev set or a test set? Ideally, could you select all hyperparameters on a separate development split and evaluate once on a strictly untouched test set to validate the 59.30 score?
- What is RIVO’s end-to-end inference and training overhead relative to the original Host and the strongest baselines? Additionally, could you provide stage-wise action statistics and a few representative successful/failed trajectories to help clarify how the controller actually drives the performance gains?
6 comments
Thank you for your response. I will keep my score at 2.
As presented, RIVO requires access to the host model’s attention weights and the ability to modify token logits during decoding. These requirements limit its applicability to APIs that expose only generated text or read-only token log probabilities. The evaluation covers two host models, and it remains unclear whether the reported gains would generalize to a broader range of models.
Have a wonderful day
dataset splitation
We thank the reviewer for raising this concern. We would like to clarify that BrowseCore contains 359 development questions and 359 separate test questions. The inconsistency in the appendix was an editorial oversight. Importantly, all hyperparameters (β, Top-K, λ, and τ) were tuned exclusively on the 359-question development set, while the reported main result (59.30) was obtained on the separate, held-out 359-question test set. Thus, no test questions were used for hyperparameter selection. To ensure transparency and reproducibility, we have publicly released the complete dataset, including both the development and test splits, which has already received 176 downloads.The complete BrowseCore dataset, including the training, development, and test splits (train.json, dev.json, and test.json), is publicly available at the anonymized link under BrowseCore/ourdata/data/.
RIVO's pipeline is actually quite heavy
We thank the reviewer for this valuable suggestion. We provide additional efficiency measurements and trajectory-level analyses on the 359 BrowseCore test questions. Inference efficiency. RIVO achieves an amortized inference time of approximately 40 seconds per question on an H200 GPU under concurrent execution. The provisional-answer and attention probes each account for approximately 13% of wall-clock time, while reroute retrieval adds less than 1%. The additional sidecar memory footprint is at most approximately 24 GB. Training cost. RIVO keeps the 4B Host frozen and trains only the lightweight 0.6B plugin, avoiding full-parameter fine-tuning of the Host. Stage-level analysis. RIVO triggers rerouting for 266 out of 359 questions, performing 483 reroutes, of which 471 successfully apply logit correction. Case studies. QID 2023 demonstrates successful evidence retrieval after rerouting, while QID 8624 illustrates premature stopping with high confidence in an incorrect answer. QID 371 shows that correct evidence can be retrieved but not committed due to insufficient stopping confidence. We will incorporate these efficiency measurements, intervention statistics, and representative case studies into the revised manuscript to better explain when and why RIVO succeeds or fails.