- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes StreamRSI: Freezing the Qwen2.5-VL-7B video question-answering model and training a 27B-parameter research agent to improve the program that runs during inference of the question-answering model. The main contribution was the design of A controlled evidence-credit protocol for diagnostic experiments: restoring the candidate program and archive before the same experiment, comparing with subsequent research branches (A and B ') that could not see the diagnostic results, and comparing with branches (A and C) that used the saved diagnostic budget for direct search. The above estimates are supervised through preference optimization for research decisions. The paper reports an Overall score of 65.50 for OVO-Bench and an accuracy rate of 80.63% for StreamingBench real-time video understanding. In the ablation experiment with eight random seeds, each seed lasting 20,000 GPU seconds, the final quality of the full objective was higher than that of the scheme that only used outcome-only supervision and a single credit term. Other experiments analyzed the differences between pre-experiment state restoration, archive-free transfer, frame selection, and evidence retention and reading.
Critical strengths
- The paper focuses on how the experiment contributes to subsequent improvement, not just whether the experiment improves the score in the moment. During the study, an experiment that does not improve accuracy may also rule out false explanations and provide directions for the next improvement. This paper tries to incorporate this kind of information value into the training objective, which has research significance.
- The paper sets up multiple controls to test whether performance improvement comes from the learning of the research agent. The authors compared the follow-up study process with and without seeing the diagnosis, and compared different training methods with the same budget and eight paired random seeds. In addition, the authors cleaned out the existing programs and archives to conduct transfer experiments, trying to test whether the research agent can find effective solutions on new tasks. These designs provide positive evidence for the effectiveness of the method.
Critical weaknesses
- The paper defines the information value of the experiment and the net value after considering the cost respectively, and shows the performance advantages of using the two together through ablation experiments. However, existing analyses mainly stop at final score comparison and have not yet fully illustrated which research decisions are improved by the two signals respectively and how they complement each other. Thus, the experiments support that the combination approach works, but the explanation for why it works is still more limited.
- Table 6 shows the advantage of the trained research agent over the untrained model on the new task, but there is no research agent trained only on the final results as a comparison. Therefore, the existing results support the transfer ability of the trained research agent, but are not sufficient to prove that the credit mechanism proposed in this paper can bring better transfer effects than normal training.
Questions for authors
- When an experiment can provide useful information, but its cost results in a final performance lower than that of a direct search, how does the complete method determine whether to conduct this experiment? Please explain how the two types of credit signals jointly influence the decision in this situation.
- Can you give a case study of an experiment that didn't immediately improve accuracy, but helped the research agent eliminate incorrect explanations and get better results later? This case will help to understand the role of the core idea of the paper in the actual research process.
- Under the same transfer conditions as in Table 6, how does the proposed method perform compared to the research agent trained only on the final results? This contrast helps to determine whether the migration gain comes from the credit mechanism proposed in this paper.
2 comments
Q2. Can you give a case of an experiment that did not immediately improve accuracy but helped the research agent rule out a wrong explanation and reach a better result later?
Case A in Table R1 is one. The current program was recent-4, and the researcher had two explanations for its errors: too few recent frames (H1) or a missing historical event (H2). It ran a controlled probe that changed recent-4 to recent-8 with the layout and prompt fixed, at a cost of 120 GPU-s, and recorded beforehand what each explanation predicts. The support score fell by 2.78 points. The extra recent frames did not fix the main errors, and the failing queries needed evidence from earlier events, which ruled out H1.
In branch A the researcher saw this result, reopened the diagnosis, and switched to query-relevant historical retrieval. Its final program uses 4 recent and 4 relevant historical frames and scores on the credit panel. The branch that ran the same probe without seeing the result reaches 78.59, and direct search with the saved budget reaches 79.94. The probe therefore has and and became a preferred action. Outcome-only training would have seen only the 2.78-point drop and labeled the probe negative.
W2 / Q3. Under the transfer conditions of Table 6, how does the method compare with a researcher trained only on final outcomes?
We ran the outcome-only researcher of Table 3 under the Table 6 protocol: weights transferred, candidate code, diagnosis archives and task memory cleared, three seeds per target.
Table R2. Weights-only transfer, mean over three seeds.
| Transferred weights | T1 | T2 | T3 |
|---|---|---|---|
| None (base search) | 78.83 | 77.87 | 76.84 |
| Outcome-only researcher | 79.18 | 78.20 | 77.10 |
| StreamRSI full | 80.32 | 79.08 | 77.57 |
| Full − Outcome-only | +1.14 | +0.88 | +0.47 |
| Paired 95% CI | [+0.64, +1.64] | [+0.63, +1.13] | [+0.10, +0.84] |
Outcome-only training gives a small transfer gain over the base search (+0.35, +0.33 and +0.26). StreamRSI is above the outcome-only researcher on every target, and every paired interval excludes zero. Relative to the base search, 77% of the weight-transfer gain on T1, 73% on T2 and 64% on T3 goes beyond what outcome-only training provides, so most of the transfer gain in Table 6 comes from the credit signals.
Thank you for the review. Below we explain how the two credit signals enter the researcher's decisions, give a case of an informative negative result, and add the outcome-only researcher to the transfer comparison of Table 6.
W1 / Q1. Which research decisions does each signal improve, how do they complement each other, and what does the full method do when an experiment is informative but still loses to direct search?
The two signals supervise different decisions, because each produces its own preference pairs:
Each decision state is forked into five branches. For each of two candidate diagnostics D0 and D1 there is a branch A, which runs the diagnostic, restores the program and archive, and continues with the result visible, and a branch B′, which does the same with the result hidden. A fifth branch C skips the diagnostic and searches directly from the same state. Net pairs compare Ai with C, and the evidence pair compares D0 with D1 by QAi−QBi′. The two kinds of pairs are combined only in the loss, L=31(Lnet+Levidence+Ledit), where Ledit is the outcome preference over program edits. A pair is kept only if the 95% bootstrap CI of the difference excludes zero (8 paired continuation seeds, 5,000 resamples over seeds and video clusters). The training pool has 144 net pairs and 80 evidence pairs.
With only one of the two signals, one of these decisions gets no supervision. Evidence credit alone teaches which probe is the most informative, but not that a probe can cost more than its information is worth. Net credit alone teaches when to probe, but gives no basis for choosing between probes. The full objective supervises both, and Table 4 shows the result: it is 1.20 [0.89, 1.51] above net credit and 2.02 [1.71, 2.34] above evidence credit.
If an experiment is informative but loses to direct search (Vevidence>0, Vnet<0), the net pair at that state prefers the direct edit. The experiment's evidence value may still rank it above a less informative probe, but the net pair trains the researcher not to run it from this state. Case B in Table R1 is an example.
Table R1. Two diagnostics from the source search. The immediate change is on the support panel; QA, QB′, QC are on the credit panel.
In Case B the probe is informative (Vevidence=+1.19): branch A uses the localized interval to improve historical sampling. The probe, however, costs 800 GPU-s. Branch C spends that budget validating more candidates and finishes higher, so Vnet=−1.35.