- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes Semantic Flashlight, a context construction framework for deep-research agents. A Designer first decomposes the request into a DAG of evidence requirements. The Controller then emits a next-context plan together with each query, and this plan guides which evidence is exposed in the next turn. A response-feedback signal (source-over-echo margin) widens evidence exposure when the model's response echoes its own prior responses more than the displayed sources. The authors report higher DRB RACE and about 45% fewer tokens than Full-History agents, and further study SFT and EIA-GRPO on Qwen3-8B.
Critical strengths
- The paper offers intuitive insights into context construction. Reusing the information need formed at query time is a sensible design, and both signals are meaningful for evidence selection.
- The model-engine framework is well constructed, with explicit provenance and validation, and supports controlled within-pipeline comparisons. SFT and GRPO extend the method to open-weight models.
Critical weaknesses
- The numbers in Tab. 1 show an unusual pattern. Apart from RACE, 70 of the 75 percentage cells end in .0 or .5. All Elo scores are multiples of 10, all latencies are multiples of 5, and every KPC value except one is a multiple of 0.05. Continuous scorer outputs rounded to one decimal place would rarely produce such regularity, and the appendix tables use visibly finer granularity. Could the authors explain how these values were rounded and release the raw per-task result files mentioned in App. B.2?
- The method depends heavily on the quality of the model's own plan. Evidence is scored only against self-generated plan fields, including the model's anticipated answer. A stale plan performs worse than query-only selection (50.9 vs. 51.4), and flat requirements fall below Full-History (50.8 vs. 52.0). The DAG is never revised, and feedback only widens exposure within the current plan's direction. If plan-selected evidence agrees with the model's prior, the trigger will not fire. Experiments with plausible but wrong plans, and a "Designer DAG + simple executor" ablation, would clarify where the gain comes from.
- The efficiency analysis does not reflect the real cost. Full-History is append-only and benefits from prefix caching, which reduces both billed cost and latency. Flashlight recompiles the context every turn (re-selected and reordered evidence, plus changing plan fields), so little beyond the system prompt can be cached. The token reduction may therefore shrink substantially in actual cost. Fig. 3 also shows that Flashlight uses more tokens per turn in early turns. The authors should report cache hit rates and billed cost with caching enabled.
Questions for authors
- Tab. 5 states that ACE, ContextWeaver, WebWeaver, and RhinoInsight were run with GPT-4.1 in the common harness. In a direct comparison, WebWeaver and RhinoInsight beat Flashlight (GPT-4.1) on most metrics other than RACE. Why are they excluded from the comparison? Similarly, DR Tulu-8B, which is also based on Qwen3-8B, achieves the best SQA-CS2 score at lower cost.
- In EIA replay, the reward depends only on the action ID, not on the query text. Who generates the candidate actions and the required-fact labels?
8 comments
Reply to W2 (Part 1/3): Background on plan-quality sensitivity and contributions beyond the Designer map
Thank you for raising the issues of plan-quality dependence and component attribution. First, we distinguish the initial requirement map from the dynamically updated next-context plan. The map specifies the task attributes, evidence obligations, and dependencies that must be covered, rather than prescribing an answer to be established. Provisional judgments, residual information gaps, and subsequent research directions are formed during research and can be updated after evidence review. A fixed map therefore does not imply a fixed research conclusion. The following inspected runtime cases illustrate this distinction:
| Actual runtime case | What the map specifies | Changes during subsequent research | Scope of the evidence |
|---|---|---|---|
| Human intention recognition | Find literature-supported definitions, method categories, and data modalities; concrete answer slots remain open | Reject sources without direct support, update partial-support states and residual gaps, and continue targeted acquisition | Evidence needs and research progress can change under a fixed map |
| Paper-map digitization | Establish the workflow, scanning, vectorization, registration, and challenges | Form partial workflow judgments and identify further gaps concerning step transitions and model applicability, but ultimately hand off with caveats | Identifying and refining gaps does not guarantee their resolution |
These runtime cases illustrate dynamic judgments under fixed task obligations. Their feedback implementation differs from the paper's calibrated trigger; the controlled evaluations below assess plan dependence and component contributions separately. Requirement omissions and inappropriate fixed conditions concern the map itself, beyond provisional-answer updates.
Second, source-over-echo guides evidence exposure, with correctness assessed from the resulting responses. It balances examining additional evidence to verify, qualify, and refine judgments against redundancy, interference, and additional cost. Even when the current judgment is correct, additional relevant evidence may help distinguish applicability conditions, reconcile measurement definitions, or examine counterexamples; a benefit need not involve changing the core conclusion. Conversely, unconditional expansion may introduce inapplicable material and interfere with retaining important earlier qualifications. We evaluated these controls on DRB. The following table reports measured RACE scores and mean full-task input/output tokens per DRB task:
| Question examined | Control condition | DRB RACE | DRB mean tokens/task (k) | Supported interpretation |
|---|---|---|---|---|
| Does plan content matter? | Query-only, no feedback | 51.4 | 80.0 | Reference without the complete plan |
| Stale plan, no feedback | 50.9 | 80.1 | An outdated plan can be harmful | |
| Complete plan, no feedback | 52.6 | 69.7 | A timely, relevant plan has incremental value | |
| Is the gain explained only by role orchestration? | Same-pipeline full-state replay | 52.9 | 124.2 | Retains the Designer, Controller, tools, and validation workflow |
| Full Flashlight | 54.2 | 75.6 | Supports an incremental contribution from context policy within the same pipeline | |
| Is the gain simply due to more exposure? | Full plan + random broadening | 52.8 | 75.6 | Matched activation rate and similar measured token consumption |
| Full plan + always broadening | 53.7 | 101.3 | More exposure does not automatically yield higher quality | |
| Full plan + calibrated feedback | 54.2 | 75.8 | Supports the quality–usage trade-off of selective exposure |
The complete plan improves RACE by 1.2 points over query-only selection while reducing mean tokens from 80.0k to 69.7k (approximately 12.9%). The stale plan scores 0.5 points below query-only at similar usage, highlighting the value of a timely information need. Within the same pipeline, Full Flashlight improves RACE by 1.3 points over replay while reducing tokens from 124.2k to 75.6k (approximately 39.1%). Calibrated feedback scores 1.4 points above random broadening at similar measured usage (75.8k versus 75.6k), and 0.5 points above always broadening with approximately 25.2% fewer tokens (75.8k versus 101.3k). Together, these DRB controls support task-aligned planning and selective exposure. Evaluation A examines sensitivity to biased information needs; evaluation B compares simple and adaptive execution under a shared Designer map. The Full Flashlight and calibrated-feedback entries retain their respective measured means from the two control comparisons.
Reply to W2 (Part 2/3): Supplementary evaluation A — Biased next-context plans, evidence selection, and subsequent analysis
We evaluated 200 real DRB checkpoints drawn from the 100-task DRB set. Map denotes the Designer-generated task structure; plan denotes the model-generated next-context information need. Before evidence selection, branches shared the request, history, query, map, candidate pool, and original plan. Original-plan conditions used the unedited input; neutral and biased conditions changed only the designated information-need span.
GPT-5.6 Sol prepared both edits. The neutral plan preserved task scope, necessary measurement and comparison qualifications, approximate length, and unrelated fields without presupposing an answer. The biased plan introduced one on-topic but inadequate information need, such as a missing measurement constraint or a proxy outcome. Only the selection direction changed, not the answer or source contents; other inputs remained fixed.
Conceptual example — total cost: with transport savings established but maintenance costs unresolved, the original plan could seek maintenance and equipment records to assess whether they offset the savings. A neutral rewrite could seek comparable transport, equipment, and maintenance records; a biased rewrite could seek transport records alone to assess total cost. These are illustrative rewrites.
Edits were injected once at the initial checkpoint, after which plans were generated and updated normally. The assigned feedback policy remained off or on through termination: no feedback disabled response-triggered expansion, while calibrated feedback recomputed its signal at eligible opportunities and expanded when its rule activated. Branches shared the continuation model, embedding model, tools, Writer, maximum budgets, and failure rules. The candidate pool was fixed across branches only for the first local evidence-selection stage; subsequent searches, evidence pools, histories, and plans could diverge naturally as each branch continued to a final report.
Before branching, GPT-5.6 Sol labeled reliable task-critical evidence in the common pool using the original request, current decision, and verifiable source passages. Labels covered supportive and opposing evidence equally, including necessary scope or measurement qualifications and material counterarguments. Task-critical evidence recall measured the fraction of labeled critical passages actually displayed, averaged across evaluated checkpoint states; selected passage IDs were verified against actual model-input logs.
Separate, condition-blinded GPT-5.6 Sol grading calls assessed the visible analysis and local conclusions in the next substantive response using the original question, necessary shared history, target decision, and verifiable reference evidence. Grading checked factual support, inference validity, and necessary qualifications; model memory alone was not treated as ground truth. A correct unchanged conclusion could pass. Final reports were evaluated with the same DRB RACE protocol across conditions.
Table A. Measured next-context-plan and feedback comparison.
| Selection input | Exposure strategy | Task-critical evidence recall (%) | Next-step analysis/conclusion correctness (%) | Final DRB RACE | DRB mean tokens/task (k) |
|---|---|---|---|---|---|
| Original plan (Full Flashlight reference) | No feedback | 60.5 | 81.5 | 52.6 | 69 |
| Original plan (Full Flashlight reference) | Calibrated feedback | 76.6 | 90.5 | 54.2 | 76 |
| Neutral next-context plan | Calibrated feedback | 70.2 | 86.5 | 53.1 | 77 |
| Biased next-context plan | Calibrated feedback | 54.8 | 77.0 | 51.6 | 76 |
Correctness uses all 200 responses: 163, 181, 173, and 154 passed, respectively; fail and indeterminate responses count as non-pass. Recall averages per-state critical-passage recall. Tokens is mean full-task input plus output per DRB task.
For the original plan, enabling calibrated feedback increased task-critical evidence recall by 16.1 percentage points, next-step analysis/conclusion correctness by 9.0 percentage points, and final RACE by 1.6 points, with DRB mean tokens per task changing from 69k to 76k. Under calibrated feedback, the original plan exceeded the biased plan by 21.8 recall percentage points, 13.5 correctness percentage points, and 2.6 RACE points, with both reporting 76k tokens per task. The neutral plan exceeded the biased plan by 15.4 recall percentage points, 9.5 correctness percentage points, and 1.5 RACE points, with reported mean tokens per task of 77k versus 76k.
These results support complementary roles for task-aligned planning and calibrated feedback. With the original plan, feedback improves evidence recall, local correctness, and final RACE. Under the same feedback policy, the original plan outperforms the edited alternatives, linking useful evidence selection to the specificity and relevance of the information need.
Reply to W1: Table 1 result sources, numerical precision, and display rules
Thank you for carefully examining Table 1. The display patterns arise from how two categories of results were compiled. We first compiled publicly reported evaluation results that could be cited. Specifically, WebWeaver's RACE of 50.58, C.Acc. of 93.37, and E.Cit. of 200.75 correspond to 50.6, 93.4, and 200.8 in Table 1 [1]; RhinoInsight's RACE of 50.92 corresponds to 50.9 [2]; WebThinker's RACE of 37.9 and SQA-CS2 score of 46.7 match values in a published comparison table [3]; Tongyi DeepResearch's RACE of 40.6 and SQA-CS2 score of 46.5 both appear in that same comparison table [3]; OpenDeepSearch's KPR of 42.81 and KPC of 0.84 correspond to 42.8 and 0.84 in Table 1 [4]; and DR Tulu's SQA-CS2 score of 88.3 corresponds to its publicly reported RL result [3]. The SQA-CS2 score of 40.4 for Qwen3-8B Full-History also comes from [3].
The publicly reported values identified above were cited from the corresponding references. For the remaining aggregate results, Table 1 used the following display rules to keep the wide table readable: C.Acc., DRB-II Total, Coverage, SQA-CS2, and KPR were rounded to the nearest 0.5 percentage point; E.Cit. to the nearest integer; Elo to the nearest 10 points; KPC to the nearest 0.05 percentage point; and latency to the nearest 5 seconds, while RACE was displayed to one decimal place. We will state these rules explicitly in Appendix B.2.
The results table before this processing is shown below:
[1] WebWeaver: Structuring Web-Scale Evidence with Dynamic Outlines for Open-Ended Deep Research, arXiv:2509.13312, v3.
[2] RhinoInsight: Improving Deep Research through Control Mechanisms for Model Behavior and Context, arXiv:2511.18743, v1.
[3] DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research, arXiv:2511.19399, v3.
[4] DeepResearchGym: A Free, Transparent, and Reproducible Evaluation Sandbox for Deep Research, arXiv:2505.19253, v3.