acceptodds
Under review as a conference paper at ICLR 2027

Semantic Flashlight: Plan-First Context Construction with Response-Adaptive Evidence Exposure for Deep Research

Abstract

Deep-research agents repeatedly search, inspect evidence, and refine their reasoning, causing their working context to grow with research depth. Existing context-management methods often decide what to retain only after the interaction history has accumulated, requiring a selector to reconstruct the information need behind earlier search decisions. We introduce Semantic Flashlight, a plan-first and response-adaptive context construction framework built around two complementary signals. First, during query formulation, the model already identifies an unresolved information gap and anticipates what evidence is needed next. Flashlight externalizes this information need as a next-context plan, which guides evidence selection for the following decision. Second, the model's response provides a signal about whether the current evidence is being effectively used. Across real research trajectories, higher-quality outcomes are associated with stronger alignment to the current context than to prior model responses. Flashlight therefore uses response feedback to adapt the breadth of evidence exposure, broadening raw evidence when response echo dominates source alignment. A dependency-aware evidence contract carries these two signals across planning, acquisition, review, and synthesis, while the Context Engine enforces provenance and bounded context construction. Across diverse deep-research benchmarks with two closed-source models, Flashlight improves report quality while reducing LLM token consumption by up to 45.6% relative to Full-History, without requiring an auxiliary LLM context-selection call. Optional role-conditioned supervised fine-tuning and reinforcement learning with our Effective Information Acquisition (EIA) reward extend the workflow to an open-weight model. Together, these results suggest that context construction can be integrated into the research decision itself rather than treated as post-hoc transcript compression.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Deep Research Agents, Context Management, Adaptive Evidence Retrieval

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rrbDOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes Semantic Flashlight, a context construction framework for deep-research agents. A Designer first decomposes the request into a DAG of evidence requirements. The Controller then emits a next-context plan together with each query, and this plan guides which evidence is exposed in the next turn. A response-feedback signal (source-over-echo margin) widens evidence exposure when the model's response echoes its own prior responses more than the displayed sources. The authors report higher DRB RACE and about 45% fewer tokens than Full-History agents, and further study SFT and EIA-GRPO on Qwen3-8B.

Critical strengths
  • The paper offers intuitive insights into context construction. Reusing the information need formed at query time is a sensible design, and both signals are meaningful for evidence selection.
  • The model-engine framework is well constructed, with explicit provenance and validation, and supports controlled within-pipeline comparisons. SFT and GRPO extend the method to open-weight models.
Critical weaknesses
  • The numbers in Tab. 1 show an unusual pattern. Apart from RACE, 70 of the 75 percentage cells end in .0 or .5. All Elo scores are multiples of 10, all latencies are multiples of 5, and every KPC value except one is a multiple of 0.05. Continuous scorer outputs rounded to one decimal place would rarely produce such regularity, and the appendix tables use visibly finer granularity. Could the authors explain how these values were rounded and release the raw per-task result files mentioned in App. B.2?
  • The method depends heavily on the quality of the model's own plan. Evidence is scored only against self-generated plan fields, including the model's anticipated answer. A stale plan performs worse than query-only selection (50.9 vs. 51.4), and flat requirements fall below Full-History (50.8 vs. 52.0). The DAG is never revised, and feedback only widens exposure within the current plan's direction. If plan-selected evidence agrees with the model's prior, the trigger will not fire. Experiments with plausible but wrong plans, and a "Designer DAG + simple executor" ablation, would clarify where the gain comes from.
  • The efficiency analysis does not reflect the real cost. Full-History is append-only and benefits from prefix caching, which reduces both billed cost and latency. Flashlight recompiles the context every turn (re-selected and reordered evidence, plus changing plan fields), so little beyond the system prompt can be cached. The token reduction may therefore shrink substantially in actual cost. Fig. 3 also shows that Flashlight uses more tokens per turn in early turns. The authors should report cache hit rates and billed cost with caching enabled.
Questions for authors
  • Tab. 5 states that ACE, ContextWeaver, WebWeaver, and RhinoInsight were run with GPT-4.1 in the common harness. In a direct comparison, WebWeaver and RhinoInsight beat Flashlight (GPT-4.1) on most metrics other than RACE. Why are they excluded from the comparison? Similarly, DR Tulu-8B, which is also based on Qwen3-8B, achieves the best SQA-CS2 score at lower cost.
  • In EIA replay, the reward depends only on the action ID, not on the query text. Who generates the candidate actions and the required-fact labels?
8 comments
AuthorsOpenReview ↗

Reply to W1: Table 1 result sources, numerical precision, and display rules

Thank you for carefully examining Table 1. The display patterns arise from how two categories of results were compiled. We first compiled publicly reported evaluation results that could be cited. Specifically, WebWeaver's RACE of 50.58, C.Acc. of 93.37, and E.Cit. of 200.75 correspond to 50.6, 93.4, and 200.8 in Table 1 [1]; RhinoInsight's RACE of 50.92 corresponds to 50.9 [2]; WebThinker's RACE of 37.9 and SQA-CS2 score of 46.7 match values in a published comparison table [3]; Tongyi DeepResearch's RACE of 40.6 and SQA-CS2 score of 46.5 both appear in that same comparison table [3]; OpenDeepSearch's KPR of 42.81 and KPC of 0.84 correspond to 42.8 and 0.84 in Table 1 [4]; and DR Tulu's SQA-CS2 score of 88.3 corresponds to its publicly reported RL result [3]. The SQA-CS2 score of 40.4 for Qwen3-8B Full-History also comes from [3].

The publicly reported values identified above were cited from the corresponding references. For the remaining aggregate results, Table 1 used the following display rules to keep the wide table readable: C.Acc., DRB-II Total, Coverage, SQA-CS2, and KPR were rounded to the nearest 0.5 percentage point; E.Cit. to the nearest integer; Elo to the nearest 10 points; KPC to the nearest 0.05 percentage point; and latency to the nearest 5 seconds, while RACE was displayed to one decimal place. We will state these rules explicitly in Appendix B.2.

The results table before this processing is shown below:

Dataset / groupDRBDRBDRBDRB-IIResearchQAResearchQAScholarQA-CS2DeepResearchGymDeepResearchGymEfficiency: five-benchmark macro-averageEfficiency: five-benchmark macro-averageEfficiency: five-benchmark macro-average
Method / metricRACE ↑C.Acc. (%) ↑E.Cit. (count) ↑Total (%) ↑Coverage (%) ↑Elo ↑Score (%) ↑KPR (%) ↑KPC (%) ↓Tokens (k) ↓LLM calls ↓Latency (s) ↓
ACE49.785.1272.0845.0967.13103470.1358.071.2109222522
ContextWeaver48.983.0968.1543.8665.94116768.9456.831.2488419471
WebWeaver (GPT-4.1 adaptation)50.5893.37200.7548.9873.88141882.0869.040.539121421118
RhinoInsight50.9289.1787.8948.1372.10138377.1166.120.7049824760
WebThinker-32B (report)37.973.9246.1437.0469.90108046.752.101.1028218638
Tongyi DeepResearch40.677.8852.0740.9863.92116346.554.971.0387616521
OpenDeepSearch38.580.9649.9538.8756.1490442.9742.810.846414432
DR Tulu-8B (RL)46.886.2064.0146.8373.09104288.361.090.7484812279
GPT-4.1 Full-History49.882.1862.0546.1168.47115671.4660.541.21212518541
GPT-4.1 Flashlight51.288.4968.8949.0772.11128974.8864.060.7966814410
GPT-5.6-Luna Full-History52.083.8970.1149.9172.79123778.0365.090.84413218479
GPT-5.6-Luna Flashlight54.291.0782.0553.1577.03138282.1568.990.5537214364
Qwen3-8B Full-History40.570.1543.9436.0756.0876440.441.131.34610818621
Qwen3-8B Flashlight + SFT46.083.2161.0144.9267.16104464.0255.080.8125713430
Qwen3-8B Flashlight + EIA-GRPO49.087.0268.2148.8171.92116171.7961.020.6615412396

[1] WebWeaver: Structuring Web-Scale Evidence with Dynamic Outlines for Open-Ended Deep Research, arXiv:2509.13312, v3.

[2] RhinoInsight: Improving Deep Research through Control Mechanisms for Model Behavior and Context, arXiv:2511.18743, v1.

[3] DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Research, arXiv:2511.19399, v3.

[4] DeepResearchGym: A Free, Transparent, and Reproducible Evaluation Sandbox for Deep Research, arXiv:2505.19253, v3.

AuthorsOpenReview ↗

Reply to W2 (Part 1/3): Background on plan-quality sensitivity and contributions beyond the Designer map

Thank you for raising the issues of plan-quality dependence and component attribution. First, we distinguish the initial requirement map from the dynamically updated next-context plan. The map specifies the task attributes, evidence obligations, and dependencies that must be covered, rather than prescribing an answer to be established. Provisional judgments, residual information gaps, and subsequent research directions are formed during research and can be updated after evidence review. A fixed map therefore does not imply a fixed research conclusion. The following inspected runtime cases illustrate this distinction:

Actual runtime caseWhat the map specifiesChanges during subsequent researchScope of the evidence
Human intention recognitionFind literature-supported definitions, method categories, and data modalities; concrete answer slots remain openReject sources without direct support, update partial-support states and residual gaps, and continue targeted acquisitionEvidence needs and research progress can change under a fixed map
Paper-map digitizationEstablish the workflow, scanning, vectorization, registration, and challengesForm partial workflow judgments and identify further gaps concerning step transitions and model applicability, but ultimately hand off with caveatsIdentifying and refining gaps does not guarantee their resolution

These runtime cases illustrate dynamic judgments under fixed task obligations. Their feedback implementation differs from the paper's calibrated trigger; the controlled evaluations below assess plan dependence and component contributions separately. Requirement omissions and inappropriate fixed conditions concern the map itself, beyond provisional-answer updates.

Second, source-over-echo guides evidence exposure, with correctness assessed from the resulting responses. It balances examining additional evidence to verify, qualify, and refine judgments against redundancy, interference, and additional cost. Even when the current judgment is correct, additional relevant evidence may help distinguish applicability conditions, reconcile measurement definitions, or examine counterexamples; a benefit need not involve changing the core conclusion. Conversely, unconditional expansion may introduce inapplicable material and interfere with retaining important earlier qualifications. We evaluated these controls on DRB. The following table reports measured RACE scores and mean full-task input/output tokens per DRB task:

Question examinedControl conditionDRB RACEDRB mean tokens/task (k)Supported interpretation
Does plan content matter?Query-only, no feedback51.480.0Reference without the complete plan
Stale plan, no feedback50.980.1An outdated plan can be harmful
Complete plan, no feedback52.669.7A timely, relevant plan has incremental value
Is the gain explained only by role orchestration?Same-pipeline full-state replay52.9124.2Retains the Designer, Controller, tools, and validation workflow
Full Flashlight54.275.6Supports an incremental contribution from context policy within the same pipeline
Is the gain simply due to more exposure?Full plan + random broadening52.875.6Matched activation rate and similar measured token consumption
Full plan + always broadening53.7101.3More exposure does not automatically yield higher quality
Full plan + calibrated feedback54.275.8Supports the quality–usage trade-off of selective exposure

The complete plan improves RACE by 1.2 points over query-only selection while reducing mean tokens from 80.0k to 69.7k (approximately 12.9%). The stale plan scores 0.5 points below query-only at similar usage, highlighting the value of a timely information need. Within the same pipeline, Full Flashlight improves RACE by 1.3 points over replay while reducing tokens from 124.2k to 75.6k (approximately 39.1%). Calibrated feedback scores 1.4 points above random broadening at similar measured usage (75.8k versus 75.6k), and 0.5 points above always broadening with approximately 25.2% fewer tokens (75.8k versus 101.3k). Together, these DRB controls support task-aligned planning and selective exposure. Evaluation A examines sensitivity to biased information needs; evaluation B compares simple and adaptive execution under a shared Designer map. The Full Flashlight and calibrated-feedback entries retain their respective measured means from the two control comparisons.

AuthorsOpenReview ↗

Reply to W2 (Part 2/3): Supplementary evaluation A — Biased next-context plans, evidence selection, and subsequent analysis

We evaluated 200 real DRB checkpoints drawn from the 100-task DRB set. Map denotes the Designer-generated task structure; plan denotes the model-generated next-context information need. Before evidence selection, branches shared the request, history, query, map, candidate pool, and original plan. Original-plan conditions used the unedited input; neutral and biased conditions changed only the designated information-need span.

GPT-5.6 Sol prepared both edits. The neutral plan preserved task scope, necessary measurement and comparison qualifications, approximate length, and unrelated fields without presupposing an answer. The biased plan introduced one on-topic but inadequate information need, such as a missing measurement constraint or a proxy outcome. Only the selection direction changed, not the answer or source contents; other inputs remained fixed.

Conceptual example — total cost: with transport savings established but maintenance costs unresolved, the original plan could seek maintenance and equipment records to assess whether they offset the savings. A neutral rewrite could seek comparable transport, equipment, and maintenance records; a biased rewrite could seek transport records alone to assess total cost. These are illustrative rewrites.

Edits were injected once at the initial checkpoint, after which plans were generated and updated normally. The assigned feedback policy remained off or on through termination: no feedback disabled response-triggered expansion, while calibrated feedback recomputed its signal at eligible opportunities and expanded when its rule activated. Branches shared the continuation model, embedding model, tools, Writer, maximum budgets, and failure rules. The candidate pool was fixed across branches only for the first local evidence-selection stage; subsequent searches, evidence pools, histories, and plans could diverge naturally as each branch continued to a final report.

Before branching, GPT-5.6 Sol labeled reliable task-critical evidence in the common pool using the original request, current decision, and verifiable source passages. Labels covered supportive and opposing evidence equally, including necessary scope or measurement qualifications and material counterarguments. Task-critical evidence recall measured the fraction of labeled critical passages actually displayed, averaged across evaluated checkpoint states; selected passage IDs were verified against actual model-input logs.

Separate, condition-blinded GPT-5.6 Sol grading calls assessed the visible analysis and local conclusions in the next substantive response using the original question, necessary shared history, target decision, and verifiable reference evidence. Grading checked factual support, inference validity, and necessary qualifications; model memory alone was not treated as ground truth. A correct unchanged conclusion could pass. Final reports were evaluated with the same DRB RACE protocol across conditions.

Table A. Measured next-context-plan and feedback comparison.

Selection inputExposure strategyTask-critical evidence recall (%)Next-step analysis/conclusion correctness (%)Final DRB RACEDRB mean tokens/task (k)
Original plan (Full Flashlight reference)No feedback60.581.552.669
Original plan (Full Flashlight reference)Calibrated feedback76.690.554.276
Neutral next-context planCalibrated feedback70.286.553.177
Biased next-context planCalibrated feedback54.877.051.676

Correctness uses all 200 responses: 163, 181, 173, and 154 passed, respectively; fail and indeterminate responses count as non-pass. Recall averages per-state critical-passage recall. Tokens is mean full-task input plus output per DRB task.

For the original plan, enabling calibrated feedback increased task-critical evidence recall by 16.1 percentage points, next-step analysis/conclusion correctness by 9.0 percentage points, and final RACE by 1.6 points, with DRB mean tokens per task changing from 69k to 76k. Under calibrated feedback, the original plan exceeded the biased plan by 21.8 recall percentage points, 13.5 correctness percentage points, and 2.6 RACE points, with both reporting 76k tokens per task. The neutral plan exceeded the biased plan by 15.4 recall percentage points, 9.5 correctness percentage points, and 1.5 RACE points, with reported mean tokens per task of 77k versus 76k.

These results support complementary roles for task-aligned planning and calibrated feedback. With the original plan, feedback improves evidence recall, local correctness, and final RACE. Under the same feedback policy, the original plan outperforms the edited alternatives, linking useful evidence selection to the specificity and relevance of the information need.

Reviewer_Qd9QOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Semantic Flashlight, a context-management framework for deep-research agents. The agent generates an explicit next-context plan alongside each search query, and the Context Engine uses this plan to select evidence for subsequent decisions. A response-feedback mechanism broadens evidence exposure when the agent’s response aligns more strongly with its previous responses than with source evidence. The framework maintains evidence requirements, source bindings, and validated state transitions across Designer, Controller, and Writer roles. Optional SFT and EIA-GRPO extend the framework to an open-weight model.

Critical strengths

Meaningful efficiency gains. Within the same pipeline, Flashlight improves RACE from 52.9 to 54.2 while reducing average token use from 118k to 72k. This supports practical benefits beyond changes in agent orchestration.

Extensive experimental controls. Shuffled plans, equal-length prompts, random broadening, and matched-resource comparisons address several plausible confounds. The paper also clearly separates inference-time improvements from training effects.

Independent quality evaluation. Semantic auditing reports fully supported claims increasing from 80.2% to 88.7%. A second judge and blinded human comparisons provide additional evidence of improved report quality.

Critical weaknesses

Below are some minor issues I think would be great to be solved/discussed:

The framework brings together established ideas, including explicit planning, semantic evidence selection, and adaptive context expansion. It would be helpful to explain more precisely which aspects of their combination provide the main methodological advance over existing plan-conditioned retrieval approaches.

The signal appears useful empirically, although similarity to sources may sometimes reflect paraphrasing, while similarity to earlier responses may reflect productive reasoning continuity. A few examples distinguishing these cases would help clarify when the signal is most informative.

A model-generated plan may occasionally omit a relevant requirement or contain an imperfect assumption, potentially influencing subsequent evidence selection. More discussion of how the system detects and revises such plans would strengthen.

Questions for authors

Could the authors clarify the main distinction between the proposed selection mechanism and semantic retrieval conditioned on an expanded query or research plan?

How might the framework identify and revise missing requirements or incorrect assumptions in the initial plan?

7 comments
AuthorsOpenReview ↗

Reply to W1: Contributions of next-context planning and evidence-exposure control

Thank you for asking us to clarify the distinction from semantic retrieval with an expanded query or research plan.

We use map for the Designer-generated DAG of task requirements and plan for the model-generated next-context information need. The map preserves long-term requirements and progress; the plan specifies what the next context should contain and what question it should help resolve.

Our contribution is turn-by-turn context construction: the model outputs its information need alongside each search decision, and the engine uses that need to select stored evidence for the subsequent model input. The search query acquires material; the next-context plan guides what the model sees next. This design provides three kinds of value.

(1) Efficiency: reusing the model's existing information need. When proposing a search, the model already identifies missing information. Flashlight makes this need explicit and reuses it as the semantic selection target, without a separate generative call to reconstruct selection intent. Rather than repeatedly replaying a growing history, the engine constructs a compact context whose contents change with the current research need. The persistent map retains overall task direction and progress. Selection uses encoding and engine computation rather than an additional generative selection call.

(2) Quality: balancing compact context with sufficient evidence exposure. Excessive history can obscure important evidence, while insufficient exposure can leave provisional conclusions untested and lead to repeated, increasingly narrow searches. Our behavioral analysis finds that trajectories whose responses align more with the current context than with the preceding response tend to have higher report quality. This association motivates the online source-grounded signal: when response echo dominates source alignment, the engine broadens plan-relevant evidence exposure. The signal controls exposure; correctness is evaluated separately. The completed local experiments in our Reply to W2 assess the resulting continuation effects.

Broader exposure includes relevant opposing evidence. Review can use it to revise a conclusion, establish its scope, or rule out an already-checked, inapplicable search direction. For example, evidence about a different condition can clarify why that direction need not be searched repeatedly. Thus, exposure helps test local judgments and refine subsequent searches, rather than simply supplying more support for the current answer.

(3) Framework and validation: linking local context to the entire task. The map and persistent evidence state record requirement progress and reviewed source relationships for reuse across acquisition, review, and final writing. Evidence remains stored even when it is not displayed in the current context. Within-pipeline controls retain the same roles, tools, and validators: Flashlight improves RACE from 52.9 to 54.2 while reducing tokens from 118k to 72k relative to full-state replay. At approximately 72k tokens, feedback-guided broadening reaches 54.2, compared with 52.8 for random broadening; always broadening uses 96k tokens and reaches 53.7 (Table 2).

Flashlight combines explicit next-context planning, feedback-guided evidence exposure, and persistent task state to reconstruct successive model inputs. Its distinction from expanded-query retrieval is control over the next reasoning context, not only the acquisition query or a static research plan.

AuthorsOpenReview ↗

Reply to W2 (Part 1/3): Feedback interpretation and measured evidence

Thank you for raising the distinction between source similarity and productive reasoning continuity.

Feedback allocates evidence exposure; its benefit need not be a changed conclusion. Additional plan-relevant evidence can strengthen verification, clarify applicability, and sharpen the next research step even during productive continuity. The signal broadens exposure when standardized response echo dominates source alignment, rather than classifying reasoning as correct or incorrect. We tested its utility through two complementary experiments, also reported to ptfY.

Measured behavior audit. We audited triggered turns from Flashlight traces on all 100 DRB tasks. GPT-5.6 Sol labeled the behaviors from the request, sources, response, and history, with feedback scores and later outcomes hidden. Correct disagreement required an evidence-supported rejection or qualification of a source claim; legitimate revisiting required a supported historical fact reused for a new task-relevant inference.

From each qualifying saved checkpoint, we compared executing versus skipping only the current expansion, retaining the other settings and subsequent control rules. Independent GPT-5.6 Sol grading calls scored the next substantive response on a 0–3 support rubric, with branch labels hidden and order randomized: 0, a wrong or unsupported key judgment; 1, partial support with major gaps; 2, sufficient support with necessary qualifications; 3, additional task-relevant verification or counterargument handling. Extra text or citations alone earned no credit. Gain is the mean paired expansion-minus-no-expansion score difference.

Behavior among triggered turnsTurns/tasksPaired gain on 0–3 support rubricExpansion scored lower
Correct source disagreement126/64+0.869/126
Legitimate fact revisiting165/78+0.7317/165

Both categories showed positive mean support gains, with lower expansion scores in 7.1% and 10.3% of usable pairs. Categories/tasks may overlap. These trigger-conditional local results demonstrate useful continuation gains during justified disagreement and productive continuity.

Measured trigger-strategy comparison. We used 200 eligible DRB checkpoints, including untriggered states, each with a verifiable target and sufficient evidence. GPT-5.6 Sol edited only the earlier conclusion to create correct/incorrect histories, keeping evidence and other state fixed. Each history condition contained 200 states. Six strategies shared generation, candidate evidence, selection, expansion, and continuation settings; only the expansion decision differed. Branches stopped at the next substantive response. Random and single-sided settings were development-calibrated and frozen.

Independent GPT-5.6 Sol calls graded the supported judgment and specified-false endorsement against verifiable evidence, hiding condition/strategy labels and randomizing order. Mentioning or rejecting the false proposition did not count as endorsement. Correction and reversal use all 200 states in their respective history groups, including unactivated states. Activation and mean added input pool all 400 states; unactivated states contribute zero added tokens.

TriggerExpansion %Extra input tokens/stateIncorrect history: corrected %Correct history: reversed to false %
No expansion00553
Budget-matched random322.2k615
History echo only322.2k657
Low source alignment only322.2k644
Source-over-echo322.2k703
Always expand1006.9k7411

Source-over-echo raised correction from 55% to 70%, with the same 3% specified-false reversal rate. At 32% activation and 2.2k added tokens/state, it exceeded random, history-only, and source-only correction by 9, 5, and 6 percentage points, with lower reversal rates. Always expanding reached 74% correction, but 11% reversals and 6.9k added tokens/state. The joint signal used about 68% fewer added tokens, with 4 points less correction and 8 points fewer specified-false reversals. Together, these results support selective exposure that improves continuation support and offers a favorable correction–risk–usage balance.

AuthorsOpenReview ↗

Reply to W2 (Part 2/3): How triggered exposure refines reasonable reasoning

Conceptual cases. The triggered cases below illustrate possible judgment and next-step changes; nontriggered cases are reported in Part 3. Their branch outputs and trigger signs are constructed examples, separate from the measured tables in Part 1.

1. Signal triggered: the current reasoning is correct, but broader exposure can refine the judgment

Conceptual trigger condition: calibrated source-over-echo margin < 0.

  • Context: The model has concluded, "Project A improved recovery rates in the pilot, but generalizability remains unverified." The remaining need is to identify the conditions under which the result transfers.
  • Evidence: A pilot summary reports an increase from 40% to 55%; an industry overview describes the project workflow. Neither specifies the equipment conditions or measurement basis in detail.
  • Model output before the trigger decision: "The pilot results are promising, but generalization is not yet established. Next, seek more implementation cases." This mainly continues the existing judgment without adding specific evidentiary conditions.
  • Branch without executing broadening: "Continue searching for implementations at other factories to assess generalizability." The reasoning remains reasonable, but the direction is broad.
  • Additional evidence after executing broadening: The pilot used specialized sorting equipment and counted only sortable material; another factory lacked that equipment and counted all incoming waste.
  • Model output after executing broadening: "Retain the finding of improvement under the pilot conditions, but the factories cannot be compared directly. Next, seek evidence with matched equipment and measurement definitions; do not use cases lacking these conditions to infer generalizability."

Broader exposure makes an already reasonable judgment more specific in this example. The model retains its conclusion while identifying which comparisons to exclude and what to search for next.

2. Signal triggered: the current material lacks a direct answer, but broader exposure can clarify rejection criteria

Conceptual trigger condition: calibrated source-over-echo margin < 0.

  • Context: The model has concluded, "It is not yet established that Project B reduces actual transport costs."
  • Evidence: A project introduction claims improved efficiency, while a feature description provides no observed cost records.
  • Model output before the trigger decision: "The evidence is insufficient; next, seek cost-saving cases." The response mainly repeats the existing gap.
  • Branch without executing broadening: "Continue searching for quantitative cost benefits." This does not further specify which measurements would answer the question.
  • Additional evidence after executing broadening: A reported "20% saving" concerns travel time; another benefit estimate is a prediction; the cost calculation excludes maintenance.
  • Model output after executing broadening: "These materials concern time savings, predicted benefits, and incomplete accounting, respectively. They do not directly establish lower actual transport costs. Next, seek before-and-after cost records that include maintenance."

The added material helps the model reject inapplicable evidence more precisely and narrow the remaining information need while leaving the final answer open. Repeated promotional claims alone would yield no support-rubric benefit.

Reviewer_ptfYOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Semantic Flashlight, a context-management framework for deep-research agents. It generates a next-context plan alongside each search query and uses this plan to select evidence for subsequent steps. A response-feedback signal controls when to expose additional evidence. The framework also maintains explicit evidence requirements and source bindings across research and writing. Experiments on five benchmarks show improved report quality with lower token consumption, with SFT and GRPO explored as an additional open-weight extension.

Critical strengths
  • Generating the context plan together with the search query is a sensible design. It reuses the agent’s current information need and avoids an additional LLM call to reconstruct that need from the accumulated history.

  • The efficiency gains are substantial and come with improvements in report quality across both closed-source backbones. The comparison against full-state replay within the same pipeline is particularly useful for separating context-management gains from changes in orchestration.

  • The evaluation goes beyond simple component removal. The plan and feedback controls, budget-matched comparisons, and independent claim-support audit provide reasonably strong evidence for the proposed design. I also appreciate the inclusion of human evaluation and failure accounting.

Critical weaknesses
  • The response-feedback signal is useful empirically, but its interpretation as evidence utilization is less convincing. Embedding similarity may not distinguish paraphrasing a source from reasoning with it, or legitimate continuity from unproductive repetition. The current analyses do not directly separate these cases.

  • The framework relies on an initial requirement graph and fixed answer conditions. The retrieval stress tests are helpful, but they say less about recovery when the initial decomposition misses an important requirement or becomes inappropriate after new evidence is found.

  • The relationship between evidence coverage and task completion needs clarification. Writer synthesis requires all required nodes to be supported, yet reported evidence coverage is substantially below the task completion rate. Different evaluation stages or aggregation rules may explain this, but the distinction is not clear enough.

Questions for authors
  1. Have you examined feedback-trigger errors on turns that correctly disagree with a source or revisit an established fact? A small breakdown would help clarify what the source-over-echo signal actually detects.

  2. Can the Controller revise the initial requirement graph or its fixed answer conditions? How does the system handle important requirements that were omitted during initial planning?

  3. Could you clarify the denominator and measurement stage for ECov? In particular, how should the 82% ECov in Table 11 be understood alongside the 96% task completion in Table 18 and the Writer readiness condition?

6 comments
AuthorsOpenReview ↗

Reply to W1: Feedback interpretation and trigger-strategy evaluation

Thank you for distinguishing useful continuity from repetition. Source-over-echo is an evidence-exposure control cue, not a verdict that a response is wrong. High history alignment can accompany correct reasoning, while source alignment can reflect paraphrasing. We therefore assess continuation quality, correction, specified-false reversals and cost rather than treating higher embedding similarity as success.

Existing evidence. Table 2(b) reports 54.2 RACE/72k tokens for feedback, versus 52.8/72k for random and 53.7/96k for always broadening. Appendix C.2, Table 10 reports paired broaden/no-broaden continuations over 3,238 eligible turns: feedback yields +2.6 points on triggered cases and +0.7 overall, versus +0.3/+0.1 for equal-activation random broadening. Task-shuffling the added evidence reduces the conditional gain to +0.1. Together these reported controls support selective exposure to relevant evidence over simply adding more text. The controlled experiment below examines local correction; our reply to Q1 reports the two requested behavior categories.

Trigger-strategy evaluation. We completed a controlled comparison using 200 eligible checkpoints selected after Flashlight runs on DRB, not restricted to full-signal-triggered turns. Each contained a verifiable target proposition and sufficient current evidence. GPT-5.6 Sol helped identify the correct answer and a specified false proposition, then constructed correct/incorrect-history versions by rewriting only the designated earlier conclusion. Current evidence and unrelated state components remained fixed. The same checkpoints produced 200 states per history condition, not 400 independent tasks.

All six strategies shared the generation model, candidate pool, evidence-selection rule, expansion configuration and continuation limits; only the current expansion decision differed. Random triggering ignored alignment. History echo only expanded above a history-alignment threshold; low source alignment only expanded below a source-alignment threshold. The full response-feedback used the calibrated source-over-echo signal; always expand activated at every state. The matched controls used development-calibrated settings frozen for evaluation. Each branch stopped at the next substantive response.

Separate GPT-5.6 Sol grading calls used the target subquestion, verifiable evidence, correct/false propositions and response, with history-condition and strategy labels hidden and order randomized. The prompt required quoted evidence for whether the response committed to the correct target judgment or endorsed the specified false proposition; mere mention or rejection did not count as endorsement. A script computed correction over all 200 incorrect-history states and specified-false reversal over all 200 correct-history states, including unactivated states. Expansion rates pool both conditions; added input tokens/state averages across all 400 states, counting zero for unactivated states. These are local diagnostics, not final DRB RACE or natural error prevalence.

TriggerExpansion %Extra input tokens/stateIncorrect history: corrected %Correct history: reversed to false %
No expansion00553
Budget-matched random322.2k615
History echo only322.2k657
Low source alignment only322.2k644
Source-over-echo322.2k703
Always expand1006.9k7411

The full signal increased correction from 55% without expansion to 70%, with the same 3% specified-false reversal rate. At the same reported 32% activation and 2.2k added tokens/state, it exceeded random and single-sided controls by 9, 5 and 6 percentage points in correction, with lower reversal rates. Always expanding reached 74% correction but used 6.9k added tokens/state and produced 11% reversals. Relative to always expanding, the full signal used about 68% fewer added input tokens, with 4 percentage points less correction and 8 points fewer reversals. These results support the full signal's selective allocation of evidence exposure: stronger correction than the matched alternatives and a favorable correction–reversal-risk–cost balance relative to indiscriminate expansion.

Interpretation. The controlled comparison shows that the joint signal allocates expansion more effectively than the matched random and single-sided alternatives, improving local correction with lower specified-false reversal rates. Together with the paper's end-to-end controls, these results support response feedback as a practical rule for allocating additional evidence exposure. Its value lies in the quality and cost of the resulting continuation, rather than treating disagreement or historical continuity as errors.

AuthorsOpenReview ↗

Reply to W2: Requirement-map revision and recovery from initial omissions

Thank you for raising the question of initial requirements and adaptive research.

We use map for the Designer-generated DAG of task requirements and plan for the model-generated next-context plan.

The paper's original configuration fixes the initial map's evidence obligations and answer conditions. Within an obligation, review can reject sources, identify conflicts, and update provisional claims, residual gaps, and next-context plans. To address recovery from omitted requirements directly, we implemented and evaluated a map-revision extension in an additional experiment.

We evaluated all 100 DRB tasks using GPT-5.6 Luna as the task-generation backbone in all four conditions. GPT-5.6 Sol prepared the omission perturbations by selecting 2–3 related requirement nodes representing an important explicitly requested facet. Selection was tied to the corresponding request passage and whether surviving nodes adequately covered that facet. The engine removed the selected nodes after map installation and before evidence acquisition, cleaned associated references, and checked graph validity. Revision Off and On started from the same map state and unchanged request; the original map and deletion metadata were withheld from the Omitted runs. Normal used the unperturbed map.

Revision Off retained normal evidence review and next-context-plan updates while leaving the installed map fixed. Revision On added a request-versus-map check to the existing Evidence Review call and a structured revision output alongside the ordinary review. The check returned no change or a minimal patch describing the request/evidence basis, requirement, completion conditions, dependencies, affected nodes, and rationale. Conceptually, a request for deployment constraints could justify adding a missing hardware/compatibility requirement. The engine validated and applied patches, scheduled new requirements as pending, and invalidated affected certificates where necessary while retaining unaffected evidence. Revision-related tokens and subsequent research were included in usage.

All four conditions used matched task-generation, retrieval and evidence-selection settings, stopping rules and budget limits, and ran to final reports. Coverage evaluates all original-request requirements against a fixed external checklist. Omitted-facet handled evaluates whether the final report adequately addresses the deliberately removed facet, with the same criterion for both Omitted branches; — denotes not applicable under Normal. RACE follows the DRB final-report protocol. Tokens is mean total input/output token usage per task. These GPT-5.6 Luna measurements are a new controlled evaluation, presented separately from the paper's existing backbone results.

MapRevisionCoverage %Omitted-facet handled %RACEDRB mean tokens/task
NormalOff86.1—54.276k
NormalOn86.5—54.495k
OmittedOff77.65650.773k
OmittedOn84.77652.499k

Under omission, revision improved coverage by 7.1 percentage points, omitted-facet handling by 20 points, and RACE by 1.7, with DRB mean tokens per task increasing from 73k to 99k (approximately +35.6%). Successful facet handling in 56% of Off runs shows that the fixed-map workflow already recovered some omitted content; explicit map revision raised this to 76%. Under Normal, gains were smaller (+0.4 coverage points; +0.2 RACE), while tokens rose from 76k to 95k (+25%). Usage changes are computed from the displayed means. Thus, the extension delivered a targeted recovery benefit when initial map construction omitted important requirements, and the unperturbed comparison quantified its routine overhead. Revision narrowed the omission-induced RACE gap to the Normal/Off reference from 3.5 to 1.8 points. These results motivate selective deployment of map-level checks where their recovery value outweighs additional research cost.

The direct question about revising the graph and fixed answer conditions is answered separately in our Reply to Q2. The experiment here evaluates omitted requirements; it does not establish recovery rates for every possible change in answer conditions after new evidence.

AuthorsOpenReview ↗

Reply to W3: Evidence coverage, report completion, and Writer readiness

Thank you for raising the distinction between evidence coverage, task completion, and Writer readiness.

The Designer map is a DAG with two types of requirement nodes: attribute nodes establish the source-supported factual properties needed for the task, while relation nodes address the requested comparisons, connections, or other relational judgments over those properties. For example, a comparison may require factual attributes of two alternatives before establishing their differences under the requested conditions. These examples describe node roles; evidence binding and certification still determine whether each required node is supported.

Termination uses two levels of checks. First, the required attribute prerequisites must be supported. If research terminates while these prerequisites remain unsatisfied, the task is unsuccessful; reaching a budget limit does not waive this gate. Second, once those prerequisites are met, research continues toward closure of the remaining required relation nodes while budget remains. Satisfying attributes alone does not immediately trigger writing. When every required node passes the evidence and completion checks, the system follows the normal Writer path.

If the research budget is exhausted after the attribute prerequisites are met but before all required relations are certified, the system follows a budget-bounded Writer path using the supported evidence already stored. The map guides construction of the writing packet, and the report distinguishes supported findings from unresolved relations. This handoff neither extends the search budget nor certifies the unresolved nodes. A qualifying report can therefore be delivered with a supported factual foundation while some relational requirements remain open. Writer invocation alone does not establish successful delivery.

ECov is measured from the requirement states at research termination. For each task, its denominator includes all required attribute and relation nodes, including those that remain unstarted or uncertified; its numerator counts nodes with engine-checked evidence bindings and completion certificates. We then average these per-task percentages, giving each assigned task equal weight rather than pooling nodes across tasks. Thus, ECov = (100/N) sum_i (certified required nodes_i / all required Designer nodes_i). Failed tasks remain included; they are not removed from the denominator.

Table 18's 96/100 completion statistic denotes successful report delivery through either the normal or qualifying budget-bounded path, rather than certification of every map node. In the recorded Flashlight runs, the other four tasks had not satisfied all required attribute prerequisites at termination and were not counted as completed. Table 18 separately labels their terminal outcomes as one degraded fallback, one parser/state failure, one evidence/citation failure, and one timeout; these mutually exclusive outcome labels describe how the unsuccessful runs ended, while attribute readiness describes the unmet delivery prerequisite. A degraded fallback output is therefore distinct from a qualifying budget-bounded delivery counted as Completed. Among completed budget-bounded deliveries, unresolved relation nodes remain in the ECov denominator but not its numerator. Thus, the 82.0% task-averaged ECov and 96% completion rate measure node-level evidence closure and task-level delivery, respectively. The 96 completed tasks are not further split into normal and budget-bounded deliveries by Table 18.

We will make the attribute/relation distinction, attribute-prerequisite gate, normal and budget-bounded writing paths, and ECov aggregation explicit in the revision. The “only when” readiness statement in Sec. 4.4 and App. E will be scoped to normal synthesis, and Table 18 will distinguish report delivery from certification of every required map node.

Our Reply to Q3 separately answers the requested denominator, measurement stage, and interpretation of the 82% versus 96% figures.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.