acceptodds
Under review as a conference paper at ICLR 2027

ClueFlow: Learning Long-Horizon Search through Attributed Research-State Transitions

Abstract

Long-horizon search requires agents to acquire relevant evidence and integrate it into an evolving research state. Existing methods maintain research states to guide exploration or assign turn-level progress signals, yet grounding these signals in explicit, attributable state changes remains underexplored. We introduce ClueFlow, which learns long-horizon search from attributed research-state transitions. ClueFlow constructs Source–State–Final attribution chains that connect evidence-acquiring browser actions, Evidence and Candidate updates, and the final answer supported by those updates. During search, ClueFlow records typed state changes and potentially supporting browser actions, then applies gold-conditioned semantic validation and source tracing after execution. We use this attribution to guide both supervised fine-tuning and reinforcement learning. During SFT, attribution selects which decisions receive supervision with their original interaction contexts. During Turn-PPO, the same attribution augments outcome-based turn advantages with credit assigned along the Source–State–Final chains. Transition-level validation also recovers useful supervision from gold-consistent state changes in trajectories with incorrect final answers. With Qwen3.5-35B-A3B, ClueFlow achieves 59.1% accuracy on BrowseComp and 72.0% on BrowseComp-Plus. In controlled BrowseComp-Plus experiments with Qwen3-30B-A3B, attribution-guided SFT matches all-turn exact-context SFT using only about 30% of the training tokens, while attribution-guided PPO outperforms outcome-only PPO by 4.1 percentage points. These results demonstrate that state-transition attribution enables efficient supervision and effective credit assignment for long-horizon search agents.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, long-horizon search, research-state transitions, credit assignment, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_AnHvOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper presents ClueFlow, a training framework for long-horizon search agents. Instead of treating every turn equally, it keeps evidence and candidate records, then after each trajectory it checks which state changes really support the answer and traces them back to the browser actions that provided the evidence. These Source–State–Final chains decide which turns are trained in their original context and how Turn-PPO assigns credit; validated partial progress from wrong final answers is also reused. With Qwen3.5-35B-A3B, ClueFlow reaches 59.1% on BrowseComp and 72.0% on BrowseComp-Plus.

Critical strengths
  1. The core idea is easy to buy: instead of treating every turn equally, the method asks which state changes actually moved the research forward and which browser actions supplied the evidence.

  2. ClueFlow gives a concrete learning signal for both SFT and PPO, and the failure-recovery angle is particularly practical because long search trajectories often contain useful intermediate work even when the final answer is wrong.

  3. The appendices are very detailed, and the qualitative traces help the reader clearly see what an attribution chain actually looks like.

Critical weaknesses
  1. The source-tracing step is heuristic, yet we do not see its precision and recall or how often valid evidence transitions are dropped because no reliable browser source is found. This matters because a wrong source link or a wrong validation label directly changes which turns get supervised or credited.

  2. The comparisons also have several confounds: different teacher trajectories, different checkpoints for fixed-corpus and live-web settings, different context management, etc. The controlled ablations are better, but they do not fully isolate attribution from serialization format and data volume.

  3. The PPO attribution budget hypeparameters are hand-tuned and there is no analysis or comparison with other turn-level credit-assignment methods.

Questions for authors
  1. As mentioned in weakness-1, what is the precision and recall of deterministic source tracing, and what fraction of validated Evidence transitions are dropped for lack of a reliable source?

  2. How much of the gain comes from the persistent state itself versus the clue tools, readiness rules, and prompting?

  3. Can failure recovery reinforce evidence that is correct but followed by a wrong conclusion? More qualitative cases would help.

2 comments

Thank you for the detailed response and additional experiments. They address my main concerns and strengthen the paper. I think I will maintain my rating.

We sincerely thank the reviewer for taking the time to revisit our responses and additional experiments. We are glad that these additions have addressed your main concerns and strengthened the paper. We greatly appreciate your constructive feedback throughout the review process.

Reviewer_w6YeOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

ClueFlow is a post-training framework for long-horizon search agents that learns from attributed research-state transitions. It maintains explicit Evidence and Candidate records, then applies post-execution, gold-conditioned validation and source tracing to construct Source–State–Final attribution chains. These chains guide the selection of turns for exact-context supervision during SFT and provide turn-level advantage bonuses during PPO. The framework also recovers useful intermediate supervision from trajectories with incorrect final answers. Experiments evaluate its effectiveness on live-web and fixed-corpus search benchmarks.

Critical strengths
  1. The framework coherently connects evidence acquisition, research-state updates, and final answers to guide both SFT and reinforcement learning. Its ability to recover valid intermediate progress from unsuccessful trajectories addresses a practical limitation of learning exclusively from successful trajectories or terminal outcomes.
  2. Controlled experiments demonstrate meaningful benefits. Attribution-guided SFT achieves comparable accuracy to all-turn exact-context SFT using approximately 30% of its serialized training tokens. On BrowseComp-Plus, the complete PPO method improves accuracy by 4.10 percentage points over outcome-only PPO under matched initialization and training settings.
Critical weaknesses
  1. The advantage over existing fine-grained supervision methods remains insufficiently established. Comparisons against competitive direct step-scoring or critical-step selection methods under matched data, judge access, and computational budgets are missing. Consequently, the experiments do not establish whether the gains specifically require the proposed Source–State–Final structure.
  2. The component ablations do not adequately isolate the source of the PPO improvements. The Source-only and Full-chain variants differ in both bonus recipients and budget allocation. The current ablations do not distinguish the benefit of assigning credit to useful state updates from the benefit of giving extra credit to correct final answers. Separate ablations of the State and Final bonuses, with the total bonus budget held constant, would help identify which component drives the PPO improvement.
Questions for authors

I have three questions about the method and its evaluation:

  1. The current baselines do not isolate the value of the attribution chain. How does ClueFlow compare with a direct LLM-based turn scorer using the same judge and state representation?
  2. The PPO results combine bonuses for state updates and correct final answers. Does the improvement remain when the Final bonus is removed and its budget is redistributed to Source and State?
  3. The state representation and readiness rules are designed around constraint-based QA. Which parts of the method would need to change for open-ended research tasks?
Reviewer_FrvtOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ClueFlow is a agentic harness that maintains a persistent research state

  • evidences (facts retreived)
  • candidate records (proposed answers) with metadata below
    • with supporting evidence IDs,
    • satisfied/missing constraints,
    • Contraditions
    • Status (active/ready/rejected) At each step, the harness constructs the context from the state and previous tool calls, the agent makes a batch of tool calls to edit state (clue.add/update/discard) and get more information (browser.search/open/find).

Authors also train both Qwen3-30B-A3B and Qwen3.5-35B-A3B using SFT and PPO. Authors use per turn attribution (GPT5.5 is used to label Evidences if they are supporting the final answer and the browser turns that resulted in these evidences as positive steps). These attributed steps are used in SFT with original context (incorrect also) and other steps of correct trajectories are trained with chunked context. The attribution gives steps additional advantage in PPO.

Critical strengths
  • An interesting approach to obtain good training signals in this crowded research direction of agentic search training recipes. Thorough experiments/ablations to measure effectiveness.
  • Table 1 shows consistent good improvement with just SFT on ClueFlow harness on both the base models.
  • Used the incorrect trajectories by selecting the attributed turns. This has real world application where there are multiple runs often generated during experimentation/training/inference that can be later leveraged for continual improvement.
  • Table 3 shows the ClueFlow harness itself is competitive with other context management harness when using GLM-5.1. Which partially highlights that gain is not purely from GLM - 5.1.
Critical weaknesses
  • Many results appear to be single runs with no reported variance, given there are places where the comparison is in 1-3 pts range difference.
  • No OOD Benchmarks. As in it would help to evaluate on factual correctness + reasoning benchmarks such as HLE, GAIA, GPQA, MedXpert-QA etc to measure if there are new performance improvement due to better search capabilities and also to measure if there is any regression in reasoning capability.
Questions for authors
  • What happens if SFT is done only on attributed turns + Failure recovery turns?
  • Any particular reasoning for choosing GPT5.5 and GLM 5.1 for their respective tasks? Any study on effectiveness of GPT 5.5 as a judge given this is central to?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.