acceptodds
Under review as a conference paper at ICLR 2027

DR Pixel: Page-Native Deep Research with Vision-Language Models

Abstract

Complex documents such as scientific papers contain rich information in tables, figures, and equations; however, existing deep research (DR) agents typically reduce papers to OCR-extracted text. We instead ask whether vision-language models (VLMs) can enable page-native deep research: directly retrieving and reasoning over rendered document page images. We introduce DR Pixel, the first end-to-end multimodal DR agent that uses rendered page images as its sole retrieval and reasoning unit, operating over an open corpus of 1.06M pages. To evaluate this setting, we construct DRPixelBench, a benchmark on scientific papers requiring reasoning over text, figures, tables, and equations. Surprisingly, page-native DR already works well without training, outperforming OCR-based baselines across state-of-the-art models like GPT-5.4, Gemini-3.5-Flash, and Qwen3.6-27B, while requiring substantially fewer tool calls. We next turn to DR training. A key challenge is that demonstration traces are inherently noisy: some reasoning and action steps are critical to reaching the correct answer, while others are redundant or failed attempts, yet SFT treats them all equally. To address this, we introduce counterfactual credit reweighting, which weights individual reasoning and action steps according to their contribution to the final answer. Our training improves retrieval recall by about 20% and final accuracy by over 10% on DRPixelBench, and likewise outperforms the untrained baseline on all four additional benchmarks. On out-of-distribution ViDoRe-V3, DR Pixel comes within 1.8% of an oracle that is given the gold pages directly. Our results suggest that the conventional OCR-then-retrieve pipeline is not only unnecessary but suboptimal: directly retrieving and reasoning over visual document pages makes complex DR both more effective and more efficient.

Primary area:
foundation or frontier models, including LLMs
Keywords:
vision-language models, deep research agents, visual document retrieval

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_tbxzOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces DR PIXEL, a page-native Deep Research framework that directly uses document page images as the units for retrieval and reasoning, together with DRPIXELBENCH for evaluation. Experiments show that this paradigm improves both accuracy and retrieval efficiency over OCR-based pipelines. Building on this, applying counterfactual credit reweighting to agent training for Deep Research achieves substantial gains on Qwen3.6-27B.

Critical strengths
  1. The paper is clearly written and generally well presented.

  2. The paper presents a complete research pipeline, from dataset and benchmark construction, to systematic evaluation and analysis across multiple models, and finally to model training. The authors have invested substantial effort throughout, with extensive experiments and analyses.

Critical weaknesses
  1. The construction process of DRPIXELBENCH introduces selection bias. Filtering out samples that can be answered without multimodal evidence is reasonable for tasks requiring visual information. However, supporting the broader claim that page-native DR generally outperforms OCR-based DR risks circular reasoning. Moreover, results on three external benchmarks do not include these comparisons.

  2. DRPIXELBENCH may be relatively limited in the reasoning complexity typically emphasized in Deep Research. All queries are grounded in exactly one gold page. Further, retrieval and reasoning are likely coupled in Deep Research: as reasoning becomes more difficult, generating subsequent searches from intermediate conclusions may also become more challenging.

  3. Results of the Text+Figure setting directly show that agents have difficulty invoking the figure-search tool, which does not seem equivalent to that cropped visual evidence is inherently inferior to full-page images.

  4. The prior work motivating counterfactual credit reweighting is based on on-policy RL. Under SFT, once the student policy deviates from the teacher trajectory at inference time, it may enter states that it has never encountered during training, potentially leading to compounding errors.

Questions for authors
  1. What's the necessity of sampling from the enriched abstract? This may bias the questions toward the paper's main conclusions and thereby limit query diversity and miss the details.

  2. Text+Figure baseline rarely invokes figure search does not sufficiently demonstrate that surrounding text is useless. Are there controlled results using only visual elements for retrieval?

  3. Reasoning trajectories of GPT-5.4 are not publicly exposed. How were them used for training? For example, the summary of the reasoning process?

If the authors can address my concerns, I will consider raising the score.

3 comments

Thank you very much for your insightful and constructive feedback on our submission. The following are our detailed responses.

Weakness 1

  1. The construction process of DRPIXELBENCH introduces selection bias. Filtering out samples that can be answered without multimodal evidence is reasonable for tasks requiring visual information. However, supporting the broader claim that page-native DR generally outperforms OCR-based DR risks circular reasoning. Moreover, results on three external benchmarks do not include these comparisons.

Thank you for your question. Here, we clarify as follows. At a high level, page-native deep research works well across question types and benchmarks, and its gains are larger when tasks place more emphasis on visual evidence.

Within DRPixelBench, the multimodal filter is applied only to visual questions (i.e., questions about tables, figures, or equations), while the remaining 216 of the 500 questions are text questions that are not filtered for visual necessity. On those Text questions, our page-native deep research performs similarly to the OCR-based pipelines, so the advantage of our method is not an artifact of filtering (detailed in Paper Figure 3, line 270). On visual questions that require multimodal evidence, the advantage of page-native deep research is much larger (detailed in Paper Figure 3, Line 270). This shows that the advantage of page-native deep research grows as tasks place more emphasis on visual evidence.

Beyond DRPixelBench, we add a new experiment that runs the OCR-based pipelines on the three external benchmarks (IRPapers, M3SciQA, and AirQA). With the same agent, judge, and budget, page-native deep research outperforms the OCR-based pipeline by 3.9 to 22.1 points while using fewer tool calls. Training the model to use the page-native search tool (DR Pixel-Trained) further widens this advantage on IRPapers (17.4 vs 22.1) and AirQA (3.9 vs 7.8).

InputModelIRPapersM3SciQAAirQA
Searches/qAccSearches/qAccSearches/qAcc
PageDR Pixel (base)5.665.47.849.87.942.7
PageDR Pixel (trained)5.079.49.464.27.052.3
Text (OCR)DR Pixel (base)9.148.011.530.510.538.8
Text (OCR)DR Pixel (trained)9.757.314.450.910.444.5

Weakness 2

2.DRPIXELBENCH may be relatively limited in the reasoning complexity typically emphasized in Deep Research. All queries are grounded in exactly one gold page. Further, retrieval and reasoning are likely coupled in Deep Research: as reasoning becomes more difficult, generating subsequent searches from intermediate conclusions may also become more challenging.

Thank you for raising this point. We clarify it from two aspects including benchmark construction and model behavior.

In terms of benchmark construction, we use single-page, short answer to keep evaluation simple and reliable, following common practice in agentic and deep research benchmarks such as BrowseComp [1], where answers are hard to find but easy to verify. However, a single gold page does not make reasoning easier. The gold page is located among 1.06M pages, and reaching it is challenging, it often requires first gathering evidence from multiple pages then filtering them to find the answer.

Our quantitative analysis of model behavior further supports this. Among the 52% of questions where the gold page is found, 57% of the gold pages are first retrieved at the second search or later, and 24% at the fifth search or later. Moreover, among questions whose gold page is first found after the first search, 76% of queries contain a specific discovery absent from the original question but surface from previous observations in the trajectory, such as a paper name, a table / figure number, or the name of the benchmark / method. This is exactly where the model generates subsequent searches from intermediate conclusions.

Search that first retrieved gold page123-45-910+
% of questions43%18%15%13%11%

[1] Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, Amelia Glaese. BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents. arXiv:2504.12516, 2025.

Weakness 3 + Question 2

  1. Results of the Text+Figure setting directly show that agents have difficulty invoking the figure-search tool, which does not seem equivalent to that cropped visual evidence is inherently inferior to full-page images.
  1. Text+Figure baseline rarely invokes figure search does not sufficiently demonstrate that surrounding text is useless. Are there controlled results using only visual elements for retrieval?

Thank you for your concern.

We agree that the weak Text+Figure results could come from the agent underusing the figure tool rather than from cropped figures being inherently inferior. In fact, all reported Text+Figure results already use a system prompt that explicitly instructs the agent to use the figure tool whenever a question refers to a figure, chart, or table. We further add few-shot demonstrations of figure search and the results are as follows.

Setup (DR Pixel-Base)PromptText searchesFigure searchesAcc ↑
Text + FigureExplicit Instruct15.20.828.6
Text + FigureExplicit Instruct + Few-Shot Demo15.01.330.8

Even so, figure searches rise only from 0.8 to 1.3 per question, and its accuracy rises only from 28.6 to 30.8, still far below page-native retrieval. These results suggest that imbalanced tool use is an inherent challenge when an agent must choose among multiple tools, and it is not easily fixed by prompting. Our page-native retrieval avoids this challenge by design.

A figure-only baseline is not meaningful in our setting, as the figure corpus contains only images extracted by the OCR parser, and only 20% of our questions are anchored in figures. Finally, we do not claim that the surrounding text is useless. Instead, our page-native approach keeps visual elements together with their surrounding text and layout, which is what makes it stronger.

Weakness 4

  1. The prior work motivating counterfactual credit reweighting is based on on-policy RL. Under SFT, once the student policy deviates from the teacher trajectory at inference time, it may enter states that it has never encountered during training, potentially leading to compounding errors.

Thank you for raising this concern. This problem of "distribution shift" is a general limitation of SFT rather than something specific to our counterfactual credit reweighting, since vanilla SFT on teacher trajectories faces the same issue.

Actually, we argue that counterfactual credit reweighting does not make the model more fragile once it deviates from the teacher path. As shown below, after missing the gold page at the first search, the student recovers more often than its teacher (29.6% vs. 27.2%), and this recovery behavior persists at deeper search depths. Moreover, repeated queries, which are a typical sign of degeneration, rise to 11.2% at searches 10-20 in the base model but stay at 0.2% in the student, close to the teacher’s 0%.

AgentRecovered after missing at search 1 ↑Recovered only at search ≥ 5 ↑Repeated queries at searches 10-20 ↓
DR Pixel-Base14.6%7.0%11.2%
DR Pixel-Trained29.6%12.5%0.2%
GPT-5.4 (Teacher)27.2%16.1%0.0%

This robustness comes from the teacher trajectories that already contain error-correction behavior. Our counterfactual credit preserves and emphasizes this recovery behavior, as it keeps and upweights the search that recovers from earlier failures in 98.6% of cases. Since masking removes a step from the loss instead of the context, the student still learns to recover from the failed attempts it sees during teacher forcing.

Question 1

  1. What's the necessity of sampling from the enriched abstract? This may bias the questions toward the paper's main conclusions and thereby limit query diversity and miss the details.

Thank you for mentioning this.

We sample from the enriched abstract to keep the question close to the workflow of real researchers. This design follows Qasper [1], where questions are written by researchers who have read only the title and abstract of a paper. Focusing questions on the paper’s core contributions is therefore an intended feature rather than a limitation. It keeps out minor questions that are too niche or do not reflect what real researchers care about. Meanwhile, to preserve fine-grained details, we enrich the abstract with content from the paper’s text chunks, figures, and tables before generating questions, as mentioned in line 187.

[1] Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, Matt Gardner. A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers. NAACL 2021. arXiv:2105.03011.

Question 3

  1. Reasoning trajectories of GPT-5.4 are not publicly exposed. How were them used for training? For example, the summary of the reasoning process?

Thanks for your question. We didn’t use GPT-5.4’s hidden reasoning or using its returned reasoning summary. We explicitly ask the model to show its reasoning inside <think></think> in its output. When generating trajectories, the teacher was additionally instructed to use this visible reasoning to recap the evidence it had retrieved and prepare its next step.

Reviewer_pTWmOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

DRPixel tackles the popular deep research problem from a vision-native perspective. The author first argues the page as the retrieval unit is superior than text chunks by comparing the performance using GPT, Gemini3.5-Flash, Qwen3.6 on the collected page-native benchmark. It further optimizes the qwen model with supervised fine-tuning. In addition, it proposes a counterfactual credit token-level mechanism to increase the effective retrieval invocations, the optimized model achieves better performance in IID datasets and yield consistent OOD improvements on ViDoRe V3 dataset.

Critical strengths
  1. The author not only proposes a popular hypothesis that many people could have "whether deep research can be page-native" but demonstrate systematic experiment design from evaluation to training. Therefore, it convinces audience the practical value of the problem setting.
  2. The visual-only question only versus text-only fine-tuning raises a very interesting point on how fine-tuning affects the hidden model states. It is a known fact that small-to-medium scale VLMs prune to attend on textual tokens instead of visual tokens. Visual queries apparently alleviates it by emphasizing visual attentions.
Critical weaknesses
  1. Although the story-telling of the paper is quite comprehensive, I found the paper may lack some discussions on any comparable baselines on training or problem setting. For example, SFT is chosen over RL without some further justifications.
Questions for authors
  1. In terms of training, how many samples are neccesary for model to converge since the credit calculation seems to be very time consuming?

  2. In the appendix, the system prompt used for evaluation, is the same prompt used for training?

3 comments

Thank you very much for your insightful and constructive feedback on our submission. The following are our detailed responses.

Weakness 1

  1. Although the story-telling of the paper is quite comprehensive, I found the paper may lack some discussions on any comparable baselines on training or problem setting. For example, SFT is chosen over RL without some further justifications.

Thank you for this suggestion. We first explain why we focus on SFT and then discuss comparable training methods and baselines.

SFT Justification. We do not claim that SFT is better than RL. We focus on SFT for two reasons. First, on-policy RL for long-horizon deep research agents is extremely expensive, as there are multiple long-horizon rollouts per question. Under the limited compute of academia, we were not able to run RL at scale (over 27B Qwen models), and we believe this constraint is shared by many academic labs. Second, SFT distills the search behavior of a strong teacher into the student, and is commonly used as the initialization before RL. Improving the SFT stage is therefore valuable on its own, both as a standalone recipe and as a better starting point for RL.

Method Comparison. IGPO [1] and TIPS [2] are the closest to our method, with both rewarding each turn from search agents by its contribution to the likelihood of the gold answer, but within on-policy RL. Beyond Uniform Credit [3] is also RL-based, but uses token-level counterfactual credit and is for math reasoning. CFT [4] is the closest SFT-based method compared with ours, but its token-level counterfactuals are designed for short math reasoning and become prohibitively expensive on long agentic trajectories. In contrast, our method measures and weights the contribution of each turn in the SFT stage. The credits are computed once on fixed teacher trajectories with no rollouts and no online recomputation, which is far cheaper than on-policy RL and token-level counterfactuals.

Comparable Baselines. VRAG-RL [5] and VISOR [6] are page-native agents designed for visual RAG, while DR Pixel targets visual deep research, which requires iteratively reformulating queries based on intermediate findings. On our benchmark under comparable settings, they issue far fewer tool calls and fail to retrieve the evidence that deep research requires.

MethodSearches/qPgHitAcc|PgAcc
VRAG-RL1.617.219.811.0
VISOR3.321.656.530.8
DR Pixel (base)8.333.863.938.6
DR Pixel (trained)8.952.067.752.1

We would greatly appreciate it if the reviewer could point us to other papers on training methods or problem settings that we may have missed, and we will discuss them in the revised version.

[1] Guoqing Wang, Sunhao Dai, Guangze Ye, Zeyu Gan, Wei Yao, Yong Deng, Xiaofeng Wu, Zhenzhe Ying. Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents. ICLR 2026. arXiv:2510.14967.

[2] Yutao Xie, Nathaniel Thomas, Nicklas Hansen, Yang Fu, Li Erran Li, Xiaolong Wang. TIPS: Turn-Level Information-Potential Reward Shaping for Search-Augmented LLMs. ICLR 2026. arXiv:2603.22293.

[3] Mykola Khandoga, Rui Yuan, Vinay Kumar Sankarapu. Beyond Uniform Credit: Causal Credit Assignment for Policy Optimization. arXiv:2602.09331, 2026.

[4] Zhiwen Ruan, Yixia Li, He Zhu, Yun Chen, Peng Li, Yang Liu, Guanhua Chen. Enhancing Large Language Model Reasoning via Selective Critical Token Fine-Tuning. arXiv:2510.10974, 2025.

[5] Qiuchen Wang, Ruixue Ding, Yu Zeng, Zehui Chen, Lin Chen, Shihang Wang, Pengjun Xie, Fei Huang, Feng Zhao. VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learning. NeurIPS 2025. arXiv:2505.22019.

[6] Yucheng Shen, Jiulong Wu, Jizhou Huang, Dawei Yin, Lingyong Yan, Min Cao. VISOR: Agentic Visual Retrieval-Augmented Generation via Iterative Search and Over-horizon Reasoning. ACM MM 2026. arXiv:2604.09508.

Question 1

  1. In terms of training, how many samples are neccesary for model to converge since the credit calculation seems to be very time consuming?

Thank you for this question. Here we provide the training curve and the required time for counterfactual credit calculation. Overall, the model learns efficiently from a small number of trajectories and credit computation is much cheaper than RL.

StepTrajectoriesSearches/qPgHitAcc
0 (base)08.333.838.6
203208.049.046.4
406408.950.849.0
609607.650.649.2
801,2809.553.050.8
1001,6009.353.348.9
1201,9207.949.749.7
1402,2408.551.951.4
156 (DR Pixel-Trained)2,4968.952.052.1

The above table shows the training curve. With only 320 trajectories (20 steps), PgHit already rises from 33.8 to 49.0 and the accuracy rise from 38.6 to 46.4. Further training continues to improve accuracy, which the model reaches the best accuracy at the final step.

Computing the counterfactual credit for all 2496 trajectories takes 23 GPU hours on H200. For reference, a single evaluation pass over the 500 evaluation questions takes about 5 GPU hours, so one rollout pass over the 2496 training questions will at most take 25 GPU hours. As on-policy RL samples multiple rollouts per question, our counterfactual credit calculation is much cheaper.

Question 2

2.In the appendix, the system prompt used for evaluation, is the same prompt used for training?

Thank you for your question. Yes, all training trajectories use the same system prompt for evaluation.

Thanks for the clarification, I will maintain my positive score.

Reviewer_UdizOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

It is well known that searching relevant documents using only extracted text is not optimal because the layout and the visual telements get lost and might lead to bad retrival and bad answers. The paper presents DR Pixel, an agent that answers questions about scientific papers by searching and reading page images instead of extracted text. It iteratively searches for relevant pages and uses what it finds to guide the next search. The authors also introduce a benchmark of 500 questions over about one million pages and a training method that gives more weight to useful reasoning and search steps. Their experiments show better accuracy than the tested text-based pipelines and fewer searches than standard supervised training.

Critical strengths

The problem is an important problem in the RAG community. preserving visual evidence and document structure during scientific information seeking was shown to be beneficial. They show an improvement over OCR based methods. They introduce a benchmark to show their results on. the benchmark itself is a nice contribution to the community. They report meaningful gains over vanilla SFT for training the retriever both in accuracy and efficiency.

Critical weaknesses

My main concerns are as follows:

  1. RAG or deep research by page images is not novel and some of the literature was even cited by the authors. Iterative visual retrieval and reasoning already appear in VRAG-RL, VISOR, and, in a different document setting, Doc-React.
  2. There are no direct comparisons to the methods mensioned above despite their relevance.
  3. There are also very close relevant works for search credit, counterfactual selective SFT, and reasoning credit. so both the loop and the trainig are not quite novel.
  4. Training novelty needs narrower positioning. the specific combination of offline scoring, channel-specific credit, ranking, and coupling may be useful, but the underlying credit-assignment principles are established. The closest training precedents above are not discussed.
  5. There are no comparisons to simpler stopping criteria.
  6. The authors mension that the difference is the depth of titerations needed 10-20 vs shallow but i did not see sufficient support for it in the experiments. there is a comparison to one shot in the appendix.
  7. Overall contribution remains uncertain. the benchmark could independently justify the paper, but stronger evidence is needed that it measures a distinct capability beyond existing benchmarks.
Questions for authors

What inference capability distinguishes DR Pixel from existing visual ReAct agents?

How does the training method differ from other works like Beyond Uniform Credit, TIPS, IGPO, and CFT?

What happens to accuracy and evidence recall at budgets of 1, 2, 5, 10, and 20 searches?

How much does adaptive query generation improve over queries generated independently before observing retrieval results?

Can you provide examples or annotations showing necessary intermediate discoveries, and report when sufficient evidence first becomes available?

What capability does DRPixelBench measure beyond existing benchmarks when all are evaluated with matched agents and retrieval configurations?

6 comments

Thank you very much for your insightful and constructive feedback on our submission. The following are our detailed responses.

Weakness 1 + Weakness 2 + Question 1

  1. RAG or deep research by page images is not novel and some of the literature was even cited by the authors. Iterative visual retrieval and reasoning already appear in VRAG-RL, VISOR, and, in a different document setting, Doc-React.
  1. There are no direct comparisons to the methods mensioned above despite their relevance.

1.What inference capability distinguishes DR Pixel from existing visual ReAct agents?

Thank you for this valuable feedback. To the best of our knowledge, DR Pixel is the first open-source end-to-end page-native Deep Research agent. VRAG-RL [1] and VISOR [2] are designed for visual RAG, while ours is designed for visual deep research. In conventional RAG, the original question can often serve directly as the retrieval query, requiring limited search depth. In contrast, as discussed in Section 4.1, deep research is a more challenging task, where the agent must break down complex questions into sub-queries, perform multiple rounds of search, and decide what to do next based on intermediate discoveries. As a result, methods that work well for conventional RAG may not necessarily perform well in deep research, where the agent must also learn how to search and act effectively.

This difference in capability requirements is directly reflected in their performance on our benchmark. We evaluate VRAG-RL and VISOR on our benchmark under comparable settings.

MethodSearches/qPgHitAcc|PgAcc
VRAG-RL1.617.219.811.0
VISOR3.321.656.530.8
DR Pixel (base)8.333.863.938.6
DR Pixel (trained)8.952.067.752.1

VRAG-RL and VISOR show limited capability in iterative retrieval with significantly fewer tool calls and final accuracy, showing that an agent design for visual RAG is not enough for visual deep research tasks.

[1] Qiuchen Wang, Ruixue Ding, Yu Zeng, Zehui Chen, Lin Chen, Shihang Wang, Pengjun Xie, Fei Huang, Feng Zhao. VRAG-RL: Empower Vision-Perception-Based RAG for Visually Rich Information Understanding via Iterative Reasoning with Reinforcement Learning. NeurIPS 2025. arXiv:2505.22019.

[2] Yucheng Shen, Jiulong Wu, Jizhou Huang, Dawei Yin, Lingyong Yan, Min Cao. VISOR: Agentic Visual Retrieval-Augmented Generation via Iterative Search and Over-horizon Reasoning. ACM MM 2026. arXiv:2604.09508.

Weakness 3 + Weakness 4 + Question 2

3.There are also very close relevant works for search credit, counterfactual selective SFT, and reasoning credit. so both the loop and the trainig are not quite novel.

4.Training novelty needs narrower positioning. the specific combination of offline scoring, channel-specific credit, ranking, and coupling may be useful, but the underlying credit-assignment principles are established. The closest training precedents above are not discussed.

2.How does the training method differ from other works like Beyond Uniform Credit, TIPS, IGPO, and CFT?

Thank you for pointing out these works. Here, we give further comparison with those methods on credit assignment and will add this to our related work in the revised version.

IGPO [1] and TIPS [2] are conceptually closest to our method, with both rewarding each turn of a search agent by its contribution to the likelihood of the gold answer. However, these two methods are under on-policy RL setting, which is very expensive because it samples many rollouts of the current policy per update and recomputes the credit online. Beyond Uniform Credit [3] also works inside RL, with token-level credit for math reasoning. CFT [4] is the closest SFT-based method compared with ours. It perturbs individual tokens and trains only on those whose replacement flips the final answer. However, as this method is designed for math reasoning with relatively shorter output, applying token-level counterfactual credit to agentic search tasks with much longer trajectories can be prohibitively expensive.

In summary, existing turn-level credit methods are specific to on-policy RL, whereas our method introduces turn-level counterfactual credit to SFT and is significantly more cost-efficient. For all 2,496 training trajectories, this takes 23 GPU hours on H200, roughly the cost of a single rollout pass over the training set (at most about 25 GPU hours), whereas on-policy RL samples multiple rollouts per question at every update.

[1] Guoqing Wang, Sunhao Dai, Guangze Ye, Zeyu Gan, Wei Yao, Yong Deng, Xiaofeng Wu, Zhenzhe Ying. Information Gain-based Policy Optimization: A Simple and Effective Approach for Multi-Turn Search Agents. ICLR 2026. arXiv:2510.14967.

[2] Yutao Xie, Nathaniel Thomas, Nicklas Hansen, Yang Fu, Li Erran Li, Xiaolong Wang. TIPS: Turn-Level Information-Potential Reward Shaping for Search-Augmented LLMs. ICLR 2026. arXiv:2603.22293.

[3] Mykola Khandoga, Rui Yuan, Vinay Kumar Sankarapu. Beyond Uniform Credit: Causal Credit Assignment for Policy Optimization. arXiv:2602.09331, 2026.

[4] Zhiwen Ruan, Yixia Li, He Zhu, Yun Chen, Peng Li, Yang Liu, Guanhua Chen. Enhancing Large Language Model Reasoning via Selective Critical Token Fine-Tuning. arXiv:2510.10974, 2025.

Weakness 5

5.There are no comparisons to simpler stopping criteria.

Thank you for this suggestion. Here we add a simple stopping baseline where the training trajectory is truncated immediately after the gold page is retrieved and followed directly by the final answer. All the other training parameter remains unchanged.

MethodSearches/qPgHitStop after Retrieve Gold PageFurther SearchAcc
DR Pixel (vanilla SFT)14.053.66%9.349.0
DR Pixel (trained)8.952.010%4.252.1
DR Pixel (vanilla truncation)7.348.927%2.547.2

While vanilla truncation reduces approximately one tool call (7.3 vs 8.9), it also hurts the retrieval where the PgHit drops by 3.1 points. Also, the truncated model stops immediately after retrieving the gold page 27% compared with 10% for DR Pixel-Trained. This behavior weakens the model in verifying the answer, which its accuracy drops by 4.9 points. DR Pixel-Trained outperforms both vanilla SFT and vanilla truncation by reaching a balance between efficiency (8.9 searches) and verification (4.2 further searches).

Question 3

3.What happens to accuracy and evidence recall at budgets of 1, 2, 5, 10, and 20 searches?

Thank you for your question.

Here we compare both DR Pixel-Base and DR Pixel-Trained under those five different search budgets.

ModelBudgetSearches/qPgHitAcc
DR Pixel-Base11.020.824.6
DR Pixel-Base21.828.234.0
DR Pixel-Base53.434.438.8
DR Pixel-Base105.935.638.0
DR Pixel-Base208.333.838.6
DR Pixel-Trained11.029.230.4
DR Pixel-Trained21.941.444.4
DR Pixel-Trained54.248.846.6
DR Pixel-Trained106.450.849.0
DR Pixel-Trained208.952.052.1

The base model saturates very early. Its PgHit peaks at 35.6 at a budget of 10, while its accuracy peaks at 38.8 at a budget of 5. With no further gains from additional budget, the base model shows limited capability to use the search tool to dig for further evidence. In contrast, the PgHit and Acc of our DR Pixel-Trained model continue to increase with larger search budget, from 29.2 to 52.0 and from 30.4 to 52.1 respectively. This shows that our training teaches the agent to make effective use of additional search budget.

Question 4

4.How much does adaptive query generation improve over queries generated independently before observing retrieval results?

Thank you for this question. We compare adaptive search with a non-adaptive independent generation baseline where the model writes 8 queries independently for a single time.

SettingModelSearches/qPgHitAcc
Queries written up independently (8 queries)Base8.036.236.8
Adaptive, budget 5Base3.434.438.8
Adaptive, budget 10Base5.935.638.0
Adaptive, budget 20Base8.333.838.6
Queries written up independently (8 queries)Trained8.035.337.7
Adaptive, budget 5Trained4.248.846.6
Adaptive, budget 10Trained6.450.849.0
Adaptive, budget 20Trained8.952.052.1

DR Pixel-Trained benefits substantially from adaptively generating queries based on previous retrieval results. Even under a maximum search budget of 5, adaptive search has surpassed independent queries by nearly 9 points in accuracy (37.7 vs 46.6). This reflects that the DR Pixel-Trained has effectively learned to refine its search query based on the retrieved results. In contrast, DR Pixel-Base only benefits slightly from adaptive search (36.8 vs 38.8).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.