- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces DR PIXEL, a page-native Deep Research framework that directly uses document page images as the units for retrieval and reasoning, together with DRPIXELBENCH for evaluation. Experiments show that this paradigm improves both accuracy and retrieval efficiency over OCR-based pipelines. Building on this, applying counterfactual credit reweighting to agent training for Deep Research achieves substantial gains on Qwen3.6-27B.
Critical strengths
-
The paper is clearly written and generally well presented.
-
The paper presents a complete research pipeline, from dataset and benchmark construction, to systematic evaluation and analysis across multiple models, and finally to model training. The authors have invested substantial effort throughout, with extensive experiments and analyses.
Critical weaknesses
-
The construction process of DRPIXELBENCH introduces selection bias. Filtering out samples that can be answered without multimodal evidence is reasonable for tasks requiring visual information. However, supporting the broader claim that page-native DR generally outperforms OCR-based DR risks circular reasoning. Moreover, results on three external benchmarks do not include these comparisons.
-
DRPIXELBENCH may be relatively limited in the reasoning complexity typically emphasized in Deep Research. All queries are grounded in exactly one gold page. Further, retrieval and reasoning are likely coupled in Deep Research: as reasoning becomes more difficult, generating subsequent searches from intermediate conclusions may also become more challenging.
-
Results of the Text+Figure setting directly show that agents have difficulty invoking the figure-search tool, which does not seem equivalent to that cropped visual evidence is inherently inferior to full-page images.
-
The prior work motivating counterfactual credit reweighting is based on on-policy RL. Under SFT, once the student policy deviates from the teacher trajectory at inference time, it may enter states that it has never encountered during training, potentially leading to compounding errors.
Questions for authors
-
What's the necessity of sampling from the enriched abstract? This may bias the questions toward the paper's main conclusions and thereby limit query diversity and miss the details.
-
Text+Figure baseline rarely invokes figure search does not sufficiently demonstrate that surrounding text is useless. Are there controlled results using only visual elements for retrieval?
-
Reasoning trajectories of GPT-5.4 are not publicly exposed. How were them used for training? For example, the summary of the reasoning process?
If the authors can address my concerns, I will consider raising the score.
3 comments
Weakness 3 + Question 2
- Results of the Text+Figure setting directly show that agents have difficulty invoking the figure-search tool, which does not seem equivalent to that cropped visual evidence is inherently inferior to full-page images.
- Text+Figure baseline rarely invokes figure search does not sufficiently demonstrate that surrounding text is useless. Are there controlled results using only visual elements for retrieval?
Thank you for your concern.
We agree that the weak Text+Figure results could come from the agent underusing the figure tool rather than from cropped figures being inherently inferior. In fact, all reported Text+Figure results already use a system prompt that explicitly instructs the agent to use the figure tool whenever a question refers to a figure, chart, or table. We further add few-shot demonstrations of figure search and the results are as follows.
| Setup (DR Pixel-Base) | Prompt | Text searches | Figure searches | Acc ↑ |
|---|---|---|---|---|
| Text + Figure | Explicit Instruct | 15.2 | 0.8 | 28.6 |
| Text + Figure | Explicit Instruct + Few-Shot Demo | 15.0 | 1.3 | 30.8 |
Even so, figure searches rise only from 0.8 to 1.3 per question, and its accuracy rises only from 28.6 to 30.8, still far below page-native retrieval. These results suggest that imbalanced tool use is an inherent challenge when an agent must choose among multiple tools, and it is not easily fixed by prompting. Our page-native retrieval avoids this challenge by design.
A figure-only baseline is not meaningful in our setting, as the figure corpus contains only images extracted by the OCR parser, and only 20% of our questions are anchored in figures. Finally, we do not claim that the surrounding text is useless. Instead, our page-native approach keeps visual elements together with their surrounding text and layout, which is what makes it stronger.
Weakness 4
- The prior work motivating counterfactual credit reweighting is based on on-policy RL. Under SFT, once the student policy deviates from the teacher trajectory at inference time, it may enter states that it has never encountered during training, potentially leading to compounding errors.
Thank you for raising this concern. This problem of "distribution shift" is a general limitation of SFT rather than something specific to our counterfactual credit reweighting, since vanilla SFT on teacher trajectories faces the same issue.
Actually, we argue that counterfactual credit reweighting does not make the model more fragile once it deviates from the teacher path. As shown below, after missing the gold page at the first search, the student recovers more often than its teacher (29.6% vs. 27.2%), and this recovery behavior persists at deeper search depths. Moreover, repeated queries, which are a typical sign of degeneration, rise to 11.2% at searches 10-20 in the base model but stay at 0.2% in the student, close to the teacher’s 0%.
| Agent | Recovered after missing at search 1 ↑ | Recovered only at search ≥ 5 ↑ | Repeated queries at searches 10-20 ↓ |
|---|---|---|---|
| DR Pixel-Base | 14.6% | 7.0% | 11.2% |
| DR Pixel-Trained | 29.6% | 12.5% | 0.2% |
| GPT-5.4 (Teacher) | 27.2% | 16.1% | 0.0% |
This robustness comes from the teacher trajectories that already contain error-correction behavior. Our counterfactual credit preserves and emphasizes this recovery behavior, as it keeps and upweights the search that recovers from earlier failures in 98.6% of cases. Since masking removes a step from the loss instead of the context, the student still learns to recover from the failed attempts it sees during teacher forcing.
Question 1
- What's the necessity of sampling from the enriched abstract? This may bias the questions toward the paper's main conclusions and thereby limit query diversity and miss the details.
Thank you for mentioning this.
We sample from the enriched abstract to keep the question close to the workflow of real researchers. This design follows Qasper [1], where questions are written by researchers who have read only the title and abstract of a paper. Focusing questions on the paper’s core contributions is therefore an intended feature rather than a limitation. It keeps out minor questions that are too niche or do not reflect what real researchers care about. Meanwhile, to preserve fine-grained details, we enrich the abstract with content from the paper’s text chunks, figures, and tables before generating questions, as mentioned in line 187.
[1] Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, Matt Gardner. A Dataset of Information-Seeking Questions and Answers Anchored in Research Papers. NAACL 2021. arXiv:2105.03011.
Question 3
- Reasoning trajectories of GPT-5.4 are not publicly exposed. How were them used for training? For example, the summary of the reasoning process?
Thanks for your question. We didn’t use GPT-5.4’s hidden reasoning or using its returned reasoning summary. We explicitly ask the model to show its reasoning inside <think></think> in its output. When generating trajectories, the teacher was additionally instructed to use this visible reasoning to recap the evidence it had retrieved and prepare its next step.
Thank you very much for your insightful and constructive feedback on our submission. The following are our detailed responses.
Weakness 1
Thank you for your question. Here, we clarify as follows. At a high level, page-native deep research works well across question types and benchmarks, and its gains are larger when tasks place more emphasis on visual evidence.
Within DRPixelBench, the multimodal filter is applied only to visual questions (i.e., questions about tables, figures, or equations), while the remaining 216 of the 500 questions are text questions that are not filtered for visual necessity. On those Text questions, our page-native deep research performs similarly to the OCR-based pipelines, so the advantage of our method is not an artifact of filtering (detailed in Paper Figure 3, line 270). On visual questions that require multimodal evidence, the advantage of page-native deep research is much larger (detailed in Paper Figure 3, Line 270). This shows that the advantage of page-native deep research grows as tasks place more emphasis on visual evidence.
Beyond DRPixelBench, we add a new experiment that runs the OCR-based pipelines on the three external benchmarks (IRPapers, M3SciQA, and AirQA). With the same agent, judge, and budget, page-native deep research outperforms the OCR-based pipeline by 3.9 to 22.1 points while using fewer tool calls. Training the model to use the page-native search tool (DR Pixel-Trained) further widens this advantage on IRPapers (17.4 vs 22.1) and AirQA (3.9 vs 7.8).
Weakness 2
Thank you for raising this point. We clarify it from two aspects including benchmark construction and model behavior.
In terms of benchmark construction, we use single-page, short answer to keep evaluation simple and reliable, following common practice in agentic and deep research benchmarks such as BrowseComp [1], where answers are hard to find but easy to verify. However, a single gold page does not make reasoning easier. The gold page is located among 1.06M pages, and reaching it is challenging, it often requires first gathering evidence from multiple pages then filtering them to find the answer.
Our quantitative analysis of model behavior further supports this. Among the 52% of questions where the gold page is found, 57% of the gold pages are first retrieved at the second search or later, and 24% at the fifth search or later. Moreover, among questions whose gold page is first found after the first search, 76% of queries contain a specific discovery absent from the original question but surface from previous observations in the trajectory, such as a paper name, a table / figure number, or the name of the benchmark / method. This is exactly where the model generates subsequent searches from intermediate conclusions.
[1] Jason Wei, Zhiqing Sun, Spencer Papay, Scott McKinney, Jeffrey Han, Isa Fulford, Hyung Won Chung, Alex Tachard Passos, William Fedus, Amelia Glaese. BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents. arXiv:2504.12516, 2025.