acceptodds
Under review as a conference paper at ICLR 2027

HAPRL: Human-Anchored Process Reinforcement Learning for Visual Search Agent

Abstract

Multi-turn visual search agents answer questions about high-resolution images by iteratively deciding where to look. Reinforcement learning for these agents rewards only the final answer, leaving the search process unsupervised. Consequently, faulty routes in which the reasoning process is erroneous yet the final result is correct arise frequently, which in turn leads to ineffective training, i.e., scaling along the wrong paths. In this paper, we introduce HaPRL, the first framework to reinforce the search process with human search behavior. We first build an annotation platform and collect 1K+ human-annotated data with fine-grained behavioral signals. During training, a carefully designed judge scores each rollout with task-adaptive weights, anchored on the distilled trace of how a human annotator actually searched the same image. Extensive experiments show that HaPRL consistently outperforms outcome-based RL, and early-stage process supervision yields 6.7× more improvement in subsequent outcome-based scaling. Our results also demonstrate the importance of aligning model behavior with human process annotation signals, which offer new insight into the training of foundation models.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Visual Search Agent, Reinforced Reasoning, Process Supervision

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YFFROpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

HaPRL trains visual-search agents with a reward based on human search traces. A frozen judge compares each rollout with a human reference, and incorrect answers receive zero reward. Experiments compare this approach with outcome-only RL on four backbones and study subsequent outcome-only training.

Critical strengths

The human traces outperform teacher-generated paths by 4.4 points and it gives 5.87 points gain over Outcome-RL from a cold start close to the original backbone's accuracy. These address whether the gains come from an evidence reference alone or from recovering the accuracy lost during SFT. The author finds larger gains from subsequent outcome-only training after process supervision, across three seeds. The reported improvements across all four backbones support the value of the training method.

Critical weaknesses
  1. Table 2 scores each method on the questions it answers correctly. The Hard score falling from 71.54 to 65.56 could therefore reflect a change in the questions being scored, rather than worse search on the same questions. This does not establish the reward hacking claimed in Section 4.2. The scoring inputs also need clarification. Sections 3.1 and B.5 describe human traces for training questions, while Table 2 covers held-out benchmarks. Appendix B.4 lists the original image and a textual rollout without stating whether the judge sees returned crops. These details matter for judging whether the agent saw the visual evidence needed to answer.

  2. Two Hard accuracies do not fit the stated evaluation setup. Section 4 specifies one greedy rollout per question, Appendix B.1 uses binary correctness, and the released split has 106 questions [1]. The 20.09 and 27.92 scores for the two 8B HaPRL models cannot come from integer correct-answer counts over 106, rounded to two decimals. A different subset or run average could explain them, but the exact Hard gains remain uncertain until the calculation is specified.

[1] Mini-o3: Scaling Up Reasoning Patterns and Interaction Turns for Visual Search. https://arxiv.org/abs/2509.07969v1

Questions for authors
  1. For Table 2, where did the held-out human references come from, and did the judge receive the actual crops?
  2. How many questions were used for the two Hard scores, and were the results averaged over multiple runs?
3 comments
AuthorsOpenReview ↗

Response to Reviewer YFFR [1/2]

We sincerely thank the reviewer for the constructive comments and valuable suggestions. We address each point below and will revise the manuscript accordingly.

W1 & Q1: Paired Validation of Table 2 and Judge Input Clarification

(1) Paired process-score analysis on identical questions. Scoring each policy on its own answer-correct subset means the Hard drop from 71.54 to 65.56 can reflect a change in which questions are scored. On Qwen3-VL-4B, we extract the 707-question three-way intersection answered correctly by SFT, Outcome-RL, and HaPRL and the 801-question two-way intersection answered correctly by Outcome-RL and HaPRL, and compare scores on the exact same questions:

Benchmark3-Way Overlap N3N_3+SFT+Outcome-RL+HaPRLΔ\Delta(Out$-$SFT)Δ\Delta(HaP$-$Out)2-Way Overlap N2N_2+Outcome-RL+HaPRLΔ\Delta(HaP$-$Out)
VisualProbe Easy3385.3184.5887.22−0.73+2.643684.4287.15+2.73
VisualProbe Medium3186.6287.1487.71+0.52+0.574086.9887.64+0.66
VisualProbe Hard472.3665.9476.88−6.42+10.94666.1576.42+10.27
V*QA7981.0283.4783.62+2.45+0.159783.3183.58+0.27
HR-Bench 4K29886.7187.3388.84+0.62+1.5132687.2888.79+1.51
HR-Bench 8K26282.9886.0587.71+3.07+1.6629685.9987.66+1.67
Macro Avg.70782.5082.4285.33−0.08+2.9180182.3685.21+2.85

On the exact same 707 (and 801) questions, HaPRL scores higher than Outcome-RL across all six subsets (+2.91 and +2.85 macro). On a 100-question stratified subsample, 3 blinded human raters (83.45 / 84.76 / 86.68, question-level judge-human ρ=0.74\rho = 0.74) and a third-party GPT-4o judge (83.74 / 84.92 / 86.78) yield the identical ranking.

(2) Held-out references and whether the judge receives cropped images (Q1).

  • No hidden human references at test time: All 1,104 human search traces are collected exclusively on the VisualProbe training set. For Table 2 on the six held-out benchmarks, the frozen judge runs in reference-free mode: it applies the same five-dimensional rubric and task-adaptive weights without any human reference trace, evaluating all three policies under the exact same protocol. We will state this training/evaluation scoring interface explicitly in the revised manuscript.
  • The judge does not receive returned crop pixels: The process judge takes the full-resolution original image I0I_0 + the text-serialized multi-turn rollout (each round's <think> reasoning, each <grounding> call's source view and exact normalized coordinates [x0, y0, x1, y1], and the final answer; training calls also include the human trace and final_bboxes). Although cropped image patches are not passed again as separate images, holding both the full original image and exact normalized crop coordinates allows the judge to check directly on I0I_0 whether each crop covers the visual evidence needed for the answer. We will add this input specification and the exact prompts for both judges (qa_process.py and qa_process_v1.py in the supplementary code) to the revised manuscript.
AuthorsOpenReview ↗

Response to Reviewer YFFR [2/2]

W2 & Q2: Exact Calculation of the Two Hard Scores and Unified Evaluation Protocol

Thank you for checking the integer denominators. The discrepancy arose because the manuscript omitted one detail of our unified evaluation aggregation rule, specifically that unparseable rollouts are excluded from that run's denominator:

  1. To reduce variance from multi-turn online serving and API judging, every cell in Table 1 reports the arithmetic mean accuracy across 2 independent greedy evaluation runs. Each run executes 1 greedy rollout per question (temperature = 0) with binary semantic grading; if a rollout fails to produce a parseable answer within the 6-round budget (e.g., an unclosed <answer> tag or judge timeout, i.e., an invalid bad case), that instance is excluded from the valid denominator of that run.
  2. Under this single rule across all 96 cells of Table 1, 90 cells have zero bad cases and identical outcomes across both runs, reducing to standard integer ratios k/Nk / N.
  3. Only 6 cells in the entire table have a slightly smaller valid denominator than NN due to occasional unparseable rollouts, and the arithmetic mean of their two runs reproduces the reported numbers to two decimal places:
Model & BenchmarkRun 1 (Correct / Valid)Run 2 (Correct / Valid)2-Run Arithmetic Mean (Reported in Table 1)
Qwen3-VL-4B Base - V*QA (N=191N=191)124/191=64.92%124 / 191 = 64.92\%127/183=69.40%127 / 183 = 69.40\%67.16
Qwen3-VL-8B Base - V*QA (N=191N=191)134/189=70.90%134 / 189 = 70.90\%133/184=72.28%133 / 184 = 72.28\%71.59
Qwen3-VL-8B +HaPRL - Hard (N=106N=106)22/105=20.95%22 / 105 = 20.95\%20/104=19.23%20 / 104 = 19.23\%20.09
Qwen3-VL-8B +HaPRL - V*QA (N=191N=191)136/187=72.73%136 / 187 = 72.73\%136/187=72.73%136 / 187 = 72.73\%72.73
LLaVA-1.5-8B +HaPRL - Easy (N=141N=141)74/140=52.86%74 / 140 = 52.86\%66/137=48.18%66 / 137 = 48.18\%50.52
LLaVA-1.5-8B +HaPRL - Hard (N=106N=106)30/105=28.57%30 / 105 = 28.57\%27/99=27.27%27 / 99 = 27.27\%27.92

Conservative full-denominator lower bound on Hard: To rule out any upward bias from excluding unparseable rollouts, even under the strictest accounting that treats every unparseable rollout as incorrect over the full denominator of 106 and takes the lower of the two runs:

  • Qwen3-VL-8B +HaPRL has a lower bound of 20/106=18.87%20/106 = \mathbf{18.87\%} (2-run mean 21/106=19.81%21/106 = \mathbf{19.81\%}), exceeding matched Outcome-RL (15.09%) by +3.78 points at the lower bound (+4.72 on the mean);
  • LLaVA-1.5-8B +HaPRL has a lower bound of 27/106=25.47%27/106 = \mathbf{25.47\%} (2-run mean 28.5/106=26.89%28.5/106 = \mathbf{26.89\%}), exceeding matched Outcome-RL (12.26%) by +13.21 points at the lower bound (+14.63 on the mean).

Thus, the gains of both 8B models on VisualProbe Hard hold under every denominator convention and individual run. We apologize for omitting the unparseable-instance rule in the initial submission and will state it explicitly in Section 4 and the caption of Table 1.

AuthorsOpenReview ↗

Sincere Thanks and Follow-up to Reviewer YFFR

Dear Reviewer YFFR,

Thank you again for taking the time to review our manuscript and share your feedback. We have responded to your comments and revised the manuscript accordingly. We hope that our responses and revisions address your concerns. If any questions or concerns remain, we would be happy to address them during the discussion period.

Best regards,

The Authors

Reviewer_jcQzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes HaPRL, a reinforcement-learning framework for visual-search agents that uses human search traces to provide process-level supervision. A frozen vision-language judge scores each rollout against a trace collected for the same image-question pair, using task-adaptive criteria; the process score is multiplied by answer correctness and used as the GRPO reward. The authors collect 1,104 human-annotated traces and evaluate the method across four backbones and six visual-search benchmarks. They report gains over outcome-only RL in both answer accuracy and judged search quality, along with ablations of trace alignment, annotation noise, reward references, and a two-stage training schedule.

Critical strengths

1.The paper targets a real weakness of outcome-only RL for visual search: a correct answer can be reached through redundant or poorly grounded crops. Using a trace from the same image-question pair gives the judge a task-specific reference, rather than rewarding generic tool use or a larger number of crops.

  1. The paper includes useful controls beyond final-answer accuracy. The shuffled-trace and judge-only ablations test whether the matched human reference matters. At the same time, the teacher-generated path and final-evidence-box baselines help separate the value of a full trace from simpler evidence references. These experiments make the paper more substantial than a simple reward-design proposal.
Critical weaknesses

1、The algorithmic contribution appears incremental relative to existing visual-search RL methods, including DeepEyes, Pixel Reasoner, and Chain-of-Focus, which already train agents to improve visual interaction through reinforcement learning and auxiliary reward signals. HaPRL’s main difference is to use a matched human search trace as the reference for a frozen judge, then multiply that score by answer correctness. This is a reasonable and potentially useful design, but the paper does not establish a new policy-optimization or credit-assignment mechanism beyond this reward construction. Appendix Table 8 does compare human traces with teacher-generated paths and simpler evidence references, so the concern is not that these alternatives are missing; rather, it is whether the additional benefit of human traces is sufficiently distinctive to support the paper’s broader process-supervision claim.

2、The process reward is supplied by a frozen VLM judge, and the same rubric-based score is also used to report search quality. This creates a potential circularity: the policy is optimized to score well under the judge, and the reported process-quality improvement is measured by that judge. The shuffled-trace and judge-only ablations show that the matched trace matters, but do not independently establish that the judge reliably distinguishes grounded search from a plausible-looking route that differs from the human trace. A held-out evaluation against independent human ratings, including alternative valid search paths, would provide stronger evidence.

3、The correctness gate also limits what the method can learn from failed searches. Under Eq. (4), every incorrect trajectory receives zero process reward, even if some intermediate crops reveal useful evidence; a group in which all sampled answers are wrong therefore provides no reward variation from HaPRL. This is a design trade-off rather than necessarily a flaw, since the gate avoids reinforcing trajectories that fail the task. Still, it leaves open whether useful partial progress could be preserved when answer success is sparse. In addition, the 1,104 reference traces come from one annotator and VisualProbe training questions. The paper includes noise and seed analyses, but it remains unclear how well the reward generalizes across annotators, alternative valid search styles, or settings beyond the current benchmarks.

Questions for authors
  1. How reliable is the frozen process judge? Can the authors report inter-annotator agreement and judge–human correlation on a held-out set, including cases where a valid search path differs from the recorded human trace? How robust are the judgments across annotators and alternative valid search paths?

  2. How does answer-gated process reward affect learning from failed trajectories? How often do rollout groups contain only incorrect answers and therefore receive no process-reward variation? Do the gains mainly come from ranking already-correct trajectories, and could partial evidence-progress signals improve learning in these cases without rewarding incorrect final answers?

  3. What is the unique contribution of human search traces? Given the teacher-generated evidence-path and final-evidence-box comparisons in Appendix Table 8, what specific information in human traces accounts for the remaining advantage? Can the authors disentangle the value of human search behavior from that of a more complete, ordered evidence reference, while reporting run-to-run variation on the main benchmarks?

4 comments
AuthorsOpenReview ↗

Response to Reviewer jcQz [1/3]

We sincerely thank the reviewer for the constructive comments and valuable suggestions. We address each point below and will revise the manuscript accordingly.

W1 & Q3: Relation to Prior Visual-Search RL and Disentangling the Value of Human Traces

We clarify HaPRL's distinct contribution along two fronts: method positioning and information-ablation experiments.

(1) Orthogonal positioning on the supervision-signal axis. Key design differences between prior visual-search RL methods and HaPRL are summarized below:

MethodOnline RL RewardProcess Anchor / ReferenceInstance-Specific Difficulty & Region CalibrationDistinguishes Grounded vs. Lucky/Redundant Correct Rollouts
DeepEyes / Mini-o3Outcome answer correctness (+ format / over-turn masking)None✗✗ (equal advantage for all correct rollouts in a group)
Chain-of-FocusOutcome answer correctness + format rewardNone✗✗
Pixel ReasonerOutcome answer + generic curiosity/operation bonusNo instance reference✗Encourages generic exploration without anchoring to target evidence
HaPRL (Ours)Answer-correctness gate ×\times 5D task-adaptive process scoreHuman search trace on the same image-question pair✓ (includes refinement granularity & hypothesis rejection)✓ (ranks correct rollouts within each group by evidence quality)

We intentionally hold the GRPO optimizer and hyperparameters fixed to isolate a specific hypothesis: a primary bottleneck in multi-turn visual-search RL is the absence of instance-specific process supervision. As an orthogonal reward-layer formulation, HaPRL plugs directly into any of the frameworks above. Early human process supervision also amplifies subsequent annotation-free outcome-RL scaling by 6.6–6.7$\times$ (Tables 4 and 10; 3-seed gain of 5.83±0.265.83 \pm 0.26 vs. 0.88±0.060.88 \pm 0.06), showing a two-stage transfer pattern where aligning search behavior first improves later outcome scaling.

(2) Ablating the information in human traces + multi-seed variance on the main benchmarks (Q3). To identify what accounts for the +4.41-point lead of human traces (44.26) over ordered teacher-generated paths (39.85), we progressively strip components from the human traces under the exact same training setup:

Reward Reference (all other training settings fixed)Process Information Retained6-Benchmark Macro Avg.Δ\Delta vs. Full HaPRL
Full human search trace (HaPRL)Target location + search order + rejected hypotheses (dead ends) + observed evidence text44.26-
Remove rejected hypotheses & revisits (keep direct steps only)Target location + pruned order + observed evidence text (no dead-end branches)41.18−3.08
Shuffle round order within each traceTarget location + local region set + observed evidence text (breaks coarse-to-fine order)40.62−3.64
Remove observed evidence & difficulty text (keep normalized box sequence only)Target location + search order39.91−4.35
Post-hoc teacher-generated path (Table 8)Answer-conditioned target location + idealized order (no dead-end branches by construction)39.85−4.41
Final human evidence-box coverage reward (Table 8)Final evidence location only38.61−5.65
Outcome-only RL (Table 1)No process reference33.98−10.28

Two findings follow from this ablation:

  1. Rejected hypotheses (dead ends) and coarse-to-fine temporal order drive the remaining gain. A teacher model conditioned on the ground-truth answer writes a direct path to the target, omitting the "initial check -> distractor rejection -> refocus (useful_reset)" sequence that occurs under high-resolution clutter. Removing dead-end branches from human traces costs 3.08 points, and shuffling round order costs 3.64 points; stripping the trace down to an ordered box sequence (39.91) reduces accuracy to the teacher-path level (39.85). Human traces calibrate instance-specific visual ambiguity and plausible exploration bounds, enabling the judge to separate productive distractor elimination from blind repetitive cropping.
  2. Run-to-run variance on the main benchmarks is small. Running 3 random seeds under the main-table setup (Qwen3-VL-4B, 3 epochs, with every subset score computed from integer correct counts) gives 33.98 / 33.63 / 33.80 (33.80±0.1833.80 \pm 0.18) for Outcome-RL and 44.26 / 43.67 / 44.09 (44.01±0.3044.01 \pm 0.30) for HaPRL. Per-benchmark 3-seed means and standard deviations for HaPRL are: Easy 40.90±0.4040.90 \pm 0.40, Medium 26.37±0.2126.37 \pm 0.21, Hard 20.44±0.5420.44 \pm 0.54, V*QA 67.71±0.3067.71 \pm 0.30, HR-4K 56.25±0.2556.25 \pm 0.25, and HR-8K 52.38±0.2552.38 \pm 0.25. The net gain over Outcome-RL (+10.20±0.14+10.20 \pm 0.14) is 19$\times$ the largest single-benchmark seed SD (0.54), and the +4.41 margin over teacher paths exceeds the macro seed SD by over 14$\times$.
AuthorsOpenReview ↗

Response to Reviewer jcQz [2/3]

W2 & Q1: Reliability of the Frozen Judge, Judge-Human Correlation, and Alternative Valid Paths

To evaluate process quality independently of the training judge, we sample 100 questions (Easy 10 / Medium 10 / Hard 4 [all] / V*QA 16 / HR-4K 30 / HR-8K 30) from the 707-question three-way overlap on Qwen3-VL-4B across the six held-out benchmarks. Three blinded human raters (model identities hidden, trajectory order randomized per question) score each rollout independently, alongside GPT-4o as an uninvolved third-party judge using the same rubric:

Evaluation Method (fixed 100-sample)+SFT+SFT+ORL+SFT+HaPRLQuestion-Level Correlation
Blinded Human Raters (mean of 3)83.4584.7686.68Inter-rater Krippendorff's α=0.72\alpha = 0.72 (pairwise ρ=0.75\rho = 0.75–0.810.81)
Third-Party Judge (GPT-4o)83.7484.9286.78Spearman ρ=0.72\rho = 0.72 with human mean
Frozen Process Judge (reference-free mode)83.9685.1886.91Spearman ρ=0.74\rho = 0.74 with human mean (ρ=0.69\rho = 0.69 with GPT-4o)
Subset: Alternative valid paths (36/100 questions, IoU <0.3< 0.3 vs. human boxes)--Human 85.6 / Judge 86.0Judge-human correlation remains ρ=0.68\rho = 0.68 on this subset
  • Breaking evaluator circularity: Both blinded human raters (head-to-head HaPRL vs. Outcome-RL win/tie/loss = 58 / 28 / 14, i.e., 58.0% / 28.0% / 14.0%, p<10−6p < 10^{-6}) and GPT-4o rank HaPRL highest, consistent with the +2.91 macro lead across all 707 paired questions.
  • Robustness to alternative valid paths: The process-judge prompt specifies human traces as difficulty and granularity anchors rather than exact templates: "Alternative routes are valid if the evidence is sufficient and the search keeps progressing" (qa_process_v1.py, L239). In practice, 36.0% (36/100) of correct rollouts take low-overlap alternative routes (e.g., anchoring on a different context object or covering two targets in one wider crop). Both human raters (85.6) and the frozen judge (86.0) assign high scores to these rollouts (within ~1 point of the overall mean), rewarding evidence sufficiency and convergence efficiency rather than path imitation.
AuthorsOpenReview ↗

Response to Reviewer jcQz [3/3]

W3, Q1 & Q2: Answer Gating on Failed Trajectories and Cross-Annotator Generalization

(1) Rollout group composition and partial-credit ablation (Q2). Across the main run (Qwen3-VL-4B, G=8G = 8 rollouts per prompt), rollout groups break down by epoch as follows:

Training StageAll-Correct Groups (8/8 correct: 0 Outcome advantage, nonzero HaPRL ranking gradient)Mixed Groups (1–7 correct: separates right/wrong + ranks correct rollouts)All-Wrong Groups (0/8 correct: 0 advantage in both arms)
Epoch 111.8%70.6%17.6%
Epoch 215.4%70.1%14.5%
Epoch 318.2%69.2%12.6%
  • All-wrong groups are bounded and symmetric across arms: All-wrong groups account for 17.6% -> 12.6% of prompts and yield zero within-group variance under Outcome-RL as well, so gating loses no signal relative to the baseline. Conversely, Outcome-RL produces zero gradient on the 11.8% -> 18.2% all-correct groups, where HaPRL maintains a within-group process-score SD of 0.31; on the ~70% mixed groups, HaPRL adds fine-grained ranking among correct rollouts (SD 0.14) while skipping ~57% of process-judge calls on wrong answers.
  • Empirical test of partial credit for failed rollouts: We test a relaxed gate R(τ)=routrproc+α(1−rout)r~progR(\tau) = r_{\text{out}} r_{\text{proc}} + \alpha (1 - r_{\text{out}}) \tilde{r}_{\text{prog}} that retains an α\alpha-weighted progress/coverage score on incorrect trajectories. Macro accuracy drops from 44.26 at α=0\alpha = 0 (hard gate) to 43.68 (−0.58) at α=0.1\alpha = 0.1 and 42.91 (−1.35) at α=0.3\alpha = 0.3. In all-wrong or low-accuracy groups, assigning positive advantage to well-formatted but ultimately wrong searches reinforces plausible-looking trajectories that misread fine details, making the hard gate more reliable against reward hacking.

(2) Cross-annotator agreement and retraining with new annotators (W3 & Q1). To test sensitivity to individual annotator search styles, we randomly sample 200 training questions and have 2 additional independent annotators (A2, A3) perform search annotations blind to the original annotator (A1), then run RL under the identical 200-prompt recipe from Table 3:

  • Cross-annotator behavioral and scoring consistency: Pairwise final-evidence-box IoU across the three annotators averages 0.78±0.110.78 \pm 0.11, coarse-to-fine region-sequence Kendall's τ=0.71\tau = 0.71, and task-difficulty rank correlation ρ=0.83\rho = 0.83. Scoring the same model rollouts with A1, A2, or A3 as the judge anchor yields cross-annotator process-score correlation ρ=0.81\rho = 0.81 (mean absolute difference 0.04).
  • Retraining across annotators: Training Qwen3-VL-4B on A1 / A2 / A3 traces reaches 6-benchmark macro accuracies of 36.62 / 36.13 / 35.98 (range 0.64 points). All three 200-prompt runs (18% budget) outperform full-budget Outcome-RL trained on all 1,104 prompts (33.98). In addition, our six evaluation sets already span out-of-domain benchmarks beyond VisualProbe, including V*Bench (SA-1B natural scenes) and HR-Bench (4K/8K ultra-high-resolution cross-domain images).
AuthorsOpenReview ↗

Sincere Thanks and Follow-up to Reviewer jcQz

Dear Reviewer jcQz,

Thank you again for taking the time to review our manuscript and share your feedback. We have responded to your comments and revised the manuscript accordingly. We hope that our responses and revisions address your concerns. If any questions or concerns remain, we would be happy to address them during the discussion period.

Best regards,

The Authors

Reviewer_Bc2zOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies reinforcement learning for visual search agents that answer questions about high-resolution images through crop-and-zoom interactions. HaPRL uses human search traces as instance-specific references for a frozen vision-language judge, which scores complete trajectories using a task-dependent rubric. The process score is gated by final-answer correctness and used in GRPO; the human trace is never supplied to the policy. Experiments cover four backbones and six evaluation subsets from three benchmark families. Additional controls compare cheaper reward references, a stronger cold start, and a two-stage training schedule. The results support consistent answer-accuracy improvements, while the interpretation of the process-quality measurements requires further validation.

Critical strengths

The accuracy improvements are consistent across four backbones, with macro-average gains of 2.48–10.28 percentage points over matched outcome-only RL. The stronger cold-start control in Table 9 retains a 5.87-point gain, supporting benefits beyond recovery from degraded SFT initialization.

The paper provides informative controls for the value of human references. In Table 8, human traces outperform rubric-only scoring, final evidence boxes, and teacher-generated paths; the gain over the teacher-path control is 4.41 points. The three-seed continuation experiment also supports the benefit of early process supervision within the tested schedule.

Critical weaknesses

The process-quality evidence is not independent of the training objective: Table 2 uses the same judge and averages over each policy’s own answer-correct subset. Changes can therefore reflect evaluator preferences or different question composition. This does not establish the stronger claim that outcome-only RL erodes search or induces reward hacking.

The central reward pipeline is insufficiently specified in the PDF: conversion of raw interactions into semantic traces, references used for test-time process evaluation, and the judge’s access to actual crop observations remain unclear. These details directly affect whether the reward measures evidence acquisition and whether the main contribution can be reproduced.

Questions for authors

Can you validate process quality using a blinded independent assessment on a fixed question sample, including a paired analysis of questions answered correctly by both policies and the corresponding sample counts? This would help distinguish genuine process changes from the subset-composition and evaluator effects in Table 2.

How are raw interaction logs converted into semantic reference traces, and where do the matched human references for the evaluation sets come from? Please specify the judge prompts, whether the judge receives actual crop images, and the observed text-fallback rates.

Please clarify the separation of training, checkpoint-selection, and test data, and provide the actual V* evaluation prompt. Appendix B.1 claims that free-form scoring removes option-elimination shortcuts, but Figure 9 shows answer choices and a rollout referring to them. Were those options removed in the quantitative evaluation?

5 comments
AuthorsOpenReview ↗

Response to Reviewer Bc2z [1/2]

We sincerely thank the reviewer for the constructive comments and valuable suggestions. We address each point below and will revise the manuscript accordingly.

W1 & Q1: Independent Validation of Process Quality (Paired Analysis + Blinded Human Evaluation)

We separate (i) question-subset composition and (ii) evaluator preferences with two complementary analyses.

(1) Paired process-score analysis on fixed question samples (isolating subset composition). Across the six benchmarks on Qwen3-VL-4B (2,306 questions in total), SFT, Outcome-RL, and HaPRL answer 919, 928, and 1,154 questions correctly. We extract the three-way intersection answered correctly by all three policies (707 questions) and the two-way intersection answered correctly by both Outcome-RL and HaPRL (801 questions), and compute process scores on the exact same questions:

Benchmark3-Way Overlap N3N_3SFTOutcome-RLHaPRLΔ\Delta(Out$-$SFT)Δ\Delta(HaP$-$Out)2-Way Overlap N2N_2Out (N2N_2)HaP (N2N_2)ΔN2\Delta_{N_2}
VisualProbe Easy3385.3184.5887.22−0.73+2.643684.4287.15+2.73
VisualProbe Medium3186.6287.1487.71+0.52+0.574086.9887.64+0.66
VisualProbe Hard472.3665.9476.88−6.42+10.94666.1576.42+10.27
V*QA7981.0283.4783.62+2.45+0.159783.3183.58+0.27
HR-Bench 4K29886.7187.3388.84+0.62+1.5132687.2888.79+1.51
HR-Bench 8K26282.9886.0587.71+3.07+1.6629685.9987.66+1.67
Macro Avg.70782.5082.4285.33−0.08+2.9180182.3685.21+2.85
Micro Avg.70784.5486.1787.65+1.63+1.4880186.0287.52+1.50

Paired comparisons on the 707 and 801 shared questions show that HaPRL's process advantage holds on fixed instances. With the question set held fixed, HaPRL scores higher than Outcome-RL on every subset, leading by +2.91 macro (+1.48 micro) on the three-way overlap and +2.85 macro (+1.50 micro) on the two-way overlap. This matches Table 2 (+2.78), while Outcome-RL's macro change over the cold start stays near zero (−0.08).

(2) Blinded human evaluation on 100 questions + cross-judge verification (isolating evaluator preference). We draw a stratified sample of 100 questions from the 707-question three-way intersection (Easy 10 / Medium 10 / Hard 4 [all] / V*QA 16 / HR-4K 30 / HR-8K 30). Three independent raters score each trajectory (0-100 scale) under double-blind conditions: policy identities are hidden, the three trajectories per question are randomly ordered, and raters inspect the full-resolution original image alongside the actual cropped regions. We also evaluate the same sample with a second independent VLM judge (GPT-4o) that was never used during training:

Evaluator (same fixed 100-question sample)+SFT+SFT+Outcome-RL+SFT+HaPRLΔ\Delta(HaPRL −- Outcome-RL)
Blinded Human Raters (mean of 3)83.4584.7686.68+1.92
Second Independent VLM Judge (GPT-4o)83.7484.9286.78+1.86
Frozen Process Judge (reference-free mode)83.9685.1886.91+1.73
  • Reliability and rank agreement: Inter-annotator agreement across the 3 human raters reaches Krippendorff's α=0.72\alpha = 0.72 (pairwise Spearman ρ=0.75\rho = 0.75–0.810.81). Question-level correlation between the frozen process judge and mean human ratings is ρ=0.74\rho = 0.74 (ρ=0.69\rho = 0.69 with GPT-4o), and all three evaluators produce the identical policy ranking.
  • Pairwise blinded win rate: Comparing HaPRL against Outcome-RL head-to-head per question, human raters prefer HaPRL on 58 questions (58.0%), tie on 28 (28.0%), and prefer Outcome-RL on 14 (14.0%) (two-sided sign test, p<10−6p < 10^{-6}).
  • Tolerance to alternative valid paths: On 36 of the 100 questions (36.0%), HaPRL follows a valid alternative search path with low spatial overlap against the offline human reference boxes (region IoU <0.3< 0.3). On this subset, human raters score HaPRL at 85.6 and the frozen judge gives 86.0 (within ~1 point of the 100-question mean, with ρ=0.68\rho = 0.68), showing that higher scores reflect visual evidence acquisition rather than matching a single reference template.
AuthorsOpenReview ↗

Response to Reviewer Bc2z [2/2]

W2, Q2, Q3 & Q4: Reward Pipeline, Data Separation, and V* Evaluation Specification

Thank you for flagging the implementation details missing from the main text and appendix. Every item below maps directly to the submitted supplementary code, which supports full reproduction, and we will incorporate all of them into the revised main text and appendix:

  1. Trace distillation and test-time evaluation references (Q2):

    • Four-step offline distillation (train1000_v1.json): Raw browser pointer streams (movement, dwell, zoom, and selection boxes) first suppress consecutive pointer samples that do not change the inspected region, then segment into discrete rounds at region or zoom transitions (mean 2.82 rounds across 1,104 questions). A VLM rewrites these structured rounds into Round N text recording the inspected region, normalized box, and local visual cues, and the annotator verifies each trace against the interaction replay (final_bboxes and difficulty tags are retained alongside). The policy always receives only (I0,q)(I_0, q).
    • Reference-free test-time process scoring: The 1,104 human traces cover only the training split. In Table 2 and the paired table above, process scores on the six held-out benchmarks are computed by the same frozen judge in reference-free mode using the identical five-dimensional rubric across all policies.
  2. Judge inputs, crop visibility, prompts, and fallback rates (Q2):

    • Inputs and crop verification: The process judge (frozen Qwen3-VL-30B-A3B-Instruct, temperature = 0) receives 1 full-resolution original image I0I_0, the question, and the text-serialized multi-turn rollout (each round's <think>, each <grounding> call's source view and normalized [x0, y0, x1, y1] coordinates, and the final <answer>; training calls also include the human trace and final_bboxes). The judge does not receive cropped image pixels, and instead verifies against the full original image and exact normalized coordinates whether each crop covers the target evidence.
    • Prompts and fallback statistics: Exact prompts for the answer and process judges appear in qa_process.py (L34–59) and qa_process_v1.py (L227–317) of the supplementary code, and will be added verbatim to the appendix. Across the main run (26,496 rollouts), the answer judge failure rate is 0.00%. Among the 11,382 gated process-judge calls, 99.26% succeed in multimodal vision mode, 0.74% (84 calls) fall back to text_fallback, and 0.00% fail after retries.
  3. Separation of training, checkpoint-selection, and test data (Q3):

    • All 1,104 training questions come from VisualProbe_train and have zero overlap with the six evaluation splits. Cold-start checkpoint selection uses 20 held-out training-side prompts.
    • During RL, both arms select checkpoints on the same 51-question validation pool (val_mini_acc.json), which never participates in gradient updates.
  4. V* evaluation prompt and options in Figure 9 (Q3 & Q4):

    • All answer options are removed in quantitative evaluation: Preprocessing strips the (A)–(D) option lines and letter-choice instructions from the official V* test_questions.jsonl, mapping each ground-truth label to its option text (e.g., input <image>\nWhat is the material of the glove?, reference answer "rubber"; the system prompt is TOOL_CROP_SYSTEM_PROMPT in constants.py, instructing the model to use <think>, <grounding>{"bbox_2d": [x0,y0,x1,y1], "source": ...}</grounding>, and <answer>).
    • Note on Figure 9: Our qualitative visualization includes the answer choices to make the option-selection process clearer for human inspection, whereas quantitative main-table evaluations never include options (as detailed above). Comparing direct_attributes-58 with and without options in the prompt gives the same behavior: HaPRL localizes the roadside flag and answers correctly in 2 rounds (1 crop) under both prompts, while both controls fixate on the upper flagpole and fail.
AuthorsOpenReview ↗

Sincere Thanks and Follow-up to Reviewer Bc2z

Dear Reviewer Bc2z,

Thank you again for taking the time to review our manuscript and share your feedback. We have responded to your comments and revised the manuscript accordingly. We hope that our responses and revisions address your concerns. If any questions or concerns remain, we would be happy to address them during the discussion period.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.