acceptodds
Under review as a conference paper at ICLR 2027

VeriFine: Scaling Verification for Self-Improvement in Physical Reasoning

22 views

Abstract

Self-improving policies continually expose new failure patterns, changing what their judges must be able to verify. However, current fixed judges constrain both optimization feedback and the discovery of useful training examples, limiting further self-improvement. This challenge is even more acute in embodied reasoning, where reliable evaluation must account for spatial grounding, causal reasoning, and safety-aware decision-making. We introduce VeriFine, an agent harness framework that scales verification through the co-evolution of the policy, training curriculum, and judge. The Policy Improvement Loop uses a rubric judge to diagnose recurring failures, construct an adaptive curriculum, and optimize the policy. When progress plateaus and verification becomes a bottleneck, the Judge Improvement Loop selectively queries human guidance on informative failure cases and refines the judge through coactive calibration, in which humans and agents resolve disagreements and converge toward the objective rubric of physical reasoning. The revised judge then guides the next stage of data selection and policy optimization. Experiments on driving and robot navigation tasks demonstrate continuous self-improvement in both policy and judge capability across reinforcement and supervised fine-tuning. These results show how scaling verification supports continuous self-improvement as policy failure patterns evolve.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Physical Reasoning, Self-improvement, Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_knBdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VeriFine iteratively refines an embodied-reasoning policy, curriculum, and rubric judge. The judge diagnoses failures and selects data; human coactive calibration updates the rubric after stagnation, followed by student-judge distillation. Driving uses GRPO and navigation uses SFT. Each controlled policy run starts from the same base model, so supervision state accumulates across rounds. Driving RB reasoning scores improve from 60.56 to 71.61, navigation scores from 68.09 to 77.59, and judge–human alignment improves.

Critical strengths
  1. S1: The framework distinguishes valuable difficult contexts from incorrect demonstrations and supports both reward-based optimization and data filtering.
  2. S2: A fixed RB evaluator, common initialization and per-policy budgets, curriculum ablations with fixed reward judges, and same-pool test-time selection provide useful controls.
  3. S3: Later challenging cases expose specific weaknesses of the old judge that refinement improves. Backbone and student ablations add supporting evidence.
Critical weaknesses
  1. W1: Main policy benefits rely on offline evaluators sharing rubric and LLM-label sources. Keeping RB fixed avoids a moving metric but does not replace independent policy assessment. Driving ADE changes only from 2.139 to 2.117 and worsens in early rounds. Navigation lacks task success, SPL, or collision outcomes. Reasoning scores do not establish improved closed-loop capability (Table 1; §4.4; Appendix A.3.3).
  2. W2: Coactive calibration can revise both rules and human calibration labels, without reporting label changes, annotator agreement, human time, or an equal-budget ordinary calibration control. Independent frozen test labels are a strength, but improved agreement does not establish convergence to a unique objective physical rubric (§3.4.2; Appendix A.4.3).
  3. W3: Per-policy budget matching excludes earlier policy runs, human queries, rubric search, and judge distillation. Resetting policies also limits an interpretation of continuous policy self-improvement. Equal-total-cost one-shot strong-judge/curriculum and sequential-training baselines are missing (§§3.1 and 4.1).
  4. W4: Appendix A.4.3 does not fully specify how the 0.5 trigger threshold combines signals; curriculum scores and tie-breaking remain abstract, and the Pearson 0.2 stagnation threshold needs units. The claim that curriculum advantage “grows” also conflicts with absolute gains declining from 8.59 to 5.27 to 3.08.
Questions for authors
  1. Q1 → W1: Can policies receive independent assessment blind to their versions, with closed-loop driving collisions/violations and navigation success/SPL, including scene-clustered intervals?
  2. Q2 → W2: Can human time, label revisions, and annotator agreement be quantified, and ordinary fixed-label calibration compared with coactive calibration under equal feedback budgets and the same frozen blind test set?
  3. Q3 → W3: Can all training, API, and human costs be included when comparing the iterative system with one-shot strong-judge/curriculum and sequential policy training?
5 comments

Thank you for the detailed review.

Weakness 1 and Question 1

Independent policy evaluation. We would like to clarify the role of VeriFine-Judge-RB. We introduced this reference-based model to make evaluation efficient and scalable without requiring human experts to score every policy rollout, which is too expensive and time-consuming. It compares rollout reasoning with ground-truth reasoning that human experts annotated and checked. Its rubric is designed around that reference and includes criteria specific to reference-based evaluation, such as semantic agreement between the reference and the rollout. We therefore use VeriFine-Judge-RB as a proxy for the upper bound on scene-level reasoning scores. Fig. 3 shows strong agreement with human scores, with Pearson r = 0.82 and MAE = 0.16. We attribute the remaining gap to the limited availability of human-scored examples for fine-tuning, which are costly to obtain.

To address the concern about independent policy assessment, we have compared with LingoJudge in the paper, which we use as a state-of-the-art driving-reasoning baseline. In Fig. 3, both VeriFine-Judge-RB and the final VeriFine-Judge student improve Pearson correlation by about 26% relative to LingoJudge. Policies improved with LingoJudge also score lower than those improved with VeriFine-Judge under both the RB and RF evaluators.

JudgePearson r ↑MAE ↓Latency (s/sample) ↓
LingoJudge0.650.23—
VeriFine-Judge-RB0.820.16—
VeriFine-Judge-R3 student0.820.170.07
VeriFine-Judge-R3 teacher0.850.1320.92

We additionally use independent, blinded human scores in the test-time scaling experiment in the upper-right panel of Fig. 3. Each judge selects the candidate it considers best, and we compare the selected candidate's human score with the human oracle. These scores show improvements across the VeriFine-Judge rounds from R1 to R3 and also show that VeriFine-Judge-RB selects better candidates than LingoJudge.

From Base to R3, open-loop ADE improves from 2.139 to 2.117 m and minADE₆ from 1.049 to 1.029 m. We attribute the modest action gains to our decision to optimize the reasoning reward alone. The goal was to measure gains in reasoning independently of a separate action reward. Reasoning and actions can diverge in a VLA policy, so a correct action could give positive feedback to a rollout whose reasoning is incorrect. Even under this objective without action reward, our final policy achieves the lowest ADE, suggesting that improved reasoning can benefit the existing action expert. A next step is to use the improved reasoning to train the action expert to produce higher-quality trajectories that better align with the reasoning.

The same applies to navigation. We focus on reasoning quality and use only reasoning supervision for SFT. Reasoning quality is harder to assess than navigation success, which can be checked using verifiable rules. Improving trajectory quality requires further training of the corresponding action expert, which is our next step.

   

Weakness 2 and Question 2

During coactive calibration, only labels in the judge evaluation set can be revised. For final evaluation, we use an independent judge test set that multiple trained annotators labeled independently. Coactive calibration starts from the premise that human feedback can contain errors and that annotators may be less familiar with the reasoning required in particular scenarios. The agent can draw on its world knowledge and common sense to help experts revisit their judgments, and experts can correct the agent in turn.

Consider a highway scenario in which another vehicle cuts in from the right. The ego vehicle may need no response to a distant cut-in, while it may need to slow down for a close one. The boundary is how far away the cut-in vehicle can be before the ego vehicle needs to slow down. Human judgments about that distance can vary across scenes. The agent may be too conservative, while the human may be too optimistic. By correcting each other, they can reach a better judgment and move closer to an objective rubric for physical reasoning. In other scenarios, the agent may make basic errors that the human expert can correct.

In Round 3 of the driving reasoning task, we additionally compare ordinary fixed-label calibration with coactive calibration under equal feedback budgets on the same frozen, blinded judge test set.

SettingPearson r ↑
Fixed-label calibration0.76
Coactive calibration0.85

Human effort and intervention timing. A human intervention takes about five minutes on average, depending on the failure pattern. In the Table 3 inference setup, the student takes 0.07 s/sample and the API teacher takes 20.92 s/sample.

Weakness 3 and Question 3

In terms of cost, the reported comparisons match per-policy training budgets.

  1. The driving policy is trained from the same base checkpoint with the same 2,700 GRPO steps to ensure a fair comparison.
  2. In the Table 3 latency setup, the student takes 0.07 s/sample and the API teacher takes 20.92 s/sample.
  3. Each judge improvement round allows up to 20 rubric versions. We request human review when the average improvement in Pearson correlation stays below 0.2 for more than ten rounds.
  4. The time required depends on the failure pattern and averages about five minutes per intervention.
  5. End-to-end costs include failure analysis, data scoring, human review, rubric search, teacher labeling, and student distillation. Including agent iterations, policy fine-tuning, and human guidance where applicable, it takes about two days for a complete policy improvement loop and one day for a complete judge improvement loop.

Additionally, for the question "Resetting policies also limits an interpretation of continuous policy self-improvement. Equal-total-cost one-shot strong-judge/curriculum and sequential-training baselines are missing".

  • Actually, in each improvement round, we train the policy from the same base initialization with a different judge, which lets us compare successive supervision states at a fixed per-policy training budget.
  • We view VeriFine-Judge-R3 (Third Round) in Table 1 as the setting corresponding to this comparison. Tables 1 and 2 report its results and the relevant comparisons.
  • In a continuous training setting, updates in earlier and later rounds affect one another, so the ablations cannot isolate the contribution of each module. In our setting, the progressively stronger judge preserves the knowledge gained and the priorities for improvement identified in previous rounds. On the other hand, without the full loop improvement, we have no final strong judge.

   

Weakness 4

Driving round transitions.

  1. During policy training, we compute the mean policy reward over each 150-step window using the current reward judge and evaluate the checkpoint at the end of that window using the fixed evaluation judge. In Eq. (7), ΔP‾t\overline{\Delta P}_t averages the changes in checkpoint performance over the last four windows, and GtG_t measures the absolute difference between the changes in mean reward and checkpoint performance from one window to the next. We scale both scores to 0–100 before computing these quantities. The transition criterion is ΔP‾t<0.5\overline{\Delta P}_t < 0.5 or Gt>1G_t > 1.

  2. For the controlled comparison, each policy starts from the same base and trains for the full 2,700 steps. After training, we apply the transition criterion to select the checkpoint at the end of a qualifying window. We use that checkpoint for the judge improvement loop and report the policy at step 2,700 for the matched-budget comparison.

Curriculum selection.

  1. After each round, we measure policy performance by scenario category on the policy evaluation set. We shift part of the data quota from stronger categories to weaker ones according to how far the weaker categories fall below the mean. We retain examples from stronger categories to preserve coverage and add diverse examples from weaker ones. The updated curriculum blends the previous category targets (60%) with newly added examples (40%). The curriculum judge applies a target quality threshold of 0.8 to filter out candidates with unreliable annotations.
  2. For each retained candidate, we generate a rollout with the current policy and score its reasoning with the current curriculum judge. We define learnability as the ground-truth reasoning score minus the policy rollout score. We rank candidates by this gap, prioritizing larger gaps because they indicate greater room for improvement and a stronger potential learning signal.

Additional, Pearson's r is a dimensionless correlation coefficient. As the reviewer noted, the original sentence omitted the subject of “grows.” We will revise it to read “The curriculum advantage persists and the reasoning score grows across judge versions.”

    We hope the clarifications can address your concerns, and we would be grateful if you would consider raising your overall rating. We also would be happy to address any further questions.

AuthorsOpenReview ↗

Follow-Up to Your Review

Dear Reviewer knBd,

Thank you again for the time and effort you put into reviewing our work!

We understand that this might be a super busy period for you. We wanted to briefly let you know that we have posted point-by-point responses to your questions and concerns.

Whenever you have a chance, we would be grateful for your feedback on whether the responses address your concerns. We would be happy to clarify anything that remains open.

Thank you!

Reviewer_QePYOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

VeriFine alternates policy training with human-assisted refinement of a rubric judge for driving and navigation reasoning. The judge selects training data and supplies rewards for driving; its evaluations are distilled into a smaller student. Reported reasoning scores rise from 60.56 to 71.61 for driving and from 68.09 to 77.59 for navigation. Across driving rounds, policies restart from the same base, while the judge and curriculum evolve. Evaluation uses a fixed policy evaluator and a separate human-scored candidate-selection experiment.

Critical strengths
  • The experiments cover two embodied reasoning tasks and include several baseline comparisons and ablation studies.
  • The reported results show consistent improvements in policy reasoning scores and judge–human agreement across refinement rounds.
Critical weaknesses
  • The navigation experiment lacks a matched fixed-judge or random-selection SFT baseline (figure 4). It demonstrates that the framework can be applied to navigation, but does not isolate how much of the improvement comes from evolving verification.
  • Appendix A.4.3 gives a checking interval, window, and threshold, but leaves the trigger logic and curriculum-ranking rule unclear. These choices implement the central loop; executable rules and an example update trace are needed to reproduce it.
  • Table 2 supports curriculum selection and figure 5 supports calibration, but neither isolates the value of boundary-triggered human intervention.
Questions for authors
  • Could you give the exact trigger and curriculum-ranking rules, with one example showing how an observed failure changes the selected data or rubric?
  • Beyond the reported calibration-set sizes and rubric revision limits, what were the actual human annotation hours and model-query costs per round? Does boundary-triggered refinement outperform periodic refinement at a matched human-feedback budget?
  • Can you provide a matched SFT control for navigation and uncertainty estimates for the main policy comparisons?
6 comments

Thank you for your detailed feedback.

Weakness 1 and Question 3 · Navigation SFT control

We ran the main component ablations on driving reasoning because its larger, more diverse dataset supports a fuller analysis of each part of VeriFine. Navigation tests whether the framework also transfers to supervised fine-tuning (SFT). In navigation, the judge selects demonstrations for SFT and have no reward judge, so the ablation varies the curriculum judge. The policy is trained with the SFT objective.

The 68.09 score in Fig. 4 is the random-selection SFT baseline. It uses the same candidate pool and training budget as the judge-guided runs. The final guided curriculum reaches 77.59, a gain of 9.50 points.

Curriculum judgePolicy reasoning score ↑
Random selection68.09
R174.14
R277.59

Weakness 2 and Questions 1–2 · Execution protocol and human cost

We will expand Appendix A.4.3 to make the execution protocol clearer.

Driving round transitions.

  1. During policy training, we compute the mean policy reward over each 150-step window using the current reward judge and evaluate the checkpoint at the end of that window using the fixed evaluation judge. In Eq. (7), ΔP‾t\overline{\Delta P}_t averages the changes in checkpoint performance over the last four windows, and GtG_t measures the absolute difference between the changes in mean reward and checkpoint performance from one window to the next. We scale both scores to 0–100 before computing these quantities. The transition criterion is ΔP‾t<0.5\overline{\Delta P}_t < 0.5 or Gt>1G_t > 1.

  2. For the controlled comparison, each policy starts from the same base and trains for the full 2,700 steps. After training, we apply the transition criterion to select the checkpoint at the end of a qualifying window. We use that checkpoint for the judge improvement loop and report the policy at step 2,700 for the matched-budget comparison.

Curriculum selection.

  1. After each round, we measure policy performance by scenario category on the policy evaluation set. We shift part of the data quota from stronger categories to weaker ones according to how far the weaker categories fall below the mean. We retain examples from stronger categories to preserve coverage and add diverse examples from weaker ones. The updated curriculum blends the previous category targets (60%) with newly added examples (40%). The curriculum judge applies a target quality threshold of 0.8 to filter out candidates with unreliable annotations.
  2. For each retained candidate, we generate a rollout with the current policy and score its reasoning with the current curriculum judge. We define learnability as the ground-truth reasoning score minus the policy rollout score. We rank candidates by this gap, prioritizing larger gaps because they indicate greater room for improvement and a stronger potential learning signal.

Additional, Pearson's r is a dimensionless correlation coefficient. As the reviewer noted, the original sentence omitted the subject of “grows.” We will revise it to read “The curriculum advantage persists and the reasoning score grows across judge versions.”

Example rubric update. In R2, the agent reviewed videos and reasoning traces from R1 and found that the policy often failed to make a lateral adjustment around obstacles encroaching into the ego lane or stopped without proceeding. The agent first proposed rubric edits. When human review was requested, the expert examined the proposals, their evaluation history, and the video and reasoning traces. The expert clarified that a slight lateral adjustment around the obstacle can be an appropriate way to proceed when the scene permits it. The agent incorporated that guidance into the existing rubric so the judge could recognize a safe pass as a valid response.

Human effort and intervention timing.

  1. Each judge improvement round allows up to 20 rubric versions. Appendix A.4.3 describes the human review criterion as an average improvement in Pearson correlation below 0.2 for more than ten iterations. A human intervention takes about five minutes on average, depending on the failure pattern. In the Table 3 inference setup, the student takes 0.07 s/sample and the API teacher takes 20.92 s/sample. End-to-end costs include failure analysis, data scoring, human review, rubric search, teacher labeling, and student distillation. Including agent iterations, policy fine-tuning, and human guidance where applicable, it takes about two days for a complete policy improvement loop and one day for a complete judge improvement loop.

  2. In the driving R3 judge improvement loop, we compared periodic and boundary-triggered refinement over 20 rubric iterations. We assigned two human guidance prompts for periodic refinement, and boundary-triggered guidance was queried only once. Boundary-triggered refinement achieved a higher final Pearson correlation in this comparison with less human guidance. The trigger lets the agent request guidance when its own refinement stalls and fully leverage the knowledge from humans.

Intervention scheduleRubric iterations with human guidanceFinal judge Pearson r ↑
Periodic refinement7, 140.83
Boundary-triggered refinement130.85

Weakness 3 · Effect of human guidance

Fig. 6 shows how human-assisted refinement improves the judge across rounds. On the frozen challenging test subset, student–human Pearson correlation rises from 0.07 with the R1 judge to 0.74 with the R2 judge in the judge test set R2. In R3, with the R1 test subset held fixed, teacher–human Pearson correlation rises from 0.70 to 0.86 with the human guidance. To isolate the role of human guidance more directly, we start from the same policy within each round and compare judge refinement with and without human guidance on the whole judge test set.

RoundHuman guidancePearson r
R1No0.37
R1Yes0.55
R2No0.59
R2Yes0.73
R3No0.76
R3Yes0.85

We hope the clarifications can address your concerns, and we would be grateful if you would consider raising your overall rating. We also would be happy to address any further questions.

Reviewer_wxCKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

VeriFine couples policy optimization, adaptive curriculum selection, and human-assisted rubric refinement for embodied reasoning, distilling a frontier teacher into an efficient student judge. Experiments demonstrate improvements in judge–human alignment and policy reasoning scores across driving with GRPO and robot navigation with SFT.

Critical strengths
  1. The framework addresses an important limitation of static judges by connecting failure discovery, curriculum adaptation, and selective human calibration. Teacher–student distillation makes this approach practical for frequent reward computation, with the final student achieving human-score correlation close to the teacher’s.

  2. The evaluation includes fixed reporting evaluators, separate test sets, within-round curriculum ablations, and human-scored candidate selection. These controls provide meaningful evidence of improved verification and reasoning across two tasks and optimization methods.

Critical weaknesses
  1. Insufficient isolation of the co-evolution benefit and its cost. The distinctive benefit of co-evolution is not sufficiently isolated from stronger rubrics, additional human supervision, and improved data selection, particularly relative to related methods such as DynamicRubric and EvoRubrics. Matched per-policy training budgets do not account for the full iterative process, while missing human-effort and end-to-end cost measurements weaken the scalability claim.

  2. Limited independent validation and practical gains. Validation relies primarily on internal or custom reasoning evaluations, and the fixed RB evaluator also uses rubric-derived model supervision, leaving shared evaluation biases insufficiently excluded. Final ADE and minADE6 improve over the base policy by only approximately 1.0% and 1.9%, without uncertainty estimates or closed-loop validation, so the practical significance of the reasoning gains remains unclear.

  3. Insufficient analysis of reward robustness and optimization stability. Aggregate judge–human correlation and the fixed-candidate selection experiment provide limited evidence about within-context ranking accuracy on evolving policy outputs or robustness under sustained reward optimization. Table 2 shows that the R1 judge with random selection degrades both RB reasoning scores and trajectory metrics relative to the base policy, but the paper does not investigate whether this reflects reward-model errors, data selection, or optimization instability.

Questions for authors
  1. Protocol clarification: How do the 150-step checks, four-check window, and 0.5 threshold determine transitions from R1 to R3 while each reported policy receives 2,700 training steps? Please clarify the actual trigger events, student-judge update procedure, and whether three rounds were prescribed or selected by a stopping rule.

  2. Contribution and cost: Is there evidence comparing the full loop against a strong fixed judge with adaptive curriculum under matched total annotation and compute budgets? Please report expert hours, annotation/revision counts, and total runtime across rounds.

  3. Validation: Can you provide independent blind human comparisons of the base and R1–R3 policies, together with uncertainty estimates for the main results? These would help establish whether the gains extend beyond shared evaluator preferences and whether the small trajectory improvements are reliable.

6 comments

Thank you for your detailed comments. We are addressing your concerns below.

Weakness 1, Question 2

Contribution and cost. Our ablations separately examine the reward judge, the training curriculum, and human guidance for judge refinement.

  1. Effect of a stronger reward judge. Table 2 compares different reward judges with a random training curriculum, the same base policy, and the same training budget. The reward judge is the only variable in this comparison. As we move from R1 to R3, the RB reasoning score rises from 58.71 to 65.32 to 68.53, showing the benefit of an improved judge in the policy improvement loop.
  2. Effect of improved data selection. Within each round in Table 2, we hold the reward judge fixed and compare random selection with the curriculum selected by that round's judge. The RB reasoning scores rise from 58.71 to 67.30 in R1, 65.32 to 70.59 in R2, and 68.53 to 71.61 in R3. This isolates the benefit of guided data selection under each reward judge.
Reward judgeCurriculum JudgeRB ↑RF ↑ADE (m) ↓minADE₆ (m) ↓
Base policy—60.5668.052.1391.049
LingoJudgeLingoJudge65.7572.602.2421.172
VeriFine-Judge-RBVeriFine-Judge-RB67.7075.202.2351.164
R1Random58.7168.902.5941.349
R1R167.3077.512.1771.106
R2Random65.3276.672.3171.119
R2R270.5979.612.1391.078
R3Random68.5381.592.2221.053
R3R371.6183.132.1171.029
  1. Effect of human guidance. Fig. 6 shows how human-assisted refinement improves the judge across rounds. On the frozen challenging test subset, student–human Pearson correlation rises from 0.07 with the R1 judge to 0.74 with the R2 judge in the judge test set R2. In R3, with the R1 test subset held fixed, teacher–human Pearson correlation rises from 0.70 to 0.86 with the human guidance. To isolate the role of human guidance more directly, we start from the same policy within each round and compare judge refinement with and without human guidance on the whole judge test set.
RoundHuman guidancePearson r
R1No0.37
R1Yes0.55
R2No0.59
R2Yes0.73
R3No0.76
R3Yes0.85

The results show the value of human guidance in all three rounds. DynamicRubric generates rubrics conditioned on candidate sets, and EvoRubrics updates a rubric generator through adversarial training. We see a remaining question about how a policy can improve beyond the performance ceiling imposed by its current verifier. VeriFine uses scene-grounded failure diagnostics to guide both curriculum construction and human-assisted verifier refinement.

Additionally, for the question" Is there evidence comparing the full loop against a strong fixed judge with adaptive curriculum under matched total annotation and compute budgets?".

  • Actually, in each improvement round, we train the policy from the same base initialization with a different judge, which lets us compare successive supervision states at a fixed per-policy training budget.
  • We view VeriFine-Judge-R3 (Third Round) in Table 1 as the setting corresponding to this comparison. Tables 1 and 2 report its results and the relevant comparisons.
  • On the other hand, without the full loop improvement, we have no final strong judge.

Moreover, in terms of cost, the reported comparisons match per-policy training budgets.

  1. The driving policy is trained from the same base checkpoint with the same 2,700 GRPO steps to ensure a fair comparison.
  2. In the Table 3 latency setup, the student takes 0.07 s/sample and the API teacher takes 20.92 s/sample.
  3. Each judge improvement round allows up to 20 rubric versions. We request human review when the average improvement in Pearson correlation stays below 0.2 for more than ten rounds.
  4. The time required depends on the failure pattern and averages about five minutes per intervention. The average human intervention count is 2, with additional human guidance at the beginning of the round.
  5. End-to-end costs include failure analysis, data scoring, human review, rubric search, teacher labeling, and student distillation. Including agent iterations, policy fine-tuning, and human guidance where applicable, it takes about two days for a complete policy improvement loop and one day for a complete judge improvement loop.

Weakness 2, Question 3

Independent policy evaluation. We would like to clarify the role of VeriFine-Judge-RB. We introduced this reference-based model to make evaluation efficient and scalable without requiring human experts to score every policy rollout, which is too expensive and time-consuming. It compares rollout reasoning with ground-truth reasoning that human experts annotated and checked. Its rubric is designed around that reference and includes criteria specific to reference-based evaluation, such as semantic agreement between the reference and the rollout. We therefore use VeriFine-Judge-RB as a proxy for the upper bound on scene-level reasoning scores. Fig. 3 shows strong agreement with human scores, with Pearson r = 0.82 and MAE = 0.16. We attribute the remaining gap to the limited availability of human-scored examples for fine-tuning, which are costly to obtain.

To address the concern about relying on internal or custom evaluations, we have compared with LingoJudge in the paper, which we use as a state-of-the-art driving-reasoning baseline. In Fig. 3, both VeriFine-Judge-RB and the final VeriFine-Judge student improve Pearson correlation by about 26% relative to LingoJudge. Policies improved with LingoJudge also score lower than those improved with VeriFine-Judge under both the RB and RF evaluators.

JudgePearson r ↑MAE ↓Latency (s/sample) ↓
LingoJudge0.650.23—
VeriFine-Judge-RB0.820.16—
VeriFine-Judge-R3 student0.820.170.07
VeriFine-Judge-R3 teacher0.850.1320.92

We additionally use independent, blinded human scores in the test-time scaling experiment in the upper-right panel of Fig. 3. Each judge selects the candidate it considers best, and we compare the selected candidate's human score with the human oracle. These scores show improvements across the VeriFine-Judge rounds from R1 to R3 and also show that VeriFine-Judge-RB selects better candidates than LingoJudge.

From Base to R3, open-loop ADE improves from 2.139 to 2.117 m and minADE₆ from 1.049 to 1.029 m. We attribute the modest action gains to our decision to optimize the reasoning reward alone. The goal was to measure gains in reasoning independently of a separate action reward. Reasoning and actions can diverge in a VLA policy, so a correct action could give positive feedback to a rollout whose reasoning is incorrect. Even under this objective without action reward, our final policy achieves the lowest ADE, suggesting that improved reasoning can benefit the existing action expert. A next step is to use the improved reasoning to train the action expert to produce higher-quality trajectories that better align with the reasoning.

Weakness 3

Thank you for pointing out the R1 degradation. In Table 2, with the R1 reward judge and random selection, RB drops from 60.56 for Base to 58.71 and ADE rises from 2.139 to 2.594 m. With the same judge and guided selection, RB rises to 67.30 and ADE falls to 2.177 m. The change shows that curriculum selection materially affects the outcome under the same reward judge. The R2 judge also improves the policy's RB reasoning score to 65.32 with random selection, which indicates a limitation of the R1 judge. Together, these results support evolving both data selection and the reward judge. With R3 guiding both, the policy achieves the best reported reasoning and trajectory metrics.

Question 1

Executed round transitions.

  1. During policy training, we compute the mean policy reward over each 150-step window using the current reward judge and evaluate the checkpoint at the end of that window using the fixed evaluation judge. In Eq. (7), ΔP‾t\overline{\Delta P}_t averages the changes in checkpoint performance over the last four windows, and GtG_t measures the absolute difference between the changes in mean reward and checkpoint performance from one window to the next. We scale both scores to 0–100 before computing these quantities. The transition criterion is ΔP‾t<0.5\overline{\Delta P}_t < 0.5 or Gt>1G_t > 1.

  2. For the controlled comparison, each policy starts from the same base and trains for the full 2,700 steps. After training, we apply the transition criterion to select the checkpoint at the end of a qualifying window. We use that checkpoint for the judge improvement loop and report the policy at step 2,700 for the matched-budget comparison.

Student judge update. We identify failure patterns from the selected checkpoint's performance on the policy evaluation set. For example, the R2 refinement targeted challenging scenarios involving pedestrians and obstacles encroaching into the ego lane, where the policy scored lower. We use the corresponding tags to retrieve new examples from our database for student training, judge evaluation, and judge testing. We use 20K new student training examples in R1 and 15K in R2. After refining the rubric on the expanded judge evaluation set, we use the updated teacher to label all newly added training examples, so that the student learns judgments made under the revised rubric.

Number of rounds. We chose three rounds as part of the experimental design. R2 expanded the judge test set to capture newly exposed failures and illustrate why the test set needs to evolve. R3 kept the test set fixed to assess the role of human guidance on the same cases. The loop can continue to R4 and R5 with additional GPU compute and agent token budgets.

We hope the clarifications can address your concerns, and we would be grateful if you would consider raising your overall rating. We also would be happy to address any further questions.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.