- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
VeriFine iteratively refines an embodied-reasoning policy, curriculum, and rubric judge. The judge diagnoses failures and selects data; human coactive calibration updates the rubric after stagnation, followed by student-judge distillation. Driving uses GRPO and navigation uses SFT. Each controlled policy run starts from the same base model, so supervision state accumulates across rounds. Driving RB reasoning scores improve from 60.56 to 71.61, navigation scores from 68.09 to 77.59, and judge–human alignment improves.
Critical strengths
- S1: The framework distinguishes valuable difficult contexts from incorrect demonstrations and supports both reward-based optimization and data filtering.
- S2: A fixed RB evaluator, common initialization and per-policy budgets, curriculum ablations with fixed reward judges, and same-pool test-time selection provide useful controls.
- S3: Later challenging cases expose specific weaknesses of the old judge that refinement improves. Backbone and student ablations add supporting evidence.
Critical weaknesses
- W1: Main policy benefits rely on offline evaluators sharing rubric and LLM-label sources. Keeping RB fixed avoids a moving metric but does not replace independent policy assessment. Driving ADE changes only from 2.139 to 2.117 and worsens in early rounds. Navigation lacks task success, SPL, or collision outcomes. Reasoning scores do not establish improved closed-loop capability (Table 1; §4.4; Appendix A.3.3).
- W2: Coactive calibration can revise both rules and human calibration labels, without reporting label changes, annotator agreement, human time, or an equal-budget ordinary calibration control. Independent frozen test labels are a strength, but improved agreement does not establish convergence to a unique objective physical rubric (§3.4.2; Appendix A.4.3).
- W3: Per-policy budget matching excludes earlier policy runs, human queries, rubric search, and judge distillation. Resetting policies also limits an interpretation of continuous policy self-improvement. Equal-total-cost one-shot strong-judge/curriculum and sequential-training baselines are missing (§§3.1 and 4.1).
- W4: Appendix A.4.3 does not fully specify how the 0.5 trigger threshold combines signals; curriculum scores and tie-breaking remain abstract, and the Pearson 0.2 stagnation threshold needs units. The claim that curriculum advantage “grows” also conflicts with absolute gains declining from 8.59 to 5.27 to 3.08.
Questions for authors
- Q1 → W1: Can policies receive independent assessment blind to their versions, with closed-loop driving collisions/violations and navigation success/SPL, including scene-clustered intervals?
- Q2 → W2: Can human time, label revisions, and annotator agreement be quantified, and ordinary fixed-label calibration compared with coactive calibration under equal feedback budgets and the same frozen blind test set?
- Q3 → W3: Can all training, API, and human costs be included when comparing the iterative system with one-shot strong-judge/curriculum and sequential policy training?
5 comments
Weakness 3 and Question 3
In terms of cost, the reported comparisons match per-policy training budgets.
- The driving policy is trained from the same base checkpoint with the same 2,700 GRPO steps to ensure a fair comparison.
- In the Table 3 latency setup, the student takes 0.07 s/sample and the API teacher takes 20.92 s/sample.
- Each judge improvement round allows up to 20 rubric versions. We request human review when the average improvement in Pearson correlation stays below 0.2 for more than ten rounds.
- The time required depends on the failure pattern and averages about five minutes per intervention.
- End-to-end costs include failure analysis, data scoring, human review, rubric search, teacher labeling, and student distillation. Including agent iterations, policy fine-tuning, and human guidance where applicable, it takes about two days for a complete policy improvement loop and one day for a complete judge improvement loop.
Additionally, for the question "Resetting policies also limits an interpretation of continuous policy self-improvement. Equal-total-cost one-shot strong-judge/curriculum and sequential-training baselines are missing".
- Actually, in each improvement round, we train the policy from the same base initialization with a different judge, which lets us compare successive supervision states at a fixed per-policy training budget.
- We view VeriFine-Judge-R3 (Third Round) in Table 1 as the setting corresponding to this comparison. Tables 1 and 2 report its results and the relevant comparisons.
- In a continuous training setting, updates in earlier and later rounds affect one another, so the ablations cannot isolate the contribution of each module. In our setting, the progressively stronger judge preserves the knowledge gained and the priorities for improvement identified in previous rounds. On the other hand, without the full loop improvement, we have no final strong judge.
Weakness 4
Driving round transitions.
-
During policy training, we compute the mean policy reward over each 150-step window using the current reward judge and evaluate the checkpoint at the end of that window using the fixed evaluation judge. In Eq. (7), averages the changes in checkpoint performance over the last four windows, and measures the absolute difference between the changes in mean reward and checkpoint performance from one window to the next. We scale both scores to 0–100 before computing these quantities. The transition criterion is or .
-
For the controlled comparison, each policy starts from the same base and trains for the full 2,700 steps. After training, we apply the transition criterion to select the checkpoint at the end of a qualifying window. We use that checkpoint for the judge improvement loop and report the policy at step 2,700 for the matched-budget comparison.
Curriculum selection.
- After each round, we measure policy performance by scenario category on the policy evaluation set. We shift part of the data quota from stronger categories to weaker ones according to how far the weaker categories fall below the mean. We retain examples from stronger categories to preserve coverage and add diverse examples from weaker ones. The updated curriculum blends the previous category targets (60%) with newly added examples (40%). The curriculum judge applies a target quality threshold of 0.8 to filter out candidates with unreliable annotations.
- For each retained candidate, we generate a rollout with the current policy and score its reasoning with the current curriculum judge. We define learnability as the ground-truth reasoning score minus the policy rollout score. We rank candidates by this gap, prioritizing larger gaps because they indicate greater room for improvement and a stronger potential learning signal.
Additional, Pearson's r is a dimensionless correlation coefficient. As the reviewer noted, the original sentence omitted the subject of “grows.” We will revise it to read “The curriculum advantage persists and the reasoning score grows across judge versions.”
We hope the clarifications can address your concerns, and we would be grateful if you would consider raising your overall rating. We also would be happy to address any further questions.
Follow-Up to Your Review
Dear Reviewer knBd,
Thank you again for the time and effort you put into reviewing our work!
We understand that this might be a super busy period for you. We wanted to briefly let you know that we have posted point-by-point responses to your questions and concerns.
Whenever you have a chance, we would be grateful for your feedback on whether the responses address your concerns. We would be happy to clarify anything that remains open.
Thank you!
Thank you for the detailed review.
Weakness 1 and Question 1
Independent policy evaluation. We would like to clarify the role of VeriFine-Judge-RB. We introduced this reference-based model to make evaluation efficient and scalable without requiring human experts to score every policy rollout, which is too expensive and time-consuming. It compares rollout reasoning with ground-truth reasoning that human experts annotated and checked. Its rubric is designed around that reference and includes criteria specific to reference-based evaluation, such as semantic agreement between the reference and the rollout. We therefore use VeriFine-Judge-RB as a proxy for the upper bound on scene-level reasoning scores. Fig. 3 shows strong agreement with human scores, with Pearson r = 0.82 and MAE = 0.16. We attribute the remaining gap to the limited availability of human-scored examples for fine-tuning, which are costly to obtain.
To address the concern about independent policy assessment, we have compared with LingoJudge in the paper, which we use as a state-of-the-art driving-reasoning baseline. In Fig. 3, both VeriFine-Judge-RB and the final VeriFine-Judge student improve Pearson correlation by about 26% relative to LingoJudge. Policies improved with LingoJudge also score lower than those improved with VeriFine-Judge under both the RB and RF evaluators.
We additionally use independent, blinded human scores in the test-time scaling experiment in the upper-right panel of Fig. 3. Each judge selects the candidate it considers best, and we compare the selected candidate's human score with the human oracle. These scores show improvements across the VeriFine-Judge rounds from R1 to R3 and also show that VeriFine-Judge-RB selects better candidates than LingoJudge.
From Base to R3, open-loop ADE improves from 2.139 to 2.117 m and minADE₆ from 1.049 to 1.029 m. We attribute the modest action gains to our decision to optimize the reasoning reward alone. The goal was to measure gains in reasoning independently of a separate action reward. Reasoning and actions can diverge in a VLA policy, so a correct action could give positive feedback to a rollout whose reasoning is incorrect. Even under this objective without action reward, our final policy achieves the lowest ADE, suggesting that improved reasoning can benefit the existing action expert. A next step is to use the improved reasoning to train the action expert to produce higher-quality trajectories that better align with the reasoning.
The same applies to navigation. We focus on reasoning quality and use only reasoning supervision for SFT. Reasoning quality is harder to assess than navigation success, which can be checked using verifiable rules. Improving trajectory quality requires further training of the corresponding action expert, which is our next step.
Weakness 2 and Question 2
During coactive calibration, only labels in the judge evaluation set can be revised. For final evaluation, we use an independent judge test set that multiple trained annotators labeled independently. Coactive calibration starts from the premise that human feedback can contain errors and that annotators may be less familiar with the reasoning required in particular scenarios. The agent can draw on its world knowledge and common sense to help experts revisit their judgments, and experts can correct the agent in turn.
Consider a highway scenario in which another vehicle cuts in from the right. The ego vehicle may need no response to a distant cut-in, while it may need to slow down for a close one. The boundary is how far away the cut-in vehicle can be before the ego vehicle needs to slow down. Human judgments about that distance can vary across scenes. The agent may be too conservative, while the human may be too optimistic. By correcting each other, they can reach a better judgment and move closer to an objective rubric for physical reasoning. In other scenarios, the agent may make basic errors that the human expert can correct.
In Round 3 of the driving reasoning task, we additionally compare ordinary fixed-label calibration with coactive calibration under equal feedback budgets on the same frozen, blinded judge test set.
Human effort and intervention timing. A human intervention takes about five minutes on average, depending on the failure pattern. In the Table 3 inference setup, the student takes 0.07 s/sample and the API teacher takes 20.92 s/sample.