- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper presents a method that finetunes DINOv2 or DINOv3 with LoRA adapters for semantic keypoint matching with REINFORCE. The reward is a frozen VLM's score of a candidate match drawn as markers on the image pair, so no target keypoint labels are used. A geo-aware variant fuses cached Stable Diffusion features through an aggregation network, with a residual initialization that keeps the DINO features untouched at the beginning of training.
Experiments on SPair-71k and PF-PASCAL compare several VLM judges (Kimi K3, Gemini 3.8 Flash, GLM-4.6V, GPT-6 Astra, Opus 5.5 among others) against frozen features, Top-K, DPO, direct prediction and a ground truth reward. The method is extended to text-queried keypoints on KeypointNet: one judge call scores for 8 candidates, queries use CLIP or learned name embeddings, and 2D predictions are lifted to 3D by multi-view fusion. At test-time, the VLMs are not used.
Critical strengths
-
Simple, general idea with cheap inference. The VLM's judgment is used only at training time, so inference needs no VLM call. The same loop works for two tasks, pair matching and text-queried keypoints. The experiments show that it also generalizes to many different VLMs.
-
Useful analysis of the VLM judge. The calibration study in section 4.3 shows that the reward stops improving below about 20 pixels, which might be due to the patch representation. The paper also ablates marker shape (Table 5) and judge format (Table 12) and states these limits openly. The gain over Top-K grows on hard pairs (Table 2, +2.90 points on hard pairs). These findings are useful for anyone using VLMs as rewards for dense prediction.
Critical weaknesses
- The main gains are shown on the validation split, and the test results do not support the method.
- Table 1 reports SPair validation scores, taking the best seed for 3 runs (l.309-309), and the same split is used for ablations, which seems like a confounded factor for fair evaluation against other methods. It indeed seems the case as the test scores drop sharply: geo-aware + Kimi goes from 85.89 on val to 78.44 on test (Table 1 and 4). Frozen DINOv3 on test reaches 73.27 on test (Table 7) and beats DINOv3 + LoRA at 70.07 (l.79 in appendix), which points to validation overfitting as well.
- In Figure 6, the curve starts close to their final value. PF-PASCAL geo-aware starts at about 92.5, then decreases, and goes back to 92.5 at the end of training. Same, for train_dino, PCK stays around 77-78 on SPair, so the "22% of the budget" claim therefore says little. These curves also do not match Table 1 (85.65). Is there missing data in Figure 6? Otherwise, I might not have understood this plot.
- Numbers are inconsistent across tables, and part of the work is unfinished.
- Table 1 mixes training budgets and evaluation splits within a column (l.311), which makes the comparison irrelevant.
- DINOv3 + LoRA (Kimi K3) gets 62.16 on SPair-all in Table 1, and 58.57 in Table 6. From l.166 in the appendix, "evaluation protocols vary" which makes it very difficult to understand or interpret.
Overall, the tables are a bit confused, it is very difficult to understand why the evaluation protocols are different across tables. The whole point of evaluation is to be able to compare models against identical settings. An explanation is clearly needed here to motivate these different settings. I might not have fully understood the evaluation protocol as well, but the results are a bit all over the place, and it is not clear why there are inconsistencies.
Questions for authors
- Please provide one SPair cat test table with the same procotol and resolution for every row, reporting mean and std without seed selection on validation. It should include frozen DINOv3, DINOv3 + LoRA + Kimi with REINFORCE, Top-K, DPO, the ground truth reward, and geo-aware + LoRA + resid. (Kimi K3).
- Why does the ground truth reward (82.56) do worse than the Kimi reward (85.65) on SPair cat val? A binary PCK reward should be close to an upper bound for a judge that cannot resolve errors below 20 pixels.
- Overall, could you please explain the choice of not having the same evaluation protocol for all methods?
- This is a suggestion: the judge cannot tell a 5 pixel error from a perfect match (l.409), likely because of patch-level visual tokens. You could test giving the judge the raw pixel coordinates instead, and/or in addition to the markers. Another option is a zoomed crop around the candidate. Coordinates alone may not be enough, since direct prediction is quite weak (Table 1), but combined with the marker, they might sharpen the reward below 20 pixels. This is also quite cheap to test.
2 comments
Response to Reviewer Crfc (Submission #30584)
We thank Reviewer Crfc for carefully auditing our tables, figures, and protocols, and for the constructive suggestion in Q4 on testing pixel coordinates and zoomed crops.
In the submission, Tables 1 and 6 were trained with different training budgets based on their convergence. The evaluation for all methods within each comparison remains the same and fair. To compare all methods under identical settings while keeping edits to the main text and Appendices A–D minimal, we report controlled unified evaluations directly below and in Appendix E of the revised Supplementary Material.
Below we address each weakness and question directly with the unified tables and new experiments.
Q1 & W1: Unified SPair cat test table (same protocol & resolution, mean std without seed selection) and test-split results
Table A below provides the exact controlled comparison requested in Q1 on SPair cat validation (272 pairs) and test (600 pairs) at a fixed budget of 1,000 judge calls (1,000 steps for REINFORCE and DPO at 1 call/step; 200 steps for Top- at calls/step), with DINOv3 ViT-B/16, LoRA (), , final checkpoints, and no validation seed selection. We also include the ground-truth (GT) binary [email protected] reward run under the exact same command and budget, and the untrained () control for the geo-aware stack:
Method (SPair cat, 1,000 calls) | Val @0.10 | Test @0.10 | Test @0.05 | Test @0.01 | Seeds |
|---|---|---|---|---|---|
| Plain path (, pixel-level NN) | |||||
| Frozen DINOv3 (no training) | 77.98 | 70.05 | 61.20 | 13.81 | – |
| + LoRA, Top- (, Kimi K2.5) | 78.06 0.04 | 70.14 0.07 | 61.30 0.04 | 13.75 0.06 | 3 |
| + LoRA, DPO (Kimi K2.5) | 78.79 0.31 | 71.49 0.37 | 62.25 0.25 | 13.94 0.21 | 3 |
| + LoRA, REINFORCE (Kimi K2.5) | 78.94 0.17 | 71.46 0.03 | 62.26 0.09 | 14.05 0.11 | 3 |
| + LoRA, REINFORCE (GPT-5.5) | 79.50 0.23 | 72.07 0.27 | 62.62 0.27 | 14.05 0.06 | 3 |
| + LoRA, REINFORCE (Gemini 3 Flash) | 79.34 0.27 | 72.03 0.31 | 62.86 0.30 | 14.37 0.07 | 3 |
| + LoRA, REINFORCE (GT reward, same protocol) | 79.78 0.17 | 72.32 0.36 | 63.49 0.40 | 14.45 0.24 | 3 |
| Geo-aware path ( NN) | |||||
| Geo-aware at (untrained control) | 81.57 | 75.91 | 65.02 | 16.54 | – |
| Geo-aware + LoRA + resid., REINFORCE (Kimi K2.5) | 82.60 0.11 | 77.38 0.25 | 66.53 0.28 | 18.19 0.48 | 3 |
| Geo-aware + LoRA + resid., REINFORCE (GT reward, same protocol) | 84.19 0.21 | 79.07 0.25 | 68.68 0.22 | 19.09 0.35 | 3 |
Response to Reviewer Crfc (Submission #30584) Cont.
We thank Reviewer Crfc for carefully auditing our tables, figures, and protocols, and for the constructive suggestion in Q4 on testing pixel coordinates and zoomed crops.
Q2: Why did the ground-truth reward (
82.56) do worse than Kimi (85.65) on SPaircatval?Thank you—your intuition is right in the strict short-budget setting (at a matched 1,000-step budget in Table A / Appendix E.1 Table 15, the binary GT [email protected] reward is slightly ahead on both paths, and in Table 1
82.56used a shorter training budget than85.65).However, we did observe VLM-reward policies matching or outperforming the binary GT [email protected] reward across multiple controlled converged experiments under the exact same protocol and stopping rule (e.g.,
86.24/86.06/85.82VLM vs.84.22GT on SPaircatGeo-aware in Appendix E.2 Table 16;71.93/71.09VLM vs.69.55GT on SPairall; and88.58VLM vs.87.58GT on PF-PASCAL). Our hypothesis is that the VLM judge provides more meaningful semantic feedback than a hard binaryGTindicator when keypoint definitions are inherently ambiguous:GTreward (1[d≤0.10]) gives zero directional signal outside the threshold (0whether a prediction is a near-miss at0.11or on the background) and a flat1inside.0on background, while still preferring closer matches within the part; Appendix E.3, Tables 18–19), providing smoother optimization and natural regularization against ambiguous point labels (though training the binary GT reward much longer to 36k steps eventually reaches a slightly higher score on the exact GT metric, Figure 7).Q3 & W2: Why were protocols different across tables (
62.16vs.58.57)?Thank you for pointing out this inconsistency in the Supplementary Material; we have uploaded the corrected version. There were no different protocols across tables; different components of the project used separate checkpoints based on their convergence. The evaluation for all methods within each comparison remains the same and fair.
Q4: Testing pixel coordinates in the prompt and a zoomed crop around the candidate
Thank you for this suggestion! we compared four prompting setups on the same SPair
catkeypoints across controlled pixel errors (0–160 px, 2,950 calls/arm, Gemini 3.8 Flash; cells report mean VLM score):0.972 > 0.968) and improves 3-seed 1k-step validation [email protected] from 78.63 ± 0.21 to 79.00 ± 0.10.