acceptodds
Under review as a conference paper at ICLR 2027

Is It a Good Match? 2D Keypoint Matching with Feedback of a VLM

Abstract

Two-dimensional (2D) keypoint matching is a fundamental problem in computer vision, used in applications such as image registration, 3D reconstruction, and visual localization. Recent Vision Foundation Models, such as DINOv3, have demonstrated strong transferable representations across a range of vision tasks, including keypoint correspondence estimation. However, their performance remains limited by imperfect feature distinctiveness and weak geometric awareness. Their practical usage often relies on task-specific fine-tuning with carefully engineered losses. Vision-Language Models (VLMs) have shown strong capabilities in evaluating and reasoning on multiple vision tasks, including assessing the quality of vision model predictions. However, their high inference cost makes them impractical for direct use in matching pipelines. In this work, we leverage the evaluative strength of VLMs to guide the fine-tuning of vision foundation models for 2D keypoint matching. Specifically, we construct a feedback signal based on a VLM-based evaluation score (VLM-as-Judge) of predicted correspondences. We introduce a feedback-driven training framework using the VLM score to fine-tune a DINOv3 backbone for keypoint matching. By integrating this reward-based feedback loop, our method improves correspondence quality without requiring explicit ground-truth supervision or loss engineering. At test time, we rely only on the fine-tuned model, running 50-160x faster than a single VLM call.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
2D keypoint matching, vision foundation models, DINOv3, vision-language models, VLM-as-a-Judge, feedback-driven fine-tuning, reward-based learning, correspondence estimation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_CrfcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper presents a method that finetunes DINOv2 or DINOv3 with LoRA adapters for semantic keypoint matching with REINFORCE. The reward is a frozen VLM's score of a candidate match drawn as markers on the image pair, so no target keypoint labels are used. A geo-aware variant fuses cached Stable Diffusion features through an aggregation network, with a residual initialization that keeps the DINO features untouched at the beginning of training.

Experiments on SPair-71k and PF-PASCAL compare several VLM judges (Kimi K3, Gemini 3.8 Flash, GLM-4.6V, GPT-6 Astra, Opus 5.5 among others) against frozen features, Top-K, DPO, direct prediction and a ground truth reward. The method is extended to text-queried keypoints on KeypointNet: one judge call scores for 8 candidates, queries use CLIP or learned name embeddings, and 2D predictions are lifted to 3D by multi-view fusion. At test-time, the VLMs are not used.

Critical strengths
  1. Simple, general idea with cheap inference. The VLM's judgment is used only at training time, so inference needs no VLM call. The same loop works for two tasks, pair matching and text-queried keypoints. The experiments show that it also generalizes to many different VLMs.

  2. Useful analysis of the VLM judge. The calibration study in section 4.3 shows that the reward stops improving below about 20 pixels, which might be due to the patch representation. The paper also ablates marker shape (Table 5) and judge format (Table 12) and states these limits openly. The gain over Top-K grows on hard pairs (Table 2, +2.90 points on hard pairs). These findings are useful for anyone using VLMs as rewards for dense prediction.

Critical weaknesses
  1. The main gains are shown on the validation split, and the test results do not support the method.
  • Table 1 reports SPair validation scores, taking the best seed for 3 runs (l.309-309), and the same split is used for ablations, which seems like a confounded factor for fair evaluation against other methods. It indeed seems the case as the test scores drop sharply: geo-aware + Kimi goes from 85.89 on val to 78.44 on test (Table 1 and 4). Frozen DINOv3 on test reaches 73.27 on test (Table 7) and beats DINOv3 + LoRA at 70.07 (l.79 in appendix), which points to validation overfitting as well.
  • In Figure 6, the curve starts close to their final value. PF-PASCAL geo-aware starts at about 92.5, then decreases, and goes back to 92.5 at the end of training. Same, for train_dino, PCK stays around 77-78 on SPair, so the "22% of the budget" claim therefore says little. These curves also do not match Table 1 (85.65). Is there missing data in Figure 6? Otherwise, I might not have understood this plot.
  1. Numbers are inconsistent across tables, and part of the work is unfinished.
  • Table 1 mixes training budgets and evaluation splits within a column (l.311), which makes the comparison irrelevant.
  • DINOv3 + LoRA (Kimi K3) gets 62.16 on SPair-all in Table 1, and 58.57 in Table 6. From l.166 in the appendix, "evaluation protocols vary" which makes it very difficult to understand or interpret.

Overall, the tables are a bit confused, it is very difficult to understand why the evaluation protocols are different across tables. The whole point of evaluation is to be able to compare models against identical settings. An explanation is clearly needed here to motivate these different settings. I might not have fully understood the evaluation protocol as well, but the results are a bit all over the place, and it is not clear why there are inconsistencies.

Questions for authors
  1. Please provide one SPair cat test table with the same procotol and resolution for every row, reporting mean and std without seed selection on validation. It should include frozen DINOv3, DINOv3 + LoRA + Kimi with REINFORCE, Top-K, DPO, the ground truth reward, and geo-aware + LoRA + resid. (Kimi K3).
  2. Why does the ground truth reward (82.56) do worse than the Kimi reward (85.65) on SPair cat val? A binary PCK reward should be close to an upper bound for a judge that cannot resolve errors below 20 pixels.
  3. Overall, could you please explain the choice of not having the same evaluation protocol for all methods?
  4. This is a suggestion: the judge cannot tell a 5 pixel error from a perfect match (l.409), likely because of patch-level visual tokens. You could test giving the judge the raw pixel coordinates instead, and/or in addition to the markers. Another option is a zoomed crop around the candidate. Coordinates alone may not be enough, since direct prediction is quite weak (Table 1), but combined with the marker, they might sharpen the reward below 20 pixels. This is also quite cheap to test.
2 comments
AuthorsOpenReview ↗

Response to Reviewer Crfc (Submission #30584) Cont.

We thank Reviewer Crfc for carefully auditing our tables, figures, and protocols, and for the constructive suggestion in Q4 on testing pixel coordinates and zoomed crops.

Q2: Why did the ground-truth reward (82.56) do worse than Kimi (85.65) on SPair cat val?

Thank you—your intuition is right in the strict short-budget setting (at a matched 1,000-step budget in Table A / Appendix E.1 Table 15, the binary GT [email protected] reward is slightly ahead on both paths, and in Table 1 82.56 used a shorter training budget than 85.65).

However, we did observe VLM-reward policies matching or outperforming the binary GT [email protected] reward across multiple controlled converged experiments under the exact same protocol and stopping rule (e.g., 86.24 / 86.06 / 85.82 VLM vs. 84.22 GT on SPair cat Geo-aware in Appendix E.2 Table 16; 71.93 / 71.09 VLM vs. 69.55 GT on SPair all; and 88.58 VLM vs. 87.58 GT on PF-PASCAL). Our hypothesis is that the VLM judge provides more meaningful semantic feedback than a hard binary GT indicator when keypoint definitions are inherently ambiguous:

  • Single-pixel human keypoint annotations on deformable categories (e.g., exact boundary placement on ears, limbs, or paws) carry natural label ambiguity, and the binary GT reward (1[d≤0.10]\mathbb{1}[d \le 0.10]) gives zero directional signal outside the threshold (0 whether a prediction is a near-miss at 0.11 or on the background) and a flat 1 inside.
  • By contrast, the VLM judge returns graded semantic scores in [0,1][0, 1] (giving partial credit on semantically close placements vs. 0 on background, while still preferring closer matches within the part; Appendix E.3, Tables 18–19), providing smoother optimization and natural regularization against ambiguous point labels (though training the binary GT reward much longer to 36k steps eventually reaches a slightly higher score on the exact GT metric, Figure 7).

Q3 & W2: Why were protocols different across tables (62.16 vs. 58.57)?

Thank you for pointing out this inconsistency in the Supplementary Material; we have uploaded the corrected version. There were no different protocols across tables; different components of the project used separate checkpoints based on their convergence. The evaluation for all methods within each comparison remains the same and fair.


Q4: Testing pixel coordinates in the prompt and a zoomed crop around the candidate

Thank you for this suggestion! we compared four prompting setups on the same SPair cat keypoints across controlled pixel errors (00–160160 px, 2,9502{,}950 calls/arm, Gemini 3.8 Flash; cells report mean VLM score):

  1. Marker only (baseline): green circle markers drawn on the source and target images.
  2. Marker + coordinates: visual markers plus fractional coordinates (x,y)∈[0,1]2(x, y) \in [0, 1]^2 in the prompt.
  3. Coordinates only: unmarked images with source and target coordinates in the prompt.
  4. Marker + 2×2\times zoom: visual markers with both views cropped at 2×2\times magnification around the marked points.
JudgeError: 0 px5 px10 px20 px40 px80 px160 px
1. Marker only (baseline)0.9580.9350.8730.4790.1210.0390.005
2. Marker + coordinates0.9500.9290.8790.4800.1170.0290.008
3. Coordinates only0.8970.8820.8930.6770.2840.0880.029
4. Marker + 2×2\times zoom0.9480.9390.7840.3170.0720.0170.004
  • Coordinates alone (Arm 3) are substantially coarser than visual markers, confirming your expectation.
  • Marker + coordinates (Arm 2) produces virtually the same curve as the marker alone. When a visual marker is present, the VLM relies on visual tokens rather than coordinate arithmetic.
  • Marker + 2×2\times zoomed crop (Arm 4) sharply tightens the reward drop-off between 10 px and 40 px; on a coarser open-weight judge (Qwen3-VL-32B), 2×2\times zoom restores strict 0 px > 5 px ordering (0.972 > 0.968) and improves 3-seed 1k-step validation [email protected] from 78.63 ±\pm 0.21 to 79.00 ±\pm 0.10.
AuthorsOpenReview ↗

Response to Reviewer Crfc (Submission #30584)

We thank Reviewer Crfc for carefully auditing our tables, figures, and protocols, and for the constructive suggestion in Q4 on testing pixel coordinates and zoomed crops.

In the submission, Tables 1 and 6 were trained with different training budgets based on their convergence. The evaluation for all methods within each comparison remains the same and fair. To compare all methods under identical settings while keeping edits to the main text and Appendices A–D minimal, we report controlled unified evaluations directly below and in Appendix E of the revised Supplementary Material.

Below we address each weakness and question directly with the unified tables and new experiments.


Q1 & W1: Unified SPair cat test table (same protocol & resolution, mean ±\pm std without seed selection) and test-split results

Table A below provides the exact controlled comparison requested in Q1 on SPair cat validation (272 pairs) and test (600 pairs) at a fixed budget of 1,000 judge calls (1,000 steps for REINFORCE and DPO at 1 call/step; 200 steps for Top-KK at K=5K=5 calls/step), with DINOv3 ViT-B/16, LoRA (r=16,α=8r=16, \alpha=8), τ=0.1\tau=0.1, final checkpoints, and no validation seed selection. We also include the ground-truth (GT) binary [email protected] reward run under the exact same command and budget, and the untrained (t=0t=0) control for the geo-aware stack:

Method (SPair cat, 1,000 calls)Val @0.10Test @0.10Test @0.05Test @0.01Seeds
Plain path (5122512^2, pixel-level NN)
Frozen DINOv3 (no training)77.9870.0561.2013.81–
+ LoRA, Top-KK (K=5K=5, Kimi K2.5)78.06 ±\pm 0.0470.14 ±\pm 0.0761.30 ±\pm 0.0413.75 ±\pm 0.063
+ LoRA, DPO (Kimi K2.5)78.79 ±\pm 0.3171.49 ±\pm 0.3762.25 ±\pm 0.2513.94 ±\pm 0.213
+ LoRA, REINFORCE (Kimi K2.5)78.94 ±\pm 0.1771.46 ±\pm 0.0362.26 ±\pm 0.0914.05 ±\pm 0.113
+ LoRA, REINFORCE (GPT-5.5)79.50 ±\pm 0.2372.07 ±\pm 0.2762.62 ±\pm 0.2714.05 ±\pm 0.063
+ LoRA, REINFORCE (Gemini 3 Flash)79.34 ±\pm 0.2772.03 ±\pm 0.3162.86 ±\pm 0.3014.37 ±\pm 0.073
+ LoRA, REINFORCE (GT reward, same protocol)79.78 ±\pm 0.1772.32 ±\pm 0.3663.49 ±\pm 0.4014.45 ±\pm 0.243
Geo-aware path (60×60→256260\times 60 \to 256^2 NN)
Geo-aware at t=0t=0 (untrained control)81.5775.9165.0216.54–
Geo-aware + LoRA + resid., REINFORCE (Kimi K2.5)82.60 ±\pm 0.1177.38 ±\pm 0.2566.53 ±\pm 0.2818.19 ±\pm 0.483
Geo-aware + LoRA + resid., REINFORCE (GT reward, same protocol)84.19 ±\pm 0.2179.07 ±\pm 0.2568.68 ±\pm 0.2219.09 ±\pm 0.353
Reviewer_AVffOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper fine-tunes DINOv2 or DINOv3 with LoRA for semantic keypoint correspondence. Source keypoints are given. For each source point, the model samples a target location from its dense feature similarity, a frozen VLM scores the marked pair from 0 to 1, and REINFORCE updates the adapter. At test time only the vision model is used. A geo-aware variant fuses cached Stable Diffusion features. The same loop is applied to text-queried keypoints on KeypointNet. Experiments cover SPair-71k, PF-PASCAL, several VLM judges, and supervised reward baselines

Critical strengths
  1. The label-free setup is clear and coherent. A source keypoint is given, while the target has no coordinate label. The VLM sees the marked pair and returns a score in [0,1], and REINFORCE uses that score to update LoRA. Test-time matching uses only the fine-tuned vision model. Extending the same loop to named keypoints is also clear: the text query replaces the source point, and one call can score eight marked candidates.
  2. The method is written clearly, and the experiments cover a wide set of judges. Section 3 separates feature adaptation, reward parsing, and the text-query policy. Table 1 includes frozen DINO, direct VLM prediction, Top-K, DPO, several commercial judges, and supervised reward training.
Critical weaknesses
  1. The VLM-reward policy still trails supervised TLFR, and the training cost is left unmeasured. On 18-category SPair-71k, supervised TLFR with DINOv3 reaches 78.24 [email protected], while the best VLM-reward entry in Table 1 reaches 71.20. The training expense makes this gap more consequential. The main loop sends every sampled correspondence to a VLM, and the paper puts a single call at about six seconds, so training proceeds at roughly ten samples per minute before API rate limits. The reported 50–160× speedup applies only at test time. The paper mentions runs of 1,000 epochs or longer but does not report the number of calls, wall-clock time, or API cost, and it does not discuss this cost as a limitation.
  2. The qualitative evidence does not show the method being claimed. Figure 3 is the main correspondence visualization, and its caption states that the adapted model was trained with supervised REINFORCE. It therefore does not show matches learned from the VLM reward. Figure 4 adds one left/right example for the geo-aware variant, without showing a VLM-reward correspondence field. The main text and the appendix do not provide further correspondence visualizations, and they do not visualize text-queried keypoints, paraphrased queries, or the multi-view 3D predictions.
Questions for authors
  1. For each main result, how many VLM calls, how much wall-clock time, and what API cost were required? Please also compare against supervised TLFR on the same split and at a matched training budget.
  2. Please add qualitative results for the VLM-reward policy and for text-queried keypoints, including failure cases. The current Figure 3 shows supervised REINFORCE.
1 comment

We thank Reviewer AVff for the clear, constructive review and for highlighting two critical gaps in the submitted manuscript: (1) the missing training cost accounting (VLM calls, wall-clock time, and API cost) and the source of the performance gap to supervised TLFR, and (2) the fact that Figure 3 visualized a supervised REINFORCE model rather than a policy trained with VLM rewards.

We have addressed both points directly below with training cost measurements, controlled Ground-Truth (GT) reward and matched-budget supervised TLFR comparisons, and rule-selected qualitative results (including failure cases) for our VLM-reward policies.


Q1 & W1: Training cost and comparison against supervised TLFR

1. Training cost

Each training step makes 1 VLM call:

  • 1,000-call run (Table 1): 1,000 VLM calls, 1.5 hours, $1.
  • Converged run (45 runs: 5 judges ×\times 3 models ×\times 3 benchmarks): mean 6,689 VLM calls, 15.0 hours, $64 across all 5 judges ($7 mean with Gemini 3.8 Flash).

2. Comparison with supervised TLFR and isolating the source of the gap

To separate reward noise (VLM vs. ground-truth labels) from the training recipe (1-point zeroth-order REINFORCE vs. TLFR's dense first-order loss over all keypoints per pair), we compare our converged VLM-reward policies, a GT-reward control in our exact REINFORCE pipeline, and supervised TLFR-DINOv3 at matched pair budgets on SPair-71k test ([email protected]; Appendix E.2, Table 16):

Method (Geo-aware + DINOv3)SupervisionPair Steps (cat / all)SPair cat test (600)SPair all test (12,234)
Untrained control (t=0t=0)None075.9158.67
Ours (Gemini 3.8 Flash)VLM bandit (1 pt/pair, 0 GT)12k / 12k86.0669.49
Ours (Claude Opus 5.5)VLM bandit (1 pt/pair, 0 GT)7k / 11k85.8269.63
Ours (GPT-6.1 Sol)VLM bandit (1 pt/pair, 0 GT)2k / 18k83.4571.09
Ours (GPT-6 Astra)VLM bandit (1 pt/pair, 0 GT)10k / 18k86.2471.93
GT-reward control (same REINFORCE rule)GT bandit (1 pt/pair)5k / 12k84.2269.55
Supervised TLFR-DINOv3 (matched budget)Dense GT loss (all kps/pair)10k / 18k82.9676.03
Supervised TLFR-DINOv3 (converged)Dense GT loss (all kps/pair)9.9k / 320k83.4078.24
  1. Supervision and training budget (320k vs. 16k steps): Converged TLFR is trained for 320k steps with supervised dense all-keypoint supervision, whereas our policies are label-free at a much smaller 16k-step budget.
  2. Better VLM judges continue to push the boundary: Our latest Gemini 4 Argon (14k steps) →\to Carbon (+2k steps) run already lifts SPair all test [email protected] further to 72.42.

Q2 & W2: Qualitative results for the VLM-reward policy (including failure cases) and text-queried keypoints

You are right that Figure 3 in Section 4.2 showed a supervised REINFORCE model rather than a VLM-reward policy. Please see Appendix E.4 of the updated Supplementary Material for rule-selected visualizations of our label-free VLM-reward policies:

  • 2D correspondence (Figures 8–9): Across the SPair-71k test, Figure 8 shows the top rule-selected gains (dog, tvmonitor, chair) and Figure 9 shows a separate failure case (pottedplant, where features drift along a radially symmetric rim without distinct sub-parts).
  • Text-queried keypoints (Figure 10 & Table 21): Figure 10 shows rule-selected 2D gains under both standard and unseen paraphrased queries (car, chair, bathtub) alongside paired 14-view 3D consensus lifts (chair, knife; ), together with the complete 10,000-iteration test evaluations in Table 21.

We will include those results in the final paper.

Reviewer_5XYYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to adapt vision foundation models without using ground-truth keypoint correspondences as training supervision, so as to improve 2D semantic keypoint matching capability.

Given a source keypoint, the model uses DINO features to define a categorical probability distribution over candidate locations in the target image. A frozen vision-language model (VLM) then scores the sampled correspondences, and this score is used as a reward to update the LoRA parameters via RL. The authors also propose a geometry-aware variant that uses a learnable aggregation network to fuse DINO features and pre-cached Stable Diffusion features. Furthermore, the paper extends the framework to keypoint localization specified by natural language names through CLIP-derived text query representations and multi-token VLM feedback. Results on SPair-71k, PF-PASCAL, and KeypointNet demonstrate that VLM feedback can improve matching while avoiding VLM calls during inference deployment.

Critical strengths
  1. This paper uses a VLM as a judge during training rather than as a coordinate predictor at test time, this method explores whether expensive, non-differentiable semantic judgment capabilities can be transferred to an efficient dense visual matching model, and it does so without requiring training on ground-truth target keypoint correspondences.

  2. The paper compares alternative optimization methods such as Top-K and DPO, various VLM judges, a control using ground-truth label rewards, and text-query keypoint localization. The authors observe that VLM scores plateau within a small range of localization errors, revealing the applicable boundaries of the proposed feedback signal.

Critical weaknesses
  1. Insufficient alignment between VLM rewards and fine-grained geometric correspondence objectives.

The judge's prompt requires determining whether two points lie on the same semantic part, while matching performance is evaluated by the error between predicted locations and manually annotated correspondences. The two are correlated but not equivalent: two locations may belong to the same semantic part yet differ significantly in geometric precision. The paper reports that when localization error is below approximately 20 pixels, the judge's score plateaus (Section 4.3), indicating its ability to distinguish neighboring but geometrically distinct candidate locations may be limited. Furthermore, the proposed reward targets individual keypoint pairs and does not explicitly enforce geometric consistency across the entire set of correspondences. Therefore, current evidence more directly supports improved semantic matching selection, without adequately demonstrating improvements in precise and geometrically consistent correspondences. Analyzing high-probability, hard candidates, especially symmetric parts, errors near the PCK threshold, and using object-scale-normalized distances, would significantly strengthen the persuasiveness of this methodological claim.

  1. The core experimental comparisons fail to adequately isolate the incremental contribution of feedback-driven optimization.

The main results conflate different training budgets and evaluation splits (Section 4.1); the baseline using direct VLM predictions and some REINFORCE results also explicitly adopt different data splits (Section 4.4). In the same-budget comparisons provided by the authors, REINFORCE improves over Top-K by only 0.88 ± 0.15 PCK percentage points, while the gap between DPO and REINFORCE falls within the fluctuation range of random seeds (Section 4.2). The geo-aware variant simultaneously introduces Stable Diffusion features, a learnable fusion module, and setup differences beyond mere resolution adjustments. Additionally, Figure 3 shows qualitative matching results for REINFORCE under supervised rewards, rather than the VLM reward training results claimed in the paper. These phenomena do not negate the value of VLM feedback, but make it difficult for readers to judge the incremental contribution of the proposed method and the specific source of the larger gains in Table 1. Controlled evaluations under unified training and computational budgets are needed to disentangle the respective roles of reward sources, optimization rules, and feature structures.

Questions for authors
  1. The granularity of rewards is aligned with the difficulty. Could the authors evaluate the VLM judge on candidate points actually sampled by the learned matching strategy, and report results separately by semantic confusion type (e.g., left-right symmetric parts), object scale, and geometric distance to the ground-truth keypoint? After VLM reward training, how does performance change under stricter PCK thresholds? This would help determine whether the feedback signal can not only suppress distant incorrect matches but also improve fine-grained localization already within the correct semantic part.

  2. Controlled comparison and full cost accounting. Could the authors report results for frozen DINO, Top-K, DPO, REINFORCE, and Geo-aware variants under the same visual backbone, input resolution, train/evaluation splits, VLM judge, and the same candidate point scoring/API call budget, providing mean ± std over multiple random seeds? Additionally, please distinguish between inference speed when Stable Diffusion features are already cached and the full end-to-end processing time for a completely new image.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 5XYY

We thank Reviewer 5XYY for the supportive assessment and constructive questions. We conducted all requested analyses and controlled experiments, and report the complete tables, cost/latency breakdowns, and rule-selected visualizations in Appendix E of the updated Supplementary Material (keeping the main paper and Appendices A–D minimally changed). Below we summarize the key conclusions.


Q1 & W1: On-policy judge analysis, stricter PCK thresholds, and viewpoint difficulty

  • Clarification & fine-grained on-policy judge breakdown (Appendix E.3): The judge–policy alignment analysis in our paper (Section 4.3 / Appendix B.2) is already evaluated directly under the same learned matching strategy rather than only synthetic displacements. To further break down this behavior by semantic confusion type, object scale, and geometric distance, we ran an additional stratified evaluation on candidates sampled by the learned policy on SPair cat test (summary below; full in Appendix E.3, Tables 18–19). It shows a clear within-part distance reward gradient, strong separation over symmetric and off-target confusions:
On-Policy Candidate Region (t=0.1max⁡(h,w)t = 0.1\max(h,w))Frozen ArgmaxTrained ArgmaxMean Judge Score
Correct part (d≤0.10d \le 0.10) (by distance 0–.01 / .01–.05 / .05–.10: .964 / .868 / .571)70.1%82.8%0.773
Left/right counterpart14.1%7.9%0.397
Other annotated part6.6%4.1%0.252
Elsewhere in box8.5%5.2%0.269
Background0.8%0.0%0.041
  • Stricter PCK thresholds and viewpoint difficulty (Appendix E.1 & Figure 7): Under a unified validation-patience convergence rule across all architectures and benchmarks, VLM-reward policies consistently outperform their untrained test controls under both standard (@0.10) and strict (@0.05) PCK thresholds, with gains growing monotonically from Easy to Medium to Hard viewpoint pairs. Ground-truth reward controls confirm that ultra-fine (@0.01) saturation on plain ViTs stems from coarse patch resolution and a region-level objective rather than VLM noise, while the Geo-aware stack improves even at @0.01.

Q2 & W2: Budget comparison, training cost, and cached vs. uncached latency

API budget without seed selection (table below), resolution scaling alone has negligible effect, the untrained Geo-aware stack provides a fixed structural boost prior to RL, single-sample REINFORCE matches DPO while Top-KK stalls, and VLM feedback recovers roughly half of the matched GT gain. Matched-split direct VLM comparisons (Appendix E.3) and rule-selected VLM-reward correspondence / failure-case / KeypointNet visualizations (Appendix E.4) are in the updated Supplementary Material.

Method (SPair cat, 1,000 calls, 3 seeds)Val @0.10Test @0.10Test @0.05Test @0.01
Frozen DINOv3 (5122512^2 NN; 78.51 val at 9602960^2)77.9870.0561.2013.81
DINOv3 + LoRA, Top-KK (K=5K{=}5, Kimi K2.5)78.06 ±\pm 0.0470.14 ±\pm 0.0761.30 ±\pm 0.0413.75 ±\pm 0.06
DINOv3 + LoRA, DPO (Kimi K2.5)78.79 ±\pm 0.3171.49 ±\pm 0.3762.25 ±\pm 0.2513.94 ±\pm 0.21
DINOv3 + LoRA, REINFORCE (Kimi K2.5)78.94 ±\pm 0.1771.46 ±\pm 0.0362.26 ±\pm 0.0914.05 ±\pm 0.11
DINOv3 + LoRA, REINFORCE (GT reward)79.78 ±\pm 0.1772.32 ±\pm 0.3663.49 ±\pm 0.4014.45 ±\pm 0.24
Geo-aware at t=0t{=}0 (602→256260^2{\to}256^2 NN)81.5775.9165.0216.54
Geo-aware + LoRA + resid., REINFORCE (Kimi K2.5)82.60 ±\pm 0.1177.38 ±\pm 0.2566.53 ±\pm 0.2818.19 ±\pm 0.48
Geo-aware + LoRA + resid., REINFORCE (GT reward)84.19 ±\pm 0.2179.07 ±\pm 0.2568.68 ±\pm 0.2219.09 ±\pm 0.35
Training cost & cached vs. uncached new-image latency (Appendix E.2 & Table 17):
  • Training cost (1 VLM call/step): 1,000-call run: 1,000 VLM calls, 1.5 hours, $1; Converged run (45 runs): mean 6,689 VLM calls, 15.0 hours, $64 across all 5 judges ($7 mean with Gemini 3.8 Flash).
  • Test-time latency (Table 17 below): Matching requires zero VLM calls; on an uncached new image, single-step Stable Diffusion extraction dominates Geo-aware latency but runs once per image across all keypoints.
Inference Regime (50 SPair cat pairs, RTX 3080 Laptop)SD / CacheDINOAggNetMatchms / img (FPS)ms / pair
Plain DINOv3 + LoRA (5122512^2, merged weights)0.0058.290.0014.1165.35 ±\pm 4.25 (15.3)130.69
Geo-aware + LoRA (9602960^2, SD cached in RAM / .pt disk)0.01 / 5.58202.2 / 203.16.110.2213.4 / 219.9 (4.69 / 4.55)426.8 / 439.8
Geo-aware + LoRA (9602960^2, live SD v1-5, uncached image)2,321.5237.56.610.72,570.9 ±\pm 124.3 (0.39)5,141.8

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.