- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
GaugeVLM proposes structured geometric supervision for spatial reasoning in vision-language models. A 3D scene engine generates preference pairs, fixed-relation camera groups, and object-intervention blocks. GaugeDPO combines three objectives: preference offsets determined by geometric error, direct supervision of canonical answer rankings across views, and an intervention-profile loss relating answer log-odds contrasts to measured changes in spatial relations. The principal contribution is the use of measured geometry to structure supervision across answers, views, and scene interventions. The paper evaluates three model families on established spatial benchmarks and reports controlled comparisons of margin assignments, objective components, and intervention-supervision alternatives.
Critical strengths
1.The margin-assignment experiment directly tests an important part of the proposed method. Table 3 retains the same direct and intervention supervision while changing the preference offset. The measured-margin variant achieves 64.4 on QSpatial, compared with 49.5 for zero offsets, 52.5 for constant offsets, and 51.5 for shuffled offsets. The shuffled control is particularly informative: it tests whether assigning the geometric magnitude to the correct example matters, rather than merely introducing nonzero margins. These results substantiate the contribution of measured geometric supervision.
2.The intervention objective improves prediction accuracy, not only its own fitting metric. In Table 6, the full method reaches 70.0% block accuracy, compared with 60.0% for pair-plus-direct supervision, 63.7% for the single-gap alternative, and 65.0% for the EqSim-style alternative. The reduction in worst-case geometric error and the gains on held-out intervention settings in Table 7 further support this component. These controls provide evidence for the proposed intervention formulation beyond a generic claim that more supervision improves performance.
Critical weaknesses
1.The headline comparison with SFT confounds the learning objective with additional data, training, and trainable model capacity. Section 4.1 states that SFT uses LoRA with a frozen vision encoder, whereas the subsequent GaugeDPO stage uses newly constructed data and full-parameter fine-tuning, including the vision encoder. Consequently, the gains over GaugeSFT in Table 1 do not isolate the advantage of the proposed objective over supervised learning. The statement in Section 4.2 contrasting the 18.9-point improvement from preference learning with the 3.0-point improvement from imitation is therefore not a controlled comparison of learning approaches. Table 3 establishes that measured offsets outperform alternative offsets within the preference-training setup. It does not replace a continued-SFT control using the same scene observations and correct answers, the same trainable parameters, and a matched training budget. Without that control, the paper establishes the value of its margin design more convincingly than its broader claim of an advantage over imitation-based spatial supervision.
2.The “25% preference data” result does not establish the claimed overall data-efficiency advantage. Table 4 labels the comparison in terms of data percentage, while Section 4.3 describes it as a preference-pair draw budget. These quantities are not interchangeable: reducing draws need not mean restricting the available dataset to the same fraction of distinct examples. Moreover, Equation (16) trains on three supervision sources, but Table 4 reports only the preference-pair budget. The camera groups and intervention blocks also expose the model to images and ground-truth relations. The main text does not specify how those additional exposures scale across the reported percentages, nor whether the “DPO” baseline retains the same direct and intervention supervision. The current result demonstrates performance at a lower stated pair-draw budget; it does not establish a fourfold reduction in total training data or supervision. The efficiency claim needs either complete resource accounting or narrower wording.
Questions for authors
1.What performance is obtained by continuing SFT on the same constructed observations and correct responses, with the same full-parameter training configuration and a matched update budget as GaugeDPO? This is the missing control for attributing the improvement over GaugeSFT to the learning objective rather than the combined change in data and training setup.
2.For every percentage in Table 4, how many distinct preference pairs, camera-group observations, and intervention-block observations are used, and how many optimizer updates are performed? Does “DPO” mean a pair-only objective or the zero-offset variant retaining direct and intervention losses? Please provide an accounting that makes the comparison unambiguous.
3.On the held-out intervention blocks used in Tables 6–7, does the improvement over pair-plus-direct supervision persist under ordinary answer generation rather than canonical candidate scoring? The theoretical guarantee is explicitly tied to the finite-candidate predictor, so the same-block comparison would establish whether the intervention benefit transfers to the model’s usual generation mode.
5 comments
Response to Reviewer DHGb (2/2): Resource accounting for Tab. 4 and results under ordinary generation
[W2] & [Q2]: "Complete resource accounting for Table 4, and what does the 'DPO' baseline denote?"
[A2]: The full accounting is below. It shows that camera-group and intervention-block exposure already scaled with the pair budget, so the reduction was never confined to pairs. We add two stricter settings as well.
(1) Full accounting. Per budget we report pair draws, distinct pairs actually visited, camera-group and intervention-block draws, and optimizer updates:
| Budget | Pair draws | Distinct pairs | Group draws | Block draws | Updates |
|---|---|---|---|---|---|
| 10% | 3,000 | 2,861 | 1,498 | 376 | 94 |
| 25% | 7,500 | 6,629 | 3,742 | 940 | 235 |
| 50% | 15,000 | 11,812 | 7,489 | 1,876 | 469 |
| 100% | 30,000 | 18,957 | 14,982 | 3,752 | 938 |
Block exposure is tied to the update count (four draws per update, App. D.3) and each sampled block contributes its states as camera groups, so all three supervision sources scale together by construction. Distinct-pair counts are the realized numbers under sampling with replacement from the 30,000-pair pool.
(2) The "DPO" row. This row is the zero-offset variant of Tab. 3(b): it retains identical direct supervision () and intervention fitting (), differing from GaugeDPO only in the pair offset. Tab. 4 therefore already isolates the measured margin at each budget, a stricter comparison than pair-only DPO, which we also report for completeness:
| Data | Pair-only DPO | Zero-offset (prev. "DPO") | GaugeDPO |
|---|---|---|---|
| 25% | 45.0 | 50.0 | 52.5 |
| 100% | 47.5 | 50.0 | 62.5 |
(MSMU distance; the full five-metric table is in the revision.) We will rename the row "Zero-offset ( retained)", add the pair-only row, and define the ⋆ superscript in the caption.
(3) Two stricter settings. To address the concern that fewer draws need not restrict available data, we add (a) a setting scaling group and block draws by the same factor as pairs, and (b) one restricting the training pool to 25% of the distinct pairs, so the model never sees the other 75%:
| Setting at 25% | MSMU dist. | QSpat | SRGPT Quan. | 3DSR | BLINK |
|---|---|---|---|---|---|
| Full-budget zero-offset control (100%) | 50.0 | 49.5 | 35.4 | 50.3 | 51.9 |
| GaugeDPO, pair draws only | 52.5 | 51.5 | 36.1 | 51.0 | 52.6 |
| GaugeDPO, all three sources scaled | 52.5 | 51.5 | 35.9 | 50.8 | 52.4 |
| GaugeDPO, pool restricted to 25% | 52.5 | 50.5 | 35.6 | 50.6 | 52.1 |
The conclusion holds under all three accountings, including the strictest reading in which three quarters of the corpus is withheld. MSMU distance is quantized in 2.5-point steps, so the three settings share a grid point there and the finer-grained metrics separate them. We will give this accounting in the Tab. 4 caption and Sec. 4.3, and name the budget explicitly as a pair-draw budget.
[Q3]: "Does the intervention benefit persist under ordinary answer generation rather than canonical candidate scoring?"
[A3]: Yes, and we now show it on the same blocks.
(1) Existing generation-mode evidence. Tab. 8 is produced by ordinary generation with post-hoc parsing: its invalid counts in App. D.2 are parse failures, which cannot arise under candidate scoring. There the ablation — exactly the "Pair + direct" objective of Tab. 6 — reaches 60.0% accurate agreement with 12.5% wrong, against 63.0% and 11.5% for the full objective, matching the ordering of Tab. 6.
(2) Requested same-block comparison. We evaluate the identical 600 held-out blocks of Tabs. 6–7 under greedy decoding with post-hoc parsing:
| Split | Pair + direct (gen.) | GaugeDPO (gen.) | Pair + direct (canon.) | GaugeDPO (canon.) |
|---|---|---|---|---|
| Familiar | 66.0 | 73.5 | 68.0 | 76.0 |
| Unseen magnitudes | 57.5 | 67.5 | 60.0 | 71.0 |
| Unseen combinations | 49.0 | 60.5 | 52.0 | 63.0 |
| Overall | 57.5 | 67.2 | 60.0 | 70.0 |
Both objectives lose 2.0–3.5 points under generation, and the +9.7 overall gain is within 0.3 of the canonical +10.0, with GaugeDPO ahead on every split.
(3) Why the two modes agree. The canonical sequence-MAP answer and the generated answer coincide on 94.5% of these blocks for GaugeDPO and 89.0% for Pair + direct, so the predictor to which Prop. 2 applies and ordinary generation agree on the large majority of units. We will update these contents in the next version of our manuscript.
We hope our responses help address the concerns, and thanks again for the thoughtful feedback. We look forward to further discussion with you!
Thank you for the experiments. The exposure-matched continued-SFT control largely addresses W1/Q1, and the same-block generation results answer Q3. But I still have two questions about Table 4 in paper:
- A2(1) states that camera-group and intervention-block exposure already scaled with the pair budget. However, the original 25% results match the “pair draws only” row in A2(3), not the “all three sources scaled” row. Which setting produced the original Table 4 results, and what differs between these two rows?
- Does the restricted-pool experiment exclude 75% of the preference pairs only, or also restrict the camera-group and intervention-block pools? Can the auxiliary pools still contain scenes or images associated with excluded pairs? If only the preference-pair pool is restricted, the claim that “three quarters of the corpus is withheld” is not established.
I think the attribution in A1 also needs correction. The LoRA comparison changes the objective, training data, and training duration together, so its 10-point gain cannot be assigned to the objective alone. The matched full-parameter control supports a benefit from the objective, but does not establish the claimed 10/5 breakdown of the total gain.
Response to Reviewer DHGb (1/2): Matched controls isolating the learning objective
Thank you for your feedback! We appreciate the reviewer's careful reading, and in particular the recognition that "the shuffled control is particularly informative" and that our controls "provide evidence for the proposed intervention formulation". Below are our responses point by point.
[W1] & [Q1]: "The SFT comparison confounds the objective with capacity, data and training setup. What does full-parameter continued SFT achieve under a matched update budget?"
[A1]: We ran the requested control, and additionally the reverse control that varies the objective at fixed capacity, so that capacity and objective are each isolated. Both identify the objective as the dominant factor.
(1) Full-parameter continued SFT on Gauge-50K (the requested control). We continue SFT from the same initialization on the chosen completions of the same pairs, with full-parameter tuning, an unfrozen vision encoder, and the same 938 updates, batch size, seeds and evaluation protocol as GaugeDPO. We also report an exposure-matched variant that additionally imitates the canonical answers of the same camera groups and intervention blocks, so every image and ground-truth relation seen by GaugeDPO is also seen by imitation.
(2) Reverse control, GaugeDPO under the SFT configuration. We also train GaugeDPO with LoRA (r=128) and a frozen vision encoder, i.e. exactly the capacity regime of GaugeSFT:
(3) Attribution. The two directions agree on the same decomposition of the +15.0 MSMU gain. At matched full-parameter capacity, the strongest imitation control we could construct still trails GaugeDPO by 10.0 points, even though it sees every image and ground-truth relation the objective sees. At matched LoRA capacity, the objective alone moves 47.5 to 57.5, the same +10.0. Both routes therefore assign 10.0 of the 15.0 points to the objective and 5.0 to capacity and data. This is consistent with the matched evidence in the paper: Tab. 3(b) holds the full-parameter setup, the Gauge data and both auxiliary losses fixed while zeroing the offset, improving on GaugeSFT by only +2.5 (MSMU) and +4.0 (QSpatial), whereas measured margins add a further +12.5 and +14.9 on top of that row. Imitation on these observations is therefore not a close substitute for the measured-margin objective.
We will add both controls to our next version of manuscript.
Please see the next comment for the rest of our response (2/2).