- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
It has been proposed that visual reasoning accuracy improves if a judge LLM separately scores the accuracies of three stages of reasoning: perception, decomposition, and inference. This paper proposes that the weight given to each stage should be adapted based on the utility of that stage in discriminating between accurate versus inaccurate rollouts in recent groups.
Critical strengths
- The use of utility-dependent weighting is shown to improve performance relative to uniform or fixed weighting
- Proposition 1 shows that the proposed leaky integration of utility-dependent weighting (Eq. 4) guarantees a graceful degradation: the proposed allocation converges to uniform as the fraction of answer-discriminative groups goes to zero
Critical weaknesses
- The proposed closed-loop harness decides between four actions (rollback, diversity, stop, and continue) that are widely used in most harnesses. Table 2 shows that, on some corpora, using this harness with PRM achieves results as good as the results of proposed CALIPER.
- The proposed utility-dependent weighting has an impact on the timing of the harness' diversify action, but its impact is not separately measured using an ablation.
- Figure 5 shows that many of the benefits of the proposed training algorithm come from steps that are pretty well-established and have been used by other authors, e.g., the independent judge of each segment, the physics prior, the solved-group gate.
- Notation is often unnecessarily confusing. For example, why introduce a new letter (d) for |S| in Proposition 1, only to use that letter in exactly one location in Eq. (7)?
Questions for authors
- Can you measure the impact of utility-dependent weighting on the harness?
2 comments
Follow-up on remaining concerns
As the discussion period draws to a close, we would appreciate knowing whether our responses have addressed your concerns. If so, we would be grateful if you could reconsider your rating accordingly. We would also be happy to address any remaining questions during the remaining discussion period.
Authors' Rebuttal
Thank you for your feedback and valuable comments. Below is our point-by-point response.
W1 [Standard harness actions and the Harness PRM row of Tab. 2]
CALIPER's harness rules depend on its utility estimator, and the harness alone does not reproduce CALIPER's gains. We address this from two aspects below:
(1) Estimator-Dependent Rules. The paper presents the harness as an audit of the utility estimate rather than as a new control scheme. Eq. 5 conditions DIVERSIFY and STOP on the allocation drift δk, the number nkM of answer-discriminative groups and the utility separation Δuk, and ROLLBACK restores the policy, allocation and rollout temperature of the best validation round. With fixed weights, δk=0 in every round, so the drift condition no longer indicates whether the allocation has stabilized. Prop. 1 shows that the pooled target converges to the uniform allocation as p↓0, whereas CALIPER's mixed-group target does not depend on p. CALIPER therefore keeps its stage contrast when answer-discriminative groups become scarce, and the update threshold Nmin holds the current allocation until enough samples accumulate. nkM detects this scarcity and DIVERSIFY counteracts it. The same four actions lower PRM on every backbone (Tab. 1) and add at most 0.54 points under frozen allocation, but add 1.13 to 3.24 points under adaptive allocation (Table R1 in our response to W2), so their value depends on the estimator state on which they act.
(2) Gains beyond the Harness. Harness PRM uses "the separate harness protocol" (Sec. 5.1), whereas the CALIPER row of Tab. 2 is trained without it. Harness PRM matches CALIPER only on RealWorldQA, scores lower on BLINK, EmbSpatial and ViewSpatial, and its mean is 1.50 points lower. On the eight benchmarks of Tab. 1, harness training lowers PRM and raises CALIPER on all three backbones. Under the same harness, CALIPER exceeds PRM by 1.50, 3.77 and 2.06 points.
W2 & Q1 [Impact of utility-dependent weighting on the harness]
Utility-dependent weighting determines both the timing of the harness interventions and the gain they provide. We address this from two aspects below:
(1) Factorial Ablation. We train adaptive and frozen allocation each with and without the harness rules (Table R1).
Table R1. Eight-benchmark mean (%) under the harness protocol.
With at most 30 optimizer steps in both settings, the rules add 1.13 to 3.24 points under adaptive allocation but only 0.23 to 0.54 points under frozen allocation. The largest gain, 3.24 points on Qwen2.5-VL-7B, comes mostly from ROLLBACK, whose removal alone costs 2.17 points.
(2) Timing of DIVERSIFY. We repeat CALIPER and frozen allocation with three seeds and track the decision Dk of every round. Under frozen allocation, DIVERSIFY is triggered in 7 of 9 runs at round 2.3 on average. Under adaptive allocation, it is triggered in 5 of 9 runs at round 3.4 on average, after the allocation stabilizes. Under frozen allocation, the extra groups cannot change the fixed weights, consistent with the small rule gain in Table R1. We have added Table R1 and these timing statistics to App. D.
W3 [Contribution of established components in Fig. 5]
The frozen-allocation control isolates the gain from weight adaptation, and the component ablations identify measured utility as the largest single contributor on every backbone. The control preserves the judge and reference access (App. B.1), and CALIPER exceeds it by 1.93, 1.03 and 1.59 points (Fig. 5b), so components shared with the control cannot account for this margin.
In Fig. 5a, removing measured utility lowers the mean by 2.66, 2.45 and 1.73 points, whereas the largest drop from removing independent judging, the length penalty, the physics prior or the solved-group gate is 2.24, 1.71 and 1.16 points. Replacing measured utility with failure-only or dispersion-only heuristics also lowers the mean on every backbone, by up to 3.64 and 2.09 points. The gain therefore requires the regression estimate of stage utility rather than any adaptive weighting, and the supporting components are not among the paper's stated contributions.
W4 [Notation]
Prop. 1 and Eq. 7 now use ∣S∣ instead of d, which remains only in App. C, where the proof uses it throughout. The harness thresholds, which shared ϵ with the allocation offset, now have distinct symbols, and App. B adds a notation table.
We hope this addresses your concerns and are happy to discuss further.